Develop Reference

sqlalchemy, can I get around creating the table classes?

I recently found out about sqlalchemy in Python. I'd like to use it for data science rather than website applications.
I've been reading about it and I like that you can translate the sql queries into Python.
The main thing that I am confused about that I'm doing is:
Since I'm reading data from an already well established schema, I wish I didn't have to create the corresponding models myself.
I am able to get around that reading the metadata for the table and then just querying the tables and columns.
The problem is when I want to join to other tables, this metadata reading is taking too long each time, so I'm wondering if it makes sense to pickle cache it in an object, or if there's another built in method for that.
Edit: Include code.
Noticed that the waiting time was due to an error in the loading function, rather than how to use the engine. Still leaving the code in case people comment something useful. Cheers.
The code I'm using is the following:
def reflect_engine(engine, update):
store = f'cache/meta_{engine.logging_name}.pkl'
if update or not os.path.isfile(store):
meta = alq.MetaData()
meta.reflect(bind=engine)
with open(store, "wb") as opened:
pkl.dump(meta, opened)
else:
with open(store, "r") as opened:
meta = pkl.load(opened)
return meta
def begin_session(engine):
session = alq.orm.sessionmaker(bind=engine)
return session()
Then I use the metadata object to get my queries...
def get_some_cars(engine, metadata):
session = begin_session(engine)
Cars = metadata.tables['Cars']
Makes = metadata.tables['CarManufacturers']
cars_cols = [ getattr(Cars.c, each_one) for each_one in [
'car_id',
'car_selling_status',
'car_purchased_date',
'car_purchase_price_car']] + [
Makes.c.car_manufacturer_name]
statuses = {
'selling' : ['AVAILABLE','RESERVED'],
'physical' : ['ATOURLOCATION'] }
inventory_conditions = alq.and_(
Cars.c.purchase_channel == "Inspection",
Cars.c.car_selling_status.in_( statuses['selling' ]),
Cars.c.car_physical_status.in_(statuses['physical']),)
the_query = ( session.query(*cars_cols).
join(Makes, Cars.c.car_manufacturer_id == Makes.c.car_manufacturer_id).
filter(inventory_conditions).
statement )
the_inventory = pd.read_sql(the_query, engine)
return the_inventory

Updating a custom field using ASANA Python API

I'm trying to update the values of custom fields in my Asana list. I'm using the Official Python client library for the Asana API v1.
My code currently looks like this;
project = "Example Project"
keyword = "Example Task"
print "Logging into ASANA"
api_key = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
client = asana.Client.basic_auth(api_key)
me = client.users.me()
all_projects = next(workspace for workspace in me['workspaces'])
projects = client.projects.find_by_workspace(all_projects['id'])
for project in projects:
if 'Example Project' not in project['name']:
continue
print "Project found."
print "\t"+project['name']
print
tasks = client.tasks.find_by_project(project['id'], {"opt_fields":"this.name,custom_fields"}, iterator_type=None)
for task in tasks:
if keyword in task['name']:
print "Task found:"
print "\t"+str(task)
print
for custom_field in task['custom_fields']:
custom_field['text_value'] = "New Data!"
print client.tasks.update(task['id'], {'data':task})
But when I run the code, the task doesn't update. The return of print client.tasks.update returns all the details of the task, but the custom field has not been updated.

I think the problem is that our API is not symmetrical with respect to custom fields... which I kind of find to be a bummer; it can be a real gotcha in cases like this. Rather than being able to set the value of a custom field within the block of values as you're doing above, which is intuitive, you have to set them with a key:value dictionary-like setup of custom_field_id:new_value - not as intuitive, unfortunately. So above, where you have
for custom_field in task['custom_fields']:
custom_field['text_value'] = "New Data!"
I think you'd have to do something like this:
new_custom_fields = {}
for custom_field in task['custom_fields']:
new_custom_fields[custom_field['id']] = "New Data!"
task['custom_fields'] = new_custom_fields
The goal is to generate JSON for the POST request that looks something like
{
"data": {
"custom_fields":{
"12345678":"New Data!"
}
}
}
As a further note, the value should be the new text string if you have a text custom field, a number if it's a number custom field, and the ID of the enum_options choice (take a look at the third example under this header on our documentation site) if it's an enum custom field.

Thanks to Matt, I got to the solution.
new_custom_fields = {}
for custom_field in task['custom_fields']:
new_custom_fields[custom_field['id']] = "New Data!"
print client.tasks.update(task['id'], {'custom_fields':new_custom_fields})
There were two problems in my original code, the first was that I was trying to treat the API symmetrically and this was identified and solved by Matt. The second was that I was trying to update in an incorrect format. Note the difference between client.tasks.update in my original and updated code.

Search document in MongoDB by _id using Flask-pymongo extension

I am baffled with the following problem.
I am using Flask, flask-pymongo extension, mongodb version v2.2.0-rc0, pdfile version 4.5
This is my route:
#app.route("/check/<id>")
def check(id):
doc=conn.db.msg.find_one({'_id':id})
return render_template('base.html',name=doc)
the id is a valid _id from a document in the msg collection, but ALWAYS return None.
I have tried:
pasing the ObjectId(id) but returns errortype: ObjectId not callable
pasing the id as str(id) returns None
Any thoughts?
UPDATE:
this how the full url looks like:
http://127.0.0.1:8000/check/5030b628895b800de1a9a792
UPDATE2:
I found a similar question with (answer) for ruby. Not sure how I can translate it to python, what sort imports/modules do I need?
How can I retrieve a document by _id?
UPDATE3:
I tried:
import bson
#app.route("/check/<id>")
def check(id):
id2="'"+id+"'"
doc=conn.db.msg.find_one({'_id':bson.objectid(id2) })
return render_template('base.html',name=doc)
but I get TypeError: 'module' object is not callable (it doesnt work with id either)
when I reached 1500 I will suggest a frustation tag :-S
UPDATE4:
Finally I got it up & running!
here it is my solution:
import bson
#app.route("/check/<id>")
def check(id):
doc=conn.db.msg.find_one({'_id':bson.ObjectId(oid=str(id))})
return render_template('base.html',name=doc)

You might also want to try using ObjectId from the bson.objectid module, like so:
from bson.objectid import ObjectId
In that case, you won't need to provide the oid kwarg. You'll just do something like this:
db_conn.msg.find_one({'_id': ObjectId(my_oid)})

web.py - How to turn result of db.query(sql) into list of dict?

I'm new to both Python and web.py (which I am currently using) so please bear with me.
In the official document:
import web
db = web.database(dbn='postgres', db='mydata', user='dbuser', pw='')
results = db.query("SELECT COUNT(*) AS total_users FROM users")
print results[0].total_users # -> prints number of entries in 'users' table
Looks like the result of the query is a list of dictionaries {total_user: num} right?
My situation is very similar: running a SELECT against the database, hoping to get a list of key:value data.
In models.py:
def get_items:
return self.db.query("SELECT title FROM news")
In code.py:
items = model.get_items
return render.list(items)
In templates/list.html:
$def with (items)
$for item in items:
<p>$item.title</p>
However, the code triggers an error as "'tuple' object has no attribute 'title'".
What have I done wrong? Thanks in advance for your help.

It will trigger, because you are just selecting only the title attribute from news, which obvioulsy will give tuple like (a,b,c,d) as result.
if you want a object as result,do something like this
self.db.query("SELECT * FROM news")

Store a list of dictionaries in GAE

I have a list of about 20 objects and for each object I return a list of 10 dictionaries.
I am trying to store the list of 10 dictionaries for each object in the list on GAE; I do not think I am writing the code correctly to store this information to GAE.
Here is what I have:
Before my main request handler I have this class:
class Tw(db.Model):
tags = db.ListProperty()
ip = db.StringProperty()
In my main request handler I have the following:
for city in lst_of_cities: # this is the list of 20 objects
dict_info = hw12.twitter(city) # this is the function to get the list of 10 dictionaries for each object in the list
datastore = Tw() # this is the class defined for db.model
datastore.tags.append(dict_info) #
datastore.ip = self.request.remote_addr
datastore.put()
data = Data.gql("") #data entities we need to fetch
I am not sure if this code is write at all. If anyone could please help it would be much appreciated.

Welcome to Stack Overflow!
I see a few issues:
Dictionaries are not supported value types for App Engine properties.
You're only storing the last entity; the rest are discarded.
You're using a ListProperty, but instead of appending each element of dict_info, you're doing a single append of the entire list.
Since you can't store a raw dictionary inside a property, you need to serialize it to some other format, like JSON or pickle. Here's a revised example using pickle:
from google.appengine.ext import db
import pickle
class Tw(db.Model):
tags = db.BlobProperty()
ip = db.StringProperty()
entities = []
for city in lst_of_cities:
dict_info = hw12.twitter(city)
entity = Tw()
entity.tags = db.Blob(pickle.dumps(dict_info))
entity.ip = self.request.remote_addr
entities.append(entity)
db.put(entities)
When you fetch the entity later, you can retrieve your list of dictionaries with pickle.loads(entity.tags).

When I deal with data types that are not directly supported by Google App Engine like dictionaries or custom data type, I usually adopt the handy PickleProperty.
from google.appengine.ext import db
import pickle
class PickleProperty(db.Property):
def get_value_for_datastore(self, model_instance):
value = getattr(model_instance, self.name, None)
return pickle.dumps(value)
def make_value_from_datastore(self, value):
return pickle.loads(value)
Once declared the PickleProperty class in your commons.py module, you can use it to store your custom data with something like this:
from google.appengine.ext import db
from commons import PickleProperty
class Tw(db.Model):
tags = PickleProperty()
ip = db.StringProperty()
entities = []
for city in lst_of_cities:
dict_info = hw12.twitter(city)
entity = Tw()
entity.tags = dict_info
entity.ip = self.request.remote_addr
entities.append(entity)
db.put(entities)
To retrieve the data back go with:
entity.tags

Since this was written, the App Engine has pushed out their experimental "ndb" Python database model, which contains in particular the JsonProperty, something that pretty well directly implements what you want.
Now, you need to be running the Python 2.7 version of the App Engine, which is still not quite ready for production, but it all seems pretty stable these days, GvR himself seems to be writing a lot of the code which bodes well for the code quality, and I'm intending to use this in production sometime this year...