I'm having some problems using SQLAlchemy in Pyramid. Although I can find examples of what I need, they're normally very short and lacking. So I've ended up with patchy code that barely makes any sense. So I'm hoping someone could give a fuller example of what I need to do.
I have 4 databases all with the same schema. I want to be able to work on them from one Pyramid app, sometimes listing all "orders" from all 4 databases, sometimes just listing all "orders" from "site1". As the schemas are the same, I also use the same model classes for the databases.
I've tried it with both sqlahelper and plain SQLAlchemy with no luck. The code below uses sqlahelper but I'm happy to use anything that works:
__init__.py
site1_eng = engine_from_config(settings, prefix='site1.')
site2_eng = engine_from_config(settings, prefix='site2.')
site3_eng = engine_from_config(settings, prefix='site3.')
sqlahelper.add_engine(site1_eng, 'site1_eng')
sqlahelper.add_engine(site2_eng, 'site2_eng')
views.py
def site_orders(request):
site = request.matchdict['site']
db_eng = sqlahelper.get_engine(("%s_eng" % (site)))
conn = db_eng.connect()
dbsession = sqlahelper.get_session()
dbsession.configure(bind=conn)
orders = dbsession.query(Order).order_by(Order.cdate.desc())[:100]
return dict(orders=orders, pagetitle=(site+" Orders"))
What Happens?
Well I'd hoped it would switch database depending on the URL and it does! However, it seems completely random as to which is chooses. So /orders/site1/ will sometimes go to site2 database and sometimes site3. Refreshing will often switch the database it chooses each time. Same for other URL's.
Its almost as if the session isn't binding to the database and its picking whichever happens to be in the session at the time? That may not make sense - my understanding of SQLAlchemy isn't great.
Really hope someone can help as it all hinges on the ability to quickly and easily switch databases within a view and at the moment it seems completely impossible to control it.
NOTE:
I did originally try following and altering the Pyramid SQLA+URL Dispatcher tutorial which used:
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
But I removed that when finding sqlahelper. If I should be using it let me know.
Configuring and connection for each request seems like a lot of work to me. I would create four session handlers in my model module and just choose from them.
Example:
models/__init__.py
DBSession1 = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
DBSession2 = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
DBSession3 = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
DBSession4 = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
metadata1 = MetaData()
metadata2 = MetaData()
metadata3 = MetaData()
metadata4 = MetaData()
def initialize_sql(engines, drop_db=False):
DBSession1.configure(bind=engine[0])
DBSession2.configure(bind=engine[1])
DBSession3.configure(bind=engine[2])
DBSession4.configure(bind=engine[3])
metadata1.bind = engine[0]
metadata2.bind = engine[1]
metadata3.bind = engine[2]
metadata4.bind = engine[3]
and then in your view:
from mypackage.models import DBSession1, DBSession2, DBSession3, DBSession4
def site_orders(request)
site = request.matchdict['site']
dbsession = globals().get("DBSession%d" % site)
orders = dbsession.query(Order).order_by(Order.cdate.desc())[:100]
return dict(orders=orders, pagetitle=(site+" Orders"))
You can set engine to the sqlalchemy session directly
Example for listing all "orders" from all 4 databases:
def site_orders(request):
...
orders = []
for engine in engines:
dbsession.bind = engine
orders += dbsession.query(Order).order_by(Order.cdate.desc())[:100]
return dict(orders=orders, pagetitle=(site+" Orders"))
Related
The firestore db has a structure like this:
collection: users
documents with id an user_id
each document has a sub-collection: games
each sub-collection has documents with id a game_id
My problem is that I can't loop over the first collection (---> for user in ref_users:) even my firestore db is already full with data organized like I said. Can someone help me understanding how can I retrieve the infos of these documents? Thanks a lot.
from flask import Flask, render_template
from google.cloud import firestore
db = firestore.Client()
#app.route('/', methods=['GET'])
def index():
ref_users = db.collection(u'users').get()
games = []
for user in ref_users:
print(f'{user.id} => {user.to_dict()}')
ref_game = db.collection(u'users').document(f'{user.id}').collection(u'games').get()
for rg in ref_game:
print(f'{rg.id} => {rg.to_dict()}')
tmp = rg.to_dict()
tmp['game_id'] = rg.id
tmp['user_id'] = user.id
games.append(tmp)
return render_template('game_list.html', title='Game list', games=games)
Implemented this code on my side with some minor changes and its working fine. So I think that you are maybe searching for solution to decrease computational complexity.
I think that collection_group might be helpful. Please check general documentation and Python reference.
I have created code for the same result as the sample (just index part). It has one loop less:
def index():
games = []
ref_game = db.collection_group(u'games').get()
for rg in ref_game:
print(f'{rg.id} => {rg.to_dict()}')
tmp = rg.to_dict()
tmp['game_id'] = rg.id
tmp['user_id'] = rg.reference.parent.parent.id
games.append(tmp)
return str(games)
I would use more sophisticated name for this subcollection like userGames or something. This is because of the fact that collection_group works on every collection with this name in whole database. If your system with grow, there is a chance that you will use the same name games in some other path not related to this logic. Than it will be included to this collection_group and may cause unexpected issues.
I recently found out about sqlalchemy in Python. I'd like to use it for data science rather than website applications.
I've been reading about it and I like that you can translate the sql queries into Python.
The main thing that I am confused about that I'm doing is:
Since I'm reading data from an already well established schema, I wish I didn't have to create the corresponding models myself.
I am able to get around that reading the metadata for the table and then just querying the tables and columns.
The problem is when I want to join to other tables, this metadata reading is taking too long each time, so I'm wondering if it makes sense to pickle cache it in an object, or if there's another built in method for that.
Edit: Include code.
Noticed that the waiting time was due to an error in the loading function, rather than how to use the engine. Still leaving the code in case people comment something useful. Cheers.
The code I'm using is the following:
def reflect_engine(engine, update):
store = f'cache/meta_{engine.logging_name}.pkl'
if update or not os.path.isfile(store):
meta = alq.MetaData()
meta.reflect(bind=engine)
with open(store, "wb") as opened:
pkl.dump(meta, opened)
else:
with open(store, "r") as opened:
meta = pkl.load(opened)
return meta
def begin_session(engine):
session = alq.orm.sessionmaker(bind=engine)
return session()
Then I use the metadata object to get my queries...
def get_some_cars(engine, metadata):
session = begin_session(engine)
Cars = metadata.tables['Cars']
Makes = metadata.tables['CarManufacturers']
cars_cols = [ getattr(Cars.c, each_one) for each_one in [
'car_id',
'car_selling_status',
'car_purchased_date',
'car_purchase_price_car']] + [
Makes.c.car_manufacturer_name]
statuses = {
'selling' : ['AVAILABLE','RESERVED'],
'physical' : ['ATOURLOCATION'] }
inventory_conditions = alq.and_(
Cars.c.purchase_channel == "Inspection",
Cars.c.car_selling_status.in_( statuses['selling' ]),
Cars.c.car_physical_status.in_(statuses['physical']),)
the_query = ( session.query(*cars_cols).
join(Makes, Cars.c.car_manufacturer_id == Makes.c.car_manufacturer_id).
filter(inventory_conditions).
statement )
the_inventory = pd.read_sql(the_query, engine)
return the_inventory
I was trying to implement a dashboard by following the instructions of https://github.com/talpor/django-dashing/ using django-dashing.
So far I have successfully customised my widget and displayed with some random data on my own web server, while I have no clue where to start if I'd like to pull some real data from DB(MySQL) and display. (like where to do the DB connecting,..etc)
Could anyone show me steps I should follow to implement it?
If it's still relevant, you can start by connecting to the database with sqlalchemy.
import sqlalchemy as sq
from sqlalchemy.engine import url as sq_url
db_connect_url = sq_url.URL(
drivername='mysql+mysqldb',
username=DB_username,
password=DB_password,
host=DB_hostname,
port=DB_port,
database=DB_name,
)
engine = sq.create_engine(db_connect_url)
From there you can manipulate the data by checking the available methods on engine. What I normally do is use pandas in situations like these.
import pandas as pd
df = pd.read_sql_table(table_name, engine)
I also had to do this recently.... I managed to get it sorted - but it is a little too clunky.
I created a Separate CherryPy REST Api in a separate Project. The Entry point looks like
#cherrpy.expose
def web_api_to_call(self, table,value):
#Do SQL Query
return str(sql_table_value)
Then in Django Created a New app, then created a Widget.py. Inside the Widget.py I wrote something like this.
import requests
class webquery(NumberWidget):
classparams=[("widget1","web_api_to_call","table","values"),
("widget2","web_api_to_call","table2","values2"),
("widget3","web_api_to_call","table3","values3")]
def myget(self):
for tup in self.classparams:
if tup[0]==type(self).__name__:
url=tup[1]
table=tup[2]
value=tup[3]
url = "http://127.0.0.1:8000/"+url
# Do Web Call Error Checking Omitted
return requests.get(url,params={"table":table,"values":value)}).text()
def get_value(self):
#Override default
return self.my_get()
#Now create new Widgets as per the static definition at the top
class widget1(web query):
id=1
class widget2(web query):
id=1
class widget3(web query):
id=1
You now just add your new widgets - as you normally would in the urls.py and then in the dashing-config.js and you are done.
Hope this assists someone.
I am trying to add memcache to my webapp deployed on GAE, and to do this I am using memcache.Client() to prevent damage from any racing conditions:
from google.appengine.api import memcache
client = memcache.Client()
class BlogFront(BlogHandler):
def get(self):
global client
val = client.gets(FRONT_PAGE_KEY)
posts = list()
if val is not None:
posts = list(val)
else:
posts = db.GqlQuery("select * from Post order by created desc limit 10")
client.cas(FRONT_PAGE_KEY, list(posts))
self.render('front.html', posts = posts)
To test the problem I have a front page for a blog that displays the 10 most recent entries. If there is nothing in the cache, I hit the DB with a request, otherwise I just present the cached results to the user.
The problem is that no matter what I do, I always get val == None, thus meaning that I always hit the database with a useless request.
I have sifted through the documentation:
https://developers.google.com/appengine/docs/python/memcache/
https://developers.google.com/appengine/docs/python/memcache/clientclass
http://neopythonic.blogspot.pt/2011/08/compare-and-set-in-memcache.html
And it appears that I am doing everything correctly. What am I missing?
(PS: I am a python newb, if this is a retarded error, please bear with me xD )
from google.appengine.api import memcache
class BlogFront(BlogHandler):
def get(self):
client = memcache.Client()
client.gets(FRONT_PAGE_KEY)
client.cas(FRONT_PAGE_KEY, 'my content')
For a reason I cannot yet possible understand, the solution lies in having a gets right before having a cas call ...
I think I will stick with the memcache non-thread-safe version of the code for now ...
I suspect that the client.cas call is failing because there is no object. Perhaps client.cas only works to update and existing object (not to set a new object if there is none currently)? You might try client.add() (which will fail if an object already exists with the specified key, which I think is what you want?) instead of client.cas()
The test still writes to my MySQL database instead of a sqlite tempfile db. Why does this happen? Thanks!
Here's my code:
class UserTests(unittest.TestCase):
def setUp(self):
self.app = get_app()
#declare testing state
self.app.config["TESTING"] = True
self.db, self.app.config["DATABASE"] = tempfile.mkstemp()
#spawn test client
self.client = self.app.test_client()
#temp db
init_db()
def tearDown(self):
os.close(self.db)
os.unlink(self.app.config["DATABASE"])
def test_save_user(self):
#create test user with 3 friends
app_xs_token = get_app_access_token(APP_ID, APP_SECRET)
test_user = create_test_user(APP_ID, app_xs_token)
friend_1 = create_test_user(APP_ID, app_xs_token)
friend_2 = create_test_user(APP_ID, app_xs_token)
friend_3 = create_test_user(APP_ID, app_xs_token)
make_friend_connection(test_user["id"], friend_1["id"], test_user["access_token"], friend_1["access_token"])
make_friend_connection(test_user["id"], friend_2["id"], test_user["access_token"], friend_2["access_token"])
make_friend_connection(test_user["id"], friend_3["id"], test_user["access_token"], friend_3["access_token"])
save_user(test_user["access_token"])
This line might be the problem:
self.db, self.app.config["DATABASE"] = tempfile.mkstemp()
print out the values of self.db and self.app.config["DATABASE"] and makes sure they are what you expect them to be.
You probably want to investigate where your config self.app.config["DATABASE"] is referenced in your database code.
The Flask example code usually does a lot of work when the module is first imported. This tends to break things when you try to dynamically change values at run time, because by then its too late.
You probably will need to use an application factory so your app isn't built before the test code can run. Also, the app factory pattern implies you are using the Blueprint interface instead of the direct app reference, which is acquired using a circular import in the example code.