I've updated django from 1.6 to 1.8.3. I create test models in test setUp method in unit tests, something like that
class MyTestCase(LiveServerTestCase):
reset_sequences = True
def setUp(self):
self.my_model = models.MyModel.objects.create(name='test)
And I have code in the application, which relies on the primary key == 1. I've noted, that sequences aren't actually reseted. In each next test the pk is greater, that in previous.
This works ok in django 1.6, but after migration to 1.8 problems appears.
Should I reset sequence manually?
P.s. I know about fixtures, but my models are more complicated and for me it's easier to create models in the code.
The problem was in sqlite3. The tests have been run with other settings file, where sqlite3 is configured as database.
The django checks, if the database supports sequences:
# django.test.testcases.py:809
def _reset_sequences(self, db_name):
conn = connections[db_name]
if conn.features.supports_sequence_reset:
sql_list = conn.ops.sequence_reset_by_name_sql(
no_style(), conn.introspection.sequence_list())
# ....
So I've switched test settings to the postgresql and now it works normally
Related
We had a problem with a website which uses Django. Each time we upgrade Django, if a user is logged in with two or more different browsers, and then they login again from one browser - they are automatically logged out from all other sessions (browsers). Since we upgraded Django to new major versions about 5 times in the last year, this caused us a headache. We don't want to force users to have to login again and again between sessions. How can we solve this problem?
We checked and found out that this problem is caused due to a change in PBKDF2PasswordHasher.iterations between versions of Django. Each time we upgrade Django to a new major version (such as from 3.0 to 3.1), PBKDF2PasswordHasher.iterations changes. This causes the user's hashed password to be calculated again the next time the user logs in, which forces the user to remain logged out in all other sessions. I even created a ticket with Django's tracking system.
There are two options to fix this issue. First, we can patch class PBKDF2PasswordHasher to keep the number of iterations constant, and also update def must_update:
from django.contrib.auth.hashers import PBKDF2PasswordHasher
def patch():
def must_update(self, encoded):
# Update the stored password only if the iterations diff is at least 250,000.
algorithm, iterations, salt, hash = encoded.split('$', 3)
iterations_diff = abs(self.iterations - int(iterations))
return ((int(iterations) != self.iterations) and (iterations_diff >= 250000))
PBKDF2PasswordHasher.iterations = 180000 # Django 3.0.x
PBKDF2PasswordHasher.must_update = must_update
And then in our base AppConfig class:
class SpeedyCoreBaseConfig(AppConfig):
name = 'speedy.core.base'
verbose_name = _("Speedy Core Base App")
label = 'base'
def ready(self):
locale_patches.patch() # Another patch
session_patches.patch() # This patch
Or, you can inherit a new class from PBKDF2PasswordHasher, change iterations and def must_update, and use your new class in the settings (PASSWORD_HASHERS). We used the first option, although it might be better to use the second option (inherit a new class) from a software engineering perspective. They both work.
I have a simple library application. In order to force 3 actions to commit as one action, and rollback if any of the actions fail, I made the following code changes:
In settings.py:
AUTOCOMMIT=False
In forms.py
from django.db import IntegrityError, transaction
class CreateLoan(forms.Form):
#Fields...
def save(self):
id_book = form.cleaned_data.get('id_book', None)
id_customer = form.cleaned_data.get('id_customer', None)
start_date = form.cleaned_data.get('start_date', None)
book = Book.objects.get(id=id_book)
customer = Customer.objects.get(id=id_customer)
new_return = Return(
book=book
start_date=start_date)
txn=Loan_Txn(
customer=customer,
book=book,
start_date=start_date
)
try
with transaction.atomic():
book.update(status="ON_LOAN")
new_return.save(force_insert=True)
txn.save(force_insert=True)
except IntegrityError:
raise forms.ValidationError("Something occured. Please try again")
Am I still missing anything with regards to this? I'm using Django 1.9 with Python 3.4.3 and the database is MySQL.
You're using transaction.atomic() correctly (including putting the try ... except outside the transaction) but you should definitely not be setting AUTOCOMMIT = False.
As the documentation states, you set that system-wide setting to False when you want to "disable Django’s transaction management"—but that's clearly not what you want to do, since you're using transaction.atomic()! More from the documentation:
If you do this, Django won’t enable autocommit, and won’t perform any commits. You’ll get the regular behavior of the underlying database library.
This requires you to commit explicitly every transaction, even those started by Django or by third-party libraries. Thus, this is best used in situations where you want to run your own transaction-controlling middleware or do something really strange.
So just don't do that. Django will of course disable autocommit for that atomic block and re-enable it when the block finishes.
sometimes we have to do a Model instance.save() regardless if some field changed, just for security and fast development.
how expensive is this with django ORM?
signals are always sent?
any SQL query is executed?
I tested with django debug toolbar to do 10 .save() in different points where anything in the model has changed, and the log does not register sql queries.
other way to test it or some article?
thank you in advance.
Im not entirely sure how you application handles this.
But i ran a small test:
a = Blog.objects.get(pk=1)
for b in range(1, 100):
a.save()
This gave me a result of:
87.04 ms (201 queries)
Be ware as well that a save will do two queries:
SELECT ••• FROM `fun_blog` WHERE `fun_blog`.`id` = 1 LIMIT 1
UPDATE `fun_blog` SET `title` = 'This is my testtitle', `body` = 'This is a testbody' WHERE `fun_blog`.`id` = 1
I've got a web-application which is built with Pyramid/SQLAlchemy/Postgresql and allows users to manage some data, and that data is almost completely independent for different users. Say, Alice visits alice.domain.com and is able to upload pictures and documents, and Bob visits bob.domain.com and is also able to upload pictures and documents. Alice never sees anything created by Bob and vice versa (this is a simplified example, there may be a lot of data in multiple tables really, but the idea is the same).
Now, the most straightforward option to organize the data in the DB backend is to use a single database, where each table (pictures and documents) has user_id field, so, basically, to get all Alice's pictures, I can do something like
user_id = _figure_out_user_id_from_domain_name(request)
pictures = session.query(Picture).filter(Picture.user_id==user_id).all()
This is all easy and simple, however there are some disadvantages
I need to remember to always use additional filter condition when making queries, otherwise Alice may see Bob's pictures;
If there are many users the tables may grow huge
It may be tricky to split the web application between multiple machines
So I'm thinking it would be really nice to somehow split the data per-user. I can think of two approaches:
Have separate tables for Alice's and Bob's pictures and documents within the same database (Postgres' Schemas seems to be a correct approach to use in this case):
documents_alice
documents_bob
pictures_alice
pictures_bob
and then, using some dark magic, "route" all queries to one or to the other table according to the current request's domain:
_use_dark_magic_to_configure_sqlalchemy('alice.domain.com')
pictures = session.query(Picture).all() # selects all Alice's pictures from "pictures_alice" table
...
_use_dark_magic_to_configure_sqlalchemy('bob.domain.com')
pictures = session.query(Picture).all() # selects all Bob's pictures from "pictures_bob" table
Use a separate database for each user:
- database_alice
- pictures
- documents
- database_bob
- pictures
- documents
which seems like the cleanest solution, but I'm not sure if multiple database connections would require much more RAM and other resources, limiting the number of possible "tenants".
So, the question is, does it all make sense? If yes, how do I configure SQLAlchemy to either modify the table names dynamically on each HTTP request (for option 1) or to maintain a pool of connections to different databases and use the correct connection for each request (for option 2)?
After pondering on jd's answer I was able to achieve the same result for postgresql 9.2, sqlalchemy 0.8, and flask 0.9 framework:
from sqlalchemy import event
from sqlalchemy.pool import Pool
#event.listens_for(Pool, 'checkout')
def on_pool_checkout(dbapi_conn, connection_rec, connection_proxy):
tenant_id = session.get('tenant_id')
cursor = dbapi_conn.cursor()
if tenant_id is None:
cursor.execute("SET search_path TO public, shared;")
else:
cursor.execute("SET search_path TO t" + str(tenant_id) + ", shared;")
dbapi_conn.commit()
cursor.close()
Ok, I've ended up with modifying search_path in the beginning of every request, using Pyramid's NewRequest event:
from pyramid import events
def on_new_request(event):
schema_name = _figire_out_schema_name_from_request(event.request)
DBSession.execute("SET search_path TO %s" % schema_name)
def app(global_config, **settings):
""" This function returns a WSGI application.
It is usually called by the PasteDeploy framework during
``paster serve``.
"""
....
config.add_subscriber(on_new_request, events.NewRequest)
return config.make_wsgi_app()
Works really well, as long as you leave transaction management to Pyramid (i.e. do not commit/roll-back transactions manually, letting Pyramid to do that at the end of request) - which is ok as committing transactions manually is not a good approach anyway.
What works very well for me it to set the search path at the connection pool level, rather than in the session. This example uses Flask and its thread local proxies to pass the schema name so you'll have to change schema = current_schema._get_current_object() and the try block around it.
from sqlalchemy.interfaces import PoolListener
class SearchPathSetter(PoolListener):
'''
Dynamically sets the search path on connections checked out from a pool.
'''
def __init__(self, search_path_tail='shared, public'):
self.search_path_tail = search_path_tail
#staticmethod
def quote_schema(dialect, schema):
return dialect.identifier_preparer.quote_schema(schema, False)
def checkout(self, dbapi_con, con_record, con_proxy):
try:
schema = current_schema._get_current_object()
except RuntimeError:
search_path = self.search_path_tail
else:
if schema:
search_path = self.quote_schema(con_proxy._pool._dialect, schema) + ', ' + self.search_path_tail
else:
search_path = self.search_path_tail
cursor = dbapi_con.cursor()
cursor.execute("SET search_path TO %s;" % search_path)
dbapi_con.commit()
cursor.close()
At engine creation time:
engine = create_engine(dsn, listeners=[SearchPathSetter()])
How can I connect to MS Access with SQLAlchemy? In their website, it says connection string is access+pyodbc. Does that mean that I need to have pyodbc for the connection? Since I am a newbie, please be gentle.
In theory this would be via create_engine("access:///some_odbc_dsn"), but the Access backend hasn't been in service at all since SQLAlchemy 0.5, and it's not clear how well it was working back then either (this is why it's noted as "development" at http://docs.sqlalchemy.org/en/latest/core/engines.html#supported-databases - "development" means, "a development version of the dialect exists, but is not yet usable"). There's just not enough interest/volunteers to keep this dialect running right now. (when/if it is, you'll see it at http://docs.sqlalchemy.org/en/latest/dialects/access.html).
Your best bet for Access right now would be to export the data into a SQLite database file (or of course some other database, though SQLite is file-based in a similar way at least), then use that.
Update, September 2019:
The sqlalchemy-access dialect has been resurrected. Details here.
Usage example:
engine = create_engine("access+pyodbc://#some_odbc_dsn")
I primarily needed read access and some simple queries. The latest version of sqlalchemy has the (broken) access back end modules, but it isn't registered as an entrypoint.
It needed a few fixups, but this worked for me:
def fixup_access():
import sqlalchemy.dialects.access.base
class FixedAccessDialect(sqlalchemy.dialects.access.base.AccessDialect):
def _check_unicode_returns(self, connection):
return True
def do_execute(self, cursor, statement, params, context=None, **kwargs):
if params == {}:
params = ()
super(sqlalchemy.dialects.access.base.AccessDialect, self).do_execute(cursor, statement, params, **kwargs)
class SomeObject(object):
pass
fixed_dialect_mod = SomeObject
fixed_dialect_mod.dialect = FixedAccessDialect
sqlalchemy.dialects.access.fix = fixed_dialect_mod
fixup_access()
ENGINE = sqlalchemy.create_engine('access+fix://admin#/%s'%(db_location))