I have been doing some testing and have been able to confirm that using Django with PostgreSQL and PGBouncer that it does not automatically reconnect on losing it's connection. To be honest, I am not sure if this is a bug or if this is by design. If it is a bug I will happily report it, if not I would like some explanation of why and how to get around it other than another custom back end.
I did these tests fairly easily by doing the following on Django 1.8.4 and on Django 1.8.6:
>>>Model.objects.all().count()
24
# Restart postgres with `sudo service postgres restart`
>>>Model.objects.all().count()
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 318, in count
return self.query.get_count(using=self.db)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 466, in get_count
number = obj.get_aggregation(using, ['__count'])['__count']
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 447, in get_aggregation
result = compiler.execute_sql(SINGLE)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
It does not matter how long I wait after restarting PostgreSQL to run the query I still get the same results. I see that in Django there is code to run a Select 1 to check for a connection but that doesn't seem to be working. Back in Django 1.5.x we had written a custom PostgreSQL back end to do this same thing but removed it since it seemed Django had it built in when we upgraded. Is this a bug?
Edit 11/6/2015: In Django, the PostgreSQL backend implements the is_usable function that does a SELECT 1 on the database. However, I am not able to find any usages of is_usable other than in close_if_unusable_or_obsolete but I couldn't find any usages of that anywhere. I would think it would be used within some database wrapper that catches exceptions and retries/reconnects based on is_usable.
The code mentioned above is found at django/db/backends/postgresql_psychopg2/base.py in Django 1.8.
Edit 2 11/6/2015: Ok, I wrote my own custom wrapper for the database and just overrode ensure_connection method to try and reconnect to the database when the connection is lost. However, on the first attempt to hit the database for a session I get this other gnarly traceback. But if I immediately query again it works. If I wrap what I am doing in a try/except: pass block it all seems to be working fine but cannot really tell if it will cause problems later on. Here is the traceback and code.
>>> c = Model.objects.all().count()
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 318, in count
return self.query.get_count(using=self.db)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 466, in get_count
number = obj.get_aggregation(using, ['__count'])['__count']
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 447, in get_aggregation
result = compiler.execute_sql(SINGLE)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 838, in execute_sql
cursor = self.connection.cursor()
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 164, in cursor
cursor = self.make_cursor(self._cursor())
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 135, in _cursor
self.ensure_connection()
File "/usr/lib/python2.7/dist-packages/custom/db/backends/postgresql_psycopg2/base.py", line 73, in ensure_connection
self._reconnect()
File "/usr/lib/python2.7/dist-packages/custom/db/backends/postgresql_psycopg2/base.py", line 63, in _reconnect
self.connect()
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 120, in connect
self.set_autocommit(self.settings_dict['AUTOCOMMIT'])
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 295, in set_autocommit
self._set_autocommit(autocommit)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 218, in _set_autocommit
self.connection.autocommit = autocommit
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 97, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 218, in _set_autocommit
self.connection.autocommit = autocommit
ProgrammingError: autocommit cannot be used inside a transaction
Now for the code:
from django.db.backends.postgresql_psycopg2.base import DatabaseError, \
IntegrityError, DatabaseWrapper as PostgresWrapper
class DatabaseWrapper(PostgresWrapper):
def _reconnect(self):
try:
self.connect()
except (DatabaseError, OperationalError):
pass
def ensure_connection(self):
"""
Guarantees that a connection to the database is established.
"""
if self.connection is None:
with self.wrap_database_errors:
self._reconnect()
else:
try:
self.connection.cursor().execute('SELECT 1')
except (DatabaseError, OperationalError):
self._reconnect()
I think I figured it out...finally. Not entirely sure what was wrong with my first approach but this one seems to be working a ton better.
class DatabaseWrapper(PostgresWrapper):
def _cursor(self):
if self.connection is not None:
if not self.is_usable():
self.connection.close()
self.connection = None
return super(DatabaseWrapper, self)._cursor()
Edit: Ended up open sourcing this. I am not sure if it is 100% needed, but it is working correctly after restarting the Postgres service on the server. You can find it on pypi as django-postgreconnect, and on GitHub: https://github.com/EccoTheFlintstone/django-postgreconnect.
Related
One of my python modules(which is not directly associated with Django application, same project though) is supposed to use Django models(that belong to original Django application) for convenient DB queries via Django-ORM.
However, this only works well until there is a DB or site-to-site connection issue.
Reproduced 100% times with following steps:
Create models.
Import models by setting up Django in any python module
os.environ['DJANGO_SETTINGS_MODULE'] = 'ServiceName.settings'
django.setup()
from ServiceName.models import ModelName
Use models via django ORM to fetch some data from DB on service startup(lets call it step 1) and later on by some triggers(lets call it step 2). In my case data is fetched once on startup, and all the subsequent times on certain Kafka event.
e.g.
query_results = ModelName.objects.filter(your_filter=some_value)
4.Shutdown PostgreSQL instance after step one, but before step 2
Not sure if related to the issue, but we are using pg-bouncer as a connection point.
5.Start PostgreSQL, trigger step2.
First attemp to fetch the data fails with the following trace:
Traceback (most recent call last):
File "/opt/services/_init_.py", line 113, in observe_events
if callback(event):
File "services_logparser_core.py", line 193, in process_event
r_handler.update_rules()
File "services_logparser_core.py", line 48, in update_rules
self.rules_list = self._fetch_enabled_rules()
File "services_logparser_core.py", line 69, in _fetch_enabled_rules
for sr in enabled_search_rules:
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 280, in _iter_
self._fetch_all()
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 1324, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 51, in _iter_
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.8/dist-packages/django/db/utils.py", line 90, in _exit_
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.8/dist-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.OperationalError: SSL SYSCALL error: EOF detected
All the subsequent attempts to fetch the data fail with:
Traceback (most recent call last):
File "/opt/services/__init__.py", line 113, in observe_events
if callback(event):
File "services_logparser_core.py", line 193, in process_event
r_handler.update_rules()
File "services_logparser_core.py", line 48, in update_rules
self.rules_list = self._fetch_enabled_rules()
File "services_logparser_core.py", line 69, in _fetch_enabled_rules
for sr in enabled_search_rules:
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 280, in __iter__
self._fetch_all()
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 1324, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/usr/local/lib/python3.8/dist-packages/django/db/models/query.py", line 51, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py", line 1173, in execute_sql
cursor = self.connection.cursor()
File "/usr/local/lib/python3.8/dist-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/django/db/backends/base/base.py", line 259, in cursor
return self._cursor()
File "/usr/local/lib/python3.8/dist-packages/django/db/backends/base/base.py", line 237, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/usr/local/lib/python3.8/dist-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.8/dist-packages/django/db/backends/base/base.py", line 237, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "/usr/local/lib/python3.8/dist-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/django/db/backends/postgresql/base.py", line 236, in create_cursor
cursor = self.connection.cursor()
django.db.utils.InterfaceError: connection already closed
Application restart fixes the problem.
I don't set up a connection to the db manually anywhere(I only setup DB credentials), all the connections are handled by Django ORM.
Is there any django/psycopg setting can be configured inside the project so that new connection is established each time db query is required?
Django==3.2.6
from django import db
And
db.close_old_connections()
before
query_results = ModelName.objects.filter(your_filter=some_value)
Did the trick.
close_old_connections() method does not just close old connections, but also check if current connection is still usable. In case connection returns an error it will be closed. New connection opens automatically.
I am trying to reflect an existing MS Access database into a new model. To link the database to SQLAlchemy, I am using
engine = create_engine("access+pyodbc://#db-dns")
Since the database already exists, I am following the basic use scheme from this article: https://docs.sqlalchemy.org/en/14/orm/extensions/automap.html
# reflect the table
Base.prepare(engine, reflect=True)
# mapped classes are now created with names by default
# matching that of the table name
Personen = Base.classes.ref_personen #here, 'ref_personen' is a table from my database
session = Session(engine)
When I try to access the table using Personen.query.all() I get an AttributeError (error log see below).
EDIT: To raise the error, I dont even have to query anything. I assume that the error is already caused at this line: Personen = Base.classes.ref_personen
Often, the reason for this error is due to not hanving defined a primary key in the external database. However, this is not the case for me. In my MS Access DB rel_personen table, a primary key is defined.
Thanks, I appreciate any help on this topic!
(appenv4) C:\Users\Code\webapps4>flask run
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
c:\users\code\webapps4\appenv4\lib\site-packages\sqlalchemy\engine\reflection.py:919: SAWarning: The Access ODBC driver does not support the ODBC "SQLPrimaryKeys" function. get_pk_constraint() is returning an empty list.
pk_cons = self.get_pk_constraint(
c:\users\code\webapps4\appenv4\lib\site-packages\sqlalchemy\engine\reflection.py:947: SAWarning: The Access ODBC driver does not support the ODBC "SQLForeignKeys" function. get_foreign_keys() is returning an empty list.
fkeys = self.get_foreign_keys(
Traceback (most recent call last):
File "c:\users\code\webapps4\appenv4\lib\site-packages\sqlalchemy\util\_collections.py", line 186, in __getattr__
return self._data[key]
KeyError: 'ref_personen'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\anaconda3\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\anaconda3\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\Code\webapps4\appenv4\Scripts\flask.exe\__main__.py", line 7, in <module>
File "c:\users\code\webapps4\appenv4\lib\site-packages\flask\cli.py", line 990, in main
cli.main(args=sys.argv[1:])
File "c:\users\code\webapps4\appenv4\lib\site-packages\flask\cli.py", line 596, in main
return super().main(*args, **kwargs)
File "c:\users\code\webapps4\appenv4\lib\site-packages\click\core.py", line 1062, in main
rv = self.invoke(ctx)
File "c:\users\code\webapps4\appenv4\lib\site-packages\click\core.py", line 1668, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\code\webapps4\appenv4\lib\site-packages\click\core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\users\code\webapps4\appenv4\lib\site-packages\click\core.py", line 763, in invoke
return __callback(*args, **kwargs)
File "c:\users\code\webapps4\appenv4\lib\site-packages\click\decorators.py", line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "c:\users\code\webapps4\appenv4\lib\site-packages\click\core.py", line 763, in invoke
return __callback(*args, **kwargs)
File "c:\users\code\webapps4\appenv4\lib\site-packages\flask\cli.py", line 845, in run_command
app = DispatchingApp(info.load_app, use_eager_loading=eager_loading)
File "c:\users\code\webapps4\appenv4\lib\site-packages\flask\cli.py", line 321, in __init__
self._load_unlocked()
File "c:\users\code\webapps4\appenv4\lib\site-packages\flask\cli.py", line 346, in _load_unlocked
self._app = rv = self.loader()
File "c:\users\code\webapps4\appenv4\lib\site-packages\flask\cli.py", line 406, in load_app
app = locate_app(self, import_name, None, raise_if_not_found=False)
File "c:\users\code\webapps4\appenv4\lib\site-packages\flask\cli.py", line 273, in locate_app
return find_best_app(script_info, module)
File "c:\users\code\webapps4\appenv4\lib\site-packages\flask\cli.py", line 68, in find_best_app
app = call_factory(script_info, app_factory)
File "c:\users\code\webapps4\appenv4\lib\site-packages\flask\cli.py", line 119, in call_factory
return app_factory(*args, **kwargs)
File "C:\Users\Code\webapps4\app\__init__.py", line 19, in create_app
from .zkl import zkl as zkl_bp
File "C:\Users\Code\webapps4\app\zkl\__init__.py", line 6, in <module>
from . import zkl_forms, zkl_views
File "C:\Users\Code\webapps4\app\zkl\zkl_views.py", line 3, in <module>
from .zkl_database import db_session, init_db
File "C:\Users\Code\webapps4\app\zkl\zkl_database.py", line 12, in <module>
Personen = Base.classes.ref_personen
File "c:\users\code\webapps4\appenv4\lib\site-packages\sqlalchemy\util\_collections.py", line 188, in __getattr__
raise AttributeError(key)
AttributeError: ref_personen
Update: sqlalchemy-access version 1.1.0 now supports reflection of Primary Key (PK) and Foreign Key (FK) constraints.
(previous answer)
As noted by one of the warning messages in your stack trace:
SAWarning: The Access ODBC driver does not support the ODBC "SQLPrimaryKeys" function. get_pk_constraint() is returning an empty list.
This is a known issue with the Access ODBC driver. It is also discussed here
https://github.com/gordthompson/sqlalchemy-access/issues/9
and here
pyodbc - read primary keys from MS Access (MDB) database
I'm trying to use Apscheduler with a postgresql db via an asyncpg connection. I thought it would working, because asyncpg supports sqlalchemy ref. But yeah, it isn't working. And to make it even worst, I don't understand the error message, so I have not even a guess what to google for.
import asyncio
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore
def simple_job():
print('This was an easy job!')
scheduler = AsyncIOScheduler()
jobstore = SQLAlchemyJobStore(url='postgresql+asyncpg://user:password#localhost:5432/public')
scheduler.add_jobstore(jobstore)
# schedule a simple job
scheduler.add_job(simple_job, 'cron', second='15', id='heartbeat',
coalesce=True, misfire_grace_time=5, replace_existing=True)
scheduler.start()
Versions:
python 3.7
APScheduler==3.7.0
asyncpg==0.22.0
SQLAlchemy==1.4.3
Error Message and traceback:
Traceback (most recent call last):
File "C:/Users/d/PycharmProjects/teamutils/utils/automation.py", line 320, in <module>
scheduler.start()
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\apscheduler\schedulers\asyncio.py", line 45, in start
super(AsyncIOScheduler, self).start(paused)
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\apscheduler\schedulers\base.py", line 163, in start
store.start(self, alias)
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\apscheduler\jobstores\sqlalchemy.py", line 68, in start
self.jobs_t.create(self.engine, True)
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\sql\schema.py", line 940, in create
bind._run_ddl_visitor(ddl.SchemaGenerator, self, checkfirst=checkfirst)
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\engine\base.py", line 2979, in _run_ddl_visitor
with self.begin() as conn:
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\engine\base.py", line 2895, in begin
conn = self.connect(close_with_result=close_with_result)
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\engine\base.py", line 3067, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\engine\base.py", line 91, in __init__
else engine.raw_connection()
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\engine\base.py", line 3146, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\engine\base.py", line 3113, in _wrap_pool_connect
return fn()
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\pool\base.py", line 301, in connect
return _ConnectionFairy._checkout(self)
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\pool\base.py", line 755, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\pool\base.py", line 419, in checkout
rec = pool._do_get()
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\pool\impl.py", line 145, in _do_get
self._dec_overflow()
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\util\langhelpers.py", line 72, in __exit__
with_traceback=exc_tb,
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\util\compat.py", line 198, in raise_
raise exception
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\pool\impl.py", line 142, in _do_get
return self._create_connection()
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\pool\base.py", line 247, in _create_connection
return _ConnectionRecord(self)
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\pool\base.py", line 362, in __init__
self.__connect(first_connect_check=True)
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\pool\base.py", line 605, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\util\langhelpers.py", line 72, in __exit__
with_traceback=exc_tb,
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\util\compat.py", line 198, in raise_
raise exception
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\pool\base.py", line 599, in __connect
connection = pool._invoke_creator(self)
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\engine\create.py", line 578, in connect
return dialect.connect(*cargs, **cparams)
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\engine\default.py", line 548, in connect
return self.dbapi.connect(*cargs, **cparams)
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\dialects\postgresql\asyncpg.py", line 744, in connect
await_only(self.asyncpg.connect(*arg, **kw)),
File "C:\Users\d\PycharmProjects\teamutils\venv\lib\site-packages\sqlalchemy\util\_concurrency_py3k.py", line 48, in await_only
"greenlet_spawn has not been called; can't call await_() here. "
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_() here. Was IO attempted in an unexpected place? (Background on this error at: http://sqlalche.me/e/14/xd2s)
sys:1: RuntimeWarning: coroutine 'connect' was never awaited
I looked up the provided link, but not getting smart of it. So it would be nice, if somebody can tell me what is going on, so I can search for a solution by my own. (a solution would okay too, of course xD)
Sorry for this "open" question, but my understanding is so bad, that I dont know what to ask for.
I think problem is in ApScheduler.
What is happening is that scheduler.start() will attempt to create the job table in your database. But since your database url is specified as +asyncpg and there is no async coroutine running (ie: async def) when ApScheduler tries to create the table. Hence the "coroutine 'connect' was never awaited" error.
After reading the ApScheduler code, I think "integrates with asyncio" is a little misleading - specifically the scheduler can run asyncio, but the JobStore itself has no provision for an asyncio database connection.
You can get it working by removing +asyncpg in the connection url used with ApScheduler.
Note it would still be possible to use async db calls within job functions with a separate asyncpg connection.
I set up a database with mongolab where the db had the same name as the collection. I decided I didnt like this deleted it and made a better naming scheme. I added a new user/password to for the new db and tried to authenticated. It keeps failing. Not sure why. I have double check my credentials and they are correct. I have checked the URI that mongolab provides and that is correct also. This code worked well for the first db.
this is my login code
def __init__(self, user_id, password, database, collection):
# mongodb://<dbuser>:<dbpassword>#ds017205.mlab.com:17205/words
mongodb_uri = "mongodb://" + user_id + ":" + password + "#ds017205.mlab.com:17205/" + database
client = pymongo.MongoClient(mongodb_uri)
db = client[database]
self.collection = db[collection] # is declared in class
this generates the following errors
Error
Traceback (most recent call last):
File "C:\Users\Austin\PycharmProjects\Words\test_mongodb.py", line 15, in testUpdate
results = self.mongodb_obj.update(test)
File "C:\Users\Austin\PycharmProjects\Words\mongodb.py", line 38, in update
upsert=True
File "C:\Python33\lib\site-packages\pymongo\collection.py", line 2235, in update
with self._socket_for_writes() as sock_info:
File "C:\Python33\lib\contextlib.py", line 48, in __enter__
return next(self.gen)
File "C:\Python33\lib\site-packages\pymongo\mongo_client.py", line 718, in _get_socket
with server.get_socket(self.__all_credentials) as sock_info:
File "C:\Python33\lib\contextlib.py", line 48, in __enter__
return next(self.gen)
File "C:\Python33\lib\site-packages\pymongo\server.py", line 152, in get_socket
with self.pool.get_socket(all_credentials, checkout) as sock_info:
File "C:\Python33\lib\contextlib.py", line 48, in __enter__
return next(self.gen)
File "C:\Python33\lib\site-packages\pymongo\pool.py", line 541, in get_socket
sock_info.check_auth(all_credentials)
File "C:\Python33\lib\site-packages\pymongo\pool.py", line 306, in check_auth
auth.authenticate(credentials, self)
File "C:\Python33\lib\site-packages\pymongo\auth.py", line 436, in authenticate
auth_func(credentials, sock_info)
File "C:\Python33\lib\site-packages\pymongo\auth.py", line 416, in _authenticate_default
return _authenticate_scram_sha1(credentials, sock_info)
File "C:\Python33\lib\site-packages\pymongo\auth.py", line 216, in _authenticate_scram_sha1
res = sock_info.command(source, cmd)
File "C:\Python33\lib\site-packages\pymongo\pool.py", line 213, in command
read_concern)
File "C:\Python33\lib\site-packages\pymongo\network.py", line 99, in command
helpers._check_command_response(response_doc, None, allowable_errors)
File "C:\Python33\lib\site-packages\pymongo\helpers.py", line 196, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: Authentication failed.
Help with this would be appreciated. Thanks
Nothing wrong with the code... the db user set up entered the incorrect password the same twice in the setup box so it was accepted then when the code was trying to use the actual correct answer it wouldn't work for obvious reasons. thanks all for looking
I have a cron job that downloads data from MySQL every night at 3. I can test this connection and download and it works. Occasionally the download partially fails. (partial download) If I try and re-run the py script, it barks. Duplicate entry error for key 2.
I want to be able to run a script and erase only the entries for the previous night so I can rerun the script that populates the db. There are three other tables tied to this one. What is django going to do if is create a SQL script that deletes yesterdays records? Will it automatically delete necessary additions to the other tables, or should I do this in the script also?
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.6/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.6/site-packages/django/core/management/__init__.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.6/site-packages/django/core/management/base.py", line 196, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.6/site-packages/django/core/management/base.py", line 232, in execute
output = self.handle(*args, **options)
File "/usr/local/django/grease/greaseboard/management/commands/import_patients.py", line 27, in handle
mrn = row.MRN,
File "/usr/local/lib/python2.6/site-packages/django/db/models/manager.py", line 134, in get_or_create
return self.get_query_set().get_or_create(**kwargs)
File "/usr/local/lib/python2.6/site-packages/django/db/models/query.py", line 449, in get_or_create
obj.save(force_insert=True, using=self.db)
File "/usr/local/lib/python2.6/site-packages/django/db/models/base.py", line 463, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "/usr/local/lib/python2.6/site-packages/django/db/models/base.py", line 551, in save_base
result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
File "/usr/local/lib/python2.6/site-packages/django/db/models/manager.py", line 203, in _insert
return insert_query(self.model, objs, fields, **kwargs)
File "/usr/local/lib/python2.6/site-packages/django/db/models/query.py", line 1576, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/local/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 910, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.6/site-packages/django/db/backends/util.py", line 40, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.6/site-packages/django/db/backends/mysql/base.py", line 114, in execute
return self.cursor.execute(query, args)
File "/usr/local/lib/python2.6/site-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/local/lib/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
django.db.utils.IntegrityError: (1062, "Duplicate entry '000xxxxxxxx' for key 2")
No idea how large the work is, but for similar problem I used transactions alongside savepoints - https://docs.djangoproject.com/en/dev/topics/db/transactions/#savepoints
so given something like:
transaction.auto_commit(False)
try:
for raw_input in streaming_filelikeobject.readline():
product = do_work(raw_input)
MyTable(**product).save()
except SomeFileIOError:
transaction.rollback()
else:
transaction.commit()
Another idea would be to have a batch_id column and assign that on the start of each batch.
For very large data sets, you could use something like Memcache/Redis for managing inventory.
transaction.auto_commit(False)
try:
for raw_input in streaming_filelikeobject.readline():
product = do_work(raw_input)
if redis_conn.sadd("my_input_set", product['some_unique_id']):
MyTable(**product).save()
except SomeFileIOError:
transaction.rollback()
else:
transaction.commit()
.sadd() is a Redis command that returns true if an element doesn't exist already in a redis set.
Please note I am typing this stuff of the top of my head so method django transaction method signatures may not be authoritative.