I'm using Peewee as ORM and connect to a Postgres database (psycopg2) using the Playhouse extension db_url.connect. My URL is a vanilla postgresql://username:pass#host:port/dbname?options=... so not using pooling or anything advanced at the moment.
Some times when I call connect it hangs for a long time and doesn't come back. So I appended to my database URL the parameter &connect_timeout=3 meaning to try for at most 3 seconds and fail-fast with a timeout rather than hanging forever. However, I am not sure whether this argument is supported by Peewee/Playhouse/Psycopg2 ... can anyone confirm?
Furthermore, where can I find all the URL parameters supported by Peewee/Playhouse/Psycopg2?
The psycopg2 doc links in turn to the libpq list of supported parameters:
https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS
connect_timeout is supported by both peewee and psycopg2:
>>> from playhouse.db_url import *
>>> db = connect('postgresql://../peewee_test?connect_timeout=3')
>>> conn = db.connection()
>>> conn.get_dsn_parameters()
{'user': 'postgres',
'passfile': '...',
'channel_binding': 'prefer',
'connect_timeout': '3', # Our connect timeout
'dbname': 'peewee_test',
'host': 'localhost',
'port': '5432',
...}
Peewee passes the parameters, including arbitrary ones like connect_timeout, back to the constructor of the DB-API Connection class.
I have a couple of services which query objects from the database.
Event.objects.filter
Connection.objects.filter
and other methods to retrieve different objects from MySQL database.
I come from JVM background, and I know that to setup a JDBC connection, you need a connector. A programmer can open a connection, query the database and close the connection. A programmer also can use Hibernate, which handles connection according to the configuration. Also it is possible to use pooled connections, so connections are not closed and removed, but stored in the pool untill they are needed.
However, I checked my teams Python Django code, and I did not find how db connection is configured. The only thing I got is which does not configure connections.
# Database
# https://docs.djangoproject.com/en/1.11/ref/settings/#databases
try:
import database_password
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': "mydb",
'USER': 'user',
'PASSWORD': database_password.password,
'HOST': '10.138.67.149',
'PORT': '3306'
}
}
Each thread maintains its own connection. See the docs for full details.
PostgreSQL + PgBouncer (connection pooler) + Django is a common setup. I'm not sure whether there's a similar connection pooler you could use with MySQL.
I use Python 3.4 from the Anaconda distribution. Within this distribution, I found the pymysql library to connect to an existing MySQL database, which is located on another computer.
import pymysql
config = {
'user': 'my_user',
'passwd': 'my_passwd',
'host': 'my_host',
'port': my_port
}
try:
cnx = pymysql.connect(**config)
except pymysql.err.OperationalError :
sys.exit("Invalid Input: Wrong username/database or password")
I now want to write test code for my application, in which I want to create a very small database at the setUp of every test case, preferably in memory. However, when I try this out of the blue with pymysql, it cannot make a connection.
def setUp(self):
config = {
'user': 'test_user',
'passwd': 'test_passwd',
'host': 'localhost'
}
cnx = pymysql.connect(**config)
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 61] Connection refused)")
I have been googling around, and found some things about SQLite and MySQLdb. I have the following questions:
Is sqlite3 or MySQLdb suitable for creating quickly a database in memory?
How do I install MySQLdb within the Anaconda package?
Is there an example of a test database, created in the setUp? Is this even a good idea?
I do not have a MySQL server running locally on my computer.
You can mock a mysql db using testing.mysqld (pip install testing.mysqld)
Due to some noisy error logs that crop up, I like this setup when testing:
import testing.mysqld
from sqlalchemy import create_engine
# prevent generating brand new db every time. Speeds up tests.
MYSQLD_FACTORY = testing.mysqld.MysqldFactory(cache_initialized_db=True, port=7531)
def tearDownModule():
"""Tear down databases after test script has run.
https://docs.python.org/3/library/unittest.html#setupclass-and-teardownclass
"""
MYSQLD_FACTORY.clear_cache()
class TestWhatever(unittest.TestCase):
#classmethod
def setUpClass(cls):
cls.mysql = MYSQLD_FACTORY()
cls.db_conn = create_engine(cls.mysql.url()).connect()
def setUp(self):
self.mysql.start()
self.db_conn.execute("""CREATE TABLE `foo` (blah)""")
def tearDown(self):
self.db_conn.execute("DROP TABLE foo")
#classmethod
def tearDownClass(cls):
cls.mysql.stop() # from source code we can see this kills the pid
def test_something(self):
# something useful
Both pymysql, MySQLdb, and sqlite will want a real database to connect too.
If you want just to test your code, you should just mock the pymysql module on the module you want to test, and use it accordingly
(in your test code: you can setup the mock object to return hardcoded results to predefined SQL statements)
Check the documentation on native Python mocking library at:
https://docs.python.org/3/library/unittest.mock.html
Or, for Python 2:
https://pypi.python.org/pypi/mock
I've just spent a week on the problems recorded in this question: https://stackoverflow.com/questions/21315427/why-does-the-ca-information-need-to-be-in-a-tuple-for-mysqldb
Have now boiled it down to one problem. Here's a script that connects to the MySQL server I have on Amazon RDS:
#! /usr/bin/env python
import MySQLdb
ssl = ({'ca': '/home/James/Downloads/mysql-ssl-ca-cert-copy.pem'},)
conn = MySQLdb.connect(host='db.doopdoop.eu-west-n.rds.amazonaws.com', user='user', passwd='pass', ssl=ssl)
cursor = conn.cursor()
cursor.execute("SHOW STATUS LIKE 'Ssl_Cipher'")
print cursor.fetchone()
This gives me back ('Ssl_cipher', ''), which I gather means the connection is not encrypted. Have also tried this using Django's manage.py shell function, and got the same result. Why, then, am I getting no exceptions? Data is flowing, certainly, but apparently the security is just being ignored. Any help on where I'm going wrong here would be appreciated.
I have tried updating MySQL-Python to 1.2.5 with no success.
Possible workaround for your issue: Use a default file. It will look something like this:
[mysql]
host = db.doopdoop.eu-west-n.rds.amazonaws.com
user = user
password = pass
ssl-ca = /home/James/Downloads/mysql-ssl-ca-cert-copy.pem
Then change your connect() call to:
conn = MySQLdb.connect(read_default_file=options_path)
where options_path is the path to the file above. This also keeps authentication data out of your code.
Django settings will look like this:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'OPTIONS': {
'read_default_file': '/path/to/my.cnf',
},
}
}
Ref: https://docs.djangoproject.com/en/dev/ref/databases/#connecting-to-the-database
Original: I have recently started getting MySQL OperationalErrors from some of my old code and cannot seem to trace back the problem. Since it was working before, I thought it may have been a software update that broke something. I am using python 2.7 with django runfcgi with nginx. Here is my original code:
views.py
DBNAME = "test"
DBIP = "localhost"
DBUSER = "django"
DBPASS = "password"
db = MySQLdb.connect(DBIP,DBUSER,DBPASS,DBNAME)
cursor = db.cursor()
def list(request):
statement = "SELECT item from table where selected = 1"
cursor.execute(statement)
results = cursor.fetchall()
I have tried the following, but it still does not work:
views.py
class DB:
conn = None
DBNAME = "test"
DBIP = "localhost"
DBUSER = "django"
DBPASS = "password"
def connect(self):
self.conn = MySQLdb.connect(DBIP,DBUSER,DBPASS,DBNAME)
def cursor(self):
try:
return self.conn.cursor()
except (AttributeError, MySQLdb.OperationalError):
self.connect()
return self.conn.cursor()
db = DB()
cursor = db.cursor()
def list(request):
cursor = db.cursor()
statement = "SELECT item from table where selected = 1"
cursor.execute(statement)
results = cursor.fetchall()
Currently, my only workaround is to do MySQLdb.connect() in each function that uses mysql. Also I noticed that when using django's manage.py runserver, I would not have this problem while nginx would throw these errors. I doubt that I am timing out with the connection because list() is being called within seconds of starting the server up. Were there any updates to the software I am using that would cause this to break/is there any fix for this?
Edit: I realized that I recently wrote a piece of middle-ware to daemonize a function and this was the cause of the problem. However, I cannot figure out why. Here is the code for the middle-ware
def process_request_handler(sender, **kwargs):
t = threading.Thread(target=dispatch.execute,
args=[kwargs['nodes'],kwargs['callback']],
kwargs={})
t.setDaemon(True)
t.start()
return
process_request.connect(process_request_handler)
Sometimes if you see "OperationalError: (2006, 'MySQL server has gone away')", it is because you are issuing a query that is too large. This can happen, for instance, if you're storing your sessions in MySQL, and you're trying to put something really big in the session. To fix the problem, you need to increase the value of the max_allowed_packet setting in MySQL.
The default value is 1048576.
So see the current value for the default, run the following SQL:
select ##max_allowed_packet;
To temporarily set a new value, run the following SQL:
set global max_allowed_packet=10485760;
To fix the problem more permanently, create a /etc/my.cnf file with at least the following:
[mysqld]
max_allowed_packet = 16M
After editing /etc/my.cnf, you'll need to restart MySQL or restart your machine if you don't know how.
As per the MySQL documentation, your error message is raised when the client can't send a question to the server, most likely because the server itself has closed the connection. In the most common case the server will close an idle connection after a (default) of 8 hours. This is configurable on the server side.
The MySQL documentation gives a number of other possible causes which might be worth looking into to see if they fit your situation.
An alternative to calling connect() in every function (which might end up needlessly creating new connections) would be to investigate using the ping() method on the connection object; this tests the connection with the option of attempting an automatic reconnect. I struggled to find some decent documentation for the ping() method online, but the answer to this question might help.
Note, automatically reconnecting can be dangerous when handling transactions as it appears the reconnect causes an implicit rollback (and appears to be the main reason why autoreconnect is not a feature of the MySQLdb implementation).
This might be due to DB connections getting copied in your child threads from the main thread. I faced the same error when using python's multiprocessing library to spawn different processes. The connection objects are copied between processes during forking and it leads to MySQL OperationalErrors when making DB calls in the child thread.
Here's a good reference to solve this: Django multiprocessing and database connections
For me this was happening in debug mode.
So I tried Persistent connections in debug mode, checkout the link: Django - Documentation - Databases - Persistent connections.
In settings:
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'dbname',
'USER': 'root',
'PASSWORD': 'root',
'HOST': 'localhost',
'PORT': '3306',
'CONN_MAX_AGE': None
},
Check if you are allowed to create mysql connection object in one thread and then use it in another.
If it's forbidden, use threading.Local for per-thread connections:
class Db(threading.local):
""" thread-local db object """
con = None
def __init__(self, ...options...):
super(Db, self).__init__()
self.con = MySQLdb.connect(...options...)
db1 = Db(...)
def test():
"""safe to run from any thread"""
cursor = db.con.cursor()
cursor.execute(...)
This error is mysterious because MySQL doesn't report why it disconnects, it just goes away.
It seems there are many causes of this kind of disconnection. One I just found is, if the query string too large, the server will disconnect. This probably relates to the max_allowed_packets setting.
I've been struggling with this issue too. I don't like the idea of increasing timeout on mysqlserver. Autoreconnect with CONNECTION_MAX_AGE doesn't work either as it was mentioned. Unfortunately I ended up with wrapping every method that queries the database like this
def do_db( callback, *arg, **args):
try:
return callback(*arg, **args)
except (OperationalError, InterfaceError) as e: # Connection has gone away, fiter it with message or error code if you could catch another errors
connection.close()
return callback(*arg, **args)
do_db(User.objects.get, id=123) # instead of User.objects.get(id=123)
As you can see I rather prefer catching the exception than pinging the database every time before querying it. Because catching an exception is a rare case. I would expect django to reconnect automatically but they seemed to refused that issue.
This error may occur when you try to use the connection after a time-consuming operation that doesn't go to the database. Since the connection is not used for some time, MySQL timeout is hit and the connection is silently dropped.
You can try calling close_old_connections() after the time-consuming non-DB operation so that a new connection is opened if the connection is unusable. Beware, do not use close_old_connections() if you have a transaction.
The most common issue regarding such warning, is the fact that your application has reached the wait_timeout value of MySQL.
I had the same problem with a Flask app.
Here's how I solved:
$ grep timeout /etc/mysql/mysql.conf.d/mysqld.cnf
# https://support.rackspace.com/how-to/how-to-change-the-mysql-timeout-on-a-server/
# wait = timeout for application session (tdm)
# inteactive = timeout for keyboard session (terminal)
# 7 days = 604800s / 4 hours = 14400s
wait_timeout = 604800
interactive_timeout = 14400
Observation: if you search for the variables via MySQL batch mode, the values will appear as it is. But If you perform SHOW VARIABLES LIKE 'wait%'; or SHOW VARIABLES LIKE 'interactive%';, the value configured for interactive_timeout, will appear to both variables, and I don't know why, but the fact is, that the values configured for each variable at '/etc/mysql/mysql.conf.d/mysqld.cnf', will be respected by MySQL.
How old is this code? Django has had databases defined in settings since at least .96. Only other thing I can think of is multi-db support, which changed things a bit, but even that was 1.1 or 1.2.
Even if you need a special DB for certain views, I think you'd probably be better off defining it in settings.
SQLAlchemy now has a great write-up on how you can use pinging to be pessimistic about your connection's freshness:
http://docs.sqlalchemy.org/en/latest/core/pooling.html#disconnect-handling-pessimistic
From there,
from sqlalchemy import exc
from sqlalchemy import event
from sqlalchemy.pool import Pool
#event.listens_for(Pool, "checkout")
def ping_connection(dbapi_connection, connection_record, connection_proxy):
cursor = dbapi_connection.cursor()
try:
cursor.execute("SELECT 1")
except:
# optional - dispose the whole pool
# instead of invalidating one at a time
# connection_proxy._pool.dispose()
# raise DisconnectionError - pool will try
# connecting again up to three times before raising.
raise exc.DisconnectionError()
cursor.close()
And a test to make sure the above works:
from sqlalchemy import create_engine
e = create_engine("mysql://scott:tiger#localhost/test", echo_pool=True)
c1 = e.connect()
c2 = e.connect()
c3 = e.connect()
c1.close()
c2.close()
c3.close()
# pool size is now three.
print "Restart the server"
raw_input()
for i in xrange(10):
c = e.connect()
print c.execute("select 1").fetchall()
c.close()
I had this problem and did not have the option to change my configuration. I finally figured out that the problem was occurring 49500 records in to my 50000-record loop, because that was the about the time I was trying again (after having tried a long time ago) to hit my second database.
So I changed my code so that every few thousand records, I touched the second database again (with a count() of a very small table), and that fixed it. No doubt "ping" or some other means of touching the database would work, as well.
Firstly, You should make sure the MySQL session & global enviroments wait_timeout and interactive_timeout values. And secondly Your client should try to reconnect to the server below those enviroments values.