Develop Reference

Locust.io Load Testing getting "Connection aborted BadStatusLine" Errors

I'm using Locust.io to load test an application. I will get a random error that I am unable to pinpoint the problem:
1)
ConnectionError(ProtocolError(\'Connection aborted.\', BadStatusLine("\'\'",)),)
2)
ConnectionError(ProtocolError('Connection aborted.', error(104, 'Connection reset by peer')),)
The first one is the one that happens a few times every 1,000,000 requests or so and seems to happen in groups where there will be 5-20 all at once and then it is fine. the second only happens every couple days or so.
The CPU and memory are well below all the servers max load for the database server, app server, and the machine running locust.io.
The servers are medium-sized Linode servers running Ubuntu 14.04. The app is Django and the database in PostgreSQL. I have already increased the maximum open file limit but am wondering if something else needs to be increased on the server that could be leading to the occasional errors.
From what I have been able to gather from searching the error is that it might have something to do with the python requests library.
-Any help would be greatly appreciated.

BadStatusLine is most likely a server side issue. See for example this answer https://stackoverflow.com/a/1767954/1591921 It could be some sort of flood/DoS protection on the server.
Connection reset by peer could also be any number of things, but it is most likely a server/network issue, not an issue on the loadgen side (perhaps connections are idle for too long, or there is a max connection age somewhere)
I dont think there are any general answers to this question, it all depends on your system under test.

How to get instance of database and close it? Tornado

I'm having trouble with MySQL timing out and going away after 8 hours. I am using google app engine as a host. My Python script uses the Tornado framework.
Right now I instantiate my MySQL db connection before any functions right at the top of the main server script. Once I deploy that, the clock starts ticking and 8 hours or so later, MySQL will go away and I will have to deploy my script again.
I haven't been using db.close() at all because I hear that restarting the database connection takes a long time. Is this true? Or is there a proper way to use db.close()?
One of my friends suggested I try getting the database instance and then closing it after each function.. is that recommended and where might I find some tutorials on that?
I'm mostly looking for resources here, but if someone wants to lay it out for me that would be awesome.
Thank you all in advance.

The connection is going away because of the wait_timeout session variable which
is the number of seconds the server waits for activity on a noninteractive connection
before closing it.
http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_wait_timeout
Good way is to close the connection each time and create a new one if you are not reusing the same connection so frequently, otherwise you can increase the value of wait_timeout

Establishing a connection to a MySQL database should be quite fast and it is certainly good practice to keep the connection open only for as long as you need it.
I am not certain why your connection should be non-responsive for 8 hours - have you tried checking your settings?
The correct command in Python is connection.close().

Recover from dropped connection in redis pub/sub

I am running client that is connecting to a redis db. The client is on a WiFi connection and will drop the connection at times. Unfortunately, when this happens, the program just keeps running without throwing any type of warning.
r = redis.StrictRedis(host=XX, password=YY...)
ps = r.pubsub()
ps.subscribe("12345")
for items in ps.listen():
if items['type'] == 'message':
data = items['data']
Ideally, what I am looking for is a catch an event when the connection is lost, try and reestablish the connection, do some error correcting, then get things back up and running. Should this be done in the python program? Should I have an external watchdog?

Unfortunately, one have to 'ping' Redis to check if it is available. If You try to put a value to Redis storage, it will raise an ConnectionError exception if connection is lost. But the listen() generator will not close automatically when connection is lost.
I think that hacking Redis' connection pool could help, give it a try.
P.S. In is very insecure to connect to redis in an untrusted network environment.

This is an old, old question but I linked one of my own questions to it and happened to run across it again. It turned out there was a bug in the redis library that caused the client to enter an infinite loop attempting to reconnect if it lost connection to the redis server. I debugged the issue and PR'd the change. it was merged a long time ago now. Once surfaced the maintainer also knew of a second location that had the same issue.
This problem shouldn't occur anymore.
To fully answer the question, I can't remember which error it is given the time since I fixed this but there is now a specific error returned you can catch and reconnect on.

Getting OperationalError: FATAL: sorry, too many clients already using psycopg2

I am getting the error OperationalError: FATAL: sorry, too many clients already when using psycopg2. I am calling the close method on my connection instance after I am done with it. I am not sure what could be causing this, it is my first experience with python and postgresql, but I have a few years experience with php, asp.net, mysql, and sql server.
EDIT: I am running this locally, if the connections are closing like they should be then I only have 1 connection open at a time. I did have a GUI open to the database but even closed I am getting this error. It is happening very shortly after I run my program. I have a function I call that returns a connection that is opened like:
psycopg2.connect(connectionString)
Thanks
Final Edit:
It was my mistake, I was recursively calling the same method on mistake that was opening the same method over and over. It has been a long day..

This error means what it says, there are too many clients connected to postgreSQL.
Questions you should ask yourself:
Are you the only one connected to this database?
Are you running a graphical IDE?
What method are you using to connect?
Are you testing queries at the same time that you running the code?
Any of these things could be the problem. If you are the admin, you can up the number of clients, but if a program is hanging it open, then that won't help for long.
There are many reasons why you could be having too many clients running at the same time.

Make sure your db connection command isn't in any kind of loop. I was getting the same error from my script until I moved my db.database() out of my programs repeating execution loop.

It simple means many clients are making transaction to PostgreSQL at same time.
I was running Postgis container and Django in different docker container. Hence for my case restarting both db and system container solved the problem.

Pylons error - 'MySQL server has gone away'

I'm using Pylons (a python framework) to serve a simple web application, but it seems to die from time to time, with this in the error log: (2006, 'MySQL server has gone away')
I did a bit of checking, and saw that this was because the connections to MySQL were not being renewed. This shouldn't be a problem though, because the sqlalchemy.pool_recycle in the config file should automatically keep it alive. The default was 3600, but I dialed it back to 1800 because of this problem. It helped a bit, but 3600 should be fine according to the docs. The errors still happen semi-regularly. I don't want to lower it too much though and DOS my own database :).
Maybe something in my MySQL config is goofy? Not sure where to look exactly.
Other relevant details:
Python 2.5
Pylons: 0.9.6.2 (w/ sql_alchemy)
MySQL: 5.0.51

I think I fixed it. It's turns out I had a simple config error. My ini file read:
sqlalchemy.default.url = [connection string here]
sqlalchemy.pool_recycle = 1800
The problem is that my environment.py file declared that the engine would only map keys with the prefix: sqlalchemy.default so pool_recycle was ignored.
The solution is to simply change the second line in the ini to:
sqlalchemy.default.pool_recycle = 1800

You might want to check MySQL's timeout variables:
show variables like '%timeout%';
You're probably interested in wait_timeout (less likely but possible: interactive_timeout). On Debian and Ubuntu, the defaults are 28800 (MySQL kills connections after 8 hours), but maybe the default for your platform is different or whoever administrates the server has configured things differently.
AFAICT, pool_recycle doesn't actually keep the connections alive, it expires them on its own before MySQL kills them. I'm not familiar with pylons, but if causing the connections to intermittently do a SELECT 1; is an option, that will keep them alive at the cost of basically no server load and minimal network traffic. One final thought: are you somehow managing to use a connection that pylons thinks it has expired?