Develop Reference

Python bottle process reach timeout connection

I have a problem that after certain amount of time my bottle server is not reachable and you get connection reset- timeout connection error.
When checking if the process is running, I found it running, but after killing the process and running it again the server return to serve requests.
Any idea what it could be?
I wrapped most of my functions with exception catching , but didn't helped me to understand the problem.
I wonder if anybody has used bottle and had encountered such problem

My guess is because bottle is single threaded, and it's hanging on a request. I would suggest trying a multi-threaded server, such as cherrypy, to see if that resolves the issue. Then go back and see where the hangup was at.
Install cherrypy
pip install cherrypy
Update your python file
bottle.run(myapp, server='cherrypy')
Would need to see more code to identify any specific issue.

Slow Flask on mod_wsgi on openshift

I have some performance issues with my Flask application on openshift.
There is a need to get some images from database and display them on the web page. And
for this taks, I have created a simple method :
#app.route('/getImage/')
def getImageFromUrl(url=None):
return make_response(getImageFromDb(request.args['url']));
There are maximum 10 images per page. And the problem is that this is slow.... veerry slow.
On my local machine, started with app.run() (even in debug mode) it is super fast, so I asume there is something in mod_wsgi.
Also there are these error messages in log files:
Exception KeyError: KeyError(140116433057760,) in <module 'threading' from '/usr/lib64/python2.6/threading.pyc'> ignored
and
[error] server reached MaxClients setting, consider raising the MaxClients setting
What is happening and what should I do to speed the things up?

Exception KeyError is caused by gevent I guess, should be more code in question :) To avoid it import gevent before all.
server reached MaxClients setting seems to be Apache error and should be investigated with logs and settings MaxClients and ServerLimt.

The KeyError is usually because you are using an old version of mod_wsgi. Use mod_wsgi 3.3 or later, which has changes to accommodate the changes which were made in Python that caused this.

How to setup WSGI server to run similarly to Apache?

I'm coming from PHP/Apache world where running an application is super easy. Whenever PHP application crashes Apache process running that request will stop but server will be still ruining happily and respond to other clients. Is there a way to have Python application work in a smilar way. How would I setup wsgi server like Tornado or CherryPy so it will work similarly? also, how would I run several applications from one server with different domains?

What you are after would possibly happen anyway for WSGI severs. This is because any Python exception only affects the current request and the framework or WSGI server would catch the exception, log it and translate it to a HTTP 500 status page. The application would still be in memory and would continue to handle future requests.
What we get down to is what exactly you mean by 'crashes Apache process'.
It would be rare for your code to crash, as in cause the process to completely exit due to a core dump, the whole process. So are you being confused in your terminology in equating an application language level error to a full process crash.
Even if you did find a way to crash a process, Apache/mod_wsgi handles that okay and the process will be replaced. The Gunicorn WSGI server will also do that. CherryPy will not unless you have a process manager running which monitors it and the process monitor restarts it. Tornado in its single process mode will have the same problem. Using Tornado as the worker in Gunicorn is one way around that plus I believe Tornado itself may have some process manager in it now for running multiple process which allow it to restart processes if they die.
Do note that if your application bug which caused the Python exception is bad enough and it corrupts state within the process, subsequent requests may possibly have issues. This is the one difference with PHP. With PHP, after any request, whether successful or not, the application is effectively thrown away and doesn't persist. So buggy code cannot affect subsequent requests. In Python, because the process with loaded code and retained state is kept between requests, then technically you could get things in a state where you would have to restart the process to fix it. I don't know of any WSGI server though that has a mechanism to automatically restart a process if one request returned an error response.

If you're in an UNIX-like environment, you can run mod_wsgi under Apache in Daemon Mode. This means there will be a separate process for the Python code, and even if it crashes the server will continue running normally (and hopefully the WSGI process will restart itself). A WSGI application can run under multiple processes and multiple threads per process.
As for running multiple domains in the same server, check Name-Based Virtual Hosts.

How do I keep a python HTTP Server up forever?

I wrote a simple HTTP server in python to manage a database hosted on a server via a web UI. It is perfectly functional and works as intended. However it has one huge problem, it won't stay put. It will work for an hour or so, but if left unused for long periods of time when returning to use it I have to re-initialize it every time. Right now the method I use to make it serve is:
def main():
global db
db = DB("localhost")
server = HTTPServer(('', 8080), MyHandler)
print 'started httpserver...'
server.serve_forever()
if __name__ == '__main__':
main()
I run this in the background on a linux server so I would run a command like sudo python webserver.py & to detach it, but as I mentioned previously after a while it quits. Any advice is appreciated cause as it stands I don't see why it shuts down.

You can write a UNIX daemon in Python using the python-daemon package, or a Windows service using the pywin32.
Unfortunately, I know of no "portable" solution to writing daemon / service processes (in Python, or otherwise).

Here's one piece of advice in a story about driving. You certainly want to drive safely (figure out why your program is failing and fix it). In the (rare?) case of a crash, some monitoring infrastructure, like monit, can be helpful to restart crashed processes. You probably wouldn't want to use it to paper over a crash just like you wouldn't want to deploy your air bag every time you stopped the car.

Well, first step is to figure out why it's crashing. There's two likely possibilities:
The serve_forever call is throwing an exception.
The python process is crashing/being terminated.
In the former case, you can make it live forever by wrapping it in a loop, with a try-except. Probably a good idea to log the error details.
The latter case is a bit trickier, because it could be caused by a variety of things. Does it happen if you run the script in the foreground? If not, maybe there's some kind of maintenance service running that is terminating your script?
Not really a complete answer, but perhaps enough to help you diagnose the problem.

Have you tried running it from inside a screen session?
$ screen -L sudo python webserver.py

As an alternative to screen there is NoHup which will ensure the process carries on running after your logged out.
Its worth checking the logs to see why its killed/quitting as well as it may not be related to the operating system but an internal fault.

Print statements on server give IOError: failed to write data

I am running Pylons on my local machine with paster, and on a Debian server using WSGI. I want to add some print statements to debug a problem: am not a Pylons or Python expert.
On my local machine this works fine: print statements go to the terminal. On the server, the statements don't print to the log files: instead the log file says "IOError: failed to write data" whenever a print statement is called.
Until I can fix this, I can't debug anything on the server.
Could someone advise how to get printing running on the server? Thanks!

It's wrong for a WSGI application to use sys.stdout or sys.stderr. If you want to spit debug to a server error log, use environ['wsgi.errors'].write().

Don't use print statements, use the logging module. We can't help you without knowing the setup of the server.