I've written some code for an image reconstruction server. I use a python library but it doesn't work for server so I made some changes so that it would work but it never reloads the library. I make the changes and it ignores them. I don't understand how this can be possible? Where does it get this information from and how can I make it get a fresh copy of the library every time?
The Python process started by uWSGI only interprets the source once at startup, any code changes after that will not affect the in-memory process. You should probably just manually restart the uWSGI process when this happens. Alternatively you can tell uWSGI to auto-reload if files or directories change: http://uwsgi-docs.readthedocs.org/en/latest/Options.html#touch-reload.
Related
The Issue
I am using my laptop with Apache to act as a server for a local project involving tensorflow and python which uses an API written in Flask to service GET and POST requests coming from an app and maybe another user on the local network.The problem is that the initial page keeps loading when I specifically import tensorflow or the object detection folder within the research folder in the tensorflow github folder, and it never seems to finish doing so, effectively getting it stuck. I suspect the issue has to do with the packages being large in size, but I didn't have any issue with that when running the application on the development server provided with Flask.
Are there any pointers that I should look for when trying to solve this issue? I checked the memory usage, and it doesn't seem to be rising substantially, as well as the CPU usage.
Debugging process
I am able to print basic hello world to the root page quite quickly, but I isolated the issue to the point when the importing takes place where it gets stuck.
The only thing I can think of is to limit the number of threads that are launched, but when I limited the number of threads per child to 5 and number of connections to 5 in the httpd-mpm.conf file, it didn't help.
The error/access logs don't provide much insight to the matter.
A few notes:
Thus far, I used Flask's development server with multi-threading enabled to serve those requests, but I found it to be prone to crashing after 5 minutes of continuous run, so I am now trying to use Apache using the wsgi interface in order to use Python scripts.
I should also note that I am not servicing html files, just basic GET and POST requests. I am just viewing them using the browser.
If it helps, I also don't use virtual environments.
I am using Windows 10, Apache 2.4 and mod_wsgi 4.5.24
The tensorflow module being a C extension module, may not be implemented so it works properly in Python sub interpreters. To combat this, force your application to run in the main Python interpreter context. Details in:
http://modwsgi.readthedocs.io/en/develop/user-guides/application-issues.html#python-simplified-gil-state-api
I have experience with PHP but now have to manage a python application. It is already developed and running on live server. On live server we have beta.domain.com as well. which has its own copy of same source, other than the main domain application.
Now, when we print something within template it effects but before that, following MVC, if we try to print or use sys.exit() in manage.py or later imported "settings.py" or then views.py, nothing effects these files at all. After any change in just mentioned files, Website still renders everything and display related template.
Another thing that if even we remove
return render_to_response('home.html', RequestContext(request, context))
inside the views.py for testing, nothing effects and website still gets rendered with template.
Do I need to think that if website code is already hosted then in order to have changes to reflect new one in the code in any file (.py), needs to be reinitialized by executing any project related file?
I am not sure how python code needs to be updated at all, any quick help is much appreciated.
Anytime you change your code, you need to restart Apache server.
<path to apache>/bin/apachectl restart
The python interpreter of the process has already loaded your python modules in previous web requests. And once the module is loaded, it is stored in memory. Next time when a request comes, the Python interpreter will simply use the version of the module that is already loaded in memory. So your changed code will not be picked up.
Please, please, please don't try and edit files on your live server. You're only going to get yourself into terrible difficulties. Use a local copy for development, preferably cloned via a version control system, and deploy at regular intervals rather than on every change.
To answer your question, you usually need to restart the WSGI process in order to see Python code changes. An easy way to do this is to touch the .wsgi file, at which point mod_wsgi will detect that it has changed, and reload everything. Otherwise you can simply reload Apache.
Unfortunately I don't understand your references to manage.py or sys.exit.
The solution is excatly restarting the apache server as answered by Sudipta .Each time the py files are interpretted .pvc files are created for each corresponding .py files.When the Apache server is restarted all the python source files are interprretted again and new .pvc files are generated and only these files will run till the apache restarts again.
When the code for my python WSGI applicaiton changes should I use apache2's reload or graceful restart feature?
Currently we use reload, but have noticed that sometimes the application does not load properly and errors pertaining to missing modules are logged to the error files even though the modules have existed for a long time.
If you can, you should probably use graceful. But if your application is not exiting correctly you may have to force it with just restart.
For wsgi, you should try running in daemon mode. When it is running in daemon mode, you can restart your service just by touching the wsgi file and updating its timestamp. This will reload all the code without restarting apache.
Here is more info: http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIDaemonProcess
This is for django, but may be useful for your project: http://code.google.com/p/modwsgi/wiki/IntegrationWithDjango
The 'reload' and 'graceful' would have the same effect as far as reloading your web application. If you are seeing issues with imports like you describe, it is likely to be an issue in your application code with you having import order dependencies or import cycles. One sees this a lot with people using Django. Suggest you actually post an example of the error you are getting.
We currently run a small shared hosting service for a couple of hundred small PHP sites on our servers. We'd like to offer Python support too, but from our initial research at least, a server restart seems to be required after each source code change.
Is this really the case? If so, we're just not going to be able to offer Python hosting support. Giving our clients the ability to upload files is easy, but we can't have them restart the (shared) server process!
PHP is easy -- you upload a new version of a file, the new version is run.
I've a lot of respect for the Python language and community, so find it hard to believe that it really requires such a crazy process to update a site's code. Please tell me I'm wrong! :-)
Python is a compiled language; the compiled byte code is cached by the Python process for later use, to improve performance. PHP, by default, is interpreted. It's a tradeoff between usability and speed.
If you're using a standard WSGI module, such as Apache's mod_wsgi, then you don't have to restart the server -- just touch the .wsgi file and the code will be reloaded. If you're using some weird server which doesn't support WSGI, you're sort of on your own usability-wise.
Depends on how you deploy the Python application. If it is as a pure Python CGI script, no restarts are necessary (not advised at all though, because it will be super slow). If you are using modwsgi in Apache, there are valid ways of reloading the source. modpython apparently has some support and accompanying issues for module reloading.
There are ways other than Apache to host Python application, including the CherryPy server, Paste Server, Zope, Twisted, and Tornado.
However, unless you have a specific reason not to use it (an since you are coming from presumably an Apache/PHP shop), I would highly recommed mod_wsgi on Apache. I know that Django recommends modwsgi on Apache and most of the other major Python frameworks will work on modwsgi.
Is this really the case?
It Depends. Code reloading is highly specific to the hosting solution. Most servers provide some way to automatically reload the WSGI script itself, but there's no standardisation; indeed, the question of how a WSGI Application object is connected to a web server at all differs widely across varying hosting environments. (You can just about make a single script file that works as deployment glue for CGI, mod_wsgi, passenger and ISAPI_WSGI, but it's not wholly trivial.)
What Python really struggles with, though, is module reloading. Which is problematic for WSGI applications because any non-trivial webapp will be encapsulating its functionality into modules and packages rather than simple standalone scripts. It turns out reloading modules is quite tricky, because if you reload() them one by one they can easily end up with bad references to old versions. Ideally the way forward would be to reload the whole Python interpreter when any file is updated, but in practice it seems some C extensions seem not to like this so it isn't generally done.
There are workarounds to reload a group of modules at once which can reliably update an application when one of its modules is touched. I use a deployment module that does this (which I haven't got around to publishing, but can chuck you a copy if you're interested) and it works great for my own webapps. But you do need a little discipline to make sure you don't accidentally start leaving references to your old modules' objects in other modules you aren't reloading; if you're talking loads of sites written by third parties whose code may be leaky, this might not be ideal.
In that case you might want to look at something like running mod_wsgi in daemon mode with an application group for each party and process-level reloading, and touch the WSGI script file when you've updated any of the modules.
You're right to complain; this (and many other WSGI deployment issues) could do with some standardisation help.
I'm running a Django app on Apache + mod_python. When I make some changes to the code, sometimes they have effect immediately, other times they don't, until I restart Apache. However I don't really want to do that since it's a production server running other stuff too. Is there some other way to force that?
Just to make it clear, since I see some people get it wrong, I'm talking about a production environment. For development I'm using Django's development server, of course.
If possible, you should switch to mod_wsgi. This is now the recommended way to serve Django anyway, and is much more efficient in terms of memory and server resources.
In mod_wsgi, each site has a .wsgi file associated with it. To restart a site, just touch the relevant file, and only that code will be reloaded.
As others have suggested, use mod_wsgi instead. To get the ability for automatic reloading, through touching the WSGI script file, or through a monitor that looks for code changes, you must be using daemon mode on UNIX. A slight of hand can be used to achieve same on Windows when using embedded mode. All the details can be found in:
http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode
You can reduce number of connections to 1 by setting "MaxRequestsPerChild 1" in your httpd.conf file. But do it only on test server, not production.
or
If you don't want to kill existing connections and still restart apache you can restart it "gracefully" by performing "apache2ctl gracefully" - all existing connections will be allowed to complete.
Use a test server included in Django. (like ./manage.py runserver 0.0.0.0:8080) It will do most things you would need during development. The only drawback is that it cannot handle simultaneous requests with multi-threading.
I've heard that there is a trick that setting Apache's max instances to 1 so that every code change is reflected immediately--but because you said you're running other services, so this may not be your case.