I am using uWSGI to deploy my WSGI application. Are the Python file compiled for every request, or are they precompiled once? I don't see any .pyc files.
Python caches modules bytecode - directly in the same location for python2.x, under a __pycache__ subfolder for python3 - but scripts (the difference is in usage - if you import it it's a module, if you execute it it's a script) are always recompiled (which is why main scripts are usually very very short an simple).
IOW, your main wsgi script will be recompiled once for each new server process. Typically a wsgi app is served as a long running process which will handle much more than one single request, so even then the initial compilation overhead is really not an issue (short script + only compiled once per process)...
Also, once a Python process is started, imported modules are cached in memory so they are only really imported (loaded) once per process.
Just note that the user under which the process is running must have write permissions on your app's directory in order to create the .pyc files... and of course read permissions on the .pyc files too.
If you're using CPython then by default, no it does not compile every time a request is received unless you manually configure it to do this.
By the first request, the uWSGI will load the python scripts bytecode and will reload it in 2 different scenarios I know of:
There is a new import that has not been loaded before (only the new module will be parsed and converted to bytecode).
You explicitly ran service uwsgi restart
Yet, there is still a way to keep reloading python scripts in every request by suppressing it by the interpreter, example: python -B my_amazing_view.py
For more details check here:
What is pycache
Should I generate .pyc
Related
I know the similar questions were already answered:
What will happen if I modify a Python script while it's running?
When are .pyc files refreshed?
Is it possible to replace a python file while its running
changing a python script while it is running
but I still can't find the clear answer to my question.
I have main.py file and the other *.py modules that I import from main file. I run python.exe main.py from (Win) console and python interpreter generates *.pyc files. Than I change some *.py source files and in another console run again python.exe main.py (while the first instance is still running). Python interpreter regenerates only *.pyc files for *.py source files I changed, while the other *.pyc files remains intact.
As I understand and as answers to those questions suggest, the first instance of the running program loaded all (first version of) *.pyc files in memory, the second instance of the running program loaded all (second version of) *.pyc files in memory.
My question is:
Are there any circumstances where the first instance will need/want to reload *.pyc files to memory from disk again (some swap memory/disk or something) or it loaded *.pyc files to memory for good (until the end of running the first instance of the program). Because if there are, the first instance will then reload some of the new *.pyc files and it can probably crash.
I know I can deliberately reload the modules in my python source files:
How do I unload (reload) a module?
but I don't do that in my source files. Is there some other danger.
Why am I asking this question. I made a strategy to upgrade my python GUI program by just copying *.py files through LAN to client's shared folders. On the client (Win) PC, user can have opened for example two or three instances of python GUI program. While user is running those instances on his/her client PC, I make the upgrade (just copy some changed *.py files to their PCs through LAN). He/She closes one of those programs (aware of the upgrade or not, it doesn't matter), loads it again and python interpreter regenerates some *.pyc files. Again, is there any danger that the first two instances will ever need to reload *.pyc files or (as far as they are concerned) they are loaded into memory for good?
Just for fun, I did exactly that for test and even deleted all *.pyc files while all three instances were running and tested it and they never needed any of the *.pyc files again (they never regenerated them) in those sessions.
I just need confirmation that it works that way in order to be sure to make upgrades safely that way.
Is it possible to tell the Django development server to listen to changes on some files (not the regular Python files in the project, they already creates a reload) and reload if these files have changed?
Why do I want this? When my gulp-script builds the JS and CSS, it takes all files and appends a hash to them. My working file frontend/css/abc.css will be built to build/css/abc-hash123.css. This so that browsers are forced to reload the content if anything has changed since last deploy. When this builds, a manifest file is created that contains the mappings. In the settings in Django, I read this manifest file at start and get all these mappings (from this example here). If I change a file, it gets a new hash, and a new manifest is written. The mappings currently loaded into the memory are then wrong, and thus I need to force a reload of the development server to get the new mappings.
Clarification: I got comments on using a VCS. So just for clarification: I use a VCS and check in my abc.css. abc-hash123.css is automatically built as a part of the build system.
As far as doc goes this is not supported.
This is a bit hacky, but just touching (or changing the timestamp in any other way) some .py file or some other file that development server observes (e.g. settings.py) in your build will do the job.
Also as comments suggest, versioning is better left to a VCS.
A bit of a hacky way, but I found a way to make force the Django server to reload on changes to the build directory. After some research on the Django autoreload module, I found that it only listen to changes on Python files, from modules that are loaded into sys.modules. So, I only needed to make the build directory a Python module:
In my build script (gulp), after I've built my files, I added a step to create the __init__.py file, to make the build directory a module. Then I also wrote a comment in this file that contains a time stamp, making it unique for each build:
fs.writeFile('./build/__init__.py', '#' + Date.now())
And then in my local.py (the settings to use locally), I simply import this new module, so it appears in sys.modules:
try:
import build
except ImportError:
print("Could not import from build folder. Will not auto reload when building new files.")
I have a python application running in an embedded Linux system. I have realized that the python interpreter is not saving the compiled .pyc files in the filesystem for the imported modules by default.
How can I enable the interpreter to save it ? File system permission are right.
There are a number of places where this enabled-by-default behavior could be turned off.
PYTHONDONTWRITEBYTECODE could be set in the environment
sys.dont_write_bytecode could be set through an out-of-band mechanism (ie. site-local initialization files, or a patched interpreter build).
File permissions could fail to permit it. This need not be obvious! Anything from filesystem mount flags to SELinux tags could have this result. I'd suggest using strace or a similar tool (as available for your platform) to determine whether any attempts to create these files exist.
On an embedded system, it makes much more sense to make this an explicit step rather than runtime behavior: This ensures that performance is consistent (rather than having some runs take longer than others to execute). Use py_compile or compileall to explicitly run ahead-of-time.
Are there any benefits, performance or otherwise for avoiding .pyc files, except for the convenience of not having a bunch of these files in the source folder?
I don't think there really is. .pyc files are cached bytecode files, and you save startup time as Python does not have to recompile your python files every time you start the interpreter.
At most, switching off bytecode compilation lets you measure how much time the interpreter spends on this step. If you want to compare how much time is saved, remove all the .pyc files in your project, and time Python by using the -B switch to the interpreter:
$ time python -B yourproject
then run again without the -B switch:
$ time python yourproject
It could be that the user under which you want to run your program does not have write access to the source code directories; for example a web server where you do not want remote users to have any chance of altering your source code through a vulnerability. In such cases I'd use the included compileall module to bytecompile everything using a priviledged user rather than forgo writing .pyc files.
One reason I can think of: during development, if you remove or rename a .py file, the .pyc file will stay around in your local copy with the old name and the old bytecode. Since you don't normally commit .pyc files to version control, this can lead to stray ImportErrors that don't happen on your machine, but do on others.
It is possible that .pyc files could encourage people to think they need not maintain/ship the original source. pyc files might not be portable between operating systems and versions of Python. When moving Python modules it is safer to leave the pyc files behind and just copy or ship the source and let the host python generate new pyc files.
By the way, from Python 3.2 the .pyc files no longer go into the source folder, but in __pycache__ (in the source folder).
I have a python project with this directory structure and these files:
/home/project_root
|---__init__.py
|---setup
|---__init__.py
|---configs.py
|---test_code
|---__init__.py
|---tester.py
The tester script imports from setup/configs.py with the reference "setup.configs". It runs fine on my development machine.
This works on the development (Linux) computer. When I move this to another (Linux) computer, I set the PYTHONPATH with
PYTHONPATH = "/home/project_root"
But when I run tester.py, it can't find the configs module. And when I run the interactive Python interpreter, sys.path doesn't include the /home/project_root directory. But /home/project_root does appear when I echo $PYTHPATH.
What am I doing wrong here?
(I don't want to rely on the .bashrc file to set the PYTHONPATH for the target machine -- the code is for a Django application, and will eventually be run by www-data. And, I know that the apache configuration for Django includes a specification of the PYTHONPATH, but I don't want to use that here as I'm first trying to make sure the code passes its unit tests in the target machine environment.)
CURIOUSER AND CURIOUSER
This seems to be a userid and permissions problem.
- When launched by a command from an ordinary user, the interpreter can import modules as expected.
- When launched by sudo (I'm running Ubuntu here), the interpreter cannot import modules as expected.
- I've been calling the test script with sudo, as the files are owned by www-data (b/c they'll be called by the user running apache as part of the Django application).
- After changing the files' ownership to that of an ordinary user, the test script does run without import errors (albeit, into all sorts of userid related walls).
Sorry to waste your time. This question should be closed.
Stick this in the tester script right before the import setup.configs
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), os.path.pardir))
sys.path is a list of all the directories the python interpreter looks for when importing a python module.
This will add the parent directory which contains setup module to the beginning of that list which means that the local directory will be checked first. That is important if you have your module installed system wide. More info on that here: sys doc.
EDIT: You could also put a .pth file in /usr/local/lib/python2.X/site-packages/ A .pth file is simply a text file with a directory path on each line that the python interpreter will search in. So just add a file with this line in it:
/home/project_root
Try explicitly setting your python path in your scripts. If you don't want to have to change it, you could always add something like "../" to the path in tester. That is to say:
sys.path.append("../")
(I don't want to rely on the .bashrc file to set the PYTHONPATH for the target machine -- the code is for a Django application, and will eventually be run by www-data. And, I know that the apache configuration for Django includes a specification of the PYTHONPATH, but I don't want to use that here as I'm first trying to make sure the code passes its unit tests in the target machine environment.)
If the code is for a Django application, is there a reason you're not testing it in the context of a Django project? Testing it in the context of a Django project gives a couple benefits:
Django's manage.py will set up your Python environment for you. It'll add the appropriate project paths to sys.path, and it'll set the environment variable DJANGO_SETTINGS_MODULE correctly.
Django's libraries include ample unit testing facilities, and you can easily extend that functionality to include your own testing facilities. Executing tests in a Django project is as easy as executing a single command via manage.py.