Core Problem:
I've run into situations multiple times where pyc files break my local application (especially unit tests).
For example I have a folder utils/ which contains a poorly named sqlalchemy.py (which just contains utilities for sql alchemy). I add another file utilities/whatever.py which contains
import sqlalchemy.orm.session
and that breaks because well...the python2 importer at least thinks "I'm going to look at the relative path first". So I rename or delete utilities/sqlalchemy.py and everything works hunky dory.
Until my teammates pull down changes and everything breaks. Because while their copy no longer has utilities/sqlalchemy.py, it still has the git ignored utilities/sqlalchemy.pyc file
What I Want:
Is there a way to get the python importer to ignore pyc files? I would of course only want this to be active locally when running unit test, so I'm hoping for like an environment variable or a configuration for the python importer, but hoping other people have run into this problem enough that there's some sort of "quasi-official" solution.
Note that I specifically don't want to do something like find . -name '*.pyc' -delete in a post-checkout git hook since our repo is large enough that this takes several minutes and would really slow us down every time someone switches branches.
Related
I would like to be able to run a Python script from any folder. However, this Python script, named statistics.py calls upon two other files, colorbar.py and language_bytes.py. It also reads data from a folder that is located beneath it, in data/languages.yml.
Because it has these dependencies, along with the .yml file, I have been unable to have it be executable from anywhere. Even if I add #!/usr/bin/env python3 to the first line of statistics.py and drag it, along with the other two files and the data folder into usr/local/bin, this doesn't work.
With a little bit of digging I found out that this was because the data folder was being deleted as soon as it went into usr/local/bin. This means that the Python script is unable to access the data that it needs, and so cannot be run.
Is there any other way to run a Python script globally?
Edit: Because this script is made to by modifiable by users and will be open-source, it would be optimal if it could be run without having to be bundled into a PyPI package. As well as this, it doesn't make sense that every time one would want to make a globally runnable script with resources they'd have to upload it to PyPI, even if it was just for personal use.
Chris and Pat collaborate on a Python project. The source code is under version control. They each choose their own checkout location on their respective drives. The collaborative application resides in <root>/app.py.
They both use PyCharm, and since the .idea folders on their checkout drives have some amount of effort invested, they are reluctant not to check it into the repository.
Yet because the checkout locations are distinct, they cannot share one .idea folder. They have resolved the problem by storing their .idea folders in <root>/pycharm/chris/.idea and <root>/pycharm/pat/.idea. Both folders are checked into the repository. They can go home with the confidence that if their desktops' disks fail, the effort they spent on their PyCharm configuration is safe.
But something is missing. They write (pytest-based) tests for their own respective subsets of the code, and they would like to share those tests.
Is there a workflow, from within PyCharm or without, that enables them to work on a unified set of test configurations? I may be missing something basic. It may be the case, for example, that they ought to make an effort to checkout into exactly the same paths, as well as to use venvs located at exactly the same paths, and share the same .idea all the way. If you've found that that's the only sensible solution, arguing for it would also be an answer.
They should just have one .idea file in the git repo and .gitignore the parts they want to keep unique to their systems.
They can also mark a given Run Configuration as "sharable" by enabling "Share" option in it (top right corner of the configuration window). This way it will be stored in .idea/runConfigurations.
You can check in pytest related configuration in git. Pytest configurations can be stored in pytest.ini file which can be version controlled.
pytest.ini file should be at the root of pytest tests directory. Pytest would pick all the test run configuration at the time of test run from that file.
Example config for increasing pytest log verbosity:
#pytest.ini
[pytest]
addopts= -vv
Documentation here: https://docs.pytest.org/en/latest/reference.html#ini-options-ref
Are there any benefits, performance or otherwise for avoiding .pyc files, except for the convenience of not having a bunch of these files in the source folder?
I don't think there really is. .pyc files are cached bytecode files, and you save startup time as Python does not have to recompile your python files every time you start the interpreter.
At most, switching off bytecode compilation lets you measure how much time the interpreter spends on this step. If you want to compare how much time is saved, remove all the .pyc files in your project, and time Python by using the -B switch to the interpreter:
$ time python -B yourproject
then run again without the -B switch:
$ time python yourproject
It could be that the user under which you want to run your program does not have write access to the source code directories; for example a web server where you do not want remote users to have any chance of altering your source code through a vulnerability. In such cases I'd use the included compileall module to bytecompile everything using a priviledged user rather than forgo writing .pyc files.
One reason I can think of: during development, if you remove or rename a .py file, the .pyc file will stay around in your local copy with the old name and the old bytecode. Since you don't normally commit .pyc files to version control, this can lead to stray ImportErrors that don't happen on your machine, but do on others.
It is possible that .pyc files could encourage people to think they need not maintain/ship the original source. pyc files might not be portable between operating systems and versions of Python. When moving Python modules it is safer to leave the pyc files behind and just copy or ship the source and let the host python generate new pyc files.
By the way, from Python 3.2 the .pyc files no longer go into the source folder, but in __pycache__ (in the source folder).
I'm just getting into programming; I've made some simple scripts and am hoping to learn how to do things with them. I have a question about where these files "live" on my computer, and if that's important. Let's say I have a script in the directory users/me/desktop/project/skeleton/ called webproject.py. In the skeleton/ directory I also have tests/; does tests/ have to be in skeleton/ or can it be in some random place like my desktop/?
Further, if one of my scripts imports a module that I've created, does it matter where that's located? If I make a script with a function that outputs the nth fibonacci number, and save it on my desktop, can my webproject.py script from users/me/desktop/project/skeleton/ import it?
Any links/resources would be helpful.
It of course matters where you place your files. There sure is more than one directory called "tests", so your Python interpreter cannot guess what "tests" directory it should pick. There is no magic in your computer, I am sorry. ;) But there is something called "Python path". All modules which are saved in a directory specified in the Python path can be imported from anywhere. To use webproject.py from "project/skeleton/" you would have to include "project/skeleton/" in your Python path or make it a package.
There is a nice chapter in the official Python tutorial about modules. :)
i have a following solution structure in python:
main_app
main_app/template_processor/
main_app/template_processor/models
main_app/template_processor/views
everything works just fine on my local machine. as soon as code gets to server (it stopped working after i removed all .pyc files from svn), it doesn't see the assembly (if it could be called like that in terms of python) with models. during syncdb command, it creates no tables except for admin ones. but in runtime it finds models themselves, but does not find tables (since they were not created by syncdb)
i've added application to settings.py as installed app, everything seems to be fine.
the only difference that i can see for now is that on my local machine i have main_app/template_processor/models/models.pyc file, but it doesn't precompile it on server for some reason (might be a hint??)
__init__.py files are existing in every folder/subfolder.
have anyone faced an issue like that?
Hm, looking at your directory structure, I think that you need to import all your models in template_processor/models/__init__.py, because Django looks only in <app_name>.models module when loading models automatically (i.e. for syncdb).
My guess would be that you renamed a file, and didn't delete the oldname.pyc file.
So if you try to
import oldname
then you rename oldname to rename, but don't update your import statement, the code will work on systems where oldname.pyc exists, however python won't be able to recreate oldname.pyc if oldname.py doesn't exist.
try
find . | grep py | xargs grep import | grep -v django | sort -u
that should give you a list of all imports in your project, see if one of those imports is pointing at a module for which you have a pyc file but not a .py file.
In general, python quickly compiles .py files into .pyc files, and it does this once. I wouldn't worry about the time it takes to generate new .pyc files.
Sounds like Django isn't seeing that module (folder, in this case) for some reason. Make sure all the folders have a file called __init__.py (notice the two underscores before and after). Once that's done, make sure it's listed in your installed apps.
Maybe you made some change to it that's causing it to stop loading. You can also try moving the .pyc files out of the directory on your local machine and see whether they're regenerated or not when you runserver.
Maybe the most helpful thing: ./manage.py shell to bring up an interactive shell and then 'import app_name' (without quotes) to see whether django is having trouble finding the module.
Ok, if anyone's interested in what really was happening, i've got a story to tell ya:
http://code.djangoproject.com/ticket/4470
that's basically what i was going to implement.
in order to really get this thing work i still should have a file models.py, which will have a proper list of classess, with Pass inside of it. Then i should have taken all the files (my models) and changed their meta for syncdb to understand they are from the certain "assembly".
source code is available (pls see url above).
thx for helping out!