I have inherited quite a bit of python code and all over it is the following snippet which adds the file path of the parent directory to the system path.
from os.path import join, dirname
sys.path.insert(0, join(dirname(sys.argv[0]), "..\\"))
from utilities import find, execute
My understanding of this is that it adds a path to the search path. Which during the running of a program adds numerous pathsto the search path and presumably make it slower. As each file adds it's own parent directory.
I prefer the syntax
from scm_tools.general.utilities import find, execute
because this is easier to understand and far less code. This might have implications if I am moving the code around but it's all in a single package.
Am I right in assuming that inside a package that the latter syntax is the more pythonic way of doing things ?
or does it not really matter as under the hood python is doing some magic ?
Use relative imports when you can:
from ..utilities import find, execute
This requires that you stay within the module space, which means each directory you traverse requires an __init__.py file.
There are cases where this breaks down, for example if your tests directory isn't inside the module structure. In these cases you need to edit the path, but you shouldn't edit the path blindly like the above example.
Either add to the PYTHONPATH environment variable before you code starts so you can always reference the root of the directory or only add paths that aren't already in the sys.path and try avoiding adding anything but module roots.
The PYTHONPATH change is a bit risky for code you wish to distribute. It's easy to have a change in PYTHONPATH you can't control or for you to not define that addition in a way that transfers to distributed code. It also adds an annoying module requirement that other's have to deal with -- so reserve this for adding whole swaths of modules that you want to include, like custom site-package directories. It's almost always better to use virtualenv for such situations.
If you do need to change a sys.path inside code you should try to at least avoid clobbering it all over the place or you'll have a headache trying to fix it when it goes awry. To avoid this try to only add root module paths so you can always import in a root.submodule.desiredmodule pattern. Additionally check if a path is already present before you insert it into sys.path to avoid very long sys.paths. In my test directories I oftentimes have an importable file that fixes the sys.path to the root of the directory structures I am testing:
# Add parent import capabilities
parentdir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if parentdir not in sys.path:
sys.path.insert(0, parentdir)
Related
I am looking for a solution that will allow me to use py file as a little library.
Short description:
I have a py script and there are a lot of common functions in it.
Instead of each time use a big file I want to throw all common functions into separate py file, put into folder Tools and use it when I need.
My problems is that I cannot import this file from Tools, because my script does not see it.
My folder structure:
C:\some\folders\here\my\folder\script.py
C:\some\folders\here\Tools\Library\library.py
Also, it is not good for me to user init.py, because I haven't any python project, it is just one file without any other things.
Are there any normal solutions?
Python interpreter searches for modules in 3 places:
current directory, that means the directory from which you run the script with your import statement
list of directories from PYTHONPATH
installation-dependent list of directories, this is configures when Python in installed
You can also modify sys.path at runtime and include a directory where your module is located, but this is the worst solution and it's usually discouraged to do so.
Setting PYTHONPATH will most likely be a solution in your situation.
There is some python code at my company that uses sys.path[0] in some file paths. Generally, everybody's sys.path[0] points to c:\\<user.name>\\workspace and I believe mine used to as well. However, now my sys.path[0] is c:\\users\\<user.name>\\repos\\venv\\lib\\site-packages\\git\\ext\\gitdb so obviously these paths are now incorrect on my local and causing errors.
My questions are what would cause this change in my local sys.path and if it is bad practice to use sys.path because of situations like this. What should we use in this case instead?
This is all for python code running on a flask server.
It is not safe, because, as it says in comments, lots of things can modify the order of sys.path.
As a workaround, you can import some file in workspace and then get the directory:
import something_in_workspace
workspace = os.path.dirname(something_in_workspace.__file__)
Usually it is needed to retrieve data files. Better is to package your project and use importlib.resources.
See How to read a (static) file from inside a Python package?
There are many of similar questions about PYTHONPATH and imports but I didn't find exactly what I needed.
I have a git repository that contains a few python helper scripts. The scripts are naturally organized in a few packages. Something like:
scripts/main.py
scripts/other_main.py
scripts/__init__.py
a/foo.py
a/bar.py
a/__init__py
b/foo.py
b/bar.py
b/__init__.py
__init__.py
scripts depends on a and b. I'm using absolute import in all modules. I run python3 scripts/main.py. Everything works as long as I set up PYTHONPATH to the root of my project.
However, I'd like to avoid users the hassle of setting up an environment variable.
What would be the right way to go? I expected this to work like in java, where the current dir is in the classpath by default but it doesn't seem to be the case. I've also tried relative import without success.
EDIT: it seems to work if I remove the top-level __init__.py
Firstly, you're right in that I don't think you need the top-level __init__.py. Removing it doesn't solve any import error for me though.
You won't need to set PYTHONPATH and there are a few alternatives that I can think of:
Use a virtual environment (https://virtualenv.pypa.io/en/latest/). This would also require you to package up your code into an installable package (https://packaging.python.org/). I won't explain this option further since it's not directly related to your question.
Move your modules under your scripts directory. Python automatically adds the script's directory into the Python path.
Modify the sys.path variable in your scripts so they can find your local modules.
The second option is the most straightforward.
The third option would require you to add some python code to the top of your scripts, above your normal imports. In your main.py it would look like:
#!/usr/bin/env python
import os.path, sys
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
import a
import b
What this does is:
Take the filename of the script
calculate the parent directory of the directory of the script
Prepend that directory to sys.path
Then do normal imports of your modules
I have a Django app and I'm getting an error whenever I try to run my code:
Error: No module named django_openid
Let me step back a bit and tell you how I came about this:
I formatted my computer and completely re-installed everything -- including virtualenv, and all dependent packages (in addition to Django) required for my project based on settings in my requirements.txt folder
I tried doing python manage.py syncdb and got the error
I googled the issue, and many people say it could be a path problem.
I'm confused as to how I go about changing the path variables though, and what exactly they mean. I found some documentation, but being somewhat of a hack-ish noob, it kind of goes over my head.
So my questions are:
What exactly is their purpose -- and are they on a system based level based on the version of Python or are they project dependent?
How can I see what mine are set to currently?
How can I change them (ie. where is this .profile file they talk of and can I just use a text editor)
Any input you would have would be great as this one is stumping me and I just want to get back to writing code :-)
The path is just the locations in your filesystem in which python will search for the modules you are trying to import. For example, when you run import somemodule, Python will perform a search for somemodule in all the locations contained in the path (sys.path variable).
You should check the path attribute in sys module:
import sys
print sys.path
It is just a regular list, sou you could append/remove elements from it:
sys.path.append('/path/to/some/module/folder/')
If you want to change your path for every python session you start, you should create a file to be loaded at startup, doing so:
Create a PYTHONSTARTUP environment variable and setting it to your startup file. E.g.: PYTHONSTARTUP=/home/user/.pythonrc (in a unix shell);
Edit the startup file so it contains the commands you want to be auto-executed when python is loaded;
An example of a .pythonrc could be:
import sys
sys.path.append('/path/to/some/folder/')
Do you really need to alter the path? It's always best to actually think about your reasons first. If you're only going to be running a single application on the server or you just don't care about polluting the system packages directory with potentially unnecessary packages, then put everything in the main system site-packages or dist-packages directory. Otherwise, use virtualenv.
The system-level package directory is always on the path. Virtualenv will add its site-packages directory to the path when activated, and Django will add the project directory to the path when activated. There shouldn't be a need to add anything else to the path, and really it's something you should never really have to worry about in practice.
I'm attempting to write a game. I therefore have lots of different types of code and want to arrange them in a useful hierarchy.
I've looked at solutions that involve placing __init__.py in each folder but I'm still somewhat confused, though not as much as the python interpreter.
Now suppose resource1.py wants to import a function from physics1.py, or indeed any other .py file in the Game directory, how would I go about doing so?
I've tried from bin.physics.physics1 import function but obviously that doesn't work.
Thanks for your help.
/Game
launcher.py
/bin
game.py
__init__.py
/physics
__init__.py
physics1.py
physics2.py
/resources
__init__.py
resource1.py
It is not possible with the normal import mechanism unless you make Game a package (i.e., by putting an __init__.py inside the Game directory). The python relative import system only works within packages. It is not a general system for referring to arbitrary modules by their location in the directory structure. If you make Game a package, then you could do from ..bin.physics.physics1 import function.
Edit: Note also that relative imports don't work from a script executed as the main program. If you try to run resource.py directly and it uses relative imports, you'll get a "relative import attempted in non-package" error. It will work if you import resource from another module. This is because the relative import system is based on the "name" of the executing module, and when you run a script directly its name is __main__ instead of whatever it would usually be named. It's possible to get around this using the __package__ keyword if you really need to, but it can be a bit tricky.