I have two Python packages where one needs to be imported by the other. The directory structure is like follows:
workspace/
management/
__init__.py
handle_management.py
other_management.py
utils/
__init__.py
utils_dict.py
I'm trying to import functionality from the utils project in the handle_management.py file:
import utils.utils_dict
Error I'm getting when trying to run handle_management.py:
ImportError: No module named utils.utils_dict
I've read a lot about how to resolve this problem and I can't seem to find a solution that works.
I started with Import a module from a relative path - I tried the applicable solutions but none worked.
Is the only solution to make workspace/ available via site_packages? If so, what is the best way to do this?
EDIT:
I've tried to add the /home/rico/workspace/ to the PYTHONPATH - no luck.
EDIT 2:
I was able to successfully use klobucar's solution but I don't think it will work as a general solution since this utility is going to be used by several other developers. I know I can use some Python generalizations to determine the relative path for each user. I just feel like there is a more elegant solution.
Ultimately this script will run via cron to execute unit testing on several Python projects. This is also going to be available to each developer to ensure integration testing during their development.
I need to be able to make this more general.
EDIT 3:
I'm sorry, but I don't really like any of these solutions for what I'm trying to accomplish. I appreciate the input - and I'm using them as a temporary fix. As a complete fix I'm going to look into adding a script available in the site_packages directory that will add to the PYTHONPATH. This is something that is needed across several machines and several developers.
Once I build my complete solution I'll post back here with what I did and mark it as a solution.
EDIT 4:
I feel as though I didn't do a good job expressing my needs with this question. The answers below addressed this question well - but not my actual needs. As a result I have restructured my question in another post. I considered editing this one, but then the answers below (which would be very helpful for others) wouldn't be meaningful to the change and would seem out of place and irrelevant.
For the revised version of this question please see Unable to import Python package universally
You have 2 solutions:
Either put workspace in your PYTHONPATH:
import sys
sys.path.append('/path/to/workspace')
from utils import utils_dict
(Note that if you're running a script inside workspace, that is importing handle_management, most probably workspace is already in your PYTHONPATH, and you wouldn't need to do that, but it seems it's not the case HERE).
Or, make "workspace" a package by adding an empty (or not) __init__.py file in the workspace directory. Then:
from ..utils import utils_dict
I would prefer the second, because you would have a problem if there's another module called "utils" in you PYTHONPATH
Apart from that, you are importing wrong here: import utils.utils_dict.py. You don't need to include the extension of the file ".py". You are not importing the file, you are importing the module (or package if it's a folder), so you don't want the path to that file, you need its name.
What you need to do is add workspace to your import path. I would make a wrapper that does this for you in workspace or just put workspace in you PYTHONPATH as an environment variable.
import sys
# Add the workspace folder path to the sys.path list
sys.path.append('/path/to/workspace/')
from workspace.utils import utils_dict
Put "workspace/" in your PYTHONPATH to make the packages underneath available to you when it searches.
This can be done from your shell profile (if you are on *nix, or environment variables on windows.
For instance, on OSX you might add to your ~/.profile (if workspace is in your home directory):
export PYTHONPATH=$HOME/workspace:$PYTHONPATH
Another option is to use virtualenv to make your project area its own contained environment.
Related
I currently have a project folder structured as this
Project/
main.py
.gitignore
.env
requirements.txt
reports/
report1.py
report2.py
utils/
__init__.py
support_functions.py
For this program, I am not trying to run a main function. I have several reports that I run once a week/month, and just run them straight from its file.
I was having a lot of trouble importing the functions from the utils module - however, I did find a workaround on stack overflow which is essentially adding the folder to the path
import sys
sys.path.append('/Project/utils')
import support_functions
However, I don't love this workaround as it uses an absolute path, and wouldn't work when I am switching computers/os/enviroments.
So my main question is how is this handled on larger projects? I have run into a couple other scenarios where I want to do this, even when I am using the "correct" structure and running a main function.
A test folder, which would import functions from another module for testing
Any situation when I want to import a function or class from another internal module
Is this just not commonly done in python? Am I missing something obvious regarding architecture?
I FINALLY found the answer.
Solution: Edit your settings.json with
"terminal.integrated.env.osx": {"PYTHONPATH": "${workspaceFolder}"}
Note - osx will need to be changed to "linux" or "windows" on those operating systems.
Documentation on the issue and solutions can be found here under the "Use of the PYTHONPATH variable" section.
There are alternative solutions of changing your PYTHONPATH by setting it in your .env, and then pointing to that in settings.json, however, neither worked for me. Solution ended up being to have VScode point the PYTHONPPATH to the current working directory, something that is done automatically in Pycharm.
I have multiple python projects which should use a number of shared files but I can not figure out how to do this in python.
If I just copy the file into the pyhton working directory it works fine with:
from HBonds import calculateHBondsForSeveralReplicas, compareSameShapeHBMatrices, calculateHBonds
But I do not want to copy it. I want to include it from: /home/b/data/pythonWorkspace/util/HBonds
For me it would make sense to do it like this (but it does not work):
from /home/b/data/pythonWorkspace/util/HBonds/HBonds.py import calculateHBondsForSeveralReplicas, compareSameShapeHBMatrices, calculateHBonds
How can I do this?
You have to make sure the PYTHONPATH includes the path to that directory as it was pointed out in previous answer.
Or you can use a nastier way: make it available at runtime with piece of code like this.
import os
import sys
folder = os.path.dirname('/home/b/data/pythonWorkspace/util/')
if dossier not in sys.path:
sys.path.append(folder)
from HBonds import HBonds
For 3rd-party libraries, it's best to install them the stock way - either to the system's site-packages or into a virtualenv.
For project(s) you're actively developing at the machine where it's running, a maintainable solution is to add their root directory(ies) to PYTHONPATH so that you can import <top_level_module>.<submodule>.<etc> from anywhere. That's what we did at my previous occupation. The main plus here is trivial code base update and switch.
Another way is to use relative imports, but it's intended for intra-package references, so that you don't have to reiterate the package's name everywhere. If many otherwise unrelated parts of the code use the same module(s), it's probably more convenient to make the shared part a separate package which would be a dependency for all of them.
I have a case for needing to add a path to a python package to sys.path (instead of its parent directory), but then refer to the package normally by name.
Maybe that's weird, but let me exemplify what I need and maybe you guys know how to achieve that.
I have all kind of experimental folders, modules, etc inside a path like /home/me/python.
Now I don't want to add that folder to my sys.path (PYTHONPATH) since there are experimental modules which names could clash with something useful.
But, inside /home/me/python I want to have a folder like pyutils. So I want to add /home/me/python/pyutils to PYTHONPATH, but, be able to refer to the package by its name pyutils...like I would have added /home/me/python to the path.
One helpful fact is that adding something to the python path is different from importing it into your interpreter. You can structure your modules and submodules such that names will not clash.
Look here regarding how to create modules. I read the documents, but I think that module layout takes a little learning-by-doing, which means creating your modules, and then importing them into scripts, and see if the importing is awkward or requires too much qualification.
Separately consider the python import system. When you import something you can use the "import ... as" feature to name it something different as you import, and thereby prevent naming clashes.
You seem to have already understood how you can change the PYTHONPATH using sys.path(), as documented here.
You have a number of options:
Make a new directory pyutilsdir, place pyutils in pyutilsdir,
and then add pyutilsdir to PYTHONPATH.
Move the experimental code outside of /home/me/python and add python to
your PYTHONPATH.
Rename the experimental modules so their names do not clash with
other modules. Then add python to PYTHONPATH.
Use a version control system like git or hg to make the
experimental modules available or unavailable as desired.
You could have a master branch without the experimental modules,
and a feature branch that includes them. With git, for example, you could switch between
the two with
git checkout [master|feature]
The contents of /home/me/python/pyutils (the git repo directory) would
change depending on which commit is checked out. Thus, using version control, you can keep the experimental modules in pyutils, but only make them present when you checkout the feature branch.
I'll answer to my own question since I got an idea while writing the question, and maybe someone will need that.
I added a link from that folder to my site-packages folder like that:
ln -s /home/me/python/pyutils /path/to/site-packages/pyutils
Then, since the PYTHONPATH contains the /path/to/site-packages folder, and I have a pyutils folder in it, with init.py, I can just import like:
from pyutils import mymodule
And the rest of the /home/me/python is not in the PYTHONPATH
I realize this may be considered a duplicate question to some of the other questions out there, but I've spent over an hour now reading through various pages and documentation and still don't understand what's going on here.
I'm trying to work with python files in multiple directories; I have essentially this:
myproject/
__init__.py
some_file.py
some_data.dat
tests/
__init__.py
test_some_file.py
test_some_file.py is run from the command line, as the name implies, intended to run the code contained in some_file.py, and needs to import it. However, I can't seem to do so.
I've tried:
from myproject import some_file
and also
from .. import some_file
I did manage to make it run using sys.path, but that doens't seem to be the correct way to do things based on what I've read.
Secondly, when I did make it run, using sys.path, I got an error that it couldn't find some_data.dat which is used by some_file.py.
This is a perennial question from Python programmers. The issue is that Python doesn't play nicely with scripts that are inside of packages. The situation has improved a bit over the last few releases, but it still doesn't do the right thing a lot of the time.
I think the best answer is to restrict where you run your test_some_file.py from, and use the Python interpreter's -m parameter. That is, change into the directory above myproject, and then run python -m myproject.tests.test_some_file. This is the only way that will work without messing around with sys.path.
This will allow either of your import lines to work correctly. PEP 8 currently recommends using absolute imports always, so the first version is probably better than the relative version using ...
For cases like yours, I add the directory of some_file.py to sys.path (temporarily).
Code:
import sys, os
dirname = os.path.dirname( # going up by 1 directory
os.path.dirname( # going up by 2 directories
sys.argv[0]))
sys.path.append(dirname)
import some_test
I do a lot of work on different projects (I'm a scientist) in a fairly standardised directory structure. e.g.:
project
/analyses/
/lib
/doc
/results
/bin
I put all my various utility scripts in /bin/ because cleanliness is next to godliness. However, I have to hard code paths (e.g. ../../x/y/z) and then I have to run things within ./bin/ or they break.
I've used Django and that has /manage.py which runs various django-things and automatically handles the path. I've also used fabric to run various user defined functions.
Question: How do I do something similar? and what's the best way? I can easily write something in /manage.py to inject the root dir into sys.path etc, but then I'd like to be able to do "./manage.py foo" which would run /bin/foo.py. Or is it possible to get fabric to call executables from a certain directory?
Basically - I want something easy and low maintenance. I want to be able to drop an executable script/file/whatever into ./bin/ and not have to deal with path issues or import issues.
What is the best way to do this?
Keep Execution at TLD
In general, try to keep your runtime at top-level. This will straighten out your imports tremendously.
If you have to do a lot of import addressing with relative imports, there's probably a
better way.
Modifying The Path
Other poster's have mentioned the PYTHONPATH. That's a great way to do it permanently in your shell.
If you don't want to/aren't able to manipulate the PYTHONPATH project path directly you can use sys.path to get yourself out of relative import hell.
Using sys.path.append
sys.path is just a list internally. You can append to it to add stuff to into your path.
Say I'm in /bin and there's a library markdown in lib/. You can append a relative paths with sys.path to import what you want.
import sys
sys.path.append('../lib')
import markdown
print markdown.markdown("""
Hello world!
------------
""")
Word to the wise: Don't get too crazy with your sys.path additions. Keep your schema simple to avoid yourself a lot confusion.
Overly eager imports can sometimes lead to cases where a python module needs to import itself, at which point execution will halt!
Using Packages and __init__.py
Another great trick is creating python packages by adding __init__.py files. __init__.py gets loaded before any other modules in the directory, so it's a great way to add imports across the entire directory. This makes it an ideal spot to add sys.path hackery.
You don't even need to necessarily add anything to the file. It's sufficient to just do touch __init__.py at the console to make a directory a package.
See this SO post for a more concrete example.
In a shell script that you source (not run) in your current shell you set the following environment variables:
PATH=$PATH:$PROJECTDIR/bin
PYTHONPATH=$PROJECTDIR/lib
Then you put your Python modules and package tree in your projects ./lib directory. Python automatically adds the PYTHONPATH environment variable to sys.path.
Then you can run any top-level script from the shell without specifying the path, and any imports from your library modules are looked for in the lib directory.
I recommend very simple top-level scripts, such as:
#!/usr/bin/python
import sys
import mytool
mytool.main(sys.argv)
Then you never have to change that, you just edit the module code, and also benefit from the byte-code caching.
You can easily achieve your goals by creating a mini package that hosts each one of your projects. Use paste scripts to create a simple project skeleton. And to make it executable, just install it via setup.py develop. Now your bin scripts just need to import the entry point to this package and execute it.