I have a Python script that uses built-in modules but also imports a number of custom modules that exist in the same directory as the main script itself.
For example, I would call
python agent.py
and agent.py has a number of imports, including:
import checks
where checks is in a file in the same directory as agent.py
agent/agent.py
agent/checks.py
When the current working directory is agent/ then everything is fine. However, if I call agent.py from any other directory, it is obviously unable to import checks.py and so errors.
How can I ensure that the custom modules can be imported regardless of where the agent.py is called from e.g.
python /home/bob/scripts/agent/agent.py
Actually your example works because checks.py is in the same directory as agent.py, but say checks.py was in the preceeding directory, eg;
agent/agent.py
checks.py
Then you could do the following:
path = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not path in sys.path:
sys.path.insert(1, path)
del path
Note the use of __file__.
You should NOT need to fiddle with sys.path. To quote from the Python 2.6 docs for sys.path:
As initialized upon program startup, the first item of this list,
path[0], is the directory containing the script that was used to
invoke the Python interpreter. If the script directory is not
available (e.g. if the interpreter is invoked interactively or if the
script is read from standard input), path[0] is the empty string,
which directs Python to search modules in the current directory first.
Notice that the script directory is inserted before the entries
inserted as a result of PYTHONPATH.
=== amod.py ===
def whoami():
return __file__
=== ascript.py ===
import sys
print "sys.argv", sys.argv
print "sys.path", sys.path
import amod
print "amod __file__", amod.whoami()
=== result of running ascript.py from afar ===
C:\somewhere_else>\python26\python \junk\timport\ascript.py
sys.argv ['\\junk\\timport\\ascript.py']
sys.path ['C:\\junk\\timport', 'C:\\WINDOWS\\system32\\python26.zip', SNIP]
amod __file__ C:\junk\timport\amod.py
and if it's re-run, the last line changes of course to ...amod.pyc. This appears not to be a novelty, it works with Python 2.1 and 1.5.2.
Debug hints for you: Try two simple files like I have. Try running Python with -v and -vv. Show us the results of your failing tests, including full traceback and error message, and your two files. Tell us what platform you are running on, and what version of Python. Check the permissions on the checks.py file. Is there a checks.something_else that's causing interference?
You need to add the path to the currently executing module to the sys.path variable. Since you called it on the command line, the path to the script will always be in sys.argv[0].
import sys
import os
sys.path.append(os.path.split(sys.argv[0])[0])
Now when import searches for the module, it will also look in the folder that hosts the agent.py file.
There are several ways to add things to the PYTHONPATH.
Read http://docs.python.org/library/site.html
Set the PYTHONPATH environment variable prior to running your script.
You can do this python -m agent to run agent.py from your PYTHONPATH.
Create .pth files in your lib/site-packages directory.
Install your modules in lib/site-packages.
I think you should consider making the agent directory into a proper Python package. Then you place this package anywhere on the python path, and you can import checks as
from agent import checks
See http://docs.python.org/tutorial/modules.html
If you know full path to check.py use this recipe (http://code.activestate.com/recipes/159571/)
If you want to add directory to system path -- this recipe (http://code.activestate.com/recipes/52662/). In this case I have to determine application directory (sys.argv[0]) an pass this value to AddSysPath function. If you want to look at production sample please leave a comment on this thread so I post it later.
Regards.
To generalize my understanding of your goal, you want to be able to import custom packages using import custom_package_name no matter where you are calling python from and no matter where your python script is located.
A number of answers mention what I'm about to describe, but I feel like most of the answers assume a lot of previous knowledge. I'll try to be as explicit as I can.
To achieve the goal of allowing custom packages to be imported via the import statement, they have to be discoverable somewhere through the path that python uses to search for packages. Python actually uses multiple paths, but we'll only focus on the one that can be found by combining the output of sys.prefix (in your python interpreter) with /lib/pythonX.Y/site-packages (or lib/site-packages if you're using windows) where X.Y is your python version.
Concretely, find the path that your python uses by running:
import sys
your_path = sys.prefix + '/lib/pythonX.Y/site-packages'
print(your_path)
This path should look something like /usr/local/lib/python3.5/site-packages if you're using python 3.5, but it could be much different depending on your setup.
Python uses this path (and a few others) to find the packages that you want to import. So what you can do is put your custom packages in the /usr/local/lib/python3.5/site-packages folder. Don't forget to add an init.py file to the folder.
To be concrete again, in terminal type:
cd your_path
cp path_to_custom_package/custom_package ./
Now you should be able to import everything your custom package like you would if the package was located in the same directory (i.e. import package.subpackage for each subpackage file in your package should work).
Related
I have a project directory that looks like this
/work
/venv
/project1
app.py
/package1
/package2
etc
where the script app.py runs the project I'm working on. When I run
python -m site from the /project1 directory I get, as expected
sys.path = [
'/work/project1',
plus the relevant /lib/pythonX.Y paths. However, when running app.py (from within project1) this import
from package1 import ...
fails with a ModuleNotFoundError, whereas from .package1 import ... works fine since it tells python to search directory that app.py is in. So, I added this to the beginning of app.py
import sys
print(sys.path)
and the result is
sys.path = [
'/work',
Instead of /work/project1 the directory that is being searched when importing to app.py is /work. What is the cause of this discrepancy between python -m site and print(sys.path) and what is the best way to go about making sure that /work/project1 is always part of sys.path?
I would like to avoid using something like site.addsitedir() and perhaps use a .pth file instead. From what I've read though a .pth file belongs in /lib/pythonX.Y/sitepackages, but I also need this solution to be system independent so that a collaborator who clones /project1 from github wont have to add their own .pth file.
I don't know if it applies to you, but I've helped a lot of others by pointing out what our shop does. We don't worry about the path upon execution at all. All we rely upon is that from the initial script location, we know where all the dependencies are. Then we add directories to sys.path by computing their positions relative the directory containing the initial script, which is always available.
We put this logic in a file named init.py in the same dir as the main script. Here's what one looks like:
repoPath = os.path.dirname(__file__) + "/../../Development/repo1"
sys.path.append(repoPath + "/common/common-pyutil")
sys.path.append(repoPath + "/common/common-pyutil/thirdparty")
and then in the main script do import init. We have directories with lots of scripts in them, and they all just do import init to have their sys.path fixed up to find all of our utility modules and such.
When we run a python script/module .py file, then the interpreter looks for any imports in the directory in which the running script is located, and not in the current working directory.
When we run python module using the -m switch then it loads the current working directory into the path.
However, when we run python package using the -m switch then which directory is loaded into the path? The current directory or directory containing the package or the package itself?
Can someone throw light on this concept.
The current directory.
Seems likes the behavior is similar between a module and a package in this regard. The documentation on running with -m:
https://docs.python.org/3/using/cmdline.html#cmdoption-m
... the current directory will be added to the start of sys.path.
Running a package with -m will add the current directory to the path, not the package's directory or the directory containing the package.
From the -m docs:
As with the -c option, the current directory will be added to the start of sys.path.
This does not depend on whether the specified module is a package or a normal module. (It could even be a namespace package, which does not have a single directory for its contents or a single containing directory.)
Note that the current directory is added to the path before attempting to resolve the name of the package or module to run, which is why it is even possible to use -m to run modules and packages in the current directory.
From my last experience it is from the directory containing the package.
However to check that, run this to see all directories where python would look into:
import sys
print('\n'.join(sys.path))
Tip: If you would like to load custom .py files from other directories, do:
import sys
sys.path.insert(1, "<path">)
OR
import sys
sys.path.append("<path>")
I have a large repository with some fixed structure and I have extended it by some folders and python scripts to add extra functionality to it as a whole. The structure looks as follows:
toplevelfolder
featureA
someModuleA.py
__ init __.py
featureB
someModuleB.py
__ init __.py
application
__ init __.py
app.py
Now someModuleA.py and someModuleB.py can be invoked via app.py but at the same time also have be able to be invoked directly, however this invocation must come from the toplevelfolder for the relative paths in the file to resolve correctly, i.e. via python ./featureA/someModuleA.py.
This all works well, but now I need some function definitions from someModuleB in someModuleA and hence I want to import this module. I have tried both absolute and relative imports, but both fail with different errors, the absolute import with
from toplevelfolder.featureA import someModuleA as A
# ModuleNotFoundError: No module named 'toplevelfolder'
and the relative import with
from toplevelfolder.featureA import someModuleA as A
# ImportError: attempted relative import with no known parent package
Now I can see that the relative import would cause problems when python is invoked from the toplevelfolder, as .. would represent the latter's parent directory, rather than the parent directory of featureA. However, I cannot get a hold of the first error message, especially since toplevelfolder should not be a module but a package.
Is there another way to import in Python that I'm not aware of, if possibly without modifying PYTHONPATH or sys.path or something like that?
Not 100% sure on what the goal is here. My advice would be:
Identify clearly what you want your top level modules and packages to be.
Make all imports absolute.
Either:
make your project a real installable project, so that those top level modules and packages are installed in the environment's site-packages directory;
or make sure that the current working directory is the one containing the top level modules and packages.
Make sure to call your code via the executable module or package method instead of the script method, if the "entry point" you want to execute is part of a package
DO (executable module or package):
path/to/pythonX.Y -m toplevelpackage.module
path/to/pythonX.Y -m toplevelpackage.subpackage (assuming there is a toplevelpackage/subpackage/__main__.py file)
DON'T (script within a package):
path/to/pythonX.Y toplevelpackage/module.py
(Optional) Later on, once it all works well and everything is under control, you might decide to change some or all imports to relative. (If things are done right, I believe it could be possible to make it so that it is possible to call the executable modules from any level within the directory structure as the current working directory.)
References:
Old reference, possibly outdated, but assuming I interpreted it right, it says that running scripts that live in a package is an anti pattern, and one should use python -m package.module instead: https://mail.python.org/pipermail/python-3000/2007-April/006793.html -- https://www.python.org/dev/peps/pep-3122/
Don't include the top level directory. In featureB.someModuleB:
from featureA.someModuleA import two
Sample directory.
Try pasting this above your import:
import os,sys,inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0,parentdir)
Then you should be able to import a file from the parent-folder.
I have just joined a project with a rather large existing code base. We develop in linux and do not use and IDE. We run through the command line. I'm trying to figure out how to get python to search for the right path when I run project modules. For instance, when I run something like:
python someprojectfile.py
I get
ImportError: no module named core.'somemodule'
I get this for all of my imports to I assume it's an issue with the path.
How do I get Python to search ~/codez/project/ and all the files and folders for *.py files during import statements?
There are a few possible ways to do this:
Set the environment variable PYTHONPATH to a colon-separated list of directories to search for imported modules.
In your program, use sys.path.append('/path/to/search') to add the names of directories you want Python to search for imported modules. sys.path is just the list of directories Python searches every time it gets asked to import a module, and you can alter it as needed (although I wouldn't recommend removing any of the standard directories!). Any directories you put in the environment variable PYTHONPATH will be inserted into sys.path when Python starts up.
Use site.addsitedir to add a directory to sys.path. The difference between this and just plain appending is that when you use addsitedir, it also looks for .pth files within that directory and uses them to possibly add additional directories to sys.path based on the contents of the files. See the documentation for more detail.
Which one of these you want to use depends on your situation. Remember that when you distribute your project to other users, they typically install it in such a manner that the Python code files will be automatically detected by Python's importer (i.e. packages are usually installed in the site-packages directory), so if you mess with sys.path in your code, that may be unnecessary and might even have adverse effects when that code runs on another computer. For development, I would venture a guess that setting PYTHONPATH is usually the best way to go.
However, when you're using something that just runs on your own computer (or when you have nonstandard setups, e.g. sometimes in web app frameworks), it's not entirely uncommon to do something like
import sys
from os.path import dirname
sys.path.append(dirname(__file__))
You should also read about python packages here: http://docs.python.org/tutorial/modules.html.
From your example, I would guess that you really have a package at ~/codez/project. The file __init__.py in a python directory maps a directory into a namespace. If your subdirectories all have an __init__.py file, then you only need to add the base directory to your PYTHONPATH. For example:
PYTHONPATH=$PYTHONPATH:$HOME/adaifotis/project
In addition to testing your PYTHONPATH environment variable, as David explains, you can test it in python like this:
$ python
>>> import project # should work if PYTHONPATH set
>>> import sys
>>> for line in sys.path: print line # print current python path
...
I know this thread is a bit old, but it took me some time to get to the heart of this, so I wanted to share.
In my project, I had the main script in a parent directory, and, to differentiate the modules, I put all the supporting modules in a sub-folder called "modules". In my main script, I import these modules like this (for a module called report.py):
from modules.report import report, reportError
If I call my main script, this works. HOWEVER, I wanted to test each module by including a main() in each, and calling each directly, as:
python modules/report.py
Now Python complains that it can't find "a module called modules". The key here is that, by default, Python includes the folder of the script in its search path, BUT NOT THE CWD. So what this error says, really, is "I can't find a modules subfolder". The is because there is no "modules" subdirectory from the directory where the report.py module resides.
I find that the neatest solution to this is to append the CWD in Python search path by including this at the top:
import sys
sys.path.append(".")
Now Python searches the CWD (current directory), finds the "modules" sub-folder, and all is well.
The easiest way I find is to create a file any_name.pth and put it in your folder \Lib\site-packages. You should find that folder wherever python is installed.
In that file, put a list of directories where you want to keep modules for importing. For instance, make a line in that file like this:
C:\Users\example\...\example
You will be able to tell it works by running this in python:
import sys
for line in sys.path:
print line
You will see your directory printed out, amongst others from where you can also import. Now you can import a mymodule.py file that sits in that directory as easily as:
import mymodule
This will not import subfolders. For that you could imagine creating a python script to create a .pth file containing all sub folders of a folder you define. Have it run at startup perhaps.
I read this question looking for an answer, and didn't like any of them.
So I wrote a quick and dirty solution. Just put this somewhere on your sys.path, and it'll add any directory under folder (from the current working directory), or under abspath:
#using.py
import sys, os.path
def all_from(folder='', abspath=None):
"""add all dirs under `folder` to sys.path if any .py files are found.
Use an abspath if you'd rather do it that way.
Uses the current working directory as the location of using.py.
Keep in mind that os.walk goes *all the way* down the directory tree.
With that, try not to use this on something too close to '/'
"""
add = set(sys.path)
if abspath is None:
cwd = os.path.abspath(os.path.curdir)
abspath = os.path.join(cwd, folder)
for root, dirs, files in os.walk(abspath):
for f in files:
if f[-3:] in '.py':
add.add(root)
break
for i in add: sys.path.append(i)
>>> import using, sys, pprint
>>> using.all_from('py') #if in ~, /home/user/py/
>>> pprint.pprint(sys.path)
[
#that was easy
]
And I like it because I can have a folder for some random tools and not have them be a part of packages or anything, and still get access to some (or all) of them in a couple lines of code.
the first task is to find out where the current environment gets its standard libraries from. And that's where I would store my OOP modules.
Import numpy
numpy.__file__
You get a catalog on your environment. For another computer, it is a good idea to create a separate directory for OOP modules.
Import sys
sys.path.append('F:/Python/My_libra/')
New option for old question.
Installing fail2ban package on Debian, looks like it's hardcoded to install on /usr/lib/python3/dist-packages/fail2ban a path not on python3 sys.path.
> python3
Python 3.7.3 (v3.7.3:ef4ec6ed12, Jun 25 2019, 18:51:50)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/lib/python3.7/site-packages']
>>>
so, instead of just copying, I (bash) linked the library to newer versions. Future updates to the original app, will also be automatically applied to the linked versions.
if [ -d /usr/lib/python3/dist-packages/fail2ban ]
then
for d in /usr/lib/python3.*
do
[ -d ${d}/fail2ban ] || \
ln -vs /usr/lib/python3/dist-packages/fail2ban ${d}/
done
fi
I'm developing/testing a package in my local directory. I want to import it in the interpreter (v2.5), but sys.path does not include the current directory. Right now I type in sys.path.insert(0,'.'). Is there a better way?
Also,
from . import mypackage
fails with this error:
ValueError: Attempted relative import in non-package
You can use relative imports only from in a module that was in turn imported as part of a package -- your script or interactive interpreter wasn't, so of course from . import (which means "import from the same package I got imported from") doesn't work. import mypackage will be fine once you ensure the parent directory of mypackage is in sys.path (how you managed to get your current directory away from sys.path I don't know -- do you have something strange in site.py, or...?)
To get your current directory back into sys.path there is in fact no better way than putting it there.
Keep it simple:
try:
from . import mymodule # "myapp" case
except:
import mymodule # "__main__" case
See the documentation for sys.path:
http://docs.python.org/library/sys.html#sys.path
To quote:
If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first.
So, there's no need to monkey with sys.path if you're starting the python interpreter from the directory containing your module.
Also, to import your package, just do:
import mypackage
Since the directory containing the package is already in sys.path, it should work fine.
If you want to run an unmodified python script so it imports libraries from a specific local directory you can set the PYTHONPATH environment variable - e.g. in bash:
export PYTHONPATH=/home/user/my_libs
python myscript.py
If you just want it to import from the current working directory use the . notation:
export PYTHONPATH=.
python myscript.py
Inside a package if there is setup.py, then better to install it
pip install -e .
A simple way to make it work is to run your script from the parent directory using python's -m flag, e.g. python -m packagename.scriptname. Obviously in this situation you need an __init__.py file to turn your directory into a package.
Using sys.path should include current directory already.
Try:
import .
or:
from . import sth
however it may be not a good practice, so why not just use:
import mypackage
A bit late to the party, but this is what worked for me:
>>> import sys
>>> sys.path.insert(0, '')
Apparently, if there is an empty string, Python knows that it should look in the current directory. I did not have the empty string in sys.path, which caused this error.
Speaking for python3.. I wanted to use an improved version of a library that's installed in my environment. There are some extra print statements it makes to show that it and not the original lib is being used.
I placed the lib's folder next to the python script. Ran the script.. it ran with the local lib with the modifications.
Removed the folder and ran it again - this time it ran with the installed lib.
So, solution was simple : place the lib's folder (with same name as in your import statement) in your project folder. That does the job, at least at my end.
This is on a standard Linux Mint 20.04 system, with a python 3.8 virutal environment activated (so "(py3.8)" appears in my terminal when I'm in the virtual env)
You can import package_name if the package is a module: this needs you have init.py under the package and things that you want to use are needed to import in the init.py
Or if you want to import class under the package, you can use from package_name import class_name