Desiring to improve my Python style, I ran a PEP-8 style checker on one of my script and it complained about something I don't know how to fix. The prologue of the script is something like:
#! /bin/env python3
import sys
import os
exe_name = os.path.basename(os.path.realpath(__file__))
bin_dir = os.path.dirname(os.path.realpath(__file__))
inst_dir = os.path.dirname(bin_dir)
sys.path.insert(0, inst_dir+'/path/to/packages')
import mypackage.mymodule
and the style checker complain on the import mymodule line, stating that it should be a top of file. But I obviously can't move it before setting the path where it will be found. Is there a good way to achieve this (mandating an environment variable or a shell wrapper are not what I find better than my current code) while respecting PEP-8 recommendations at the same time?
If you want to avoid path manipulation, you may be able to do so by using the under-known .pth feature.
sys.path should begin with the directory containing the main program either by name or by reference as ''. I assume that the file importing mymodule is not part of mypackage, so that the '' entry is not useful for importing mymodule.
sys.path should end with the site-packages directory for the executing binary. That is the normal place for added packages. If you do not want to move mypackage into site-packages, you can extend the latter 'vitually' by putting a mystuff.pth file in it. It should contain one line: the path to the directory containing mypackage. Call it myprojects. Then mypackage and any other package in myprojects can be imported as if they were in site-packages.
One advantage of .pth files is that you can put identical copies in multiple site-packages directories. For instance, I have multiple projects in F:/python. I have multiple versions of Python installed. So I have put python.pth containing that one line in the site-packages for each.
The best strategy would be to put the sys.path related code in separate file and import it in working code file.
So, I will split above code in two files. One named my_module.py and other named working_module.py
Now my_module will contain below lines
import sys
import os
exe_name = os.path.basename(os.path.realpath(__file__))
bin_dir = os.path.dirname(os.path.realpath(__file__))
inst_dir = os.path.dirname(bin_dir)
sys.path.insert(0, inst_dir+'/path/to/packages')
and working_module will contain
import my_module
import mypackage.mymodule
because we are importing my_module before mypackage, it will execute the path related code before and your package will be available in path.
You can use importlib module (python 3.5) or imp for python 2.7 to load mypackage.mymodule programatically. Both have the same purpose:
mechanisms used to implement the import statement
This question might help you more:
How to import a module given the full path?
https://docs.python.org/3/library/importlib.html#examples
https://docs.python.org/2/library/imp.html
I have the following directory structure
project/
bin/
script
stuff/
__init__.py
mainfile.py
Inside of script, I have the following to enable command line execution:
#!/usr/bin/env python
from stuff import mainfile
This works, but I would have expected needing to jump up one level...
from ..stuff import mainfile
or
from project.stuff import mainfile
What am I missing here?
Actually none of your examples should work out of the box.
Let's modify bin/script.py somewhat:
#! /usr/bin/env python
import sys
print sys.path
from stuff import mainfile
This should yield something like
['.../project/bin', ...]
Traceback (most recent call last):
File "bin/script.py", line 6, in <module>
from stuff import mainfile
ImportError: No module named stuff
Only the directory of the script (not the current directory) is added to sys.path automatically:
As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter.* If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory is inserted before the entries inserted as a result of PYTHONPATH.
Hence there's no stuff module on sys.path. I'm not sure about your environment setup, but this is the canonical result when no additional parameters are set up (e.g. PYTHONPATH).
Similarily,
from ..stuff import mainfile
will result in the classic ValueError: Attempted relative import in non-package. You are informed of the fact that you can only do relative imports relative to actual modules. Since inside script .. does not refer to a module (because the script itself is the top level module so to say), a relative import does not work here. In a manner of speaking from Python's perspective, there is no module above the script and thus .. does not refer to something tangible when used in the context of the script.
Note that this also means that it does not help to only make project and project/bin into modules themselves by dropping __init__.py marker files. Relative imports to a parent of the script are only possible if the parent of the script is actually something python has a concept of.
This is one of the reasons, why the -m command line switch exists making it possible to run a module from the command line. For example, given the above relative import
python -m project.bin.script
does the trick, but only if executed from the proper directory (projectdir/..).
That problem is even worse with
from project.stuff import mainfile
because the project directory is only automatically on sys.path when you start a script from the directory above project and do not specify a main script to run:
cd <projectdir>/..
python -m project.bin.script
# works
cd <projectdir>
python -m bin.script
# does not work, because `sys.path` starts with <projectdir>
# and it's `stuff.mainfile` now, not `project.stuff.mainfile`.
If you want to import modules from project in your script, fix up sys.path to your needs:
import sys
import os
sys.path.insert(0, os.path.dirname(sys.path[0]))
from stuff import mainfile
You need to first add the parent directory into sys.path. Try the following:
# This is a file in bin/ directory.
this_file_path = os.path.dirname(__file__)
sys.path.append(os.path.join(this_file_path, '..'))
import stuff.mainfile
I had a small problem with importing a script that was in a parent folder, but I managed to resolve it using:
import sys
sys.path.append("../")
My directory is like this:
Data
|->->code
|->->script1.py
|->->->->subfolder
|->->->script2.py
When I run script2 (which imports script1) from the subfolder directory, the script runs without problems. But if I try to run script2 from the code directory using:
:~ ./subfolder/script2.py
I get an error :
ImportError: No module named script1
I tried using relative imports but because my code is not structured in packages it doesn't work. Is there a way I can run script2 from both directories (the parent and the child) and still be able to import script1 everytime?
Thank you in advance,
Georgi Nikolov
EDIT: Ok, after I read through all the suggestions, I did a "simple" hack which is quite ugly in my opinion but works quite well:
import sys
parent_folder = sys.path[0].split("/subfolder")[0]
sys.path.append(parent_folder)
import script1
Now I can even call script2 from the root and it will manage to import script1
To import a module that is up a level, you can use this.
import os, sys
sys.path.append(os.path.join(os.path.dirname(__file__), ".."))
An Explanation
__file__ # The full path to your running file.
os.path.dirname # See below. (1)
os.path.join # See below. (2)
sys.path.append # See below. (3)
".." # Universal for "up a level".
os.path.dirname(path) -
Return the directory name of pathname path.
os.path.join(path, *paths) -
Join one or more path components intelligently.
sys.path -
A list of strings that specifies the search path for modules.
You can add a string with the append method.
You should use absolute paths:
import sys, os
HERE = os.path.abspath(os.path.dirname(__file__))
sys.path.append(os.path.join(HERE, ".."))
But better do not do this: Guido views running scripts within a package as an anti-pattern
You should put the standalone scripts in the root folder of the project. A script should not be used as a module and as the main script.
If you cannot move the script to the root, make another bootstrap script at the root which will import your service script running a dedicated function in it.
How do I import a Python module given its relative path?
For example, if dirFoo contains Foo.py and dirBar, and dirBar contains Bar.py, how do I import Bar.py into Foo.py?
Here's a visual representation:
dirFoo\
Foo.py
dirBar\
Bar.py
Foo wishes to include Bar, but restructuring the folder hierarchy is not an option.
Assuming that both your directories are real Python packages (do have the __init__.py file inside them), here is a safe solution for inclusion of modules relatively to the location of the script.
I assume that you want to do this, because you need to include a set of modules with your script. I use this in production in several products and works in many special scenarios like: scripts called from another directory or executed with python execute instead of opening a new interpreter.
import os, sys, inspect
# realpath() will make your script run, even if you symlink it :)
cmd_folder = os.path.realpath(os.path.abspath(os.path.split(inspect.getfile( inspect.currentframe() ))[0]))
if cmd_folder not in sys.path:
sys.path.insert(0, cmd_folder)
# Use this if you want to include modules from a subfolder
cmd_subfolder = os.path.realpath(os.path.abspath(os.path.join(os.path.split(inspect.getfile( inspect.currentframe() ))[0],"subfolder")))
if cmd_subfolder not in sys.path:
sys.path.insert(0, cmd_subfolder)
# Info:
# cmd_folder = os.path.dirname(os.path.abspath(__file__)) # DO NOT USE __file__ !!!
# __file__ fails if the script is called in different ways on Windows.
# __file__ fails if someone does os.chdir() before.
# sys.argv[0] also fails, because it doesn't not always contains the path.
As a bonus, this approach does let you force Python to use your module instead of the ones installed on the system.
Warning! I don't really know what is happening when current module is inside an egg file. It probably fails too.
Be sure that dirBar has the __init__.py file -- this makes a directory into a Python package.
You could also add the subdirectory to your Python path so that it imports as a normal script.
import sys
sys.path.insert(0, <path to dirFoo>)
import Bar
import os
import sys
lib_path = os.path.abspath(os.path.join(__file__, '..', '..', '..', 'lib'))
sys.path.append(lib_path)
import mymodule
Just do simple things to import the .py file from a different folder.
Let's say you have a directory like:
lib/abc.py
Then just keep an empty file in lib folder as named
__init__.py
And then use
from lib.abc import <Your Module name>
Keep the __init__.py file in every folder of the hierarchy of the import module.
If you structure your project this way:
src\
__init__.py
main.py
dirFoo\
__init__.py
Foo.py
dirBar\
__init__.py
Bar.py
Then from Foo.py you should be able to do:
import dirFoo.Foo
Or:
from dirFoo.Foo import FooObject
Per Tom's comment, this does require that the src folder is accessible either via site_packages or your search path. Also, as he mentions, __init__.py is implicitly imported when you first import a module in that package/directory. Typically __init__.py is simply an empty file.
The easiest method is to use sys.path.append().
However, you may be also interested in the imp module.
It provides access to internal import functions.
# mod_name is the filename without the .py/.pyc extention
py_mod = imp.load_source(mod_name,filename_path) # Loads .py file
py_mod = imp.load_compiled(mod_name,filename_path) # Loads .pyc file
This can be used to load modules dynamically when you don't know a module's name.
I've used this in the past to create a plugin type interface to an application, where the user would write a script with application specific functions, and just drop thier script in a specific directory.
Also, these functions may be useful:
imp.find_module(name[, path])
imp.load_module(name, file, pathname, description)
This is the relevant PEP:
http://www.python.org/dev/peps/pep-0328/
In particular, presuming dirFoo is a directory up from dirBar...
In dirFoo\Foo.py:
from ..dirBar import Bar
The easiest way without any modification to your script is to set PYTHONPATH environment variable. Because sys.path is initialized from these locations:
The directory containing the input script (or the current
directory).
PYTHONPATH (a list of directory names, with the same
syntax as the shell variable PATH).
The installation-dependent default.
Just run:
export PYTHONPATH=/absolute/path/to/your/module
You sys.path will contains above path, as show below:
print sys.path
['', '/absolute/path/to/your/module', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client', '/usr/lib/python2.7/dist-packages/ubuntuone-client', '/usr/lib/python2.7/dist-packages/ubuntuone-control-panel', '/usr/lib/python2.7/dist-packages/ubuntuone-couch', '/usr/lib/python2.7/dist-packages/ubuntuone-installer', '/usr/lib/python2.7/dist-packages/ubuntuone-storage-protocol']
In my opinion the best choice is to put __ init __.py in the folder and call the file with
from dirBar.Bar import *
It is not recommended to use sys.path.append() because something might gone wrong if you use the same file name as the existing python package. I haven't test that but that will be ambiguous.
The quick-and-dirty way for Linux users
If you are just tinkering around and don't care about deployment issues, you can use a symbolic link (assuming your filesystem supports it) to make the module or package directly visible in the folder of the requesting module.
ln -s (path)/module_name.py
or
ln -s (path)/package_name
Note: A "module" is any file with a .py extension and a "package" is any folder that contains the file __init__.py (which can be an empty file). From a usage standpoint, modules and packages are identical -- both expose their contained "definitions and statements" as requested via the import command.
See: http://docs.python.org/2/tutorial/modules.html
from .dirBar import Bar
instead of:
from dirBar import Bar
just in case there could be another dirBar installed and confuse a foo.py reader.
For this case to import Bar.py into Foo.py, first I'd turn these folders into Python packages like so:
dirFoo\
__init__.py
Foo.py
dirBar\
__init__.py
Bar.py
Then I would do it like this in Foo.py:
from .dirBar import Bar
If I wanted the namespacing to look like Bar.whatever, or
from . import dirBar
If I wanted the namespacing dirBar.Bar.whatever. This second case is useful if you have more modules under the dirBar package.
Add an __init__.py file:
dirFoo\
Foo.py
dirBar\
__init__.py
Bar.py
Then add this code to the start of Foo.py:
import sys
sys.path.append('dirBar')
import Bar
Relative sys.path example:
# /lib/my_module.py
# /src/test.py
if __name__ == '__main__' and __package__ is None:
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../lib')))
import my_module
Based on this answer.
Well, as you mention, usually you want to have access to a folder with your modules relative to where your main script is run, so you just import them.
Solution:
I have the script in D:/Books/MyBooks.py and some modules (like oldies.py). I need to import from subdirectory D:/Books/includes:
import sys,site
site.addsitedir(sys.path[0] + '\\includes')
print (sys.path) # Just verify it is there
import oldies
Place a print('done') in oldies.py, so you verify everything is going OK. This way always works because by the Python definition sys.path as initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter.
If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory is inserted before the entries inserted as a result of PYTHONPATH.
Another solution would be to install the py-require package and then use the following in Foo.py
import require
Bar = require('./dirBar/Bar')
Simply you can use: from Desktop.filename import something
Example:
given that the file is name test.py in directory
Users/user/Desktop , and will import everthing.
the code:
from Desktop.test import *
But make sure you make an empty file called "__init__.py" in that directory
Here's a way to import a file from one level above, using the relative path.
Basically, just move the working directory up a level (or any relative location), add that to your path, then move the working directory back where it started.
#to import from one level above:
cwd = os.getcwd()
os.chdir("..")
below_path = os.getcwd()
sys.path.append(below_path)
os.chdir(cwd)
I'm not experienced about python, so if there is any wrong in my words, just tell me. If your file hierarchy arranged like this:
project\
module_1.py
module_2.py
module_1.py defines a function called func_1(), module_2.py:
from module_1 import func_1
def func_2():
func_1()
if __name__ == '__main__':
func_2()
and you run python module_2.py in cmd, it will do run what func_1() defines. That's usually how we import same hierarchy files. But when you write from .module_1 import func_1 in module_2.py, python interpreter will say No module named '__main__.module_1'; '__main__' is not a package. So to fix this, we just keep the change we just make, and move both of the module to a package, and make a third module as a caller to run module_2.py.
project\
package_1\
module_1.py
module_2.py
main.py
main.py:
from package_1.module_2 import func_2
def func_3():
func_2()
if __name__ == '__main__':
func_3()
But the reason we add a . before module_1 in module_2.py is that if we don't do that and run main.py, python interpreter will say No module named 'module_1', that's a little tricky, module_1.py is right beside module_2.py. Now I let func_1() in module_1.py do something:
def func_1():
print(__name__)
that __name__ records who calls func_1. Now we keep the . before module_1 , run main.py, it will print package_1.module_1, not module_1. It indicates that the one who calls func_1() is at the same hierarchy as main.py, the . imply that module_1 is at the same hierarchy as module_2.py itself. So if there isn't a dot, main.py will recognize module_1 at the same hierarchy as itself, it can recognize package_1, but not what "under" it.
Now let's make it a bit complicated. You have a config.ini and a module defines a function to read it at the same hierarchy as 'main.py'.
project\
package_1\
module_1.py
module_2.py
config.py
config.ini
main.py
And for some unavoidable reason, you have to call it with module_2.py, so it has to import from upper hierarchy.module_2.py:
import ..config
pass
Two dots means import from upper hierarchy (three dots access upper than upper,and so on). Now we run main.py, the interpreter will say:ValueError:attempted relative import beyond top-level package. The "top-level package" at here is main.py. Just because config.py is beside main.py, they are at same hierarchy, config.py isn't "under" main.py, or it isn't "leaded" by main.py, so it is beyond main.py. To fix this, the simplest way is:
project\
package_1\
module_1.py
module_2.py
config.py
config.ini
main.py
I think that is coincide with the principle of arrange project file hierarchy, you should arrange modules with different function in different folders, and just leave a top caller in the outside, and you can import how ever you want.
This also works, and is much simpler than anything with the sys module:
with open("C:/yourpath/foobar.py") as f:
eval(f.read())
Call me overly cautious, but I like to make mine more portable because it's unsafe to assume that files will always be in the same place on every computer. Personally I have the code look up the file path first. I use Linux so mine would look like this:
import os, sys
from subprocess import Popen, PIPE
try:
path = Popen("find / -name 'file' -type f", shell=True, stdout=PIPE).stdout.read().splitlines()[0]
if not sys.path.__contains__(path):
sys.path.append(path)
except IndexError:
raise RuntimeError("You must have FILE to run this program!")
That is of course unless you plan to package these together. But if that's the case you don't really need two separate files anyway.