I have the following directory structure
project/
bin/
script
stuff/
__init__.py
mainfile.py
Inside of script, I have the following to enable command line execution:
#!/usr/bin/env python
from stuff import mainfile
This works, but I would have expected needing to jump up one level...
from ..stuff import mainfile
or
from project.stuff import mainfile
What am I missing here?
Actually none of your examples should work out of the box.
Let's modify bin/script.py somewhat:
#! /usr/bin/env python
import sys
print sys.path
from stuff import mainfile
This should yield something like
['.../project/bin', ...]
Traceback (most recent call last):
File "bin/script.py", line 6, in <module>
from stuff import mainfile
ImportError: No module named stuff
Only the directory of the script (not the current directory) is added to sys.path automatically:
As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter.* If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory is inserted before the entries inserted as a result of PYTHONPATH.
Hence there's no stuff module on sys.path. I'm not sure about your environment setup, but this is the canonical result when no additional parameters are set up (e.g. PYTHONPATH).
Similarily,
from ..stuff import mainfile
will result in the classic ValueError: Attempted relative import in non-package. You are informed of the fact that you can only do relative imports relative to actual modules. Since inside script .. does not refer to a module (because the script itself is the top level module so to say), a relative import does not work here. In a manner of speaking from Python's perspective, there is no module above the script and thus .. does not refer to something tangible when used in the context of the script.
Note that this also means that it does not help to only make project and project/bin into modules themselves by dropping __init__.py marker files. Relative imports to a parent of the script are only possible if the parent of the script is actually something python has a concept of.
This is one of the reasons, why the -m command line switch exists making it possible to run a module from the command line. For example, given the above relative import
python -m project.bin.script
does the trick, but only if executed from the proper directory (projectdir/..).
That problem is even worse with
from project.stuff import mainfile
because the project directory is only automatically on sys.path when you start a script from the directory above project and do not specify a main script to run:
cd <projectdir>/..
python -m project.bin.script
# works
cd <projectdir>
python -m bin.script
# does not work, because `sys.path` starts with <projectdir>
# and it's `stuff.mainfile` now, not `project.stuff.mainfile`.
If you want to import modules from project in your script, fix up sys.path to your needs:
import sys
import os
sys.path.insert(0, os.path.dirname(sys.path[0]))
from stuff import mainfile
You need to first add the parent directory into sys.path. Try the following:
# This is a file in bin/ directory.
this_file_path = os.path.dirname(__file__)
sys.path.append(os.path.join(this_file_path, '..'))
import stuff.mainfile
Related
If I have a project with two files main.py and main2.py. In main2.py I do import sys; sys.path.append(path), and I do import main, which main.py will be imported? The one in my project or the one in path (the question poses itself only if there are two main.py obviously)?
Can I force Python to import a specific one?
The one that is in your project.
According to python docs, the import order is:
Built-in python modules. You can see the list in the variable sys.modules
The sys.path entries
The installation-dependent default locations
Your added directory to sys.path, so it depends on the order of entries in sys.path.
You can check the order of imports by running this code:
import sys
print(sys.path)
As you will observe, the first entry of sys.path is your module's directory ("project directory") and the last is the one you appended.
You can force python to use the outer directory by adding it in the beginning of your sys.path:
sys.path.insert(0,path)
Although I would recommend against having same names for the modules as it is often hard to manage.
Another thing to keep in mind is that you can't import relative paths to your sys.path. Check this answer for more information on how to work around that: https://stackoverflow.com/a/35259170/6599912
When two paths contain the same module name, using sys.path.append(path) isn't going to change the priority of the imports, since append puts path at the end of the python path list
I'd use
sys.path.insert(0,path)
so modules from path come first
Example: my module foo.py imports main and there's one main.py in the same directory and one main.py in the sub directory
foo.py
main.py
sub/main.py
in foo.py if I do:
import sys
sys.path.append("sub")
import main
print(main.__file__)
I get the main.py file path of the current directory because sys.path always starts by the main script directory.
So I could remove that directory from sys.path but it's a bad idea because I could need other modules from there.
Now if I use insert instead:
import sys
sys.path.insert(0,"sub")
import main
print(main.__file__)
I get sub/main.py as sub is the first directory python looks into when searching modules.
I have a very simple scenario like this:
example/
common.py
folder1/
script.py
where script.py file should import common.py module.
If I cd into folder1/ and run the script (i.e. by calling python3 script.py on the command line), the import breaks with the usual error ModuleNotFoundError: No module named 'common'.
I know I could turn the whole parent folder into a package by adding __init__.py files in each subdirectory, however this solution still prevents me to run the script directly from inside folder1/.
How can I fix this?
If you turn both directories into python packages, and the top directory is in your PYTHONPATH, you can run script.py like so:
python3 -m example.folder1.script
If you need to add a parent directory to the search path, you can always do:
import os
import sys
DIR = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
sys.path.insert(0, DIR)
os.path.realpath(__file__) gets the realpath to the file, and os.path.dirname gets the directory name of this file. Doing it twice gets the example directory here. Now, since this is added to the search path, you can import common.
However, you should really consider having a dedicated scripts or bin directory, rather than try to run scripts from subdirectories. Think about writing a library, and import this library rather than individual files.
As per screen print, import shows error in Python 3.7 version, earlier it was working fine in version Python 2.7 and I am using IntelliJ Idea.
If you see, EOC related .py files are in the same folder and have classes which are being called in Main_EOC.py by passing objects which are inter-related. It's amazing to see the red line while importing files from same folder.
Please help me why it's showing such error
"This inspection detects names that should resolve but don't. Due to dynamic dispatch and duck typing, this is possible in a limited but useful number of cases. Top-level and class-level items are supported better than instance items.`"
Also, if you see the line which have full path, is not showing error
from EOC_Module.eoc.script.config import Config
Please help me if there is a way to add this full path on top of the code or other option.
The behavior of import path search changed between python2 and python3. The import path always includes the directory from which the main module was loaded, but it no longer includes directories from which modules were imported.
You need to change your import statement syntax as follows, if you want to import a module that lives in the same directory as the module in which you do the import:
# old way, import works if the named module is in this module's directory
import x
# new (Python3) way:
from . import x
For the second part: adding a path so all code can import from a certain directory: if that directory is (and will always be) relative to your main: you can add a few lines in the main module to make it available. Something like this:
import sys # if you haven't imported it already
import os.path
home = os.path.dirname(sys.argv[0])
sys.path.append( os.path.join(home, "EOC_Module/eoc/script") )
# now, you can import straight from the script directory
import EOC_Intraction
When using pycharm the root directory for your python executable is the same as the root directory of your project, this means that python will start looking for files in the root directory with this files:
.idea/
EOC_module/
logs/
reports/
sql/
This is the reason of why: from EOC_Module.eoc.script.config import Config works.
If you execute your code from the terminal with: python3 Main_EOC.py (not pycharm) the root directory for your python will be the same as the one containing the file, all the other imports will work but from EOC_Module.eoc.script.config import Config not.
So you need to make your imports from project directory if you are using pycharm.
I've read the following post:
Relative imports for the billionth time
It explains the difference between running a file as a top-level script and importing it as a module in remarkable clarity and detail.
I can't figure out, however, how to write the imports of a runnable script (if __name__ == '__main__' et cetera) in a way which I'd also be able to import it from my tests, all the while keeping my runnable scripts stand-alone (i.e. not requiring package installation).
Say I have the following project hierarchy:
/reporoot
/mypkg
/mysubpkg
__init__.py
subpkgmodule.py
__init__.py
main1.py
main2.py
/tests
test_main1.py
test_main2.py
test_main1 needs to import main1 in some manner, and main1 needs to import subpkgmodule in some manner.
If, for instance, main1 imports subpkgmodule like so:
import mysubpkg.subpkgmodule
This will work fine when running the script as the top-level script because the top level script has its directory added to sys.path. This import will break, however, when the module is imported rather than run (because its directory will not be added to sys.path).
If main1 were to import subpkgmodule like so:
import mypkg.mysubpkg.subpkgmodule
This will only work when the package is an installed package (including python setup.py develop) and not running as a standalone script (i.e. chuck it on some filesystem and run python main1.py).
I saw that the standard library's http.server module has an import http.client and therefore cannot be run as a standalone script as I mentioned.
What would a clean solution for the imports of my runnable scripts be?
Typical way would be to extend sys.path in runnable script with location of your package.
A list of strings that specifies the search path for modules.
Initialized from the environment variable PYTHONPATH, plus an
installation-dependent default.
As initialized upon program startup, the first item of this list,
path[0], is the directory containing the script that was used to
invoke the Python interpreter. If the script directory is not
available (e.g. if the interpreter is invoked interactively or if the
script is read from standard input), path[0] is the empty string,
which directs Python to search modules in the current directory first.
Notice that the script directory is inserted before the entries
inserted as a result of PYTHONPATH.
A program is free to modify this list for its own purposes.
Sample test_main1.py:
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), os.pardir))
import mypkg.mysubpkg.subpkgmodule # should import fine
if __name__ == "__main__":
pass # runnable code here
Libraries such as the built-in unittest have built-in test discovery. By default, it expects tests to be in files named with the pattern test*.py, where the asterisk (*) may be whatever you'd like. You'd then use discover and run the package using:
python -m unittest discover
This assumes that
you are using unittest,
your tests subclass unittest.TestCase, and
you use the default class unittest.TestLoader.
I am running into pathing issues with some scripts that I wrote to test parsers I've written. It would appear that Python (27 and 3) both do not act like IronPython when it comes to using the current working directory as part of sys.path. As a result my inner package pathings do not work.
This is my package layout.
MyPackage
__init__.py
_test
__init__.py
common.py
parsers
__init__.py
_test
my_test.py
I am attempting to call the scripts from within the MyPackage directory.
Command Line Statements:
python ./parsers/_test/my_test.py
ipy ./parsers/_test/my_test.py
The import statement located in my my_test.py file is.
from _test.common import TestClass
In the python scenario I get ONLY the MyPackage/parsers/_test directory appended to sys.path so as a result MyPackage cannot be found. In the IronPython scenario both the MyPackage/parsers/_test directory AND MyPackage/ is in sys.path. I could get the same bad reference if I called the test from within the _test directory (it would have no idea about MyPackage). Even if i consolidated the test files into the _test directory I would still have this pathing issue.
I should note, that I just tested and if i did something like
import sys
import os
sys.path.append(os.getcwd())
It will load correctly. But I would have to do this for every single file.
Is there anyway to fix this kind of pathing issue without doing the following.
A. Appending a relative pathing for sys.path in every file.
B. Appending PATH to include MyPackage (I do not want my package as part of the python "global" system pathing)
Thanks alot for any tips!
Two options spring to mind:
Use the environment variable PYTHONPATH and include .
Add it to your sys.path at the beginning of your program
import sys.path
sys.path.append('.')
If you need to import a relative path dynamically you can always do something like
import somemodule
import sys
dirname = os.path.dirname(os.path.abspath(somemodule.__file__))
sys.path.append(dirname)