I have the following directory structure:
base_folder
methods_folder
method_1.py
method_2.py
.
.
.
method_n.py
class_methods.py
top_class.py
class_methods.py imports the other files in the same directory, like this:
from method_1 import method_1
from method_2 import method_2
.
.
.
from method_n import method_n
(obs: these methods files has a method with its own file names inside them)
If I run class_methods.py by myself, no problem. But if I try to run top_class.py, which imports class_methods.py, I get the error no module named method_1
So, when executing top_class.py, it is not seeing the files in methods_folder/. Why?
the correct import inside top_class.py would be from methods_folder.method_n import method_n. This is because you are treating methods_folder as a package. If you are running a version of Python that is before 3.3 you must also unclude __init__.py file inside the methods_folder in order to turn it into a package.
Files only have direct access to things they import. Say we have a.py which imports b.py, and b.py imports c.py. When running within a.py functions in b.py that use c.py, this will work fine, because a has access to b, and b has access to c. This does not mean, however, that the imports chain (as in C++) and you can use functions from c in a. You will get an error, because a can only see the contents of b, which it imported.
So if you want to use all your method_i.py files from within top_class.py, you need to import them directly in the same file.
Edit: You also have some other issues. To import other files in a subfolder, you would need to, inside top_class.py, call import methods_folder.method_i. To import something in the same directory, just use import method_i. Since you have a method of the same name in each file, what you have works fine in class_methods.py You also need to create an empty file called __init__.py in any folder containing python files you intend to import to/from which lets python know it's allowed to look there.
You can create an importable package in python in one of two ways. The first way is what you are doing: you create a file called my_package.py and import it with import my_package. This is commonly used for simpler packages that don't need to be further broken up. For this to work, your .py file has to be on the PYTHONPATH which is an environment variable that tells python where to look for packages. If not defined, there are some default places that python will use to look for packages. One of these default locations is the current working directory, which is why your first set of imports works.
In theory you should be able to run the second piece of code from the same location (python ../top_class.py) and use the same import style, but I assume you are changing directories to run that file. This means your files are no longer in the current working directory and no longer findable by python.
One way to get your code to work using the existing style would be to define PYTHONPATH with the location of your methodX.py files. You will typically add to the python search path like this:
PYTHONPATH=$PYTHONPATH:./methods_folder python top_class.py
This tells python to look in methods_folder, in addition to the standard places, when you try to import something. Playing with the PYTHONPATH gets a little annoying after a while, so I actually prefer the next approach.
A second way to create a package is by creating a folder with an __init__.py file inside. This tells python that you want it to treat that directory as a package. This is the preferred style for more complicated pieces of code that might benefit from organization across multiple files. For your example, you could organize your code in the following way:
base_folder
methods_folder
__init__.py
method_1.py
method_2.py
.
.
.
method_n.py
class_methods.py
top_class.py
And then your import in top_class.py would look like this:
from methods_folder.method1 import method1
from methods_folder.method2 import method2
from methods_folder.method3 import method3
This has the effect of creating a top level methods_folder package with modules method1, method2, etc. Because methods_folder is in the same directory as the one you are running top_class.py from, python picks that up as a package using the default PYTHONPATH and lets you import from within it.
I assume you are running them from their respective directories? Unless they are installed in your Python path (I'm going to assume they are not), then Python will by default look for imports in your current directory. So when you run class_methods.py from its directory, then it can find methods_1.py in order to satisfy from methods_1 import method_1. But when you execute top_class.py, it looks for methods_1.py or methods_1/__init__.py, neither of which it fines from that directory.
Assuming Python 3, you would need to use relative imports in class_methods.py.
# class_methods.py
from .methods_1 import method_1
from .methods_2 import method_2
This will let you run it from top_class.py. Unfortunately, you can't use relative imports when running a script in the same package, so you wouldn't be able to run class_methods.py directly.
Another option in that case is to keep the absolute imports in class_methods.py and add the methods folder to the path in top_class.py.
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), 'methods'))
import class_methods
from methods_1 import method
Of course editing sys.path is fine for small scripts and standalone things. But if this grows past being a script, you'll need to be careful about doing it, and you'll probably just want to come up with another solution. Likely, the best thing is to create a package that you install (you can still do this from your source directory while you are developing) and then you can import the same way from anywhere.
The recommended way of running a python script is using the -m switch from the parent of your root package - so in your case:
$ cd base_folder
$ python -m top_class
Python will automatically add the base_folder to its sys.path and you don't need to do any sys.path/PYTHOPATH hacks that are just this - hacks that bloat the code with boilerplate and will blow when least expected.
Now to run the class_methods the correct way is also
$ cd base_folder
$ python -m methods_folder.class_methods
but then the imports in class_methods should be modified to either absolute:
from methods_folder.method1 import method1
...
or relative:
from .method1 import method1
...
Related
Consider this folder structure:
main.py
module_a/
aa.py
bb.py
__init__.py
In main.py, I import aa as:
from module_a import aa
aa.yyy()
Then in aa.py, I import bb and include its functions as:
import bb
bb.xxx()
However, when I run main.py, python says "No module named 'bb'".
May I know why this happens. What is the correct way to import bb.
Thanks!!!
I have tried to write aa.py as:
import .bb
bb.xxx()
But it still does not work.
why this happens
Because the aa folder is not a place that Python is searching for modules.
Python's imports are absolute by default. They only look in specific places determined by sys.path. In main.py, import module_a.aa works because the root folder of the project happens to be on the sys.path; that folder contains a module_a folder; and that folder contains aa.py.
What is the correct way to import bb.
Please use relative imports between files in your package. In this case, the necessary import in aa.py looks like:
from . import bb
Absolute imports are error-prone; a project that uses two packages whose contents have overlapping names will run into namespace collisions. (Sadly, the standard library uses absolute imports in most places, such that projects need to ban certain module names for safety.) Relative imports will also require much less maintenance, should you later rename a sub-package.
The only thing relative imports require is that the package gets loaded, which typically will happen automatically with the first (yes, absolute) import of any of the package contents. When the package is loaded, it automatically sets a __package__ attribute on the modules in that package, which Python can use to resolve the relative imports. It's important to note that relative imports are relative to the package hierarchy, not the directory structure, which is why this is necessary; imports like from .. import example do not work by figuring out the current file location and then going up a level in the directory hierarchy. Instead, they check the __package__ to figure out what the containing package is, then check the file/folder location for that, and work from there.
If the "driver" script is within the package, run it as a module, using the -m switch for Python. For example, from the root folder, if module_a/aa.py is the driver, use python -m module_a.aa. This instructs Python that module_a is the containing package for aa.py, and ensures it gets loaded even though no import in the code has loaded it.
Contrary to what many people will wrongly tell you, it is almost never required to manipulate sys.path; there are popular Python projects on GitHub, running hundreds of thousands of lines of code, which either do not use it at all or use it only once in an ancillary role (perhaps because of a special requirement for a documentation tool). Just don't do it.
Also contrary to what many people will wrongly tell you, __init__.py files are not required to create packages in Python. They are simply a place where additional code can be placed for package initialization - for example, to create aliases for sub-package contents, or to limit what will be imported with a *-import (by setting __all__).
import .bb
This is just invalid. Relative imports only use the from syntax. See above for the correct syntax.
Suppose the same file structure as stated in OP, and:
main.py:
import module_a.aa
module_a.aa.thisFile()
module_a.aa.module_a.bb.thisFile()
aa.py:
import module_a.bb
def thisFile():
print("aa")
bb.py:
def thisFile():
print("bb")
Then, this will print
aa
bb
like you would expect. The main difference here, is that bb.py is imported in aa.py via module_a.bb. Running main.py gives no problem, however, running aa.py does not work in this way. That is why you might want to add folders to your path, such that you can call a function from a different file without this trouble. This is done via:
import os, inspect, sys
current_folder = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parent_folder = os.path.dirname(current_folder)
sys.path.insert(0,parent_folder)
Then you can import your files such as import file. If you consider this option, I would suggest to do some reading about how this works. Tip: make sure you avoid cyclic import problems.
I have a large repository with some fixed structure and I have extended it by some folders and python scripts to add extra functionality to it as a whole. The structure looks as follows:
toplevelfolder
featureA
someModuleA.py
__ init __.py
featureB
someModuleB.py
__ init __.py
application
__ init __.py
app.py
Now someModuleA.py and someModuleB.py can be invoked via app.py but at the same time also have be able to be invoked directly, however this invocation must come from the toplevelfolder for the relative paths in the file to resolve correctly, i.e. via python ./featureA/someModuleA.py.
This all works well, but now I need some function definitions from someModuleB in someModuleA and hence I want to import this module. I have tried both absolute and relative imports, but both fail with different errors, the absolute import with
from toplevelfolder.featureA import someModuleA as A
# ModuleNotFoundError: No module named 'toplevelfolder'
and the relative import with
from toplevelfolder.featureA import someModuleA as A
# ImportError: attempted relative import with no known parent package
Now I can see that the relative import would cause problems when python is invoked from the toplevelfolder, as .. would represent the latter's parent directory, rather than the parent directory of featureA. However, I cannot get a hold of the first error message, especially since toplevelfolder should not be a module but a package.
Is there another way to import in Python that I'm not aware of, if possibly without modifying PYTHONPATH or sys.path or something like that?
Not 100% sure on what the goal is here. My advice would be:
Identify clearly what you want your top level modules and packages to be.
Make all imports absolute.
Either:
make your project a real installable project, so that those top level modules and packages are installed in the environment's site-packages directory;
or make sure that the current working directory is the one containing the top level modules and packages.
Make sure to call your code via the executable module or package method instead of the script method, if the "entry point" you want to execute is part of a package
DO (executable module or package):
path/to/pythonX.Y -m toplevelpackage.module
path/to/pythonX.Y -m toplevelpackage.subpackage (assuming there is a toplevelpackage/subpackage/__main__.py file)
DON'T (script within a package):
path/to/pythonX.Y toplevelpackage/module.py
(Optional) Later on, once it all works well and everything is under control, you might decide to change some or all imports to relative. (If things are done right, I believe it could be possible to make it so that it is possible to call the executable modules from any level within the directory structure as the current working directory.)
References:
Old reference, possibly outdated, but assuming I interpreted it right, it says that running scripts that live in a package is an anti pattern, and one should use python -m package.module instead: https://mail.python.org/pipermail/python-3000/2007-April/006793.html -- https://www.python.org/dev/peps/pep-3122/
Don't include the top level directory. In featureB.someModuleB:
from featureA.someModuleA import two
Sample directory.
Try pasting this above your import:
import os,sys,inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0,parentdir)
Then you should be able to import a file from the parent-folder.
I have trouble importing package.
My file structure is like this:
filelib/
__init__.py
converters/
__init__.py
cmp2locus.py
modelmaker/
__init__.py
command_file.py
In module command_file.py I have a class named CommandFile which i want to call in the cmp2locus.py module.
I have tried the following in cmp2locus.py module:
import filelib.modelmaker.command_file
import modelmaker.command_file
from filelib.modelmaker.command_file import CommandFile
All these options return ImportError: No modules named ...
Appreciate any hint on solving this. I do not understand why this import does not work.
To perform these imports you have 3 options, I'll list them in the order I'd prefer. (For all of these options I will be assuming python 3)
Relative imports
Your file structure looks like a proper package file structure so this should work however anyone else trying this option should note that it requires you to be in a package; this won't work for some random script.
You'll also need to run the script doing the importing from outside the package, for example by importing it and running it from there rather than just running the cmp2locus.py script directly
Then you'll need to change your imports to be relative by using ..
So:
import filelib.modelmaker.command_file
becomes
from ..modelmaker import command_file
The .. refers to the parent folder (like the hidden file in file systems).
Also note you have to use the from import syntax because names starting with .. aren't valid identifiers in python. However you can of course import it as whatever you'd like using from import as.
See also the PEP
Absolute imports
If you place your package in site-packages (the directories returned by site.getsitepackages()) you will be able to use the format of imports that you were trying to use in the question. Note that this requires any users of your package to install it there too so this isn't ideal (although they probably would, relying on it is bad).
Modifying the python path
As Meera answered you can also directly modify the python path by using sys.
I dislike this option personally as it feels very 'hacky' but I've been told it can be useful as it gives you precise control of what you can import.
To import from another folder, you have to append that path of the folder to sys.path:
import sys
sys.path.append('path/filelib/modelmaker')
import command_file
I just got set up to use pytest with Python 2.6. It has worked well so far with the exception of handling "import" statements: I can't seem to get pytest to respond to imports in the same way that my program does.
My directory structure is as follows:
src/
main.py
util.py
test/
test_util.py
geom/
vector.py
region.py
test/
test_vector.py
test_region.py
To run, I call python main.py from src/.
In main.py, I import both vector and region with
from geom.region import Region
from geom.vector import Vector
In vector.py, I import region with
from geom.region import Region
These all work fine when I run the code in a standard run. However, when I call "py.test" from src/, it consistently exits with import errors.
Some Problems and My Solution Attempts
My first problem was that, when running "test/test_foo.py", py.test could not "import foo.py" directly. I solved this by using the "imp" tool. In "test_util.py":
import imp
util = imp.load_source("util", "util.py")
This works great for many files. It also seems to imply that when pytest is running "path/test/test_foo.py" to test "path/foo.py", it is based in the directory "path".
However, this fails for "test_vector.py". Pytest can find and import the vector module, but it cannot locate any of vector's imports. The following imports (from "vector.py") both fail when using pytest:
from geom.region import *
from region import *
These both give errors of the form
ImportError: No module named [geom.region / region]
I don't know what to do next to solve this problem; my understanding of imports in Python is limited.
What is the proper way to handle imports when using pytest?
Edit: Extremely Hacky Solution
In vector.py, I changed the import statement from
from geom.region import Region
to simply
from region import Region
This makes the import relative to the directory of "vector.py".
Next, in "test/test_vector.py", I add the directory of "vector.py" to the path as follows:
import sys, os
sys.path.append(os.path.realpath(os.path.dirname(__file__)+"/.."))
This enables Python to find "../region.py" from "geom/test/test_vector.py".
This works, but it seems extremely problematic because I am adding a ton of new directories to the path. What I'm looking for is either
1) An import strategy that is compatible with pytest, or
2) An option in pytest that makes it compatible with my import strategy
So I am leaving this question open for answers of these kinds.
The issue here is that Pytest walks the filesystem to discover files that contain tests, but then needs to generate a module name that will cause import to load that file. (Remember, files are not modules.)
Pytest comes up with this test package name by finding the first directory at or above the level of the file that does not include an __init__.py file and declaring that the "basedir" for the module tree containing a module generated from this file. It then adds the basedir to sys.path and imports using the module name that will find that file relative to the basedir.
There are some implications of this of which you should beware:
The basepath may not match your intended basepath in which case the module will have a name that doesn't match what you would normally use. E.g., what you think of as geom.test.test_vector will actually be named just test_vector during the Pytest run because it found no __init__.py in src/geom/test/ and so added that directory to sys.path.
You may run into module naming collisions if two files in different directories have the same name. For example, lacking __init__.py files anywhere, adding geom/test/test_util.py will conflict with test/test_util.py because both are loaded as import test_util.py, with both test/ and geom/test/ in the path.
The system you're using here, without explicit __init__.py modules, is having Python create implicit namespace packages for your directories. (A package is a module with submodules.) Ideally we'd configure Pytest with a path from which it would also generate this, but it doesn't seem to know how to do that.
The easiest solution here is simply to add empty __init__.py files to all of the subdirectories under src/; this will cause Pytest to import everything using package/module names that start with directory names under src/.
The question How do I Pytest a project using PEP 420 namespace packages? discusses other solutions to this.
import looks in the following directories to find a module:
The home directory of the program. This is the directory of your root script. When you are running pytest your home directory is where it is installed (/usr/local/bin probably). No matter that you are running it from your src directory because the location of your pytest determines your home directory. That is the reason why it doesn't find the modules.
PYTHONPATH. This is an environment variable. You can set it from the command line of your operating system. In Linux/Unix systems you can do this by executing: 'export PYTHONPATH=/your/custom/path' If you wanted Python to find your modules from the test directory you should include the src path in this variable.
The standard libraries directory. This is the directory where all your libraries are installed.
There is a less common option using a pth file.
sys.path is the result of combining the home directory, PYTHONPATH and the standard libraries directory. What you are doing, modifying sys.path is correct. It is something I do regularly. You could try using PYTHONPATH if you don't like messing with sys.path
If you include an __init__.py file inside your tests directory, then when the program is looking to set a home directory it will walk 'upwards' until it finds one that does not contain an init file. In this case src/.
From here you can import by saying :
from geom.region import *
you must also make sure that you have an init file in any other subdirectories, such as the other nested test directory
I was wondering what to do about this problem too. After reading this post, and playing around a bit, I figured out an elegant solution. I created a file called "test_setup.py" and put the following code in it:
import sys, os
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
I put this file in the top-level directory (such as src). When pytest is run from the top-level directory, it will run all test files including this one since the file is prefixed with "test". There are no tests in the file, but it is still run since it begins with "test".
The code will append the current directory name of the test_setup.py file to the system path within the test environment. This will be done only once, so there are not a bunch of things added to the path.
Then, from within any test function, you can import modules relative to that top-level folder (such as import geom.region) and it knows where to find it since the src directory was added to the path.
If you want to run a single test file (such as test_util.py) instead of all the files, you would use:
pytest test_setup.py test\test_util.py
This runs both the test_setup and test_util code so that the test_setup code can still be used.
Are so late to answer that question but usining python 3.9 or 3.10 u just need to add __init__.py folder in tests folders.
When u add this file python interprets this folders as a module.
Wold be like this
src/
main.py
util.py
test/
__init__.py
test_util.py
geom/
vector.py
region.py
test/
__init__.py
test_vector.py
test_region.py
so u just run pytest.
Sorry my poor english
Not the best solution, but maybe the fastest one:
cd path/python_folder
python -m pytest python_file.py
I have some code in a project which I'd like to reuse in another project. What do I need to do (in both folders) so that I can do this?
The directory structure is something like:
Foo
Project1
file1.py
file2.py
Bar
Project2
fileX.py
fileY.py
I want to use functions from file1.py and file2.py in fileX.py and fileY.py.
Ideally both projects will be an installable python package, replete with __init__.py and setup.py. They could then be installed with python setup.py install or similar.
If that is not possible, don't use execfile()! Manipulate the PYTHONPATH to add Foo so that import Project1.file1 works.
For example, from Project2/fileX.py:
from os import path
import sys
sys.path.append(path.abspath('../Foo'))
from Project1.file1 import something
However, the real answer is to make each a discrete installable package.
There's a lot going on here. you should read about python packages and module management http://docs.python.org/2/tutorial/modules.html#packages but the basic idea is that fileX needs to know where file1 and file2 are in order to use them.
To turn a folder into a package, it just needs to contain an __init__.py file. What I would suggest you do is (in a terminal)
$ touch Foo/__init__.py
$ touch Foo/Project1/__init__.py
(assuming you're using unix/linux).
Then somehow, fileX needs to know where the Foo package is. You can call sys.path.append(PATH) where PATH is the location of Foo.
finally inside fileX.py you'd have
import sys
sys.path.append(PATH) #replace PATH with the path to Foo
from Foo.Project1 import file1
#use its functions
file1.function_name(argument)
if you really want to just say function_name without the preceeding file1. you can import all of its functions by saying from Foo.Project1.file1 import * however please note that from module import * is highly frowned upon as it mixes names and make code less readable and understandable
You may want to make a module out of it and then import whatever you need from it.
You take the code you want to use in both projects, and you put it into a module, which you extract into a third separate project. That project you make into a package, which you can work on separately. You then release version of it, and reuse them in your other projects.
It is important that you have versions that you "release" so that you can keep track of which version of the module each project uses. Otherwise you will end up in a situation where an old project stops working because you have made incompatible changes to the common module.
If it's generically usable not only for you but for others, consider uploading it to PyPI.
I think you can add Foo to the current Python path using os.path.join() or os.path.append() and do from import Project1.file1 import function_name.