Given the directory structure:
program/
setup.py
ilm/
__init__.py
app/
__init__.py
bin/
script.py
Note: the setup.py is not a typical setup.py, rather it is a custom-made setup uniquely for py2app.
program/ilm/app/__init__.py is non-empty: it contains a main() function, which instantiates a class in the same file. My question: In program/ilm/bin/script.py, if I want to import and execute the main() function in program/ilm/app/__init__.py, what are the valid ways of achieving this? The reason I ask is that script.py is doing so thus:
import ilm.app as app
if __name__ == '__main__':
app.main()
Based on my (admittedly limited) understanding of packaging and importing, this shouldn't work, since we have not explicitly told script.py where to look for project/ilm/app/__init__.py using ... And indeed, I get:
MacBook-Pro-de-Pyderman:program Pyderman$ python ./bin/script.py
Traceback (most recent call last):
File "./bin/script.py", line 5, in <module>
import ilm.app as app
ImportError: No module named ilm.app
In contrast, when the Python interpreter is started in /project, import ilm.app as app works fine.
This is apparently fully-functional production code which I should not have to change to get running.
Is the import statement valid, given the directory structure, and if so, what am I missing?
If not, what is the recommended way of getting the import to work? Add the path using sys.path.append() above the import statement? Or use .. notation in the import statement to explicitly point to program to pick up project/ilm/app/__init__.py? Is the fact that it is an __init__.py I am trying to import significant?
Two things. You need to make sure the iml directory is in the python path. Either make sure you are running python from the right directory or add the right path to sys.path list. And you need to make sure that both iml and app directory both have
__init__.py
file, since python needs to interpret the whole thing as a hierarchy of modules rather than just dirs. Then you should be able to do
from iml import app
The obvious conclusion would seem to be that the iml directory has an __init__.py inside it, but why that would happen in your production setup is hard to say. Have you checked in the production environment whether this is the case?
Assuming that the production environment is importing the package at iml/app (which you can check by examining app.__file__) then the program will indeed execute the main function from the __init__.py file - but __init__.py might easily be importing it from sonewhere else rather than defining it locally.
The Python package needs proper setup.py which defines the package structure. Furthermore bin/ scripts should be defined as console_scripts entry points.
This all must be installed in a proper Python environment, not just any folder.
Related
First of all, there are a bunch of solutions on stackoverflow regarding this but from the ones I tried none of them is working. I am working on a remote machine (linux). I am prototyping within the dir-2/module_2.py file using an ipython interpreter. Also I am trying to avoid using absolute paths as the absolute path in this remote machine is long and ugly, and I want my code to run on other machines upon download.
My directory structure is as follows:
/project-dir/
-/dir-1/
-/__ init__.py
-/module_1.py
-/dir-2/
-/__ init__.py
-/module_2.py
-/module_3.py
Now I want to import module_1 from module_2. However the solution mentioned in this stackoverflow post: link of using
sys.path.append('../..')
import module_2
Does not work. I get the error: ModuleNotFoundError: No module named 'module_1'
Moreover, within the ipython interpreter things like import .module_3 within module_2 throws error:
import .module_3
^ SyntaxError: invalid syntax
Isn't the dot operator supposed to work within the same directory as well. Overall I am quite confused by the importing mechanism. Any help with the initial problem is greatly appreciated! Thanks a lot!
Why it didn't work?
If you run the module1.py file and you want to import module2 then you need something like
sys.path.append("../dir-2")
If you use sys.path.append("../..") then the folder you added to the path is the folder containing project-dirand there is notmodule2.py` file inside it.
The syntax import .module_3 is for relative imports. if you tried to execute module2.py and it contains import .module_3 it does not work because you are using module2.py as a script. To use relative imports you need to treat both module2.py and module_3.py as modules. That is, some other file imports module2 and module2 import something from module3 using this syntax.
Suggestion on how you can proceed
One possible solution that solves both problems is property organizing the project and (optionally, ut a good idea) packaging your library (that is, make your code "installable"). Then, once your library is installed (in the virtual environment you are working) you don't need hacky sys.path solutions. You will be able to import your library from any folder.
Furthermore, don't treat your modules as scripts (don't run your modules). Use a separate python file as your "executable" (or entry point) and import everything you need from there. With this, relative imports in your module*.py files will work correctly and you don't get confused.
A possible directory structure could be
/project-dir/
- apps/
- main.py
- yourlib/
-/__ init__.py
-/dir-1/
-/__ init__.py
-/module_1.py
-/dir-2/
-/__ init__.py
-/module_2.py
-/module_3.py
Notice that the the yourlib folder as well as subfolders contain an __init__.py file. With this structure, you only run main.py (the name does not need to be main.py).
Case 1: You don't want to package your library
If you don't want to package your library, then you can add sys.path.append("../") in main.py to add "the project-dir/ folder to the path. With that your yourlib library will be "importable" in main.py. You can do something like from yourlib import module_2 and it will work correctly (and module_2 can use relative imports). Alternatively, you can also directly put main.py in the project-dir/ folder and you don't need to change sys.path at all, since project-dir/ will be the "working directory" in that case.
Note that you can also have a tests folder inside project-dir and to run a test file you can do the same as you did to run main.py.
Case 2: You want to package your library
The previous solution already solves your problems, but going the extra mile adds some benefits, such as dependency management and no need to change sys.path no matter where you are. There are several options to package your library and I will show one option using poetry due to its simplicity.
After installing poetry, you can run the command below in a terminal to create a new project
poetry new mylib
This creates the following folder structure
mylib/
- README.rst
- mylib/
- __init__.py
- pyproject.toml
- tests
You can then add the apps folder if you want, as well as subfolders inside mylib/ (each with a __init__.py file).
The pyproject.toml file specifies the dependencies and project metadata. You can edit it by hand and/or use poetry to add new dependencies, such as
poetry add pandas
poetry add --dev mypy
to add pandas as a dependency and mypy as a development dependency, for instance. After that, you can run
poetry build
to create a virtual environment and install your library in it. You can activate the virtual environment with poetry shell and you will be able to import your library from anywhere. Note that you can change your library files without the need to run poetry build again.
At last, if you want to publish your library in PyPi for everyone to see you can use
poetry publish --username your_pypi_username --password _passowrd_
TL; DR
Use an organized project structure with a clear place for the scripts you execute. Particularly, it is better if the script you execute is outside the folder with your modules. Also, don't run a module as a script (otherwise you can't use relative imports).
I'm trying to import a class in a different directory to another file, but can't seem to get it to work. I know this question has been asked a lot and I have looked through multiple stackoverflow solutions and at https://docs.python.org/3/tutorial/modules.html#packages
1: Importing files from different folder
2: import python file in another directory failed
I want to try to just use the method containing just __init__.py file instead of doing an import sys
My directory structure is as follows:
django_vue/
__init__.py
devices/
__init__.py
models.py
lib/
__init__.py
my_file.py
I'm trying to import the class Device from /django_vue/devices/models.py to /django_vue/lib/my_file.py by:
from devices.models import Device
However when I do that I still get the error:
from devices.models import Device
ModuleNotFoundError: No module named 'devices'
I'm not sure what I'm dong wrong since I already have the __init__ file in both directories. Any help is appreciated. Also I'm running python 3.6.
This is the folder structure I'm working with.
.
└── django_vue
├── devices
│ └── models.py
└── lib
└── file.py
When you run
$ python file.py
python has no way of knowing what's outside the directory.
python can't go back and then into devices/ just like that.
The easiest way to solve this would be to add the folder devices/ to sys.path. When python imports a module, it searches for the module from sys.path. Adding the path to devices/ would make it available for imports.
Here are my files.
# models.py
Device = 'device'
# file.py
import sys
sys.path.append('..') # adds the parent dir (which is django-vue/) to path
# django-vue dir has devices/ so now this is available for imports
# importing this works now
from devices.models import Device
print(Device)
Output
django_vue/lib$ python3 file.py
device
Think about it your are inside my_file.py and import something called devices.
How can python know where the name devices has come from.
It won't search your entire Drive for that module/package
Relative Import
use a relative import instead. write from ..devices.models import Device. This is telling python to go up one directory to the parent directory and that's where it will find the devices package. Your lib module should now work as a module
If you however run the my_file.py package directly (as in python C:/django_vue/lib/my_file.py)
You will still get an error. Not the same error; the new error will be something like
ImportError: attempted relative import with no known parent package
This is happening because you are actually running my_file.py
If you think about it why would you want to run my_file.py by itself when it is clearly a part of a package. Maybe you are just testing to see if the import works when you use your package. The problem with this is that it makes it seem like your packages relative imports don't work even though this actually works.
Create a main.py in django_vue and write from lib import my_file. This will run your my_file.py and you will notice there is no error.
What's happening here
Have you heard of __package__?
if you put print(str(__package__)) in your my_file.py and run my_file.py directly you will see that it prints None.
However if you run main.py (that you just created) you will see that when It reaches my_file.py, __package__ will actually be defined to something.
Ahhh... you see now it all makes sense; The error you originally got said something about no known parent package. If __package__ is undefined that means there is no relative parent package because the file was obviously run directly instead of as part of a package.
Consider Absolute imports
you also might want to consider using absolute imports because if you are working on the package you might change it directory structure while developing. so you need to keep changing the import references on the affected files.
Although you can find IDE's with python extensions that automatically to this as you change your directory. I believe VS Code does this automatically.
Replace the __init__ files with __main__.
I'm developing a system in Python that includes a calculation engine and a front end. I've split them up into two projects as the calculation engine can be used for other front ends as well.
I'm using Eclipse and PyDev. Everything works perfectly in Eclipse/PyDev, but as soon as I try to run it outside of PyDev (from command line) I get importing errors. I've done quite a bit of research to find the problem, but I just don't see a solution that works nicely. I believe that PyDev modifies the Python path.
In my project layout below I have two packages (package1 and tests) within one project (Calculations). I can't seem to import anything from package1 in tests. I also have another project (Frontend). Here I also can't import anything from package1.
What I want to understand is the proper way of calling my script/tests files from the command line? Both for two separate projects and two packages in the same project. I assume it would be similar to how PyDev does it. So far I think I have the following options:
Create python code to append to sys.path (seems hacky/not good practice)
Modify the PYTHONPATH when I call the test_some_calc.py like this: PYTHONPATH= python test_some_calc.py (I think this is how PyDev does it, but it seems a bit long - there must be a simpler way?
Make a install package (eventually I might go this method, but not yet.)
I have the following project layout.
CodeSolution/
Calculations/
package1/
__init__.py
subpackage/
__init__.py
some_calc.py
subpackage2/
__init__.py
another_calc.py
tests/
__init__.py
subpackage/
__init__.py
test_some_calc.py # Unable to import from package1
subpackage2/
__init__.py
test_another_calc.py # Unable to import from package1
Frontend/
some_script.py # Unable to import from package1
Comments on my project layout will also be appreciated.
A clean, quick and modular way to include certain python from anywhere on your system is to make a file named mymodule.pth and put it inside the path of site-packages
mymodule.pth should have the path of your project. Project folder must have an __init__.py file.
for example put:
for Linux:
/home/user/myproject
inside
/usr/lib/python2.7/site-packages/mymodule.pth
or
for Windows
C:\\Users\myUsername\My Documents\myproject
inside
C:\PythonXY\Lib\site-packages\mymodule.pth
I wrote a script to load PYTHONPATHs from PyDev's project properties. It's allow you to run your code from console without problems like "ModuleNotFoundError: No module named ...".
import sys
from xml.dom import minidom
import os
print(sys.path)
def loadPathsFromPyDev():
sys_path = sys.path[0]
# Load XML
xmldoc = minidom.parse(sys_path+'\.pydevproject')
# Get paths
xmlpaths = xmldoc.getElementsByTagName('path')
# Get paths' values
paths = list()
for xmlpath in xmlpaths:
paths.append(xmlpath.firstChild.data)
# set path variable
for path in paths:
# Set backslashes to forwardslashes
path = os.path.normpath(path)
# Set string's sys_path
path = path.replace("\${PROJECT_DIR_NAME}", sys_path)
if path not in sys.path:
# Add to system path
sys.path.insert(1,path)
loadPathsFromPyDev()
print(sys.path)
I hope it will help You :)
I just got set up to use pytest with Python 2.6. It has worked well so far with the exception of handling "import" statements: I can't seem to get pytest to respond to imports in the same way that my program does.
My directory structure is as follows:
src/
main.py
util.py
test/
test_util.py
geom/
vector.py
region.py
test/
test_vector.py
test_region.py
To run, I call python main.py from src/.
In main.py, I import both vector and region with
from geom.region import Region
from geom.vector import Vector
In vector.py, I import region with
from geom.region import Region
These all work fine when I run the code in a standard run. However, when I call "py.test" from src/, it consistently exits with import errors.
Some Problems and My Solution Attempts
My first problem was that, when running "test/test_foo.py", py.test could not "import foo.py" directly. I solved this by using the "imp" tool. In "test_util.py":
import imp
util = imp.load_source("util", "util.py")
This works great for many files. It also seems to imply that when pytest is running "path/test/test_foo.py" to test "path/foo.py", it is based in the directory "path".
However, this fails for "test_vector.py". Pytest can find and import the vector module, but it cannot locate any of vector's imports. The following imports (from "vector.py") both fail when using pytest:
from geom.region import *
from region import *
These both give errors of the form
ImportError: No module named [geom.region / region]
I don't know what to do next to solve this problem; my understanding of imports in Python is limited.
What is the proper way to handle imports when using pytest?
Edit: Extremely Hacky Solution
In vector.py, I changed the import statement from
from geom.region import Region
to simply
from region import Region
This makes the import relative to the directory of "vector.py".
Next, in "test/test_vector.py", I add the directory of "vector.py" to the path as follows:
import sys, os
sys.path.append(os.path.realpath(os.path.dirname(__file__)+"/.."))
This enables Python to find "../region.py" from "geom/test/test_vector.py".
This works, but it seems extremely problematic because I am adding a ton of new directories to the path. What I'm looking for is either
1) An import strategy that is compatible with pytest, or
2) An option in pytest that makes it compatible with my import strategy
So I am leaving this question open for answers of these kinds.
The issue here is that Pytest walks the filesystem to discover files that contain tests, but then needs to generate a module name that will cause import to load that file. (Remember, files are not modules.)
Pytest comes up with this test package name by finding the first directory at or above the level of the file that does not include an __init__.py file and declaring that the "basedir" for the module tree containing a module generated from this file. It then adds the basedir to sys.path and imports using the module name that will find that file relative to the basedir.
There are some implications of this of which you should beware:
The basepath may not match your intended basepath in which case the module will have a name that doesn't match what you would normally use. E.g., what you think of as geom.test.test_vector will actually be named just test_vector during the Pytest run because it found no __init__.py in src/geom/test/ and so added that directory to sys.path.
You may run into module naming collisions if two files in different directories have the same name. For example, lacking __init__.py files anywhere, adding geom/test/test_util.py will conflict with test/test_util.py because both are loaded as import test_util.py, with both test/ and geom/test/ in the path.
The system you're using here, without explicit __init__.py modules, is having Python create implicit namespace packages for your directories. (A package is a module with submodules.) Ideally we'd configure Pytest with a path from which it would also generate this, but it doesn't seem to know how to do that.
The easiest solution here is simply to add empty __init__.py files to all of the subdirectories under src/; this will cause Pytest to import everything using package/module names that start with directory names under src/.
The question How do I Pytest a project using PEP 420 namespace packages? discusses other solutions to this.
import looks in the following directories to find a module:
The home directory of the program. This is the directory of your root script. When you are running pytest your home directory is where it is installed (/usr/local/bin probably). No matter that you are running it from your src directory because the location of your pytest determines your home directory. That is the reason why it doesn't find the modules.
PYTHONPATH. This is an environment variable. You can set it from the command line of your operating system. In Linux/Unix systems you can do this by executing: 'export PYTHONPATH=/your/custom/path' If you wanted Python to find your modules from the test directory you should include the src path in this variable.
The standard libraries directory. This is the directory where all your libraries are installed.
There is a less common option using a pth file.
sys.path is the result of combining the home directory, PYTHONPATH and the standard libraries directory. What you are doing, modifying sys.path is correct. It is something I do regularly. You could try using PYTHONPATH if you don't like messing with sys.path
If you include an __init__.py file inside your tests directory, then when the program is looking to set a home directory it will walk 'upwards' until it finds one that does not contain an init file. In this case src/.
From here you can import by saying :
from geom.region import *
you must also make sure that you have an init file in any other subdirectories, such as the other nested test directory
I was wondering what to do about this problem too. After reading this post, and playing around a bit, I figured out an elegant solution. I created a file called "test_setup.py" and put the following code in it:
import sys, os
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
I put this file in the top-level directory (such as src). When pytest is run from the top-level directory, it will run all test files including this one since the file is prefixed with "test". There are no tests in the file, but it is still run since it begins with "test".
The code will append the current directory name of the test_setup.py file to the system path within the test environment. This will be done only once, so there are not a bunch of things added to the path.
Then, from within any test function, you can import modules relative to that top-level folder (such as import geom.region) and it knows where to find it since the src directory was added to the path.
If you want to run a single test file (such as test_util.py) instead of all the files, you would use:
pytest test_setup.py test\test_util.py
This runs both the test_setup and test_util code so that the test_setup code can still be used.
Are so late to answer that question but usining python 3.9 or 3.10 u just need to add __init__.py folder in tests folders.
When u add this file python interprets this folders as a module.
Wold be like this
src/
main.py
util.py
test/
__init__.py
test_util.py
geom/
vector.py
region.py
test/
__init__.py
test_vector.py
test_region.py
so u just run pytest.
Sorry my poor english
Not the best solution, but maybe the fastest one:
cd path/python_folder
python -m pytest python_file.py
Relative import not working properly in python2.6.5 getting "ValueError: Attempted relative import in non-package".
I am having all those __init__.py in proper place.
I have seen that error before when running a script that is actually inside a package. To the interpreter, it appears as though the package is not a package.
Try taking the script into another directory, putting your package inside your pythonpath, and import absolutely. Then, relative imports inside your package will work. NOTE: you can STILL not relatively import inside the end script - the easiest thing to do in this case is to make a "wrapper" script, that simply calls some entry point in your package.
You can go even further here by using setuptools to create a setup.py for your package, to make it distributable. Then, as a part of that, entry points would allow you to autogenerate scripts that called your package's code.
EDIT:
From your comment, it appears as though I wasn't quite clear. I'm not 100% sure of your directory structure because your comment above wasn't formatted, but I took it to be like this:
PythonEvent/
main.py
__init__.py
DBConnector/
__init__.py
connector.py
service/
__init__.py
myservice.py
When in myservice.py you have the line from ..DBConnector.connector import DBUpdate, the interpreter tries to import it relatively, UNLESS you are running myservice.py directly. This is what it appears you are doing.
Try making another dummy script outside of PythonEvent/ that is simply as follows:
from PythonEvent.service import myservice
if __name__ == '__main__':
myservice.main() # or whatever the entry point is called in myservice.
Then, set your PYTHONPATH environment variable to point to the parent directory of PythonEvent/ (or move PythonEvent/ to your site-packages).
main.py
setup.py
Main Package/ ->
__init__.py
subpackage_a/ ->
__init__.py
module_a.py
subpackage_b/ ->
__init__.py
module_b.py
i)
1.You run python main.py
2.main.py does: import app.package_a.module_a
3.module_a.py does import app.package_b.module_b
ii)
Alternatively 2 or 3 could use: from app.package_a import module_a
That will work as long as you have app in your PYTHONPATH. main.py could be anywhere then.
So you write a setup.py to copy (install) the whole app package and subpackages to the target system's python folders, and main.py to target system's script folders.
Thanks to https://stackoverflow.com/a/1083169