I've got a python package written locally with a structure like
package
├── __init__.py
├── __main__.py
├── tests
│ ├── __init__.py
│ └── package_tests.py
└── package
├── __init__.py
├── package.py
This works great when run using python -m package in a Python3 virtualenv from the root of the project (parent dir of the first package dir in that tree)
However when run in a Python2.7 virtualenv, I get an ImportError in the __main__.py script as it tries to import the functions from package.py
__main__.py:
import sys
from package.package.package import foo, bar
def main(args):
f = foo(args)
bar(f)
if __name__ == "__main__":
main(sys.argv[1:])
The error:
ImportError: No module named package
What do I need to change to make it compatible with both?
(Obviously the package isn't actually called package)
Despite looking for an explanation for so long, immediately after posting this question I found a solution.
After looking over the changes to imports between python 2 and 3 I found that I only needed to use relative imports.
So the import line in my __main__.py became
from .package.package import foo, bar
Related
I have a project structure as follows:
python-test » tree
.
├── asdf
│ ├── __init__.py
│ ├── mycode2.py
│ ├── mycode.py
│ └── scripts
│ └── __init__.py
├── __init__.py
└── scripts
├── __init__.py
└── mymod.py
asdf/mycode.py contains:
import scripts.mymod
and scripts/mymod.py contains:
print('hello world')
All the __init__.pys are empty.
I would like to import scripts.mymod.
I am running asdf/mycode.py which fails because it is looking inside asdf/scripts instead of the root scripts folder for mymod.
> PYTHONPATH=: python asdf/mycode.py
Traceback (most recent call last):
File "asdf/mycode.py", line 1, in <module>
import scripts.mymod
ModuleNotFoundError: No module named 'scripts.mymod'
As a workaround, I can manually modify the path, which I have done in asdf/mycode2.py.
import sys
sys.path = [p for p in sys.path if 'asdf' not in p]
import scripts.mymod
This works correctly:
> PYTHONPATH=: python asdf/mycode2.py
hello world
Is there a way to import scripts.mymod in asdf/mycode.py without modifying the sys.path manually, and without changing the PYTHONPATH which has to be set to :?
TL;DR
Python imports are weird and cumbersome and you are probably best off either creating an entrypoint in the top of the directory or using a tool to handle the path manipulations.
Python Imports
Python has two different ways of handling imports, the first is the standard: import numpy as np, which is resolved against the environment PYTHONPATH in order until it finds the requested module. Included in that path is the directory of the file that is currently being executed, which you see in your example in that you are needing to manually remove that from the sys.path.
The other way that python handles imports is via relative paths, which always lead with . like from . import a or import .a or from .. import b. Unfortunately, relative import only work if the file that they are in is not being directly run (This happens if you have two files within a module where one of them imports an object from another, but both of them are intended to be imported by and external script). This is because python uses the builtin name global to resolve the relative path, which if the file is being directly run from a shell gets overridden as "__main__".
Consider a file structure:
.
├── a.py
└── b.py
where a.py is
import b
and b.py is
print(__name__)
If you run:
python3 b.py # prints "__main__"
python3 a.py # prints "b"
So, if you want to be able to import the scripts/mymod.py from asdf/mycode2.py, you could put change the import to:
from .. import scripts.mymod
But, if you do that, you will not be able to run the asdf/mycode2.py file directly, you will need to create a third file somewhere else that imports asdf/mycode2.py and run that third script (these files are called entrypoints). For example:
python-test » tree
.
├── asdf
│ ├── __init__.py
│ ├── mycode2.py
│ ├── mycode.py
│ └── scripts
│ └── __init__.py
├── __init__.py
├── entrypoint.py
└── scripts
├── __init__.py
└── mymod.py
where entrypoint.py is
import asdf.mycode2
An Alternative Approach
The alternative approach would be to develop a tool which handles manipulating python's sys.path to allow relative imports even when the current file is being run a "__main__". Full disclosure, I am currently working on such a tool based off nodejs's require function, called repyrer, which is pip-installable as
pip3 install repyrer
It allows you to do this kind of thing:
from repyrer import repyre
mymod = repyre('../scripts/mymod')
which will work even if you run the file directly.
Hope that helps!
My folder structure is as follows
./fff
├── __init__.py
├── fg
│ ├── __init__.py
│ └── settings
│ ├── __init__.py
│ └── settings.py
└── obng
└── test.py
I want to import the settings.py inside fg/settings as a module into the test.py
I have added the line
from ..fg.settings import settings
But when I run it, it gives me the following error
Traceback (most recent call last):
File "/mnt/d/Repos/fff/obng/test.py", line 1, in
from ..fg.settings import settings
ImportError: attempted relative import with no known parent package
This style of relative importing is supported as per https://docs.python.org/3/reference/import.html#package-relative-imports
What am I doing wrong here?
It is a matter of how you run your project - you should run from the parent directory of the top-level package as in
$ cd ../fff
$ python -m fff.obng.test # note no py
Then relative imports will be resolved correctly. It is an antipattern running a script directly from its folder
Normally you can't use relative imports when you run your python module as main module like python filename.py but there is a hack using __package__ to achieve this. Remember __package__ is how python resolves relative imports:
1- Create a file called __init__.py in your root directory - fff. ( I can see that you have it, I mentioned for completeness)
2- Put this code on top of your test.py module:
if __name__ == '__main__' and not __package__:
import sys
sys.path.insert(0, <path to parent directory of root directory - fff>)
__package__ = 'fff.obng'
Note: sys.path is where python searches for modules to import them.
3- Now place your relative import statement after the code above (inside the if statement, because we don't wanna mess when your test.py is being imported) :
from ..fg.settings import settings
Now you can call you test.py, it will run without problem. I don't recommend using these hacks but showing the flexibility of the language and doing exactly what you wanna do in some cases is beneficial.
Other good solutions: Absolute import I think is easier and cleaner than this. In addition take a look at #Mr_and_Mrs_D's answer another good solution would be to run your module with -m command-line flag.
Relative imports are based on the name of the current module. When running
python fff/obng/test.py
the name of test.py will be __main__ and the import will not work.
What will work is having another script called "test.py" outside the fff module that imports the fff.obng.test
fff_top
├── fff
│ ├── fg
│ │ ├── __init__.py
│ │ └── settings
│ │ ├── __init__.py
│ │ └── settings.py
│ ├── __init__.py
│ └── obng
│ ├── __init__.py
│ └── test.py
└── test.py
with fff_top/test.py:
import fff.obng.test
Then, running the "external" test.py should be ok:
python fft_top/test.py
Alternatively, I would recommend dropping relative imports entirely. One way to do this is using a virtual environment for every package you write, using for example the venv library:
python -m venv venv
Then, add a setup.py in the root folder with the content:
from setuptools import setup, find_packages
setup(name="fff", packages=find_packages())
and change the imports in obng/test.py:
from fff.fg.settings import settings
Finally, activate your virtual environment:
source venv/bin/activate
and install your package in editable mode:
pip install -e .
Then, after you have completed all the steps above:
python fff/obng/test.py
should work.
In Linux, you could create a symbolic link:
$ ln -s ../folder1 mymodules
$ python
>>> import mymodules.myfancymodule as fancy
This question has been asked in different contexts, but I still am unable to get a successful absolute import.
I have this directory structure
21:48:07$ tree
start_dir
└── top_package
├── README.md
├── __init__.py
├── __pycache__
│ ├── __init__.cpython-37.pyc
│ └── secrets.cpython-37.pyc
├── subpackage1
│ ├── __init__.py
│ └── test.py
├── requirements.txt
├── secrets.py
└── subpackage2
└── __init__.py
All I want to be able to do is from test.py be able to import from secrets. Here is the contents of test.py:
# import top_package # ImportError: No module named top_package
# from top_package import secrets # ImportError: No module named top_package
import top_package.secrets # ImportError: No module named top_package
None of these have worked. I've read through multiple pages on having multiple subpackages in a top level package but still no dice.
Is this simply because I'm trying to execute the file directly by running python test.py instead of calling this from start_dir? If so, what's the best way to develop and work around this?
Goal: I want to be able to absolute import secrets from test.py. If possible, I do not want to do anything with the site-packages or with the setup.py. I want to be able to run python test.py and have the line: import top_package not give a ImportError no module found. Please advise the best course of action.
So you created top_package and want to import it anywhere within your system. You can do so by copying the package folder to the site-packages folder of your environment. This is a path where python would look while importing packages.
Assume a Python package (e.g., MyPackage) that consists of several modules (e.g., MyModule1.py and MyModule2.py) and a set of unittests (e.g., in MyPackage_test.py).
.
├── MyPackage
│ ├── __init__.py
│ ├── MyModule1.py
│ └── MyModule2.py
├── README.md
├── requirements.txt
├── setup.py
└── tests
└── MyPackage_test.py
I would like to import functions of MyModule1.py within the unittests of MyPackage_test.py. Specifically, I would like to import the functions both before as well as after package installation via setup.py install MyPackage.
Currently, I am using two separate commands, depending on the state before or after package installation:
# BEFORE
import sys, os
sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'MyPackage'))
# AFTER
import MyPackage
Can this be done with a single command?
Option 1:
It seems that the following command does what I need:
sys.path.append(os.path.join(__file__.split(__info__)[0] + __info__), __info__)
Option 2:
Depending on the location of __init__.py, this also works:
sys.path.append(os.path.dirname(os.path.split(inspect.getfile(MyPackage))[0]))
Option 3:
Moreover, the ResourceManager API seems to offer additional methods.
in pycharm IDE you can can import method easily.by setting the working directory to the folder which contains all files.And, then in MyPackage __init__.py file import all the function from MyModule1.py and MyModule2.py .
then in MyPackage_test.py you can use
import MyPackage
from MyPackage import xyz
I am confused how to structure a project containing a package and a unittest.
I have this directory structure:
TestProject/
├── package
│ ├── constants.py
│ ├── __init__.py
│ ├── package.py
│ └── tests
│ └── test_package.py
└── usage.py
constants.py
A = 1
__init__.py
from .package import give_A
package.py
from .constants import *
def give_A():
return A
usage.py
from package import give_A
print(give_A())
test_package.py
import unittest
from package import give_A
class Test_package(unittest.TestCase):
def test_simple(self):
self.assertEqual(give_A(), 1)
if __name__ == '__main__':
unittest.main()
Everything works until I try to run the test_package.py module, which understandably cannot import package:
~/Python/TestProject/package/tests $ python3 test_package.py
Traceback (most recent call last):
File "test_package.py", line 3, in <module>
from package import give_A
ImportError: No module named 'package'
However, this seems to be preferred structure of package / tests. How can I make it work? Or should I do it completely differently (structure, directories, imports, test execution...)?
Either:
Setup your PYTHONPATH such that TestProject is in it.
Run tests using unittest like this:
~/Python/TestProject $ python3 -m package.tests.test_package
Use unittest commendline interface:
~/Python/TestProject $ python3 -m unittest package/tests/test_package.py
For more information see: https://docs.python.org/3/library/unittest.html#command-line-interface