Develop Reference

A consistent way to import files such that the import statement is valid for normal python scripts and also within packages

I'm working on a package that has the following directory structure:
setup.py
README.md
src/
__init__.py
file.py
subpackage/
...
tests/
...
I want to import file.py in __init__.py
This import statement works when I directly run __init__.py:
import file
.
.
.
object = file.class()
But it fails when I install the package using pip by using python -m pip install . (the . refers to the root package directory) and call __init__.py using an entry point defined in setup.py
I get the following error for the same import statement:
ModuleNotFoundError: No module named 'file'
I observed that the import succeeds in the entry script if I do:
from . import file
But it fails when I run init.py the regular way with this error:
ImportError: attempted relative import with no known parent package
For reference, here's the entry script defined in setup.py:
entry_points={
'gui_scripts': [
'runpack =packagename.__init__:main',
],
}
Any ideas as to what I'm doing wrong?
Thank you.

First, your file directory structure does not seem correct. I would recommend you to rename the src directory to packagename (or something like that), which will be your "top-level importable package".
I also recommend to only ever use absolute imports, which means that all imports should start from the top-level importable packages. So to import from your own project the import statements should start from packagename. For example import packagename.file or from packagename import file.
It seems to me like your entry point should be runpack=packagename:main, the __init__ is not needed here.
I would also add a packagename/__main__.py file with a content similar to this:
import packagename
def my_main():
return packagename.main()
if __name__ == '__main__':
my_main()
Finally you could run your code as either python -m packagename or runpack but you definitely should not run your code as python packagename/__init__.py or python packagename/__main__.py.

Import a module from a sub package not working

My file structure is
project/
__init__.py
features/
__init__.py
CompareText.py
tests/
test.py
in test.py I am trying to import CompareText
from project.features import CompareText
I get an error of:
ModuleNotFoundError: No module named 'features'`
I checked the documentation and I think my import statement is correct. How can I fix it?

Add an __init__ file in test. Your project directory should look like this:
project/
__init__.py
features/
__init__.py
CompareText.py
tests/
__init__.py
test.py
Then in project/tests/test.py the following import statement will work:
from ..features import CompareText
Oh, and this will still raise an error if you try to run it directly. In the question you said you tried to import it like this:
from project.features import CompareText
This will only work if the parent directory of project is in Python's module search path. So, if you want to run the tests directly then modify the module search path as needed (See: sys.path).

Your import statement is supposed to look like this :
(But make sure your working directory is the same directory as your project folder is located during execution)
from project.features import CompareText
This is supposed to work if your current path while executing the script has the project folder
If you execute it while inside project folder you can use:
from .features import CompareText
Hope this helps!

I assume you are running test.py as a script. test.py needs to find the project package and two ways to do that are to make your project installable or to hack sys.path.
Installable
First, change your directory structure a bit so that project is a subdirectory of some anonymous directory you happen to be using for development. If you are checking this stuff into source control, it needs to be written so that it can be checked out anywhere. Move tests down one directory.
mydevdir/
setup.py
project/
__init__.py
features/
__init__.py
CompareText.py
tests/
test.py
How write a setup.py. This can get quite complicated. You can read Building and Distributing Packages with Setuptools and lookup other resources on the net, but a minimalist setup.py is
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(name='project',
version='0.1',
description='This is project: project',
packages=find_packages(),
)
Now, while in mydevdir do python setup.py develop. Or you can actually produce an install package and put it in a virtual env for test.
Hack sys.path
It may be easier to hack paths in test.py. Note that this will need to be undone if you make project installable later. Just add to the top of test.py
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).absolute().parents[2]))
This puts the parent directory in the python path and now project will be found on import. It runs the risk that a .py file in the same directory as project can mask an installed module. If you have a local csv.py and you import csv, you'd get the local.

python import module fails

custom/
scripts/
__init__.py
file1.py
utils/
__init__.py
utilFile1.py
utilFile2.py
utilFile3.py
I have the above file structure I'm struggling trying to figure out how to import modules in python. I have gone through stackoverflow posts that have the same question but I still couldn't figure out why I can't get it to work.
I am trying to import these modules in maya and when I run
from utils import utilFile1 I get
ImportError: cannot import name utilFile1.
Running from custom.scripts.utils import utilFile1 gives me this error ImportError: no module named custom.scripts.utils. However if I run import file1 it imports without any errors
I have appended custom/scripts to sys.path and when that didn't work, tried appending custom/scripts/utils as well, but that didn't work either. Based off some of the posts on stackoverflow, I saw some people suggesting to run "python -m" but I'm not sure if I should run this or where to execute it from.
I'm really at a loss as to why I can't get it to work at all.

you have make it a python package by following steps,
in your root dir create setup.py,
in your case
custom/
setup.py
scripts/
...
...
in setup.py
from setuptools import setup, find_packages
setup(
name='your_package_name',
description='bla bla bla',
version='0.0.1-dev',
install_requires=[
],
)
and then hit
pip install -e .
for more info refer to this docs

Please update the directory structure as below.
Python might not be considering custom as a module as it doesn't have __init__.py file,
custom/
__init__.py
scripts/
__init__.py
file1.py
utils/
__init__.py
utilFile1.py
utilFile2.py
utilFile3.py

Yash's method works, I believe that's what companies use to setup their environments. An alternative way is to add your scripts path as a PYTHONPATH in your maya.env file instead of using sys.path.append. I'm not sure why there's a difference or if it's coz I'm trying this on windows, but oddly enough it worked in my case.

Importing modules with same name

Basically, I have two python projects, one is located under myapp/screening and the other myapp/server. I'm currently developing the server module and want to import functions from screening using myapp.screening.
My folder structure is as shown bellow:
myapp/
screening/
screening-env/
myapp/
__init__.py
screening/
__init__.py
screening_task.py
submodule1/
# __init__.py and ub module files
submodule2/
# __init__.py and sub module files
submodule3/
# __init__.py and sub module files
tests/
# __init__.py and test scripts
setup.py
server/
server-env/
myapp/
__init__.py
server/
__init__.py
server_task.py
tests/
__init__.py
server_test.py
I structured my project following this answer.
My setup.py is basically as bellow:
from setuptools import setup
setup(
name='myapp-screening',
version='0.1.0',
packages=[
'myapp.screening',
'myapp.screening.submodule1',
'myapp.screening.submodule2',
'myapp.screening.submodule3'
],
)
I activated my server-env and installed the screening project by navigating to myapp/screening/ (same directory as setup.py) and ran python setup.py develop.
Finally, both server_test.py and server_task are such as bellow:
from myapp.screening.screening_test import foo
foo()
When I run python -m myapp.server.server_task or python -m test.server_test I get:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'myapp.screening'
This error makes sense if I'm running python -m myapp.server.server_task because local myapp existis and might be overwriting the installed myapp that contains the screening modules.
Is there a way to import stuff from screening using from myapp.screening.screening_task import foo?!

So, after some more research I found this similar (in a way) question that leads to import python modules with the same name and How do I create a namespace package in Python?.
The answer for "importing modules with same name" is not useful since it says to rename one module or turn the project directory into a package.
The other question is exactly what I want. It basically talks about the pkgutil with which you can 'append' modules to a given namespace.
I understand and share some opinions against this technique for some cases (such as this) but since it's presented here sometimes you want separated structures so you don't patch everything togheter even when you don't want everything

trying to make paths work - attempted relative import beyond top-level package

I can't make this work..
My structure is:
program_name/
__init__.py
setup.py
src/
__init__.py
Process/
__init__.py
thefile.py
tests/
__init__.py
thetest.py
thetest.py:
from ..src.Process.thefile.py import sth
Running: pytest ./tests/thetest.py from program_name gives :
ValueError: attempted relative import beyond top-level package
I tried also other approaches but i am receiving various errors.
But I would expect for the above to work.

ValueError: Attempted relative import in non-package
States that you're trying to use relative import in the module, which are to be used for packages i.e. to make it a package add __init__.py and call the thetest.py from some file outside the package.
Directly running thetest.py from interpreter won't work.
Relative imports require that the module which uses them is being
imported itself either as package module.
Suggestion 1:
The current tests directory has a __init__.py file but that doesn't allow you to run it as a module (via shell) - to make your current (relative) import work, you need to import it in an external (to package) file/module - let's create a main.py (can name it anything you like):
main.py
program_name/
__init__.py
setup.py
src/
__init__.py
Process/
__init__.py
thefile.py
tests/
__init__.py
thetest.py
src/Process/thefile.py:
s = 'Hello world'
tests/thetest.py:
from ..src.Process.thefile import s
print s
main.py:
from program_name.tests.thetest import s
Executing main.py:
[nahmed#localhost ~]$ python main.py
Hello world
Suggestion 2:
Execute the file just above root dir i.e. one level up the program_name/ , in the following fashion:
[nahmed#localhost ~]$ python -m program_name.tests.thetest
Hell World
P.S. relative imports are for packages, not modules.

Just solved a similar problem with a lot of googling.
Here's two solutions without changing the existing file structor:
1
The way to import module from parent folder from ..src.Process.thefile.py import sth is called "relative import".
It's only supported when launching as a package from the top-level package. In your case, that is launching command line from the directory which contains program_name/ and type (for win environment)
python -m program_name.tests.thetest
or simply (useful for many pytest files):
python -m pytest
2
Otherwise -- when trying to run a script alone or from a non top-level package --
you could manually add directory to the PYTHONPATH at run time.
import sys
from os import path
sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))
from src.Process.thefile import s
Try the first one first see if it's compatiable with the pytest framework. Otherwise the second one should always solve the problem.
Reference (How to fix "Attempted relative import in non-package" even with __init__.py)

When importing a file, Python only searches the current directory, the directory that the entry-point script is running from.
you can use sys.path to include different locations
import sys
sys.path.insert(0, '/path/to/application/app/folder')
import thefile