I have had it up to here with Python's import system... thought I'd finally got something reliable and then the inexplicable happens!
This is the directory structure of my application:
/
- my-application/
- subpackage/
- __init__.py
- my_module.py
- __init__.py
- tests/
- subpackage/
- __init__.py
- test_my_module.py
- __init__.py
- conftest.py
- run.py
- spark.py
I run all my tests through tests/run.py, which looks like the following (in an attempt to resolve all the import problems):
import os
import pytest
import sys
rootdir = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
sys.path.insert(0, os.path.abspath(os.path.join(rootdir, "my-application")))
sys.exit(pytest.main([os.path.join(rootdir, "tests")]))
This worked like an absolute charm, until I made one modification to the file /my-application/subpackage/my-module.py - I added a local function call. So e.g. my_module.py:
def foo():
pass
def run_my_module():
def bar():
foo() <---- Added this line
bar()
print("Ran")
UPDATE: THIS works fine:
def foo():
pass
def run_my_module():
def bar():
pass
foo()
bar()
print("Ran")
As soon as I added that local function call, the tests stop working, with the error No module named "subpackage".
The test_my_module.py looks like this (basically):
from subpackage.my_module import run_my_module
def basic_test():
run_my_module()
Note that in test_my_module.py I am using subpackage as the first part of my import statement, because I am using the run.py file that sets my-application as a system path. If I change the import to start with my_application I get the same error referring to my_application.py.
I am still learning python, so suggest any change to my application structure you like. I can't believe the hassle of this import system - I do feel like I'm missing something basic here...
Thank you in advance!
Managing import paths manually is difficult.
A setup.py is the best way to manage python packages.
by convention package names should use _ not -.
Create a setup.py with this content next to my_application/
from setuptools import find_packages, setup
setup(
name='my_application',
version='0.0.1',
packages=find_packages(),
)
I recommend an application structure like the following:
$ tree
├── my_application
│ ├── __init__.py
│ ├── spark.py
│ └── subpackage
│ ├── __init__.py
│ └── my_module.py
├── setup.py
└── tests
├── conftest.py
└── subpackage
├── __init__.py
└── test_my_module.py
Install package locally
python setup.py develop
This will symlink (magically) install your package into the python package path
Now in any scripts you can use paths as you'd expect e.g.
from my_application.subpackage.my_module import run_my_module
also recommend you use a virtualenv
More on setup.py here
Related
I am trying to make a package that has 2 subpackages. The directory structure looks like this:
project_folder
└── package_name
├── __init__.py
├── subpackage1
├── __init__.py
├── script1.py
├── subpackage2
├── __init__.py
├── script2.py
The __init__.py file in package_name is empty. The __init__.py in subpackage1 looks like this:
from .subpackage1 import *
and let's say script1.py has this function:
def add(a, b):
return a+b
Now script2.py in subpackage2 has a collection of functions that may use some of the functions of script1.py. Let's say script2.py looks like this:
def add_extended(a, b):
return add(a,b) + a
I have read many posts on StackOverflow, but so far I have not managed how to import a submodule that has dependencies on other submodules. I have tried to make __init__.py in subpackage2 look like this:
from .subpackage2 import *
from ..subpackage1.script1 import add
In my mind, I am setting subpackage2 in a way that when it is imported, the function add from subpackage1 is also imported because .. is looking for subpackage1 in the parent directory. However, when I run a script like this:
from package_name.subpackage2 import add_1
a, b = 2, 4
add_extended(a,b)
It raises a NameError: name 'add' is not defined, so it pretty much is telling me that the function add from submodule subpackage1 is not being imported properly when importing subpackage2.
I am using Python 3.8 on Windows 10.
EDIT
Interestingly, if I import the functions from subpackage1 on script2.py itself instead of on __init__.py:
from ..subpackage1.script1 import add
def add_extended(a, b):
return add(a,b) + a
the script below works:
from package_name.subpackage2 import add_1
a, b = 2, 4
add_extended(a,b)
However, I believe that is not how the package is supposed to work. Am I missing any file that should be added to enable me do what I want to do?
I am struggling with nested __init__.py in a Python package I am writting. The Package has the following architecture:
module/
├── __init__.py
├── submodule1
│ ├── __init__.py
│ └── source.py
└── submodule2
├── __init__.py
├── source.py
└── subsubmodule2
├── __init__.py
└── source.py
My intent is to be able to access functions defined in submodule2/source.py through module.submodule2.function and in subsubmodules2/source.py through module.submodule2.subsubmodule2.function.
The first thing I tried was to define __init__.py in submodule2 this way:
from .subsubmodule2 import *
But doing so, I get the functions defined in subsubmodules2/source.py through module.submodule2.function (and module.function).
If I do:
from . import subsubmodule2
I get these functions through module.subsubmodule2.function.
I also tried to define __all__ keyword in __init__, with no more success. If I understand well Python documentation, I guess I could leave empty __init__.py files and it could work, but from my understanding that is not the best practice either.
What would be the best way to access these functions as intended in my module?
in module __init__.py file write the module which you want to import as
from . import submodule1
from . import submodule2
__all__ = ['submodule1', 'submodule2']
Now, in submodule1 __init__.py file write
from . import source
from . import subsubmodule
# if you want to import functions from source then import them or in source.py
# define __all__ and add function which you want to expose
__all__ = ['source', 'subsubmodule']
now in subsubmodule __init__ file define function or class which you want to expose
from . source import *
__all__ = ['source']
# if you want to use * as import, i suggest you to use __all__ in source.py and mention all exposed function there
The __init__.py file represents its respective package. For example, module/submodule2/__init__.py represents module. submodule2 .
In order to pull objects defined in submodules into their package namespace, import them:
# module/submodule2/__init__.py
from .source import *
Since __init__.py is a regular Python module, one can also forgo a separate .source module and define objects directly inside __init__.py:
# module/submodule2/__init__.py
def function():
...
Note that subpackages themselves are already available as their respective name. One does not have to – and in fact should not – import them in the parent module. They will be imported if code using the package imports them.
I have a beeware project and also want to use my own modules in it like Models and Controllers. Also, a module which creates some objects I can test with.
But when I want to import the module to create the test objects and use the method it just throws an error:
ImportError: attempted relative import beyond top-level package
After some research, I know that the path (directory) structures, where I put my modules in, and where the package is, are important. But where ever I put the modules it has the same (or kinda like this) errors. But I can import my Models to create objects of these classes. I also can't decide where the start point of the briefcase is.
Here my structure currently:
/Project_Dir (own created)
/briefcase_project (created from briefcase)
/src
/Models (own created)
/app_directory (created from briefcase)
here is the __main__.py and the __init__.py (the start point I guess) and the app.py (where beeware code is, and also my module import from Test)
/Test (own created, here is a file with a method I want to call)
Sadly there is not so much stuff to find about beeware so I could find a solution.
Please help. Thanks ^^
I did the following to workaround the issue. The example using the Beeware Tutorial 2 source code is on Github
.
├── __init__.py
├── __main__.py
├── app.py
├── mylib <--- # my lib.
│ ├── __init__.py
│ └── testlib.py
└── resources
├── __init__.py
├── beewarecustomlibexample.icns
├── beewarecustomlibexample.ico
└── beewarecustomlibexample.png
2 directories, 9 files
The mylib/testlib.py
def test(text: str) -> str:
return f"Hello: {text}"
In the app.py:
import toga
from toga.style import Pack
from toga.style.pack import COLUMN, ROW
from beewarecustomlibexample.mylib.testlib import test # Import custom lib
class BeewareCustomLibExample(toga.App):
def startup(self):
...
def say_hello(self, widget):
# Calling my test method
result = test(self.name_input.value)
self.main_window.info_dialog("Test Dialog", result)
def main():
return BeewareCustomLibExample()
The above is how I got it working. I've built it on MacOS and works fine.
Take your project folder name and then import from there, so if you're tinkering with the tutorial and you've set up a module folder called myModule in the same directory as your app.py and you have a file called file.py with a class called myClass, you might type:
from helloworld.myModule.file import myClass
I have a project which I want to structure like this:
myproject
├── api
│ ├── __init__.py
│ └── api.py
├── backend
│ ├── __init__.py
│ └── backend.py
├── models
│ ├── __init__.py
│ └── some_model.py
└── __init__.py
Now, I want to import the module some_model.py in both api.py and backend.py. How do I properly do this?
I tried:
from models import some_model
but that fails with ModuleNotFoundError: No module named 'models'.
I also tried:
from ..models import some_model
which gave me ValueError: attempted relative import beyond top-level package.
What am I doing wrong here? How can I import a file from a different directory, which is not a subdirectory?
Firstly, this import statement:
from models import some_model
should be namespaced:
# in myproject/backend/backend.py or myproject/api/api.py
from myproject.models import some_model
Then you will need to get the directory which contains myproject, let's call this /path/to/parent, into the sys.path list. You can do this temporarily by setting an environment variable:
export PYTHONPATH=/path/to/parent
Or, preferably, you can do it by writing a setup.py file and installing your package. Follow the PyPA packaging guide. After you have written your setup.py file, from within the same directory, execute this to setup the correct entries in sys.path:
pip install --editable .
Unfortunately, Python will only find your file if your file is in the systems path. But fear not! There is a way around this!
Using python's sys module, we can add a directory to the path just while Python is running, and once Python stops running, it will remove it from the path.
You can do this by:
import sys
sys.path.insert(0, '/path/to/application/app/folder')
import [file]
It is important to import sys and set the directory path before you import the file however.
Good luck!
Jordan.
I would lay out two approaches:
Simply import some_model via absolute importing:
from myproject.models import some_model
Note that the myproject should be treated as an module (i.e. having __init__.py)
Or
You can add the previous path to the sys.path which I use in such parallel level modules:
import sys
sys.path.append('../')
from models import some_model
I have a question regarding one single module that is distributed over multiple directories.
Let's say I have these two file and directories:
~/lib/python
xxx
__init__.py
util
__init__.py
module1.py
module2.py
~/graphics/python
xxx
__init__.py
misc
__init__.py
module3.py
module4.py
So then in my Python modules, I did this:
import sys
pythonlibpath = '~/lib/python'
if pythonlibpath not in sys.path: sys.path.append(pythonlibpath)
import xxx.util.module1
which works.
Now, the problem is that I need xxx.misc.module3, so I did this:
import sys
graphicslibpath = '~/graphics/python'
if graphicslibpath not in sys.path: sys.path.append(graphicslibpath)
import xxx.misc.module3
but I get this error:
ImportError: No module named misc.module3
It seems like it somehow still remembers that there was a xxx package in ~/lib/python and then tries to find misc.module3 from there.
How do I get around this issue?
You can't without an extreme amount of trickery that pulls one package structure into the other. Python requires that all modules in a package be under a single subdirectory. See the os source to learn how it handles os.path.
Python does indeed remember that there was a xxx package. This is pretty much necessary to achieve acceptable performance, once modules and packages are loaded they are cached. You can see which modules are loaded by looking the the dictionary sys.modules.
sys.modules is a normal dictionary so you can remove a package from it to force it to be reloaded like below:
import sys
print sys.modules
import xml
print sys.modules
del sys.modules['xml']
print sys.modules
Notice that after importing the xml package it is the dictionary, however it is possible to remove it from that dictionary too. This is a point I make for pedagogical purposes only, I would not recommend this approach in a real application. Also if you need to use your misc and util packages together this would not work so great. If at all possible rearrange your source code structure to better fit the normal Python module loading mechanism.
This is addressed by Implicit Namespace Packages in Python 3.3. See PEP-420.
This is an adaptation of an answer to a similar question.
Following up on #Gary's answer, the PEP 420 page says to use the following code on shared __init__.py packages.
__init__.py:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
This code should be placed inside the xxx directory's __init__.py.
See the *s below
someroot/
├── graphics
│ └── python
│ └── xxx
│ ├── ****__init__.py****
│ └── misc
│ ├── __init__.py
│ ├── module3.py
│ └── module4.py
└── lib
└── python
└── xxx
├── ****__init__.py****
└── util
├── __init__.py
├── module1.py
└── module2.py
Some setup.sh file to add to the Python Path:
libPath=someroot/lib/python/
graphicsPath=someroot/graphics/python/
export PYTHONPATH=$PYTHONPATH:$libPath:$graphicsPath
Python test code (tested on Python versions 2.7.14 and 3.6.4 using pyenv):
import xxx.util.module1
import xxx.misc.module3 # No errors