I've got a PyCharm "project" in Python, which is to say that I have a folder that is a conglomeration of all sorts of experimental python files, convenience methods/classes, and Jupyter notebooks from following along with online classes.
I've actually just written something that I'm proud of and would like to re-use. I'm finding it difficult to import. I have looked at and attempted implementing the answers to the following questions to no avail:
Can't import my own modules in Python
How to import my own modules in python 3.6?
How to import my own modules the elegant way?
Project structure:
learning_project
|
├───.idea
│ ├───dictionaries
│ └───inspectionProfiles
|
├───decision_trees
├───linear_algebra
├───neural_networks
| ├───based_sequential.py <---------------------------- # Module location #
│ ├───cross-entropy-gradient-descent
│ └───learning pytorch
| ├─── class_notebook.ipynb <---------------------- # Want to import for use here #
| └───Cat_Dog_data
|
└───venv
├───Include
├───Lib
│ └───site-packages
└───Scripts
I have tried the following:
import based_sequential
from based_sequential import ClassName
import based_sequential.ClassName
import neural_networks
from neural_networks import based_sequential
import neural_networks.based_sequential
from neural_networks.based_sequential import ClassName
All result in the error No module named '<pick your poison>'
Question 1: Obviously, what am I missing?
Question 2: Is my organization here part of the problem? I'm starting to suspect that it is.
I also suspect I have some work to do learning the administrative aspects of writing code that is bigger than just one .py file.
I hope you're returning some value in the function/module you're trying to import. If not, check that.
Otherwise, just use sys.path from sys module and direct it towards the file you want to import.
>>> import sys
>>> sys.path
['',
'C:\\Python33\\Lib\\idlelib',
'C:\\Windows\\system32\\python33.zip',
'C:\\Python33\\DLLs',
'C:\\Python33\\lib',
'C:\\Python33',
'C:\\Python33\\lib\\site-packages']
I suggest you check on relative imports.
https://realpython.com/absolute-vs-relative-python-imports/
The way you can solve this problem is shown below:
FILE STRUCTURE I'LL BE USING:
└── project
├── package1
│ ├── module1.py
│ └── module2.py
└── package2
├── __init__.py
├── module3.py
├── module4.py
└── subpackage1
└── module5.py
To import from module 2 while in module 1 use:
from .module2 import function1
In the case of importing the module3 from module2 use:
from ..package2 import module3
IN YOUR OWN CASE, IT SHOULD BE:
from ..based_sequential import whatever
Related
I am trying to import a util package one directory up from where my code is, but I get an ImportError which I don't understand.
I have a number of different variations on the import syntax in Python, none of which are working.
There are a number of similar questions on Stack Overflow, but none have helped me understand or fix this issue.
Of the top of my head, I have tried the following variations:
import util
import ..util
from .. import util
from ..util import parser
from AdventOfCode2022 import util
from ..AdventOfCode2022 import util
from ...AdventOfCode2022 import util
Most of these I guessed wouldn't work, but I tried them anyway to be sure.
Error message:
ImportError: attempted relative import with no known parent package
Directory structure:
.
├── day03
│ ├── input.txt
│ ├── part1.py
│ ├── part2.py
│ └── test_input.txt
└── util
├── __init__.py
└── parser.py
I just want to import my util package from any "day0*/" directory - not sure why Python makes it so hard!
Two options:
Add the full path to ./util/ to your PYTHONPATH environment variable.
For example on Bash, your ~/.bashrc might have export PYTHONPATH="${PYTHONPATH}:/Users/foobar/projects/advent-of-code/util/".
Add sys.path.append('/path/to/application/app/folder') before the import.
The other solutions don't work because:
day03 and the parent directory are not modules with their own __init__.py. Lines like from ..util import parser only work if everything involved is a module.
You are presumably running the code from within ./day03/.
View this as 'I have a bunch of independent Python projects (day01, day02 etc) that all want to share a common piece of code I have living in a different project (util) that lives somewhere else on my computer.'
I'm working on a python 3.8.5 project which has this folder structure:
project/
├── __init__.py
|── foo.py
|── bar.py
├── utils/
│ ├── __init__py
│ └── configurator.py
└── server/
├── __init__.py
├── models.py
In server/models.py I need to use a class Configurator declared inside utils/configurator.py:
# server/models.py
from utils.configurator import Configurator
# some code here ...
Unfortunately I get this error: ModuleNotFoundError: No module named 'utils'. I dont't understand how it cannot find the utils module since I've properly set the init.py file inside that folder.
I've also tried with relative import:
# server/models.py
from ..utils.configurator import Configurator
# some code here ...
this time getting the following error: ImportError: attempted relative import with no known parent package
Finally, I've attempted with an absolute import, with no results again:
# server/models.py
from projects.utils.configurator import Configurator
# some code here ...
The error this time is: ModuleNotFoundError: No module named 'projects'
Based on this previous question it seems correct to me, I can't understand what I'm messing with.
My code editor is Visual Studio Code and when I type from utils. ... it gives the right suggestion, although I don't think this piece of information could be of any utility.
I have a project which I want to structure like this:
myproject
├── api
│ ├── __init__.py
│ └── api.py
├── backend
│ ├── __init__.py
│ └── backend.py
├── models
│ ├── __init__.py
│ └── some_model.py
└── __init__.py
Now, I want to import the module some_model.py in both api.py and backend.py. How do I properly do this?
I tried:
from models import some_model
but that fails with ModuleNotFoundError: No module named 'models'.
I also tried:
from ..models import some_model
which gave me ValueError: attempted relative import beyond top-level package.
What am I doing wrong here? How can I import a file from a different directory, which is not a subdirectory?
Firstly, this import statement:
from models import some_model
should be namespaced:
# in myproject/backend/backend.py or myproject/api/api.py
from myproject.models import some_model
Then you will need to get the directory which contains myproject, let's call this /path/to/parent, into the sys.path list. You can do this temporarily by setting an environment variable:
export PYTHONPATH=/path/to/parent
Or, preferably, you can do it by writing a setup.py file and installing your package. Follow the PyPA packaging guide. After you have written your setup.py file, from within the same directory, execute this to setup the correct entries in sys.path:
pip install --editable .
Unfortunately, Python will only find your file if your file is in the systems path. But fear not! There is a way around this!
Using python's sys module, we can add a directory to the path just while Python is running, and once Python stops running, it will remove it from the path.
You can do this by:
import sys
sys.path.insert(0, '/path/to/application/app/folder')
import [file]
It is important to import sys and set the directory path before you import the file however.
Good luck!
Jordan.
I would lay out two approaches:
Simply import some_model via absolute importing:
from myproject.models import some_model
Note that the myproject should be treated as an module (i.e. having __init__.py)
Or
You can add the previous path to the sys.path which I use in such parallel level modules:
import sys
sys.path.append('../')
from models import some_model
I realize that there are a lot of questions about this on StackOverflow already, but I find the solutions to be entirely unclear and often contradicting of each other.
I have the following scenario: I am writing a python (Python 3.4) package with the following structure:
mypackage/
__init__.py (empty file)
mydir1/
__init__.py (empty file)
mymodule1.py
mydir2/
__init__.py (empty file)
mymodule21.py
mymodule22.py
mydir3/
__init__.py (empty file)
mymodule31.py
mymodule32.py
tests/
__init__.py (empty file)
test_something_in_mydir1.py
test_something_in_mydir2.py
test_something_in_mydir3.py
I want the files within the tests/ directory to contain unit tests of everything inside the package. The problem is that no matter which approach I try, I get this error:
SystemError: Parent module '' not loaded, cannot perform relative import
I should note that I want this to work "out of the box" which means that I do not want to import things using their absolute paths and I do not want to change my path variable. None of these solutions seem very pythonic and its driving me slowly up the wall. Is there really no way to do this?
I've tried a bunch of stuff already, including examples below, but they all seem to produce the same result:
from ..mydir1.mymodule1 import *
from .mypackage.mydir1.mymodule1 import *
from ...mypackage.mydir1.mymodule1 import *
And I've tried overwriting the __all__ variable in each of the __init__.py files as well as doing the imports as shown here: https://docs.python.org/3/reference/import.html#submodules
Can anyone tell me how to structure the imports (for example) in the file test_something_in_mydir1.py in such a way as to have access to all the classes/functions that are found in mymodule1.py for unit testing the package?
If test_something_in_mydir1.py contains:
import unittest
from ..mydir1 import mymodule1
class Test1(unittest.TestCase):
def test1(self):
self.assertEqual(mymodule1.x,1) # mymodule1 trivially contains x=1
And if the unit tests were launched from the directory containing the package (in this case, C:\Test) using the following command:
py -m unittest discover -v
The tests are discovered and run correctly with relative imports:
C:\Test>tree /f
Folder PATH listing
Volume serial number is CE8B-D448
C:.
└───mypackage
│ __init__.py
│
├───mydir1
│ mymodule1.py
│ __init__.py
│
├───mydir2
│ mymodule21.py
│ mymodule22.py
│ __init__.py
│
├───mydir3
│ mymodule31.py
│ mymodule32.py
│ __init__.py
│
└───tests
test_something_in_mydir1.py
__init__.py
C:\Test>py -m unittest discover -v
test1 (mypackage.tests.test_something_in_mydir1.Test1) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
I have a question regarding one single module that is distributed over multiple directories.
Let's say I have these two file and directories:
~/lib/python
xxx
__init__.py
util
__init__.py
module1.py
module2.py
~/graphics/python
xxx
__init__.py
misc
__init__.py
module3.py
module4.py
So then in my Python modules, I did this:
import sys
pythonlibpath = '~/lib/python'
if pythonlibpath not in sys.path: sys.path.append(pythonlibpath)
import xxx.util.module1
which works.
Now, the problem is that I need xxx.misc.module3, so I did this:
import sys
graphicslibpath = '~/graphics/python'
if graphicslibpath not in sys.path: sys.path.append(graphicslibpath)
import xxx.misc.module3
but I get this error:
ImportError: No module named misc.module3
It seems like it somehow still remembers that there was a xxx package in ~/lib/python and then tries to find misc.module3 from there.
How do I get around this issue?
You can't without an extreme amount of trickery that pulls one package structure into the other. Python requires that all modules in a package be under a single subdirectory. See the os source to learn how it handles os.path.
Python does indeed remember that there was a xxx package. This is pretty much necessary to achieve acceptable performance, once modules and packages are loaded they are cached. You can see which modules are loaded by looking the the dictionary sys.modules.
sys.modules is a normal dictionary so you can remove a package from it to force it to be reloaded like below:
import sys
print sys.modules
import xml
print sys.modules
del sys.modules['xml']
print sys.modules
Notice that after importing the xml package it is the dictionary, however it is possible to remove it from that dictionary too. This is a point I make for pedagogical purposes only, I would not recommend this approach in a real application. Also if you need to use your misc and util packages together this would not work so great. If at all possible rearrange your source code structure to better fit the normal Python module loading mechanism.
This is addressed by Implicit Namespace Packages in Python 3.3. See PEP-420.
This is an adaptation of an answer to a similar question.
Following up on #Gary's answer, the PEP 420 page says to use the following code on shared __init__.py packages.
__init__.py:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
This code should be placed inside the xxx directory's __init__.py.
See the *s below
someroot/
├── graphics
│ └── python
│ └── xxx
│ ├── ****__init__.py****
│ └── misc
│ ├── __init__.py
│ ├── module3.py
│ └── module4.py
└── lib
└── python
└── xxx
├── ****__init__.py****
└── util
├── __init__.py
├── module1.py
└── module2.py
Some setup.sh file to add to the Python Path:
libPath=someroot/lib/python/
graphicsPath=someroot/graphics/python/
export PYTHONPATH=$PYTHONPATH:$libPath:$graphicsPath
Python test code (tested on Python versions 2.7.14 and 3.6.4 using pyenv):
import xxx.util.module1
import xxx.misc.module3 # No errors