I have a bunch of scripts for static code analysis.
The get a directory as the command line argument, and they run on all files inside that directory.
Here's the structure of my project:
__init__.py
run.py
message.py
globals.py
react
__init__.py
run.py
check_imports.py
analyze_states.py
next
__init__.py
check_routes.py
analyze_images.py
git
__init__.py
check_size.py
ensure_branch_name.py
run.py => how can I call this and still access message.py?
Now, if I use top-level run.py as the orchestrator to call sub-modules inside sub-packages, everything works great and I can use import message from each sub-module.
But for git package, I want to call it directly. And I want to use functions defined inside message.py. I'm stuck at this point.
I saw Python import from parent package and tried from .. import message but it does not work.
The way I did find out is for the git parent folder to be in your python path.
In general when I am developing I would add this lines in your git/run.py
import sys
sys.path.append('..')
Then you will be able to import message
When you finish with development and it is ready for pip install you need to make sure that message is an importable module in your setup.py for example.
Update
The '..' will only work if you execute the git/run.py script or module from within the git repository as '..' is a relative path.
If you want to execute run.py from anywhere you can do:
import os
import sys
parent_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
sys.path.append(parent_dir)
Just have in mind that __file__ will only exist if you execute run.py in batch mode not from an interactive shell or a notebook.
Related
The directory I have looks like this:
repository
/src
/main.py
/a.py
/b.py
/c.py
I run my program via python ./main.py and within main.py there's an important statement from a import some_func. I'm getting a ModuleNotFoundError: No module named 'a' every time I run the program.
I've tried running the Python shell and running the commands import b or import c and those work without any errors. There's nothing particularly special about a either, it just contains a few functions.
What's the problem and how can I fix this issue?
repository/
__init__.py
/src
__init__.py
main.py
a.py
b.py
c.py
In the __init__.py in repository, add the following line:
from . import repository
In the __init__.py in src, add the following line:
from . import main
from . import a
from . import b
from . import c
Now from src.a import your_func is going to work on main.py
Maybe you could try using a relative import, which allows you to import modules from other directories relative to the location of the current file.
Note that you will need to add a dot (.) before the module name when using a relative import, this indicates that the module is in the same directory as the current file:
from . import a
Or try running it from a different directory and appending the /src path like this:
import sys
sys.path.append('/src')
You could also try using the PYTHONPATH (environment variable) to add a directory to the search path:
Open your terminal and navigate to the directory containing the main.py file (/src).
Set the PYTHONPATH environment variable to include the current directory, by running the following command
export PYTHONPATH=$PYTHONPATH:$(pwd)
At last you could try to use the -m flag inside your command, so that Python knows to look for the a module inside the /src directory:
python -m src.main
I've had similar problems in the past. Imports in Python depend on a lot of things like how you run your program, as a script or as a module and what is your current working directory.
Thus I've created a new import library: ultraimport It gives the programmer more control over imports and lets you do file system based, relative imports.
Your main.py could look like this:
import ultraimport
a = ultraimport('__dir__/a.py')
This will always work, no matter how you run your code, no matter what is your sys.path and also no init files are necessary.
I have a big project:
main
golang
src
file1.go
python
src
file1.py
file2.py
__init__.py
java
src
file1.java
scripts
script.py
validator.sh
venv
bin
pip
python3
pyyaml
dateutil
Python project will use interpreter from:
ven/bin/python3
So anywhere inside
file1.py
file2.py
I can use imports:
import pyyaml
import dateutil
And this will work, by running from CLI:
venv/bin/python3 python/src/file1.py
However I wish to use some functions from file1.py inside file2.py
And have "relative reference" like this (inside file2.py)
from src.file1 import some_function
But having this kind of import and running the same way as before from CLI fails with error:
ModuleNotFoundError: No module named 'src'
What should I do? Pay attention that I have init.py file.
When you do from src.file1 this is relative to your sys.path.
Usually your current working directory is the first element in sys.path.
Thus you need to cd to main/python and then run ../venv/bin/python3 src/file1.py to make the imports work.
I've had similar issues and thus I have created an experimental, new import library: ultraimport.
It gives you more control over your imports and allows file system based imports.
You could then write in file2.py:
import ultraimport
file1 = ultraimport('__dir__/file1.py)'
This will always work, no matter how you run your code
My file structure is
project/
__init__.py
features/
__init__.py
CompareText.py
tests/
test.py
in test.py I am trying to import CompareText
from project.features import CompareText
I get an error of:
ModuleNotFoundError: No module named 'features'`
I checked the documentation and I think my import statement is correct. How can I fix it?
Add an __init__ file in test. Your project directory should look like this:
project/
__init__.py
features/
__init__.py
CompareText.py
tests/
__init__.py
test.py
Then in project/tests/test.py the following import statement will work:
from ..features import CompareText
Oh, and this will still raise an error if you try to run it directly. In the question you said you tried to import it like this:
from project.features import CompareText
This will only work if the parent directory of project is in Python's module search path. So, if you want to run the tests directly then modify the module search path as needed (See: sys.path).
Your import statement is supposed to look like this :
(But make sure your working directory is the same directory as your project folder is located during execution)
from project.features import CompareText
This is supposed to work if your current path while executing the script has the project folder
If you execute it while inside project folder you can use:
from .features import CompareText
Hope this helps!
I assume you are running test.py as a script. test.py needs to find the project package and two ways to do that are to make your project installable or to hack sys.path.
Installable
First, change your directory structure a bit so that project is a subdirectory of some anonymous directory you happen to be using for development. If you are checking this stuff into source control, it needs to be written so that it can be checked out anywhere. Move tests down one directory.
mydevdir/
setup.py
project/
__init__.py
features/
__init__.py
CompareText.py
tests/
test.py
How write a setup.py. This can get quite complicated. You can read Building and Distributing Packages with Setuptools and lookup other resources on the net, but a minimalist setup.py is
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(name='project',
version='0.1',
description='This is project: project',
packages=find_packages(),
)
Now, while in mydevdir do python setup.py develop. Or you can actually produce an install package and put it in a virtual env for test.
Hack sys.path
It may be easier to hack paths in test.py. Note that this will need to be undone if you make project installable later. Just add to the top of test.py
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).absolute().parents[2]))
This puts the parent directory in the python path and now project will be found on import. It runs the risk that a .py file in the same directory as project can mask an installed module. If you have a local csv.py and you import csv, you'd get the local.
I have a very simple scenario like this:
example/
common.py
folder1/
script.py
where script.py file should import common.py module.
If I cd into folder1/ and run the script (i.e. by calling python3 script.py on the command line), the import breaks with the usual error ModuleNotFoundError: No module named 'common'.
I know I could turn the whole parent folder into a package by adding __init__.py files in each subdirectory, however this solution still prevents me to run the script directly from inside folder1/.
How can I fix this?
If you turn both directories into python packages, and the top directory is in your PYTHONPATH, you can run script.py like so:
python3 -m example.folder1.script
If you need to add a parent directory to the search path, you can always do:
import os
import sys
DIR = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
sys.path.insert(0, DIR)
os.path.realpath(__file__) gets the realpath to the file, and os.path.dirname gets the directory name of this file. Doing it twice gets the example directory here. Now, since this is added to the search path, you can import common.
However, you should really consider having a dedicated scripts or bin directory, rather than try to run scripts from subdirectories. Think about writing a library, and import this library rather than individual files.
I can't make this work..
My structure is:
program_name/
__init__.py
setup.py
src/
__init__.py
Process/
__init__.py
thefile.py
tests/
__init__.py
thetest.py
thetest.py:
from ..src.Process.thefile.py import sth
Running: pytest ./tests/thetest.py from program_name gives :
ValueError: attempted relative import beyond top-level package
I tried also other approaches but i am receiving various errors.
But I would expect for the above to work.
ValueError: Attempted relative import in non-package
States that you're trying to use relative import in the module, which are to be used for packages i.e. to make it a package add __init__.py and call the thetest.py from some file outside the package.
Directly running thetest.py from interpreter won't work.
Relative imports require that the module which uses them is being
imported itself either as package module.
Suggestion 1:
The current tests directory has a __init__.py file but that doesn't allow you to run it as a module (via shell) - to make your current (relative) import work, you need to import it in an external (to package) file/module - let's create a main.py (can name it anything you like):
main.py
program_name/
__init__.py
setup.py
src/
__init__.py
Process/
__init__.py
thefile.py
tests/
__init__.py
thetest.py
src/Process/thefile.py:
s = 'Hello world'
tests/thetest.py:
from ..src.Process.thefile import s
print s
main.py:
from program_name.tests.thetest import s
Executing main.py:
[nahmed#localhost ~]$ python main.py
Hello world
Suggestion 2:
Execute the file just above root dir i.e. one level up the program_name/ , in the following fashion:
[nahmed#localhost ~]$ python -m program_name.tests.thetest
Hell World
P.S. relative imports are for packages, not modules.
Just solved a similar problem with a lot of googling.
Here's two solutions without changing the existing file structor:
1
The way to import module from parent folder from ..src.Process.thefile.py import sth is called "relative import".
It's only supported when launching as a package from the top-level package. In your case, that is launching command line from the directory which contains program_name/ and type (for win environment)
python -m program_name.tests.thetest
or simply (useful for many pytest files):
python -m pytest
2
Otherwise -- when trying to run a script alone or from a non top-level package --
you could manually add directory to the PYTHONPATH at run time.
import sys
from os import path
sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))
from src.Process.thefile import s
Try the first one first see if it's compatiable with the pytest framework. Otherwise the second one should always solve the problem.
Reference (How to fix "Attempted relative import in non-package" even with __init__.py)
When importing a file, Python only searches the current directory, the directory that the entry-point script is running from.
you can use sys.path to include different locations
import sys
sys.path.insert(0, '/path/to/application/app/folder')
import thefile