I am currently working in the following project, and running into some difficulty with imports. I previously came from a Ruby background before Python, so I suspect I'm just missing something.
-src
--project
---actions
----some .py files
---config
----some .py files
---db
----some .py files
-tests
--some .py files
-run.py
Some of the actions I'd like to do are:
import src/project/config/file.py from run.py
import between second level folder in project (ie. file in actions imports something from config)
import any file into a test
Would anyone have any advice on how to accomplish this?
So it's quite easy to reference something going deeper into the directory . For example, from run.py, you would be able to use something such as:
from src.project.config.file import foo
It's easy to access something in the same directory as well, say you're in src/file1.py trying to access src/file2.py, this would just be:
from file2 import foo
This works too. However if you were trying to import upward say from file.py to run.py and tried running something like this:
from ... import run.foo
You would get the following error:
ValueError: attempted relative import beyond top-level package
The problem is that Python is assuming that your top-level package is wherever the file being run is located. That's why you can import everything on the same level and deeper but can't get at anything above.
An easy hack to get past this is to just append the path to your file to your sys.path so:
import sys
sys.append('path/to/run')
from run import foo
So that's the answer to you question.
Based on the way your project is setup though it seems like you want to make a module and that's a bit more hands on, I suggest reading the documentation it's pretty good.
Related
am beginner in python and working on multi folder structure and struggling to refer the files using relative path. tried in different ways to refer the files in other folders but ended up in error "attempted relative import with no known parent package". Can somebody explain how the relative paths will work in python?
here is the example what am trying to achieve..
folder structure
Main
source
Function1
inputs
1.py
2.py
uts
test_1.py
code_1.py
code_2.py
Now inside test_1.py am trying to refer code_1 using relative path like
from ..code_1 import Code
from .code_1 import Code
but, ended up in error..
some where I found that we need to have init.py in each folder to use relative path and I created empty init.py in each folder but, no use..
I may get the answer but more importantly need to know how this relative path works in python.
Thanks in advance
I've also had issues with relative imports in Python and thus I've created an experimental, new import library: ultraimport
It gives you more control over your imports and lets you do file system based imports.
You could then write in your test_1.py:
import ultraimport
Code = ultraimport('__dir__/code_1.py', 'Code')
This would import code_1.py from the same directory as test_1.py.
This will always work, no matter how you run your code.
I have looked at I think 5 different answers to this problem, yet none of them have worked for me yet. For reference, I've looked through all of these posts:
Relative imports for the billionth time
Attempted relative import with no known parent package
"Attempted relative import with no known parent package"
From what I've gathered, there are two solutions to this problem:
Move the .py file you're trying to import functions from into the same directory as the script you're trying to run (this works, but it is not a good solution, I should be able to import from a parent directory without this error)
Create a __init__.py file in the directory of the .py file you're trying to import from, and use import package_name to it. (I have tried this, but same issue)
Here is my project's structure:
I'm trying to run the test.py script, which (attempts) to import the function add_technical_indicators from the add_technical_indicators.py file. My import statement looks like this:
from ..utils.add_technical_indicators import add_technical_indicators
Looking at the folder structure again, I have to go UP one directory, then into the utils folder to bring in the add_technical_indicators .py file, and finally the function add_technical_indicators.
Here's what I have tried so far:
from ..utils.add_technical_indicators import add_technical_indicators
from .utils.add_technical_indicators import add_technical_indicators
from utils.add_technical_indicators import add_technical_indicators (this doesn't work of course because add_technical_indicators is not in the same folder as the script being run)
Created an __init__.py file in the utils folder that reads import add_technical_indicators
Created an __init__.py file in the misc folder that reads import test
None of it works. I need a concise and actionable answer as to why this is still not working. I'm running Python 3.7.9, Windows 10, and VS code in case that matters.
I have looked through previous, repeat answers but none of them have worked for me, so although this IS a duplicate question, please do not close it until I have a solution because linking to the already "answered" questions didn't help me.
short solution
In test.py import as from utils.add_technical_indicators import add_technical_indicators
and run as python -m misc.test (not test.py) in parent directory(in STOCK_PEAKS_ADN_THROUGHS_2)
explained
python cannot import files from upper directory of your working directory unless you add them to PATH, so from ..utils.add_technical_indicators import add_technical_indicators won't work.
python finds files starting from your working directory, so from utils.add_technical_indicators import add_technical_indicators in test.py can find utils/add_technical_indicators.py if you run script at parent directory.
python -m misc.test will run misc/test.py in parent directory. Note that python misc/test.py will run script in misc directory and will give you same error.
You did the right thing here: from ..utils.add_technical_indicators import add_technical_indicators
The main issue must be in the imports of utils/add_technical_indicators.py file
Did you try importing stuffs inside add_technical_indicators.py relative to itself (if so, those imports only work when add_technical_indicators.py is run instead of test.py)
In contrast to what minolee has said Python actually can import files from upper directories. Python has a built-in SourceFileLoader that can load Python code from any file in the file system, also from any upper directory of your working directory.
It requires some boilerplate code, so I have decided to wrap this approach into an experimental import library: ultraimport
ultraimport can do file system based imports. In your test.py you could then write:
import ultraimport
add_technical_indicators = ultraimport('__dir__/../utils/add_technical_indicators.py')
This will always work, no matter what is your current working directory and no matter what is your current sys.path. You don't need to create __init__.py files and it also works if you run the code as a script or as a module.
One caveat when importing files like this is if the imported code contains further relative imports. ultraimport has a built-in preprocessor to automatically rewrite subsequent relative imports so they continue to work.
I have the following directory structure:
base_folder
methods_folder
method_1.py
method_2.py
.
.
.
method_n.py
class_methods.py
top_class.py
class_methods.py imports the other files in the same directory, like this:
from method_1 import method_1
from method_2 import method_2
.
.
.
from method_n import method_n
(obs: these methods files has a method with its own file names inside them)
If I run class_methods.py by myself, no problem. But if I try to run top_class.py, which imports class_methods.py, I get the error no module named method_1
So, when executing top_class.py, it is not seeing the files in methods_folder/. Why?
the correct import inside top_class.py would be from methods_folder.method_n import method_n. This is because you are treating methods_folder as a package. If you are running a version of Python that is before 3.3 you must also unclude __init__.py file inside the methods_folder in order to turn it into a package.
Files only have direct access to things they import. Say we have a.py which imports b.py, and b.py imports c.py. When running within a.py functions in b.py that use c.py, this will work fine, because a has access to b, and b has access to c. This does not mean, however, that the imports chain (as in C++) and you can use functions from c in a. You will get an error, because a can only see the contents of b, which it imported.
So if you want to use all your method_i.py files from within top_class.py, you need to import them directly in the same file.
Edit: You also have some other issues. To import other files in a subfolder, you would need to, inside top_class.py, call import methods_folder.method_i. To import something in the same directory, just use import method_i. Since you have a method of the same name in each file, what you have works fine in class_methods.py You also need to create an empty file called __init__.py in any folder containing python files you intend to import to/from which lets python know it's allowed to look there.
You can create an importable package in python in one of two ways. The first way is what you are doing: you create a file called my_package.py and import it with import my_package. This is commonly used for simpler packages that don't need to be further broken up. For this to work, your .py file has to be on the PYTHONPATH which is an environment variable that tells python where to look for packages. If not defined, there are some default places that python will use to look for packages. One of these default locations is the current working directory, which is why your first set of imports works.
In theory you should be able to run the second piece of code from the same location (python ../top_class.py) and use the same import style, but I assume you are changing directories to run that file. This means your files are no longer in the current working directory and no longer findable by python.
One way to get your code to work using the existing style would be to define PYTHONPATH with the location of your methodX.py files. You will typically add to the python search path like this:
PYTHONPATH=$PYTHONPATH:./methods_folder python top_class.py
This tells python to look in methods_folder, in addition to the standard places, when you try to import something. Playing with the PYTHONPATH gets a little annoying after a while, so I actually prefer the next approach.
A second way to create a package is by creating a folder with an __init__.py file inside. This tells python that you want it to treat that directory as a package. This is the preferred style for more complicated pieces of code that might benefit from organization across multiple files. For your example, you could organize your code in the following way:
base_folder
methods_folder
__init__.py
method_1.py
method_2.py
.
.
.
method_n.py
class_methods.py
top_class.py
And then your import in top_class.py would look like this:
from methods_folder.method1 import method1
from methods_folder.method2 import method2
from methods_folder.method3 import method3
This has the effect of creating a top level methods_folder package with modules method1, method2, etc. Because methods_folder is in the same directory as the one you are running top_class.py from, python picks that up as a package using the default PYTHONPATH and lets you import from within it.
I assume you are running them from their respective directories? Unless they are installed in your Python path (I'm going to assume they are not), then Python will by default look for imports in your current directory. So when you run class_methods.py from its directory, then it can find methods_1.py in order to satisfy from methods_1 import method_1. But when you execute top_class.py, it looks for methods_1.py or methods_1/__init__.py, neither of which it fines from that directory.
Assuming Python 3, you would need to use relative imports in class_methods.py.
# class_methods.py
from .methods_1 import method_1
from .methods_2 import method_2
This will let you run it from top_class.py. Unfortunately, you can't use relative imports when running a script in the same package, so you wouldn't be able to run class_methods.py directly.
Another option in that case is to keep the absolute imports in class_methods.py and add the methods folder to the path in top_class.py.
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), 'methods'))
import class_methods
from methods_1 import method
Of course editing sys.path is fine for small scripts and standalone things. But if this grows past being a script, you'll need to be careful about doing it, and you'll probably just want to come up with another solution. Likely, the best thing is to create a package that you install (you can still do this from your source directory while you are developing) and then you can import the same way from anywhere.
The recommended way of running a python script is using the -m switch from the parent of your root package - so in your case:
$ cd base_folder
$ python -m top_class
Python will automatically add the base_folder to its sys.path and you don't need to do any sys.path/PYTHOPATH hacks that are just this - hacks that bloat the code with boilerplate and will blow when least expected.
Now to run the class_methods the correct way is also
$ cd base_folder
$ python -m methods_folder.class_methods
but then the imports in class_methods should be modified to either absolute:
from methods_folder.method1 import method1
...
or relative:
from .method1 import method1
...
I have trouble importing package.
My file structure is like this:
filelib/
__init__.py
converters/
__init__.py
cmp2locus.py
modelmaker/
__init__.py
command_file.py
In module command_file.py I have a class named CommandFile which i want to call in the cmp2locus.py module.
I have tried the following in cmp2locus.py module:
import filelib.modelmaker.command_file
import modelmaker.command_file
from filelib.modelmaker.command_file import CommandFile
All these options return ImportError: No modules named ...
Appreciate any hint on solving this. I do not understand why this import does not work.
To perform these imports you have 3 options, I'll list them in the order I'd prefer. (For all of these options I will be assuming python 3)
Relative imports
Your file structure looks like a proper package file structure so this should work however anyone else trying this option should note that it requires you to be in a package; this won't work for some random script.
You'll also need to run the script doing the importing from outside the package, for example by importing it and running it from there rather than just running the cmp2locus.py script directly
Then you'll need to change your imports to be relative by using ..
So:
import filelib.modelmaker.command_file
becomes
from ..modelmaker import command_file
The .. refers to the parent folder (like the hidden file in file systems).
Also note you have to use the from import syntax because names starting with .. aren't valid identifiers in python. However you can of course import it as whatever you'd like using from import as.
See also the PEP
Absolute imports
If you place your package in site-packages (the directories returned by site.getsitepackages()) you will be able to use the format of imports that you were trying to use in the question. Note that this requires any users of your package to install it there too so this isn't ideal (although they probably would, relying on it is bad).
Modifying the python path
As Meera answered you can also directly modify the python path by using sys.
I dislike this option personally as it feels very 'hacky' but I've been told it can be useful as it gives you precise control of what you can import.
To import from another folder, you have to append that path of the folder to sys.path:
import sys
sys.path.append('path/filelib/modelmaker')
import command_file
I'm developing a python framework and want to do imports based on the top-level package (the project name). Users will use the framework by copying the entire framework and writing their own modules within.
My current structure looks like this:
myapp/
config.py
docs/
framework/
main.py
utils.py
file.py
lib/
some_module.py
unit_tests/
test_utils.py
I want to be able to use the following import in python files in lib and unit_tests in the following way:
from myapp.framework import utils
Is there a simple way to do this? I've tried doing sys.path.append() hacks but they don't really work. If there is a truly pythonic way to achieve this, I don't mind going extra lengths to get it working.
EDIT: Well I tried the sys.path.append() again and it actually works but it really is an in-elegant solution and I'd really like to hear if there's another way.
For short: You can't have two modules with the same name one inside the other. You have the myapp folder and the myapp.py file
That's because of the order of importing modules.
Inside the current directory
The current package
Global package folder
So, if you're trying to import myapp.lib.some_module in config.py the interpreter will first see that you have myapp.py in the same folder and literaly try to import lib.some_module from that file, which doesn't exist.
Would be something as trying to import myapp.myapp.lib.some_module from outside that module.
So, the best thing for you to do is to rename the myapp.py file