I had a small problem with importing a script that was in a parent folder, but I managed to resolve it using:
import sys
sys.path.append("../")
My directory is like this:
Data
|->->code
|->->script1.py
|->->->->subfolder
|->->->script2.py
When I run script2 (which imports script1) from the subfolder directory, the script runs without problems. But if I try to run script2 from the code directory using:
:~ ./subfolder/script2.py
I get an error :
ImportError: No module named script1
I tried using relative imports but because my code is not structured in packages it doesn't work. Is there a way I can run script2 from both directories (the parent and the child) and still be able to import script1 everytime?
Thank you in advance,
Georgi Nikolov
EDIT: Ok, after I read through all the suggestions, I did a "simple" hack which is quite ugly in my opinion but works quite well:
import sys
parent_folder = sys.path[0].split("/subfolder")[0]
sys.path.append(parent_folder)
import script1
Now I can even call script2 from the root and it will manage to import script1
To import a module that is up a level, you can use this.
import os, sys
sys.path.append(os.path.join(os.path.dirname(__file__), ".."))
An Explanation
__file__ # The full path to your running file.
os.path.dirname # See below. (1)
os.path.join # See below. (2)
sys.path.append # See below. (3)
".." # Universal for "up a level".
os.path.dirname(path) -
Return the directory name of pathname path.
os.path.join(path, *paths) -
Join one or more path components intelligently.
sys.path -
A list of strings that specifies the search path for modules.
You can add a string with the append method.
You should use absolute paths:
import sys, os
HERE = os.path.abspath(os.path.dirname(__file__))
sys.path.append(os.path.join(HERE, ".."))
But better do not do this: Guido views running scripts within a package as an anti-pattern
You should put the standalone scripts in the root folder of the project. A script should not be used as a module and as the main script.
If you cannot move the script to the root, make another bootstrap script at the root which will import your service script running a dedicated function in it.
Related
If I have a project with two files main.py and main2.py. In main2.py I do import sys; sys.path.append(path), and I do import main, which main.py will be imported? The one in my project or the one in path (the question poses itself only if there are two main.py obviously)?
Can I force Python to import a specific one?
The one that is in your project.
According to python docs, the import order is:
Built-in python modules. You can see the list in the variable sys.modules
The sys.path entries
The installation-dependent default locations
Your added directory to sys.path, so it depends on the order of entries in sys.path.
You can check the order of imports by running this code:
import sys
print(sys.path)
As you will observe, the first entry of sys.path is your module's directory ("project directory") and the last is the one you appended.
You can force python to use the outer directory by adding it in the beginning of your sys.path:
sys.path.insert(0,path)
Although I would recommend against having same names for the modules as it is often hard to manage.
Another thing to keep in mind is that you can't import relative paths to your sys.path. Check this answer for more information on how to work around that: https://stackoverflow.com/a/35259170/6599912
When two paths contain the same module name, using sys.path.append(path) isn't going to change the priority of the imports, since append puts path at the end of the python path list
I'd use
sys.path.insert(0,path)
so modules from path come first
Example: my module foo.py imports main and there's one main.py in the same directory and one main.py in the sub directory
foo.py
main.py
sub/main.py
in foo.py if I do:
import sys
sys.path.append("sub")
import main
print(main.__file__)
I get the main.py file path of the current directory because sys.path always starts by the main script directory.
So I could remove that directory from sys.path but it's a bad idea because I could need other modules from there.
Now if I use insert instead:
import sys
sys.path.insert(0,"sub")
import main
print(main.__file__)
I get sub/main.py as sub is the first directory python looks into when searching modules.
I have the following folder format
Main Folder
-DataManagement
-Libraries
-TransformLibrary
__init__.py
transform.py
-DataUsage
-TransformData
main.py
I wish to call Transform.py from main.py. However I need this to be relative, meaning that if somebody clones my project, it can run without changing any paths.
Thanks!
You could use sys.path to both find your current directory, and include your module in sys.path so that Python can find it. However, this solution is a bit messy:
import sys
import os
# Get current directory
current_dir = sys.path[0]
# Create path to 'Main Folder/DataManagement/Libraries'
search_dir = os.path.join(current_dir.split('DataUsage')[0], 'Libraries')
# Make Python search for new path
sys.path.append(search_dir)
from TransformLibrary import transform # transform.py is imported
Desiring to improve my Python style, I ran a PEP-8 style checker on one of my script and it complained about something I don't know how to fix. The prologue of the script is something like:
#! /bin/env python3
import sys
import os
exe_name = os.path.basename(os.path.realpath(__file__))
bin_dir = os.path.dirname(os.path.realpath(__file__))
inst_dir = os.path.dirname(bin_dir)
sys.path.insert(0, inst_dir+'/path/to/packages')
import mypackage.mymodule
and the style checker complain on the import mymodule line, stating that it should be a top of file. But I obviously can't move it before setting the path where it will be found. Is there a good way to achieve this (mandating an environment variable or a shell wrapper are not what I find better than my current code) while respecting PEP-8 recommendations at the same time?
If you want to avoid path manipulation, you may be able to do so by using the under-known .pth feature.
sys.path should begin with the directory containing the main program either by name or by reference as ''. I assume that the file importing mymodule is not part of mypackage, so that the '' entry is not useful for importing mymodule.
sys.path should end with the site-packages directory for the executing binary. That is the normal place for added packages. If you do not want to move mypackage into site-packages, you can extend the latter 'vitually' by putting a mystuff.pth file in it. It should contain one line: the path to the directory containing mypackage. Call it myprojects. Then mypackage and any other package in myprojects can be imported as if they were in site-packages.
One advantage of .pth files is that you can put identical copies in multiple site-packages directories. For instance, I have multiple projects in F:/python. I have multiple versions of Python installed. So I have put python.pth containing that one line in the site-packages for each.
The best strategy would be to put the sys.path related code in separate file and import it in working code file.
So, I will split above code in two files. One named my_module.py and other named working_module.py
Now my_module will contain below lines
import sys
import os
exe_name = os.path.basename(os.path.realpath(__file__))
bin_dir = os.path.dirname(os.path.realpath(__file__))
inst_dir = os.path.dirname(bin_dir)
sys.path.insert(0, inst_dir+'/path/to/packages')
and working_module will contain
import my_module
import mypackage.mymodule
because we are importing my_module before mypackage, it will execute the path related code before and your package will be available in path.
You can use importlib module (python 3.5) or imp for python 2.7 to load mypackage.mymodule programatically. Both have the same purpose:
mechanisms used to implement the import statement
This question might help you more:
How to import a module given the full path?
https://docs.python.org/3/library/importlib.html#examples
https://docs.python.org/2/library/imp.html
I am using py.test to test my python code. The relevant structure of my project is
-myproject
file.py
file2.py
-test/
input.txt
input2.dat
test_part.py
-regress/
file2_old.py
__init__.py
-test/
test_file2_regression.py
test_part.py imports file and file2 and test_file2_regression.py imports file2 and regress.file2_old. If I run pytest in the console, I get the import errors, that the packages don't exist. Running python -m pytest on the other hand works perfectly fine, but only if I run it from the myproject/ directory.
What is the correct way to do this, to get it working from anywhere in my project? I already tried modifying the PYTHONPATH but I honestly have no idea how to properly do it.
Further information:
I don't have any setup files, and my __init__s are just empty files. If manipulating the PYTHONPATH is necessary it needs to be done relative to myproject as I use it on several machines. I am running python 2.7.
I already check out:
Importing correctly with pytest
Good Integration Practices
but it didn't really help me.
The solution which works with the pytest command from any directory in the project is to include before the imports in the test*.py files:
import os
from sys import path
PATH = os.path.abspath(os.path.dirname(__file__))
path.append(os.path.join(PATH, os.pardir, os.pardir))
where the proper number of os.pardir is used to navigate to the project directory, from there the __init__.py files allow to import the modules.
Both argv[0] and inspect.getsourcefile don't provide the necessary information. argv[0] contains the location of the used py.test module and the getsourcefile method simply returns None.
Edit: since Python 3.4 we can use instead of os.path the modern pathlib:
from pathlib import Path
from sys import path
PATH = Path(__file__).resolve()
path.append(PATH.parents[2])
Having the same issue and similar success in searching for "the best way of doing this", I concluded for myself to avoid the situation whenever possible (by running the actual scripts consequently from top level), but to answer your question, my current approach (for instance for unit testing from a parallel folder) is
from sys import argv, path
from os.path import dirname, join
path.append(join(dirname(argv[0]), ".."))
This makes the interpreter also search in the folder above where the script is started. Another approach (instead of using argv) is to use the introspect module to obtain the filename. These work better for me than using __file__, as the latter is not always defined.
Edit 29.10.:
Alternative to argv[0] is to use
from inspect import getsourcefile
from sys import path
from os.path import dirname, join
this_file = getsourcefile(lambda _: None)
path.append(join(dirname(this_file), ".."))
I hope this will work at least for the requested purpose, see also How do I get the path of the current executed file in Python?.
The simplest - if it works in your case - is of course:
from os.path import dirname, join
path.append(join(dirname(__file__), ".."))
I have a folder called Script and inside I have temp.py script. My temp script imports module from sub-folder called lib.
Lib folder has inside empty __init__.py and my parent_computer_test.py script.
in my temp.py script is the following code:
import lib.parent_computer_test as parent_computer_test
parent_computer_test.mainChunk()
parent_computer_test.splitChunks()
I managed to import module from sub-folder without any bigger problems.
This workflow/script works fine, BUT for a specific reason, my lib folder has to be somewhere else on my computer. There is a long story why, but it has to be that way.
Long story short. I want that my temp.py from the /Script folder imports modules from folder lib (or any other name) with parent_computer_test.py, but at the same time this folder is no sub-folder of /Script - so it is somewhere else in the computer. It can be C:/development/... or something.
So my question is how to import a module from a specific folder?
Append the path to lib folder to the SYS PATH Environment Variable. Then it can be imported from anywhere
import os, sys
lib_path = os.path.abspath(os.path.join('..', '..', '..', 'lib'))
sys.path.append(lib_path)
import mymodule
import imp
yourModule = imp.load_source('yourModuleName', '/path/to/yourModule.py')
foo = yourModule.YourFunction("You", "get", "the", "idea.")
I realize this is a special case, but in general, I would avoid stuff like this. Things can get ugly when you use absolute paths, especially if you move things around, and I would only use this for throwaway scripts or on systems that aren't going to change very much.
EDIT: Aswin's answer is a much better longterm solution.
You have to use sys.path.append("path") . But use this just one time. Then try
import "my_module". IT should be fine.
If you want to remove the path appended you can use,
sys.path.remove("path").