Lets assume the following structure of directories for a project
<root>
__init__.py
helloworld.py
<moduleOne>
f.txt
__init__.py
printfile.py
where root and moduleOne are directories
Content of helloworld.py:
#!/usr/bin/python
import helloworld.printfile
printf()
Content of moduleOne/printfile
#!/usr/bin/python
f = open('f.txt')
def printf():
print 'print file'
print f
if __name__ == '__main__':
printf()
My issue:
From moduleOne/ the execution of printfile is ok, but from root/, if i run helloworld.py the following error happens:
import moduleOne.printfile
File "/root/moduleOne/printfile.py", line 5, in <module>
f = open('f.txt')
IOError: [Errno 2] No such file or directory: 'f.txt'
How to solve this in python?
[Edited]
I solved (more or less) this issue with a "workaround", but stil have a problem:
My solution:
In moduleOne/printfile
import sys
fname = 'moduloOne/f.txt'
def printf():
f = open(fname)
print 'print file'
print f
if __name__ == '__main__':
fname = 'f.txt'
printf()
But....
lets say i have a new directory, from the root, called etc, then the new structure is:
<root>
__init__.py
helloworld.py
<moduleOne>
f.txt
__init__.py
printfile.py
<etc>
f2.txt
And now i need to acess etc/f2.txt from moduleOne/printfile. how?
You need more abstraction.
Don't hard-code the file path in printfile.py
Don't access a global in the printf function.
Do accept a file handle as a parameter to the printf function:
def printf(file_handle):
print 'print file'
print file_handle
In a script that does actually need to know the path of f.txt (I guess helloworld.py in your case), put it there, open it, and pass it to printf:
from moduleOne.printfile import printf
my_f_file = open('/path/to/f.txt')
printf(my_f_file)
Better yet, get the file path from the command line
import sys
from moduleOne.printfile import printf
input_file_path = sys.argv[1]
my_f_file = open(input_file_path)
printf(my_f_file)
EDIT: You said on your Google+ cross-post:
full path is problem, the program will run on differents environments.
If you're trying to distribute your program to other users and machines, you should look into making a distribution package (see side note 3 below), and using package_data to include your configuration file, and pkgutil or pkg_resources to access the configuration file. See How do I use data in package_data from source code?
Some side-notes:
Represent directories as the directory name with a trailing slash, à la the conventions of the tree command: / instead of <root>, moduleOne/ instead of <moduleOne>
You're conflating "module" with "package". I suggest you rename moduleOne/ to packageOne/. A directory with an __init__.py file constitutes a package. A file ending in a .py extension is a module. Modules can be part of packages by physically existing inside a directory with an __init__.py file. Packages can be part of other packages by being a physical subdirectory of a parent directory with an __init__.py file.
Unfortunately, the term "package" is overloaded in Python and also can mean a collection of Python code for distribution and installation. See the Python Packaging Guide glossary.
Related
This question already has answers here:
How do you properly determine the current script directory?
(16 answers)
Closed 6 months ago.
I'm building a simple helper script for work that will copy a couple of template files in our code base to the current directory. I don't, however, have the absolute path to the directory where the templates are stored. I do have a relative path from the script but when I call the script it treats that as a path relative to the current working directory. Is there a way to specify that this relative url is from the location of the script instead?
In the file that has the script, you want to do something like this:
import os
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'relative/path/to/file/you/want')
This will give you the absolute path to the file you're looking for. Note that if you're using setuptools, you should probably use its package resources API instead.
UPDATE: I'm responding to a comment here so I can paste a code sample. :-)
Am I correct in thinking that __file__ is not always available (e.g. when you run the file directly rather than importing it)?
I'm assuming you mean the __main__ script when you mention running the file directly. If so, that doesn't appear to be the case on my system (python 2.5.1 on OS X 10.5.7):
#foo.py
import os
print os.getcwd()
print __file__
#in the interactive interpreter
>>> import foo
/Users/jason
foo.py
#and finally, at the shell:
~ % python foo.py
/Users/jason
foo.py
However, I do know that there are some quirks with __file__ on C extensions. For example, I can do this on my Mac:
>>> import collections #note that collections is a C extension in Python 2.5
>>> collections.__file__
'/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-
dynload/collections.so'
However, this raises an exception on my Windows machine.
It's 2018 now, and Python has already evolved to the __future__ long time ago. So how about using the amazing pathlib coming with Python 3.4 to accomplish the task instead of struggling with os, os.path, glob , shutil, etc.
So we have 3 paths here (possibly duplicated):
mod_path: which is the path of the simple helper script
src_path: which contains a couple of template files waiting to be copied.
cwd: current directory, the destination of those template files.
and the problem is: we don't have the full path of src_path, only know its relative path to the mod_path.
Now let's solve this with the amazing pathlib:
# Hope you don't be imprisoned by legacy Python code :)
from pathlib import Path
# `cwd`: current directory is straightforward
cwd = Path.cwd()
# `mod_path`: According to the accepted answer and combine with future power
# if we are in the `helper_script.py`
mod_path = Path(__file__).parent
# OR if we are `import helper_script`
mod_path = Path(helper_script.__file__).parent
# `src_path`: with the future power, it's just so straightforward
relative_path_1 = 'same/parent/with/helper/script/'
relative_path_2 = '../../or/any/level/up/'
src_path_1 = (mod_path / relative_path_1).resolve()
src_path_2 = (mod_path / relative_path_2).resolve()
In the future, it's just that simple.
Moreover, we can select and check and copy/move those template files with pathlib:
if src_path != cwd:
# When we have different types of files in the `src_path`
for template_path in src_path.glob('*.ini'):
fname = template_path.name
target = cwd / fname
if not target.exists():
# This is the COPY action
with target.open(mode='wb') as fd:
fd.write(template_path.read_bytes())
# If we want MOVE action, we could use:
# template_path.replace(target)
you need os.path.realpath (sample below adds the parent directory to your path)
import sys,os
sys.path.append(os.path.realpath('..'))
As mentioned in the accepted answer
import os
dir = os.path.dirname(__file__)
filename = os.path.join(dir, '/relative/path/to/file/you/want')
I just want to add that
the latter string can't begin with the backslash , infact no string
should include a backslash
It should be something like
import os
dir = os.path.dirname(__file__)
filename = os.path.join(dir, 'relative','path','to','file','you','want')
The accepted answer can be misleading in some cases , please refer to this link for details
Consider my code:
import os
def readFile(filename):
filehandle = open(filename)
print filehandle.read()
filehandle.close()
fileDir = os.path.dirname(os.path.realpath('__file__'))
print fileDir
#For accessing the file in the same folder
filename = "same.txt"
readFile(filename)
#For accessing the file in a folder contained in the current folder
filename = os.path.join(fileDir, 'Folder1.1/same.txt')
readFile(filename)
#For accessing the file in the parent folder of the current folder
filename = os.path.join(fileDir, '../same.txt')
readFile(filename)
#For accessing the file inside a sibling folder.
filename = os.path.join(fileDir, '../Folder2/same.txt')
filename = os.path.abspath(os.path.realpath(filename))
print filename
readFile(filename)
See sys.path
As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter.
Use this path as the root folder from which you apply your relative path
>>> import sys
>>> import os.path
>>> sys.path[0]
'C:\\Python25\\Lib\\idlelib'
>>> os.path.relpath(sys.path[0], "path_to_libs") # if you have python 2.6
>>> os.path.join(sys.path[0], "path_to_libs")
'C:\\Python25\\Lib\\idlelib\\path_to_libs'
Instead of using
import os
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'relative/path/to/file/you/want')
as in the accepted answer, it would be more robust to use:
import inspect
import os
dirname = os.path.dirname(os.path.abspath(inspect.stack()[0][1]))
filename = os.path.join(dirname, 'relative/path/to/file/you/want')
because using __file__ will return the file from which the module was loaded, if it was loaded from a file, so if the file with the script is called from elsewhere, the directory returned will not be correct.
These answers give more detail: https://stackoverflow.com/a/31867043/5542253 and https://stackoverflow.com/a/50502/5542253
From what suggest others and from pathlib documentation, a simple (but not ideal) solution is the following (suppose the file we need to refer to is Test/data/users.csv):
# Current file location: Tests/src/long/module/subdir/some_script.py
from pathlib import Path
# back to Tests/
PROJECT_ROOT = Path(__file__).parents[4]
# then down to Test/data/users.csv
CSV_USERS_PATH = PROJECT_ROOT / 'data' / 'users.csv'
with CSV_USERS_PATH.open() as users:
print(users.read())
This works but looks a bit odd because if you move some_script.py around, the path to the root of our project may change (and we would therefore need to change the parents[4] part).
I think I found a better solution that, based on the same idea.
We will use a file paths.py to store where the root of the project is, this file will remain at the same location compared to the root directory.
Tests
├── data
│ └── users.csv
└── src
├── long
│ └── module
│ └── subdir
│ └── some_script.py
├── main.py
└── paths.py
Where paths.py's only responsability is to provide PROJECT_ROOT:
from pathlib import Path
PROJECT_ROOT = Path(__file__).parents[1]
All scripts can now use paths.PROJECT_ROOT to express absolute paths from the root of the project. For example in src/long/module/subdir/some_script.py we could have:
from paths import PROJECT_ROOT
CSV_USERS_PATH = PROJECT_ROOT / 'data' / 'users.csv'
def hello():
with CSV_USERS_PATH.open() as f:
print(f.read())
And everything goes as expected:
~/Tests/src/$ python main.py
/Users/cglacet/Tests/data/users.csv
hello, user
~/Tests/$ python src/main.py
/Users/cglacet/Tests/data/users.csv
hello, user
The main.py script simply is:
from long.module.subdir import some_script
some_script.hello()
summary of the most important commands
>>> import os
>>> os.path.join('/home/user/tmp', 'subfolder')
'/home/user/tmp/subfolder'
>>> os.path.normpath('/home/user/tmp/../test/..')
'/home/user'
>>> os.path.relpath('/home/user/tmp', '/home/user')
'tmp'
>>> os.path.isabs('/home/user/tmp')
True
>>> os.path.isabs('/tmp')
True
>>> os.path.isabs('tmp')
False
>>> os.path.isabs('./../tmp')
False
>>> os.path.realpath('/home/user/tmp/../test/..') # follows symbolic links
'/home/user'
A detailed description is found in the docs.
These are linux paths. Windows should work analogous.
Hi first of all you should understand functions os.path.abspath(path) and os.path.relpath(path)
In short os.path.abspath(path) makes a relative path to absolute path. And if the path provided is itself a absolute path then the function returns the same path.
similarly os.path.relpath(path) makes a absolute path to relative path. And if the path provided is itself a relative path then the function returns the same path.
Below example can let you understand the above concept properly:
suppose i have a file input_file_list.txt which contains list of input files to be processed by my python script.
D:\conc\input1.dic
D:\conc\input2.dic
D:\Copyioconc\input_file_list.txt
If you see above folder structure, input_file_list.txt is present in Copyofconc folder and the files to be processed by the python script are present in conc folder
But the content of the file input_file_list.txt is as shown below:
..\conc\input1.dic
..\conc\input2.dic
And my python script is present in D: drive.
And the relative path provided in the input_file_list.txt file are relative to the path of input_file_list.txt file.
So when python script shall executed the current working directory (use os.getcwd() to get the path)
As my relative path is relative to input_file_list.txt, that is "D:\Copyofconc", i have to change the current working directory to "D:\Copyofconc".
So i have to use os.chdir('D:\Copyofconc'), so the current working directory shall be "D:\Copyofconc".
Now to get the files input1.dic and input2.dic, i will read the lines "..\conc\input1.dic" then shall use the command
input1_path= os.path.abspath('..\conc\input1.dic') (to change relative path to absolute path. Here as current working directory is "D:\Copyofconc", the file ".\conc\input1.dic" shall be accessed relative to "D:\Copyofconc")
so input1_path shall be "D:\conc\input1.dic"
This code will return the absolute path to the main script.
import os
def whereAmI():
return os.path.dirname(os.path.realpath(__import__("__main__").__file__))
This will work even in a module.
An alternative which works for me:
this_dir = os.path.dirname(__file__)
filename = os.path.realpath("{0}/relative/file.path".format(this_dir))
Example
Here's an example, tested in Python '3.9.5`:
your current directory: 'c:\project1\code\'
and you want to access the following folder: 'c:\project1\dataset\train\'.
Then you can access the folder using the following address: '../dataset/train/'
References
If you want some more information about path in Python, read this:
PEP - 355
PEP - 519
What worked for me is using sys.path.insert. Then I specified the directory I needed to go. For example I just needed to go up one directory.
import sys
sys.path.insert(0, '../')
I think to work with all systems use "ntpath" instead of "os.path". Today, it works well with Windows, Linux and Mac OSX.
import ntpath
import os
dirname = ntpath.dirname(__file__)
filename = os.path.join(dirname, 'relative/path/to/file/you/want')
A simple solution would be
import os
os.chdir(os.path.dirname(__file__))
From C:\Users\xyz\myFolder to C:\Users\xyz\testdata :
import os
working_dir = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
# C:\Users\xyz\myFolder
print(working_dir)
updated_working_dir = os.path.join(os.path.realpath(working_dir + '/../'), 'testdata')
# C:\Users\xyz\testdata
print(updated_working_dir)
Output
C:\Users\xyz\myFolder
C:\Users\xyz\testdata
Here is my sumup:
First, define the tool function named relpath, which convert a relative path to current file into a relative path to cwd
import os
relpath = lambda p: os.path.normpath(os.path.join(os.path.dirname(__file__), p))
Then we use it to wrap paths which is relative to current file
path1 = relpath('../src/main.py')
And you can also call sys.path.append() to import file relative to current file position
sys.path.append(relpath('..')) # so that you can import from upper dir
The full example code : https://gist.github.com/luochen1990/9b1ffa30f5c4a721dab5991e040e3eb1
Say the current archive named "Helper" and the upper directory named "Workshop", and the template files are in \Workshop\Templates, then the relative path in Python is "..\Templates".
This a simple way to add a relative path to the system path set . For example, for frequent case when the target directory is one level above (thus, '/../') the working directory:
import os
import sys
workingDir = os.getcwd()
targetDir = os.path.join(os.path.relpath(workingDir + '/../'),'target_directory')
sys.path.insert(0,targetDir)
This solution was tested for:
Python 3.9.6 | packaged by conda-forge | (default, Jul 11 2021,
03:37:25) [MSC v.1916 64 bit (AMD64)]
I'm not sure if this applies to some of the older versions, but I believe Python 3.3 has native relative path support.
For example the following code should create a text file in the same folder as the python script:
open("text_file_name.txt", "w+t")
(note that there shouldn't be a forward or backslash at the beginning if it's a relative path)
The project has the same structure as in the picture: I'm trying to import from "mod.py " in "index.py "
from .. import mod
However, it gives the error: "ImportError: attempted relative import with no known parent package" If you use this option:
from pack1 import mod
Then error: "ModuleNotFoundError error: there is no module named 'pack1'"
enter image description here
PROJECT/
pack1/
__init__.py
mod.py
pack2/
__init__.py
index.py
What is the problem?
This is a recurring question on StackOverflow. And much of the confusion (in my opinion) comes from how Python interprets the files and folders it sees is based on where Python is run from. First, some terminology:
module: a file containing Python code.
package: a folder containing files with Python code and other folders.
When you start Python in a directory (folder), it doesn't "know" what the namespace of that directory should be. I.e., if you are working in Z:\path\to_my\project\ when you start Python:
it does NOT consider project to be a package.
any .py files you want to import from will be in their own namespace as modules.
any folders you want to import from will also be in their own namespace as packages.
What about __init__.py? Since version 3.3, Python has implicit namespace packages, which allows importing without needing to create an empty __init__.py file.
Consider #2: if you have two files: first.py and second.py:
path/
to_my/
project/
>>Python is running here<<
first.py
second.py
with these contents:
# first.py
first_var = 'hello'
# second.py
from .first import first_var
second_var = first_var + ' world'
if you try to import like this:
>>> import second
Python basically does the following:
"ok, I see second.py"
"Reading that in as a module, chief!"
"Ok, it wants to import .first
"The . means get the package (folder) that contains first.py"
"Wait, I don't have a parent package for first.py!"
"Better raise an error."
The same rules apply for #3 as well. If we add a few packages to the project like this:
path/
to_my/
project/
>>Python is running here<<
first.py
second.py
pack1/
mod.py
other_mod.py
pack2/
index.py
with the following contents:
# pack1/mod.py
mod_var = 1234
# pack1/other_mod.py
from .mod import mod_var
other_var = mod_var * 10
# pack2/index.py
from ..pack1 import mod
and when you try to import like this:
>>> from pack2 import index.py
The import in pack2/index.py is going to fail for the same reason second.py, Python will work its way up the import chain of dots like this:
"Reading in in index.py as a module."
"Looks like it wants to import mod from ..pack1.
"Ok, . is the pack2 parent package namespace of index.py, found that."
"So, .. is the parent package of pack2."
"But, I don't have a parent package for pack2!"
"Better raise an error."
How do we make it work? Two thing.
First, move where Python is running up one level so that all of the .py files and subfolders are considered to be part of the same package namespace, which allows the file to reference each other using relative references.
path/
to_my/
>>Python is running here now<<
project/
first.py
second.py
pack1/
mod.py
other_mod.py
pack2/
index.py
So now Python sees project as a package namespace, and all of the files within can use relative references up to that level.
This changes how you import when you are in the Python interpreter:
>>> from project.pack2 import index.py
Second, you make explicit references instead of relative references. That can make the import statements really long, but if you have several top-level modules that need to pull from one another, this is how you can do it. This is useful when you are defining your functions in one file and writing your script in another.
# first.py
first_var = 'hello'
# second.py
from first import first_var # we dropped the dot
second_var = first_var + ' world'
I hope this helps clear up some of the confusion about relative imports.
I am working on a python3 app with a fairly simple file structure, but I'm having issues reading a text file in a script, both of which are lower in the file structure than the script calling them. To be absolutely clear, the file structure is as follows:
app/
|- cli-script
|- app_core/
|- dictionary.txt
|- lib.py
cli-script calls lib.py, and lib.py requires dictionary.txt to do what I need it to, so it gets opened and read in lib.py.
The very basics of cli-script looks like this:
from app_core import lib
def cli_func():
x = lib.Lib_Class()
x.lib_func()
The problem area of lib is here:
class Lib_Class:
def __init__(self):
dictionary = open('dictionary.txt')
The problem I'm getting is that while I have this file structure, the lib file can't find the dictionary file, returning a FileNotFoundError. I would prefer to only use relative paths for portability reasons, but otherwise I just need to make the solution OS agnostic. Symlinks are a last resort option I've figured out, but I want to avoid it at all costs. What are my options?
When you run a Python script, calls involving paths are executed relative to where you run them from, not where the files are actually from.
The __file__ variable stores the path of the current file (no matter where it is), so relative files will be siblings to that.
In your structure, __file__ refers to the path app/app_core/lib.py, so to create app/app_core/dictionary.txt, you need to co up and then down again.
app/app_core/lib.py
import os.path
class Lib_Class:
def __init__(self):
path = os.path.join(os.path.dirname(__file__), 'dictionary.txt')
dictionary = open(path)
or using pathlib
path = pathlib.Path(__file__).parent / 'dictionary.txt'
Because you are expecting the dictionary.txt to be present in the same path as your lib.py file you can do the following.
Instead of dictionary = open('dictionary.txt') use
dictionary = open(Path(__file__).parent / 'dictionary.txt')
Solved see my answer below for anyone who might find this helpful.
I have two scripts a.py and b.py.
In my current directory "C:\Users\MyName\Desktop\MAIN", I run > python a.py.
The first script, a.py runs in my current directory, does something to a bunch of files and creates a new directory (testA) with the edited versions of those files which are simultaneously moved into that new directory. Then I need to run b.py for the files in testA.
As a beginner, I would just copy and paste my b.py script into testA and execute the command again "> python b.py", which runs some commands on those new files and creates another folder (testB) with those edited files.
I am trying to eliminate the hassle of waiting for a.py to finish, move into that new directory, paste b.py, and then run b.py. I am trying to write a bash script that executes these scripts while maintaining my hierarchy of directories.
#!/usr/bin/env bash
python a.py && python b.py
Script a.py runs smoothly, but b.py does not execute at all. There are no error messages coming up about b.py failing, I just think it cannot execute because once a.py is done, b.py does not exist in that NEW directory.
Is there a small script I can add within b.py that moves it into the new directory? I actually tried changing b.py directory paths as well but it did not work.
For example in b.py:
mydir = os.getcwd() # would be the same path as a.py
mydir_new = os.chdir(mydir+"\\testA")
I changed mydirs to mydir_new in all instances within b.py, but that also made no difference...I also don't know how to move a script into a new directory within bash.
As a little flowchart of the folders:
MAIN # main folder with unedited files and both a.py and b.py scripts
|
| (execute a.py)
|
--------testA # first folder created with first edits of files
|
| (execute b.py)
|
--------------testB # final folder created with final edits of files
TLDR: How do I execute a.py and b.py from the main test folder (bash script style?), if b.py relies on files created and stored in testA. Normally I copy and paste b.py into testA, then run b.py - but now I have 200+ files so copying and pasting is a waste of time.
The easiest answer is probably to change your working directory, then call the second .py file from where it is:
python a.py && cd testA && python ../b.py
Of course you might find it even easier to write a script that does it all for you, like so:
Save this as runTests.sh in the same directory as a.py is:
#!/bin/sh
python a.py
cd testA
python ../b.py
Make it executable:
chmod +x ./runTests.sh
Then you can simply enter your directory and run it:
./runTests.sh
I managed to get b.py executing and producing the testB folder where I need it to, while remaining in the MAIN folder. For anyone who might wonder, at the beginning of my b.py script I would simply use mydir = os.getcwd() which normally is wherever b.py is.
To keep b.py in MAIN while making it work on files in other directories, I wrote this:
mydir = os.getcwd() # would be the MAIN folder
mydir_tmp = mydir + "//testA" # add the testA folder name
mydir_new = os.chdir(mydir_tmp) # change the current working directory
mydir = os.getcwd() # set the main directory again, now it calls testA
Running the bash script now works!
In your batch file, you can set the %PYTHONPATH% variable to the folder with the Python module. This way, you don't have to change directories or use pushd to for network drives. I believe you can also do something like
set "PYTHONPATH=%PYTHONPATH%;c:\the path\to\my folder\which contains my module"
This will append the paths I believe (This will only work if you already have set %PYTHONPATH% in your environment variables).
If you haven't, you can also just do
set "PYTHONPATH=c:\the path\to\my folder\which contains my module"
Then, in the same batch file, you can do something like
python -m mymodule ...
despite there are already answers i still wrote a script out of fun and it still could be of help in some respects.
I wrote it for python3, so it is necessary to tweak some minor things to execute it on v2.x (e.g. the prints).
Anyways... the code creates a new folder relative to the location of a.py, creates and fills script b.py with code, executes b and displays b's results and errors.
The resulting path-structure is:
testFolder
|-testA
| |-a.py
|-testB
| |-b.py
The code is:
import os, sys, subprocess
def getRelativePathOfNewFolder(folderName):
return "../" + folderName + "/"
def getAbsolutePathOfNewFolder(folderName):
# create new folder with absolute path:
# get path of current script:
tmpVar = sys.argv[0]
# separate path from last slash and file name:
tmpVar = tmpVar[:sys.argv[0].rfind("/")]
# again to go one folder up in the path, but this time let the slash be:
tmpVar = tmpVar[:tmpVar.rfind("/")+1]
# append name of the folder to be created:
tmpVar += folderName + "/"
# for the crazy ones out there, you could also write this like this:
# tmpVar = sys.argv[0][:sys.argv[0].rfind("/", 0,
sys.argv[0].rfind("/")-1)+1] + folderName + "/"
return tmpVar
if __name__ == "__main__":
# do stuff here:
# ...
# create new folder:
bDir = getAbsolutePathOfNewFolder("testB")
os.makedirs(bDir, exist_ok=True) # makedirs can create new nested dirs at once. e.g: "./new1/new2/andSoOn"
# fill new folder with stuff here:
# ...
# create new python file in location bDir with code in it:
bFilePath = bDir + "b.py"
with open(bFilePath, "a") as toFill:
toFill.write("if __name__ == '__main__':")
toFill.write("\n")
toFill.write("\tprint('b.py was executed correctly!')")
toFill.write("\n")
toFill.write("\t#do other stuff")
# execute newly created python file
args = (
"python",
bFilePath
)
popen = subprocess.Popen(args, stdout=subprocess.PIPE)
# use next line if the a.py has to wait until the subprocess execution is finished (in this case b.py)
popen.wait()
# you can get b.py´s results with this:
resultOfSubProcess, errorsOfSubProcess = popen.communicate()
print(str(resultOfSubProcess)) # outputs: b'b.py was executed correctly!\r\n'
print(str(errorsOfSubProcess)) # outputs: None
# do other stuff
instead of creating a new code file and filling it with code you of course can simply copy an existing one as shown here:
How do I copy a file in python?
Your b.py script could take the name of the directory as a parameter. Access the first parameter passed to b.py with:
import sys
dirname = sys.argv[1]
Then iterate over the files in the named directory with:
import os
for filename in os.listdir(dirname):
process(filename)
Also see glob.glob and os.walk for more options processing files.
I have a Python project with the following directory structure:
project/
project/src/
project/src/somecode.py
project/src/mypackage/mymodule.py
project/src/resources/
project/src/resources/datafile1.txt
In mymodule.py, I have a class (lets call it "MyClass") which needs to load datafile1.txt. This sort of works when I do:
open ("../resources/datafile1.txt")
Assuming the code that creates the MyClass instance created is run from somecode.py.
The gotcha however is that I have unit tests for mymodule.py which are defined in that file, and if I leave the relative pathname as described above, the unittest code blows up as now the code is being run from project/src/mypackage instead of project/src and the relative filepath doesn't resolve correctly.
Any suggestions for a best practice type approach to resolve this problem? If I move my testcases into project/src that clutters the main source folder with testcases.
I usually use this to get a relative path from my module. Never tried in a unittest tho.
import os
print(os.path.join(os.path.dirname(__file__),
'..',
'resources'
'datafile1.txt'))
Note: The .. tricks works pretty well, but if you change your directory structure you would need to update that part.
On top of the above answers, I'd like to add some Python 3 tricks to make your tests cleaner.
With the help of the pathlib library, you can explicit your ressources import in your tests. It even handles the separators difference between Unix (/) and Windows ().
Let's say we have a folder structure like this :
`-- tests
|-- test_1.py <-- You are here !
|-- test_2.py
`-- images
|-- fernando1.jpg <-- You want to import this image !
`-- fernando2.jpg
You are in the test_1.py file, and you want to import fernando1.jpg. With the help to the pathlib library, you can read your test resource with an object oriented logic as follows :
from pathlib import Path
current_path = Path(os.path.dirname(os.path.realpath(__file__)))
image_path = current_path / "images" / "fernando1.jpg"
with image_path.open(mode='rb') as image :
# do what you want with your image object
But there's actually convenience methods to make your code more explicit than mode='rb', as :
image_path.read_bytes() # Which reads bytes of an object
text_file_path.read_text() # Which returns you text file content as a string
And there you go !
in each directory that contains Python scripts, put a Python module that knows the path to the root of the hierarchy. It can define a single global variable with the relative path. Import this module in each script. Python searches the current directory first so it will always use the version of the module in the current directory, which will have the relative path to the root of the current directory. Then use this to find your other files. For example:
# rootpath.py
rootpath = "../../../"
# in your scripts
from rootpath import rootpath
datapath = os.path.join(rootpath, "src/resources/datafile1.txt")
If you don't want to put additional modules in each directory, you could use this approach:
Put a sentinel file in the top level of the directory structure, e.g. thisisthetop.txt. Have your Python script move up the directory hierarchy until it finds this file. Write all your pathnames relative to that directory.
Possibly some file you already have in the project directory can be used for this purpose (e.g. keep moving up until you find a src directory), or you can name the project directory in such a way to make it apparent.
You can access files in a package using importlib.resources (mind Python version compatibility of the individual functions, there are backports available as importlib_resources), as described here. Thus, if you put your resources folder into your mypackage, like
project/src/mypackage/__init__.py
project/src/mypackage/mymodule.py
project/src/mypackage/resources/
project/src/mypackage/resources/datafile1.txt
you can access your resource file in code without having to rely on inferring file locations of your scripts:
import importlib.resources
file_path = importlib.resources.files('mypackage').joinpath('resources/datafile1.txt')
with open(file_path) as f:
do_something_with(f)
Note, if you distribute your package, don't forget to include the resources/ folder when creating the package.
The filepath will be relative to the script that you initially invoked. I would suggest that you pass the relative path in as an argument to MyClass. This way, you can have different paths depending on which script is invoking MyClass.