Apparently python takes every related importation in relation to the first called file.
I have the following file structure
src
|--myunittests.py
|--subfolder1
|--__init__.py
|--printFileContent.py
|--subfolder2
|--__init__.py
|--file
myunittests.py will test the behavior of functions inside printFileContent:
from subfolder1.printFileContent import printFileContent
printFileContent()
printFileContent prints the content of a file contained inside the subfolder:
def printFileContent():
with open("./subfolder2/file") as file:
for line in file:
print(line)
if __name__ == "__main__":
printFileContent()
file just contains some text.
Question:
Doing python3 printFileContent.py inside the subfolder1 will correctly output the file content.
But doing python3 myunittests.py raises the error, that the file could not be found.
Is there a way to solve this problem? (Is there a way to tell python, that files refered relative programmatically should be relative to the file they are used in?
constraints
changing content inside printFileContent.py is not an option (generated file)
Such calls of printFileContent are at arbitrary places throughout the code (A unittest file calls a dialog, that calls printFileContent vv inside subdirectory2)
When does this behavior occur?
When file is an icon that is used inside printFileContent.py, while printFileContent.py is called from myunittests.py
Sidequestion:
Is there a proper title/bulletpoint word for explaining / finding out about this behavior and problems with it?
If you cannot modify printFileContent.py, you can save the current directory, go to the directory of subfolder1 and then come back to the original directory:
import subfolder1
import os
# Save current directory (absolute path)
cdir = os.path.abspath(os.path.curdir)
# Change directory, and call printFileContent
os.chdir(os.path.dirname(subfolder1.__file__))
subfolder1.printFileContent()
# Go back to the original directory
os.chdir(cdir)
If you have to use this a lot of time, you can make this behavior a class usable with a with statement so that it's easier to use and more robust (you won't forget to chdir back):
import os
class TmpDirChanger:
def __init__(self, tmpPath):
self.currentDir = os.path.abspath(os.path.curdir)
os.chdir(tmpPath)
def __enter__(self): pass
def __exit__(self, exc_type, exc_val, exc_tb):
#change back to original dir
os.chdir(self.currentDir)
with TmpDirChanger('path/to/some/dir'):
do_something()
If you can modify printFileContent.py, it is less tricky:
import os
def printFileContent():
# This will give you path to subfolder1
sfolder1 = os.path.dirname(__file__)
# This will give you path to "file" in subfolder2
name = os.path.join(sfolder1, 'subfolder2', 'file')
with open(name) as file:
for line in file:
print(line)
Related
My file structure is as follows:
kkg/
builder/test.py
builder/data
api/api.py
__init__py
'kkg' is my package name, and in init.py some function are defined, and implementations of these function are written api.py.
In test.py, I have:
import kkg
kkg.load('builder/data/')
Inside the 'load' of the api.py, I have code:
abspath = os.path.abspath(os.path.dirname(__file__))
...
for file in files:
file_path = os.path.join(abspath, data_path)
The data_path is the parameter 'builder/data/' passed from test.py. The path.join reports an error:
Caused by: java.io.FileNotFoundException: /Users/comin/kkg/kkg/api/data/people.csv
The correct data path, if parsed properly, should be:
/Users/comin/kkg/kkg/data/people.csv
I run the 'test.py' inside the builder/ directory. I think the reason there is an unnecessary 'api' in the path generated is because the code piece where error occurs in the api.py.
Perhaps I shouldn't use the join(abspath, data_path) to get the absolute directory. How to get the path correctly parsed?
EDIT:
I changed the path parameter:
kkg.load('../builder/data/')
but then this code failed:
if not os.path.isdir(data_path):
raise ValueError('data_path must be a directory, not file name!')
Why does it raise an error when I added '..' to the path? It is not considered as a directory due to the '..'?
you want the parent directory I think
abspath = os.path.abspath(os.path.join(os.path.dirname(__file__),".."))
you seem to be struggling with this more than you should...it might be easier to just do
path_to_data_file_folder = os.path.abspath(sys.argv[1])
then call it with python main.py /path/to/data/folder
It is much easier to use pathlib over os library:
# api/api.py
from pathlib import Path
KKG_FOLDER = Path(__file__).parent.parent
DATA_FOLDER = KKG_FOLDER / 'builder/data'
print(DATA_FOLDER)
This is more verbose and easier to understand.
I have a script (dump_ora_shelve.py) which retrieves the data from shelve-storage by specifying the key, e.g.:
def get_shelve_users(field):
import shelve
db = shelve.open('oracle-shelve')
for key in db:
if key == field:
return db[key]
db.close()
The data is retrieved just fine:
print(get_shelve_users('db_users'))
> {'SYS': 'sysdba'}
print(get_shelve_users('oratab'))
> ['orcl:/u01/app']
There is another script which should do the same thing (retrieve the data with key specified) that has dump_ora_shelve imported, but the value returns is Null:
from before_OOP.dump_ora_shelve import get_shelve_users
print(get_shelve_users('db_users'))
> Null
print(get_shelve_users('oratab'))
> Null
The file being imported is located one level above from the file it is importing to.
Please note if I copy both files to the same location import and then function works just fine.
You could provide the full pathname to shelve.open. Remember that inside a module, __file__ is the path of where the source file resides. So you can use that to construct the full pathname.
Typically you will have something like this:
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
Use os.path.join to concatenate the directory and the filename. Note the use of os.path.dirname and os.path.abspath.
So you can say:
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
shelve_db = os.path.join(BASE_DIR, 'oracle-shelve')
db = shelve.open('oracle-shelve')
This assumes that the orache-shelve file is in the same folder as the module in which the get_shelve_users function is (dump_ora_shelve.py).
Don't forget the __file__. That is what makes the whole thing tick, i.e. makes your program insulated from whatever its current directory is.
When running the second script your working directory will be the directory where that script is located. That working directory is kept even when you import and use a file from a different package/directory.
So if your dump_ora_shelve.py script and shelve is located in a different directory/package it will not open the correct file.
If you provide the full path to 'oracle-shelve' in dump_ora_shelve.py it should work.
Update:
In your 'dump_ora_shelve.py' file:
ABS_DIR = os.path.dirname(os.path.abspath(__file__))
This gives you the absoulte path of the directory of ''dump_ora_shelve.py'. Join with name of your DB:
shelve_db = os.path.join(ABS_DIR, 'oracle-shelve')
And finally:
db = shelve.open(shelve_db)
This assumes that your 'oracle-shelve' is in the same directory as 'dump_ora_shelve.py'
I have created a script that writes the binary content of my files to other files in the same directory that end with a certain extension. When doing this however I have come onto an issue, when I open the file and save the new file in a folder, it creates a new file. I'll show an example. There is a folder, FOLDER1 with test.py inside, inside the folder is a folder called OPTION1 with the files, bye.sh and hello. When I try to execute test.py, my program sees the files bye.sh and hello(hello's binary data is being copied to bye.sh) in OPTION1 and reads it, but creates a new file in the FOLDER1 alongside test.py creating a new bye.sh in FOLDER1, how do I make it overwrite the bye.sh in OPTION1? Just mentioning, I am running Mac OS X.
Code here:
import os
class FileHandler:
#import all needed modules
def __init__(self):
import os
#Replaces second file with the binary data in file one
"""
Trys To Check If a File Exists
If It exists, it will open the file that will be copied in binary form and the output file in reading/writing binary form
If the input file doesnt exist it will tell the user, then exit the program
Once the files are open, it reads the input file binary and copies it to the output file
Then it checks to see if the two binaries match which they should if it all worked correctly, the program will then return true
The Files are then closed
If it failed to try it will tell the user the file was skipped
"""
def replaceBinary(self,file_in="",file_out=""):
try:
if(os.path.isfile(file_in)):
in_obj = open(file_in,"rb") #Opens The File That Will Be Copied
out_obj = open(file_out,"wb+") #Opens The File That Will Be Written To
object_data = in_obj.read()#Get Contents of In File
out_obj.write(object_data)# Write Contents of In File To Out File
print("SPECIAL FILE:"+file_out)
print(os.getcwd())
if out_obj.read() == in_obj.read():
print("Its the same...")
return True
print("Done Copying...")
else:
print("Usage: Enter an input file and output file...")
print(file_in+" Doesn't Exist...")
raise SystemExit
return False
in_obj.close()
out_obj.close()
except:
print("File, "+file_out+" was skipped.")
"""
Procedurally continues down an entire location and all its locations in a tree like manner
Then For each file found it checks if it matches of the extensions
It checks a file for the extensions and if it has one of them, it calls the replace binary function
"""
def WalkRoot(self,file1="",root_folder=""):
for root,dirs,files in os.walk(root_folder):
for f in files:
print(os.path.abspath(f))
array = [".exe",".cmd",".sh",".bat",".app",".lnk",".command",".out",".msi",".inf",".com",".bin"]
for extension in array:
if(f.endswith(extension)):
print("Match|\n"+os.path.abspath(f)+"\n")
self.replaceBinary(file1,os.path.abspath(f))
break #If The right extension is met skip rest of extensions and move to next file
print("Done...")
def main():
thing = FileHandler()
#path = os.path.join(os.getcwd())
path = input(str("Enter path:"))
thing.WalkRoot("Execs/hello","/Users/me/Documents/Test")
main()
Thanks!
The list of files returned by os.walk() does not include directory information. However, the dirpath (which you call root) is updated as you go through the list. Quoting the manual,
filenames is a list of the names of the non-directory files in dirpath. Note
that the names in the lists contain no path components. To get a full path
(which begins with top) to a file or directory in dirpath, do
os.path.join(dirpath, name).[1]
So, to get the correct, full path to the files within your top-level directory, try replacing
self.replaceBinary(file1,os.path.abspath(f))
with
self.replaceBinary(file1, os.path.join(root, f))
[1] https://docs.python.org/3.5/library/os.html#os.walk
Similar to the system path, I want to offer some convenience in my code allowing a user to specify a file name that could be in one of a handful of paths.
Say I had two or more config paths
['~/.foo-config/', '/usr/local/myapp/foo-config/']
And my user wants to open bar, (AKA bar.baz)
Is there a convenient build in way to let open('bar') or open('bar.baz') automatically search these paths for that file in LTR order of precedence? Eg, will temporary adjusting my sys.path to only be these directories do this for me?
Else, how would you suggest implementing a PATH-like searching open-wrapper?
As other people already mentioned: sys.path only affects the module search path, i.e. it's relevant for importing Python modules, but not at all for open().
I would suggest separating the logic for searching the paths in order of precedence and opening the file, because that way it's easier to test and read.
I would do something like this:
import os
PATHS = ['~/.foo-config/', '/usr/local/myapp/foo-config/']
def find_first(filename, paths):
for directory in paths:
full_path = os.path.join(directory, filename)
if os.path.isfile(full_path):
return full_path
def main():
filename = 'file.txt'
path = find_first(filename, PATHS)
if path:
with open(path) as f:
print f
else:
print "File {} not found in any of the directories".format(filename)
if __name__ == '__main__':
main()
open doesn't get into that kind of logic. If you want, write a wrapper function that uses os.path.join to join each member of sys.path to the parameter filename, and tries to open them in order, handling the error that occurs when no such file is found.
I'll add that, as another user stated, this is kind of a misuse of sys.path, but this function would work for any list of paths. Indeed, maybe the nicest option is to use the environment variables suggested by another user to specify a colon-delimited list of config directories, which you then parse and use within your search function.
environmental variables
say your app is named foo ... in the readme tell the user to use the FOO_PATH environmental variable to specify the extra paths
then inside your app do something like
for path in os.environ.get("FOO_PATH",".").split(";"):
lookfor(os.path.join(path,"somefile.txt"))
you could wrap it into a generic function
def open_foo(fname):
for path in os.environ.get("FOO_PATH",".").split(";"):
path_to_test = os.path.join(path,"somefile.txt")
if os.path.exists(path_to_test):
return open(path_to_test)
raise Exception("No File Found On FOOPATH")
then you could use it just like normal open
with open_foo("my_config.txt") as f:
print f.read()
Extract from Python Standard Library documentation for open built-in function:
open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None)
...file is either a string or bytes object giving the pathname (absolute or relative to the current working directory) of the file to be opened ...
Explicitely, open does not bring anything to automagically find a file : if path is not absolute, it is only searched in current directory.
So you will have to use a custom function or a custom class for that. For example:
class path_opener(object):
def __init__(self, path = [.]):
self.path = path
def set(self, path):
self.path = path
def append(self, path):
self.path.append(path)
def extent(self, path):
self.path.extend(path)
def find(self, file):
for folder in self.path:
path = os.path.join(folder, file)
if os.path.isfile(path):
return path
raise FileNotFoundError()
def open(self, file, *args, **kwargs):
return open(self.find(file), *args, **kwargs)
That means that a file opener will keep its own path, will be initialized by default with current path, will have methods to set, append to or extend its path, and will normaly raise a FileNotFoundError is a file is not found in any of the directories listed in its path.
Usage :
o = path_opener(['~/.foo-config/', '/usr/local/myapp/foo-config/'])
with o.open('foo') as fd:
...
I have a small text (XML) file that I want a Python function to load. The location of the text file is always in a fixed relative position to the Python function code.
For example, on my local computer, the files text.xml and mycode.py could reside in:
/a/b/text.xml
/a/c/mycode.py
Later at run time, the files could reside in:
/mnt/x/b/text.xml
/mnt/x/c/mycode.py
How do I ensure I can load in the file? Do I need the absolute path? I see that I can use os.path.isfile, but that presumes I have a path.
you can do a call as follows:
import os
BASE_DIR = os.path.dirname(os.path.realpath(__file__))
This will get you the directory of the python file you're calling from mycode.py
then accessing the xml files is as simple as:
xml_file = "{}/../text.xml".format(BASE_DIR)
fin = open(xml_file, 'r+')
If the parent directory of the two directories are always the same this should work:
import os
path_to_script = os.path.realpath(__file__)
parent_directory = os.path.dirname(path_to_script)
for root, dirs, files in os.walk(parent_directory):
for file in files:
if file == 'text.xml':
path_to_xml = os.path.join(root, file)
You can use the special variable __file__ which gives you the current file name (see http://docs.python.org/2/reference/datamodel.html).
So in your first example, you can reference text.xml this way in mycode.py:
xml_path = os.path.join(__file__, '..', '..', 'text.xml')