I am trying to read from a txt file I am including with my pyqt application.
The problem I am having is that the os.path.dirname(__file__) is returning the directory where the module that houses finding the data file exists not the directory where main() lives. In this case a subdirectory of the application directory not the application directory.
Is there a way to always return the directory where main() is being called from ?
That or can I get the directory to this text file in main() and pass it to Mainform() somehow ?
The reason for all this is because I want to put the .txt in the same directory as main.py so when I deploy my frozen application the text file can be found. (on OSX and WIN)
Thanks
A setup like this may work for you:
main.py
import other
def main():
print other.get_data_files_dir()
main()
other.py
import os
import sys
def get_data_files_dir():
namespace = sys._getframe(1).f_globals # caller's globals
return os.path.dirname(namespace['__file__'])
Related
I have this code in my_program.py:
from my_module import do_stuff_with_file
do_stuff_with_file("hi.txt")
And this is my_module.py:
def do_stuff_with_file(fileName):
print(fileName)
# do stuff with the file
A file not found error arises when my_module.py is not in the same directory as my_program.py. This problem was solved using this code (my_module.py).
def do_stuff_with_file(fileName):
fileName = os.path.join(os.path.dirname(sys.modules['__main__'].__file__), fileName)
print(fileName)
# do stuff with file
The issue arises when my_program.py is imported by a file in a different directory.
How can I fix this?
Given the following file hierarchy :
stack_overflow/
├─ q67993523/
├─ my_module/
│ ├─ __init__.py
├─ 67993523.py
├─ hi.txt
With the following file content :
# 67993523.py
from my_module import do_stuff_with_file
do_stuff_with_file("hi.txt")
# my_module/__init__.py
def do_stuff_with_file(filename):
print(f"{filename!s} content is :")
with open(filename, "rt") as file:
print(file.read())
and the file hi.txt :
Hello !
When I run C:\path\to\python.exe C:/stack_overflow/q67993523/67993523.py (with the path to q67993523 included in my PYTHONPATH), with my current directory being q67993523/, I get :
hi.txt content is :
Hello !
But if I change my current dir to q67993523/my_module/ and execute the exact same command, I get :
hi.txt content is :
Traceback:
[...]
FileNotFoundError: [Errno 2] No such file or directory: 'hi.txt'
because relative to the current working directory q67993523/my_module/ there is no file hi.txt, the file would be ../hi.txt.
I think what you are doing is an instance of the XY problem.
What you are trying to achieve is to find a file given its name but not the location. It is very difficult to do, prone to error, and would include lots of hacks to work.
I don't think it is actually what you want to do. What you want to do, I presume, is not to search for files but just to use them. So you should not lose the precious information of their location.
For example, in your main script (mine is 67993523.py), you know that the file is right there, in the same directory. But if you just send hi.txt, because the function don't know the file location of the code that called her, it does not know where to search for the file.
Instead, give the complete file location, namely the absolute path.
If I change my main script to :
# 67993523.py
from pathlib import Path
from my_module import do_stuff_with_file
the_directory_of_this_pyfile = Path(__file__).parent
do_stuff_with_file((the_directory_of_this_pyfile / "hi.txt").absolute())
And run it with my current directory being q67993523/, I get :
C:\stack_overflow\q67993523\hi.txt content is :
Hello !
And when I change my current directory to q67993523/my_module/, I get the same thing :
C:\stack_overflow\q67993523\hi.txt content is :
Hello !
The difference is that in your script, the hi.txt filename assumes that your current working directory is q67993523/. If you have a different current working directory (because Pytest, because running the script for anywhere you want, ... see the comment from #tdelaney) then there is no ./hi.txt file, so it will fail.
I encourage you to learn on the topic of current working directory
and how to express the current Python file directory.
import tempfile
tmp = tempfile.NamedTemporaryFile(delete=True)
try:
# do stuff with temp
tmp.write(b'def fun():\n\tprint("hello world!")\n')
if __name__ == '__main__':
func = __import__(tmp.name)
func.fun()
finally:
tmp.close() # deletes the file
So I want to create a temporary file, add some source code to it and then import the module and call the function, but I always run into this error:
ModuleNotFoundError: No module named '/var/folders/3w/yyp887lx4018h9s5sr0bwhkw0000gn/T/tmp84bk0bic'
It doesn't seem to find the module of the temporary file. How do I solve this?
There are a few problems with your code:
Your filename does not end with .py, but Python modules are expected to. You can fix this by setting suffix='.py' in NamedTemporaryFile().
__import__() is not the right way to load a module from a full path. See here: How to import a module given the full path?
You do not flush after writing and before importing, so even if Python does find the file, it may well be empty. Add tmp.flush() after writing to fix this.
Importing can only be done from certain directories which are part of the PYTHON_PATH. You can extend that. Then you will have to use __import__() with a module name (not a path in the file system). You will have to deal with the suffix for the temp file.
I implemented a simple version using the local directory for the temp module file and a version using a proper tempfile:
#!/usr/bin/env python3
import sys
import os
import tempfile
SCRIPT = '''\
def fun():
print("hello world!")
'''
# simple version using the local directory:
with open('bla.py', 'w') as tmp_module_file:
tmp_module_file.write(SCRIPT)
import bla
bla.fun()
# version using the tempfile module:
tmpfile = tempfile.NamedTemporaryFile(suffix='.py', delete=True)
try:
tmpfile.write(SCRIPT.encode('utf8'))
tmpfile.flush()
tmpmodule_path, tmpmodule_file_name = os.path.split(tmpfile.name)
tmpmodule_name = tmpmodule_file_name[:-3] # strip off the '.py'
sys.path.append(tmpmodule_path)
tmpmodule = __import__(tmpmodule_name)
finally:
tmpfile.close()
tmpmodule.fun()
Apparently python takes every related importation in relation to the first called file.
I have the following file structure
src
|--myunittests.py
|--subfolder1
|--__init__.py
|--printFileContent.py
|--subfolder2
|--__init__.py
|--file
myunittests.py will test the behavior of functions inside printFileContent:
from subfolder1.printFileContent import printFileContent
printFileContent()
printFileContent prints the content of a file contained inside the subfolder:
def printFileContent():
with open("./subfolder2/file") as file:
for line in file:
print(line)
if __name__ == "__main__":
printFileContent()
file just contains some text.
Question:
Doing python3 printFileContent.py inside the subfolder1 will correctly output the file content.
But doing python3 myunittests.py raises the error, that the file could not be found.
Is there a way to solve this problem? (Is there a way to tell python, that files refered relative programmatically should be relative to the file they are used in?
constraints
changing content inside printFileContent.py is not an option (generated file)
Such calls of printFileContent are at arbitrary places throughout the code (A unittest file calls a dialog, that calls printFileContent vv inside subdirectory2)
When does this behavior occur?
When file is an icon that is used inside printFileContent.py, while printFileContent.py is called from myunittests.py
Sidequestion:
Is there a proper title/bulletpoint word for explaining / finding out about this behavior and problems with it?
If you cannot modify printFileContent.py, you can save the current directory, go to the directory of subfolder1 and then come back to the original directory:
import subfolder1
import os
# Save current directory (absolute path)
cdir = os.path.abspath(os.path.curdir)
# Change directory, and call printFileContent
os.chdir(os.path.dirname(subfolder1.__file__))
subfolder1.printFileContent()
# Go back to the original directory
os.chdir(cdir)
If you have to use this a lot of time, you can make this behavior a class usable with a with statement so that it's easier to use and more robust (you won't forget to chdir back):
import os
class TmpDirChanger:
def __init__(self, tmpPath):
self.currentDir = os.path.abspath(os.path.curdir)
os.chdir(tmpPath)
def __enter__(self): pass
def __exit__(self, exc_type, exc_val, exc_tb):
#change back to original dir
os.chdir(self.currentDir)
with TmpDirChanger('path/to/some/dir'):
do_something()
If you can modify printFileContent.py, it is less tricky:
import os
def printFileContent():
# This will give you path to subfolder1
sfolder1 = os.path.dirname(__file__)
# This will give you path to "file" in subfolder2
name = os.path.join(sfolder1, 'subfolder2', 'file')
with open(name) as file:
for line in file:
print(line)
There are multiple threads about getting the current Python's script directory, for example:
import os
dir = os.path.dirname(os.path.abspath(__file__))
The question is, what should I do if I want to add this function into some utility file, while I want the returned value to be the directory of the calling file, in this case, the file executed by the user?
What would happen here?
// utils.py
def get_script_dir():
import os
return os.path.dirname(os.path.abspath(__file__))
// main_app.py
from utils import get_script_dir
print(get_script_dir())
// user's shell
python c:\path\to\somewhere\main_app.py
Will it print the directory of main_app.py or the directory of utils.py? What is an elegant solution for placing the function in the utils file while getting the directory of the file that was actually executed by the user?
Please try following and let me know whether it's exactly what you want:
# utils.py
def get_script_dir():
import sys
import os
return os.path.dirname(sys.modules['__main__'].__file__)
# main_app.py
from utils import get_script_dir
print(get_script_dir())
Supposing that your project directory tree is like the following:
Python/
main.py
modules/
utils.py
__init__.py
Being Python/modules/utils.py:
import os
get_script_dir = lambda file: os.path.dirname(os.path.abspath(file))
and Python/main.py:
from modules import utils
import os
script_dir = utils.get_script_dir(__file__)
print("[i] Currently executing file {} located at {}".format(os.path.basename(__file__), script_dir))
Executing Python/main.py is going to output something like:
[i] Currently executing file main.py located at C:\Users\BlackVirusScript\Desktop\Python
I'm trying to compile a python script. On executing the exe I got:-
C:\Python27\dist>visualn.exe
Traceback (most recent call last):
File "visualn.py", line 19, in <module>
File "MMTK\__init__.pyc", line 39, in <module>
File "Scientific\Geometry\__init__.pyc", line 30, in <module>
File "Scientific\Geometry\VectorModule.pyc", line 9, in <module>
File "Scientific\N.pyc", line 1, in <module>
ImportError: No module named Scientific_numerics_package_id
I can see the file Scientific_numerics_package_id.pyd at the location "C:\Python27\Lib\site-packages\Scientific\win32". I want to include this module file into the compilation. I tried to copy the above file in the "dist" folder but no good. Any idea?
Update:
Here is the script:
from MMTK import *
from MMTK.Proteins import Protein
from Scientific.Visualization import VRML2; visualization_module = VRML2
protein = Protein('3CLN.pdb')
center, inertia = protein.centerAndMomentOfInertia()
distance_away = 8.0
front_cam = visualization_module.Camera(position= [center[0],center[1],center[2]+distance_away],description="Front")
right_cam = visualization_module.Camera(position=[center[0]+distance_away,center[1],center[2]],orientation=(Vector(0, 1, 0),3.14159*0.5),description="Right")
back_cam = visualization_module.Camera(position=[center[0],center[1],center[2]-distance_away],orientation=(Vector(0, 1, 0),3.14159),description="Back")
left_cam = visualization_module.Camera(position=[center[0]-distance_away,center[1],center[2]],orientation=(Vector(0, 1, 0),3.14159*1.5),description="Left")
model_name = 'vdw'
graphics = protein.graphicsObjects(graphics_module = visualization_module,model=model_name)
visualization_module.Scene(graphics, cameras=[front_cam,right_cam,back_cam,left_cam]).view()
Py2exe lets you specify additional Python modules (both .py and .pyd) via the includes option:
setup(
...
options={"py2exe": {"includes": ["Scientific.win32.Scientific_numerics_package_id"]}}
)
EDIT. The above should work if Python is able to
import Scientific.win32.Scientific_numerics_package_id
There is a way to work around this types of issues that I have used a number of times. In order to add extra files to the py2exe result you can extend the media collector in order to have a custom version of it. The following code is an example:
import glob
from py2exe.build_exe import py2exe as build_exe
def get_py2exe_extension():
"""Return an extension class of py2exe."""
class MediaCollector(build_exe):
"""Extension that copies Scientific_numerics_package_id missing data."""
def _add_module_data(self, module_name):
"""Add the data from a given path."""
# Create the media subdir where the
# Python files are collected.
media = module_name.replace('.', os.path.sep)
full = os.path.join(self.collect_dir, media)
if not os.path.exists(full):
self.mkpath(full)
# Copy the media files to the collection dir.
# Also add the copied file to the list of compiled
# files so it will be included in zipfile.
module = __import__(module_name, None, None, [''])
for path in module.__path__:
for f in glob.glob(path + '/*'): # does not like os.path.sep
log.info('Copying file %s', f)
name = os.path.basename(f)
if not os.path.isdir(f):
self.copy_file(f, os.path.join(full, name))
self.compiled_files.append(os.path.join(media, name))
else:
self.copy_tree(f, os.path.join(full, name))
def copy_extensions(self, extensions):
"""Copy the missing extensions."""
build_exe.copy_extensions(self, extensions)
for module in ['Scientific_numerics_package_id',]:
self._add_module_data(module)
return MediaCollector
I'm not sure which is the Scientific_numerics_package_id module so I've assumed that you can import it like that. The copy extensions method will get a the different module names that you are having problems with and will copy all their data into the dir folder for you. Once you have that, in order to use the new Media collector you just have to do something like the following:
cmdclass['py2exe'] = get_py2exe_extension()
So that the correct extension is used. You might need to touch the code a little but this should be a good starting point for what you need.
I encountered similar probelm with py2exe and the only solution I can find ,is to use another tool to convert python to exe - pyinstaller
Its very easy tool to use and more important , it works!
UPDATE
As I understood from your comments below , running your script from command line is not working also , due to import error (My recommendation is to first check your code from command line ,and than try to convert it to EXE)
It looks like PYTHONPATH problem.
PYTHONPATH is list of paths (similar of Windows PATH) that python programs use to find import modules.
If your script run from your IDE , that means the PYTHONPATH is set correctly in the IDE ,so all imported modules are found.
In order to set PYTHONPATH you can use :
import sys|
sys.path.append(pathname)
or use the following code that add the all folders under path parameter to PYTHONPATH:
import os
import sys
def add_tree_to_pythonpath(path):
"""
Function: add_tree_to_pythonpath
Description: Go over each directory in path and add it to PYTHONPATH
Parameters: path - Parent path to start from
Return: None
"""
# Go over each directory and file in path
for f in os.listdir(path):
if f == ".bzr" or f.lower() == "dll":
# Ignore bzr and dll directories (optional to NOT include specific folders)
continue
pathname = os.path.join(path, f)
if os.path.isdir(pathname) == True:
# Add path to PYTHONPATH
sys.path.append(pathname)
# It is a directory, recurse into it
add_tree_to_pythonpath(pathname)
else:
continue
def startup():
"""
Function: startup
Description: Startup actions needed before call to main function
Parameters: None
Return: None
"""
parent_path = os.path.normpath(os.path.join(os.getcwd(), ".."))
parent_path = os.path.normpath(os.path.join(parent_path, ".."))
# Go over each directory in parent_path and add it to PYTHONPATH
add_tree_to_pythonpath(parent_path)
# Start the program
main()
startup()
The ImportError is rectified by using "Gil.I" and "Janne Karila" suggestion by setting pythonpath and by using include function. But before this I had to create __init__.py file in the win32 folder of both the modules.
BTW I still got another error for the above script - link