Consider the following python (3.9) script, called test.py:
import sys
import os
import subprocess
from pathlib import Path
# This is for pyinstaller (as main_dir = sys.path[0] won't work)
if getattr(sys, 'frozen', False):
main_dir = os.path.dirname(sys.executable)
else:
main_dir = os.path.dirname(os.path.abspath(__file__))
processes_dir = Path(main_dir, "processes")
outfile = Path(main_dir, "output.txt")
# Initialize the output text file
with open(outfile, 'w') as f:
f.write('')
# This calls A1.py (see below)
result = subprocess.run([sys.executable, Path(processes_dir, "A1.py")], input="1\n2", capture_output=True, text=True)
# If an error is raised, it's written to output.txt; stdout is written to output.txt
if result.stderr:
with open(outfile, 'a') as f:
f.write("{0}\n\n".format(result.stderr))
else:
with open(outfile, 'a') as f:
f.write("{0}\n\n".format(result.stdout))
The subprocess.run invokes the following simple script:
x1 = int(input())
x2 = int(input())
print(x1+x2)
This runs just fine. I'm trying to work out how to convert this into an executable (.exe) using Pyinstaller. In the appropriate directory, I run:
pyinstaller --onefile test.py
This builds test.exe successfully. When I run test.exe (either from cmd or double-clicking on the file) it opens with no errors, produces an empty output.txt and then simply hangs indefinitely. It appears subprocess.run doesn't work properly with pyinstaller. Any ideas/suggestions to get test.exe to work with pyinstaller?
What happens here is that, when the script is not compiled, sys.executable returns the python executable file (C:\Users\randomuser\AppData\Local\Programs\Python38), but when the code is compiled, sys.executable returns the .exe file you havce made. So, your .exe file calls itself, calling itself again infinite times, hainging.
You could solve this in two different (easy) ways:
(Less recommended if you want to distribute your exe file, because it dependes on a python installation):
Replace sys.executable with 'python'. This will ensure the script is executed with Python rather than with your own .exe file if compiled:
result = subprocess.run(['python', Path(processes_dir, "A1.py")], input="1\n2", capture_output=True, text=True)
You can import the A1.py script (ensuring the script is in the same folder as the executable, and making the main code to be in a function called main returning the result as a string):
Then, you'll be able to run import A1, and call the main procedure running A1.main():
import sys
import os
import subprocess
import A1
from pathlib import Path
# This is for pyinstaller (as main_dir = sys.path[0] won't work)
if getattr(sys, 'frozen', False):
main_dir = os.path.dirname(sys.executable)
else:
main_dir = os.path.dirname(os.path.abspath(__file__))
processes_dir = Path(main_dir, "processes")
outfile = Path(main_dir, "output.txt")
# Initialize the output text file
with open(outfile, 'w') as f:
f.write('')
# This calls A1.py (see below)
result = A1.main()
# The output is written into output.txt.
with open(outfile, 'a') as f:
f.write("{0}\n\n".format(result))
You could consider capturing error tracebacks with try-except clause and with the traceback module
The pyinstaller command then should be: pyinstaller --onefile test.py --add-data "A1.py;."
Related
I have a C file say, myfile.c.
Now to compile I am doing : gcc myfile.c -o myfile
So now to run this I need to do : ./myfile inputFileName > outputFileName
Where inputFileName and outputFileName are 2 command line inputs.
Now I am trying to execute this within a python program and I am trying this below approach but it is not working properly may be due to the >
import subprocess
import sys
inputFileName = sys.argv[1];
outputFileName = sys.argv[2];
subprocess.run(['/home/dev/Desktop/myfile', inputFileName, outputFileName])
Where /home/dev/Desktop is the name of my directory and myfile is the name of the executable file.
What should I do?
The > that you use in your command is a shell-specific syntax for output redirection. If you want to do the same through Python, you will have to invoke the shell to do it for you, with shell=True and with a single command line (not a list).
Like this:
subprocess.run(f'/home/dev/Desktop/myfile "{inputFileName}" > "{outputFileName}"', shell=True)
If you want to do this through Python only without invoking the shell (which is what shell=True does) take a look at this other Q&A: How to redirect output with subprocess in Python?
You can open the output file in Python, and pass the file object to subprocess.run().
import subprocess
import sys
inputFileName = sys.argv[1];
outputFileName = sys.argv[2];
with open(outputFileName, "w") as out:
subprocess.run(['/home/dev/Desktop/myfile', inputFileName], stdout=out)
When I run my python script via the terminal by going into the directory the python script is held and running > python toolstation.py, the script runs successfully.
Then what I try to do is run the script via a .bat file. My .bat file is set as so:
"C:\Users\xxxx\AppData\Local\Programs\Python\Python39\python.exe" "C:\Users\xxxx\Downloads\axp_solutions\python_scripts\toolstation.py"
When I run this bat file, it gives me an exception which states it cannot find the directory to open the csv file, which is one directory above the python script.
Exception:
Traceback (most recent call last):
File "C:\Users\xxx\Downloads\axp_solutions\python_scripts\toolstation.py", line 12, in <module>
f = open('../input/toolstation.csv', 'r')
FileNotFoundError: [Errno 2] No such file or directory: '../input/toolstation.csv'
The code for this in the python script is set like so:
f = open('../input/toolstation.csv', 'r')
Now I can set this via a hardcoded path like so to get around it:
f = open('C:/Users/xxxx/Downloads/axp_solutions/input/toolstation.csv', 'r')
But as I am sending this script and bat file to a friend, they will have a different path set. So my question is, how should the dynamic path be set so that it is able to recognise the directory to go to?
Instead of constructing the path to the CSV file (based on this answer), I would suggest using Python's argparse library to add an optional argument which takes the path to that CSV file.
You could give it a reasonable default value (or even use the automatically determined relative path as the default), so that you don't have to specify anything if you are on your system, while at the same time adding a lot of flexibility to your script.
You and everyone using your script, can at any moment decide which CSV file to use, while the overhead is manageable.
Here's what I would start with:
import argparse
import os
DEFAULT_PATH = r'C:\path\that\makes\sense\here\toolstation.csv'
# Alternatively, automatically determine default path:
# DEFAULT_PATH = os.path.join(
# os.path.dirname(os.path.abspath(__file__)),
# r'..\input\toolstation.csv',
# )
parser = argparse.ArgumentParser(prog='toolstation')
parser.add_argument(
'-i',
'--input-file',
default=DEFAULT_PATH,
help='Path to input CSV file for this script (default: %(default)s).'
)
args = parser.parse_args()
try:
in_file = open(args.input_file, 'r')
except FileNotFoundError:
raise SystemExit(
f"Input file '{args.input_file}' not found. "
"Please provide a valid input file and retry. "
"Exiting..."
)
Your script gets a nice and extensible interface with out-of-the-box help (you can just run python toolstation.py --help).
People using your script will love you because you provided them with the option to choose their input file via:
python toolstation.py --input-file <some_path>
The input file path passed to the script can be absolute or relative to the directory the script is executed from.
Use os module to get the script path and then add the path to the src file.
Ex:
import os
path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'input', 'toolstation.csv')
with open(path, 'r') as infile:
...
You can pass the path to that CSV in as an argument to the python script.
This tutorial may be helpful.
The batch file:
set CSV_PATH="C:\...\toolstation.csv"
"C:\...\python.exe" "C:\...\toolstation.py" %CSV_PATH%
You'll have to fill in the "..."s with the appropriate paths; this isn't magic.
The Python script:
import sys
toolstation_csv = sys.argv[1]
...
f = open(toolstation_csv, 'r')
...
I created a .exe file with Pyinstaller. The program should create a data file with the results at runtime. It works in the Python interpreter mode when I execute python test.py but it doesn't work in the .exe mode. It simply doesn't create a file and responds with an error.
Which flags should I add to the command python -m PyInstaller --hidden-import=timeit --hidden-import=bisect -F test.py to make it work?
The exception with this setup is:
Error: [Errno 2] No such file or directory: "C:\Users\Admin\AppData\Local\Temp\..."
Where the aforementioned directory is temporary and I don't have access to it.
The piece of code which is supposed to write a file is:
def write_file(data, path, encoding='utf-8'):
'''Creates report files.'''
try:
if config.python_version == 2 and type(data) in (list, str):
f = io.open(path, 'wb')
else:
f = io.open(path, 'w', encoding=encoding, newline='')
if type(data) is list:
writer = csv.writer(f)
writer.writerows(data)
else:
f.write(data)
f.close()
console(u'Report file: ' + path)
except IOError as e:
console(e, level=Level.error)
I assume there should be a setting to point to the place where the file should be saved.
I've checked here https://pyinstaller.readthedocs.io/en/stable/spec-files.html#adding-files-to-the-bundle but without success. I couldn't use the listed flags properly, nothing seemed to work.
How can I specify the place where the file would be saved?
The issue isn't with pyinstaller, it's with however you're creating your file.
You may be using some environment variable when running your python script from the command line that isn't set when you run your Exe
I have created a simple example of a program that creates a data file in the directory from which it is called:
#myscript.py
f = open("Test.txt", "w")
print("Hello World!", file=f)
Then I generate the Exe with Pyinstaller:
pyinstaller -F myscript.py
Copy the exe anywhere and you can create Test.txt, if you have permissions in that folder.
Double click myscript.exe
Test.txt appears in the same folder as myscript.exe
Hello World!
I expect a directory to be created and then a file to be opened within it for writing to when I execute my code below in Python 2.6.6,
import subprocess
def create_output_dir(work_dir):
output_dir = '/work/m/maxwell9/some_name5/'
subprocess.Popen(['mkdir', output_dir])
return output_dir
if __name__ == '__main__':
work_dir = '/work/m/maxwell9/'
output_dir = create_output_dir(work_dir)
#output_dir = '/work/m/maxwell9/some_name5/'
filename = output_dir + 'bt.sh'
with open(filename, 'w') as script:
print('there')
but instead I get the error,
Traceback (most recent call last):
File "slurm_test.py", line 13, in <module>
with open(filename, 'w') as script:
IOError: [Errno 2] No such file or directory: '/work/m/maxwell9/some_name5/bt.sh'
If I run the script, I can then see that the directory is created. If I then uncomment the line,
#output_dir = '/work/m/maxwell9/some_name5/'
and comment the line,
output_dir = create_output_dir(work_dir)
then the file is output fine. So there is something about creating the folder and then writing to it in the same script that is causing an error.
subprocess.Popen starts up an external process but doesn't wait for it to complete unless you tell it to (e.g. by calling .wait on the returned Popen instance). Most likely, mkdir is in the process of creating a directory while open(filename, 'w') attempts to create a file in that directory. This is an example of a "race condition".
The solution is to .wait on the open process (as noted above), or you can use one of the convenience wrappers subprocess.check_output, subprocess.check_call or (even better), you can avoid subprocess entirely by using os.mkdir or os.makedirs.
You could use the os library instead of subprocess, which makes for a more straightforward implementation. Try swapping out your create_output_dir function with this:
import os
def create_output_dir(work_dir):
try:
os.makedirs(work_dir)
except OSError:
pass
return work_dir
I'm running into a wall with regards to using subprocess.call in a python script running in a crontab. I have isolated this problem to be subprocess not able to find the 7z executable. I'm running this on FreeBSD 10.1, but that should not make a difference. I have tried adding PYTHONPATH=$PATH to crontab, I have tried adding shell=True to subprocess.call, and I have tried using /usr/loca/bin/7z rather than 7z. None of these have fixed the problem. The error that I get is the following:
/usr/local/bin/7z: realpath: not found
/usr/local/bin/7z: dirname: not found
exec: /../libexec/p7zip/7z: not found
Here is how I'm calling the script in crontab:
PATH=$PATH:/usr/local/bin
#every_minute $HOME/test.py >> $HOME/test.error 2>&1
Here is the contents of my script (test.py):
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import os
import subprocess
import tempfile
thing = 'blahblahblah'
errors = open('/home/myuser/error', 'wb')
with tempfile.TemporaryDirectory() as tmpdirname:
tempthing = os.path.join(tmpdirname, thing)
fh = open(tempthing, 'wb')
fh.write(b'123')
fh.close()
zipname = '{}.zip'.format(thing)
ziptempfile = os.path.join(tmpdirname, zipname)
zipper = subprocess.call(['7z', 'a', '-p{}'.format('something'), '-tzip', '-y', ziptempfile, tempthing], stdout=errors, stderr=subprocess.STDOUT)
The answer is that the PATH variable in crontab must use an absolute path like so:
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
That fixes everything.