I'm running into a wall with regards to using subprocess.call in a python script running in a crontab. I have isolated this problem to be subprocess not able to find the 7z executable. I'm running this on FreeBSD 10.1, but that should not make a difference. I have tried adding PYTHONPATH=$PATH to crontab, I have tried adding shell=True to subprocess.call, and I have tried using /usr/loca/bin/7z rather than 7z. None of these have fixed the problem. The error that I get is the following:
/usr/local/bin/7z: realpath: not found
/usr/local/bin/7z: dirname: not found
exec: /../libexec/p7zip/7z: not found
Here is how I'm calling the script in crontab:
PATH=$PATH:/usr/local/bin
#every_minute $HOME/test.py >> $HOME/test.error 2>&1
Here is the contents of my script (test.py):
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import os
import subprocess
import tempfile
thing = 'blahblahblah'
errors = open('/home/myuser/error', 'wb')
with tempfile.TemporaryDirectory() as tmpdirname:
tempthing = os.path.join(tmpdirname, thing)
fh = open(tempthing, 'wb')
fh.write(b'123')
fh.close()
zipname = '{}.zip'.format(thing)
ziptempfile = os.path.join(tmpdirname, zipname)
zipper = subprocess.call(['7z', 'a', '-p{}'.format('something'), '-tzip', '-y', ziptempfile, tempthing], stdout=errors, stderr=subprocess.STDOUT)
The answer is that the PATH variable in crontab must use an absolute path like so:
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
That fixes everything.
Related
I am trying to call a simple .cmd script with the parameters. In a windows terminal, the command below works:
cmd /c ""C:\Users\Mathieu\Desktop\NeuroDecode\scripts\nd_stream_recorder.cmd" "C:\Users\Mathieu\Desktop" "test""
However, using os.system() with the same command doesn't work.
os.system(r'cmd /c ""C:\Users\Mathieu\Desktop\NeuroDecode\scripts\nd_stream_recorder.cmd" "C:\Users\Mathieu\Desktop" "test""')
EOFError: EOF when reading a line
Could someone explain what my mistake is?
Additionally, here is a more complete code snippet I use:
# -*- coding: utf-8 -*-
import os
import sys
def start_recorder(directory, fname, neurodecode_path):
if sys.platform.startswith('win'):
script = os.path.join(neurodecode_path, 'scripts', 'nd_stream_recorder.cmd')
script = script.replace('/', '\\')
if directory.endswith('/'):
directory = directory[:-1]
elif directory.endswith('\\'):
directory = directory[:-2]
directory = directory.replace('/', '\\')
os.system(f'cmd /c ""{script}" "{directory}" "{fname}""')
else:
pass
if __name__ == '__main__':
directory = 'C:\\Users\\Mathieu\\Desktop'
fname = 'test'
neurodecode_path = 'C:/Users/Mathieu/Desktop/NeuroDecode'
start_recorder(directory, fname, neurodecode_path)
I will also add that the execution of this CMD call should occur in a separate/new terminal.
Consider the following python (3.9) script, called test.py:
import sys
import os
import subprocess
from pathlib import Path
# This is for pyinstaller (as main_dir = sys.path[0] won't work)
if getattr(sys, 'frozen', False):
main_dir = os.path.dirname(sys.executable)
else:
main_dir = os.path.dirname(os.path.abspath(__file__))
processes_dir = Path(main_dir, "processes")
outfile = Path(main_dir, "output.txt")
# Initialize the output text file
with open(outfile, 'w') as f:
f.write('')
# This calls A1.py (see below)
result = subprocess.run([sys.executable, Path(processes_dir, "A1.py")], input="1\n2", capture_output=True, text=True)
# If an error is raised, it's written to output.txt; stdout is written to output.txt
if result.stderr:
with open(outfile, 'a') as f:
f.write("{0}\n\n".format(result.stderr))
else:
with open(outfile, 'a') as f:
f.write("{0}\n\n".format(result.stdout))
The subprocess.run invokes the following simple script:
x1 = int(input())
x2 = int(input())
print(x1+x2)
This runs just fine. I'm trying to work out how to convert this into an executable (.exe) using Pyinstaller. In the appropriate directory, I run:
pyinstaller --onefile test.py
This builds test.exe successfully. When I run test.exe (either from cmd or double-clicking on the file) it opens with no errors, produces an empty output.txt and then simply hangs indefinitely. It appears subprocess.run doesn't work properly with pyinstaller. Any ideas/suggestions to get test.exe to work with pyinstaller?
What happens here is that, when the script is not compiled, sys.executable returns the python executable file (C:\Users\randomuser\AppData\Local\Programs\Python38), but when the code is compiled, sys.executable returns the .exe file you havce made. So, your .exe file calls itself, calling itself again infinite times, hainging.
You could solve this in two different (easy) ways:
(Less recommended if you want to distribute your exe file, because it dependes on a python installation):
Replace sys.executable with 'python'. This will ensure the script is executed with Python rather than with your own .exe file if compiled:
result = subprocess.run(['python', Path(processes_dir, "A1.py")], input="1\n2", capture_output=True, text=True)
You can import the A1.py script (ensuring the script is in the same folder as the executable, and making the main code to be in a function called main returning the result as a string):
Then, you'll be able to run import A1, and call the main procedure running A1.main():
import sys
import os
import subprocess
import A1
from pathlib import Path
# This is for pyinstaller (as main_dir = sys.path[0] won't work)
if getattr(sys, 'frozen', False):
main_dir = os.path.dirname(sys.executable)
else:
main_dir = os.path.dirname(os.path.abspath(__file__))
processes_dir = Path(main_dir, "processes")
outfile = Path(main_dir, "output.txt")
# Initialize the output text file
with open(outfile, 'w') as f:
f.write('')
# This calls A1.py (see below)
result = A1.main()
# The output is written into output.txt.
with open(outfile, 'a') as f:
f.write("{0}\n\n".format(result))
You could consider capturing error tracebacks with try-except clause and with the traceback module
The pyinstaller command then should be: pyinstaller --onefile test.py --add-data "A1.py;."
I've been trying to figure this out for hours with no luck. I have a list of directories that have subdirectories and other files of their own. I'm trying to traverse through all of them and move all of their content to a specific location. I tried shutil and glob but I couldn't get it to work. I even tried to run shell commands using subprocess.call and that also did not work either. I understand that it didn't work because I couldn't apply it properly but I couldn't find any solution that moves all contents of a directory to another.
files = glob.glob('Food101-AB/*/')
dest = 'Food-101/'
if not os.path.exists(dest):
os.makedirs(dest)
subprocess.call("mv Food101-AB/* Food-101/", shell=True)
# for child in files:
# shutil.move(child, dest)
I'm trying to move everything in Food101-AB to Food-101
shutil module of the standart library is the way to go:
>>> import shutil
>>> shutil.move("Food101-AB", "Food-101")
If you don't want to move Food101-AB folder itself, try using this:
import shutil
import os
for i in os.listdir("Food101-AB"):
shutil.move(os.path.join("Food101-AB", i), "Food-101")
For more information about move function:
https://docs.python.org/3/library/shutil.html#shutil.move
Try to change call function to run in order to retrieve the stdout, stderr and return code for your shell command:
from subprocess import run, CalledProcessError
source_dir = "full/path/to/src/folder"
dest_dir = "full/path/to/dest/folder"
try:
res = run(["mv", source_dir, dest_dir], check=True, capture_output=True)
except CalledProcessError as ex:
print(ex.stdout, ex.stderr, ex.returncode)
So I have written a piece of code which first runs a powershell command to generate a UTF-8 version of a DAT file (have been having special character issues with the original file, hence the step). Following which I try to open the newly created file. But the issue is, I keep getting 'FileNotFoundError: [Errno 2]' Initially I was only trying with the file name since the newly created file was in the same folder, but then i tried to generate the absolute path as well.
import os
import subprocess
subprocess.Popen('powershell.exe -Command "Get-Content .\Own.DAT | Set-Content -Encoding utf8 Own1.dat"')
filepath = __file__
filepath = filepath[:-7]
with open(filepath+"Own1.dat", "r") as f:
I can confirm that filepath+"Own1.dat" is fetching the correct filepath. Yet can't figure out what the issue could be.
Edit: Someone asked for confirmation, here is the message i am getting:
C:\Users\Debojit\MiniConda3\python.exe "E:/My Documents/Work/essbase/ownership/test.py"
Traceback (most recent call last):
File "E:/My Documents/Work/essbase/ownership/test.py", line 18, in <module>
with open(filepath+"Own1.dat", "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'E:/My Documents/Work/essbase/ownership/Own1.dat'
Process finished with exit code 1
Note: Curiously enough if i put the powershell command into a separate batch file, write a code in the python script to run it, the works without any issues. Here is the code i am talking about:
import os
import subprocess
from subprocess import Popen
p = Popen("conversion.bat", cwd=r"E:\My Documents\Work\essbase\ownership")
stdout, stderr = p.communicate()
filepath = __file__
filepath = filepath[:-7]
with open(filepath+"Own1.dat", "r") as f:
The conversion.bat file contains the following
powershell.exe -Command "Get-Content .\Own.DAT | Set-Content -Encoding utf8 Own1.DAT"
But I don't want to include a separate batch file to go with the python script.
Any idea what might be causing the issue?
Your error is unrelated to powershell. Popen runs asynchronously. In one command, you are using communicate(), but in the other, you are not.
You're using Popen() incorrectly.
If you want run a command and also pass arguments to it, you have to pass them as a list, like so:
subprocess.Popen(['powershell.exe', '-Command', ...])
In your code, popen tries to run a command literally named powershell.exe -Command "Get-Content ... which of course doesn't exist.
To use a simpler example, this code won't work:
subprocess.Popen('ls -l')
because it's trying to run a command literally named ls -l.
But this does work:
subprocess.Popen(['ls', '-l'])
I still couldn't figure out why the error was happening. But I found a workaround
with open("conversion.bat","w") as f:
f.writelines("powershell.exe -Command \"Get-Content '" +fileName+ "' | Set-Content -Encoding utf8 Own1.dat\"")
from subprocess import Popen
p = Popen("conversion.bat", cwd=os.path.dirname(os.path.realpath(__file__)))
stdout, stderr = p.communicate()
os.remove("conversion.bat")
Basically I would create the batch file, run it and then delete it once the file has been created. Don't why I have to use this route, but it works.
I would like to loop over files using subprocess.run(), something like:
import os
import subprocess
path = os.chdir("/test")
files = []
for file in os.listdir(path):
if file.endswith(".bam"):
files.append(file)
for file in files:
process = subprocess.run("java -jar picard.jar CollectHsMetrics I=file", shell=True)
How do I correctly call the files?
shell=True is insecure if you are including user input in it. #eatmeimadanish's answer allows anybody who can write a file in /test to execute arbitrary code on your machine. This is a huge security vulnerability!
Instead, supply a list of command-line arguments to the subprocess.run call. You likely also want to pass in check=True – otherwise, your program would finish without an exception if the java commands fails!
import os
import subprocess
os.chdir("/test")
for file in os.listdir("."):
if file.endswith(".bam"):
subprocess.run(
["java", "-jar", "picard.jar", "CollectHsMetrics", "I=" + file], check=True)
Seems like you might be over complicating it.
import os
import subprocess
path = os.chdir("/test")
for file in os.listdir(path):
if file.endswith(".bam"):
subprocess.run("java -jar picard.jar CollectHsMetrics I={}".format(file), shell=True)