I want to run the 'time' unix command from a Python script, to time the execution of a non Python app. I would use the os.system method.
Is there any way to save the output of this in Python? My goal is to run the app several times, save their execution times and then do some statistics on them.
Thank You
You really should be using the subprocess module to run external commands. The Popen() method lets you specify a file object where stdout should go (note, you can use any Python object that behaves like a file, you don't necessarily need to write it to a file).
For instance:
import subprocess
log_file = open('/path/to/file', 'a')
return_code = subprocess.Popen(['/usr/bin/foo', 'arg1', 'arg2'], stdout=log_file).wait()
You can use the timeit module to time code snippets (say, a function that launches external commands using subprocess module as described in the answer above) and save the data to a csv file. You can do statistics on the csv data using a stats module or externally using Excel/ LogParser/ R etc.
Another approach is to use the hotshot profiler that does the profiling and also returns stats that you can either print using print_stats() method or save to a file by iterating over.
Related
Iv'e been using the following shell command to read the image off a scanner named scanner_name and save it in a file named file_name
scanimage -d <scanner_name> --resolution=300 --format=tiff --mode=Color 2>&1 > <file_name>
This has worked fine for my purposes.
I'm now trying to embed this in a python script. What I need is to save the scanned image, as before, into a file and also capture any std output (say error messages) to a string
I've tried
scan_result = os.system('scanimage -d {} --resolution=300 --format=tiff --mode=Color 2>&1 > {} '.format(scanner, file_name))
But when I run this in a loop (with different scanners), there is an unreasonably long lag between scans and the images aren't saved until the next scan starts (the file is created as an empty file and is not filled until the next scanning command). All this with scan_result=0, i.e. indicating no error
The subprocess method run() has been suggested to me, and I have tried
with open(file_name, 'w') as scanfile:
input_params = '-d {} --resolution=300 --format=tiff --mode=Color 2>&1 > {} '.format(scanner, file_name)
scan_result = subprocess.run(["scanimage", input_params], stdout=scanfile, shell=True)
but this saved the image in some kind of an unreadable file format
Any ideas as to what may be going wrong? Or what else I can try that will allow me to both save the file and check the success status?
subprocess.run() is definitely preferred over os.system() but neither of them as such provides support for running multiple jobs in parallel. You will need to use something like Python's multiprocessing library to run several tasks in parallel (or painfully reimplement it yourself on top of the basic subprocess.Popen() API).
You also have a basic misunderstanding about how to run subprocess.run(). You can pass in either a string and shell=True or a list of tokens and shell=False (or no shell keyword at all; False is the default).
with_shell = subprocess.run(
"scanimage -d {} --resolution=300 --format=tiff --mode=Color 2>&1 > {} ".format(
scanner, file_name), shell=True)
with open(file_name) as write_handle:
no_shell = subprocess.run([
"scanimage", "-d", scanner, "--resolution=300", "--format=tiff",
"--mode=Color"], stdout=write_handle)
You'll notice that the latter does not support redirection (because that's a shell feature) but this is reasonably easy to implement in Python. (I took out the redirection of standard error -- you really want error messages to remain on stderr!)
If you have a larger working Python program this should not be awfully hard to integrate with a multiprocessing.Pool(). If this is a small isolated program, I would suggest you peel off the Python layer entirely and go with something like xargs or GNU parallel to run a capped number of parallel subprocesses.
I suspect the issue is you're opening the output file, and then running the subprocess.run() within it. This isn't necessary. The end result is, you're opening the file via Python, then having the command open the file again via the OS, and then closing the file via Python.
JUST run the subprocess, and let the scanimage 2>&1> filename command create the file (just as it would if you ran the scanimage at the command line directly.)
I think subprocess.check_output() is now the preferred method of capturing the output.
I.e.
from subprocess import check_output
# Command must be a list, with all parameters as separate list items
command = ['scanimage',
'-d{}'.format(scanner),
'--resolution=300',
'--format=tiff',
'--mode=Color',
'2>&1>{}'.format(file_name)]
scan_result = check_output(command)
print(scan_result)
However, (with both run and check_output) that shell=True is a big security risk ... especially if the input_params come into the Python script externally. People can pass in unwanted commands, and have them run in the shell with the permissions of the script.
Sometimes, the shell=True is necessary for the OS command to run properly, in which case the best recommendation is to use an actual Python module to interface with the scanner - versus having Python pass an OS command to the OS.
I want to run some command line scripts from within my python program. These scripts generates some output files. I want to grab these output files from the subprocess call as object in my python program, while canceling generation of files on disk. Problem is I don't know how to do it, or whether that is even possible.
A simple example would look like this:
#foo.py
fout1 = open("temp1.txt","w")
fout2 = open("temp2.txt","w")
fout1.write("fout1")
fout2.write("fout2")
fout1.close()
fout2.close()
#test.py
import subprocess
process = subprocess.Popen(["python","foo.py"], ????????) #what arguments to use to grab temp1.txt and temp2.txt
print(process.??????) #how to access those files
I am familiar with subprocess.Popen so that is what the example code uses, but I am open to the use of other modules too if they could do it.
I got a python file which is a code that I developed. During his execution I input from the keyboard several characters at different stages of the program itself. Also, during the execution, I need to close a notepad session which comes out when I execute into my program the command subprocess.call(["notepad",filename]). Having said that I would like to run this code several times with inputs which change according to the case and I was wondering if there is an automatic manner to do that. Assuming that my code is called 'mainfile.py' I tried the following command combinations:
import sys
sys.argv=['arg1']
execfile('mainfile.py')
and
import sys
import subprocess
subprocess.call([sys.executable,'mainfile.py','test'])
But it does not seem to work at least for the first argument. Also, as the second argument should be to close a notepad session, do you know how to pass this command?
Maybe have a look at this https://stackoverflow.com/a/20052978/4244387
It's not clear what you are trying to do though, I mean the result you want to accomplish seems to be just opening notepad for the sake of saving a file.
The subprocess.call() you have is the proper way to execute your script and pass it arguments.
As far as launching notepad goes, you could do something like this:
notepad = subprocess.Popen(['notepad', filename])
# do other stuff ...
notepad.terminate() # terminate running session
I am currently using subprocess to run a Python script inside of my current Python but it is keep giving me an error:
for dir in os.listdir(os.path.join(DIR2,dirname)):
temp = os.path.join(os.path.join(DIR2,dirname),dir)
files = [os.path.join(temp, f) for f in os.listdir(temp) if f.endswith("json")]
for lists in files:
subprocess.Popen(["python", DIR4, os.path.join(temp,lists)])
Above is what I am currently using.
DIR4 is the path of the python that I want to run.
Problem is, the python that I want to run can only take one file at a time.
However this subprocess looks like it tries to execute ALL at ONCE.
I want to run ONE at a time, instead of ALL at ONCE.
Because it is running ALL at ONCE, my python that I want to run does not work the way it is..
What do I need to do to change this?
If you want to wait first for the subprocess to terminate, before going ahead, I think you could use Popen.wait():
...
p = subprocess.Popen(["python", DIR4, os.path.join(temp,lists)])
p.wait()
...
To actually do what you're asking, and not hack it together through subprocess, you can use exec which allows you to run python code with your own provided globals and locals.
In older versions of Python (meaning pre-3), you can use execfile to achieve the same thing.
I wrote a piece of python code that calls a external program to write an intermediate file and thereafter my code reads from it. I want to run multiple instances of my code simultaneously. Will there be any conflict if I code list this?
args=['/usr/bin/program','-o','intermediate_file']
process = subprocess.Popen(args,shell=False)
process.wait()
if process.returncode ==0:
fh = open('intermediate_file', 'r')
process(fh)
...
Concurrent file access is handled by the operating system. There are several scenarios, depending on the OS and or filesystem you use. Take a look at the Wikipedia-article.
Take a look here: tempfile
You can make use of this lib to avoid conflicts - temp files have random names.