I have a python script that opens a .exe program using the subprocess module. This .exe program is an infinitely iterative script, in that it will continue to print the results of each iteration until the user closes the window. Every so often, it prints the results of the iteration into a file, replacing the previous data in the file.
My aims here are to:
Run the .exe program, and test for the existence of the file it outputs.
Once the file has been shown to exist, I need to run a test on the file to see if the iteration has converged to within a given tolerance. Once the iteration has converged, I need to kill the .exe subprocess.
This is my current code. It is designed to kill the subprocess once the iterate file has been created:
import subprocess
from subprocess import Popen, PIPE
fileexists = False
iteratecomms = Popen('iterate.exe', stdout=PIPE, stdin=PIPE, stderr=PIPE)
# Begin the iteration. Need to select options 1 and then 1 again at program menu
out, err = iteratecomms.communicate("1\n1\n".encode())
while (fileexists == False):
fileexists = os.path.exists(filelocation)
else:
Popen.kill(iteratecomms)
I know that this is incorrect; the issue is that as soon as I start the out, err = iteratecomms.communicate("1\n1\n".encode()) line, the program begins iterating, and does not move on to the next set of python code. Essentially, I need to start the .exe program, and at the same time test to see if the file has been created. I can't do this, however, because the program runs indefinitely.
How could I get around this? I have assumed that moving on to step 2 (testing the file and killing the subprocess under certain conditions) would not take too much work on top of this; if this is not true, how would I go about completing all of my aims?
Thank you very much for the help!
Edit: Clarified that the external file is overwritten.
I would use the multiprocessing module.
pool = multiprocessing.Pool()
def start_iteration():
return Popen('iterate.exe', stdout=PIPE, stdin=PIPE, stderr=PIPE)
pool.apply_async(start_iteration)
while (fileexists == False):
fileexists = os.path.exists(filelocation)
Popen.kill(???)
The only problem now is that you'll have to somehow find the PID of the process without waiting for Popen to return (because Popen should never return.)
Assuming that you're trying to continuously trying to read this file I would suggest running a tail on the file in question. This can be done from a separate terminal in any *nix family OS, but otherwise I would check out this article for a Python implementation:
http://code.activestate.com/recipes/157035-tail-f-in-python/
After that if you want to kill the program running you should just be able to call terminate on the process running:
import subprocess
sub = subprocess.popen(#Whatever)
#Do something
sub.terminate()
Related
I'm trying to have one subprocess open a program (rv in this case, using a specific tag) and then wait for this program to start before running two additional rvpush commands against this same rv session:
tag = "my-tag"
path = "/path/to/file/to/open/my_file"
cmd = f"rvpush -tag {tag} merge {path}"
cmd += f"; sleep 7; rvpush -tag {tag} py-eval \"rv.commands.setViewNode('master')\""
cmd += f"; rvpush -tag {tag} py-eval \"[rv.commands.setStringProperty(f'{{stack}}.composite.type', ['replace'], True) for stack in rv.commands.nodesOfType('RVStack')]\""
proc = subprocess.Popen(cmd, shell=True)
print("Starting the subprocess.")
proc.wait()
print(f"Subprocess ended. Deleting temporary file {path}.")
pathlib.Path(path).unlink()
I can do this by adding a sleep in-between, as above suggests, to let the program start, but I was hoping there would be a better solution for this.
I've played around with all different call, Popen, check_call as well as proc.wait(), proc.communicate(), proc.poll() etc, but I can't make it work without adding a sleep.
Do anyone know if you can somehow track via subprocess if the program has started successfuly? I feels it's not possible as subprocess only seem aware of if the subprocess is executed or finished, not the context that it runs.
Edit: For some context, the problem with using proc.wait() is that it waits for the process to finish, which is not what I want. I want my next commands to execute when the subprocess has finished opening the program via that initial command. This adds some complexity that I do not think subprocess initially handles, as it doesn't know about the context of which the command it executes has.
I'm trying to spawn a subprocess that should still be running after the main process closed. This part works fine, but if I redirect the output of this process to a file, I can't start the script a second time because the process still blocks the log file.
This short example demonstrates the problem:
In this case the second process is "notepad" and is started by "other.cmd". While the main process/script is "test_it.py" which is started by "start_it.cmd".
start_it.cmd
#python test_it.py > test.log
test_it.py
import subprocess
from subprocess import DEVNULL, STDOUT
subprocess.Popen(["other.cmd"], stdin=DEVNULL, stdout=DEVNULL, stderr=STDOUT)
other.cmd
start notepad
When start_it.cmd is executed the second time, it will fail with this error message "The process cannot access the file because it is being used by another process".
How can I start the subprocess so that it doesn't block the log file?
A solution using a pipe.
multiplexer.py
with open('log.txt', 'a') as outputFile:
while True:
data = sys.stdin.read(1024)
if None == data:
break
outputFile.write(data)
start_it.cmd
#python test_it.py | python multiplexer.py
Everything else stays the same.
I found a solution that is close to what I originally intended:
subprocess.Popen("explorer other.cmd", shell=True)
By letting the explorer start the .cmd file this succesfully detaches the called .cmd from the original process. And thus doesn't keep the log file open.
I have two scripts in Python.
sub.py code:
import time
import subprocess as sub
while 1:
value=input("Input some text or number") # it is example, and I don't care about if it is number-input or text-raw_input, just input something
proces=sub.Popen(['sudo', 'python', '/home/pi/second.py'],stdin=sub.PIPE)
proces.stdin.write(value)
second.py code:
import sys
while 1:
from_sub=sys.stdin()#or sys.stdout() I dont remember...
list_args.append(from_sub) # I dont know if syntax is ok, but it doesn't matter
for i in list_arg:
print i
First I execute sub.py, and I input something, then second.py file will execute and printing everything what I inputed and again and again...
The thing is I don't want to open new process. There should be only one process. Is it possible?
Give me your hand :)
This problem can be solved by using Pexpect. Check my answer over here. It solves a similar problem
https://stackoverflow.com/a/35864170/5134525.
Another way to do that is to use Popen from subprocess module and setting stdin and stdout as pipe. Modifying your code a tad bit can give you the desired results
from subprocess import Popen, PIPE
#part which should be outside loop
args = ['sudo', 'python', '/home/pi/second.py']
process = Popen(args, stdin=PIPE, stdout=PIPE)
while True:
value=input("Input some text or number")
process.stdin.write(value)
You need to open the process outside the loop for this to work. A similar issue is addressed here in case you want to check that Keep a subprocess alive and keep giving it commands? Python
This approach will lead to error if child process quits after first iteration and close all the pipes. You somehow need to block the child process to accept more input. This you can do by either using threads or by using the first option i.e. Pexpect
From python I am calling a java function:
os.system("java -jar example.jar run myFunction 'inFile.txt' 'outFile.txt' " )
This function is processing a file and the output is written into 'outFile.txt'. The output is dependent on the information in 'inFile.txt'. While processing the input file and writing into out file, sometimes the 'outFile.txt' grows too large (tens of GBs) and at that time, i want to quit and the current processing and move on to process another inFile.txt
Is there is way to know that my outFile.txt that is being written has grown more than say 10GB.
Edit:
As suggested by Maksym, I am using the following code and seems to be working. Thanks
import subprocess
from time import sleep
p = subprocess.Popen(["java", "-jar", "example.jar", "run", "myFunction", "'inFile.txt'", "'outFile.txt'")
rc = p.poll() #returncode
while (rc == None):
sleep(1)
if(os.path.getsize(outFileName) < 1000000000):
rc = p.poll()
continue
else:
p.kill()
break
Have a look at subprocess module. Using Popen you can fork a process and kill it when you need this:
import subprocess
from time import sleep
p = subprocess.Popen(["java", "-jar", "example.jar", "run", "myFunction", "'inFile.txt'", "'outFile.txt'")
while not check_my_conditions():
sleep(my_timeout)
p.kill()
Then, you can rotate your files and restart the process.
Instead of directly calling os.system, you should strongly considering using the multiprocessing.Process built-in class. It handles dealing with spawned processes much more gracefully.
You need to watch the output file periodically, either using something like os.stat to check the file size. You can then kill the original process (or whatever you want to do) when the threshold is exceeded.
Does the java application provide any output (for example, a count of records processed) to stdout or stderr while it runs? If so, you could invoke it using Python's Popen class (in the subprocess module) and estimate when it has processed 'too much'.
I have some code to execute the unix shell command in background in python
import subprocess
process = subprocess.Popen('find / > tmp.txt &',shell=True)
I need to capture the scenario where i come to know that process has finished successful
completion .
Please explain with sample code
Tazim
There is no need for the &: the command is launched in a separate process, and runs independently.
If you want to wait until the process terminates, use wait():
process = subprocess.Popen('find / > tmp.txt', shell = True)
exitcode = process.wait()
if exitcode == 0:
# successful completion
else:
# error happened
If your program can do something meaningful in the meantime, you can use poll() to determine if the process has finished.
Furthermore, instead of writing the output to a temporary file and then reading it from your Python program, you can read from the pipe directly. See the subprocess documentation for details.
Don't use shell=True. It is bad for your health.
proc = subprocess.Popen(['find', '/'], stdout=open('tmp.txt', 'w'))
if proc.wait() == 0:
pass
If you really need a file, use import tempfile instead of a hard-coded temporary file name. If you don't need the file, use a pipe (see the subprocess docs as Thomas suggests).
Also, don't write shell scripts in Python. Use the os.walk function instead.