I am using below function to read my python scripts output line by line and save in parallel. But getting Traceback error in end.
Code:
def myrun(cmd):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
stdout = []
while True:
line = p.stdout.readline()
stdout.append(line)
print (line),
if line == '' and p.poll() != None:
break
return ''.join(stdout)
When call function as :
myrun(os.system("./tests_run.py"))
I get below error:
Error:
Traceback (most recent call last):
File "./Portability_Tests.py", line 38, in <module>
myrun(os.system("./tests_run.py"))
File "./Tests.py", line 11, in myrun
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
File "/usr/local/lib/python3.5/subprocess.py", line 676, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.5/subprocess.py", line 1171, in _execute_child
args = list(args)
TypeError: 'int' object is not iterable
Anyone know how can i fix this error ?
The subprocess.Popen function receives a "sequence of program arguments or else a single string" as its args argument.
What you are passing in the args argument is the output of the os.system() call which according to the documentation is the "exit status of the process", thus an int number. Instead in the cmd variable you should directly pass the string (or an other iterator) of your file /tests_run.py. If you want this path to be relative to your current project you can use the os.path module.
Related
I've a main process where I open a multiprocessing.Pipe(False) and send the writing end to a worker Process. Then, in the worker process, I run a Java program using subprocces.Popen(['java', 'myprogram'], stdin=subprocess.PIPE, stdout=subprocess.PIPE). I need to redirect the error of this subprocess to the writing end of multiprocessing.Pipe
For this I referred to this answer by Ilija as this is exactly what I want to achieve, but on my machine(Windows), it throws OSError: [Errno 9] Bad file descriptor
Machine details:
OS - Windows 10 (64bit)
Python version - 3.7.4
Code:
Method 1 (Ilija's answer)
def worker(w_conn):
os.dup2(w_conn.fileno(), 2)
sp = subprocess.Popen(['java', 'myprogram'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
sp.wait()
w_conn.close()
def main():
r_conn, w_conn = multiprocessing.Pipe(False)
process = multiprocessing.Process(target=worker, args=(w_conn,))
process.start()
while not r_conn.poll() and not w_conn.closed:
# Do stuff
else:
# Read error from r_conn, and handle it
r_conn.close()
process.join()
if __name__=='__main__':
main()
Error:
Process Process-1:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\User\Desktop\Workspace\Error.py", line 14, in worker
os.dup2(w_conn.fileno(), 2)
OSError: [Errno 9] Bad file descriptor
Method 2: In worker function, sending w_conn as argument to Popen
def worker(w_conn):
sp = subprocess.Popen(['java', 'myprogram'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=w_conn)
sp.wait()
w_conn.close()
Error:
Process Process-1:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\User\Desktop\Workspace\Error.py", line 13, in worker
sp = subprocess.Popen(['java', 'myprogram'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=w_conn)
File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 728, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 1077, in _get_handles
errwrite = msvcrt.get_osfhandle(stderr.fileno())
OSError: [Errno 9] Bad file descriptor
Is there any workaround/alternate method to achive this on Windows?
I still don't know why "Method 1" is not working. Any information regarding this will be appreciated.
"Method 2" is wrong altogether as we can't use Connection object (returned by multiprocessing.Pipe()) as a file handle in subprocess.Popen.
What works is checking for data in stderr of subprocess sp and sending the data through w_conn to main process.
def worker(w_conn):
sp = subprocess.Popen(['java', 'myprogram'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
sp.wait()
if sp.stderr.seek(0, io.SEEK_END)>0:
w_conn.send(sp.stderr.read())
w_conn.close()
I want to use both multiline input for command 'cat -' and single line input in 'pwd'.
For that i am trying to get input directly in process.communicate() , I am getting broken pipe error. What shall i replace in the argument passed?
command = 'cat -'
stdout_value=''
process = subprocess.Popen(shlex.split(command),
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
shell=True)
while True:
if process.poll() is not None:
break
stdout_value = stdout_value + process.communicate(process.stdin)[0]
print(repr(stdout_value))
This code gets error:
Traceback (most recent call last):
File "try.py", line 67, in <module>
stdout_value = stdout_value + process.communicate(process.stdin)[0]
File "/usr/lib64/python3.5/subprocess.py", line 1068, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib64/python3.5/subprocess.py", line 1687, in _communicate
input_view = memoryview(self._input)
TypeError: memoryview: a bytes-like object is required, not '_io.BufferedWriter'
[vibhcool#localhost try]$ Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe
process.communicate() expects string which you have encoded() to bytes and it returns bytes which you have to decode() to string.
You use process.stdin which is not string. It even doesn't have method read() to do process.communicate(process.stdin.read()) (besides using process.stdin with the same process makes no sense). It could be rather process.communicate(other_process.stdout.read()) (if you have other process)
Working example - I send text with numbers to command sort and it returns text with sorted numbers.
input: 3\n2\n5\n-1
output: -1\n2\n3\n5\n
Code
import subprocess
import shlex
command = 'sort'
process = subprocess.Popen(shlex.split(command),
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
shell=True)
stdout_value = ''
while True:
if process.poll() is not None:
break
result = process.communicate("3\n2\n5\n-1".encode('utf-8'))[0]
stdout_value += result.decode('utf-8')
print(repr(stdout_value))
I am trying to run following code which uses pythos 'subprocess' module.
subprocess.call(cli_args, stdout=client_log, stderr=client_log, timeout=10)
I am using timeout arguement, which is mentioned here to skip from this line if the subprocess stuck at the middle, by killing it.
But when I run this, I am getting following error.
Traceback (most recent call last):
File "test.py", line 152, in <module>
ret = runServiceTest(test_name, server_executable, server_extra_args, client_executable, client_extra_args, protocol, transport, 9090, 0, 0)
File "test.py", line 102, in runServiceTest
ret = subprocess.call(cli_args, stdout=client_log, stderr=client_log, timeout=10)
File "/usr/lib/python2.7/subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
TypeError: __init__() got an unexpected keyword argument 'timeout'
What is the reason for this? How can I solve this? My complete code can be found here.
Judging by your print statements, you are using Python2.x, where subprocess.call does not have a timeout argument:
subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False)
https://docs.python.org/2/library/subprocess.html
I'm trying to collect stderr in memory, instead of directly writing it to a file or stdout. I do this so I can generated the error log file in a certain way. I found a library called StringIO that is an in-memory 'file'. I don't think it does the trick. Here's my code:
buffer = StringIO.StringIO()
status = subprocess.call(args, stdout=log_fps["trace"], stderr=buffer)
if status and self.V_LEVEL:
sys.stderr.write(buffer.getvalue())
print "generated error"
if status:
log_fps["fail"].write("==> Error with files %s and %s\n" % (domain_file, problem_file))
log_fps["fail"].write(buffer.getvalue())
I get the following error:
Traceback (most recent call last):
File "./runit.py", line 284, in <module>
launcher.run_all_cff_domain_examples("ring")
File "./runit.py", line 259, in run_all_cff_domain_examples
result = self.run_clg(in_d["domain"], in_d["problem"], in_d["prefix"])
File "./runit.py", line 123, in run_clg
status = subprocess.call(args, stdout=log_fps["trace"], stderr=buffer)
File "/usr/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 672, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "/usr/lib/python2.7/subprocess.py", line 1075, in _get_handles
errwrite = stderr.fileno()
AttributeError: StringIO instance has no attribute 'fileno'
I guess this means that I can't use StringIO to collect stderr in memory. What else can I do, short of writing to a file in /tmp?
stdout = subprocess.check_output(args)
See check_output documentation for more options.
If you don't want to capture stdout, use Popen.communicate:
from subprocess import Popen, PIPE
p = Popen(args, stdout=log_fps["trace"], stderr=PIPE)
_, stderr = p.communicate()
import subprocess
p = subprocess.Popen(args, stdout=log_fps["trace"], stderr=subprocess.PIPE)
_, stderr = p.communicate()
print stderr,
I'm having a strange issue when using subprocess.Popen.communicate(). For background, I want to execute an application from my python script. When I run the program from the command line, I do it like this (UNIX):
$ echo "input text" | /path/to/myapp
From my script, I also want to pipe the input into the application. So, I tried the following. But I get a "broken pipe" error when I try to send the input with communicate():
>>> cmd = ['/path/to/myapp']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
>>> out,err = p.communicate('input text')
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.5/subprocess.py", line 670, in communicate
return self._communicate(input)
File "/usr/lib/python2.5/subprocess.py", line 1223, in _communicate
bytes_written = self._write_no_intr(self.stdin.fileno(), buffer(input, input_offset, 512))
File "/usr/lib/python2.5/subprocess.py", line 1003, in _write_no_intr
return os.write(fd, s)
OSError: [Errno 32] Broken pipe
To make matters stranger, if I leave out the input data, I don't get any errors. However, this isn't really a good workaround because the application needs input to work.
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
>>> out,err = p.communicate()
>>> print out
[error from myapp regarding lack of input]
Any idea what I'm missing?
Your observation suggests that myapp is terminating without reading (all of the) input. Not knowing anything about myapp, that's hard to confirm, but consider for example
$ echo 'hello world' | tr 'l' 'L'
heLLo worLd
now...:
>>> cmd = ['/usr/bin/tr']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
>>> out,err = p.communicate('hello world')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.5/subprocess.py", line 668, in communicate
return self._communicate(input)
File "/usr/lib/python2.5/subprocess.py", line 1218, in _communicate
bytes_written = self._write_no_intr(self.stdin.fileno(), buffer(input, input_offset, 512))
File "/usr/lib/python2.5/subprocess.py", line 997, in _write_no_intr
return os.write(fd, s)
OSError: [Errno 32] Broken pipe
because...:
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
>>> /usr/bin/tr: missing operand
Try `/usr/bin/tr --help' for more information.
and if we fix the bug:
>>> cmd = ['/usr/bin/tr', 'l', 'L']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
>>> out,err = p.communicate('hello world')>>> print out
heLLo worLd
>>> print err
None
...it fixes everything. What happens if you omit the stderr redirection -- do you perchance see any error messages from myapp...?