Hi I have spent a few days reading at stackoverflow for an answer why I get the output listed below instead of seeing the loop in hello.py count forever. Still unclear for me and nothing works. How to start a few scripts in POpen subprocesses and get their output continously. Not when they have finished, because they will run forever. In case this is considered already answered, I'd be glad if someone can link to something functional with Python3.
I run this .py code in the hope to see hello.py execute:
import subprocess
import sys
command = [sys.executable, 'hello.py']
process_handle = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
while process_handle.poll() is None:
for line in iter(process_handle.stdout.readline, ""):
output = line.decode('utf-8')
if output != '':
print('Last number is: ' + output)
Here is what I get. No running of hello.py:
Python 3.7.3 (default, Apr 3 2019, 05:39:12) [GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
hello.py contains:
i = 0
while True:
print(i)
i=i+1
sleep(1)
I tried the first variant from the second duplicate suggestion from live output from subprocess command
It does not compile/execute:
import subprocess
import sys
with open('test.log', 'w') as f: # replace 'w' with 'wb' for Python 3
process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
for c in iter(lambda: process.stdout.read(1), ''): # replace '' with b'' for Python 3
sys.stdout.write(c)
f.write(c)
Result:
Traceback (most recent call last): File "hello.py", line 11, in
Traceback (most recent call last): File "hello5.py", line
7, in
sys.stdout.flush() NameError: name 'sys' is not defined
sys.stdout.write(c) TypeError: write() argument must be str, not bytes
Now I tried the second variant from above, modified like this:
import subprocess
import sys
command = [sys.executable, 'hello.py']
process = subprocess.Popen(command, stdout=subprocess.PIPE)
for line in iter(process.stdout.readline, b''): # replace '' with b'' for Python 3
sys.stdout.write(line)
Result:
Traceback (most recent call last): File "hello.py", line 11, in
Traceback (most recent call last): File "hello5.py", line
8, in
sys.stdout.flush() NameError: name 'sys' is not defined
sys.stdout.write(line) TypeError: write() argument must be str, not bytes
After advice from ShadowRanger I added b'' for binary, sys was already imported however:
import subprocess
import sys
command = [sys.executable, 'hello.py']
process = subprocess.Popen(command, stdout=subprocess.PIPE)
for c in iter(lambda: process.stdout.read(1), b''): # replace '' with b'' for Python 3
sys.stdout.write(c)
Result, despite adding import sys, sys is not defined? Plus now it complains about binary and wants string.
Traceback (most recent call last): File "hello.py", line 11, in
sys.stdout.flush() NameError: name 'sys' is not defined Traceback (most recent call last): File "hello5.py", line 8, in
sys.stdout.write(c) TypeError: write() argument must be str, not bytes
OK, I went with ShadowRanger's info and converted the binary output to text again. hello.py is now printing text.
sys.stdout.write(c.decode('ASCII'))
BUT its coming all at once and thats not what I need. Every printed line in hello.py must show up in realtime and not just when the script finishes.
Got it working, thanks for all help. In order to get output in realtime instead of all of it at the end of script, flushing of stdout must be done in the subprocess script (hello.py in this case) Also, like ShadowRanger said, if b'' for binary is used there will be a complaint from stdout.write. That can be handled by decoding to text.
hello.py
import sys
from time import sleep
i = 0
while True:
print(i)
sys.stdout.flush()
i=i+1
sleep(0.1)
mainscript.py
import sys
import subprocess
command = [sys.executable, 'hello.py']
process = subprocess.Popen(command, stdout=subprocess.PIPE)
for c in iter(lambda: process.stdout.read(1), b''): # replace '' with b'' for Python 3
sys.stdout.write(c.decode('utf-8')) # decode b'' back to text
Related
I'm trying to run the following command using subprocess.run, but I think the hyphen in the command ir-keytable, is confusing the parser. What's the best way to solve this?
ir-keytable -t -s rc0
The error:
stdout:
stderr: Traceback (most recent call last):
File "<string>", line 1, in <module>
NameError: name 'ir' is not defined
My code:
import subprocess
import sys
result = subprocess.run(
[sys.executable, "-c", "ir-keytable('-t -s rc0')"], capture_output=True, text=True
)
print("stdout:", result.stdout)
print("stderr:", result.stderr)
Your Syntax is giving error for any command, you should just try this :
import subprocess
# If you just want to run command then this :
result=subprocess.run("<>")
# But if you want to get result of your command the this:
result=subprocess.check_output("<>")
print(result.decode("utf-8")) # It returns byte type data, so we are converting that into utf-8
I would like to get a full, descriptive error message from failed Python script executed with subprocess module.
I have a following script sub_script.py which fails and produces IndexError: pop from empty list error when executed on it's own:
# sub_script.py
empty_list = []
empty_list.pop()
I am calling sub_script.py from sub_test.py as follows:
# sub_test.py
import subprocess
import sys
print(str(subprocess.run([sys.executable, 'sub_script.py'],
check=True,
capture_output=True)))
However I am only getting subprocess.CalledProcessError error.
Traceback (most recent call last):
File "/Users/my_user/Desktop/my_dir/sub_test.py", line 4, in <module>
print(str(subprocess.run([sys.executable, 'sub_script.py'],
File "/usr/local/Cellar/python#3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/opt/python#3.9/bin/python3.9', 'sub_script.py']' returned non-zero exit status 1.
I would like to see a full description of the error returned from sub_script.py (IndexError: pop from empty list) when it's executed with a subprocess in sub_test.py.
Is it possible to get full error when script executes and fails within subprocess.run?
Keep the return value of the subprocess.run call instead of immediately converting it to a str, and do not have check=True.
# main.py
import subprocess
import sys
command = [sys.executable, 'task.py']
outcome = subprocess.run(command, check=False, capture_output=True)
print(f"returncode = {outcome.returncode}")
if outcome.returncode != 0:
# there was an error, we assume the traceback was printed to stderr
print("there was an error :\n")
print(outcome.stderr.decode("utf-8"))
# task.py
empty_list = []
empty_list.pop()
output :
returncode = 1
there was an error :
Traceback (most recent call last):
File "task.py", line 2, in <module>
empty_list.pop()
IndexError: pop from empty list
I'm working on a short, native (do NOT recommend an outside [non-native] module such as pexpect), cross-platform, insecure remote control application for python (Windows will use py2exe and an exe file). I am using start_new_thread for the blocking calls such as readline(). For some reason, however, I get this string of ugliness as my output:
Unhandled exception in thread started by <function read_stream at 0xb6918730>Unhandled exception in thread started by <function send_stream at 0xb69186f0>
Traceback (most recent call last):
Traceback (most recent call last):
File "main.py", line 17, in read_stream
s.send(pipe.stdout.readline())
AttributeError File "main.py", line 14, in send_stream
pipe.stdin.write(s.recv(4096))
AttributeError: 'NoneType' object has no attribute 'stdin'
: 'NoneType' object has no attribute 'stdout'
Here is my program (main.py):
#!/usr/bin/env python
import socket
import subprocess as sp
from thread import start_new_thread
from platform import system
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('10.0.0.201', 49200))
shell = 'powershell.exe' if system() == 'Windows' else '/bin/bash' # is this right?
pipe = sp.Popen(shell, shell=True, stdin=sp.PIPE, stdout=sp.PIPE, stderr=sp.PIPE)
entered_command=False
def send_stream(): # send what you get from command center
while True:
pipe.stdin.write(s.recv(4096))
def read_stream(): # send back what is returned from shell command
while True:
s.send(pipe.stdout.readline())
start_new_thread(send_stream, ())
start_new_thread(read_stream, ())
Thanks for your help.
It turns out that the problem is that the program was trying to exit after the two start_new_thread calls because it had reached the end, and caused errors while trying to do so. So I replaced:
start_new_thread(send_stream, ())
start_new_thread(read_stream, ())
With:
start_new_thread(send_stream, ())
read_stream()
I'm trying to write a Python program that interacts with the bsdgames trek program. It's kind of like Zork with Klingons:
* * * S T A R T R E K * * *
Press return to continue.
What length game: short
What skill game: fair
Enter a password: hunter2
10 Klingons
2 starbases at 3,6, 0,2
It takes 400 units to kill a Klingon
Command:
I'm currently trying to use subprocess.Popen() to interact with it:
>>> import subprocess
>>> t = subprocess.Popen('trek', stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
Unfortunately:
>>> t.communicate('')
('', None)
>>> t.poll()
-2
>>> t.communicate('')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 754, in communicate
return self._communicate(input)
File "/usr/lib/python2.7/subprocess.py", line 1297, in _communicate
self.stdin.flush()
ValueError: I/O operation on closed file
It ends itself as soon as I .communicate() with it. It seems to respond to my first .communicate():
>>> t = subprocess.Popen('trek', stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
>>> t.communicate('\n')
('\n * * * S T A R T R E K * * *\n\nPress return to continue.\nWhat length game: ', None)
But I need to be able to read stdout to be able to figure out what the next stdin should be. So how do I send stuff to stdin without doing whatever is telling trek that that's the end of its input?
EDIT: Someone suggested t.stdin.write(). This works, but now I can't find a way to read the results:
>>> t.stdin.write('\n')
>>> t.poll()
>>> t.stdout.read()
This hangs forever so:
^CTraceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyboardInterrupt
>>> t.stdout.readline()
''
So now what's going on?
Assuming this is linux, Use the pexpect module. It's written to deal with interactive programs. Techniques like communicate() don't work because they wait for the program to exit. Simply reading doesn't work because the program hasn't flushed its stout so there is nothing to read. You can create your own pty and use that when calling Popen() or let pexpect do the work for you.
I have the following Python code:
import sys
import traceback
fifo_in = sys.argv[1]
while 1:
try:
exec open(fifo_in)
except:
traceback.print_exc()
sys.stdout.flush()
The first argument is a named pipe created by mkfifo. So the following prints '1':
mkfifo input
python script.py input
... in a separate terminal ...
echo "print 1" > input
Great, so far so good. But when I do something like echo "foobar" > input, the script only prints part of the traceback. It then pauses until I send it another command, and the output gets all mixed up:
echo "asdf" > input # pause here and check output
echo "print 1" > input
... in output terminal ...
Traceback (most recent call last):
File "test.py", line 8, in <module>
exec open(fifo_in)
File "in", line 1, in <module>
...PAUSES HERE...
print 1
NameError: name 'asdf' is not defined
What's going on? How can I get stdout to flush fully and why is it out of order? I've tried using traceback.format_exc instead, then printing it by hand, but I get the same result. Calling sys.stderr.flush does not fix anything either. I've also tried putting a sleep in the loop to see if that helps, but nothing.
UPDATE
One interesting piece of behavior I am seeing: If I ctrl+c it, normally the program keeps running - the try/except just catches the KeyboardInterrupt and it keeps looping. However, if I ctr+c it after sending it an error, the program exits and I get the following. It's almost like it pauses inside of print_exc:
^CTraceback (most recent call last):
File "test.py", line 10, in <module>
traceback.print_exc()
File "/usr/lib/python2.7/traceback.py", line 232, in print_exc
print_exception(etype, value, tb, limit, file)
File "/usr/lib/python2.7/traceback.py", line 125, in print_exception
print_tb(tb, limit, file)
File "/usr/lib/python2.7/traceback.py", line 69, in print_tb
line = linecache.getline(filename, lineno, f.f_globals)
File "/usr/lib/python2.7/linecache.py", line 14, in getline
lines = getlines(filename, module_globals)
File "/usr/lib/python2.7/linecache.py", line 40, in getlines
return updatecache(filename, module_globals)
File "/usr/lib/python2.7/linecache.py", line 132, in updatecache
with open(fullname, 'rU') as fp:
KeyboardInterrupt
I think you want to look at the stdlib code module
This behavior is from using exec. Exec is for evaluating python code so "print 1" executes the python code print 1, where as "asdf" will raise a NameError as it does not exist in the context. exec open(fifo_in) is strange as it shouldn't work. The while will also eat up 100% cpu.
UPDATE: fix sleep duration
Here is a modified version of your code to try.
import sys
import time
import traceback
fifo_in = sys.argv[1]
try:
fp = open(fifo_in) # will block until pipe is opened for write
except IOError:
traceback.print_exc()
except OSError:
traceback.print_exc()
data = None
while True:
try:
data = fp.read()
try:
exec data
except:
traceback.print_exc()
finally:
time.sleep(0.1)
except KeyboardInterrupt:
break