process = subprocess.check_output(BACKEND+"mainbgw setup " + str(NUM_USERS), shell=True,\
stderr=subprocess.STDOUT)
I am using the above statement to run a C program in django-python based server for some computations, there are some printf() statements whose output I would like to see on stdout while the server is running and executing the subprocess, how can that be done ?
If you actually don't need the output to be available to your python code as a string, you can just use os.system, or subprocess.call without redirecting stdout elsewhere. Then stdout of your C program will just go directly to stdout of your python program.
If you need both streaming stdout and access to the output as a string, you should use subprocess.Popen (or the old popen2.popen4) to obtain a file descriptor of the output stream, then repeatedly read lines from the stream until you exhausted it. In the mean time, you keep a concatenated version of all data you grabbed. This is an example of the loop.
Related
I would like to run a command using subprocess.run() and then get its stdout/stderr as a string, but I want the subprocess to also print its output to the console normally while running. If I do
result = subprocess.run(['ls', '-al'])
then I can see the output printed to my console but I can't access the output after the command runs. If I do
result = subprocess.run(['ls', '-al'], capture_output=True, text=True)
I can access result.stdout and result.stderr but nothing is printed to the console while the command is running. Can I have both printing to the console and saving to result.stdout?
From the documentation of subprocess.run :
Run the command described by args. Wait for command to complete, then return a CompletedProcess instance.
[...]
If capture_output is true, stdout and stderr will be captured. When used, the internal Popen object is automatically created with stdout=PIPE and stderr=PIPE.
The docs for subprocess.PIPE say :
Special value that can be used as the stdin, stdout or stderr argument to Popen and indicates that a pipe to the standard stream should be opened. Most useful with Popen.communicate().
The doc for the Popen constructor parameter stdout :
stdin, stdout and stderr specify the executed program’s standard input, standard output and standard error file handles, respectively. Valid values are PIPE, DEVNULL, an existing file descriptor (a positive integer), an existing file object, and None.
So using capture_output=True is a no-go, because the output will be stored in a pipe for you to read after the call finishes.
The simpler is for you to use subprocess.Popen as #MatBBastos suggested, with wich you can communicate (repeatedly sending content to stdin and receiving content from stdout/stderr). The solution linked is a bit dated (cf its own comments; Python 2) but should work well. A related solution is this one.
To keep using subprocess.run, you will have to provide a file descriptor as stdout parameter, which I don't know how would have to redirect to a file object that does what you want : writing to the standard stream, but also keeping a copy in memory for later use.
There are docs in the io module, and a lot of questions on Stack Overflow about doing things like that, but it is notably more difficult than the other way.
I am still fairly new to the python world and know this should be an easy question to answer. I have this section of a script in python that calls a script in Perl. This Perl script is a SOAP service that fetches data from a web page. Everything works great and outputs what I want, but after a bit of trial and error I am confused to how I can capture the data with a python variable and not just output to the screen like it does now.
Any pointers appreciated!
Thank you,
Pablo
# SOAP SERVICE
# Fetch the perl script that will request the users email.
# This service will return a name, email, and certificate.
var = "soap.pl"
pipe = subprocess.Popen(["perl", "./soap.pl", var], stdin = subprocess.PIPE)
pipe.stdin.write(var)
print "\n"
pipe.stdin.close()
I am not sure what your code aims to do (with var in particular), but here are the basics.
There is the subprocess.check_output() function for this
import subprocess
out = subprocess.check_output(['ls', '-l'])
print out
If your Python is before 2.7 use Popen with the communicate() method
import subprocess
proc = subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE)
out, err = proc.communicate()
print out
You can instead iterate proc.stdout but it appears that you want all output in one variable.
In both cases you provide the program's arguments in the list.
Or add stdin if needed
proc = subprocess.Popen(['perl', 'script.pl', 'arg'],\
stdin = subprocess.PIPE,\
stdout = subprocess.PIPE)
The purpose of stdin = subprocess.PIPE is to be able to feed the STDIN of the process that is started, as it runs. Then you would do proc.stdin.write(string) and this writes to the invoked program's STDIN. That program generally waits on its STDIN and after you send a newline it gets everything written to it (since the last newline) and runs relevant processing.
If you simply need to pass parameters/arguments to the script at its invocation then that generally doesn't need nor involve its STDIN.
Since Python 3.5 the recommended method is subprocess.run(), with a very similar full signature, and similar operation, to that of the Popen constructor.
I'm writing some code which involves running a few shell commands from Python and ideally, I would like to integrate the output from these commands into the logger that I'm using. I know I can divert stdout into a file / socket as follows:
call( '<a-shell-cmd>', shell=True, stdout=myFile )
but I'd rather not have the bind of opening a temporary file, looping over the file writing the output, closing the file, deleting the file etc. If there's anyway that I can send the output directly to the logger, it would seem a lot neater to me. Any ideas?
Use the subprocess module.
Tip: you can go to the documentation for a particular version of python via http://docs.python.org/release/<major>.<minor>/
From Python 2.7 and above:
output = subprocess.check_output(["command", "arg1"], shell=True)
In Python 2.4:
process = subprocess.Popen(["command", "arg1"], shell=True, stdout=subprocess.PIPE)
stdout,stderr = process.communicate()
# not shown: how to use Popen.poll() to wait for process death.
# while filling an output buffer
print stdout
Below Python 2.4:
output = os.popen('ls')
Use os.popen
output = os.popen('ls')
You can then log output or do it directly when calling the above.
I am using Python 3.0 in Windows and trying to automate the testing of a commandline application. The user can type commands in Application Under Test and it returns the output as 2 XML packets. One is a packet and the other one is an packet. By analyzing these packets I can verifyt he result. I ahev the code as below
p = subprocess.Popen(SomeCmdAppl, stdout=subprocess.PIPE,
shell = True, stdin=subprocess.PIPE, stderr=subprocess.STDOUT)
p.stdin.write((command + '\r\n').encode())
time.sleep(2.5)
testresult = p.stdout.readline()
testresult = testresult.decode()
print(testresult)
I cannot ge any output back. It get stuck in place where I try to read the output by using readline(). I tried read() and it get stuck too
When I run the commandline application manually and type the command I get the output back correctly as tow xml packets as below
Sent: <PivotNetMessage>
<MessageId>16f8addf-d366-4031-b3d3-5593efb9f7dd</MessageId>
<ConversationId>373323be-31dd-4858-a7f9-37d97e36eb36</ConversationId>
<SageId>4e1e7c04-4cea-49b2-8af1-64d0f348e621</SagaId>
<SourcePath>C:\Python30\PyNTEST</SourcePath>
<Command>echo</Command>
<Content>Hello</Content>
<Time>7/4/2009 11:16:41 PM</Time>
<ErrorCode>0</ErrorCode>
<ErrorInfo></ErrorInfo>
</PivotNetMessagSent>
Recv: <PivotNetMessage>
<MessageId>16f8addf-d366-4031-b3d3-5593efb9f7dd</MessageId>
<ConversationId>373323be-31dd-4858-a7f9-37d97e36eb36</ConversationId>
<SageId>4e1e7c04-4cea-49b2-8af1-64d0f348e621</SagaId>
<SourcePath>C:\PivotNet\Endpoints\Pipeline\Pipeline_2.0.0.202</SourcePath>
<Command>echo</Command>
<Content>Hello</Content>
<Time>7/4/2009 11:16:41 PM</Time>
<ErrorCode>0</ErrorCode>
<ErrorInfo></ErrorInfo>
</PivotNetMessage>
But when I use the communicate() as below I get the Sent packet and never get the Recv: packet. Why am I missing the recv packet? The communicate(0 is supposed to bring everything from stdout. rt?
p = subprocess.Popen(SomeCmdAppl, stdout=subprocess.PIPE,
shell = True, stdin=subprocess.PIPE, stderr=subprocess.STDOUT)
p.stdin.write((command + '\r\n').encode())
time.sleep(2.5)
result = p.communicate()[0]
print(result)
Can anybody help me with a sample code that should work? I don't know if it is needed to read and write in separate threads. Please help me. I need to do repeated read/write. Is there any advanced level module in python i can use. I think Pexpect module doesn't work in Windows
This is a popular problem, e.g. see:
Interact with a Windows console application via Python
How do I get 'real-time' information back from a subprocess.Popen in python (2.5)
how do I read everything currently in a subprocess.stdout pipe and then return?
(Actually, you should have seen these during creation of your question...?!).
I have two things of interest:
p.stdin.write((command + '\r\n').encode()) is also buffered so your child process might not even have seen its input. You can try flushing this pipe.
In one of the other questions one suggested doing a stdout.read() on the child instead of readline(), with a suitable amount of characters to read. You might want to experiment with this.
Post your results.
Try sending your input using communicate instead of using write:
result = p.communicate((command + '\r\n').encode())[0]
Have you considered using pexpect instead of subprocess? It handles the details which are probably preventing your code from working (like flushing buffers, etc). It may not be available for Py3k yet, but it works well in 2.x.
See: http://pexpect.sourceforge.net/pexpect.html
I want to spawn (fork?) multiple Python scripts from my program (written in Python as well).
My problem is that I want to dedicate one terminal to each script, because I'll gather their output using pexpect.
I've tried using pexpect, os.execlp, and os.forkpty but neither of them do as I expect.
I want to spawn the child processes and forget about them (they will process some data, write the output to the terminal which I could read with pexpect and then exit).
Is there any library/best practice/etc. to accomplish this job?
p.s. Before you ask why I would write to STDOUT and read from it, I shall say that I don't write to STDOUT, I read the output of tshark.
See the subprocess module
The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to replace several other, older modules and functions, such as:
os.system
os.spawn*
os.popen*
popen2.*
commands.*
From Python 3.5 onwards you can do:
import subprocess
result = subprocess.run(['python', 'my_script.py', '--arg1', val1])
if result.returncode != 0:
print('script returned error')
This also automatically redirects stdout and stderr.
I don't understand why you need expect for this. tshark should send its output to stdout, and only for some strange reason would it send it to stderr.
Therefore, what you want should be:
import subprocess
fp= subprocess.Popen( ("/usr/bin/tshark", "option1", "option2"), stdout=subprocess.PIPE).stdout
# now, whenever you are ready, read stuff from fp
You want to dedicate one terminal or one python shell?
You already have some useful answers for Popen and Subprocess, you could also use pexpect if you're already planning on using it anyways.
#for multiple python shells
import pexpect
#make your commands however you want them, this is just one method
mycommand1 = "print 'hello first python shell'"
mycommand2 = "print 'this is my second shell'"
#add a "for" statement if you want
child1 = pexpect.spawn('python')
child1.sendline(mycommand1)
child2 = pexpect.spawn('python')
child2.sendline(mycommand2)
Make as many children/shells as you want and then use the child.before() or child.after() to get your responses.
Of course you would want to add definitions or classes to be sent instead of "mycommand1", but this is just a simple example.
If you wanted to make a bunch of terminals in linux, you just need to replace the 'python' in the pextpext.spawn line
Note: I haven't tested the above code. I'm just replying from past experience with pexpect.