I'm trying to run a shell command from within a python script which needs to do several things
1. The shell command is 'hspice tran.deck >! tran.lis'
2. The script should wait for the shell command to complete before proceeding
3. I need to check the return code from the command and
4. Capture STDOUT if it completed successfully else capture STDERR
I went through the subprocess module and tried out a couple of things but couldn't find a way to do all of the above.
- with subprocess.call() I could check the return code but not capture the output.
- with subprocess.check_output() I could capture the output but not the code.
- with subprocess.Popen() and Popen.communicate(), I could capture STDOUT and STDERR but not the return code.
I'm not sure how to use Popen.wait() or the returncode attribute. I also couldn't get Popen to accept '>!' or '|' as arguments.
Can someone please point me in the right direction? I'm using Python 2.7.1
EDIT: Got things working with the following code
process = subprocess.Popen('ls | tee out.txt', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = process.communicate()
if(process.returncode==0):
print out
else:
print err
Also, should I use a process.wait() after the process = line or does it wait by default?
Just use .returncode after .communicate(). Also, tell Popen that what you're trying to run is a shell command, rather than a raw command line:
p = subprocess.Popen('ls | tee out.txt', shell=True, ...)
p.communicate()
print p.returncode
From the docs:
Popen.returncode
The child return code, set by poll() and wait() (and indirectly by communicate()). A None value indicates that the process hasn’t terminated yet.
A negative value -N indicates that the child was terminated by signal N (Unix only).
Here is example how to interact with shell:
>>> process = subprocess.Popen(['/bin/bash'], shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
>>> process.stdin.write('echo it works!\n')
>>> process.stdout.readline()
'it works!\n'
>>> process.stdin.write('date\n')
>>> process.stdout.readline()
'wto, 13 mar 2012, 17:25:35 CET\n'
>>>
Related
I'm trying to issue a command over ssh and get its return code via subprocess. I have some code that looks like this:
cmd = 'ssh user#ip_addr "some_command"'
res = subprocess.check_output(
cmd,
shell=True,
stdout=subprocess.PIPE)
Now this works great if the cmd only generates an exit code (for instance, setting cmd to "exit 1", and then doing a try/catch to see if it exits with a non-zero. However, the following hangs indefinitely:
cmd = 'ssh user#ip_addr "ls -la && exit 0;"'
res = subprocess.check_output(
cmd,
shell=True,
stdout=subprocess.PIPE)
I saw two questions that looked similar, and I did RTFM, but I'm still not sure what to do. I don't really care whether or not the command generates output; I'm more concerned with the exit code. If anyone knows what the best way of doing this is, or whether or not I'm using subprocess inappropriately, I would appreciate it.
Remove the stdout=subprocess.PIPE, and it should work; check_output itself captures the output, so redirecting it using stdout=subprocess.PIPE will cause problems. If you don't care about the output at all, just use subprocess.check_call (and again, don't use stdout=subprocess.PIPE).
DO NOT USE std{out,err}=PIPE UNLESS YOU READ FROM THE PIPE!!!
To get the return code while discarding the output of a command issued over ssh using subprocess module:
from subprocess import call, DEVNULL, STDOUT
returncode = call(['ssh', 'user#ip', 'ls -la && exit 0;'],
stdin=DEVNULL, stdout=DEVNULL, stderr=STDOUT)
See also, How to hide output of subprocess in Python 2.7.
Note: shell=True is not used.
When i am running a program in the console, i get some text output.
When i am running the same program in Popen(..), with the same parameters, stdout and stderr are empty.
I tried everything i could imagine like shell=False and shell=True, set stdout=subprocess.PIPE, did a os.chdir() to change into the directory of this program, try p.wait() and p.communicate(), set the command as a list and as a string, but nothing works.
example:
p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
out, err = p.communicate()
--> out and err are empty strings, but if i ran this command in console i get a real output. Command is with fullpath, so its regardless where the command will be started.
My question is, are there mechanisms for programms to detect they weren't run in a real console? If so, how can i cheat.
Or miss i something?
(Python 2.7.8. x32 in Win7 x64)
from subprocess import Popen, STDOUT, PIPE
p = Popen(command, shell=True, stdout=PIPE, stderr=STDOUT, stdin=PIPE)
while p.poll() is None:
print(p.stdout.read())
p.stdout.close()
p.stdin.close()
Try this and see if it makes any difference. Also make sure command is a string and not a list/touple, shell=True for whatever reason works better or only with strings.
Also note that shell=True will get you hanged because it's insecure etc.
Also skipping .communicate() you'll need to tap off stdout otherwise the buffer will get full and you might hang both yours and the child process.
If this doesn't work, please provide more information. Such as the command used and the expected output (at least first few lines)
I am writing a python script to ssh into a linux server and execute a shell script that is already stored on the linux server.
Here is what my code look like so far
command = ['ssh into the remote server',
'cd into the directory of the shell script,
'./running the shell script',
]
process = subprocess.Popen(command,
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
err, out = process.communicate()
if out:
print "standard output of subprocess is : "
print out
if err:
print "standard error of subprocess is : "
print err
print "returncode of subprocess: "
print process.returncode
1st question: I can obtain the output of my shell scripts through stderr, but I only obtain it after the entire shell script has finished executing. So if the shell script takes 10 minutes to finish, I only get to see the output of the shell script after 10 minutes.
I want to have the output of my shell scripts return line by line to me just as if I was executing the script manually in the remote server. Can this be done?
2nd question: as you can see, I have three commands in my command list (which is only a small portion of all my commands,) if I put all my commands in the list, I only obtain the output of ALL my commands through stdout ONLY when all my commands has finished executing. If my 1st question cannot be done, is there a way to at least obtain the output of each command after each one has been executed instead of receiving them all at once only when all the commands has finished being executed.
To see the output immediately, don't redirect it:
from subprocess import Popen, PIPE
p = Popen(['ssh', 'user#hostname'], stdin=PIPE)
p.communicate(b"""cd ..
echo 1st command
echo 2nd command
echo ...
""")
If you want both to capture the "live" output in a variable and to display it in the terminal then the solution depends on whether you need to handle stdin/stdout/stderr concurrently.
If input is small and you want to combine stdout/stderr then you could pass all commands at once and read the merged output line-by-line:
from subprocess import Popen, PIPE, STDOUT
p = Popen(['ssh', 'user#hostname'], stdin=PIPE,
stdout=PIPE, stderr=STDOUT, bufsize=1)
p.stdin.write(b"""cd ..
echo 1st command
echo 2nd command
echo ...
""")
p.stdin.close() # no more input
lines = [] # store output here
for line in iter(p.stdout.readline, b''): # newline=b'\n'
lines.append(line) # capture for later
print line, # display now
p.stdout.close()
p.wait()
If you want to capture "live" stdout/stderr separately, see:
Displaying subprocess output to stdout and redirecting it
Subprocess.Popen: cloning stdout and stderr both to terminal and variables
I'm not entirely sure, but maybe you get instant output if you pass the other two commands as arguments to ssh:
command = 'ssh user#example.com \'cd some/path/on/your/server; ./run-the-script.sh\''
The way I understand it, Python first reads and processes all the input and only then returns output. I'm not too familiar with Python, so I might be wrong on this, but if I'm right, this should help.
Don't call .communicate() -- that waits for the process to finish.
Instead, keep reading data from .stdout pipe.
Simple example:
In [1]: import subprocess
In [2]: p = subprocess.Popen(["find", "/"], stdout=subprocess.PIPE)
In [3]: p.stdout
Out[3]: <open file '<fdopen>', mode 'rb' at 0x7f590446dc00>
In [4]: p.stdout.readline()
Out[4]: '/\n'
In [5]: p.stdout.readline()
Out[5]: '/var\n'
In [6]: p.stdout.readline()
Out[6]: '/var/games\n'
Submitting a complex cmd string made of a full file path to an executable, the multiple flags, arguments, parameters, inputs and outputs seems to require me to set shell=True otherwise subprocess.Popen is not able understand anything more complex than just a simple path to executable (with no spaces in a filepath).
In my example I have quite a long cmd:
cmd = " '/Application/MyApp.app/Contents/MacOS/my_executable' '/Path/to/input/files' -some -flags -here -could -be -a -lot '/full/path/to/output/files' "
Submitting this cmd to subprocess.Popen " results to an error that complains on something about the path and not being able to find it.
So instead of using :
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
check_call seems workings quite well:
proc = subprocess.check_call(cmd, shell=True)
Interesting, only after shell is set to True
shell=True
the subprocess.check_call works with a supplied cmd.
The side effect is that the rest of the code seems proceeds running without waiting for subprocess.check_call(cmd, shell=True) to finish first.
The code is designed the way that the rest of the execution is dependent on a result of subprocess.check_call(cmd, shell=True).
I wonder if there is anyway to enforce the code execution to wait till subprocess.check_call(cmd, shell=True) is finished. Thanks in advance!
As #mikkas suggest just use it as a list here is a working example:
mainProcess = subprocess.Popen(['python', pyfile, param1, param2], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# get the return value from the method
communicateRes = mainProcess.communicate()
stdOutValue, stdErrValue = communicateRes
You are calling python.exe pyfile param1 param2
By using communicate() you can get the stdout and stderr as a Tuple
You can use python method split() to split your string to a list for example:
cmd = "python.exe myfile.py arg1 arg2"
cmd.split(" ")
Output:
['python.exe', 'myfile.py', 'arg1', 'arg2']
I think the check_call function should wait for the command to finish.
See the docs here
http://docs.python.org/2/library/subprocess.html
Check call does not wait. You need to do a process.wait() and check the return code explicitly to get the functionaly you want.
Process = subprocess.Popen('%s' %command_string,stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
Process.wait()
if Process1.returncode!=0:
print Process1.returncode
sendMail()
return
else:
sendMail()
is there a way to use python2.6 with either subprocess.Popen() or os.system() to run two tasks? Example the script will run "airodump-ng" first then this process is sub and is hidden(meaning will not print out from terminal) after which continue run the rest of the script which contain "sniff" function of scapy. I been researched but I only found windows version and python3. By the way I running on debian.
Use subprocess.Popen in combination with subprocess.PIPE:
p = Popen(['airodump-ng', …], stdin=PIPE, stdout=PIPE, stderr=PIPE)
If you want to wait until the process has finished use:
stdout, stderr = p.communicate()
If you omit the code above airodump-ng will run in the background and produce no visible output, while you can continue with your python code.
Another method would be to use os.devnull to redirect the output of airodump-ng to, this will completly get rid of any output produced:
devnull = os.open(os.devnull, os.O_WRONLY)
p = Popen(['airodump-n', …], stdout=devnull, stderr=devnull)
In the spot where you put the command airodump-ng replace that part with timeout 'X's airodump-ng mon'X'