Develop Reference

Python, subprocess.check_call() and pipes redirection

Why am I getting list of files when executing this command?
subprocess.check_call("time ls &>/dev/null", shell=True)
If I will paste
time ls &>/dev/null
into the console, I will just get the timings.
OS is Linux Ubuntu.

On debian-like systems, the default shell is dash, not bash. Dash does not support the &> shortcut. To get only the subprocess return code, try:
subprocess.check_call("time ls >/dev/null 2>&1", shell=True)
To get subprocess return code and the timing information but not the directory listing, use:
subprocess.check_call("time ls >/dev/null", shell=True)
Minus, of course, the subprocess return code, this is the same behavior that you would see on the dash command prompt.

The Python version is running under sh, but the console version is running in whatever your default shell is, which is probably either bash or dash. (Your sh may actually be a different shell running in POSIX-compliant mode, but that doesn't make any difference.)
Both bash and dash have builtin time functions, but sh doesn't, so you get /usr/bin/time, which is a normal program. The most important difference this makes is that the time builtin is not running as a subprocess with its own independent stdout and stderr.
Also, sh, bash, and dash all have different redirection syntax.
But what you're trying to do seems wrong in the first place, and you're just getting lucky on the console because two mistakes are canceling out.
You want to get rid of the stdout of ls but keep the stderr of time, but that's not what you asked for. You're trying to redirect both stdout and stderr: that's what >& means on any shell that actually supports it.
So why are you still getting the time stderr? Either (a) your default shell doesn't support >&, or (b) you're using the builtin instead of the program, and you're not redirecting the stderr of the shell itself, or maybe (c) both of the above.
If you really want to do exactly the same thing in Python, with the exact same bugs canceling out in the exact same way, you can run your default shell manually instead of using shell=True. Depending on which reason it was working, that would be either this:
subprocess.check_call([os.environ['SHELL'], '-c', 'time ls &> /dev/null'])
or this:
subprocess.check_call('{} -c time ls &> /dev/null'.format(os.environ(SHELL), shell=True)
But really, why are you doing this at all? If you want to redirect stdout and not stderr, write that:
subprocess.check_call('time ls > /dev/null', shell=True)
Or, better yet, why are you even using the shell in the first place?
subprocess.check_call(['time', 'ls'], stdout=subprocess.devnull)

Struggling with Python's subprocess.Popen running a Perl script

I have a Perl script that I need to run with Python, and I've been trying to use subprocess to do it, unsuccessfully. I'm able to get the command to run just fine on the command line, but subprocess isn't able to get it to work.
If I have a Perl script like this:
#!/usr/bin/perl
use strict;
use warnings;
my $name = shift;
print "Hello $name!\n";
I am able to successfully run the command on the command line like so
C:\current\path> perl test.pl world
>>>Hello world!
But when I try and invoke the same command with subprocess, I get this error
cmd = 'perl test.pl world'
pipe = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE)
>>>"'perl' is not recognized as an internal or external command, operable program or batch file."
I've tried different combinations of creating cmd as a list
cmd = ['perl','test.pl','world']
and of giving subprocess the absolute path to Perl
cmd = ['C:\path\to\perl.exe','test.pl','world']
but nothing gets it to work. I'm able to get subprocess to play well without Perl
pipe = subprocess.Popen('dir',shell=True,stdout=subprocess.PIPE)
pipe.stdout.read()
>>>Volume in drive C is Windows7_OS....
I'm absolutely sure that Perl is in my PATH, because like I said I can invoke the command just fine in the command line. I've looked around at different posts that suggested to check os.environ["COMSPEC"], adding Perl to the subprocess env, and nothing has worked.
EDIT: I also can't get the command to work with any other subprocess methods: check_output() returns an empty byte string, and call() returns 1.
Any sort of fix or alternative solution would be immensely appreciated. I'm running Python 3.4 on Windows 7 64bit. I've tried 32- and 64-bit Python as well, to no avail.
UPDATE:
I've been able to get this to work on Mac, but with one difference: I can't have shell=True. Other than that, any subprocess function I want will work, including Popen, check_output, and call.
So I guess this is a Windows problem more than anything. I've tried not setting shell=True on my Windows machine, but I always get this error:
WindowsError: [Error 193] %1 is not a valid Win32 application
UPDATE 2:
I've also tried creating a .bat file that runs the command, but I get the same error as well I try to just call Perl.
pipe = subprocess.Popen('test.bat',shell=True,stdout=subprocess.PIPE)
>>>"'test.bat' is not recognized as an internal or external command, operable program or batch file."
Where test.bat has only one line:
echo Hello, World

you could try using os.system('command') instead. Apparently its not well regarded for some reason, or at least not as well regarded as subprocess though. Alternately, you could try some of the other subprocess methods, such as call, check_call, or check_output. I've had a similar problem with Popen in the past and switching to one of these methods helped.

I got the same error while trying to run a perl script from Python.
I managed to get it work by adding the full path of the perl.exe (with an extra escape character, as '\') to the command I was trying to run as below:
pipe = subprocess.Popen('C:\\Perl64\\bin\\perl myscrip.pl',shell=True,stdout=subprocess.PIPE)
pipe.stdout.read()

This works fine for me. I know it's been a while and things may have just fixed themselves.
I'm using activestate perl 5.8.9
and python 3.9.5 on Windows 10.
from subprocess import Popen, PIPE
proc = Popen(['perl', 'test.pl', 'world'], stdout=PIPE, stderr=PIPE)
stdout, stderr = proc.communicate()
if proc.returncode == 0:
print(stdout.decode())
else:
print(stderr.decode())

Why Popen need pressing Enter to return back to shell

I want to execute shell commands with bash in my Python script, getting real-time printing message in the screen. I use the following line to do this:
subprocess.Popen(my_commands, shell=True, stdout=sys.stdout, stderr=sys.stderr, executable='/bin/bash')
Everything looks good expect that after the shell commands are finished, the input cursor is still invisible. I have to press the Enter key to active the shell again. So what is the error?

It is because you have two shells, shell=True and /bin/bash
If you set shell=False you will not see the output on the console. You will have to use PIPE and/or subprocess.communicate() to get the output (depending on what you want).

Executing an xmgrace batch with subprocess.Popen()

I have to make graphs from several files with data. I already found a way to run a simple command
xmgrace -batch batch.bfile -nosafe -hardcopy
in which batch.bfile is a text file with grace commands to print the graph I want. I already tried it manually and it works perfectly. To do this with several files I just have to edit one parameter inside batch.bfile and run the same command every time I make a change.
I have already written a python code which edits batch.bfile and goes through all the data files with a for cycle. In each cycle step I want to run the mentioned command directly in the command line.
After searching a bit I found two solutions, one with os.system() and another with subprocess.Popen() and I could only make subprocess.Popen() work without giving any errors by writing:
subprocess.Popen("xmgrace -batch batch.bfile -nosafe -hardcopy", shell=True)
Problem is, this doesn't do anything in practice, i.e., it just isn't the same as running the command directly in the command line. I already tried writing the full directory for the batch.bfile but nothing changed.
I am using Python 2.7 and Mac OS 10.7

Have you checked running xmgrace from the command line using sh? (i.e. invoke /bin/sh, then run xmgrace... which should be the same shell that Popen is using when you set shell=true).
Another solution would be to create a shell script (create a file like myscript.sh, and run chmod +x from the terminal). In the script call xmgrace:
#!/bin/bash
xmgrace -batch batch.bfile -nosafe -hardcopy
You could then test that myscript.sh works, which ought to pick up any environment variables that might be in your profile that might differ from python. If this works, you could call the script from python's subprocess.Popen('myscript.sh'). You can check what the environment variables are set in python for subprocess by running:
import os
os.environ

You may want to check out http://sourceforge.net/projects/graceplot/

When use use Popen, you can capture the application's output to stdout to stderr and print it within your application - this way you can see what is happening:
from subprocess import Popen, PIPE
ps = Popen(reportParameters,bufsize=512, stdout = PIPE, stderr = PIPE)
if ps:
while 1:
stdout = ps.stdout.readline()
stderr = ps.stderr.readline()
exitcode = ps.poll()
if (not stdout and not stderr) and (exitcode is not None):
break
if stdout:
stdout = stdout[:-1]
print stdout
if stderr:
stderr = stderr[:-1]
print stderr

Can i control PSFTP from a Python script?

i want to run and control PSFTP from a Python script in order to get log files from a UNIX box onto my Windows machine.
I can start up PSFTP and log in but when i try to run a command remotely such as 'cd' it isn't recognised by PSFTP and is just run in the terminal when i close PSFTP.
The code which i am trying to run is as follows:
import os
os.system("<directory> -l <username> -pw <password>" )
os.system("cd <anotherDirectory>")
i was just wondering if this is actually possible. Or if there is a better way to do this in Python.
Thanks.

You'll need to run PSFTP as a subprocess and speak directly with the process. os.system spawns a separate subshell each time it's invoked so it doesn't work like typing commands sequentially into a command prompt window. Take a look at the documentation for the standard Python subprocess module. You should be able to accomplish your goal from there. Alternatively, there are a few Python SSH packages available such as paramiko and Twisted. If you're already happy with PSFTP, I'd definitely stick with trying to make it work first though.
Subprocess module hint:
# The following line spawns the psftp process and binds its standard input
# to p.stdin and its standard output to p.stdout
p = subprocess.Popen('psftp -l testuser -pw testpass'.split(),
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# Send the 'cd some_directory' command to the process as if a user were
# typing it at the command line
p.stdin.write('cd some_directory\n')

This has sort of been answered in: SFTP in Python? (platform independent)
http://www.lag.net/paramiko/
The advantage to the pure python approach is that you don't always need psftp installed.