Develop Reference

Python subprocess.check_output(args) fails, while args executed via Windows command line work OK

Some problems with python subprocess.check_output.
output = subprocess.check_output(args)
where my args is:
args = "C:\\DO\\bin\\Config.exe --ChCfg7 --LFE -b1152000 C:\\DO\\PCM\\1.wav C:\\DO\\PCM\\2.wav C:\\DO\\PCM\\3.wav C:\\DO\\PCM\\4.wav C:\\DO\\PCM\\5.wav C:\\DO\\PCM\6.wav --ModeBCast -oC:\\DO\\OUT\\outfile > C:\\DO\\OUT\\log.txt
This works when executed from standard windows command line, but doesn't work when executed via Python subprocess.check_output. In win cmd case there is output file produced and log.txt too, and python script produces out file with size 0, and no log.txt at all.

output = subprocess.check_output(args,shell=True)
Run this with shell=True

Use a list of args and redirect the output to a file:
import subprocess
args = ['C:/DO/bin/Config.exe', '--ChCfg7', '--LFE', '-b1152000', 'C:/DO/PCM/1.wav', 'C:/DO/PCM/2.wav', 'C:/DO/PCM/3.wav', 'C:/DO/PCM/4.wav', 'C:/DO/PCM/5.wav', 'C:/DO/PCM/6.wav', '--ModeBCast', '-oC:/DO/OUT/outfile']
with open("C:/DO/OUT/log.txt", "w") as f:
subprocess.check_call(args, stdout=f)
You can use shell=Truebut for security reasons generally it is not a very good idea and the same can be quite easily achieved using the code above and simply redirecting the output to the file.

> is a shell redirection operator. Either run the command in a shell or (better) as #Padraic Cunningham suggested emulate it in Python:
#!/usr/bin/env python
import subprocess
args = r"C:\DO\bin\Config.exe --ChCfg7 --LFE -b1152000".split()
args += [r'C:\DO\PCM\%d.wav' % i for i in range(1, 7)]
args += ["--ModeBCast", r"-oC:\DO\OUT\outfile"]
with open(r"C:\DO\OUT\log.txt", "wb", 0) as output_file:
subprocess.check_call(args, stdout=output_file)
The code uses raw string literals for Windows paths to avoid escaping backslashes.
There is usually no point to use shell=True on Windows unless you want to run a built-in command such as dir. If args is not constructed using input from an external source then security considerations do not apply. shell=True starts additional process (%COMSPEC%) and it changes how the executable is searched and it changes what characters should be escaped (what characters are metacharacters) — do not use shell=True unless necessary.

Python: Executing a shell command

I need to do this:
paste file1 file2 file3 > result
I have the following in my python script:
from subprocess import call
// other code here.
// Here is how I call the shell command
call ["paste", "file1", "file2", "file3", ">", "result"])
Unfortunately I get this error:
paste: >: No such file or directory.
Any help with this will be great!

You need to implement the redirection yourself, if you're wisely deciding not to use a shell.
The docs at https://docs.python.org/2/library/subprocess.html warn you not to use a pipe -- but, you don't need to:
import subprocess
with open('result', 'w') as out:
subprocess.call(["paste", "file1", "file2", "file3"], stdout=out)
should be just fine.

There are two approaches to this.
Use shell=True:
call("paste file1 file2 file3 >result", shell=True)
Redirection, >, is a shell feature. Consequently, you can only access it when using a shell: shell=True.
Keep shell=False and use python to perform the redirection:
with open('results', 'w') as f:
subprocess.call(["paste", "file1", "file2", "file3"], stdout=f)
The second is normally preferred as it avoids the vagaries of the shell.
Discussion
When the shell is not used, > is just another character on the command line. Thus, consider the error message:
paste: >: No such file or directory.
This indicates that paste had received > as an argument and was trying to open a file by that name. No such file exists. Therefore the message.
As the shell command line, one can create a file by that name:
touch '>'
If such a file had existed, paste, when called by subprocess with shell=False, would have used that file for input.

If you don't mind adding an additional dependency in your code base you might consider installing the sh Python module (from PyPI:sh using pip, of course).
This is a rather clever wrapper around Python's subprocess module's functionality. Using sh your code would look something like:
#!/usr/bin/python
from sh import paste
paste('file1', 'file2', 'file3', _out='result')
... although I think you'd want some exception handling around that so you could use something like:
#!/usr/bin/python
from __future__ import print_function
import sys
from sh import paste
from tempfile import TemporaryFile
with tempfile.TemporaryFile() as err:
try:
paste('file1', 'file2', 'file3', _out='result', _err=err)
except (EnvironmentError, sh.ErrorReturnCode) as e:
err.seek(0)
print("Caught Error: %s" % err.read(), file=sys.stderr)
sh makes such things almost trivially easy although there are some tricks as you get more advanced. You also have to note the difference between _out= and other keyword arguments of that form, vs. sh's magic for most other keyword arguments.
All that sh magic make confuse anyone else who ever reads your code. You might also find that using Python modules with sh code interlaced into it makes you complacent about portability issues. Python code is generally fairly portable while Unix command line utilities can vary considerably from one OS to another and even from one Linux distribution or version to another. Having lots of shell utilities interlaced with your Python code in such a transparent way may make that problem less visible.

Python stdin filename

I'm trying to get the filename thats given in the command line. For example:
python3 ritwc.py < DarkAndStormyNight.txt
I'm trying to get DarkAndStormyNight.txt
When I try fileinput.filename() I get back same with sys.stdin. Is this possible? I'm not looking for sys.argv[0] which returns the current script name.
Thanks!

In general it is not possible to obtain the filename in a platform-agnostic way. The other answers cover sensible alternatives like passing the name on the command-line.
On Linux, and some related systems, you can obtain the name of the file through the following trick:
import os
print(os.readlink('/proc/self/fd/0'))
/proc/ is a special filesystem on Linux that gives information about processes on the machine. self means the current running process (the one that opens the file). fd is a directory containing symbolic links for each open file descriptor in the process. 0 is the file descriptor number for stdin.

You can use ArgumentParser, which automattically gives you interface with commandline arguments, and even provides help, etc
from argparse import ArgumentParser
parser = ArgumentParser()
parser.add_argument('fname', metavar='FILE', help='file to process')
args = parser.parse_args()
with open(args.fname) as f:
#do stuff with f
Now you call python2 ritwc.py DarkAndStormyNight.txt. If you call python3 ritwc.py with no argument, it'll give an error saying it expected argument for FILE. You can also now call python3 ritwc.py -h and it will explain that a file to process is required.
PS here's a great intro in how to use it: http://docs.python.org/3.3/howto/argparse.html

In fact, as it seams that python cannot see that filename when the stdin is redirected from the console, you have an alternative:
Call your program like this:
python3 ritwc.py -i your_file.txt
and then add the following code to redirect the stdin from inside python, so that you have access to the filename through the variable "filename_in":
import sys
flag=0
for arg in sys.argv:
if flag:
filename_in = arg
break
if arg=="-i":
flag=1
sys.stdin = open(filename_in, 'r')
#the rest of your code...
If now you use the command:
print(sys.stdin.name)
you get your filename; however, when you do the same print command after redirecting stdin from the console you would got the result: <stdin>, which shall be an evidence that python can't see the filename in that way.

I don't think it's possible. As far as your python script is concerned it's writing to stdout. The fact that you are capturing what is written to stdout and writing it to file in your shell has nothing to do with the python script.

Calling a subprocess with mixed data type arguments in Python

I am a bit confused as to how to get this done.
What I need to do is call an external command, from within a Python script, that takes as input several arguments, and a file name.
Let's call the executable that I am calling "prog", the input file "file", so the command line (in Bash terminal) looks like this:
$ prog --{arg1} {arg2} < {file}
In the above {arg1} is a string, and {arg2} is an integer.
If I use the following:
#!/usr/bin/python
import subprocess as sbp
sbp.call(["prog","--{arg1}","{arg2}","<","{file}"])
The result is an error output from "prog", where it claims that the input is missing {arg2}
The following produces an interesting error:
#!/usr/bin/python
import subprocess as sbp
sbp.call(["prog","--{arg1} {arg2} < {file}"])
all the spaces seem to have been removed from the second string, and equal sign appended at the very end:
command not found --{arg1}{arg2}<{file}=
None of this behavior seems to make any sense to me, and there isn't much that one can go by from the Python man pages found online. Please note that replacing sbp.call with sbp.Popen does not fix the problem.

The issue is that < {file} isn’t actually an argument to the program, but is syntax for the shell to set up redirection. You can tell Python to use the shell, or you can setup the redirection yourself.
from subprocess import *
# have shell interpret redirection
check_call('wc -l < /etc/hosts', shell=True)
# set up redirection in Python
with open('/etc/hosts', 'r') as f:
check_call(['wc', '-l'], stdin=f.fileno())
The advantage of the first method is that it’s faster and easier to type. There are a lot of disadvantages, though: it’s potentially slower since you’re launching a shell; it’s potentially non-portable because it depends on the operating system shell’s syntax; and it can easily break when there are spaces or other special characters in filenames.
So the second method is preferred.

Problems with command using * wildcard in subprocess

I'm trying to copy files from one location to another using subprocess library and Popen method. When runing following script I'm getting the error cp: cannot stat /some/dev_path/*. I was told that the * is not expanded to the file names and that's where the problem is. Also in some other posts people were suggesting to use call instead of Popen, but call will not return stderr as far as I know.
devPath = '/some/dev_path/'
productionPath = '/some/prod_path/'
p = subprocess.Popen(['cp', '-r', devPath + '*', productionPath], stdout = subprocess.PIPE, stderr = subprocess.PIPE)
pout, perr = p.communicate()
if perr != '':
sys.exit('Error: ' + perr)

Expanding the * (globbing) is a function of your shell, bash for example. Therefore you'd have to use the keyword argument shell=True in your subprocess.Popen call.
However, for this case I'd strongly suggest to use shutil.copytree instead.
(First of all, because it's much simpler (see Zen of Python) and less error-prone. Dealing with errors is much cleaner, you get nice exceptions including a list of errors (for multi file operations like yours), and you don't have to deal with spawning a subprocess and communicating with it. Second, it's an unnecessary waste of resources to fork a child process if you don't need to. Other issues include quoting / escaping and possibly introducing security vulnerabilities into your code if you fail to properly sanitize user input.)
For example:
from shutil import copytree
from shutil import Error
try:
copytree('dir_a', 'dir_b')
except (Error, OSError), e:
print "Attempt to copy failed: %s" % e
Also, you shouldn't build filesystem paths by concatenating strings together, but instead use os.path.join(). That will use the correct directory separator (os.sep) for the current OS and allow you to easily write portable code.
Example:
>>> import os
>>> os.path.join('/usr/lib', 'python2.7')
'/usr/lib/python2.7'
Note: os.path.join still only does (smart) string manipulation - it doesn't care if that path is accessible or even exists.