Develop Reference

Python subprocess module: what the code does?

What this piece of code is doing?
with open(temp_path) as f:
command = "xdg-open"
subprocess.Popen(
["im=$(cat);" + command + " $im; rm -f $im"], shell=True, stdin=f
)
I'm confused with the subprocess part...
What the shell script does?

im=$(cat)
uses cat to read the standard input, and assigns the result to the variable im. Since you use stdin=f, that reads the contents of temp_path.
command + " $im;`
executes the command xdg-open with $im as its argument. So this uses the contents of the file as the argument to xdg-open, which opens the file in its default application. Since the argument should be a filename, this implies that temp_path contains a filename.
rm -f $im
deletes the file that was opened.
This seems like a silly way to do this. A better way to write it would be:
with open(temp_path) as f:
filename = f.read().strip()
command = "xdg-open"
subprocess.Popen([command, filename])
os.remove(filename)
Although I haven't seen the rest of the script, I suspect the temp path is also unnecessary when doing it this way -- it seems like it was just using that as a way to get the filename into the shell variable.

How to get data from web in python using curl?

In bash when I used
myscript.sh
file="/tmp/vipin/kk.txt"
curl -L "myabcurlx=10&id-11.com" > $file
cat $file
./myscript.sh gives me below output
1,2,33abc
2,54fdd,fddg3
3,fffff,gfr54
When I tried to fetch it using python and tried below code -
mypython.py
command = curl + ' -L ' + 'myabcurlx=10&id-11.com'
output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE).stdout.read().decode('ascii')
print(output)
python mypython.py throw error, Can you please point out what is wrong with my code.
Error :
/bin/sh: line 1: &id=11: command not found
Wrong Parameter

command = curl + ' -L ' + 'myabcurlx=10&id-11.com'
Print out what this string is, or just think about it. Assuming that curl is the string 'curl' or '/usr/bin/curl' or something, you get:
curl -L myabcurlx=10&id-11.com
That’s obviously not the same thing you typed at the shell. Most importantly, that last argument is not quoted, and it has a & in the middle of it, which means that what you’re actually asking it to do is to run curl in the background and then run some other program that doesn’t exist, as if you’d done this:
curl -L myabcurlx=10 &
id-11.com
Obviously you could manually include quotes in the string:
command = curl + ' -L ' + '"myabcurlx=10&id-11.com"'
… but that won’t work if the string is, say, a variable rather than a literal in your source—especially if that variable might have quote characters within it.
The shlex module has helpers to quoting things properly.
But the easiest thing to do is just not try to build a command line in the first place. You aren’t using any shell features here, so why add the extra headaches, performance costs, problems with the shell getting in the way of your output and retcode, and possible security issues for no benefit?
Make the arguments a list rather than a string:
command = [curl, '-L', 'myabcurlx=10&id-11.com']
… and leave off the shell=True
And it just works. No need to get spaces and quotes and escapes right.
Well, it still won’t work, because Popen doesn’t return output, it’s a constructor for a Popen object. But that’s a whole separate problem—which should be easy to solve if you read the docs.
But for this case, an even better solution is to use the Python bindings to libcurl instead of calling the command-line tool. Or, even better, since you’re not using any of the complicated features of curl in the first place, just use requests to make the same request. Either way, you get a response object as a Python object with useful attributes like text and headers and request.headers that you can’t get from a command line tool except by parsing its output as a giant string.

import subprocess
fileName="/tmp/vipin/kk.txt"
with open(fileName,"w") as f:
subprocess.read(["curl","-L","myabcurlx=10&id-11.com"],stdout=f)
print(fileName)
recommended approaches:
https://docs.python.org/3.7/library/urllib.request.html#examples
http://docs.python-requests.org/en/master/user/install/

Quotes without backslash in subprocess command line in python

I'm trying to use ffmpeg from python. The command I need to execute is:
ffmpeg -i test_file-1kB.mp4 -i test_file.mp4 -filter_complex psnr="stats_file=test_file.mp4-1kB.psnr" -f null -
However, my output that is getting passed to subprocess looks like it is escaping the double quotes with backslashes like so:
In[1]: print(subprocess.list2cmdline(psnr_args))
ffmpeg -i test_file-1kB.mp4 -i test_file.mp4 -filter_complex psnr=\"stats_file=test_file.mp4-1kB.psnr\" -f null -
To use subprocess, I build my command line arguments one at a time into a list and then pass the list to subprocess.
psnr_args = []
psnr_args.append("ffmpeg")
#add first input, the encoded video
psnr_args.append("-i")
psnr_args.append(full_output_file_name)
#add second input, the original video
psnr_args.append("-i")
psnr_args.append(video_file)
#Setup the psnr log file
psnr_args.append("-filter_complex")
psnr_args.append('psnr="stats_file=%s.psnr"' % vstats_abs_filename )
#Output the video to null
psnr_args.append("-f")
psnr_args.append("null")
psnr_args.append("-")
print(subprocess.list2cmdline(psnr_args))
run_info_psnr = subprocess.run(psnr_args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

After more fiddling, I found a solution that works in this case but may not work in all cases. If I use double quotes as the outer quotes and the single quotes as the inner quotes, the output to subprocess uses a single quote at the same location with no backslash. This is acceptable for ffmpeg. However, for others where double quotes are the only solution, it won't be a fix.
psnr_args.append("psnr='stats_file=%s.psnr'" % vstats_abs_filename )
Output to subprocess looks like this:
In[1]: print(subprocess.list2cmdline(psnr_args))
ffmpeg -i test_file-1kB.mp4 -i test_file.mp4 -filter_complex psnr='stats_file=test_file.mp4-1kB.psnr' -f null -

In shell, the argument:
psnr="stats_file=test_file.mp4-1kB.psnr"
Is absolutely identical to:
psnr=stats_file=test_file.mp4-1kB.psnr
The quotes are removed during the shell's own processing. They are not part of the command passed to ffmpeg, which doesn't expect or understand them. Because you're directly telling the Python subprocess module to invoke a literal argument vector, there's no shell involved, so shell syntax shouldn't be present.

This has something to do with ffmpeg AV filter chain syntax too. You need to run the command like xxxx -filter_complex "psnr='stats.txt'" xxxx. To get this, you should ensure the double quote that encapsulate the filter chain reaches inside. subproces expects a flat list as the first argument, where the command is the first entry. So ['ffmpeg', '-i', "t1.mp4", "-filter_compelx", '"psnr=\'stats.txt\'"', .... and so on ].

python subprocess module can't parse filename with special characters "("

I have a Python program that reads files and then tars them into tar balls of a certain size.
One of my files not only has spaces in it but also contains parentheses. I have the following code:
cmd = "/bin/tar -cvf " + tmpname + " '" + filename + "'"
NOTE: Those are single quotes inside double quotes outside of the filename variable. It's a little difficult to see.
Where tmpname and filename are variables in a for-loop that are subject to change each iteration (irrelevant).
As you can see the filename I'm tarballing contains single quotes around the file name so that the shell (bash) interprets it literally as is and doesn't try to do variable substitution which "" will do or program execution which ` will do.
As far as I can see, the cmd variable contains the exact syntax for the shell to interpret the command as I want it to. However when I run the following subprocess command substituting the cmd variable:
cmdobj = call(cmd, shell=True)
I get the following output/error:
/bin/tar: 237-r Property Transport Request (PTR) 012314.pdf: Cannot stat: No such file or directory
/bin/tar: Exiting with failure status due to previous errors
unable to tar: 237-r Property Transport Request (PTR) 012314.pdf
I even print the command out to the console before running the subprocess command to see what it will look when running in the shell and it's:
cmd: /bin/tar -cvf tempname0.tar '237-r Property Transport Request (PTR) 012314.pdf'
When I run the above command in the shell as is it works just fine. Not really sure what's going on here. Help please!

Pass a list of args without shell=True and the full path to the file if running from a different directory:
from subprocess import check_call
check_call(["tar","-cvf",tmpname ,"Property Transport Request (PTR) 012314.pdf"])
Also use tar not 'bin/tar'. check_call will raise a CalledProcessError if the command returns a non-zero exit status.

The call method that is part of the subprocess module should have an array of strings passed.
On the command line you would call
tar -cvf "file folder with space/"
The following is equivalent in python
call(["tar", "-cvf", "file folder with space/"])
You are making this call in the shell
"tar -cvf 'file folder with space/'"
Which causes the shell to look for a program with the exact name as `tar -cvf 'file folder with space/'
This avoids string concatenation, which makes for cleaner code.

Using Tshark in Python Subprocess is giving syntax error

I am trying to develop a script to read pcap file and extract some field from it but using tshark as a subprocess. However i am getting syntax error regarding cmd. Can anyone help me out on this?
def srcDestDport (filename):
cmd = r"tshark -o column.format:"Source","%s", "Destination","%d", "dstport"," %uD"' -r %s"%(filename)
subcmd = cmd.split(' ')
lines = subprocess.Popen(subcmd,stdout=subprocess.PIPE)
return lines

As far as Python is concerned, you appear to be missing some commas in your cmd definition:
cmd = r"tshark -o column.format:"Source","%s", "Destination","%d", "dstport"," %uD"' -r %s"%(filename)
# -- no comma here -^ ----^ ----^ --^
because the first string ends when the first " is encountered at "Source"; a raw string does not preclude you from escaping embedded quotes.
If you wanted to produce a list of arguments, just make it a list directly, saves you interpolating the filename too:
cmd = ["tshark", "-o",
'column.format:"Source","%s","Destination","%d","dstport"," %uD"',
"-r", filename]
Note the single quotes around the 3rd argument to preserve the quotes in the command line argument.
This eliminates the need to split as well and preserves whitespace in the filename.