I am trying to develop a script to read pcap file and extract some field from it but using tshark as a subprocess. However i am getting syntax error regarding cmd. Can anyone help me out on this?
def srcDestDport (filename):
cmd = r"tshark -o column.format:"Source","%s", "Destination","%d", "dstport"," %uD"' -r %s"%(filename)
subcmd = cmd.split(' ')
lines = subprocess.Popen(subcmd,stdout=subprocess.PIPE)
return lines
As far as Python is concerned, you appear to be missing some commas in your cmd definition:
cmd = r"tshark -o column.format:"Source","%s", "Destination","%d", "dstport"," %uD"' -r %s"%(filename)
# -- no comma here -^ ----^ ----^ --^
because the first string ends when the first " is encountered at "Source"; a raw string does not preclude you from escaping embedded quotes.
If you wanted to produce a list of arguments, just make it a list directly, saves you interpolating the filename too:
cmd = ["tshark", "-o",
'column.format:"Source","%s","Destination","%d","dstport"," %uD"',
"-r", filename]
Note the single quotes around the 3rd argument to preserve the quotes in the command line argument.
This eliminates the need to split as well and preserves whitespace in the filename.
Related
my file name is
file_name = '19-00165_my-test - Copy (7)_Basic_sample_data'
my function is like
call("rm -rf /tmp/" + file_name + '.csv', shell=True)
but getting this error
/bin/sh: -c: line 0: syntax error near unexpected token `('
My response always is: Don't use space in files.
But if you really want this, than you should place the files in quotes as such:
call("rm -f '/tmp/{0}.csv'".format(file_name), shell=True)
Why are you using shell=True? That means the command will be passed to a shell for parsing, which is what's causing all the trouble. With shell=False, you pass a list consisting of the commands followed by its arguments, each as a separate list element (rather than all mashed together as a single string). Since the filename never goes through shell parsing, it can't get mis-parsed.
call(["rm", "-rf", "/tmp/" + file_name + '.csv'], shell=False)
In order to avoid having problems with unescaped characters, one way is to use the shlex module:
You can use the quote() function to escape the string, it returns a shell-escaped version of the string:
import shlex
file_name = "19-00165_my-test - Copy (7)_Basic_sample_'data"
call(f"rm -f /tmp/{shlex.quote(file_name)}.csv", shell=True)
# rm -rf /tmp/'19-00165_my-test - Copy (7)_Basic_sample_'"'"'data'.csv
You can also use join():
import shlex
file_name = "19-00165_my-test - Copy (7)_Basic_sample_'data"
call(shlex.join(["rm", "-f", f"/tmp/{file_name}.csv"]), shell=True)
# rm -f '/tmp/19-00165_my-test - Copy (7)_Basic_sample_'"'"'data.csv'
Note: This answer is only valid if shell=True is required to make the command work. Otherwise the answer of #Gordon Davisson is way easier.
I have an awk command that works in bash, but im now trying to put it into a python script
I have tried both os.system, and subprocess.call both return the same error. sh: 1: Syntax error: "(" unexpected
os.system('awk \'FNR<=27{print;next} ++count%10==0{print;count}\' \'{0} > {1}\'.format(inputfile, outpufile)')
So this awk command will take the large inputfile and create an output file that leaves the first 27 lines of header, but then starting on line 28 it only takes every 10th line and puts it into the output file
Im using the .format() because it is within a python script where the input file will be different every times its run.
ive also tried
subprocess.call('awk \'FNR<=27{print;next} ++count%10==0{print;count}\' \'{0} > {1}\'.format(inputfile, outpufile)')
both come up with the same error above. What am I missing?
As per the comment above, probably more pythonic (and more manageable) to directly use python.
But, if you want to use awk then one way is to format your command with your variable filenames separately.
This works using a basic test text file:
import os
def awk_runner(inputfile, outputfile):
cmd = "awk 'FNR<=27{print;next} ++count%10==0{print;count}' " + inputfile + " > " + outputfile
os.system(cmd)
awk_runner('test1.txt', 'testout1.txt')
There are two main issues with your Python code:
format() is a python method call, it should not be put into the string of awk_cmd to execute under the shell
when calling format() method, braces {} are used to identify substitution target in the format string objects, they need to be escaped using {{ ... }}
See below a modified version of your code:
awk_cmd = "awk 'FNR<=7{{print;next}} ++count%10==0{{print;count}}' {0} > {1}".format(inputfile, outpufile)
os.system(awk_cmd)
In bash when I used
myscript.sh
file="/tmp/vipin/kk.txt"
curl -L "myabcurlx=10&id-11.com" > $file
cat $file
./myscript.sh gives me below output
1,2,33abc
2,54fdd,fddg3
3,fffff,gfr54
When I tried to fetch it using python and tried below code -
mypython.py
command = curl + ' -L ' + 'myabcurlx=10&id-11.com'
output = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE).stdout.read().decode('ascii')
print(output)
python mypython.py throw error, Can you please point out what is wrong with my code.
Error :
/bin/sh: line 1: &id=11: command not found
Wrong Parameter
command = curl + ' -L ' + 'myabcurlx=10&id-11.com'
Print out what this string is, or just think about it. Assuming that curl is the string 'curl' or '/usr/bin/curl' or something, you get:
curl -L myabcurlx=10&id-11.com
That’s obviously not the same thing you typed at the shell. Most importantly, that last argument is not quoted, and it has a & in the middle of it, which means that what you’re actually asking it to do is to run curl in the background and then run some other program that doesn’t exist, as if you’d done this:
curl -L myabcurlx=10 &
id-11.com
Obviously you could manually include quotes in the string:
command = curl + ' -L ' + '"myabcurlx=10&id-11.com"'
… but that won’t work if the string is, say, a variable rather than a literal in your source—especially if that variable might have quote characters within it.
The shlex module has helpers to quoting things properly.
But the easiest thing to do is just not try to build a command line in the first place. You aren’t using any shell features here, so why add the extra headaches, performance costs, problems with the shell getting in the way of your output and retcode, and possible security issues for no benefit?
Make the arguments a list rather than a string:
command = [curl, '-L', 'myabcurlx=10&id-11.com']
… and leave off the shell=True
And it just works. No need to get spaces and quotes and escapes right.
Well, it still won’t work, because Popen doesn’t return output, it’s a constructor for a Popen object. But that’s a whole separate problem—which should be easy to solve if you read the docs.
But for this case, an even better solution is to use the Python bindings to libcurl instead of calling the command-line tool. Or, even better, since you’re not using any of the complicated features of curl in the first place, just use requests to make the same request. Either way, you get a response object as a Python object with useful attributes like text and headers and request.headers that you can’t get from a command line tool except by parsing its output as a giant string.
import subprocess
fileName="/tmp/vipin/kk.txt"
with open(fileName,"w") as f:
subprocess.read(["curl","-L","myabcurlx=10&id-11.com"],stdout=f)
print(fileName)
recommended approaches:
https://docs.python.org/3.7/library/urllib.request.html#examples
http://docs.python-requests.org/en/master/user/install/
I'm trying to use ffmpeg from python. The command I need to execute is:
ffmpeg -i test_file-1kB.mp4 -i test_file.mp4 -filter_complex psnr="stats_file=test_file.mp4-1kB.psnr" -f null -
However, my output that is getting passed to subprocess looks like it is escaping the double quotes with backslashes like so:
In[1]: print(subprocess.list2cmdline(psnr_args))
ffmpeg -i test_file-1kB.mp4 -i test_file.mp4 -filter_complex psnr=\"stats_file=test_file.mp4-1kB.psnr\" -f null -
To use subprocess, I build my command line arguments one at a time into a list and then pass the list to subprocess.
psnr_args = []
psnr_args.append("ffmpeg")
#add first input, the encoded video
psnr_args.append("-i")
psnr_args.append(full_output_file_name)
#add second input, the original video
psnr_args.append("-i")
psnr_args.append(video_file)
#Setup the psnr log file
psnr_args.append("-filter_complex")
psnr_args.append('psnr="stats_file=%s.psnr"' % vstats_abs_filename )
#Output the video to null
psnr_args.append("-f")
psnr_args.append("null")
psnr_args.append("-")
print(subprocess.list2cmdline(psnr_args))
run_info_psnr = subprocess.run(psnr_args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
After more fiddling, I found a solution that works in this case but may not work in all cases. If I use double quotes as the outer quotes and the single quotes as the inner quotes, the output to subprocess uses a single quote at the same location with no backslash. This is acceptable for ffmpeg. However, for others where double quotes are the only solution, it won't be a fix.
psnr_args.append("psnr='stats_file=%s.psnr'" % vstats_abs_filename )
Output to subprocess looks like this:
In[1]: print(subprocess.list2cmdline(psnr_args))
ffmpeg -i test_file-1kB.mp4 -i test_file.mp4 -filter_complex psnr='stats_file=test_file.mp4-1kB.psnr' -f null -
In shell, the argument:
psnr="stats_file=test_file.mp4-1kB.psnr"
Is absolutely identical to:
psnr=stats_file=test_file.mp4-1kB.psnr
The quotes are removed during the shell's own processing. They are not part of the command passed to ffmpeg, which doesn't expect or understand them. Because you're directly telling the Python subprocess module to invoke a literal argument vector, there's no shell involved, so shell syntax shouldn't be present.
This has something to do with ffmpeg AV filter chain syntax too. You need to run the command like xxxx -filter_complex "psnr='stats.txt'" xxxx. To get this, you should ensure the double quote that encapsulate the filter chain reaches inside. subproces expects a flat list as the first argument, where the command is the first entry. So ['ffmpeg', '-i', "t1.mp4", "-filter_compelx", '"psnr=\'stats.txt\'"', .... and so on ].
So, I am trying to issue this command from a python script that collects cpu information across a predetermined number of nodes in a cluster. Here I use a fanout of 2 and only run it on nodes b127 through b129 for testing purposes.
pdsh -f2 -w b[127-129] 'python -c "import multiprocessing
num_cpu = multiprocessing.cpu_count()
stat_fd = open('/proc/stat')
stat_fd.close()"'
I printed the command and this is what it shows on the terminal. Thus, telling me that the quotes and commands are properly formatted. I get this string by executing the following code:
python_command = "'python -c "\
+ "\"import multiprocessing\n"\
+ "num_cpu = multiprocessing.cpu_count()\n"\
+ "stat_fd = open(\'/proc/stat\')\n"\
+ "stat_fd.close()\"'"
command = "pdsh -f2 -w b[127-129] " + python_command
print command
Unfortunately, the line with open(\'/proc/stat\') seems to be the problem as that is the only line that causes the parser to fail due to the nested single quotes. I've tried numerous combinations of quoting and escaping in order to make it work to no avail. I've omitted some code between the opening and closing of the file to minimize the posted code.
I searched around and found this link, but found it was too simple of an example because I could replicate those commands. And yes, I know I can use bash commands to get what I want done and I may end up doing so, but this one has me beating my head on the wall. I also have scripts that gather data using top and ps so I don't need an explanation using them. I greatly appreciate the help in advance.
Try this:
python_command = """pdsh -f2 -w b[127-129] 'python -c "import multiprocessing
num_cpu = multiprocessing.cpu_count()
stat_fd = open(\\"/proc/stat\\")
stat_fd.close()"'"""
In Python, you can use triple quotes ("""...""" or '''...''') for strings containing new lines and single/double quotes.
The last level of quotes (on the open() line) will need to be escaped so that they don't conflict with outer quotes. You also need to escape the backslashes so they aren't immediately consumed by Python when interpreting the string: \\".