This question already has answers here:
How to use `subprocess` command with pipes
(7 answers)
Closed 4 years ago.
I am running a code in python which calculates the count of the files present in a directory`
hadoop fs -count /user/a909983/sample_data/ | awk '{print $2}'
This successfully returns 0 in the linux command line as the dir is empty.However when I run this in python script it returns 1.The line of code in python is:
directoryEmptyStatusCommand = subprocess.call(
["hadoop", "fs", "-count", "/user/a909983/sample_data/", "|", "awk '{print $2}'"])
How can I correct this? or what am I missing ?. I have also tried using Popen, but the result is the same.
Use subprocess.Popen and don't use the pipe | because it requires shell=True which security risk. So, use the subprocess.PIPE and use that with subprocess.check_output without pipe thats the correct method.
So, you can try something like:
command = subprocess.Popen(("hadoop", "fs", "-count", "/user/a909983/sample_data/") , stdout=subprocess.PIPE)
output = subprocess.check_output(("awk '{print $2}'"), stdin=command.stdout)
In Case You want to try Shell commands by enabling shell=True:
cmd = "hadoop fs -count /user/a909983/sample_data/ | awk '{print $2}'"
command = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
output = command.communicate()[0]
print(output)
Related
This question already has answers here:
File not found error when launching a subprocess containing piped commands
(6 answers)
How to use `subprocess` command with pipes
(7 answers)
Actual meaning of 'shell=True' in subprocess
(7 answers)
Closed 3 years ago.
I want to execute the command throughout the python and want to get output back
command is: ls /sys/class/net/ | sed -n '/e.*/p'
it returns the names of interfaces attached which starts with e
with terminal, I am getting the output as: eth0 eth1, which is expected
But with python, I am executing like this
out = subprocess.Popen(["ls", "/sys/class/net/", " | ", "sed -n '/e.*/p'"],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,)
stdout,stderr = out.communicate()
print(stdout)
the expected output is eth0 eth1
but output:
ls: cannot access ' | ': No such file or directory\nls: cannot access 'sed -n '\''/e.*/p'\''': No such file or directory /sys/class/net/:\neth0\neth1\nlo\nwifi0\nwifi1\nwifi2\n
what is the problem here?
I want to get the string output of the following linux command
systemctl show node_exporter |grep LoadState| awk '{split($0,a,"="); print a[2]}'
I tried with
import subprocess
output = subprocess.check_output("systemctl show node_exporter |grep LoadState| awk '{split($0,a,"="); print a[2]}'", shell=True)
but the output is,
output = subprocess.check_output("systemctl show node_exporter |grep LoadState| awk '{split($0,a,"="); print a[2]}'", shell=True)
SyntaxError: keyword can't be an expression
Well,
First of all, the function takes a list of strings as a command, not a single string. E.g.:
"ls -a -l" - wrong
["ls", "-a", "-l"] - good
Secondly. If the linux command is super complex or contains lots of lines - it makes sense to create a separate bash file e.g. command.sh, put your linux commands there and run the script from python with:
import subprocess
output = subprocess.check_output(["./command.sh"], shell=True)
You need to escape the double quotes (because they indicate the begin/end of the string):
import subprocess
output = subprocess.check_output("systemctl show node_exporter |grep LoadState| awk '{split($0,a,\"=\"); print a[2]}'", shell=True)
Im trying to run this bash command using python subprocess
find /Users/johndoe/sandbox -iname "*.py" | awk -F'/' '{ print $NF}'
output:-
helld.xl.py
parse_maillog.py
replace_pattern.py
split_text_match.py
ssh_bad_login.py
Here is what i have done in python2.7 way, but it gives the output where awk command filter is not working
>>> p1=subprocess.Popen(["find","/Users/johndoe/sandbox","-iname","*.py"],stdout=subprocess.PIPE)
>>> p2=subprocess.Popen(['awk','-F"/"','" {print $NF} "'],stdin=p1.stdout,stdout=subprocess.PIPE)
>>>p2.communicate()
('/Users/johndoe/sandbox/argparse.py\n/Users/johndoe/sandbox/custom_logic_substitute.py\n/Users/johndoe/sandbox/finditer_html_parse.py\n/Users/johndoe/sandbox/finditer_simple.py\n/Users/johndoe/sandbox/group_regex.py\n/Users/johndoe/sandbox/helo.py\n/Users/johndoe/sandbox/newdir/helld.xl.py\n/Users/johndoe/sandbox/parse_maillog.py\n/Users/johndoe/sandbox/replace_pattern.py\n/Users/johndoe/sandbox/split_text_match.py\n/Users/johndoe/sandbox/ssh_bad_login.py\n', None)
I could also get output by using p1 alone here like below,but i cant get the awk working here
list1=[]
result=p1.communicate()[0].split("\n")
for item in res:
a=item.rstrip('/').split('/')
list1.append(a[-1])
print list1
You are incorrectly passing in shell quoting (and extra shell quoting which isn't even required by the shell!) when you're not invoking a shell. Don't do that.
p2=subprocess.Popen(['awk', '-F/', '{print $NF}'], stdin=...
When you have shell=True you need extra quotes around some arguments to protect them from the shell, but there is no shell here, so putting them in is incorrect, and will cause parse errors by Awk.
However, you should almost never need to call Awk from Python, especially for trivial tasks which Python can easily do natively:
list1 = [line.split('/')[-1]
for line in subprocess.check_output(
["find", "/Users/johndoe/sandbox",
"-iname", "*.py"]).splitlines()]
In this particular case, note also that GNU find already has a facility to produce this result directly:
list1 = subprocess.check_output(
["find", "/Users/johndoe/sandbox",
"-iname", "*.py", "-printf", "%f\\n"]).splitlines()
Use this: p2.communicate()[0].split("\n").
It will output a list of lines.
if you don't have any reservation using shell=True , then this should be pretty simple solution
from subprocess import Popen
import subprocess
command='''
find /Users/johndoe/sandbox -iname "*.py" | awk -F'/' '{ print $NF}'
'''
process=Popen(command,shell=True,stdout=subprocess.PIPE)
result=process.communicate()
print result
I'm unable to execute the following line:
os.system("timeout 1s bash -c \"ffmpeg -i \""+path+\"+" | <some_<other_cmd>\"")
So the purpose of this command is to set a timeout for the whole command, i.e. pipelining some ffmpeg information from a path.
The problem is because bash -c "CMD" is expected, but the command also contains " ".
Is there another way of defining the \"path\", because the path can contain spaces? Or another solution which can resolve my problem?
Thanks in advance!
Triple sinqle quotes can do the trick (so that you do not have to escape doublequotes):
os.system('''timeout 1s bash -c "ffmpeg -i "+path+"+" | cat''')
But in general.. Why not use subprocess.call that has saner syntax?
Has answered similar question in other posts: 1 and 2
you can use subprocess related functions, which all support timeout parameter, to replace os.system
such as subprocess.check_output
ffmpegCmd = "ffmpeg -I %s | %s" % (path, someOtherCmd)
outputBytes = subprocess.check_output(ffmpegCmd, shell=True, timeout=1)
outputStr = outputBytes.decode("utf-8") # change utf-8 to your console encoding if necessary
This question already has answers here:
How to use `subprocess` command with pipes
(7 answers)
Closed 7 years ago.
I am trying to filter out first 3 line of /proc/meminfo using pipe and head command.
so basically i need to run this in Python:
cat /proc/meminfo | head -3
I am using below line in my code :
subprocess.call(["cat", "/proc/meminfo", "|", "head", "-3"])
While just using subprocess.call(["cat", "/proc/meminfo"]) I am getting whole list but I am just interested in first 3 line.
Using above command is giving me below error:
cat: invalid option -- '3'
Try `cat --help' for more information.
Any suggestions?
/proc/meminfo is just a file. You don't need a subprocess to read it. Simply open and read it as a file. Here is all you need:
fh = open('/proc/meminfo', 'r')
lines = fh.readlines()
fh.close()
first_lines = lines[:3]
The first_lines list will contain the first three lines (including trailing newline characters).
To use pip you have to enable shell as shell=True, however it's not advisable specifically because of security reason . You can do this alternative,
import subprocess
ps = subprocess.Popen(('cat', '/proc/meminfo'),stdout=subprocess.PIPE)
output = subprocess.check_output(('head', '-3'), stdin=ps.stdout)
print output
The pipe is a shell syntax element. You need to run the code in a shell to use a pipe:
subprocess.call(["cat /proc/meminfo | head -3"], shell=True)
From the manual:
If shell is True, the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user’s home directory.
Well head actually accepts an argument, so the pipe is not actually necessary. The following should give the expected result.
subprocess.call(["head", "-3", "/proc/meminfo"])
following this document
In default, subprocess.call with shell=False will disables all shell based features including pipe. When using shell=True, pipes.quote() can be used to properly escape whitespace and shell metacharacters in strings that are going to be used to construct shell commands.
you can use this code
subprocess.call("cat /proc/meminfo | head -3", shell=True)