Develop Reference

Cant use grep in subprocess command

I'm having a problem with my subprocess command, I like to grep out the lines that match with "Online" line.
def run_command(command):
p = subprocess.Popen(command,shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
return iter(p.stdout.readline, b'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v | grep "Online" '.split()
for line in run_command(command):
print(line)
But I will get an error
Error: Unknown option '|'.
Use 'mosquitto_sub --help' to see usage.
But when running with linux shell
user#server64:~/Pythoniscriptid$ mosquitto_sub -u example -P example -t ITT/# -v | grep "Online"
ITT/C5/link Online
ITT/IoT/tester55/link Online
ITT/ESP32/TEST/link Online
I also tried shell = True, but with no success, because I will get another error, that dosen't recognize the topic ITT/#
Error: You must specify a topic to subscribe to.
Use 'mosquitto_sub --help' to see usage.
The "possible dublicate" didn't help me at all, So I think I'm having a different problem. I tried to change code to this, put in not getting any return
def run_command(command,command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline,'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v'.split()
command2 = 'grep Online'.split()
#subprocess.getoutput(command)
for line in run_command(command,command2):
print(line)

When you split the text, the list will look like
['mosquitto_sub', ..., 'ITT/#', '-v', '|', 'grep', '"Online"']
When you pass this list to subprocess.Popen, a literal '|' will be one of the arguments to mosquitto_sub.
If you use shell=True, you must escape any special characters like # in the command, for instance with double quotes:
import subprocess
command = 'echo -e "ITT/#\\ni am Online\\nbar Online\\nbaz" | grep "Online" '
p = subprocess.Popen(
command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
print(line)
Alternatively, connect the pipes as you wrote, but make sure to iterate until b'', not u'':
import subprocess
def run_command(command, command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline, b'')
command = ['echo', '-e', 'ITT/#\\ni am Online\\nbar Online\\nbaz']
command2 = 'grep Online'.split()
for line in run_command(command,command2):
print(line)

Sed with space in python

I was trying perform replace using sed in VMkernel. I used the following command,
sed s/myname/sample name/g txt.txt
I got an error saying sed: unmatched '/'.
I replaced space with \. It worked.
When I tried the same using python,
def executeCommand(cmd):
process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
print (output.decode("utf-8"))
executeCommand('sed s/myname/sample\ name/g txt.txt')
I am getting the error sed: unmatched '/' again. I used \s instead of space I am getting the name replaced with samplesname.
How can I replace a string with space?

The simplest thing would be to not be smart about splitting the command:
executeCommand(['sed', 's/myname/sample name/g', 'txt.txt'])
Otherwise you are opening a can of worms, effectively playing a shell parser role.
Alternatively you may run the command in a shell and let the shell parse and run the command:
import subprocess
def executeCommand(cmd):
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
# Or:
# This will run the command in /bin/bash (instead of /bin/sh)
process = subprocess.Popen(['/bin/bash', '-c', cmd], stdout=subprocess.PIPE)
output, error = process.communicate()
print (output.decode("utf-8"))
executeCommand("sed 's/myname/sample name/g' txt.txt")

Python - subprocess cmd as list not expanding properly

Any ideas why this list that I'm passing as my command to subprocess() isn't expanding properly?
file_name = "some_make_file"
cmd = ['gmake', '-pn', '-f', file_name, '|', 'grep', '-A1', '"^# makefile"', '|', 'grep', '-v', '"^#\|^--"', '|', 'sort', '|', 'uniq']
proc = Popen(cmd, stdout=PIPE, stderr=PIPE)
stdout, stderr = prod.communicate()
The output is a gmake error, and if I join the cmd to test it on the cmd line myself, I get this:
#join
" ".join(cmd)
#output
('gmake -pn -f some_make_file | grep -A1 "^ makefile" | grep -v '
'"^#\\|^--" | sort | uniq')
For the life if me, I cannot seem to figure out what's wrong with this command. Any ideas? Am I not escaping something properly? Adding a special char and not realizing it?

You are not running an intermediate shell so there is nothing to interprete the | as a pipe command. Even if you set shell=True, because you passed in a list, it will be escaped. The proper way to pipeline is given in the official python docs Replacing Shell Pipeline – tdelaney

subprocess.call

I am new to the subprocess.call function and I have tried different combinations of the same call but it is not working.
I am trying to execute the following command:
cmd = 'sort -k1,1 -k4,4n -k5,5n '+outpath+fnametempout+' > '+outpath+fnameout
print cmd
If I try the call I get an error:
cmd = cmd.split(" ")
print cmd
subprocess.call(cmd)
the error I get is:
sort: stat failed: >: No such file or directory

Doing it this way, you need shell=True to allow the shell redirection to work.
subprocess.call('sort -k1,1 -k4,4n -k5,5n '+outpath+fnametempout,shell=True)
A better way is:
with open(outpath+fnameout,'w') as fout: #context manager is OK since `call` blocks :)
subprocess.call(cmd,stdout=fout)
which avoids spawning a shell all-together and is safe from shell injection type attacks. Here, cmd is a list as in your original, e.g.
cmd = 'sort -k1,1 -k4,4n -k5,5n '+outpath+fnametempout
cmd = cmd.split()
It should also be stated that python has really nice sorting facilities and so I doubt that it is actually necessary to pass the job off to sort via a subprocess.
Finally, rather than using str.split to split the arguments, from a string, it's probably better to use shlex.split as that will properly handle quoted strings.
>>> import shlex
>>> cmd = "foo -b -c 'arg in quotes'"
>>> print cmd.split()
['foo', '-b', '-c', "'arg", 'in', "quotes'"]
>>> print shlex.split(cmd)
['foo', '-b', '-c', 'arg in quotes']

it is not to compecated execute above command in python:
import subprocess
import sys
proc = subprocess.Popen(['sort','-k1','1', '-k4','4n', '-k5','5n', '+outpath+fnametempout+', '>', '+outpath+fnameout'],stdin=subprocess.PIPE)
proc.communicate()

example:
subprocess.call(['ps','aux'])
lines=subprocess.check_output(['ls','-al'])
line_list = lines.split('\n')
or
handle = subprocess.Popen('ls',stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,close_fds=True)
handle.stdout.read()

Subprocess Python.. Stringing together commands

I am writing a Python program that needs to return the active hosts scanned in one of my vulnerability scans. I have used this method before returning XML, but when I try to tack on these extra programs such as cut and grep I run into issues. Perhaps it doesn't like "pipes" | or maybe I am doing something completely wrong here with my commas but I have tried all sorts of things and cant seems to get it to return the result like it does when I run the command standalone from the command line. Thanks very much for any help that is provided.
def activeHostsQuery():
args = ['curl', '-s', '-k', '-H', 'X-Requested-With: curl demoapp', '-u','username:password', 'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv', '|', 'cut', '-d', '-f1', '|', 'sort', '|', 'uniq', '|', 'grep', '-E', '"\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\""', '|', 'wc', '-l']
activeHostsNumber = subprocess.Popen(args, stdout=subprocess.PIPE).communicate()[0]
return activeHostsNumber

The right way to string together commands -- if you want to keep the shell out of it, which you should -- is to create multiple Popen objects.
def activeHostsQuery():
args1 = ['curl', '-s', '-k',
'-H', 'X-Requested-With: curl demoapp',
'-u','username:password',
'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv']
args2 = ['cut', '-d', '-f1']
args3 = ['sort', '-u']
args4 = ['grep', '-E', '"\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\""']
args5 = ['wc', '-l']
p1 = subprocess.Popen(args1, stdout=subprocess.PIPE)
p2 = subprocess.Popen(args2, stdin=p1.stdout, stdout=subprocess.PIPE); p1.stdout.close()
p3 = subprocess.Popen(args3, stdin=p2.stdout, stdout=subprocess.PIPE); p2.stdout.close()
p4 = subprocess.Popen(args4, stdin=p3.stdout, stdout=subprocess.PIPE); p3.stdout.close()
p5 = subprocess.Popen(args5, stdin=p4.stdout, stdout=subprocess.PIPE); p4.stdout.close()
activeHostsNumber = p5.communicate()[0]
return activeHostsNumber
The advantage of this is that there's no shell involved -- you can substitute arbitrary variables into your argument lists without concern that they'll be string-split, misinterpreted, cause redirections, or anything else, and the distinctions between arguments you use in generating your lists will be honored.
Now, in this particular case, I'd do the whole thing in native Python -- there's no reason even to use curl when you have native HTTP libraries -- but knowing how to build pipelines with subprocess.Popen is useful in any event.

I would try this:
def activeHostsQuery():
args = ['curl', '-s', '-k', '-H', 'X-Requested-With: curl demoapp', '-u','username:password', 'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv', '|', 'cut', '-d', '-f1', '|', 'sort', '|', 'uniq', '|', 'grep', '-E', '"\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\""', '|', 'wc', '-l']
activeHostsNumber = subprocess.Popen(" ".join("'%s'" % a for a in args), shell=True, stdout=subprocess.PIPE).communicate()[0]
return activeHostsNumber
Edit: added quotes around arguments.
Another edit: Ok, try making the command a single string:
def activeHostsQuery():
cmd = 'curl -s -k -H \'X-Requested-With: curl demoapp\' -u username:password \'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv\' | cut -d, -f1 | sort | uniq | grep -E \'"[[:digit:]]{1,3}\\.[[:digit:]]{1,3}\\.[[:digit:]]{1,3}\\.[[:digit:]]{1,3}"\' | wc -l'
ctiveHostsNumber = subprocess.Popen(cmd, shell = True, stdout = subprocess.PIPE).communicate()[0]
return activeHostsNumber

I know this doesn't answer the question... but that's shell script.
If you want shell script pass an argument to sh -c (or bash or something)
args = ['sh', '-c', 'curl -s -k -H X-Requested-With: curl demoapp -u','username:password https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv | cut -d -f1 | sort | uniq | grep -E "\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\"" | wc -l'
count = int(cubprcess.check_output(args))
or use shell=True like some others suggested. This will definitely not work on Windows if you care about such things.
really you should probably do something like:
import requests
from csv
from StringIO import StringIO
import re
req=reqeusts.get(
'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv',
auth=('username','passoword'),
headers={'X-Requested-With': 'curl demoapp'})
reader = csv.reader(StringIO(req.text))
count = 0
for line in reader:
if re.match(r'.*\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}.*',line[0]) is not None:
count += 1
print count