I am writing a Python program that needs to return the active hosts scanned in one of my vulnerability scans. I have used this method before returning XML, but when I try to tack on these extra programs such as cut and grep I run into issues. Perhaps it doesn't like "pipes" | or maybe I am doing something completely wrong here with my commas but I have tried all sorts of things and cant seems to get it to return the result like it does when I run the command standalone from the command line. Thanks very much for any help that is provided.
def activeHostsQuery():
args = ['curl', '-s', '-k', '-H', 'X-Requested-With: curl demoapp', '-u','username:password', 'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv', '|', 'cut', '-d', '-f1', '|', 'sort', '|', 'uniq', '|', 'grep', '-E', '"\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\""', '|', 'wc', '-l']
activeHostsNumber = subprocess.Popen(args, stdout=subprocess.PIPE).communicate()[0]
return activeHostsNumber
The right way to string together commands -- if you want to keep the shell out of it, which you should -- is to create multiple Popen objects.
def activeHostsQuery():
args1 = ['curl', '-s', '-k',
'-H', 'X-Requested-With: curl demoapp',
'-u','username:password',
'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv']
args2 = ['cut', '-d', '-f1']
args3 = ['sort', '-u']
args4 = ['grep', '-E', '"\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\""']
args5 = ['wc', '-l']
p1 = subprocess.Popen(args1, stdout=subprocess.PIPE)
p2 = subprocess.Popen(args2, stdin=p1.stdout, stdout=subprocess.PIPE); p1.stdout.close()
p3 = subprocess.Popen(args3, stdin=p2.stdout, stdout=subprocess.PIPE); p2.stdout.close()
p4 = subprocess.Popen(args4, stdin=p3.stdout, stdout=subprocess.PIPE); p3.stdout.close()
p5 = subprocess.Popen(args5, stdin=p4.stdout, stdout=subprocess.PIPE); p4.stdout.close()
activeHostsNumber = p5.communicate()[0]
return activeHostsNumber
The advantage of this is that there's no shell involved -- you can substitute arbitrary variables into your argument lists without concern that they'll be string-split, misinterpreted, cause redirections, or anything else, and the distinctions between arguments you use in generating your lists will be honored.
Now, in this particular case, I'd do the whole thing in native Python -- there's no reason even to use curl when you have native HTTP libraries -- but knowing how to build pipelines with subprocess.Popen is useful in any event.
I would try this:
def activeHostsQuery():
args = ['curl', '-s', '-k', '-H', 'X-Requested-With: curl demoapp', '-u','username:password', 'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv', '|', 'cut', '-d', '-f1', '|', 'sort', '|', 'uniq', '|', 'grep', '-E', '"\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\""', '|', 'wc', '-l']
activeHostsNumber = subprocess.Popen(" ".join("'%s'" % a for a in args), shell=True, stdout=subprocess.PIPE).communicate()[0]
return activeHostsNumber
Edit: added quotes around arguments.
Another edit: Ok, try making the command a single string:
def activeHostsQuery():
cmd = 'curl -s -k -H \'X-Requested-With: curl demoapp\' -u username:password \'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv\' | cut -d, -f1 | sort | uniq | grep -E \'"[[:digit:]]{1,3}\\.[[:digit:]]{1,3}\\.[[:digit:]]{1,3}\\.[[:digit:]]{1,3}"\' | wc -l'
ctiveHostsNumber = subprocess.Popen(cmd, shell = True, stdout = subprocess.PIPE).communicate()[0]
return activeHostsNumber
I know this doesn't answer the question... but that's shell script.
If you want shell script pass an argument to sh -c (or bash or something)
args = ['sh', '-c', 'curl -s -k -H X-Requested-With: curl demoapp -u','username:password https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv | cut -d -f1 | sort | uniq | grep -E "\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\"" | wc -l'
count = int(cubprcess.check_output(args))
or use shell=True like some others suggested. This will definitely not work on Windows if you care about such things.
really you should probably do something like:
import requests
from csv
from StringIO import StringIO
import re
req=reqeusts.get(
'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv',
auth=('username','passoword'),
headers={'X-Requested-With': 'curl demoapp'})
reader = csv.reader(StringIO(req.text))
count = 0
for line in reader:
if re.match(r'.*\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}.*',line[0]) is not None:
count += 1
print count
Related
trying to figure out how to do this:
command = f"adb -s {i} shell"
proc = Popen(command, stdin=PIPE, stdout=PIPE)
out, err = proc.communicate(f'dumpsys package {app_name} | grep version'.encode('utf-8'))
but in this:
command = f"adb -s {i} shell"
proc = run(command, stdin=PIPE, stdout=PIPE, shell=True)
out, err = run(f'dumpsys package {app_name} | grep version', shell=True, text=True, stdin=proc.stdout )
The idea is to make a command which require input of some kind( for example(entering shell)) and afterwards inserting another command to shell.
I've found a way online with communicate, But I wonder how to do it with run() func.
Thanks!
You only need to call run once -- pass the remote command in the input argument (and don't use shell=True in places where you don't need it).
import subprocess, shlex
proc = subprocess.run(['adb', '-s', i, 'shell'],
capture_output=True,
input=f'dumpsys package {shlex.quote(app_name)} | grep version')
shlex.quote prevents an app name that contains $(...), ;, etc from running unwanted commands on your device.
I'm having a problem with my subprocess command, I like to grep out the lines that match with "Online" line.
def run_command(command):
p = subprocess.Popen(command,shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
return iter(p.stdout.readline, b'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v | grep "Online" '.split()
for line in run_command(command):
print(line)
But I will get an error
Error: Unknown option '|'.
Use 'mosquitto_sub --help' to see usage.
But when running with linux shell
user#server64:~/Pythoniscriptid$ mosquitto_sub -u example -P example -t ITT/# -v | grep "Online"
ITT/C5/link Online
ITT/IoT/tester55/link Online
ITT/ESP32/TEST/link Online
I also tried shell = True, but with no success, because I will get another error, that dosen't recognize the topic ITT/#
Error: You must specify a topic to subscribe to.
Use 'mosquitto_sub --help' to see usage.
The "possible dublicate" didn't help me at all, So I think I'm having a different problem. I tried to change code to this, put in not getting any return
def run_command(command,command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline,'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v'.split()
command2 = 'grep Online'.split()
#subprocess.getoutput(command)
for line in run_command(command,command2):
print(line)
When you split the text, the list will look like
['mosquitto_sub', ..., 'ITT/#', '-v', '|', 'grep', '"Online"']
When you pass this list to subprocess.Popen, a literal '|' will be one of the arguments to mosquitto_sub.
If you use shell=True, you must escape any special characters like # in the command, for instance with double quotes:
import subprocess
command = 'echo -e "ITT/#\\ni am Online\\nbar Online\\nbaz" | grep "Online" '
p = subprocess.Popen(
command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
print(line)
Alternatively, connect the pipes as you wrote, but make sure to iterate until b'', not u'':
import subprocess
def run_command(command, command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline, b'')
command = ['echo', '-e', 'ITT/#\\ni am Online\\nbar Online\\nbaz']
command2 = 'grep Online'.split()
for line in run_command(command,command2):
print(line)
Any ideas why this list that I'm passing as my command to subprocess() isn't expanding properly?
file_name = "some_make_file"
cmd = ['gmake', '-pn', '-f', file_name, '|', 'grep', '-A1', '"^# makefile"', '|', 'grep', '-v', '"^#\|^--"', '|', 'sort', '|', 'uniq']
proc = Popen(cmd, stdout=PIPE, stderr=PIPE)
stdout, stderr = prod.communicate()
The output is a gmake error, and if I join the cmd to test it on the cmd line myself, I get this:
#join
" ".join(cmd)
#output
('gmake -pn -f some_make_file | grep -A1 "^ makefile" | grep -v '
'"^#\\|^--" | sort | uniq')
For the life if me, I cannot seem to figure out what's wrong with this command. Any ideas? Am I not escaping something properly? Adding a special char and not realizing it?
You are not running an intermediate shell so there is nothing to interprete the | as a pipe command. Even if you set shell=True, because you passed in a list, it will be escaped. The proper way to pipeline is given in the official python docs Replacing Shell Pipeline – tdelaney
How do I place output of bash command to Python variable?
I am writing a Python script, which I want to enter the output of
bash command:
rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n' | grep -v 'Red Hat'|wc -l, and place it to Python variable, let say R.
After that I want to do, Python if R != 0
then run some Linux command.
How do I achieve that?
There are various options, but the easiest is probably using subprocess.check_output() with shell=True although this can be security hazard if you don't fully control what command is passed in.
import subprocess
var = subprocess.check_output('rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n' | grep -v 'Red Hat'|wc -l', shell = True)
var = int(var)
You need to use shell=True as otherwise the pipes would not be interpreted.
If you need more control you might want to look at plumbum where you can do:
from plumbum.cmd import rpm, grep, wc
chain = rpm["-qa", "--qf", r"%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n"] | grep["-v", "Red Hat"] | wc["-l"]
R = int(chain())
Although I would probably not invoke wc and get the whole output and count its length within python (easier to check that you got just the lines that you expected, piping through wc -l throws away all of the details)
I would recommend envoy primarily because the API is much more intuitive to use for 90% of the use cases.
r = envoy.run('ls ', data='data to pipe in', timeout=2)
print r.status_code # returns status code
print r.std_out # returns the output.
See the Envoy Github page for more details.
You can use stdin.
#!/usr/bin/python
import sys
s = sys.stdin.read()
print s
Then you will run a bash command like this
echo "Hello" | ./myscript.py
Output
Hello
You can replace shell pipeline using Popen:
from subprocess import PIPE,Popen
p1 = Popen(["rpm", "-qa", "--qf", '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n'],stdout=PIPE)
p2 = Popen(["grep", "-v", 'Red Hat'],stdin=p1.stdout,stdout=PIPE)
p1.stdout.close()
p3 = Popen(["wc", "-l"],stdin=p2.stdout,stdout=PIPE)
p2.stdout.close()
out,err = p3.communicate()
If you just want to check if grep returned any matches then forget the wc - l and just check what grep returns:
p1 = Popen(["rpm", "-qa", "--qf", '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n'],stdout=PIPE)
p2 = Popen(["grep", "-v", 'Red Hat'],stdin=p1.stdout,stdout=PIPE)
p1.stdout.close()
out,err = p2.communicate()
if out:
...
Or just use check_output to run the rpm command and check the string for "Red Hat":
out = check_output(["rpm", "-qa", "--qf", '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n'])
if "Red Hat" not in out:
....
Which is the same as inverting the search with grep -v then checking if there are any matches with wc.
I'm trying to grep a list of file from the "*.nasl" of "Openvas" which contains a certain port's number.
I can make it directly in the terminal with the command :
egrep --only-match '111' /home/gwvm/Openvas/var/lib/openvas/plugins/*.nasl |cut -d ":" -f1
This command return all the name of the nasl file which contains 111.
like :
/home/gwvm/Openvas/var/lib/openvas/plugins/SolarWinds_TFTP.nasl:111
/home/gwvm/Openvas/var/lib/openvas/plugins/trojan_horses.nasl:111
and after the cut :
/home/gwvm/Openvas/var/lib/openvas/plugins/SolarWinds_TFTP.nasl
/home/gwvm/Openvas/var/lib/openvas/plugins/trojan_horses.nasl
But when I'm in python(3.1.3) the output give me an error :
egrep:/home/gwvm/Openvas/var/lib/openvas/plugins/*.nasl: No such file or directory
i was thinking about a problem because of the "*.nasl" but when I'm trying with an existing file, same result.
Here is the part of code :
command = ("egrep --only-match '"+ str(port[0]) +"' "+ openvas_directory["locate"]["nasl"] + '*.nasl' + ' |cut -d ":" -f1 ')
process=sp.Popen(command,shell=True, stdout= sp.PIPE)
or
exec(command)
I was thinking too of a bad construction but wen I'm printing the command it gives me what i want :
egrep --only-match '111' /home/gwvm/Openvas/var/lib/openvas/plugins/*.nasl |cut -d ":" -f1
If there are any idea!
from subprocess import PIPE, Popen
x = Popen('egrep --only-match \'111\' /home/gwvm/Openvas/var/lib/openvas/plugins/*.nasl', stdout=PIPE, stderr=PIPE, shell=True)
y = Popen('cut -d ":" -f1', stdin=x.stdout, stdout=PIPE, stderr=PIPE, shell=True)
for row in y.stdout.readline():
print row
Or just use check_output()
And this is btw how you | in Popen ;)
Guidelines:
When using Popen, if you supply a command as a string, use shell=True.
If you however supply Popen with a list ['ls, '-l'] then use shell=False, that's just how it works.
If you're piping data, execute two different Popen's and use the output from the first command as stdin for the second command, this is equivilant to doing | in Linux.