I've read a bunch of examples but none of them work for this specific task.
Python code:
x = Popen(commands, stdout=PIPE, stderr=PIPE, shell=True)
print commands
stdout = x.stdout.read()
stderr = x.stderr.read()
print stdout, stderr
return stdout
Output:
[user#host]$ python helpers.py
['ssh', '-t', 'user#host', ' ', "'service --status-all'"]
usage: ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
[-D [bind_address:]port] [-e escape_char] [-F configfile]
[-I pkcs11] [-i identity_file]
[-L [bind_address:]port:host:hostport]
[-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]
[-R [bind_address:]port:host:hostport] [-S ctl_path]
[-W host:port] [-w local_tun[:remote_tun]]
[user#]hostname [command]
Why am i getting this error?
Using os.popen(...) it works, it executes at least but i can't retrieve the output of the remote command via the SSH tunnel.
I think your commands list is wrong:
commands = ['ssh', '-t', 'user#host', "service --status-all"]
x = Popen(commands, stdout=PIPE, stderr=PIPE)
Additionally, I don't think you should pass shell=True if you're going to pass a list to Popen.
e.g. either do this:
Popen('ls -l',shell=True)
or this:
Popen(['ls','-l'])
but not this:
Popen(['ls','-l'],shell=True)
Finally, there exists a convenience function for splitting a string into a list the same way your shell would:
import shlex
shlex.split("program -w ith -a 'quoted argument'")
will return:
['program', '-w', 'ith', '-a', 'quoted argument']
Related
I want to run the command ffprobe -i test.m4a -show_entries format=duration -v quiet -of csv="p=0". It works in the terminal and returns output code 0, but running it with subprocess, i.e.
subprocess.check_output(['ffprobe', '-i', 'test.m4a', '-show_entries', 'format=duration', '-v', 'quiet', '-of', 'csv="p=0"'])
raises a CalledProcessError - {Command} returned non-zero exit status 1.. I tried running this command in a try-except loop and printing the error details, but it just outputs as an empty byte string b''.
One way for debugging the issue is adding -report argument:
subprocess.check_output(['ffprobe', '-i', 'output.mp4', '-show_entries', 'format=duration', '-v', 'quiet', '-of', 'csv="p=0"', '-report'])
-report is used for creating a log file with name like ffprobe-20220811-232043.log.
The log files shows the following error:
[csv # 00000213297fe640] Failed to set option '"p' with value '0"' provided to writer context
The log files shows that the executed "shell command" is:
ffprobe -i output.mp4 -show_entries "format=duration" -v quiet -of "csv=\"p=0\"" -report
The solution is removing the quotes from "p=0":
subprocess.check_output(['ffprobe', '-i', 'output.mp4', '-show_entries', 'format=duration', '-v', 'quiet', '-of', 'csv=p=0'])
I recommend using the subprocess.run instead of check_output.
subprocess.run(command, stdout=output, encoding="utf-8")
Command = is the variable that houses the command you want , no need for separation using the commas.
stdout = Output = means that the output should be recorded in the file called (Output) which you can create beforehand.
encoding = just means to make sure it's encoded into texts from bytes
Using the git method from the sh package in Python, I am trying to craft the following git statement:
git grep -i -e "(example1|example2)" -- './*' ':!*.java' ':!*.xml' HEAD
In my code, I am using the following:
exclude_list = '-- \'./*\' \':!*.java\' \':!*.xml\''
result = git('grep', '-i', '-e', '"({})"'.format(r'\|'.join(self.keywords)), exclude_list, 'HEAD', _tty_out=False))
Which when I run returns the following error:
Error:
RAN: /usr/bin/git grep -i -e "(example1\|example2)" -- './*' ':!*.java' ':!*.xml' HEAD
STDOUT:
STDERR:
error: unknown option ` './*' ':!*.java' ':!*.xml''
usage: git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]
However, if I run the command locally, it returns as expected. I am not a Python dev, unfortunately, so not sure what I am doing wrong!
You pass 'grep', '-i', '-e', as separate parameters but you pass exclude_list as a single string; git complains about not understanding this single string. Split it:
result = git('grep', '-i', '-e', '"({})"'.format(r'\|'.join(self.keywords)), '--', './*', ':!*.java', ':!*.xml', 'HEAD', _tty_out=False))
This is the essential part: '--', './*', ':!*.java', ':!*.xml',
I'm having a problem with my subprocess command, I like to grep out the lines that match with "Online" line.
def run_command(command):
p = subprocess.Popen(command,shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
return iter(p.stdout.readline, b'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v | grep "Online" '.split()
for line in run_command(command):
print(line)
But I will get an error
Error: Unknown option '|'.
Use 'mosquitto_sub --help' to see usage.
But when running with linux shell
user#server64:~/Pythoniscriptid$ mosquitto_sub -u example -P example -t ITT/# -v | grep "Online"
ITT/C5/link Online
ITT/IoT/tester55/link Online
ITT/ESP32/TEST/link Online
I also tried shell = True, but with no success, because I will get another error, that dosen't recognize the topic ITT/#
Error: You must specify a topic to subscribe to.
Use 'mosquitto_sub --help' to see usage.
The "possible dublicate" didn't help me at all, So I think I'm having a different problem. I tried to change code to this, put in not getting any return
def run_command(command,command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline,'')
command = 'mosquitto_sub -u example -P example -t ITT/# -v'.split()
command2 = 'grep Online'.split()
#subprocess.getoutput(command)
for line in run_command(command,command2):
print(line)
When you split the text, the list will look like
['mosquitto_sub', ..., 'ITT/#', '-v', '|', 'grep', '"Online"']
When you pass this list to subprocess.Popen, a literal '|' will be one of the arguments to mosquitto_sub.
If you use shell=True, you must escape any special characters like # in the command, for instance with double quotes:
import subprocess
command = 'echo -e "ITT/#\\ni am Online\\nbar Online\\nbaz" | grep "Online" '
p = subprocess.Popen(
command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
print(line)
Alternatively, connect the pipes as you wrote, but make sure to iterate until b'', not u'':
import subprocess
def run_command(command, command2):
p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
return iter(p2.stdout.readline, b'')
command = ['echo', '-e', 'ITT/#\\ni am Online\\nbar Online\\nbaz']
command2 = 'grep Online'.split()
for line in run_command(command,command2):
print(line)
I am trying to call ssh from a Python program but it seems to be ignoring the arguments.
This is the Python program:
#!/usr/bin/python
from subprocess import Popen, PIPE, call
vm_name = 'vmName with-space'
vm_host = 'user#192.168.21.230'
def ssh_prefix_list(host=None):
if host:
# return ["ssh", "-v", "-v", "-v", host]
return ["scripts/ssh_wrapper", "-v", "-v", "-v", host]
else:
return []
def start(vm_name, vm_host=None): # vm_host defaults to None
print "vm_host = ", vm_host
vbm_args = ssh_prefix_list(vm_host) + ["VBoxManage", "startvm", vm_name]
print vbm_args
return call(vbm_args, shell=True)
start(vm_name, vm_host)
The wrapper prints the number of arguments, their values, and calls ssh.
#!/bin/bash
echo Number of arguments: $#
echo ssh arguments: "$#"
ssh "$#"
This is the output.
$ scripts/vm_test.py
vm_host = stephen#192.168.21.230
['scripts/ssh_wrapper', '-v', '-v', '-v', 'stephen#192.168.21.230', 'VBoxManage', 'startvm', 'atp-systest Clone']
Number of arguments: 0
ssh arguments:
usage: ssh [-1246AaCfgKkMNnqsTtVvXxY] [-b bind_address] [-c cipher_spec]
[-D [bind_address:]port] [-e escape_char] [-F configfile]
[-i identity_file] [-L [bind_address:]port:host:hostport]
[-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]
[-R [bind_address:]port:host:hostport] [-S ctl_path]
[-w local_tun[:remote_tun]] [user#]hostname [command]
This is on Python 2.5.
When you are using shell=True , you need to pass in a string, not a list of arguments. Try -
return call(' '.join(vbm_args), shell=True)
Also, you should consider constructing the string from start, rather than list.
When you pass a list to call() or Popen() with shell=True , only the first element in the list is actually called , and that is the reason you are seeing the wrapper called with 0 arguments.
You should also try first without using shell=True , since its a security hazard, as clearly stated in the documentation of subprocess -
Using shell=True can be a security hazard. See the warning under Frequently Used Arguments for details.
I think this might be it:
prefix_list = ssh_prefix_list(vm_host)
prefix_list.append(["VBoxManage startvm %s" % vm_name])
however I would strongly recommend using paramiko - it makes things much easier.
When you use call with shell=True, you need to pass a single string, not an array of strings. So:
call("scripts/ssh_wrapper -v -v -v "+host+" VBoxManage startvm "+vmname)
I am writing a Python program that needs to return the active hosts scanned in one of my vulnerability scans. I have used this method before returning XML, but when I try to tack on these extra programs such as cut and grep I run into issues. Perhaps it doesn't like "pipes" | or maybe I am doing something completely wrong here with my commas but I have tried all sorts of things and cant seems to get it to return the result like it does when I run the command standalone from the command line. Thanks very much for any help that is provided.
def activeHostsQuery():
args = ['curl', '-s', '-k', '-H', 'X-Requested-With: curl demoapp', '-u','username:password', 'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv', '|', 'cut', '-d', '-f1', '|', 'sort', '|', 'uniq', '|', 'grep', '-E', '"\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\""', '|', 'wc', '-l']
activeHostsNumber = subprocess.Popen(args, stdout=subprocess.PIPE).communicate()[0]
return activeHostsNumber
The right way to string together commands -- if you want to keep the shell out of it, which you should -- is to create multiple Popen objects.
def activeHostsQuery():
args1 = ['curl', '-s', '-k',
'-H', 'X-Requested-With: curl demoapp',
'-u','username:password',
'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv']
args2 = ['cut', '-d', '-f1']
args3 = ['sort', '-u']
args4 = ['grep', '-E', '"\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\""']
args5 = ['wc', '-l']
p1 = subprocess.Popen(args1, stdout=subprocess.PIPE)
p2 = subprocess.Popen(args2, stdin=p1.stdout, stdout=subprocess.PIPE); p1.stdout.close()
p3 = subprocess.Popen(args3, stdin=p2.stdout, stdout=subprocess.PIPE); p2.stdout.close()
p4 = subprocess.Popen(args4, stdin=p3.stdout, stdout=subprocess.PIPE); p3.stdout.close()
p5 = subprocess.Popen(args5, stdin=p4.stdout, stdout=subprocess.PIPE); p4.stdout.close()
activeHostsNumber = p5.communicate()[0]
return activeHostsNumber
The advantage of this is that there's no shell involved -- you can substitute arbitrary variables into your argument lists without concern that they'll be string-split, misinterpreted, cause redirections, or anything else, and the distinctions between arguments you use in generating your lists will be honored.
Now, in this particular case, I'd do the whole thing in native Python -- there's no reason even to use curl when you have native HTTP libraries -- but knowing how to build pipelines with subprocess.Popen is useful in any event.
I would try this:
def activeHostsQuery():
args = ['curl', '-s', '-k', '-H', 'X-Requested-With: curl demoapp', '-u','username:password', 'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv', '|', 'cut', '-d', '-f1', '|', 'sort', '|', 'uniq', '|', 'grep', '-E', '"\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\""', '|', 'wc', '-l']
activeHostsNumber = subprocess.Popen(" ".join("'%s'" % a for a in args), shell=True, stdout=subprocess.PIPE).communicate()[0]
return activeHostsNumber
Edit: added quotes around arguments.
Another edit: Ok, try making the command a single string:
def activeHostsQuery():
cmd = 'curl -s -k -H \'X-Requested-With: curl demoapp\' -u username:password \'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv\' | cut -d, -f1 | sort | uniq | grep -E \'"[[:digit:]]{1,3}\\.[[:digit:]]{1,3}\\.[[:digit:]]{1,3}\\.[[:digit:]]{1,3}"\' | wc -l'
ctiveHostsNumber = subprocess.Popen(cmd, shell = True, stdout = subprocess.PIPE).communicate()[0]
return activeHostsNumber
I know this doesn't answer the question... but that's shell script.
If you want shell script pass an argument to sh -c (or bash or something)
args = ['sh', '-c', 'curl -s -k -H X-Requested-With: curl demoapp -u','username:password https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv | cut -d -f1 | sort | uniq | grep -E "\"[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\"" | wc -l'
count = int(cubprcess.check_output(args))
or use shell=True like some others suggested. This will definitely not work on Windows if you care about such things.
really you should probably do something like:
import requests
from csv
from StringIO import StringIO
import re
req=reqeusts.get(
'https://qualysapi.qualys.com/api/2.0/fo/scan/?action=fetch&scan_ref=scan/1111111.22222&mode=brief&output_format=csv',
auth=('username','passoword'),
headers={'X-Requested-With': 'curl demoapp'})
reader = csv.reader(StringIO(req.text))
count = 0
for line in reader:
if re.match(r'.*\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}.*',line[0]) is not None:
count += 1
print count