Develop Reference

Execute chained bash commands including multiple pipes and grep in Python3

I have to use the below bash command in a python script which includes multiple pip and grep commands.
grep name | cut -d':' -f2 | tr -d '"'| tr -d ','
I tried to do the same using subprocess module but didn't succeed.
Can anyone help me to run the above command in Python3 scripts?
I have to get the below output from a file file.txt.
Tom
Jack
file.txt contains:
"name": "Tom",
"Age": 10
"name": "Jack",
"Age": 15
Actually I want to know how can run the below bash command using Python.
cat file.txt | grep name | cut -d':' -f2 | tr -d '"'| tr -d ','

This works without having to use the subprocess library or any other os cmd related library, only Python.
my_file = open("./file.txt")
line = True
while line:
line = my_file.readline()
line_array = line.split()
try:
if line_array[0] == '"name":':
print(line_array[1].replace('"', '').replace(',', ''))
except IndexError:
pass
my_file.close()

If you not trying to parse a json file or any other structured file for which using a parser would be the best approach, just change your command into:
grep -oP '(?<="name":[[:blank:]]").*(?=",)' file.txt
You do not need any pipe at all.
This will give you the output:
Tom
Jack
Explanations:
-P activate perl regex for lookahead/lookbehind
-o just output the matching string not the whole line
Regex used: (?<="name":[[:blank:]]").*(?=",)
(?<="name":[[:blank:]]") Positive lookbehind: to force the constraint "name": followed by a blank char and then another double quote " the name followed by a double quote " extracted via (?=",) positive lookahead
demo: https://regex101.com/r/JvLCkO/1

Store filtered output of cmd command in a variable

I am trying to store the output of a cmd command as a variable in python.
To achieve this i am using os.system() but os.system() just runs the process,it doesn't capture the output.
import os
PlatformName = os.system("adb shell getprop | grep -e 'bt.name'")
DeviceName = os.system("adb shell getprop | grep -e '.product.brand'")
DeviceID = os.system("adb shell getprop | grep -e 'serialno'")
Version = os.system("adb shell getprop | grep -e 'version.release'")
print(PlatformName)
print(DeviceName)
print(DeviceID)
print(Version)
Then i tried to use the subprocess module.
import subprocess
import os
PlatformName = subprocess.check_output(["adb shell getprop | grep -e 'bt.name'"])
DeviceName = subprocess.check_output(["adb shell getprop | grep -e '.product.brand'"])
DeviceID = subprocess.check_output(["adb shell getprop | grep -e 'serialno'"])
Version = subprocess.check_output(["adb shell getprop | grep -e 'version.release'"])
print(PlatformName)
print(DeviceName)
print(DeviceID)
print(Version)
I am getting the following error
FileNotFoundError: [WinError 2] The system cannot find the file
specified
How can I store the output of the command as a variable?

The issues here:
passing arguments like this (string in a list, with spaces) is really not recommended
passing arguments like this need shell=True for it to have a slight chance to work, and shell=True is known for security issues (and other issues as well, like non-portability)
grep is not standard on windows, and the pattern is a regex which means you'd probably have to escape . ("bt\.name").
when not found grep returns 1 and would make check_output fail.
when found grep returns match(es), and a newline, that you'd have to strip
I'd rewrite this:
PlatformName = subprocess.check_output(["adb shell getprop | grep -e 'bt.name'"])
as:
output = subprocess.check_output(["adb","shell","getprop"])
platform_name = next((line for line in output.decode().splitlines() if "bt.name" in line),"")
The second line is a "native" version of grep (without regexes). It returns the first occurrence of "bt.line" in the output lines or empty string if not found.
You don't need grep here (the above is not strictly equivalent, as it yields the first occurrence, not all the occurrences but that should be okay on your case). And your clients may not have grep installed on Windows.

Hey I got the same problem as you. Sub-process can do what you want even with the shell=False. The trick is the communicate() method.
with subprocess.Popen(cmdCode,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
cwd = workingDir,
bufsize=1,
universal_newlines = True) as proc:
#output is stored in proc.stdout
#errors are stored in proc.stderr
Now you just need a little function to scan the proc.stdout for the information you need: PlatformName, etc

Combine grep with and & or logic

I am trying to convert the following code which is written in python to use cat | grep instead of opening a file.
The original code:
LOG_NAME="/raft/log/{}{}{}_{}_Rx_flow.log".format(NTIME.tm_year,str(NTIME.tm_mon).zfill(2),str(NTIME.tm_mday).zfill(2),str(NTIME.tm_hour).zfill(2))
print time.strftime("%Y/%m/%d %H:%M:%S") + " Log file name, RX restart, is: " + LOG_NAME
print time.strftime("%Y/%m/%d %H:%M:%S") + " ERRTIMEMIN value: " + ERRTIMEMIN + " RXRESTART message value: " + RXRESTART
LINK_LOG_FILE = open(LOG_NAME, "r")
ISRXRESTART=0
for REPline in LINK_LOG_FILE:
***if RXRESTART in REPline and (ERRTIMEMIN in REPline or ERRTIMEMIN1 in REPline) and ISRXRESTART==0:***
#Link restarted - correct behaviour.
print time.strftime("%Y/%m/%d %H:%M:%S") + " RX restarted - This is correct behaviour"
ISRXRESTART=1
I have to delete the line which opens the file and change the following line with the *** *** to something with cat and grep
for example:
os.popen("sshpass -p ssh root#"+self.ipaddr+" cat "+LOG_NAME+" | egrep `"+device_start+" "+ERRTIMEMIN+`").read().strip()
But I don't know how to combine or & and in the same grep

"OR" can be simulated by simply grepping multiple patterns at once:
egrep -E 'thing1|thing2' <file.txt
-E tells egrep to use extended regex.
To my knowledge there is no "AND" operator in grep but again can be simulated by grepping for patterns in both forward and backward order.
egrep -E 'thing1.*thing2|thing2.*thing1' <file.txt
Use -v for "NOT".
egrep -E 'thing1' <file.txt | egrep -v 'thing2'
This will find everything with "thing1" then grab only the stuff without "thing2".
Hope this helped.

Python exec grep

I'm trying to grep a list of file from the "*.nasl" of "Openvas" which contains a certain port's number.
I can make it directly in the terminal with the command :
egrep --only-match '111' /home/gwvm/Openvas/var/lib/openvas/plugins/*.nasl |cut -d ":" -f1
This command return all the name of the nasl file which contains 111.
like :
/home/gwvm/Openvas/var/lib/openvas/plugins/SolarWinds_TFTP.nasl:111
/home/gwvm/Openvas/var/lib/openvas/plugins/trojan_horses.nasl:111
and after the cut :
/home/gwvm/Openvas/var/lib/openvas/plugins/SolarWinds_TFTP.nasl
/home/gwvm/Openvas/var/lib/openvas/plugins/trojan_horses.nasl
But when I'm in python(3.1.3) the output give me an error :
egrep:/home/gwvm/Openvas/var/lib/openvas/plugins/*.nasl: No such file or directory
i was thinking about a problem because of the "*.nasl" but when I'm trying with an existing file, same result.
Here is the part of code :
command = ("egrep --only-match '"+ str(port[0]) +"' "+ openvas_directory["locate"]["nasl"] + '*.nasl' + ' |cut -d ":" -f1 ')
process=sp.Popen(command,shell=True, stdout= sp.PIPE)
or
exec(command)
I was thinking too of a bad construction but wen I'm printing the command it gives me what i want :
egrep --only-match '111' /home/gwvm/Openvas/var/lib/openvas/plugins/*.nasl |cut -d ":" -f1
If there are any idea!

from subprocess import PIPE, Popen
x = Popen('egrep --only-match \'111\' /home/gwvm/Openvas/var/lib/openvas/plugins/*.nasl', stdout=PIPE, stderr=PIPE, shell=True)
y = Popen('cut -d ":" -f1', stdin=x.stdout, stdout=PIPE, stderr=PIPE, shell=True)
for row in y.stdout.readline():
print row
Or just use check_output()
And this is btw how you | in Popen ;)
Guidelines:
When using Popen, if you supply a command as a string, use shell=True.
If you however supply Popen with a list ['ls, '-l'] then use shell=False, that's just how it works.
If you're piping data, execute two different Popen's and use the output from the first command as stdin for the second command, this is equivilant to doing | in Linux.

Error from stderr - how to fix it?

I have a Python program (below) and when I run it I get the following error:
% python SSH_Prog.py
About to connect...
stderr: ["bash: -c: line 0: unexpected EOF while looking for matching `''\n", 'bash: -c: line 1: syntax error: unexpected end of file\n']
pwd: []
stderr: ['watch: no process found\n']
pwd: []
^CTraceback (most recent call last):
File "SSH_Prog.py", line 32, in <module>
time.sleep(3)
KeyboardInterrupt
I think it is to do with escape sequence probably, and the "\n" character from stdin, but I lack the experience to deal with it.
Here's the program:
import os
import sys
import time
import paramiko
#from ssh import SSHClient
# Define remote machine
host="<ip>"
user="<usrnm>"
passw="<passw>"
client = paramiko.SSHClient()
#client.load_system_host_keys()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
# Try SSH connection, catch exception
#if not
print('About to connect...')
client.connect(host, username=user, password=passw)
# ForLoop to iterate through the interactions
for x in range(10):
xx = str(x)
# Commands to execute on local machine
f = os.popen3('tshark -i eth0 -f snmp -F pcapng -w ~/Desktop/traf_logs/n'+(xx))
# commands to execute on remote machine
stdin, stdout, stderr = client.exec_command("watch -n 0.1 'ps -p $(pgrep -d"," -x snmpd) -o rss= | awk '\''{ i += $1 } END { print i }'\'' >> ~/Desktop/mem_logs/mem_"+(xx)+";")
print "stderr: ", stderr.readlines()
print "pwd: ", stdout.readlines()
g = os.popen3('snmpget -v 2c -c communitystring <ip> sysContact.0')
time.sleep(3)
stdin, stdout, stderr = client.exec_command('killall watch;')
print "stderr: ", stderr.readlines()
print "pwd: ", stdout.readlines()
ff = os.popen3('killall tshark')
# terminate connection
client.close()
exit(0)
Do you have any idea to fix it?
Regards.

Your first exec_command looks like this:
stdin, stdout, stderr = client.exec_command("watch -n 0.1 'ps -p $(pgrep -d"," -x snmpd) -o rss= | awk '\''{ i += $1 } END { print i }'\'' >> ~/Desktop/mem_logs/mem_"+(xx)+";")
In other words, the first argument is:
"watch -n 0.1 'ps -p $(pgrep -d"
And your second argument is:
" -x snmpd) -o rss= | awk '\''{ i += $1 } END { print i }'\'' >> ~/Desktop/mem_logs/mem_"+(xx)+";"
If you fire up bash in a terminal and type that first argument (without the quotes), followed by a newline and a ^D, it'll tell you this:
> -bash: unexpected EOF while looking for matching `''
-bash: syntax error: unexpected end of file
Which is exactly what you're getting back from Paramiko.
And the second error is just killall telling you that there is no process named watch, because your first command never started one.
If you just replace the "," with a space, that'll solve that problem… but without knowing why you thought you wanted a "," there, I'm not sure it'll do what you actually were intending to do.
I'm also not sure what the '\'' is supposed to do. Why do you want to triple-quote the arguments to awk, or why you're doing something so complicated when it ought to be equivalent to just { print $1 }, or why you're explicitly asking ps for multiple columns just to use awk to pick out the first one, or…