Develop Reference

call shell command from python

I have this shell command that I want to call from python
frontend='test'
instance_list = subprocess.call(['sudo gcloud compute instances list | grep -v TERMINA | grep +'frontend'+ | awk '{ print $1 }''])
I'm getting this error
instance_list = subprocess.call(['sudo gcloud compute instances list | grep -v TERMINA | grep +'frontend'+ | awk '{ print $1 }''])
^
SyntaxError: invalid syntax
What I'm doing wrong?

How about this:
frontend='test'
instance_list = subprocess.call(['sudo gcloud compute instances list | grep -v TERMINA | grep '+frontend+' | awk \'{ print $1 }\''])
You just did the string concatenation wrong: the plus needs to be outside of the quotes...
And the quotes for awk probably need to be escaped...

You should put the plus signs outside the quotation marks:
instance_list = subprocess.call(['sudo gcloud compute instances list | grep -v TERMINA | grep '+frontend+' | awk '{ print $1 }''])

I Think the problem is in the way you arranged your quotations and concatenation sign (+)
Concatenation works like: 'Hello' + variable + 'world'
Or when it is about escaping quotes inside (Which is probably not your case)
You can use triple quotes like xxx.call([''' You are free to use single quotes inside here ''']);
For your case, this can help:
instance_list = subprocess.call(['sudo gcloud compute instances list | grep -v TERMINA | grep ' + frontend + ' | awk '{ print $1 }''])

Execute chained bash commands including multiple pipes and grep in Python3

I have to use the below bash command in a python script which includes multiple pip and grep commands.
grep name | cut -d':' -f2 | tr -d '"'| tr -d ','
I tried to do the same using subprocess module but didn't succeed.
Can anyone help me to run the above command in Python3 scripts?
I have to get the below output from a file file.txt.
Tom
Jack
file.txt contains:
"name": "Tom",
"Age": 10
"name": "Jack",
"Age": 15
Actually I want to know how can run the below bash command using Python.
cat file.txt | grep name | cut -d':' -f2 | tr -d '"'| tr -d ','

This works without having to use the subprocess library or any other os cmd related library, only Python.
my_file = open("./file.txt")
line = True
while line:
line = my_file.readline()
line_array = line.split()
try:
if line_array[0] == '"name":':
print(line_array[1].replace('"', '').replace(',', ''))
except IndexError:
pass
my_file.close()

If you not trying to parse a json file or any other structured file for which using a parser would be the best approach, just change your command into:
grep -oP '(?<="name":[[:blank:]]").*(?=",)' file.txt
You do not need any pipe at all.
This will give you the output:
Tom
Jack
Explanations:
-P activate perl regex for lookahead/lookbehind
-o just output the matching string not the whole line
Regex used: (?<="name":[[:blank:]]").*(?=",)
(?<="name":[[:blank:]]") Positive lookbehind: to force the constraint "name": followed by a blank char and then another double quote " the name followed by a double quote " extracted via (?=",) positive lookahead
demo: https://regex101.com/r/JvLCkO/1

Combine grep with and & or logic

I am trying to convert the following code which is written in python to use cat | grep instead of opening a file.
The original code:
LOG_NAME="/raft/log/{}{}{}_{}_Rx_flow.log".format(NTIME.tm_year,str(NTIME.tm_mon).zfill(2),str(NTIME.tm_mday).zfill(2),str(NTIME.tm_hour).zfill(2))
print time.strftime("%Y/%m/%d %H:%M:%S") + " Log file name, RX restart, is: " + LOG_NAME
print time.strftime("%Y/%m/%d %H:%M:%S") + " ERRTIMEMIN value: " + ERRTIMEMIN + " RXRESTART message value: " + RXRESTART
LINK_LOG_FILE = open(LOG_NAME, "r")
ISRXRESTART=0
for REPline in LINK_LOG_FILE:
***if RXRESTART in REPline and (ERRTIMEMIN in REPline or ERRTIMEMIN1 in REPline) and ISRXRESTART==0:***
#Link restarted - correct behaviour.
print time.strftime("%Y/%m/%d %H:%M:%S") + " RX restarted - This is correct behaviour"
ISRXRESTART=1
I have to delete the line which opens the file and change the following line with the *** *** to something with cat and grep
for example:
os.popen("sshpass -p ssh root#"+self.ipaddr+" cat "+LOG_NAME+" | egrep `"+device_start+" "+ERRTIMEMIN+`").read().strip()
But I don't know how to combine or & and in the same grep

"OR" can be simulated by simply grepping multiple patterns at once:
egrep -E 'thing1|thing2' <file.txt
-E tells egrep to use extended regex.
To my knowledge there is no "AND" operator in grep but again can be simulated by grepping for patterns in both forward and backward order.
egrep -E 'thing1.*thing2|thing2.*thing1' <file.txt
Use -v for "NOT".
egrep -E 'thing1' <file.txt | egrep -v 'thing2'
This will find everything with "thing1" then grab only the stuff without "thing2".
Hope this helped.

Error using shlex and subprocess [duplicate]

This question already has answers here:
How do I use subprocess.Popen to connect multiple processes by pipes?
(9 answers)
Closed 7 years ago.
Hi I am trying to run this command in python's subprocess with shlex split, however, I haven't found anything helpful for this particular case :
ifconfig | grep "inet " | grep -v 127.0.0.1 | grep -v 192.* | awk '{print $2}'
I get an with ifconfig error because the split with the single and double quotes and even the white space before the $ sign are not correct.
Please Help.

You can use shell=True (shell will interpret |) and triple quote string literal (otherwise you need to escape ", ' inside the string literal):
import subprocess
cmd = r"""ifconfig | grep "inet " | grep -v 127\.0\.0\.1 | grep -v 192\. | awk '{print $2}'"""
subprocess.call(cmd, shell=True)
or you can do it in harder way (Replacing shell pipeline from subprocess module documentation):
from subprocess import Popen, PIPE, call
p1 = Popen(['ifconfig'], stdout=PIPE)
p2 = Popen(['grep', 'inet '], stdin=p1.stdout, stdout=PIPE)
p3 = Popen(['grep', '-v', r'127\.0\.0\.1'], stdin=p2.stdout, stdout=PIPE)
p4 = Popen(['grep', '-v', r'192\.'], stdin=p3.stdout, stdout=PIPE)
call(['awk', '{print $2}'], stdin=p4.stdout)

Handling newlines within sed; command called from Python

I am having an issue with newlines in my command which involves the use of sed.
The scenario is as follows. When I execute the following command from Bash, I get:
cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed -e s/ph/\\nPH/g | grep -v ^$
PHysical id : 0core id : 0
PHysical id : 0core id : 1
As you can see, the sed command replaced ph with \nPH, such that I get a new line for each 'physical id...'
Now, I am calling this bash command from Python. Here is a small snippet of my code containing all relevant library imports.
import subprocess
cmd = 'cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed -e s/ph/\\nPH/g | grep -v ^$ '
subprocess.call(cmd, shell=True, universal_newlines=True, stdin=subprocess.PIPE)
The problem is that I get:
nPHysical id : 0core id : 0nPHysical id : 0core id : 1
on one line. It appears that the '\n' is not processed as the letter n is printed before PH.
I need to get the output nicely printed so that I can later add | sort | uniq | wc -l to my command to count the lines.
I would appreciate some help from the Bash-and-Python gurus out there.
Thank you.

Try:
cmd = r'cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed -e s/ph/\\nPH/g | grep -v ^$ '
The r means is a raw string. You might not be escaping some characters correctly.

Simply enclose the substitution command for sed with commas ("") to ensure it is passed correctly into python subprocess:
import subprocess
cmd = 'cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed -e "s/ph/\\nPH/g" | grep -v ^$ '
subprocess.call(cmd, shell=True, universal_newlines=True, stdin=subprocess.PIPE)
Output:
PHysical id : 0core id : 0
PHysical id : 0core id : 0