Develop Reference

KeyError: 'print $2' when I use subprocess.call("ps -ef | grep wget | grep {0} | awk '{print $2}'

In my macOS, I using a wget downloading the www.test.com index page.
then I have below python code:
#-*- coding:utf-8 -*-
import subprocess
url = "https://www.test.com"
subprocess.call("ps -ef | grep wget | grep {0} | awk '{print $2}'".format(url), shell=True)
when I run it, I get issue:
subprocess.call("ps -ef | grep wget | grep {0} | awk '{print $2}'".format(url), shell=True)
KeyError: 'print $2'
and I switch the subprocess.call() to os.system(), still get this issue.

Your error is not caused by subprocess, rather the string format.
"ps -ef | grep wget | grep {0} | awk '{print $2}'".format(url)
you can use %s to forming string.
"ps -ef | grep wget | grep %s | awk '{print $2}'"%(url)

You need to use subprocess.run()
From Docs
Code needing to capture stdout or stderr should use run() instead
Note : Do not use stdout=PIPE or stderr=PIPE with this function. The child process will block if it generates enough output to a pipe to fill up the OS pipe buffer as the pipes are not being read from.

Get source file path of a running python script from process id

I have a process running in the background, a python one, with ps -ef I can see filename from running command : UID PID PPID ... python ./filename.py
How can I know where the file is located

pwdx < PID > gives full directory the process is running from.
So, the full script would be
ps -ef | grep 'your process' | awk '{print $2}' | xargs pwdx
Though, you can simplify this into
pgrep 'your process' | awk '{print $1}' | xargs pwdx

using awk with subprocess Popen

import subprocess
gpus_raw=subprocess.Popen(['sinfo -pmain -Ogres:100,nodelist -N -h -r -tidle,mix,alloc|
grep -v drng | awk -F " " '{print $1,$2}''],
stdout=subprocess.PIPE,
shell=True).communicate()[0].strip().split('\n')
I use subprocess to generate a file with two columns after a grep and awk to remove the third colunn
but I get an error
File "/usr/bin/savail", line 10
gpus_raw = subprocess.Popen(['sinfo -pmain -Ogres:100,nodelist -N -h -r -tidle,mix,alloc|grep -v drng | awk -F " " '{print $1,$2}''], stdout=subprocess.PIPE,
shell=True).communicate()[0].strip().split('\n')
^ SyntaxError: invalid syntax

python subprocess.check_output giving wrong output

I have problem execute this code here
subprocess.check_output(['ps -ef | grep ftp | wc -l'],env=environ,shell=True)
When I execute from terminal
ps -ef | grep ftp | wc -l
I get "1" as output which is fine.
Now, I execute same code from my python files as subprocess.check_output and it gives me 2. That is strange. Any Ideas why is it happening. Here is the complete code:
def countFunction():
environ = dict(os.environ)
return subprocess.check_output(['ps -ef | grep ftp | wc -l'],env=environ,shell=True)
count = countFunction()
print count
EDIT:
Just to update , I do not have any ftp connections on.So command line is printing 1 on command which is fine.
Thanks
Arvind

The grep command will find itself:
$ ps -ef | grep ftp
wallyk 12546 12326 0 16:25 pts/3 00:00:00 grep ftp
If you don't want that, exclude the grep command:
$ ps -ef | grep ftp | grep -v ftp
$
It would be better to drop the -f switch to ps so that the command line arguments are not searched. That way, it won't find the grep ftp running:
$ ps -e | grep ftp | wc -l

python subprocess.Popen vs os.popen

I'm trying to get the output of the following shell command in my python script,
hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}
I can successfully get the output through os.popen as follows:
import os
cmd = "hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}"
p = os.popen(cmd,"r")
while 1:
line = p.readline()
if not line: break
print line
But os.popen() is deprecated since python 2.6 so I wanted to replace the above snippet with the subprocess.Popen() function.
But the code snippet for subprocess.Popen() below gives a different result than the code snippet above.
import subprocess as sub
import shlex
cmd = "hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}"
args = shlex.split(cmd)
p = sub.Popen(args,stdout=sub.PIPE,stderr=sub.PIPE)
output, errors = p.communicate()
print output
The above command just gives output for 'hadoop fs -ls /projectpath/' part of the command.
I have tried consulting several references (http://docs.python.org/2/library/subprocess.html#popen-objects, Python, os.system for command-line call (linux) not returning what it should?) for subpocess.Popen() but cannot get it to execute the command in the string cmd. Can anyone point out what I'm doing wrong?

try this:
cmd = "hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}"
p = sub.Popen(cmd,stdout=sub.PIPE,stderr=sub.PIPE, shell=True)