Develop Reference

Print Subprocess.Popen

I have problem with function Popen. I try retrieving the output from command which i used.
print(subprocess.Popen("dig -x 156.17.86.3 +short", shell=True, stdout=subprocess.PIPE).communicate()[0].decode('utf-8').strip())
This part working, but when I call the variable inside Popen (for adress in IP)
print(subprocess.Popen("dig -x ",Adres," +short", shell=True, stdout=subprocess.PIPE).communicate()[0].decode('utf-8').strip())
happens something like that:
raise TypeError("bufsize must be an integer")
I thought it would be problem with command so I used this solution:
command=['dig','-x',str(Adres),'+short']
print(subprocess.Popen(command, shell=True, stdout=subprocess.PIPE).communicate()[0].decode('utf-8').strip())
But now the return values is different than from console :
dig -x 156.17.4.20 +short
vpn.ii.uni.wroc.pl.
How I can print this the above name in script ?
Thank a lot of

The error is that you're not passing a single string, but multiple separate arguments:
subprocess.Popen("dig -x ",Adres," +short", shell=True, stdout=subprocess.PIPE)
If you look at the Popen constructor in the docs, that means you're passing "dig -x" as the args string, passing Adres as the bufsize, and passing "+short" as the executable. That's definitely not what you want.
You could fix this by building a string with concatenation or string formatting:
subprocess.Popen("dig -x " + str(Adres) + " +short", shell=True, stdout=subprocess.PIPE)
subprocess.Popen(f"dig -x {Adres} +short", shell=True, stdout=subprocess.PIPE)
However, a much better fix is to just not use the shell here, and pass the arguments as a list:
subprocess.Popen(['dig', '-x', Adres, '+short'], stdout=subprocess.PIPE)
Notice that if you do this, you have to remove the shell=True, or this won't work. (It may actually work on Windows, but not on *nix, and you shouldn't do it even on Windows.) In the edited version of your question, you're not doing that, so it's still wrong.
While we're at it, you really don't need to create a Popen object and communicate with it if that's literally all you're doing. A simpler solution is:
print(subprocess.run(['dig', '-x', Adres, '+short'], stdout=subprocess.PIPE).stdout.decode('utf-8'))
Also, if you're having problems debugging a complicated expression like yours, it really helps to break it into separate pieces that you can debug separately (with extra prints, or debugger breakpoints):
proc = subprocess.run(['dig', '-x', Adres, '+short'], stdout=subprocess.PIPE)
result = proc.stdout.decode('utf-8')
print(result)
This is essentially the same thing, with nearly the same efficiency, but easier to read and easier to debug.
And when I run this with Adres = '156.17.4.20', I get exactly the output you're looking for:
vpn.ii.uni.wroc.pl.

How to call a series of bash commands in python and store output

I am trying to run the following bash script in Python and store the readlist output. The readlist that I want to be stored as a python list, is a list of all files in the current directory ending in *concat_001.fastq.
I know it may be easier to do this in python (i.e.
import os
readlist = [f for f in os.listdir(os.getcwd()) if f.endswith("concat_001.fastq")]
readlist = sorted(readlist)
However, this is problematic, as I need Python to sort the list in EXACTLY the same was as bash, and I was finding that bash and Python sort certain things in different orders (eg Python and bash deal with capitalised and uncapitalised things differently - but when I tried
readlist = np.asarray(sorted(flist, key=str.lower))
I still found that two files starting with ML_ and M_ were sorted in different order with bash and Python. Hence trying to run my exact bash script through Python, then to use the sorted list generated with bash in my subsequent Python code.
input_suffix="concat_001.fastq"
ender=`echo $input_suffix | sed "s/concat_001.fastq/\*concat_001.fastq/g" `
readlist="$(echo $ender)"
I have tried
proc = subprocess.call(command1, shell=True, stdout=subprocess.PIPE)
proc = subprocess.call(command2, shell=True, stdout=subprocess.PIPE)
proc = subprocess.Popen(command3, shell=True, stdout=subprocess.PIPE)
But I just get: subprocess.Popen object at 0x7f31cfcd9190
Also - I don't understand the difference between subprocess.call and subprocess.Popen. I have tried both.
Thanks,
Ruth

So your question is a little confusing and does not exactly explain what you want. However, I'll try to give some suggestions to help you update it, or in my effort, answer it.
I will assume the following: your python script is passing to the command line 'input_suffix' and that you want your python program to receive the contents of 'readlist' when the external script finishes.
To make our lives simpler, and allow things to be more complicated, I would make the following bash script to contain your commands:
script.sh
#!/bin/bash
input_suffix=$1
ender=`echo $input_suffix | sed "s/concat_001.fastq/\*concat_001.fastq/g"`
readlist="$(echo $ender)"
echo $readlist
You would execute this as script.sh "concat_001.fastq", where $1 takes in the first argument passed on the command line.
To use python to execute external scripts, as you quite rightly found, you can use subprocess (or as noted by another response, os.system - although subprocess is recommended).
The docs tell you that subprocess.call:
"Wait for command to complete, then return the returncode attribute."
and that
"For more advanced use cases when these do not meet your needs, use the underlying Popen interface."
Given you want to pipe the output from the bash script to your python script, let's use Popen as suggested by the docs. As I posted the other stackoverflow answer, it could look like the following:
import subprocess
from subprocess import Popen, PIPE
# Execute out script and pipe the output to stdout
process = subprocess.Popen(['script.sh', 'concat_001.fastq'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
# Obtain the standard out, and standard error
stdout, stderr = process.communicate()
and then:
>>> print stdout
*concat_001.fastq

Repo command is not running using subprocess

I'm trying to run repo command using subprocess.check_call. I don't see any error but it's not running.
Here is my code.
def repo(*args):
return subprocess.check_call(['repo'] + list(args), shell = True, stdout = subprocess.PIPE, stderr = subprocess.PIPE)
repo('forall','-pc','"','git','merge','--strategy=ours','\${REPO_REMOTE}/branch_name','"','>log.log','2>&1')
Am I missing something?
Please help.
Thanks.

I'm going on a hunch, I guess you don't see anything because the error messages are stuck in your stderr pipe. Try this;
import subprocess
def repo(command):
subprocess.check_call('repo ' + command, shell=True)
repo('forall -pc "git merge --strategy=ours \${REPO_REMOTE}/branch_name" > log.log 2>&1')
Does that look more like you imagined? Also;
when using shell=True (although I don't recommend that your do) you can just pass a str (as opposed to a list); and
there is not pressing need for the return because check_call() either raise an exception or returns 0.

If you have shell=True and the first argument a sequence, as you have, then the first element in the sequence will be passed as option -c to the shell, and the rest of the elements will be additional arguments to the shell. Example
subprocess.check_call(['ls', '-l'], shell=True)
means the following is run:
sh -c "ls" -l
Note that ls doesn't get the option -l, but the shell sh does.
So, you should not use shell=True. If you have to, use a string instead of a list as args.
Also, the fine manual warns not to use stdout=PIPE and stderr=PIPE with check_call().

How to get PID via subprocess.Popen with custom environment variable?

Using Python, how can I run a subprocess with a modified environment variable and get its PID? I assume subprocess.Popen() is along the right track...
In shell (bash), I would do this:
MY_ENV_VAR=value ./program_name arg1 arg2 etc &
This runs program_name in the background, passing in "arg1" and "arg2" and "etc", with a modified environment variable, "MY_ENV_VAR" with a value of "value". The program program_name requires the environment variable MY_ENV_VAR to be set to the proper value.
How can do the equivalent thing in Python? I absolutely need the PID of the process. (My intent is to keep the python script running and performing checks on some of the things program_name is doing in the meantime, and I need the process ID to make sure it's still running.)
I've tried:
proc = subprocess.Popen(['MY_ENV_VAR=value', './program_name', 'arg1', 'arg2', 'etc'])
But of course, it expects the first item to be the program, not an environment variable.
Also tried:
environ = dict(os.environ)
environ['MY_ENV_VAR'] = 'value'
proc = subprocess.Popen(['./program_name', 'arg1', 'arg2', 'etc', env=environ])
Close, I suppose, but no cigar. Similarly, this:
environ = dict(os.environ)
environ['MY_ENV_VAR'] = 'value'
proc = subprocess.Popen(['echo', '$MY_ENV_VAR'], env=environ)
This echoes "$MY_ENV_VAR" literally, I suppose because there's no shell to interpret it. Okay, so I try the above but with this line instead:
proc = subprocess.Popen(['echo', '$MY_ENV_VAR'], env=environ, shell=True)
And that's fine and dandy, except that the value that's echoed is blank (doesn't apparently exist). And even if it did work, I'd get the PID of the shell, not the actual process I'm trying to launch.
I need to launch a process with a custom environment variable and get its PID (not the PID of the shell). Ideas?

Your last version is very close, but not quite there.
You don't want $MY_ENV_VAR to be an argument to echo. The echo program will have MY_ENV_VAR in its environment, but echo doesn't do any env variable expansion. You need it to be expanded by the shell, before it even gets to echo.
This may actually have nothing to do with your real-life test case. You already are getting the environment variable to the child process in all of your tests, it's just that echo doesn't do anything with that environment variable. If your real program just needs the environment variable to be set, you're done:
proc = subprocess.Popen(['./program_name', 'arg1', 'arg2', 'etc'], env=environ)
But if your program needs it to be substituted, like echo, then you have to substitute it into the arguments before they get passed to your program.
The easiest way to do that is to just give the shell a command line instead of a list of arguments:
proc = subprocess.Popen('echo "$MY_ENV_VAR"', env=environ, shell=True)
People will tell you that you should never use a command string in subprocess—but the reason for that is that you always want to prevent the shell from expanding variables, etc., in a way that could be insecure/etc. On the rare occasions when you want the shell to do its shelly things, you want a command string.
Of course if you use a shell, on most platforms, you're going to end up getting the PID of the shell rather than the PID of the actual program. Short of doing some platform-specific digging to enumerate the shell's children (or wrapping the whole thing in some simple sh code that gives you the child's PID indirectly), there's no way around that. The shell is what you're running.
Another alternative is to expand the variables in Python instead of making the shell do it. Then you don't even need a shell:
proc = subprocess.Popen(['echo', os.path.expandvars('$MY_ENV_VAR')])
… or, even more simply:
proc = subprocess.Popen(['echo', os.environ['MY_ENV_VAR']])

here's a program that spits out the current environment.
#!/usr/bin/env python
##program_name
import os
for k,v in os.environ.iteritems():
print k, '=', v
Here's a program that calls the other program, but first changes the environment
#!/usr/bin/env python
import subprocess, os
newenv = os.environ.copy()
newenv['MY_ENV_VAR'] = 'value'
args = ['./program_name', 'arg1', 'arg2', 'etc']
proc = subprocess.Popen(args, env=newenv)
pid = proc.pid
proc.wait()
print 'PID =', pid

Python Subprocess can not get the output of oracle command "imp"

For example:
import subprocess
p=subprocess.Popen("imp -help",stdout=subprocess.PIPE,stdin=subprocess.PIPE)
out,err=p.communicate
the out is null
but other oracle command like "sqlplus -help","rman -help" works fine

There could be two problems why you are not getting any output in stdout:
The process is dumping all it's output to stderr.
The system does not know how to execute "imp -help".
The solution for the first problem is easy: capture stderr using the argument stderr = subprocess.PIPE.
The solution to the second is also easy, but the explanation is a bit longer: Subprocess does not guess much, it will just try to execute the whole string as one command. That means, in your case, it will try to execute "imp -help" as one command. It does not try to execute the command "imp" with the argument "-help". You have to explicitly tell subprocess the command and the arguments separately.
From the python documentation on subprocess:
args should be a string, or a sequence
of program arguments. The program to
execute is normally the first item in
the args sequence or the string if a
string is given, ...
That means you have to separate the command and the arguments and pack them together in a sequence. This: "imp -help" should look like this: ["imp", "-help"]. Read the documentation on subprocess for more details on the intricacies of spliting the command and arguments.
Here is how the code should look like:
import subprocess
p=subprocess.Popen(["imp", "-help"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
out,err=p.communicate()
Note: you also typed p.communicate instead of p.communicate(). I assume that was a typo in your question, not in your code.