Develop Reference

How to call a series of bash commands in python and store output

I am trying to run the following bash script in Python and store the readlist output. The readlist that I want to be stored as a python list, is a list of all files in the current directory ending in *concat_001.fastq.
I know it may be easier to do this in python (i.e.
import os
readlist = [f for f in os.listdir(os.getcwd()) if f.endswith("concat_001.fastq")]
readlist = sorted(readlist)
However, this is problematic, as I need Python to sort the list in EXACTLY the same was as bash, and I was finding that bash and Python sort certain things in different orders (eg Python and bash deal with capitalised and uncapitalised things differently - but when I tried
readlist = np.asarray(sorted(flist, key=str.lower))
I still found that two files starting with ML_ and M_ were sorted in different order with bash and Python. Hence trying to run my exact bash script through Python, then to use the sorted list generated with bash in my subsequent Python code.
input_suffix="concat_001.fastq"
ender=`echo $input_suffix | sed "s/concat_001.fastq/\*concat_001.fastq/g" `
readlist="$(echo $ender)"
I have tried
proc = subprocess.call(command1, shell=True, stdout=subprocess.PIPE)
proc = subprocess.call(command2, shell=True, stdout=subprocess.PIPE)
proc = subprocess.Popen(command3, shell=True, stdout=subprocess.PIPE)
But I just get: subprocess.Popen object at 0x7f31cfcd9190
Also - I don't understand the difference between subprocess.call and subprocess.Popen. I have tried both.
Thanks,
Ruth

So your question is a little confusing and does not exactly explain what you want. However, I'll try to give some suggestions to help you update it, or in my effort, answer it.
I will assume the following: your python script is passing to the command line 'input_suffix' and that you want your python program to receive the contents of 'readlist' when the external script finishes.
To make our lives simpler, and allow things to be more complicated, I would make the following bash script to contain your commands:
script.sh
#!/bin/bash
input_suffix=$1
ender=`echo $input_suffix | sed "s/concat_001.fastq/\*concat_001.fastq/g"`
readlist="$(echo $ender)"
echo $readlist
You would execute this as script.sh "concat_001.fastq", where $1 takes in the first argument passed on the command line.
To use python to execute external scripts, as you quite rightly found, you can use subprocess (or as noted by another response, os.system - although subprocess is recommended).
The docs tell you that subprocess.call:
"Wait for command to complete, then return the returncode attribute."
and that
"For more advanced use cases when these do not meet your needs, use the underlying Popen interface."
Given you want to pipe the output from the bash script to your python script, let's use Popen as suggested by the docs. As I posted the other stackoverflow answer, it could look like the following:
import subprocess
from subprocess import Popen, PIPE
# Execute out script and pipe the output to stdout
process = subprocess.Popen(['script.sh', 'concat_001.fastq'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
# Obtain the standard out, and standard error
stdout, stderr = process.communicate()
and then:
>>> print stdout
*concat_001.fastq

How to redirect python OS system call to a file?

I have no idea why the below code is not working. The file arch_list does not get created or anything written to it. The commands work fine when run in the terminal alone.
from yum.plugins import PluginYumExit , TYPE_CORE, TYPE_INTERACTIVE
import os
requires_api_version = '2.3'
plugin_type = (TYPE_CORE, TYPE_INTERACTIVE)
ip_vm = ['192.168.239.133']
def get_arch():
global ip_vm
os.system("uname -p > ~/arch_list")
for i in ip_vm:
cmd = "ssh thejdeep#"+i+" 'uname -p' >> ~/arch_list"
print cmd
os.system(cmd)
def init_hook(conduit):
conduit.info(2,'Hello World !')
get_arch()

I don't think os.system() will return to stdout in that case. You may try using subprocess.call() with the appropriate parameters.
Edit: Actually I think I remember seeing similar behaviour with ssh when running in a standard bash loop. You might try adding a -n to your ssh call.. I think that is the solution I used years ago in bash.
I just ran your code and it works fine for me, writing to the local arch file. I suspect adding more than one host to your list is where you start having problems. What version of python are you running? I'm on 2.7.6.

os.system() will not redirect stdout and stderr.
You can use subprocess modules Popen to set the stdout and stderr to a file descriptor or a pipe.
For example:
>>> import subprocess
>>> child1 = subprocess.Popen(["ls","-l"], stdout=subprocess.PIPE)
>>> print child1.stdout.readlines()
You can replace subprocess.PIPE to any valid file descriptor you opened for write. or you could pick up some lines to the file. It's your call.

Interactively writing/reading from an external program

Backstory: I'm pretty new to Python, but fairly experienced in Perl. I'm trying to diversify my scripting portfolio in the sysadmin-activities area.
I'm attempting to dynamically communicate with an external process from my python script.
What I want to do is:
Call an executable (lets call it "cli")
Write a line to cli
Read the response back internally
Iterate through the response, one by one, and write another line to the CLI
Print the response from the cli back to the console
What I'm hoping this will yield is:
(spawn process) /usr/local/bin/cli
-> show listofobjects
<- (read back list of objects internally)
-> (one by one, write a line to the cli for each of the list of objects)
-> get objectname modifiedtime
<- (print response from above command)
Here is the code that I have so far:
import shlex, subprocess, re
clicmd = "/usr/local/bin/cli -s 10.1.213.226 -n Administrator -p password"
cliargs = shlex.split(clicmd)
cliproc = subprocess.Popen(cliargs,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
tmpclicmd = "LIST objects OUTPUT csv NAME"
cliret = cliproc.communicate(input=tmpclicmd)[0]
regex = re.compile('^\".*')
for line in cliret.split('\n'):
if regex.match(line):
procline = line.replace('"','')
if 'NAME' not in procline:
clideets = subprocess.Popen(cliargs,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
clideetscmd = 'modify objects ' + procline
clideets.communicate(input=clideetscmd)
clideetscmd = 'list objectdetails'
clideetsresp = clideets.communicate(input=clideetscmd)[0]
print clideetsresp;
I'm probably going about this in the completely wrong way. Do I have to spawn a new Popen for every step of the way? How could I do this better? (another module, etc). At the end of the day, I can't have the cli close on me, which Popen seems to do after each step of the way.
Thanks in advance!

It is not necessary to start a new process (using Popen) for every interaction. You do though, when you use communicate for sending data to the process, because, as the documentation states:
Interact with process: Send data to stdin. Read data from stdout and
stderr, until end-of-file is reached. Wait for process to terminate.
Instead, simply write to cliproc.stdin and read from cliproc.stdout:
cliproc.stdin.write("LIST objects OUTPUT csv NAME")
cliret = cliproc.stdout.readline()
The process keeps alive this way.
I don't know why you use the shlex module here:
clicmd = "/usr/local/bin/cli -s 10.1.213.226 -n Administrator -p password"
cliargs = shlex.split(clicmd)
The built-in str.split method will do fine:
clicmd = "/usr/local/bin/cli -s 10.1.213.226 -n Administrator -p password"
cliargs = clicmd.split()
Or you can just write the resulting list yourself:
cliargs = ["/usr/local/bin/cli", "-s", "10.1.213.226",
"-n", "Administrator", "-p", "password"]
You don't need a semicolon here:
print clideetsresp;

How to get PID via subprocess.Popen with custom environment variable?

Using Python, how can I run a subprocess with a modified environment variable and get its PID? I assume subprocess.Popen() is along the right track...
In shell (bash), I would do this:
MY_ENV_VAR=value ./program_name arg1 arg2 etc &
This runs program_name in the background, passing in "arg1" and "arg2" and "etc", with a modified environment variable, "MY_ENV_VAR" with a value of "value". The program program_name requires the environment variable MY_ENV_VAR to be set to the proper value.
How can do the equivalent thing in Python? I absolutely need the PID of the process. (My intent is to keep the python script running and performing checks on some of the things program_name is doing in the meantime, and I need the process ID to make sure it's still running.)
I've tried:
proc = subprocess.Popen(['MY_ENV_VAR=value', './program_name', 'arg1', 'arg2', 'etc'])
But of course, it expects the first item to be the program, not an environment variable.
Also tried:
environ = dict(os.environ)
environ['MY_ENV_VAR'] = 'value'
proc = subprocess.Popen(['./program_name', 'arg1', 'arg2', 'etc', env=environ])
Close, I suppose, but no cigar. Similarly, this:
environ = dict(os.environ)
environ['MY_ENV_VAR'] = 'value'
proc = subprocess.Popen(['echo', '$MY_ENV_VAR'], env=environ)
This echoes "$MY_ENV_VAR" literally, I suppose because there's no shell to interpret it. Okay, so I try the above but with this line instead:
proc = subprocess.Popen(['echo', '$MY_ENV_VAR'], env=environ, shell=True)
And that's fine and dandy, except that the value that's echoed is blank (doesn't apparently exist). And even if it did work, I'd get the PID of the shell, not the actual process I'm trying to launch.
I need to launch a process with a custom environment variable and get its PID (not the PID of the shell). Ideas?

Your last version is very close, but not quite there.
You don't want $MY_ENV_VAR to be an argument to echo. The echo program will have MY_ENV_VAR in its environment, but echo doesn't do any env variable expansion. You need it to be expanded by the shell, before it even gets to echo.
This may actually have nothing to do with your real-life test case. You already are getting the environment variable to the child process in all of your tests, it's just that echo doesn't do anything with that environment variable. If your real program just needs the environment variable to be set, you're done:
proc = subprocess.Popen(['./program_name', 'arg1', 'arg2', 'etc'], env=environ)
But if your program needs it to be substituted, like echo, then you have to substitute it into the arguments before they get passed to your program.
The easiest way to do that is to just give the shell a command line instead of a list of arguments:
proc = subprocess.Popen('echo "$MY_ENV_VAR"', env=environ, shell=True)
People will tell you that you should never use a command string in subprocess—but the reason for that is that you always want to prevent the shell from expanding variables, etc., in a way that could be insecure/etc. On the rare occasions when you want the shell to do its shelly things, you want a command string.
Of course if you use a shell, on most platforms, you're going to end up getting the PID of the shell rather than the PID of the actual program. Short of doing some platform-specific digging to enumerate the shell's children (or wrapping the whole thing in some simple sh code that gives you the child's PID indirectly), there's no way around that. The shell is what you're running.
Another alternative is to expand the variables in Python instead of making the shell do it. Then you don't even need a shell:
proc = subprocess.Popen(['echo', os.path.expandvars('$MY_ENV_VAR')])
… or, even more simply:
proc = subprocess.Popen(['echo', os.environ['MY_ENV_VAR']])

here's a program that spits out the current environment.
#!/usr/bin/env python
##program_name
import os
for k,v in os.environ.iteritems():
print k, '=', v
Here's a program that calls the other program, but first changes the environment
#!/usr/bin/env python
import subprocess, os
newenv = os.environ.copy()
newenv['MY_ENV_VAR'] = 'value'
args = ['./program_name', 'arg1', 'arg2', 'etc']
proc = subprocess.Popen(args, env=newenv)
pid = proc.pid
proc.wait()
print 'PID =', pid

How to get environment from a subprocess?

I want to call a process via a python program, however, this process need some specific environment variables that are set by another process. How can I get the first process environment variables to pass them to the second?
This is what the program look like:
import subprocess
subprocess.call(['proc1']) # this set env. variables for proc2
subprocess.call(['proc2']) # this must have env. variables set by proc1 to work
but the to process don't share the same environment. Note that these programs aren't mine (the first is big and ugly .bat file and the second a proprietary soft) so I can't modify them (ok, I can extract all that I need from the .bat but it's very combersome).
N.B.: I am using Windows, but I prefer a cross-platform solution (but my problem wouldn't happen on a Unix-like ...)

Here's an example of how you can extract environment variables from a batch or cmd file without creating a wrapper script. Enjoy.
from __future__ import print_function
import sys
import subprocess
import itertools
def validate_pair(ob):
try:
if not (len(ob) == 2):
print("Unexpected result:", ob, file=sys.stderr)
raise ValueError
except:
return False
return True
def consume(iter):
try:
while True: next(iter)
except StopIteration:
pass
def get_environment_from_batch_command(env_cmd, initial=None):
"""
Take a command (either a single command or list of arguments)
and return the environment created after running that command.
Note that if the command must be a batch file or .cmd file, or the
changes to the environment will not be captured.
If initial is supplied, it is used as the initial environment passed
to the child process.
"""
if not isinstance(env_cmd, (list, tuple)):
env_cmd = [env_cmd]
# construct the command that will alter the environment
env_cmd = subprocess.list2cmdline(env_cmd)
# create a tag so we can tell in the output when the proc is done
tag = 'Done running command'
# construct a cmd.exe command to do accomplish this
cmd = 'cmd.exe /s /c "{env_cmd} && echo "{tag}" && set"'.format(**vars())
# launch the process
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, env=initial)
# parse the output sent to stdout
lines = proc.stdout
# consume whatever output occurs until the tag is reached
consume(itertools.takewhile(lambda l: tag not in l, lines))
# define a way to handle each KEY=VALUE line
handle_line = lambda l: l.rstrip().split('=',1)
# parse key/values into pairs
pairs = map(handle_line, lines)
# make sure the pairs are valid
valid_pairs = filter(validate_pair, pairs)
# construct a dictionary of the pairs
result = dict(valid_pairs)
# let the process finish
proc.communicate()
return result
So to answer your question, you would create a .py file that does the following:
env = get_environment_from_batch_command('proc1')
subprocess.Popen('proc2', env=env)

As you say, processes don't share the environment - so what you literally ask is not possible, not only in Python, but with any programming language.
What you can do is to put the environment variables in a file, or in a pipe, and either
have the parent process read them, and pass them to proc2 before proc2 is created, or
have proc2 read them, and set them locally
The latter would require cooperation from proc2; the former requires that the variables become known before proc2 is started.

Since you're apparently in Windows, you need a Windows answer.
Create a wrapper batch file, eg. "run_program.bat", and run both programs:
#echo off
call proc1.bat
proc2
The script will run and set its environment variables. Both scripts run in the same interpreter (cmd.exe instance), so the variables prog1.bat sets will be set when prog2 is executed.
Not terribly pretty, but it'll work.
(Unix people, you can do the same thing in a bash script: "source file.sh".)

You can use Process in psutil to get the environment variables for that Process.
If you want to implement it yourself, you can refer to the internal implementation of psutil. It adapts to some operating system.
Currently supported operating systems are:
AIX
FreeBSD, OpenBSD, NetBSD
Linux
macOS
Sun Solaris
Windows
Eg: In Linux platform, you can find one pid 7877 environment variables in file /proc/7877/environ, just open with rt mode to read it.
Of course the best way to do this is to:
import os
from typing import Dict
from psutil import Process
process = Process(pid=os.getpid())
process_env: Dict = process.environ()
print(process_env)
You can find other platform implementation in source code
Hope I can help you.

The Python standard module multiprocessing have a Queues system that allow you to pass pickle-able object to be passed through processes. Also processes can exchange messages (a pickled object) using os.pipe. Remember that resources (e.g : database connection) and handle (e.g : file handles) can't be pickled.
You may find this link interesting :
Communication between processes with multiprocessing
Also the PyMOTw about multiprocessing worth mentioning :
multiprocessing Basics
sorry for my spelling

Two things spring to mind: (1) make the processes share the same environment, by combining them somehow into the same process, or (2) have the first process produce output that contains the relevant environment variables, that way Python can read it and construct the environment for the second process. I think (though I'm not 100% sure) that there isn't any way to get the environment from a subprocess as you're hoping to do.

Environment is inherited from the parent process. Set the environment you need in the main script, not a subprocess (child).