Scenario
In my python script I need to run an executable file as a subprocess with x number of command line parameters which the executable is expecting.
Example:
EG 1: myexec.sh param1 param2
EG 2: myexec.sh param1 $MYPARAMVAL
The executable and parameters are not known as these are configured and retrieved from external source (xml config) at run time.
My code is working when the parameter is a known value (EG 1) and configured, however the expectation is that a parameter could be an environment variable and configured as such, which should be interpreted at run time.(EG 2)
In the example below I am using echo as a substitute for myexec.sh to demonstrate the scenario.
This is simplified to demonstrate issue. 'cmdlst' is built from a configuration file, which could be any script with any number of parameters and values which could be a value or environment variable.
test1.py
import subprocess
import os
cmdlst = ['echo','param1','param2']
try:
proc = subprocess.Popen(cmdlst,stdout=subprocess.PIPE)
jobpid = proc.pid
stdout_value, stderr_value = proc.communicate()
except (OSError, subprocess.CalledProcessError) as err:
raise
print stdout_value
RESULT TEST 1
python test1.py
--> param1 param2
test2.py
import subprocess
import os
cmdlst = ['echo','param1','$PARAM']
try:
proc = subprocess.Popen(cmdlst,stdout=subprocess.PIPE)
jobpid = proc.pid
stdout_value, stderr_value = proc.communicate()
except (OSError, subprocess.CalledProcessError) as err:
raise
print stdout_value
RESULT TEST 2
export PARAM=param2
echo $PARAM
--> param2
python test2.py
--> param1 $PARAM
I require Test 2 to produce the same result as Test 1, considering that $PARAM would only be known at run-time and need to be retrieved from the current environment.
I welcome your advice.
If you want to have the shell expand environment variables, you have to set shell=True
subprocess.Popen('echo param1 $PARAM', shell=True, stdout=subprocess.PIPE)
Alternatively, you could just query the environment variable yourself when constructing the command, and then there is no need for shell expansion
subprocess.Popen(['echo', 'param1', os.environ['PARAM']], stdout=subprocess.PIPE)
You could do:
cmdlist = ['echo','param',os.environ["PARAM"]]
Or:
cmdlist = ['echo','param1','$PARAM']
proc = subprocess.Popen(cmdlist,stdout=subprocess.PIPE, env={'PARAM':os.environ['PARAM'])
You can not access the environment variable using $VAR. You'd use os.environ[..] instead:
cmdlst = ['echo','param1',os.environ['PARAM']]
Related
I have written a C code where I have converted one file format to another file format. To run my C code, I have taken one command line argument : filestem.
I executed that code using : ./executable_file filestem > outputfile
Where I have got my desired output inside outputfile
Now I want to take that executable and run within a python code.
I am trying like :
import subprocess
import sys
filestem = sys.argv[1];
subprocess.run(['/home/dev/executable_file', filestem , 'outputfile'])
But it is unable to create the outputfile. I think some thing should be added to solve the > issue. But unable to figure out. Please help.
subprocess.run has optional stdout argument, you might give it file handle, so in your case something like
import subprocess
import sys
filestem = sys.argv[1]
with open('outputfile','wb') as f:
subprocess.run(['/home/dev/executable_file', filestem],stdout=f)
should work. I do not have ability to test it so please run it and write if it does work as intended
You have several options:
NOTE - Tested in CentOS 7, using Python 2.7
1. Try pexpect:
"""Usage: executable_file argument ("ex. stack.py -lh")"""
import pexpect
filestem = sys.argv[1]
# Using ls -lh >> outputfile as an example
cmd = "ls {0} >> outputfile".format(filestem)
command_output, exitstatus = pexpect.run("/usr/bin/bash -c '{0}'".format(cmd), withexitstatus=True)
if exitstatus == 0:
print(command_output)
else:
print("Houston, we've had a problem.")
2. Run subprocess with shell=true (Not recommended):
"""Usage: executable_file argument ("ex. stack.py -lh")"""
import sys
import subprocess
filestem = sys.argv[1]
# Using ls -lh >> outputfile as an example
cmd = "ls {0} >> outputfile".format(filestem)
result = subprocess.check_output(shlex.split(cmd), shell=True) # or subprocess.call(cmd, shell=True)
print(result)
It works, but python.org frowns upon this, due to the chance of a shell injection: see "Security Considerations" in the subprocess documentation.
3. If you must use subprocess, run each command separately and take the SDTOUT of the previous command and pipe it into the STDIN of the next command:
p = subprocess.Popen(cmd, stdin=PIPE, stdout=PIPE)
stdout_data, stderr_data = p.communicate()
p = subprocess.Popen(cmd, stdin=stdout_data, stdout=PIPE)
etc...
Good luck with your code!
I am using the subprocess.call to execute a shell script from another application I am integrating with. This script sets environment variables with export MY_VAR=foo. Next, I need to execute more commands over subprocess with the environment that was set by the shell script.
How to extract the state of environment from the child process? It only returns the errno code.
i.e. I want to run:
subprocess.call(["export", "MY_VAR=foo"]
subprocess.call(["echo", "$MY_VAR"]) # should print 'foo'.
I know that I can set environment with env keyword, but the point of my question is how to get the environment variables that a subprocess sets. In shell you can source any script to get it's declared environment variables. What's the alternative in python?
I ran into this issue just recently. It seems that this is a difficult problem for reasons upstream of Python: posix_spawn doesn't give a way to read the environment variables of the spawned process, nor is there any easy way to read the environment of a running process.
Bash's source is specific to running bash code in the bash interpreter: it just evals the file in the current bash interpreter rather than starting a subprocess. This mechanism can't work if you are running bash code from Python.
It is possible to make a separate mechanism specific to running bash code from Python. The following is the best that I could manage. Would be nice to have a less flimsy solution.
import json
import os
import subprocess
import sys
from contextlib import AbstractContextManager
class BashRunnerWithSharedEnvironment(AbstractContextManager):
"""Run multiple bash scripts with persisent environment.
Environment is stored to "env" member between runs. This can be updated
directly to adjust the environment, or read to get variables.
"""
def __init__(self, env=None):
if env is None:
env = dict(os.environ)
self.env: Dict[str, str] = env
self._fd_read, self._fd_write = os.pipe()
def run(self, cmd, **opts):
if self._fd_read is None:
raise RuntimeError("BashRunner is already closed")
write_env_pycode = ";".join(
[
"import os",
"import json",
f"os.write({self._fd_write}, json.dumps(dict(os.environ)).encode())",
]
)
write_env_shell_cmd = f"{sys.executable} -c '{write_env_pycode}'"
cmd += "\n" + write_env_shell_cmd
result = subprocess.run(
["bash", "-ce", cmd], pass_fds=[self._fd_write], env=self.env, **opts
)
self.env = json.loads(os.read(self._fd_read, 5000).decode())
return result
def __exit__(self, exc_type, exc_value, traceback):
if self._fd_read:
os.close(self._fd_read)
os.close(self._fd_write)
self._fd_read = None
self._fd_write = None
def __del__(self):
self.__exit__(None, None, None)
Example:
with BashRunnerWithSharedEnvironment() as bash_runner:
bash_runner.env.pop("A", None)
res = bash_runner.run("A=6; echo $A", stdout=subprocess.PIPE)
assert res.stdout == b'6\n'
assert bash_runner.env.get("A", None) is None
bash_runner.run("export A=2")
assert bash_runner.env["A"] == "2"
res = bash_runner.run("echo $A", stdout=subprocess.PIPE)
assert res.stdout == b'2\n'
res = bash_runner.run("A=6; echo $A", stdout=subprocess.PIPE)
assert res.stdout == b'6\n'
assert bash_runner.env.get("A", None) == "6"
bash_runner.env["A"] = "7"
res = bash_runner.run("echo $A", stdout=subprocess.PIPE)
assert res.stdout == b'7\n'
assert bash_runner.env["A"] == "7"
It is not possible, because the environment is changed only in the child process. You might from there return it as output to STDOUT, STDERR - but as soon as the subprocess is terminated, You can not access anything from it.
# this is process #1
subprocess.call(["export", "MY_VAR=foo"]
# this is process #2 - it can not see the environment of process #1
subprocess.call(["echo", "$MY_VAR"]) # should print 'foo'.
Not sure I see the problem here. You just need to remember the following:
each subprocess that gets started is independent of any setups done in previous subprocesses
if you want to set up some variables and use them, do both those things in ONE process
So make setupVars.sh like this:
export vHello="hello"
export vDate=$(date)
export vRandom=$RANDOM
And make printVars.sh like this:
#!/bin/bash
echo $vHello, $vDate, $vRandom
And make that executable with:
chmod +x printVars.sh
Now your Python looks like this:
import subprocess
subprocess.call(["bash","-c","source setupVars.sh; ./printVars.sh"])
Output
hello, Mon Jul 12 00:32:29 BST 2021, 8615
I have a cmd file "file.cmd" containing 100s of lines of command.
Example
pandoc --extract-media -f docx -t gfm "sample1.docx" -o "sample1.md"
pandoc --extract-media -f docx -t gfm "sample2.docx" -o "sample2.md"
pandoc --extract-media -f docx -t gfm "sample3.docx" -o "sample3.md"
I am trying to run these commands using a script so that I don't have to go to a file and click on it.
This is my code, and it results in no output:
file1 = open('example.cmd', 'r')
Lines = file1.readlines()
# print(Lines)
for i in Lines:
print(i)
os.system(i)
You don't need to read the cmd file line by line. you can simply try the following:
import os
os.system('myfile.cmd')
or using the subprocess module:
import subprocess
p = subprocess.Popen(['myfile.cmd'], shell = True, close_fds = True)
stdout, stderr = proc.communicate()
Example:
myfile.cmd:
#ECHO OFF
ECHO Grettings From Python!
PAUSE
script.py:
import os
os.system('myfile.cmd')
The cmd will open with:
Greetings From Python!
Press any key to continue ...
You can debug the issue by knowing the return exit code by:
import os
return_code=os.system('myfile.cmd')
assert return_code == 0 #asserts that the return code is 0 indicating success!
Note: os.system works by calling system() in C can only take up to 65533 arguments after a command (so it is a 16 bit issue). Giving one more argument will result in the return code 32512 (which implies the exit code 127).
The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function (os.system('command')).
since it is a command file (cmd), and only the shell can run it, then shell argument must set to be true. since you are setting the shell argument to true, the command needs to be string form and not a list.
use the Popen method for spawn a new process and the communicte for waiting on that process (you can time it out as well). if you whish to communicate with the child process, provide the PIPES (see mu example, but you dont have to!)
the code below for python 3.3 and beyond
import subprocess
try:
proc=subprocess.Popen('myfile.cmd', shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
outs, errs = proc.communicate(timeout=15) #timing out the execution, just if you want, you dont have to!
except TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
for older python versions
proc = subprocess.Popen('myfile.cmd', shell=True)
t=10
while proc.poll() is None and t >= 0:
print('Still waiting')
time.sleep(1)
t -= 1
proc.kill()
In both cases (python versions) if you dont need the timeout feature and you dont need to interact with the child process, then just, use:
proc = subprocess.Popen('myfile.cmd', shell=True)
proc.communicate()
I am running the following piece of python code (runs a command in a shell and grabs its output or reports an error)
import sys
import subprocess
def check_output(args, communicate=None, quiet=False, **kwargs):
for stream in ["stdout", "stderr"]:
kwargs.setdefault(stream, subprocess.PIPE)
proc = subprocess.Popen(args, **kwargs)
try:
out, err = proc.communicate()
finally:
for f in (proc.stdout, proc.stderr):
if f is not None:
f.close()
proc.wait()
if kwargs["stderr"] != subprocess.PIPE:
err = ""
if proc.returncode != 0:
raise Exception(args, proc.returncode, err)
else:
if not quiet:
sys.stderr.write(err)
sys.stderr.flush()
return out
with the following arguments:
env = dict(
PGHOST='{pg_host}',
PGPORT='{pg_port}',
PGDATABASE='{pg_dbname}',
PGUSER='{pg_user}',
PGPASSWORD='{pg_password}',
)
cmd = ['psql', '-c', "INSERT INTO {ft_geom} SELECT * FROM {ft_geom_in};"].format(**tables)
check_output(cmd, shell=True, env=env)
Here env simply contains the PG[HOST|USER|DATABASE|PORT|..] environment variables and tables contains just the names of those 2 tables. When I run this code, it hangs indefinitely on proc = subprocess.Popen call. I am using python 2.6.5 on Ubuntu 10.04.3 LTS
I check that no tables are locked with the following:
SELECT a.datname,
c.relname,
l.transactionid,
l.mode,
l.granted,
a.usename,
a.current_query,
a.query_start,
age(now(), a.query_start) AS "age",
a.procpid
FROM pg_stat_activity a
JOIN pg_locks l ON l.pid = a.procpid
JOIN pg_class c ON c.oid = l.relation
ORDER BY a.query_start;
And it shows that all the locks were granted. Not sure where else to look at. I need shell=True because the commands are sometimes more complex, requiring bash pipes. I know I should ideally pass stdout.PIPE of one command to the other, but its not possible to change this at the moment.
Running the very same command from bash directly works as expected, also, running without shell=True works
The problem was that I didn't filter newline characters in tokens passed to check_output, then shell would hang waiting for more input. So make sure you escape the tokens when you pass them to shell.
I'm attempting to call an outside program from my python application, but it shows no output and fails with error 127. Executing the command from the command line works fine. (and I am in the correct working directory)
def buildContris (self, startUrl, reportArray):
urls = []
for row in reportArray:
try:
url = subprocess.check_output(["casperjs", "casper.js", startUrl, row[0]], shell=True)
print (url)
urls.append(url)
break
except subprocess.CalledProcessError as e:
print ("Error: " + str(e.returncode) + " Output:" + e.output.decode())
return urls
Each loop outputs the following error: (I've also checked e.cmd. It's correct, but long, so I omitted it in this example)
Error: 127 Output:
SOLUTION:
The following code works
app = subprocess.Popen(["./casperjs/bin/casperjs", "casper.js", startUrl, row[0]], stdout=subprocess.PIPE, stderr=subprocess.PIPE, env = {"PATH" : "/usr/local/bin/:/usr/bin"}, universal_newlines=True)
app.wait()
out, errs = app.communicate()
Try adding the full path to casperjs in your subprocess.check_output() call.
Edit: Answeing your 2nd question. My apologies for the formatting as I'm on iPad.
I think you should try Popen instead of check_output so that you can specify environment variables:
p = subprocess.Popen(["/path/to/casperjs", "casper.js", startUrl, row[0]], env={"PATH": "/path/to/phantomjs"})
url, err = p.communicate()
shell=True changes the interpretation of the first argument (args) in check_output() call, from the docs:
On Unix with shell=True, ... If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
Exit status 127 might mean that the shell haven't found casperjs program or casperjs itself exited with that code.
To fix the code: drop shell=True and specify the full path to the casperjs program e.g.:
url = check_output(["./casperjs", "casper.js", startUrl, row[0]])
Try to add explicitly the path in this way.
If the file to call is in the same path (change __file__ if not):
cwd=os.path.dirname(os.path.realpath(__file__))
a = subprocess.check_output(["./casper.js", startUrl, row[0]],cwd=cwd,shell=True)
If you're experiencing this kinda nonsense on macOS: don't use aliases. Lost half a day with that. So, change:
subprocess.check_output(
"scribus-ng -g -ns -py {0} {1}".format(script_path, id),
stderr=subprocess.STDOUT,
shell=True)
to
subprocess.check_output(
"/Applications/Scribus.app/Contents/MacOS/Scribus -g -ns -py {0} {1}".format(script_path, id),
stderr=subprocess.STDOUT,
shell=True)