Develop Reference

SubProcess: Python Shell to OpenSSL drops into OpenSSL prompt instead if processing given command line arguments (Windows)

I am using Python subprocess module but the results are unexpected. I have successfully determined the correct command line strings (using another language) but cannot get a Python equivalent to work.
So I wrote some code in VBA which uses a Windows COM type library to spawn process and capture stdout, stderr. Using this I managed to write my program which generates elliptic curve keys and sign a file etc. An example command line which would run fine would be this
C:\Progra~1\OpenSSL-Win64\bin\openssl.exe ecparam -genkey -name secp384r1 -out n:\ECDSA\2017-11-30T203401\ec_key.pem
I am trying to convert the code to Python so that it runs on a linux box. I am using the same strategy in that I am spawning shells (because i have figured out all the right arguments in the VBA equivalent). I am not using a Python API like pyOpenSSL.
So I am using the subprocess module. I have never used this and am having trouble, initially I couldn't get the thing to work at all. I have managed to get something working but it looks like instead of executing the whole statement and generating a file (in the case given) it is instead dropping me into the OpenSSL prompt. That is to say it is not processing the arguments correctly. Here is my Python code
import os
import subprocess
import sys
class Expando(object):
pass
def RunShellAndWait(sExe, aArgs):
print(sExe + ' ' + ' '.join(aArgs))
# https://stackoverflow.com/questions/4760215/running-shell-command-from-python-and-capturing-the-output#answer-4760517
if (sys.platform[:3]=='win'):
suinfo = subprocess.STARTUPINFO()
suinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
SW_SHOWMAXIMIZED = 3
suinfo.wShowWindow = SW_SHOWMAXIMIZED
try:
result = subprocess.run(args=aArgs,executable=sExe,stdin=subprocess.PIPE,stdout=subprocess.PIPE, stderr=subprocess.PIPE,startupinfo=suinfo)
except:
print( "something went wrong with subprocess.run \r\nexec " + sExe + ' ' + ' '.join(aArgs))
else:
suinfo = None
retVal=Expando()
retVal.returnCode = result.returncode
retVal.sStdOut = result.stdout.decode('utf-8')
retVal.sStdErr = result.stderr.decode('utf-8')
return retVal
def OpenSSL_GenECDSAKeys(sBatchDir, sEcdsaKeyFile, sEcdsaPublicKeyDerFile,sEcdsaPublicKeyPemFile):
if os.path.isfile(sOPENSSL_BIN)!=True:
print("Cannot find OpenSSL.exe! " + sOPENSSL_BIN)
return false
if os.path.isfile(sEcdsaKeyFile)==True:
os.removefile (sEcdsaKeyFile)
try:
aGenKeyArgs = [' ecparam -genkey -name secp384r1 -out ' + sEcdsaKeyFile]
aGenKeyStatus = RunShellAndWait(sOPENSSL_BIN, aGenKeyArgs)
...
I concude that it is dropping into interactive mode because with stdout capture says "OpenSSL>" which is OpenSLL interactive prompt, i.e. I can just run OpenSSL.exe on its own and I get that prompt. Also the expected file is not generated.
I tried experiment with Popen but it did not like it. I am hoping someone has trodden this path before me and can point out a pitfall.
So the question is how to get OpenSSL to acknowledge and process the given arguments instead of dropping into interactive mode.

Import bash variables from a python script

I have seen plenty examples of running a python script from inside a bash script and either passing in variables as arguments or using export to give the child shell access, I am trying to do the opposite here though.
I am running a python script and have a separate file, lets call it myGlobalVariables.bash
myGlobalVariables.bash:
foo_1="var1"
foo_2="var2"
foo_3="var3"
My python script needs to use these variables.
For a very simple example:
myPythonScript.py:
print "foo_1: {}".format(foo_1)
Is there a way I can import them directly? Also, I do not want to alter the bash script if possible since it is a common file referenced many times elsewhere.

If your .bash file is formatted as you indicated - you might be able to just import it direct as a Python module via the imp module.
import imp
bash_module = imp.load_source("bash_module, "/path/to/myGlobalVariables.bash")
print bash_module.foo_1

You can also use os.environ:
Bash:
#!/bin/bash
# works without export as well
export testtest=one
Python:
#!/usr/bin/python
import os
os.environ['testtest'] # 'one'

I am very new to python, so I would welcome suggestions for more idiomatic ways to do this, but the following code uses bash itself to tell us which values get set by first calling bash with an empty environment (env -i bash) to tell us what variables are set as a baseline, then I call it again and tell bash to source your "variables" file, and then tell us what variables are now set. After removing some false-positives and an apparently-blank line, I loop through the "additional" output, looking for variables that were not in the baseline. Newly-seen variables get split (carefully) and put into the bash dictionary. I've left here (but commented-out) my previous idea for using exec to set the variables natively in python, but I ran into quoting/escaping issues, so I switched gears to using a dict.
If the exact call (path, etc) to your "variables" file is different than mine, then you'll need to change all of the instances of that value -- in the subprocess.check_output() call, in the list.remove() calls.
Here's the sample variable file I was using, just to demonstrate some of the things that could happen:
foo_1="var1"
foo_2="var2"
foo_3="var3"
if [[ -z $foo_3 ]]; then
foo_4="test"
else
foo_4="testing"
fi
foo_5="O'Neil"
foo_6='I love" quotes'
foo_7="embedded
newline"
... and here's the python script:
#!/usr/bin/env python
import subprocess
output = subprocess.check_output(['env', '-i', 'bash', '-c', 'set'])
baseline = output.split("\n")
output = subprocess.check_output(['env', '-i', 'bash', '-c', '. myGlobalVariables.bash; set'])
additional = output.split("\n")
# these get set when ". myGlobal..." runs and so are false positives
additional.remove("BASH_EXECUTION_STRING='. myGlobalVariables.bash; set'")
additional.remove('PIPESTATUS=([0]="0")')
additional.remove('_=myGlobalVariables.bash')
# I get an empty item at the end (blank line from subprocess?)
additional.remove('')
bash = {}
for assign in additional:
if not assign in baseline:
name, value = assign.split("=", 1)
bash[name]=value
#exec(name + '="' + value + '"')
print "New values:"
for key in bash:
print "Key: ", key, " = ", bash[key]
Another way to do it:
Inspired by Marat's answer, I came up with this two-stage hack. Start with a python program, let's call it "stage 1", which uses subprocess to call bash to source the variable file, as my above answer does, but it then tells bash to export all of the variables, and then exec the rest of your python program, which is in "stage 2".
Stage 1 python program:
#!/usr/bin/env python
import subprocess
status = subprocess.call(
['bash', '-c',
'. myGlobalVariables.bash; export $(compgen -v); exec ./stage2.py'
]);
Stage 2 python program:
#!/usr/bin/env python
# anything you want! for example,
import os
for key in os.environ:
print key, " = ", os.environ[key]

As stated in #theorifice answer, the trick here may be that such formatted file may be interpreted by both as bash and as python code. But his answer is outdated. imp module is deprecated in favour of importlib.
As your file has extension other than ".py", you can use the following approach:
from importlib.util import spec_from_loader, module_from_spec
from importlib.machinery import SourceFileLoader
spec = spec_from_loader("foobar", SourceFileLoader("foobar", "myGlobalVariables.bash"))
foobar = module_from_spec(spec)
spec.loader.exec_module(foobar)
I do not completely understand how this code works (where there are these foobar parameters), however, it worked for me. Found it here.

How do you "Save as" /"open" a document using a Python Tkinter interface [duplicate]

I need to be able to open a document using its default application in Windows and Mac OS. Basically, I want to do the same thing that happens when you double-click on the document icon in Explorer or Finder. What is the best way to do this in Python?

Use the subprocess module available on Python 2.4+, not os.system(), so you don't have to deal with shell escaping.
import subprocess, os, platform
if platform.system() == 'Darwin': # macOS
subprocess.call(('open', filepath))
elif platform.system() == 'Windows': # Windows
os.startfile(filepath)
else: # linux variants
subprocess.call(('xdg-open', filepath))
The double parentheses are because subprocess.call() wants a sequence as its first argument, so we're using a tuple here. On Linux systems with Gnome there is also a gnome-open command that does the same thing, but xdg-open is the Free Desktop Foundation standard and works across Linux desktop environments.

open and start are command-interpreter things for Mac OS/X and Windows respectively, to do this.
To call them from Python, you can either use subprocess module or os.system().
Here are considerations on which package to use:
You can call them via os.system, which works, but...
Escaping: os.system only works with filenames that don't have any spaces or other shell metacharacters in the pathname (e.g. A:\abc\def\a.txt), or else these need to be escaped. There is shlex.quote for Unix-like systems, but nothing really standard for Windows. Maybe see also python, windows : parsing command lines with shlex
MacOS/X: os.system("open " + shlex.quote(filename))
Windows: os.system("start " + filename) where properly speaking filename should be escaped, too.
You can also call them via subprocess module, but...
For Python 2.7 and newer, simply use
subprocess.check_call(['open', filename])
In Python 3.5+ you can equivalently use the slightly more complex but also somewhat more versatile
subprocess.run(['open', filename], check=True)
If you need to be compatible all the way back to Python 2.4, you can use subprocess.call() and implement your own error checking:
try:
retcode = subprocess.call("open " + filename, shell=True)
if retcode < 0:
print >>sys.stderr, "Child was terminated by signal", -retcode
else:
print >>sys.stderr, "Child returned", retcode
except OSError, e:
print >>sys.stderr, "Execution failed:", e
Now, what are the advantages of using subprocess?
Security: In theory, this is more secure, but in fact we're needing to execute a command line one way or the other; in either environment, we need the environment and services to interpret, get paths, and so forth. In neither case are we executing arbitrary text, so it doesn't have an inherent "but you can type 'filename ; rm -rf /'" problem, and if the file name can be corrupted, using subprocess.call gives us little additional protection.
Error handling: It doesn't actually give us any more error detection, we're still depending on the retcode in either case; but the behavior to explicitly raise an exception in the case of an error will certainly help you notice if there is a failure (though in some scenarios, a traceback might not at all be more helpful than simply ignoring the error).
Spawns a (non-blocking) subprocess: We don't need to wait for the child process, since we're by problem statement starting a separate process.
To the objection "But subprocess is preferred." However, os.system() is not deprecated, and it's in some sense the simplest tool for this particular job. Conclusion: using os.system() is therefore also a correct answer.
A marked disadvantage is that the Windows start command requires you to pass in shell=True which negates most of the benefits of using subprocess.

I prefer:
os.startfile(path, 'open')
Note that this module supports filenames that have spaces in their folders and files e.g.
A:\abc\folder with spaces\file with-spaces.txt
(python docs) 'open' does not have to be added (it is the default). The docs specifically mention that this is like double-clicking on a file's icon in Windows Explorer.
This solution is windows only.

Just for completeness (it wasn't in the question), xdg-open will do the same on Linux.

import os
import subprocess
def click_on_file(filename):
'''Open document with default application in Python.'''
try:
os.startfile(filename)
except AttributeError:
subprocess.call(['open', filename])

If you have to use an heuristic method, you may consider webbrowser.
It's standard library and despite of its name it would also try to open files:
Note that on some platforms, trying to open a filename using this
function, may work and start the operating system’s associated
program. However, this is neither supported nor portable.
(Reference)
I tried this code and it worked fine in Windows 7 and Ubuntu Natty:
import webbrowser
webbrowser.open("path_to_file")
This code also works fine in Windows XP Professional, using Internet Explorer 8.

If you want to go the subprocess.call() way, it should look like this on Windows:
import subprocess
subprocess.call(('cmd', '/C', 'start', '', FILE_NAME))
You can't just use:
subprocess.call(('start', FILE_NAME))
because start is not an executable but a command of the cmd.exe program. This works:
subprocess.call(('cmd', '/C', 'start', FILE_NAME))
but only if there are no spaces in the FILE_NAME.
While subprocess.call method enquotes the parameters properly, the start command has a rather strange syntax, where:
start notes.txt
does something else than:
start "notes.txt"
The first quoted string should set the title of the window. To make it work with spaces, we have to do:
start "" "my notes.txt"
which is what the code on top does.

Start does not support long path names and white spaces. You have to convert it to 8.3 compatible paths.
import subprocess
import win32api
filename = "C:\\Documents and Settings\\user\\Desktop\file.avi"
filename_short = win32api.GetShortPathName(filename)
subprocess.Popen('start ' + filename_short, shell=True )
The file has to exist in order to work with the API call.

os.startfile(path, 'open') under Windows is good because when spaces exist in the directory, os.system('start', path_name) can't open the app correctly and when the i18n exist in the directory, os.system needs to change the unicode to the codec of the console in Windows.

I am pretty late to the lot, but here is a solution using the windows api. This always opens the associated application.
import ctypes
shell32 = ctypes.windll.shell32
file = 'somedocument.doc'
shell32.ShellExecuteA(0,"open",file,0,0,5)
A lot of magic constants. The first zero is the hwnd of the current program. Can be zero. The other two zeros are optional parameters (parameters and directory). 5 == SW_SHOW, it specifies how to execute the app.
Read the
ShellExecute API docs for more info.

Here is the answer from Nick, adjusted slightly for WSL:
import os
import sys
import logging
import subprocess
def get_platform():
if sys.platform == 'linux':
try:
proc_version = open('/proc/version').read()
if 'Microsoft' in proc_version:
return 'wsl'
except:
pass
return sys.platform
def open_with_default_app(filename):
platform = get_platform()
if platform == 'darwin':
subprocess.call(('open', filename))
elif platform in ['win64', 'win32']:
os.startfile(filename.replace('/','\\'))
elif platform == 'wsl':
subprocess.call('cmd.exe /C start'.split() + [filename])
else: # linux variants
subprocess.call(('xdg-open', filename))

If you want to specify the app to open the file with on Mac OS X, use this:
os.system("open -a [app name] [file name]")

On windows 8.1, below have worked while other given ways with subprocess.call fails with path has spaces in it.
subprocess.call('cmd /c start "" "any file path with spaces"')
By utilizing this and other's answers before, here's an inline code which works on multiple platforms.
import sys, os, subprocess
subprocess.call(('cmd /c start "" "'+ filepath +'"') if os.name is 'nt' else ('open' if sys.platform.startswith('darwin') else 'xdg-open', filepath))

On mac os you can call open:
import os
os.open("open myfile.txt")
This would open the file with TextEdit, or whatever app is set as default for this filetype.

I think you might want to open file in editor.
For Windows
subprocess.Popen(["notepad", filename])
For Linux
subprocess.Popen(["text-editor", filename])

I built a small library combining the best answers here for cross-platform support:
$ pip install universal-startfile
then launch a file or URL:
from startfile import startfile
startfile("~/Downloads/example.png")
startfile("http://example.com")

I was getting an error when calling my open file() function. I was following along with a guide but the guide was written in windows while I'm on Linux. So the os.statrfile method wasn't working for me. I was able to alleviate this problem by doing the following:
Import libraries
import sys, os, subprocess
import tkinter
import tkinter.filedioalog as fd
import tkinter.messagebox as mb
After the lib imports I then called the subprocess method for opening a file in unix based OS which is "xdg-open" and the file that will be opened.
def open_file():
file = fd.askopenfilename(title='Choose a file of any type', filetypes=[('All files', "*.*")])
subprocess.call(['xdg-open', file])

Calling Python 2 script from Python 3

I have two scripts, the main is in Python 3, and the second one is written in Python 2 (it also uses a Python 2 library).
There is one method in the Python 2 script I want to call from the Python 3 script, but I don't know how to cross this bridge.

Calling different python versions from each other can be done very elegantly using execnet. The following function does the charm:
import execnet
def call_python_version(Version, Module, Function, ArgumentList):
gw = execnet.makegateway("popen//python=python%s" % Version)
channel = gw.remote_exec("""
from %s import %s as the_function
channel.send(the_function(*channel.receive()))
""" % (Module, Function))
channel.send(ArgumentList)
return channel.receive()
Example: A my_module.py written in Python 2.7:
def my_function(X, Y):
return "Hello %s %s!" % (X, Y)
Then the following function calls
result = call_python_version("2.7", "my_module", "my_function",
["Mr", "Bear"])
print(result)
result = call_python_version("2.7", "my_module", "my_function",
["Mrs", "Wolf"])
print(result)
result in
Hello Mr Bear!
Hello Mrs Wolf!
What happened is that a 'gateway' was instantiated waiting
for an argument list with channel.receive(). Once it came in, it as been translated and passed to my_function. my_function returned the string it generated and channel.send(...) sent the string back. On other side of the gateway channel.receive() catches that result and returns it to the caller. The caller finally prints the string as produced by my_function in the python 3 module.

You could run python2 using subprocess (python module) doing the following:
From python 3:
#!/usr/bin/env python3
import subprocess
python3_command = "py2file.py arg1 arg2" # launch your python2 script
process = subprocess.Popen(python3_command.split(), stdout=subprocess.PIPE)
output, error = process.communicate() # receive output from the python2 script
Where output stores whatever python 2 returned

Maybe to late, but there is one more simple option for call python2.7 scripts:
script = ["python2.7", "script.py", "arg1"]
process = subprocess.Popen(" ".join(script),
shell=True,
env={"PYTHONPATH": "."})

I am running my python code with python 3, but I need a tool (ocropus) that is written with python 2.7. I spent a long time trying all these options with subprocess, and kept having errors, and the script would not complete. From the command line, it runs just fine. So I finally tried something simple that worked, but that I had not found in my searches online. I put the ocropus command inside a bash script:
#!/bin/bash
/usr/local/bin/ocropus-gpageseg $1
I call the bash script with subprocess.
command = [ocropus_gpageseg_path, current_path]
process = subprocess.Popen(command,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
output, error = process.communicate()
print('output',output,'error',error)
This really gives the ocropus script its own little world, which it seems to need. I am posting this in the hope that it will save someone else some time.

It works for me if I call the python 2 executable directly from a python 3 environment.
python2_command = 'C:\Python27\python.exe python2_script.py arg1'
process = subprocess.Popen(python2_command.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
python3_command = 'python python3_script.py arg1'
process = subprocess.Popen(python3_command.split(), stdout=subprocess.PIPE)
output, error = process.communicate()

I ended up creating a new function in the python3 script, which wraps the python2.7 code. It correctly formats error messages created by the python2.7 code and is extending mikelsr's answer and using run() as recommended by subprocess docs.
in bar.py (python2.7 code):
def foo27(input):
return input * 2
in your python3 file:
import ast
import subprocess
def foo3(parameter):
try:
return ast.literal_eval(subprocess.run(
[
"C:/path/to/python2.7/python.exe", "-c", # run python2.7 in command mode
"from bar import foo27;"+
"print(foo27({}))".format(parameter) # print the output
],
capture_output=True,
check=True
).stdout.decode("utf-8")) # evaluate the printed output
except subprocess.CalledProcessError as e:
print(e.stdout)
raise Exception("foo27 errored with message below:\n\n{}"
.format(e.stderr.decode("utf-8")))
print(foo3(21))
# 42
This works when passing in simple python objects, like dicts, as the parameter but does not work for objects created by classes, eg. numpy arrays. These have to be serialized and re-instantiated on the other side of the barrier.

Note: This was happening when running my python 2.x s/w in the liclipse IDE.
When I ran it from a bash script on the command line it didn't have the problem.
Here is a problem & solution I had when mixing python 2.x & 3.x scripts.
I am running a python 2.6 process & needed to call/execute a python 3.6 script.
The environment variable PYTHONPATH was set to point to 2.6 python s/w, so it was choking on the followng:
File "/usr/lib64/python2.6/encodings/__init__.py", line 123
raise CodecRegistryError,\
This caused the 3.6 python script to fail.
So instead of calling the 3.6 program directly I created a bash script which nuked the PYTHONPATH environment variable.
#!/bin/bash
export PYTHONPATH=
## Now call the 3.6 python scrtipt
./36psrc/rpiapi/RPiAPI.py $1

Instead of calling them in python 3, you could run them in conda env batch by creating a batch file as below:
call C:\ProgramData\AnacondaNew\Scripts\activate.bat
C:\Python27\python.exe "script27.py"
C:\ProgramData\AnacondaNew\python.exe "script3.py"
call conda deactivate
pause

I recommend to convert the Python2 files to Python3:
https://pythonconverter.com/

Calling a subprocess within a script using mpi4py

I’m having trouble calling an external program from my python script in which I want to use mpi4py to distribute the workload among different processors.
Basically, I want to use my script such that each core prepares some input files for calculations in separate folders, then starts an external program in this folder, waits for the output, and then, finally, reads the results and collects them.
However, I simply cannot get the external program call to work. On my search for a solution to this problem I've found that the problems I'm facing seem to be quite fundamental. The following simple example makes this clear:
#!/usr/bin/env python
import subprocess
subprocess.call(“EXTERNAL_PROGRAM”, shell=True)
subprocess.call(“echo test”, shell=True)
./script.py works fine (both calls work), while mpirun -np 1 ./script.py only outputs test. Is there any workaround for this situation? The program is definitely in my PATH, but it also fails if I use the abolute path for the call.
This SO question seems to be related, sadly there are no answers...
EDIT:
In the original version of my question I’ve not included any code using mpi4py, even though I mention this module in the title. So here is a more elaborate example of the code:
#!/usr/bin/env python
import os
import subprocess
from mpi4py import MPI
def worker(parameter=None):
"""Make new folder, cd into it, prepare the config files and execute the
external program."""
cwd = os.getcwd()
dir = "_calculation_" + parameter
dir = os.path.join(cwd, dir)
os.makedirs(dir)
os.chdir(dir)
# Write input for simulation & execute
subprocess.call("echo {} > input.cfg".format(parameter), shell=True)
subprocess.call("EXTERNAL_PROGRAM", shell=True)
# After the program is finished, do something here with the output files
# and return the data. I'm using the input parameter as a dummy variable
# for the processed output.
data = parameter
os.chdir(cwd)
return data
def run_parallel():
"""Iterate over job_args in parallel."""
comm = MPI.COMM_WORLD
size = comm.Get_size()
rank = comm.Get_rank()
if rank == 0:
# Here should normally be a list with many more entries, subdivided
# among all the available cores. I'll keep it simple here, so one has
# to run this script with mpirun -np 2 ./script.py
job_args = ["a", "b"]
else:
job_args = None
job_arg = comm.scatter(job_args, root=0)
res = worker(parameter=job_arg)
results = comm.gather(res, root=0)
print res
print results
if __name__ == '__main__':
run_parallel()
Unfortunately I cannot provide more details of the external executable EXTERNAL_PROGRAM other than that it is a C++ application which is MPI enabled. As written in the comment section below, I suspect that this is the reason (or one of the resons) why my external program call is basically ignored.
Please note that I’m aware of the fact that in this situation, nobody can reproduce my exact situation. Still, however, I was hoping that someone here already ran into similar problems and might be able to help.
For completeness, the OS is Ubuntu 14.04 and I’m using OpenMPI 1.6.5.

In your first example you might be able to do this:
#!/usr/bin/env python
import subprocess
subprocess.call(“EXTERNAL_PROGRAM && echo test”, shell=True)
The python script is only facilitating the MPI call. You could just as well write a bash script with command “EXTERNAL_PROGRAM && echo test” and mpirun the bash script; it would be equivalent to mpirunning the python script.
The second example will not work if EXTERNAL_PROGRAM is MPI enabled. When using mpi4py it will initialize the MPI. You cannot spawn another MPI program once you initialized the MPI environment in such a manner. You could spawn using MPI_Comm_spawn or MPI_Comm_spawn_multiple and -up option to mpirun. For mpi4py refer to Compute PI example for spawning (use MPI.COMM_SELF.Spawn).