Before I start, some notes on my environment: python 2.7.14 installed through miniconda, macOS 10.13.3.
The Problem
I'm trying to write a data processing pipeline (call it analyze.py) in python which calls several different programs. One of those programs, EMAN2 uses its own python environment to run (located, say, ~/EMAN2/bin/python). From the command line, a call to eman2 might look something like this:
~/EMAN2/bin/e2buildstacks.py <args>
where e2buildstacks.py specifies its own python environment using the standard #!/Users/<username>/EMAN2/bin/python type formulation at the top of the file (note, there's a lot of these different .py files to run, I'm just using one name as an example).
In python, I'm using a subprocess.Popen construction for these calls. A very simple example would be:
import subprocess
stacks_cmd = '/Users/<username>/EMAN2/bin/e2buildstacks.py <args>'
process = subprocess.Popen(stacks_cmd, shell=True, stdout=subprocess.PIPE)
stacks_output, stacks_error = process.communicate()
Where I've replaced the actual args and username, for simplicity.
If this is all I'm doing, it works fine. I can run python analyze.py from the command line and it runs EMAN2 without problem.
However, this pipeline is getting wrapped up in a GUI (wxpython based). So I have to use pythonw to run my program. When I try to do:
pythonw analyze.py
It can't run the EMAN2 scripts correctly, I get an error:
Traceback (most recent call last):
File "/Users/<username>/EMAN2/bin/e2buildstacks.py", line 34, in <module>
from EMAN2 import *
ImportError: No module named EMAN2
This indicates to me that it's using my miniconda python rather than the ~/EMAN2/bin/python to run the script.
(Note: if anyone uses EMAN2, I can provide you with the full argument list and the input files. But rest assured that's not the issue, I can run the command I'm using just fine from the command line)
What I've tried
I've tried several different things to fix this problem. The simplest was specifying the python to be used in the command:
import subprocess
stacks_cmd = '/Users/<username>/EMAN2/bin/python /Users/<username>/EMAN2/bin/e2buildstacks.py <args>'
process = subprocess.Popen(stacks_cmd, shell=True, stdout=subprocess.PIPE)
stacks_output, stacks_error = process.communicate()
That didn't work, and fails with the same error.
I've tried using Popen in shell=False mode
import subprocess
stacks_cmd = ['/Users/<username>/EMAN2/bin/python', '/Users/<username>/EMAN2/bin/e2buildstacks.py', <args>]
process = subprocess.Popen(stacks_cmd, shell=False, stdout=subprocess.PIPE)
stacks_output, stacks_error = process.communicate()
or
import subprocess
stacks_cmd = ['/Users/<username>/EMAN2/bin/e2buildstacks.py', <args>]
process = subprocess.Popen(stacks_cmd, shell=False, stdout=subprocess.PIPE)
stacks_output, stacks_error = process.communicate()
Both of those fail with the same error.
I've tried specifying the executable:
import subprocess
stacks_cmd = ['/Users/<username>/EMAN2/bin/e2buildstacks.py', <args>]
process = subprocess.Popen(stacks_cmd, shell=False, stdout=subprocess.PIPE, executable='/Users/<username>/EMAN2/bin/python')
stacks_output, stacks_error = process.communicate()
This one gives a different error:
Unknown option: --
usage: /Users/<username>/EMAN2/bin/e2buildstacks.py [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.
Which makes me think that somehow the arguments aren't getting passed correctly to the script, but are rather being passed to python itself (I probably don't really understand what the executable setting does here).
I've tried setting the environment variable:
import os
my_env = os.environ.copy()
my_env['PATH'] = '/Users/<username>/EMAN2/bin:'+my_env['PATH']
my_env['PYTHONPATH'] = '/Users/<username>/EMAN2/bin'
And then passing that to subprocess.Popen in any of the above commands as env=my_env. This fails with the same errors.
At this point, I'm pretty much out of ideas and hoping someone can help. Again, this is only happening with pythonw, not python.
Things I've read looking for solutions
I couldn't find anything on stackoverflow that quite matched this problem. things I've looked at:
subprocess a program using another version of python
Never answered successfully.
Module cannot be found when using "pythonw" (instead of "python") to run an application
The problem is not which python/pythonw I'm using, both are coming from miniconda.
Python subprocess is running a different version of Python
This is kind of the exact opposite of my problem, so not much use.
https://github.com/ContinuumIO/anaconda-issues/issues/199
This suggests that the miniconda pythonw is a bit of a hack that can cause various problems, but doesn't directly offer a solution (lets say that using another version of python is strongly discouraged).
Python subprocess/Popen with a modified environment
Python Script not running in crontab calling pysaunter
These led me to trying to modify the environment variables.
Final note
Changing the python distribution being used is possible, but very much not the ideal solution. I'm writing something that will be run in several different environments (including linux and windows, which I haven't tested on yet) and by people who aren't me, so I need a more bulletproof solution.
Any help folks can provide is much appreciated.
Related
action_publisher = subprocess.Popen(
["bash", "-c", "/opt/ros/melodic/bin/rostopic pub -r 20 /robot_operation std_msgs/String start"],
env={'ROS_MASTER_URI': 'http://10.42.0.49:11311\''})
I tried to run it shell=True and shell=False. Also calling it with bash or just running my executable and I am always getting an error:
Traceback (most recent call last):
File "/opt/ros/melodic/bin/rostopic", line 34, in <module>
import rostopic
ImportError: No module named rostopic
How can I make a call of a shell executable with open through python removing this issue? Tried all combination possible and also other stack proposed solution and still, it tries to import the executable instead of running it on a shell.
I can identify several problems with your attempt, but I'm not sure I have identified them all.
You should use subprocess.check_call or subprocess.run if you just want the subprocess to run, and your Python script to wait for that to complete. If you need to use raw subprocess.Popen(), there are several additional required steps of plumbing which you need to do yourself, which check_call or run will perform for you if you use these higher-level functions.
Your use of env will replace the variables in the environment. Copy the existing environment instead so you don't clobber useful settings like PYTHONPATH etc which may well be preventing the subprocess from finding the library it needs.
The shell invocation seems superfluous.
The stray escaped single quote at the end of 'http://10.42.0.49:11311\'' definitely looks wrong.
With that, try this code instead; but please follow up with better diagnostics if this does not yet solve your problem completely.
import subprocess
import os
# ...
env = os.environ.copy()
env['ROS_MASTER_URI'] = 'http://10.42.0.49:11311'
action_publisher = subprocess.run(
["/opt/ros/melodic/bin/rostopic", "pub", "-r", "20",
"/robot_operation", "std_msgs/String", "start"],
env=env, check=True)
If rostopic is actually a Python program, a better solution altogether might be to import it directly instead.
It sounds like the code of that file is trying to import the Python module. If you're getting that import error even when you try to execute the file in bash/from a shell, then it has nothing to do with subprocess.Popen.
From the traceback, it looks like it's a Python file itself and it's trying to import that module, which would explain why you see the issue when executing it from a shell.
Did you go through the installation correctly, specifically the environment setup? http://wiki.ros.org/melodic/Installation/Ubuntu#melodic.2FInstallation.2FDebEnvironment.Environment_setup
It sounds like you need to source a particular file to have the correct paths available where the Python module is located so it can be found when you execute the script.
As far as I can see, your .Popen command will try to execute
bash -c /opt/ros/melodic/bin/rostopic pub -r 20 /robot_operation std_msgs/String start
while bash -c has to be followed by a string. Thus, you may need to add single quotes.
I am pulling my hair out here. I am spawning a process which I need the feedback from in Python.
When I run the command in the cmd window it runs fine, but when I try to run it via Python the terminal hangs.
p = subprocess.Popen(startcmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(out, err) = p.communicate()
Where startcmd is a string which when printed in the Python console looks like this:
"C:/Program Files/GRASS GIS 7.2.1/grass72.bat" --version
If I copy and paste this into a Windows cmd, it shows the version information and returns control to the command prompt about a second later, but in Python it freezes up.
I should point out, if I replace the startcmd string with something like "dir" or even "python --version", it works fine!
Additional: I have tried shell=True, this has the same result.
Additional: I have tried sending the cmd and arguments through as an array as suggested in an answer below given that shell=False, but this also hangs the same.
Additional: I have added the GRASS path to the system PATH, so that now I can simply call grass72 --version in the cmd window to get a result, however this also still freezes in Python but works fine in cmd.
Additional: I have created a basic .bat file to test if .bat files run ok via Python, here is what I created:
#echo off
title Test Batch Script
echo I should see this message
This runs fine both in cmd, and in Python.
Problem found but not solved!
So, I'm running the script which spawns the process using subprocess.Popen using Python 3.6. The .bat file which is spawned launches a Python script using a version of Python (based on 2.7) which comes shipped with GRASS:
%GRASS_PYTHON% "\BLAH\BLAH\grass72.py"
What is interesting, is that if I launch the subprocess.Popen script with Python 2.7, it works fine. Ahah, you may think, solved! But this doesn't solve my problem - because I really need Python 3.6 to be launching the process, also why does it matter what version of Python launches the batch file? The new Python script which is spawned is launched with Python 2.7 anyway.
Since I started re-directing stdout I can see that there is an error when I use Python 3.6 to launch the process:
File "C:\ProgramData\Anaconda3\lib\site.py", line 177
file=sys.stderr)
^
SyntaxError: invalid syntax
Notice its reverting to Anaconda3! Even though it is launched using python.exe from 2.7!
I experienced the same issue with Python 3.6 and 3.7 on Windows hanging for subprocess calls:
p = subprocess.Popen(startcmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(out, err) = p.communicate()
Upon closer investigation I noticed this occurs only if the process writes more than about 4 KB (4096 bytes) of output which might explain why your short script does not reproduce this.
A workaround I found is using tempfile in the standard library:
# Write to a temporary file because pipe redirection seems broken
with tempfile.NamedTemporaryFile(mode="w+") as tmp_out,
tempfile.NamedTemporaryFile(mode="w+") as tmp_err:
p = subprocess.Popen(startcmd, stdout=tmp_out, stderr=tmp_err,
universal_newlines=True)
# `run` waits for command to complete, `Popen` continues Python program
while p.poll() is None:
time.sleep(.1)
# Cursor is after the last write call, reset to read output
tmp_out.seek(0)
tmp_err.seek(0)
out = tmp_out.read()
err = tmp_err.read()
You don't specify shell=True in your arguments to Popen. The recommended usage in that case is to specify a sequence of arguments instead of a string. So you should set startcmd equal to ["C:/Program Files/GRASS GIS 7.2.1/grass72.bat", "--version"].
Try this:
p = subprocess.Popen(startcmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
When I launch a PowerShell script from Python, the delay seems to be approximately 45s, and I cannot figure out why.
I'm trying to run a PowerShell script (accessing some APIs only available to PowerShell) from a Python script.
I've tried a lot of permutations, and all incur ~45 second delay compared to just running the script from a command prompt, using an identical command line.
For example - sample.ps1 might say:
echo foo
And runner.py might say:
import subprocess
p = subprocess.Popen([POWERSHELL, '-File', 'sample.ps1'], stdout=subprocess.STDOUT)
d = p.stdout.read()
Running the .ps1 script directly is fast, running it via runner.py (Python 2.7, 32bit on a 64bit machine) incurs 45 second delay.
The exact same thing occurs if I use "os.system", or Twisted's built-in process tools. So I suspect it's some subtle interaction between the Python interpreter and the Powershell interpreter, possibly related to creation of console windows, or handling of stdin/out/err streams? (which I know don't "really exist" in the same way on Windows)
I do not see any such delays. It is pretty snappy. ( that will also depend on what your script actually does.) Try using call:
from subprocess import call
call(["powershell", "sample.ps1"])
PowerShell loads your user's profile by default. Use the -NoProfile argument to turn that behavior off:
import subprocess
p = subprocess.Popen([POWERSHELL, '-NoProfile', '-File', 'sample.ps1'], stdout=subprocess.STDOUT)
d = p.stdout.read()
So, I thought it would be cool if i could get my dev env up and running in a single fell swoop with some python magic. Various DBs, webserver etc.
However, every variation on the below that i have tried on the following seems to fail with 'file not found'.
p2 = Popen(["exec", "/path/to/redis/server"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
Running the command directly from the shell (i.e. exec /path/to/redis/server) works just fine. Strangely enough, a simple command line uptime seems to work fine.
Any clues as to what is going on? Also, whilst we are on the topic, is multiprocessing the thing to use when i want to run many of these external processes in parallel?
Thanks
exec is a builtin command in bash, not an executable. The file not found error probably comes from exec not being found in the $PATH.
I would try ommitting "exec" in the Popen call.
I'm working in a windows environment (my laptop!) and I need a couple of scripts that run other programs, pretty much like a windows batch file.
how can I run a command from python such that the program when run, will replace the script? The program is interactive (for instance, unison) and keeps printing lines and asking for user input all the time.
So, just running a program and printing the output won't suffice. The program has to takeover the script's input/output, pretty mcuh like running the command from a .bat file.
I tried os.execl but it keeps telling me "invalid arguments", also, it doesn't find the program name (doesn't search the PATH variable); I have to give it the full path ..?!
basically, in a batch script I can write:
unison profile
how can I achieve the same effect in python?
EDIT:
I found out it can be done with os.system( ... ) and since I cannot accept my own answer, I'm closing the question.
EDIT: this was supposed to be a comment, but when I posted it I didn't have much points.
Thanks Claudiu, that's pretty much what I want, except for a little thing: I want the function to end when the program exits, but when I try it on unison, it doesn't return control to the python script, but to the windows command line environment
>>> os.execlp("unison")
C:\>Usage: unison [options]
or unison root1 root2 [options]
or unison profilename [options]
For a list of options, type "unison -help".
For a tutorial on basic usage, type "unison -doc tutorial".
For other documentation, type "unison -doc topics".
C:\>
C:\>
C:\>
how to get around this?
You should create a new processess using the subprocess module.
I'm not fluent in windows processes but its Popen function is cross-platform, and should be preffered to OS specific solutions.
EDIT: I maintain that you should prefer the Subprocess module to os.* OS specific functions, it is cross-platform and more pythonic (just google it). You can wait for the result easily, and cleanly:
import os
import subprocess
unison = os.path.join(os.path.curdir, "unison")
p = subprocess.Popen(unison)
p.wait()
I found out that os.system does what I want,
Thanks for all that tried to help.
os.system("dir")
runs the command just as if it was run from a batch file
import subprocess
proc = subprocess.Popen(['unison', 'profile'], stderr=subprocess.PIPE,
stdout=subprocess.PIPE, stdin=subprocess.PIPE)
proc.stdin.write('user input')
print proc.stdout.read()
This should help you get started. Please edit your question with more information if you want a more detailed answer!
os.execlp should work. This will search your path for the command. Don't give it any args if they're not necessary:
>>> import os
>>> os.execlp("cmd")
D:\Documents and Settings\Claudiu>Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.
D:\Documents and Settings\Claudiu>