I am using python subprocess module to run some command and store its output in background. The command is deployed on my machine. Now whenever i run the command from shell prompt it works fine. But when I try to run the same command using subprocess module it gives following error
The command to be executed is vxswadm listswitch all
process = subprocess.Popen('vxswadm listswitch all > tmp.txt &',shell=True)
>>> Traceback (most recent call last):
File "/usr/bin/vxswadm", line 30, in <module>
l.uname = os.getlogin()
OSError: [Errno 25] Inappropriate ioctl for device
Can anyone help me out to fix this error . Any suggestions will be helpful. Thanks in advance
Tazim
The problem is likely due to the bash shell terminating immediately after the & and sending the SIGHUP singal to all of it's subprocesses (standard shell behavior).
You can use the subprocess module to directly execute the command and can redirect the output to tmp.txt yourself by first opening the file and then by passing it's file handle to the stdout argument of the Popen call.
There is a problem with os.getlogin() and subprocessing and python.
See http://code.activestate.com/lists/python-list/288845/
You need to use something else, such as:
pwd.getpwuid(os.getuid()).pw_name (Unix only)
See also the discussion on a portable way to get the username.
Try changing it to ['vxswadm', 'listswitch', 'all', '>', 'tmp.txt','&'] and/or changing shell to False.
I think it might be the shell bit, though (if that fixes it).
You may also try adding stdin=subprocess.PIPE, stdout=subprocess.PIPE, though I doubt that will affect it.
Related
action_publisher = subprocess.Popen(
["bash", "-c", "/opt/ros/melodic/bin/rostopic pub -r 20 /robot_operation std_msgs/String start"],
env={'ROS_MASTER_URI': 'http://10.42.0.49:11311\''})
I tried to run it shell=True and shell=False. Also calling it with bash or just running my executable and I am always getting an error:
Traceback (most recent call last):
File "/opt/ros/melodic/bin/rostopic", line 34, in <module>
import rostopic
ImportError: No module named rostopic
How can I make a call of a shell executable with open through python removing this issue? Tried all combination possible and also other stack proposed solution and still, it tries to import the executable instead of running it on a shell.
I can identify several problems with your attempt, but I'm not sure I have identified them all.
You should use subprocess.check_call or subprocess.run if you just want the subprocess to run, and your Python script to wait for that to complete. If you need to use raw subprocess.Popen(), there are several additional required steps of plumbing which you need to do yourself, which check_call or run will perform for you if you use these higher-level functions.
Your use of env will replace the variables in the environment. Copy the existing environment instead so you don't clobber useful settings like PYTHONPATH etc which may well be preventing the subprocess from finding the library it needs.
The shell invocation seems superfluous.
The stray escaped single quote at the end of 'http://10.42.0.49:11311\'' definitely looks wrong.
With that, try this code instead; but please follow up with better diagnostics if this does not yet solve your problem completely.
import subprocess
import os
# ...
env = os.environ.copy()
env['ROS_MASTER_URI'] = 'http://10.42.0.49:11311'
action_publisher = subprocess.run(
["/opt/ros/melodic/bin/rostopic", "pub", "-r", "20",
"/robot_operation", "std_msgs/String", "start"],
env=env, check=True)
If rostopic is actually a Python program, a better solution altogether might be to import it directly instead.
It sounds like the code of that file is trying to import the Python module. If you're getting that import error even when you try to execute the file in bash/from a shell, then it has nothing to do with subprocess.Popen.
From the traceback, it looks like it's a Python file itself and it's trying to import that module, which would explain why you see the issue when executing it from a shell.
Did you go through the installation correctly, specifically the environment setup? http://wiki.ros.org/melodic/Installation/Ubuntu#melodic.2FInstallation.2FDebEnvironment.Environment_setup
It sounds like you need to source a particular file to have the correct paths available where the Python module is located so it can be found when you execute the script.
As far as I can see, your .Popen command will try to execute
bash -c /opt/ros/melodic/bin/rostopic pub -r 20 /robot_operation std_msgs/String start
while bash -c has to be followed by a string. Thus, you may need to add single quotes.
When I try to call ulimit -n from subprocess, i.e.
subprocess.check_output(['ulimit', '-n'])
I get the following error:
OSError: [Errno 2] No such file or directory
This is strange, because the command is valid on the command line. Previous answers to similar questions focus on the need to input the command in the format of a list, which I have done. Other answers have mentioned that alias commands can cause problems for subprocess, but ulimit is not a alias. If I use the shell=True option the code works. But I would like to understand why.
ulimit is a wrapper around a system call to limit the resources of the current process. Because it acts on the current process, it needs to be called on the current process or it has no effect.
For this reason, the shell implements it as a built-in, so there is no such binary.
If you were to create a shell to just call ulimit, and then kill the shell, you have accomplished nothing, because the process which has the limits is then killed. This is why things like cd that affect the current process need to be implemented like that in the shell.
This means that you cannot call it as a subprocess in python. Fortunately, python has a module to wrap it: https://docs.python.org/3/library/resource.html
Before I start, some notes on my environment: python 2.7.14 installed through miniconda, macOS 10.13.3.
The Problem
I'm trying to write a data processing pipeline (call it analyze.py) in python which calls several different programs. One of those programs, EMAN2 uses its own python environment to run (located, say, ~/EMAN2/bin/python). From the command line, a call to eman2 might look something like this:
~/EMAN2/bin/e2buildstacks.py <args>
where e2buildstacks.py specifies its own python environment using the standard #!/Users/<username>/EMAN2/bin/python type formulation at the top of the file (note, there's a lot of these different .py files to run, I'm just using one name as an example).
In python, I'm using a subprocess.Popen construction for these calls. A very simple example would be:
import subprocess
stacks_cmd = '/Users/<username>/EMAN2/bin/e2buildstacks.py <args>'
process = subprocess.Popen(stacks_cmd, shell=True, stdout=subprocess.PIPE)
stacks_output, stacks_error = process.communicate()
Where I've replaced the actual args and username, for simplicity.
If this is all I'm doing, it works fine. I can run python analyze.py from the command line and it runs EMAN2 without problem.
However, this pipeline is getting wrapped up in a GUI (wxpython based). So I have to use pythonw to run my program. When I try to do:
pythonw analyze.py
It can't run the EMAN2 scripts correctly, I get an error:
Traceback (most recent call last):
File "/Users/<username>/EMAN2/bin/e2buildstacks.py", line 34, in <module>
from EMAN2 import *
ImportError: No module named EMAN2
This indicates to me that it's using my miniconda python rather than the ~/EMAN2/bin/python to run the script.
(Note: if anyone uses EMAN2, I can provide you with the full argument list and the input files. But rest assured that's not the issue, I can run the command I'm using just fine from the command line)
What I've tried
I've tried several different things to fix this problem. The simplest was specifying the python to be used in the command:
import subprocess
stacks_cmd = '/Users/<username>/EMAN2/bin/python /Users/<username>/EMAN2/bin/e2buildstacks.py <args>'
process = subprocess.Popen(stacks_cmd, shell=True, stdout=subprocess.PIPE)
stacks_output, stacks_error = process.communicate()
That didn't work, and fails with the same error.
I've tried using Popen in shell=False mode
import subprocess
stacks_cmd = ['/Users/<username>/EMAN2/bin/python', '/Users/<username>/EMAN2/bin/e2buildstacks.py', <args>]
process = subprocess.Popen(stacks_cmd, shell=False, stdout=subprocess.PIPE)
stacks_output, stacks_error = process.communicate()
or
import subprocess
stacks_cmd = ['/Users/<username>/EMAN2/bin/e2buildstacks.py', <args>]
process = subprocess.Popen(stacks_cmd, shell=False, stdout=subprocess.PIPE)
stacks_output, stacks_error = process.communicate()
Both of those fail with the same error.
I've tried specifying the executable:
import subprocess
stacks_cmd = ['/Users/<username>/EMAN2/bin/e2buildstacks.py', <args>]
process = subprocess.Popen(stacks_cmd, shell=False, stdout=subprocess.PIPE, executable='/Users/<username>/EMAN2/bin/python')
stacks_output, stacks_error = process.communicate()
This one gives a different error:
Unknown option: --
usage: /Users/<username>/EMAN2/bin/e2buildstacks.py [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.
Which makes me think that somehow the arguments aren't getting passed correctly to the script, but are rather being passed to python itself (I probably don't really understand what the executable setting does here).
I've tried setting the environment variable:
import os
my_env = os.environ.copy()
my_env['PATH'] = '/Users/<username>/EMAN2/bin:'+my_env['PATH']
my_env['PYTHONPATH'] = '/Users/<username>/EMAN2/bin'
And then passing that to subprocess.Popen in any of the above commands as env=my_env. This fails with the same errors.
At this point, I'm pretty much out of ideas and hoping someone can help. Again, this is only happening with pythonw, not python.
Things I've read looking for solutions
I couldn't find anything on stackoverflow that quite matched this problem. things I've looked at:
subprocess a program using another version of python
Never answered successfully.
Module cannot be found when using "pythonw" (instead of "python") to run an application
The problem is not which python/pythonw I'm using, both are coming from miniconda.
Python subprocess is running a different version of Python
This is kind of the exact opposite of my problem, so not much use.
https://github.com/ContinuumIO/anaconda-issues/issues/199
This suggests that the miniconda pythonw is a bit of a hack that can cause various problems, but doesn't directly offer a solution (lets say that using another version of python is strongly discouraged).
Python subprocess/Popen with a modified environment
Python Script not running in crontab calling pysaunter
These led me to trying to modify the environment variables.
Final note
Changing the python distribution being used is possible, but very much not the ideal solution. I'm writing something that will be run in several different environments (including linux and windows, which I haven't tested on yet) and by people who aren't me, so I need a more bulletproof solution.
Any help folks can provide is much appreciated.
I have a python script that builds commands based off input gotten via rest.
The commands work when printed out and copied and pasted into powershell.
My question is how to make it actually execute the commands?
So far I have tried writing the commands to a file in the same directory and running like so:
import subprocess, sys
p = subprocess.Popen(["powershell.exe", "test.ps1"],stdout=sys.stdout)
p.communicate()
But I get an error:
Traceback (most recent call last): File
"C:/Users/pschaefer/src/python/executePowerShell.py", line 2, in
p = subprocess.Popen(["powershell.exe", "test.ps1"],stdout=sys.stdout) File "C:\Python27\lib\subprocess.py",
line 703, in init
errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr) File "C:\Python27\lib\subprocess.py", line 850, in
_get_handles
c2pwrite = msvcrt.get_osfhandle(stdout.fileno()) UnsupportedOperation: fileno
Update
removing ,stdout=sys.stdout1 gets rid of the previous error and now I see
test.ps1 cannot be loaded because running \r\nscripts is disabled on
this system.
I have tried switching the command to ".\\test.ps1 Set-ExecutionPolicy -ExecutionPolicy unrestricted" and still have the issue.
Also, would it be possible to build the commands as strings and execute them one by one as built instead of creating a .ps1 file? Big kudos to anyone that can figure that out!
Is your machine you are running the script on enabled to run powershell scripts? You can set this via the Set-ExecutionPolicy cmdlet, or you can pass the -ExecutionPolicy Bypass parameter to run the script with lowered permissions.
Running such scripts from an IDE isn't supported because IDEs redirect stdout to their own stream, which doesn't support fileno properly most of the time.
Running those commands in a real system shell works, though.
The workaround which works in both shell & IDE is to remove stdout=sys.stdout but you can't get the error messages or retrieve the output, or even better: redirect the output/error streams using Popen capabilities:
import subprocess, sys
p = subprocess.Popen(["powershell.exe", "test.ps1"],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
out,err = p.communicate()
print(out,err)
on my machine, since test.ps1 doesn't exist, I get a clear powershell error message.
EDIT: my answer doesn't answer to your edit of your question which shows the actual error message now. This Q&A may help you: PowerShell says "execution of scripts is disabled on this system."
I am trying to run and interact with a vendor specific old version of Python (SunGard Arena Python) from a main Python program through Popen, as I need access to a database through the vendor version of Python.
I can run the vendor Python through the shell though I get
'import site' failed; use -v for traceback.
Now when I try to run it through Popen, if I do the standard:
proc = Popen('U:arena_python.exe',bufsize=-1,stdin=PIPE, stdout=PIPE, stderr=STDOUT)
it doesn't work at all and when I do proc.communicate() I get:
('An exception has occurred -- see the traceback log in "acm_.log".\nCreated a minidump in ".\\arena_python-20160216-092027-942.dmp".\n\'import site\' failed; use -v for traceback\n', None)
The log mentions an access violation in C:\windows\SysWOW64\ntdll.dll. Strangely, if I run the above and include a script as a command line argument, the script runs fine before the process crashes again. Playing around, it seems to work better if I run it with the close_fds=True:
Popen('U:arena_python.exe',bufsize=-1,close_fds=True)
But then I don't know how to interact with the process - the documentation says that on Windows you can't use close_fds=True and redirect input/output.
Any idea what is going on? How can I interact with the process? Thank you,
I know it's an old post, but for the record, here's my solution:
I had this problem too. I have found a tricky workaround for it, published in the following Q&A:
Howto: workaround of close_fds=True and redirect stdout/stderr on windows
Have a good day.