My python script is being called by kea-dhcp, which has capability of executing external scripts (https://kea.readthedocs.io/en/latest/arm/hooks.html#run-script-run-script-support-for-external-hook-scripts for details)
import subprocess
...
subprocess.check_output(('bridge', arg1, arg2 ...), stderr=subprocess.STDOUT)
Where does python take information about location of bridge binary? I'm getting an error:
FileNotFoundError: [Errno 2] No such file or directory: 'bridge': 'bridge'
I don't think it has anything to do with PYTHONPATH since it is not a problem of importing modules. What can be possibly wrong?
UPDATE
Following Charles Duffy's suggestion, I did:
res = subprocess.check_output([shutil.which('bridge'), '-j', 'fdb', 'show'], stderr=subprocess.STDOUT)
but got error (it points to above subprocess line):
File "/lib64/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/lib64/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/lib64/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/lib64/python3.6/subprocess.py", line 1278, in _execute_child
executable = os.fsencode(executable)
File "/lib64/python3.6/os.py", line 800, in fsencode
filename = fspath(filename) # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not NoneType
I can't see what is wrong with that, looks perfectly fine.
How unqualified names are handled by subprocess depends on which operating system you're on. Sometimes PATH is honored; never is PYTHONPATH used for the purpose.
To explicitly perform a PATH lookup, change your code to use shutil.which():
import shutil, subprocess
subprocess.check_output([shutil.which('bridge'), arg1, arg2], stderr=subprocess.STDOUT)
Quoting from https://docs.python.org/3/library/subprocess.html#popen-constructor (formatting from the original) --
Warning For maximum reliability, use a fully qualified path for the executable. To search for an unqualified name on PATH, use shutil.which(). On all platforms, passing sys.executable is the recommended way to launch the current Python interpreter again, and use the -m command-line format to launch an installed module.
Resolving the path of executable (or the first item of args) is platform dependent. For POSIX, see os.execvpe(), and note that when resolving or searching for the executable path, cwd overrides the current working directory and env can override the PATH environment variable. For Windows, see the documentation of the lpApplicationName and lpCommandLine parameters of WinAPI CreateProcess, and note that when resolving or searching for the executable path with shell=False, cwd does not override the current working directory and env cannot override the PATH environment variable. Using a full path avoids all of these variations.
The problem is that kea-dhcp does not export system environment variables to scripts it executes, it only supplies its specific environment. Discovered this via os.environ. The only solution to this is provide the full pass to bridge command (or any other launched in this context).
Related
I am trying to activate my virtualenv (already existing) using the following python code:
Test.py
import os, sys
filename = "activate"
exec(compile(open(filename, "rb").read(), filename, 'exec'), globals, locals)
print(os.system('pwd'))
if hasattr(sys, 'real_prefix'):
print('success')
else:
print('failed')
I then run this script via the terminal:
python Test.py
which then produces this error:
Traceback (most recent call last):
File "activate_this.py", line 3, in <module>
exec(compile(open(filename, "rb").read(), filename, 'exec'), globals, locals)
File "activate", line 4
deactivate () {
^
SyntaxError: invalid syntax
I can activate the virtualenv successfully by executing cd env/bin and then source activate
TLDR
Activating virtualenv from python script is throwing a syntax error from within the activate file.
The very 1st line of activate (note that VEnv is installed on Win, but this shouldn't be a problem):
# This file must be used with "source bin/activate" *from bash*
That, and the lines below should tell you that activate is a (Bourne) shell file.
[Python 3]: compile(source, filename, mode, flags=0, dont_inherit=False, optimize=-1) on the other hand, works with Python source code.
So, in order to execute the file, you'd need to use other ways, e.g. [Python 3]: subprocess - Subprocess management. You can check how I've used it: [SO]: How to effectively convert a POSIX path to Windows path with Python in Cygwin? (#CristiFati's answer).
But, I really don't see the point of doing all this, you probably misunderstood your colleague's suggestion.
Also note that even if you do manage to do it this way, all the environment variables will only be set in the calling process, so it will pretty much be unusable (well, unless you also execute your script from there too).
You should go the recommended way ([PyPA]: Virtualenv - User Guide), and that is (from bash):
source /path/to/Django/ENV/bin/activate
python your_project_startup_script.py # (as I recall, it's manage.py)
I put an operation in my python file :
subprocess.Popen(['notify-send', message])
and the error on terminal is :
subprocess.Popen(['notify-send', message])
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__ errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
how to prevent this error?
Popen uses default no environment, so not PATH. There are several solutions:
Use an env in Popen subprocess.Popen(args=['notify-send', message], env={'PATH': os.getenv('PATH')}
Use the full path to notify-send subprocess.Popen(['/full/path/here/notify-send', message])
By design, programs starting other programs do not use the PATH environment variable by default. It used to be a common attack way to subvert a legit program by changing the PATH or by installing a rogue program with a higher priority in normal PATH than the real child. As a result, the rogue program could be executed on behalf on the user without any abnormal action from him/her.
There are tons of ways of forcing the use of the PATH and it is fine for simple operations (parameter shell = TRUE is one of them). But for more serious scripts it gives the same difference as using the frowned upon system instead of fork + exec in C or C++ languages.
TL/DR: the most correct way is to pass the full PATH of the child program.
I am trying to run a linux executable on Max OS X 10.11.6 via python2.7
I would like to use subprocess.check_output.
The command, which works via the terminal is:
mosel -c "exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"
However, when I try:
subprocess.check_output(['mosel','-c',cmd])
where
cmd="exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"'
I get:
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 629, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 825, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python2.7/site-packages/subprocess32.py", line 1574, in _execute_child
raise child_exception_type(errno_num, err_msg)
OSError: [Errno 2] No such file or directory: 'mosel'
I have been able to get it to "echo" the command to an output file, but I cannot run "which mosel" via python, which leads me to believe that it has to do with check_output using "bin/sh"as the executable.
So, do I need to use "Popen" instead and set
executable=path/to/mosel
?
If so, how do use Python to get the user's path to mosel (i.e. get the output of "which mosel")?
Thanks!
UPDATE:
PyCharm was not seeing the system paths, which I fixed using this answer:
Setting environment variables in OS X?
Now, it appears that
subprocess.check_output(['mosel','-c',cmd])
Is sending the square brackets to the command line, because it now returns:
dyld: Library not loaded: libxprm_mc.dylib
Referenced from: /usr/local/opt/xpress/bin/mosel
Reason: image not found
Traceback (most recent call last):
File "/Users/nlaws/projects/sunlamp/sunlamp-ss/RunScenarios/run.py", line 70, in <module>
run(1)
File "/Users/nlaws/projects/sunlamp/sunlamp-ss/RunScenarios/run.py", line 44, in run
out = check_output(['mosel', '-c', cmd])
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 219, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['mosel', '-c', cmd]' returned non-zero exit status -5
Or is there still a path issue?! (I can run mosel -c cmd via the mac terminal, but not in pycharm via python, nor in the mac terminal via python).
The problem is you're using check_output's arguments incorrectly. Unless you pass it shell=True, check_output expects a list of parameters as its input, in the form:
check_call(['binary_name','arg1','arg2','arg3', ....])
So in this case, you should do:
subprocess.check_call(['mosel', '-c', "exec PATH/TO/SCRIPT arg1='value1', arg2='value2'"])
The root of the issue turns out to be the DYLD_LIBRARY_PATH:
The new OS X release 10.11 "El Capitan" has a "security" feature that
prevents passing DYLD_LIBRARY_PATH to child processes. Somehow, that
variable is stripped from the environment. - (quoted from https://www.postgresql.org/message-id/20151103113612.GW11897#awork2.anarazel.de)
The security feature is called SIP or "System Integrity Protection". Unfortunately, it seems that no one has come up with a solution to this issue (other than work-arounds that must be tailored to each situation).
Here is another example of this issue:
https://github.com/soumith/cudnn.torch/issues/111
Google "mac os inherit dyld_library_path" and you will find many other examples.
I'm running a webservice which operates on wav-files, but I don't want to allow just any upload, so I check the duration of the uploaded file first with the following code:
os.chdir("/home/me/bin")
proc = subprocess.Popen(['duration',wav],stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=False)
out, errors = proc.communicate()
time = int(float(out.strip()))
if time > MAX_TIME:
sys.exit(1)
This has been working fine for several months, but recently (after a migration, I should add) I get the following error:
Traceback (most recent call last):
File "/home/me/webservice.py", line 100, in <module>
proc = subprocess.Popen(['duration',wav],stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=False)
File "/usr/lib64/python2.6/subprocess.py", line 639, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1228, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
This error seems to be mostly caused by the use of a string for the command instead of a list, but that's not the case here. When I try to reproduce this error in a separate script, I can't.
Anyone have any idea what might be the cause here?
Commands without a path are always only looked up on the PATH. You cannot cd to a directory and have subprocess (or your shell for that matter) find a command located in that directory without specifying at least a relative path. This is standard command lookup behaviour.
If you want to run a command in the current working directory, you must specify a path of ./ to make it explicit the PATH search semantics are not to be used.
Relative paths in the PATH environment variable are also not supported everywhere; adding . to PATH is considered a security risk and some systems explicitly filter out . from the PATH. You must have migrated away from a system that allowed this, to a system that does disallow . in the PATH.
The error might not be pertaining to the executable you're feeding to Popen, but to an another executable that is subsequently called and is not found in the PATH of the new environment.
This exact situation happened to me, and the problem was that I had not installed the interpreter for the script I was calling. Frustrating to debug because the error message does not tell you which file it could not find.
Consider doing something like the following at the top of you file then there should be no need to chdir unless you have to for another reason.
import os
cmd = os.path.expanduser('~/bin/duration')
if not os.path.isfile(cmd):
raise IOError("command '%s' not found" % cmd)
proc = subprocess.Popen([cmd,wav ...
The docs for the subprocess module state that 'If shell is True, the specified command will be executed through the shell'. What does this mean in practice, on a Windows OS?
It means that the command will be executed using the program specified in the COMSPEC environment variable. Usually cmd.exe.
To be exact, subprocess calls the CreateProcess windows api function, passing "cmd.exe /c " + args as the lpCommandLine argument.
If shell==False, the lpCommandLine argument to CreateProcess is simply args.
When you execute an external process, the command you want may look something like "foo arg1 arg2 arg3". If "foo" is an executable, that is what gets executed and given the arguments.
However, often it is the case that "foo" is actually a script of some sort, or maybe a command that is built-in to the shell and not an actual executable file on disk. In this case the system can't execute "foo" directly because, strictly speaking, these sorts of things aren't executable. They need some sort of "shell" to execute them. On *nix systems this shell is typically (but not necessarily) /bin/sh. On windows it will typically be cmd.exe (or whatever is stored in the COMSPEC environment variable).
This parameter lets you define what shell you wish to use to execute your command, for the relatively rare case when you don't want the default.
In addition to what was said in other answers, it is useful in practice if you want to open a file in the default viewer for that file type. For instance, if you want to open an HTML or PDF file, but will not know which browser or viewer is installed on the systems it will be run on, or have no guarantees as to the path to the executable, you can simply pass the file name as the only argument for the args field, then set shell=True. This will have Windows use whatever program is associated with that file type.
One caveat, if the path to your file has spaces, you need to surround it with two ".
eg.
path = "C:\\Documents and Settings\\Bob\\Desktop\\New Folder\\README.txt"
subprocess.call('""' + path + '""', shell = True)
In using-the-subprocess-module, there is an explicit paragraph:
The executable argument specifies the program to execute. It is very seldom needed: Usually, the program to execute is defined by the args argument. If shell=True, the executable argument specifies which shell to use. On Unix, the default shell is /bin/sh. On Windows, the default shell is specified by the COMSPEC environment variable.
Windows example - the shell (cmd.exe) command date -t will not be recognized without the shell:
>>> p=subprocess.Popen(["date", "/t"], stdout=subprocess.PIPE)
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
File "C:\Python26\lib\subprocess.py", line 595, in __init__
errread, errwrite)
File "C:\Python26\lib\subprocess.py", line 804, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
>>>
Using a shell, all is well:
>>> p=subprocess.Popen(["date", "/t"], shell=True, stdout=subprocess.PIPE)
>>> p.communicate()
('Wed 04/22/2009 \r\n', None)
>>>