I'm working on a wrapper script written in Python. The wrapper is supposed to choose another Python script based on the system state and execute it (using the absolute path). There is no need to return to the parent script.
It should be noted that I have no control over the scripts being run. They can use __name__ checks, access sys.argv and it should all behave like if the script was run directly.
Right now, I'm using os.execl():
import os, sys
# ...
os.execl(sys.executable, sys.executable, new_script, *sys.argv[1:])
But I can count at least three issues with that:
Any options passed to the Python interpreter are not preserved (e.g. python -v wrapper stops being verbose on re-exec);
The Python interpreter is re-executed unnecessarily (with PyPy it adds 0,7s on my system);
It relies on sys.executable being useful, and the docs say that:
If Python is unable to retrieve the real path to its executable, sys.executable will be an empty string or None.
I'm wondering what replacement I should use for the os.execl call to solve all the issues. So far, I can tell that:
execfile() would probably work but it was removed in Python3 and re-implementing it by hand AFAICS is ugly (because of the encoding issues). I'm not sure what other implications can execfile() have;
imp.load_module() would probably work but it's a bit hacky and was deprecated in Python3.3. It probably can suffer Python3 encoding issues as well.
Which solution do you suggest I use?
Edit: I'd forget. The solution has to work with Python 2.5+, PyPy and Jython 2.5+.
I would just use execfile() instead of imp.load_module(). Although control will be returned to the executive script, one big advantage is that, quoting the docs:
It is different from the import statement in that it does not use the
module administration — it reads the file unconditionally and does not
create a new module.
This means the script file can be anywhere, can have any (or no) file extension, and resources aren't wasted doing module-import related tasks.
Doing so automatically accomplishes or avoids the things you desire:
interpreter options will be preserved
interpreter will not be re-executed unnecessarily
it does not rely on the value of sys.executable
Have you tried something like this?
### wrapped script ###
import sys
print("__name__: {0}\nsys.argv: {1}".format(__name__, sys.argv))
### wrapper script ###
import builtins, sys
my_script = "wrapped_script.py"
print("Executing the wrapped script...")
sys.argv[0] = my_script
with open(my_script, 'r') as fd:
for line in fd:
exec(line)
Result:
$ python3 wrapped_script.py --foo --bar=quux
__name__: __main__
sys.argv: ['wrapped_script.py', '--foo', '--bar=quux']
$ python3 wrapper.py --foo --bar=quux
Executing the wrapped script...
__name__: __main__
sys.argv: ['wrapped_script.py', '--foo', '--bar=quux']
Related
In bash or C, exec will terminate the current process and replace it with something new. Can this functionality be accomplished in Python? I don't wish to simply execute some code then continue running the python script (even just to immediately exit), or spawn a child process.
My specific situation is the following. I'm developing a command line application (python + curses) to manage data generation/analysis in the context of scientific computing. It will sometimes be necessary for me to terminate the application and go wrangle with the data in a given subdirectory manually. It would be convenient if I could do something like:
# within python script; d=<some directory>
if exit_and_goto_dir:
exec("pushd {}".format(d)) # C-style exec -- terminate program and execute new command
The following do not work, for example:
# attempt 1
if exit_and_goto_dir:
os.system("pushd {}".format(d))
exit(0) # pushd does not outlast the python script
# attempt 2
if exit_and_goto_dir:
os.chdir(d)
exit(0)
This behavior isn't really critical. There are plenty of work arounds (e.g. print the directory I care about to terminal then cd manually). Mostly I'm curious if it's possible. Thanks!
The os module contains Python wrappers for the various exec* functions in the C standard library:
>>> [method for method in dir(os) if method.startswith("exec")]
['execl', 'execle', 'execlp', 'execlpe', 'execv', 'execve', 'execvp', 'execvpe']
However, pushd is not an executable that you can exec but rather a bash builtin (and the same is true for cd).
What you could do would be to change directory inside the python process and then exec an interactive shell:
import os
os.chdir(d)
os.execvp("bash", ["bash", "-login"])
Python's current directory will be inherited by the shell that you exec. When you later exit from that shell, control will then return to the original login shell from which you invoked python (unless you used that shell's exec command to launch python in the first place).
What you can't do is to modify the current directory of the calling shell from inside python, in order to return to the shell prompt but in a different working directory from when python was invoked. (At least there's no straightforward way. There is a hack involving attaching gdb to it, described here, but which only worked as root when I tried it on Ubuntu.)
In the pip program, the She-bang is
#!/usr/local/bin/python
if __name__ == "__main__":
# Python program body
while in the Install Certificates.command that Python Launcher offers:
#!/bin/sh
/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 << "EOF"
# python program body
EOF
Are there any differences between those two approaches? And is there any reason to prefer one to another?
It seems to me they are all the same, except for the second one has one more bash subroutine. Is this right?
In the general case, you simply want to specify the interpreter you actually want.
Outside of this, you sometimes see workarounds like this as portability hacks. On POSIX systems, /usr/bin/env covers the majority of scenarios quite nicely; but if you need portability to older or otherwise peculiar systems, falling back to the lowest common denominator and then working your way back up to a place where you can reliably run e.g. Python on a variety of systems may require all kinds of unobvious constructs. (The previous - upvoted! - answer by Dan D. is a good example.)
There are also cases where you want sh to set something up (fetch some environment variables which are specified in a file which uses sh syntax, for example) and then hand over execution to Python;
#!/bin/sh
# source some variables
. /etc/defaults/myenv.sh
# Then run Python
exec env python -c '
# ... Your Python script here
' "$#"
There is a line length limit on the #! line. Perhaps they did that to get around that.
The options are the path to the program but only if it is short enough. Use of env python which uses the path. Or chain loading like this.
This specific code for the Install Certificates.command script was introduced in Python Issue #17128. As far as I can tell, the author hasn't explained why he wrote the code this way.
Note that .command files are Shell scripts on Mac OS X that can be executed by double-clicking on them in Finder.
I believe the likely explanation is that the author simply wanted to honour Mac OS X's expectation that .command files should be Shell scripts.
You could test this by placing the following content in a file ~/Desktop/test.command:
#!/usr/bin/env python
print "Hello world"
Then view the Desktop folder in Finder, and note that it is reported as a "shell" file:
(Although it is reported incorrectly as a Shell file, this Python script can still be executed by double-clicking on it. It doesn't break Finder or anything.)
To answer the specific question, one reason for preferring this pattern might be, as Dan D. said, to avoid a Shebang line limit.
In general, you would prefer to use #!/usr/bin/env python as your Shebang line. Creating a Bash Heredoc (i.e. the python3.6 << EOF pattern) would create all sorts of problems, such as your syntax highlighting won't work, you have to watch out for Bash variable interpolation inside the Heredoc, etc.
I have a python script that I like to run with python -i script.py, which runs the script and then enters interactive mode so that I can play around with the results.
Is it possible to have the script itself invoke this option, such that I can just run python script.py and the script will enter interactive mode after running?
Of course, I can simply add the -i, or if that is too much effort, I can write a shell script to invoke this.
From within script.py, set the PYTHONINSPECT environment variable to any nonempty string. Python will recheck this environment variable at the end of the program and enter interactive mode.
import os
# This can be placed at top or bottom of the script, unlike code.interact
os.environ['PYTHONINSPECT'] = 'TRUE'
In addition to all the above answers, you can run the script as simply ./script.py by making the file executable and setting the shebang line, e.g.
#!/usr/bin/python -i
this = "A really boring program"
If you want to use this with the env command in order to get the system default python, then you can try using a shebang like #donkopotamus suggested in the comments
#!/usr/bin/env PYTHONINSPECT=1 python
The success of this may depend on the version of env installed on your platform however.
You could use an instance of code.InteractiveConsole to get this to work:
from code import InteractiveConsole
i = 20
d = 30
InteractiveConsole(locals=locals()).interact()
running this with python script.py will launch an interactive interpreter as the final statement and make the local names defined visible via locals=locals().
>>> i
20
Similarly, a convenience function named code.interact can be used:
from code import interact
i = 20
d = 30
interact(local=locals())
This creates the instance for you, with the only caveat that locals is named local instead.
In addition to this, as #Blender stated in the comments, you could also embed the IPython REPL by using:
import IPython
IPython.embed()
which has the added benefit of not requiring the namespace that has been populated in your script to be passed with locals.
I think you're looking for this?
import code
foo = 'bar'
print foo
code.interact(local=locals())
I would simply accompany the script with a shell script that invokes it.
exec python -i "$(dirname "$0")/script.py"
I am trying to compile a C program using Python and want to give input using "<" operator but it's not working as expected.
If I compile the C program and run it by giving input though a file it works; for example
./a.out <inp.txt works
But similarly if I try to do this using a Python script, it did not quite work out as expected.
For example:
import subprocess
subprocess.call(["gcc","a.c","-o","x"])
subprocess.call(["./x"])
and
import subprocess
subprocess.call(["gcc","a.c","-o","x"])
subprocess.call(["./x","<inp.txt"])
Both script ask for input though terminal. But I think in the second script it should read from file. why both the programs are working the same?
To complement #Jonathan Leffler's and #alastair's helpful answers:
Assuming you control the string you're passing to the shell for execution, I see nothing wrong with using the shell for convenience. [1]
subprocess.call() has an optional Boolean shell parameter, which causes the command to be passed to the shell, enabling I/O redirection, referencing environment variables, ...:
subprocess.call("./x <inp.txt", shell = True)
Note how the entire command line is passed as a single string rather than an array of arguments.
[1]
Avoid use of the shell in the following cases:
If your Python code must run on platforms other than Unix-like ones, such as Windows.
If performance is paramount.
If you find yourself "outsourcing" tasks better handled on the Python side.
If you're concerned about lack of predictability of the shell environment (as #alastair is):
subprocess.call with shell = True always creates non-interactive non-login instances of /bin/sh - note that it is NOT the user's default shell that is used.
sh does NOT read initialization files for non-interactive non-login shells (neither system-wide nor user-specific ones).
Note that even on platforms where sh is bash in disguise, bash will act this way when invoked as sh.
Every shell instance created with subprocess.call with shell = True is its own world, and its environment is neither influenced by previous shell instances nor does it influence later ones.
However, the shell instances created do inherit the environment of the python process itself:
If you started your Python program from an interactive shell, then that shell's environment is inherited. Note that this only pertains to the current working directory and environment variables, and NOT to aliases, shell functions, and shell variables.
Generally, that's a feature, given that Python (CPython) itself is designed to be controllable via environment variables (for 2.x, see https://docs.python.org/2/using/cmdline.html#environment-variables; for 3.x, see https://docs.python.org/3/using/cmdline.html#environment-variables).
If needed, you can supply your own environment to the shell via the env parameter; note, however, that you'll have to supply the entire environment in that event, potentially including variables such as USER and HOME, if needed; simple example, defining $PATH explicitly:
subprocess.call('echo $PATH', shell = True, \
env = { 'PATH': '/sbin:/bin:/usr/bin' })
The shell does I/O redirection for a process. Based on what you're saying, the subprocess module does not do I/O redirection like that. To demonstrate, run:
subprocess.call(["sh","-c", "./x <inp.txt"])
That runs the shell and should redirect the I/O. With your code, your program ./x is being given an argument <inp.txt which it is ignoring.
NB: the alternative call to subprocess.call is purely for diagnostic purposes, not a recommended solution. The recommended solution involves reading the (Python 2) subprocess module documentation (or the Python 3 documentation for it) to find out how to do the redirection using the module.
import subprocess
i_file = open("inp.txt")
subprocess.call("./x", stdin=i_file)
i_file.close()
If your script is about to exit so you don't have to worry about wasted file descriptors, you can compress that to:
import subprocess
subprocess.call("./x", stdin=open("inp.txt"))
By default, the subprocess module does not pass the arguments to the shell. Why? Because running commands via the shell is dangerous; unless they're correctly quoted and escaped (which is complicated), it is often possible to convince programs that do this kind of thing to run unwanted and unexpected shell commands.
Using the shell for this would be wrong anyway. If you want to take input from a particular file, you can use subprocess.Popen, setting the stdin argument to a file descriptor for the file inp.txt (you can get the file descriptor by calling fileno() a Python file object).
By some need I was forced to correct os.environ['PATH'] to be able to run dir\to\fake\python.cmd script which adds some extra parameters to the original one before execution.
Also I have two python scripts:
test1.py:
# ...
p = subprocess.call("test2.py") # shell=True works fine
# ...
test2.py:
# ...
print "Hello from test2.py"
# ...
When I run python test1.py my "fake" python.cmd doing its stuff, refers to the original python in c:\Python25 and runs test1.py with my extra arguments. But, sadly, test2.py, script is never called. If I put shell=True as subprocess.call argument - everythin's fine, test2.py is called.
I know, Windows is trying to find python interpreter to use for the call in the real c:\Python25 working directory when shell=False is by default.
The question to you is: how can I achieve the goal without changing my code in test1.py and test2.py? Maybe virtualenv library may be very useful in this case?
Thank you very much for your help
As stated in the docs:
The shell argument (which defaults to False) specifies whether to use the shell as the program to execute.
and
On Windows with shell=True, the COMSPEC environment variable specifies the default shell. The only time you need to specify shell=True on Windows is when the command you wish to execute is built into the shell (e.g. dir or copy). You do not need shell=True to run a batch file or console-based executable.
So when you call subprocess.call("test2.py"), the system tries to call test2.py as an executable, which it is not, so it fails. However, you don't capture the return value from subprocess.open to check for error conditions, so it fails silently. When you call it with shell=True, it calls the system shell with the argument test2.py, which in turn looks up the default executable for .py files on your system and then executes the file that way.
All that said though, the deeper problem here is that your code is very poorly engineered. The probability that you have a legitimate reason to dispatch a python script via the OS from another python script is vanishingly small. Rather than patching another kludgy hack over the top of what you already have, save yourself headaches in the future and refactor the system so that it does things in an easier, more sensible, more naturally integrated way.