Recently, I came across the Linux command source and then found this answer on what it does.
My understanding was that source executes the file that is passed to it, and it did work for a simple shell script. Then I tried using source on a Python script–but it did not work.
The Python script has a shebang (e.g. #!/usr/bin/python) and I am able to do a ./python.py, as the script has executable permission. If that is possible, source python.py should also be possible, right? The only difference is ./ executes in a new shell and source executes in the current shell. Why is it not working on a .py script? Am I missing something here?
You're still not quite on-target understanding what source does.
source does indeed execute commands from a file in the current shell process. It does this effectively as if you had typed them directly into your current shell.
The reason this is necessary is because when you run a shell script without sourcing it, it will spawn a subshell—a new process. When this process exits, any changes made within that script are lost as you return to the shell from which it spawned.
It follows, then, that you cannot source Python into a shell, because the Python interpreter is always a different process from your shell. Running a Python script spawns a brand-new process, and when that process exits, its state is lost.
Of course, if your shell is actually Python (which I would not recommend!), you can still "source" into it—by using import.
source executes the files and places whatever functions/aliases/environment variables created in that script within the shell that called it. It does this by not spawning a new process, but instead executing the script in the current process.
The shabang is used by the shell to indicate what to use to spawn the new process, so for source it is ignored, and the file is interpreted as the language of the current process (bash in this case). This is why using source on a python file failed for you.
Related
In bash or C, exec will terminate the current process and replace it with something new. Can this functionality be accomplished in Python? I don't wish to simply execute some code then continue running the python script (even just to immediately exit), or spawn a child process.
My specific situation is the following. I'm developing a command line application (python + curses) to manage data generation/analysis in the context of scientific computing. It will sometimes be necessary for me to terminate the application and go wrangle with the data in a given subdirectory manually. It would be convenient if I could do something like:
# within python script; d=<some directory>
if exit_and_goto_dir:
exec("pushd {}".format(d)) # C-style exec -- terminate program and execute new command
The following do not work, for example:
# attempt 1
if exit_and_goto_dir:
os.system("pushd {}".format(d))
exit(0) # pushd does not outlast the python script
# attempt 2
if exit_and_goto_dir:
os.chdir(d)
exit(0)
This behavior isn't really critical. There are plenty of work arounds (e.g. print the directory I care about to terminal then cd manually). Mostly I'm curious if it's possible. Thanks!
The os module contains Python wrappers for the various exec* functions in the C standard library:
>>> [method for method in dir(os) if method.startswith("exec")]
['execl', 'execle', 'execlp', 'execlpe', 'execv', 'execve', 'execvp', 'execvpe']
However, pushd is not an executable that you can exec but rather a bash builtin (and the same is true for cd).
What you could do would be to change directory inside the python process and then exec an interactive shell:
import os
os.chdir(d)
os.execvp("bash", ["bash", "-login"])
Python's current directory will be inherited by the shell that you exec. When you later exit from that shell, control will then return to the original login shell from which you invoked python (unless you used that shell's exec command to launch python in the first place).
What you can't do is to modify the current directory of the calling shell from inside python, in order to return to the shell prompt but in a different working directory from when python was invoked. (At least there's no straightforward way. There is a hack involving attaching gdb to it, described here, but which only worked as root when I tried it on Ubuntu.)
I have a tcsh shell script that sets up all the necessary environment including PYTHONPATH, which then run an executable at the end of it. I also have a python script that gets sent to the shell script as an input. So the following works perfectly fine when it is ran from Terminal:
path to shell script path to python script
Now, the problem occurs when I want to do the same thing from a subprocess. The python script fails to be ran since it cannot find many of the modules that's already supposed to be set via the shell script. And clearly, the PYTHONPATH ends up having many missing paths compared to the parent environment the subprocess was ran from or the shell script itself! It seems like the subprocess does not respect the environment the shell script sets up.
I've tried all sorts of things already but none help!
cmd = [shell_script_path, py_script_path]
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=os.environ.copy())
It makes no difference if env is not given either!
Any idea how to fix this?!
Knowing the subprocess inherits all the parent process environment and they are supposed to be ran under same environment, making the shell script to not setup any environment, fixed it.
This solves the environment being retained, but now the problem is, the process just hangs! (it does not happen when it is ran directly from shell)
I'm trying to change the terminal directory through a python script. I've seen this post and others like it so I know about os.chdir, but it's not working the way I'd like. os.chdir appears to change the directory, but only for the python script. For instance I have this code.
#! /usr/bin/env python
import os
os.chdir("/home/chekid/work2/")
print os.getcwd()
Unfortunately after running I'm still in the directory of the python script (e.g. /home/chekid) rather than the directory I want to be in. See below.
gandalf(pts/42):~> pwd
/home/chekid
gandalf(pts/42):~> ./changedirectory.py
/home/chekid/work2
gandalf(pts/42):~> pwd
/home/chekid
Any thoughts on what I should do?
Edit: Looks like what I'm trying to do doesn't exist in 'normal' python. I did find a work around, although it doesn't look so elegant to me.
cd `./changedirectory.py`
You can't. The shell's current directory belongs to the shell, not to you.
(OK, you could ptrace(2) the shell and make it call chdir(2), but that's probably not a great design, won't work on Windows, and I would not begin to know how to do it in pure Python except that you'd probably have to mess around with ctypes or something similar.)
You could launch a subshell with your current working directory. That might be close enough to what you need:
os.chdir('/path/to/somewhere')
shell = os.environ.get('SHELL', '/bin/sh')
os.execl(shell, shell)
# execl() does not return; it replaces the Python process with a new shell process
The original shell will still be there, so make sure you don't leave it hanging around. If you initially call Python with the exec builtin (e.g. exec python /path/to/script.py), then the original shell will be replaced with the Python process and you won't have to worry about this. But if Python exits without launching the shell, you'll be left with no shell open at all.
You can if you cheat: Make a bash script that calls your python script. The python script returns the path you want to change directory to. Then the bash script does the acctual chdir. Of course you would have to run the bash script in your bash shell using "source".
The current working directory is an attribute of a process. It cannot be changed by another program, such as changing the current working directory in your shell by running a separate Python program. This is why cd is always a shell built-in command.
You can make your python print the directory you want to move to, and then call your script with cd "$(./python-script.py)". In condition your script actually does not print anything else.
I'd like to call a separate non-child python program from a python script and have it run externally in a new shell instance. The original python script doesn't need to be aware of the instance it launches, it shouldn't block when the launched process is running and shouldn't care if it dies. This is what I have tried which returns no error but seems to do nothing...
import subprocess
python_path = '/usr/bin/python'
args = [python_path, '&']
p = subprocess.Popen(args, shell=True)
What should I be doing differently
EDIT
The reason for doing this is I have an application with a built in version of python, I have written some python tools that should be run separately alongside this application but there is no assurance that the user will have python installed on their system outside the application with the builtin version I'm using. Because of this I can get the python binary path from the built in version programatically and I'd like to launch an external version of the built in python. This eliminates the need for the user to install python themselves. So in essence I need a simple way to call an external python script using my current running version of python programatically.
I don't need to catch any output into the original program, in fact once launched I'd like it to have nothing to do with the original program
EDIT 2
So it seems that my original question was very unclear so here are more details, I think I was trying to over simplify the question:
I'm running OSX but the code should also work on windows machines.
The main application that has a built in version of CPython is a compiled c++ application that ships with a python framework that it uses at runtime. You can launch the embedded version of this version of python by doing this in a Terminal window on OSX
/my_main_app/Contents/Frameworks/Python.framework/Versions/2.7/bin/python
From my main application I'd like to be able to run a command in the version of python embedded in the main app that launches an external copy of a python script using the above python version just like I would if I did the following command in a Terminal window. The new launched orphan process should have its own Terminal window so the user can interact with it.
/my_main_app/Contents/Frameworks/Python.framework/Versions/2.7/bin/python my_python_script
I would like the child python instance not to block the main application and I'd like it to have its own terminal window so the user can interact with it. The main application doesn't need to be aware of the child once its launched in any way. The only reason I would do this is to automate launching an external application using a Terminal for the user
If you're trying to launch a new terminal window to run a new Python in (which isn't what your question asks for, but from a comment it sounds like it's what you actually want):
You can't. At least not in a general-purpose, cross-platform way.
Python is just a command-line program that runs with whatever stdin/stdout/stderr it's given. If those happen to be from a terminal, then it's running in a terminal. It doesn't know anything about the terminal beyond that.
If you need to do this for some specific platform and some specific terminal program—e.g., Terminal.app on OS X, iTerm on OS X, the "DOS prompt" on Windows, gnome-terminal on any X11 system, etc.—that's generally doable, but the way to do it is by launching or scripting the terminal program and telling it to open a new window and run Python in that window. And, needless to say, they all have completely different ways of doing that.
And even then, it's not going to be possible in all cases. For example, if you ssh in to a remote machine and run Python on that machine, there is no way it can reach back to your machine and open a new terminal window.
On most platforms that have multiple possible terminals, you can write some heuristic code that figures out which terminal you're currently running under by just walking os.getppid() until you find something that looks like a terminal you know how to deal with (and if you get to init/launchd/etc. without finding one, then you weren't running in a terminal).
The problem is that you're running Python with the argument &. Python has no idea what to do with that. It's like typing this at the shell:
/usr/bin/python '&'
In fact, if you pay attention, you're almost certainly getting something like this through your stderr:
python: can't open file '&': [Errno 2] No such file or directory
… which is exactly what you'd get from doing the equivalent at the shell.
What you presumably wanted was the equivalent of this shell command:
/usr/bin/python &
But the & there isn't an argument at all, it's part of sh syntax. The subprocess module doesn't know anything about sh syntax, and you're telling it not to use a shell, so there's nobody to interpret that &.
You could tell subprocess to use a shell, so it can do this for you:
cmdline = '{} &'.format(python_path)
p = subprocess.Popen(cmdline, shell=True)
But really, there's no good reason to. Just opening a subprocess and not calling communicate or wait on it already effectively "puts it in the background", just like & does on the shell. So:
args = [python_path]
p = subprocess.Popen(args)
This will start a new Python interpreter that sits there running in the background, trying to use the same stdin/stdout/stderr as your parent. I'm not sure why you want that, but it's the same thing that using & in the shell would have done.
Actually I think there might be a solution to your problem, I found a useful solution at another question here.
This way subprocess.popen starts a new python shell instance and runs the second script from there. It worked perfectly for me on Windows 10.
You can try using screen command
with this command a new shell instance created and the current instance runs in the background.
# screen; python script1.py
After running above command, a new shell prompt will be seen where we can run another script and script1.py will be running in the background.
Hope it helps.
I want to run a csh file from a python script,
example,
#!/usr/bin/python
import os
os.system("source path/to/file.csh")
and I want this file to run in the same shell as I am running the python script, because the file.csh script is settings some environment variables that I need.
Does anyone know how to do this in Python?
A child process cannot affect the environment of the parent process. The best you can do is to run your csh script in a separate process, get the environment variables that it defines, then set each environment variable in your python script.
Even with that, the python script won't be able to affect the shell in which you run the python script.
The common way to solve this (AFAIK) is to have your script emit shell commands to set the environment, then from the main shell you run the script and eval what you get back.
For more information you might want to check out this question: can a shell script set environment variables of the calling shell
You can kludge it this way:
#!/usr/bin/env python
# This is kludge.py
print "setenv VARNAME \"the value\""
In your case, you can have the file.sh print the setenv line.
Then from csh:
$ eval `./kludge.py`
$ echo $VARNAME
the value
This isn't clean, but it is the only way to have a child process effect the environment of its parent. This is only because the parent process is explicitly letting it happen with eval.