Develop Reference

launch external shell python instance in shell from python

I'd like to call a separate non-child python program from a python script and have it run externally in a new shell instance. The original python script doesn't need to be aware of the instance it launches, it shouldn't block when the launched process is running and shouldn't care if it dies. This is what I have tried which returns no error but seems to do nothing...
import subprocess
python_path = '/usr/bin/python'
args = [python_path, '&']
p = subprocess.Popen(args, shell=True)
What should I be doing differently
EDIT
The reason for doing this is I have an application with a built in version of python, I have written some python tools that should be run separately alongside this application but there is no assurance that the user will have python installed on their system outside the application with the builtin version I'm using. Because of this I can get the python binary path from the built in version programatically and I'd like to launch an external version of the built in python. This eliminates the need for the user to install python themselves. So in essence I need a simple way to call an external python script using my current running version of python programatically.
I don't need to catch any output into the original program, in fact once launched I'd like it to have nothing to do with the original program
EDIT 2
So it seems that my original question was very unclear so here are more details, I think I was trying to over simplify the question:
I'm running OSX but the code should also work on windows machines.
The main application that has a built in version of CPython is a compiled c++ application that ships with a python framework that it uses at runtime. You can launch the embedded version of this version of python by doing this in a Terminal window on OSX
/my_main_app/Contents/Frameworks/Python.framework/Versions/2.7/bin/python
From my main application I'd like to be able to run a command in the version of python embedded in the main app that launches an external copy of a python script using the above python version just like I would if I did the following command in a Terminal window. The new launched orphan process should have its own Terminal window so the user can interact with it.
/my_main_app/Contents/Frameworks/Python.framework/Versions/2.7/bin/python my_python_script
I would like the child python instance not to block the main application and I'd like it to have its own terminal window so the user can interact with it. The main application doesn't need to be aware of the child once its launched in any way. The only reason I would do this is to automate launching an external application using a Terminal for the user

If you're trying to launch a new terminal window to run a new Python in (which isn't what your question asks for, but from a comment it sounds like it's what you actually want):
You can't. At least not in a general-purpose, cross-platform way.
Python is just a command-line program that runs with whatever stdin/stdout/stderr it's given. If those happen to be from a terminal, then it's running in a terminal. It doesn't know anything about the terminal beyond that.
If you need to do this for some specific platform and some specific terminal program—e.g., Terminal.app on OS X, iTerm on OS X, the "DOS prompt" on Windows, gnome-terminal on any X11 system, etc.—that's generally doable, but the way to do it is by launching or scripting the terminal program and telling it to open a new window and run Python in that window. And, needless to say, they all have completely different ways of doing that.
And even then, it's not going to be possible in all cases. For example, if you ssh in to a remote machine and run Python on that machine, there is no way it can reach back to your machine and open a new terminal window.
On most platforms that have multiple possible terminals, you can write some heuristic code that figures out which terminal you're currently running under by just walking os.getppid() until you find something that looks like a terminal you know how to deal with (and if you get to init/launchd/etc. without finding one, then you weren't running in a terminal).

The problem is that you're running Python with the argument &. Python has no idea what to do with that. It's like typing this at the shell:
/usr/bin/python '&'
In fact, if you pay attention, you're almost certainly getting something like this through your stderr:
python: can't open file '&': [Errno 2] No such file or directory
… which is exactly what you'd get from doing the equivalent at the shell.
What you presumably wanted was the equivalent of this shell command:
/usr/bin/python &
But the & there isn't an argument at all, it's part of sh syntax. The subprocess module doesn't know anything about sh syntax, and you're telling it not to use a shell, so there's nobody to interpret that &.
You could tell subprocess to use a shell, so it can do this for you:
cmdline = '{} &'.format(python_path)
p = subprocess.Popen(cmdline, shell=True)
But really, there's no good reason to. Just opening a subprocess and not calling communicate or wait on it already effectively "puts it in the background", just like & does on the shell. So:
args = [python_path]
p = subprocess.Popen(args)
This will start a new Python interpreter that sits there running in the background, trying to use the same stdin/stdout/stderr as your parent. I'm not sure why you want that, but it's the same thing that using & in the shell would have done.

Actually I think there might be a solution to your problem, I found a useful solution at another question here.
This way subprocess.popen starts a new python shell instance and runs the second script from there. It worked perfectly for me on Windows 10.

You can try using screen command
with this command a new shell instance created and the current instance runs in the background.
# screen; python script1.py
After running above command, a new shell prompt will be seen where we can run another script and script1.py will be running in the background.
Hope it helps.

Running a python app alongside the live interpereter

I'm making a drawing program with python and pygame. I am trying to incorporate a script-fu thing in which the program opens a python live interpreter upon startup and allows the user to execute commands in the interpreter alongside the graphical interface.
My current strategy is to run the main loop inside its own thread, then have the application opened using a bash script that does 'python -i main.py'
Is this a safe/effective/ideal way of doing this? How can I use locks to ensure that commands coming in from the interpreter are executed between main loop iterations?
This is my first time using threads, so please explain to me like I am 7.
Thanks :)

The interpreter won't cooperate with locks you set (since it doesn't know about them). Thus, you cannot guarantee when the code entered by the user will execute.
Consider using the code module to build your own interactive console (it's really easy!). Then you can do the locking every time you go to execute user input.

Why are you using a third party live interpreter? Do you realize that pygame comes with one built in? The documentation is here. This will eliminate all of your problems quite easily.

Send commands to an interactive shell from Python

Is there a way to send command to another interactive shell ? Let's take the example of the meterpreter shell used in metasploit. Could it be a way to say command to this shell from python code, as soon as I get control of a computer and have a meterpreter shell to play with ?
I mean All this from python code.

pexpect may be useful: http://pypi.python.org/pypi/pexpect/2.4

It will not be easy at all.
You will have to know if meterpreter has any means for other programs to communicate with it.
If it doesn't, you might want to go through hacking through it, e.g using OS pipes, etc to be able to get it to work.
In any case, the code needed for such communication might be beyond Python's power.

Run pdb without stdin/stdout using FIFO

I am developing FUSE filesystem with python. The problem is that after mounting a filesystem I have no access to stdin/stdout/stderr from my fuse script. I don't see anything, even tracebacks. I am trying to launch pdb like this:
import pdb
pdb.Pdb(None, open('pdb.in', 'r'), open('pdb.out', 'w')).set_trace()
All works fine but very inconvenient. I want to make pdb.in and pdb.out as fifo files but don't know how to connect it correctly. Ideally I want to type commands and see output in one terminal, but will be happy even with two terminals (in one put commands and see output in another). Questions:
1) Is it better/other way to run pdb without stdin/stdout?
2) How can I redirect stdin to pdb.in fifo (All what I type must go to pdb.in)? How can I redirect pdb.out to stdout (I had strange errors with "cat pdb.out" but maybe I don't understand something)

Ok. Exactly what I want, has been done in http://pypi.python.org/pypi/rpdb/0.1.1 .

Before starting the python app
mkfifo pdb.in
mkfifo pdb.out
Then when pdb is called you can interact with it using these two cat commands, one running in the background
cat pdb.out & cat > pdb.in
Note the readline support does not work (i.e. up arrow)

I just ran into a similar issue in a much simpler use-case:
debug a simple Python program running from the command line that had a file piped into sys.stdin, meaning, no way to use the console for pdb.
I ended up solving it by using wdb.
Quick rundown for my use-case. In the shell, install both the wdb server and the wdb client:
pip install wdb.server wdb
Now launch the wdb server with:
wdb.server.py
Now you can navigate to localhost:1984 with your browser and see an interface listing all Python programs running. The wdb project page above has instructions on what you can do if you want to debug any of these running programs.
As for a program under your control, you can you can debug it from the start with:
wdb myscript.py --script=args < and/stdin/redirection
Or, in your code, you can do:
import wdb; wdb.set_trace()
This will pop up an interface in your browser (if local) showing the traced program.
Or you can navigate to the wdb.server.py port to see all ongoing debugging sessions on top of the list of running Python programs, which you can then use to access the specific debugging session you want.
Notice that the commands for navigating the code during the trace are different from the standard pdb ones, for example, to step into a function you use .s instead of s and to step over use .n instead of n. See the wdb README in the link above for details.

Python IDE on Linux Console

This may sound strange, but I need a better way to build python scripts than opening a file with nano/vi, change something, quit the editor, and type in python script.py, over and over again.
I need to build the script on a webserver without any gui. Any ideas how can I improve my workflow?

put this line in your .vimrc file:
:map <F2> :w\|!python %<CR>
now hitting <F2> will save and run your python script

You should give the screen utility a look. While it's not an IDE it is some kind of window manager on the terminal -- i.e. you can have multiple windows and switch between them, which makes especially tasks like this much easier.

You can execute shell commands from within vim.

Using emacs with python-mode you can execute the script with C-c C-c

you can try ipython. using its edit command, it will bring up your editor (nano/vim/etc), you write your script, and then on exiting you're returned to the ipython prompt and the script is automatically executed.

When working with Vim on the console, I have found that using "tabs" in Vim, instead of having multiple Vim instances suspended in the background, makes handling multiple files in Vim more efficient. It takes a bit of getting used to, but it works really well.

You could run XVNC over ssh, which is actually passably responsive for doing this sort of thing and gets you a windowing GUI. I've done this quite effectively over really asthmatic Jetstart DSL services in New Zealand (128K up/ 128K down =8^P) and it's certainly responsive enough for gvim and xterm windows. Another option would be screen, which lets you have multiple textual sessions open and switch between them.

There are actually 2 questions. First is polling for a console IDE for python and the second is a better dev/test/deploy workflow.
For while there are many ways you can write python code in the console, I find a combination of screen, vim and python/ipython is the best as they are usually available on most servers. If you are doing long sessions, I find emacs + python-mode typically involves less typing.
For a better workflow, I would suggest setting up a development environment. You can easily setup a Linux VM on your desktop/laptop easily these days - there isn't a excuse not to even if it's for hobby projects. That opens up a much larger selection of IDEs available to you, such as:
GUI versions of VI and friends
Remote file editing with tramp and testing locally with python-mode inside Emacs
http://www.netbeans.org
and of course http://eclipse.org with the PyDev plugin
I would also setup a SCM to keep track of changes so that you do
better QA and use it to deploy tested changes onto the server.
For example I use Mercurial for my pet projects and I simply tag my repo when it's ready and update the production server to the tag when I deploy. On the devbox, I do:
(hack hack hack, test test test)
hg ci -m 'comment'
hg tag
hg push
Then I jump onto the server and do the following when I deploy:
hg update
restart service/webserver as needed

Well, apart from using one of the more capable console editors (Emacs or vi would come to mind), why do you have to edit it on the web server itself? Just edit it remotely if constant FTP/WebDAV transfer would seem to cumbersome.
Emacs has Tramp Mode, gedit on Linux and bbedit on the Mac support remote editing, too. Probably quite a large number of other editors. In that case you would just edit in on a more capable desktop and restart the script from a shell window.

For what it's worth, VIM alone can do the same tasks as previously posted. I have had the same problem with testing Python from the command line.
My solution was to use the screen command. I split screens vertically, I run Python in one instance of a shell, and on the second screen, I usually edit Python code with VIM.
Command to install screen:
sudo apt-get install screen
The screen package has a bit of a learning curve but there isn't any mystery if you can remember the "Ctrl-Alt ?" command that contains all knowledge.
No GUI is required!