Develop Reference

running command line in multiple running processes from python

I'm not sure if what I'm wanting to do is possible, but:
I have a python script (lets call it PY) that calls a batch script to start a tool in terminal mode (lets call it A). This tool gets passed a starting script (tcl script) that sets up its environment and launches a second tool (lets call it B). The two tools communicate over a TCP connection locally.
My question is, with these two programs running (A and B), can I switch back to the python script to run commands in either A or B's TCL interface?
The scripts look sort of like this:
#python PY
def ReadConigAndSetup():
#read some data
...
#run bat
subprocess.run("./some_bat.bat some_data_args")
#bat start program A and pass it a startup script
some_program_A -mode tcl -source ./some_source.tcl
#tcl some_source.tcl
setup environment
open TCP port
start program B
#program B setup tcl
some more setup
after program B has run I'd like to be able to run more commands in program B from python as parsing some of the config files is much easier in the python environment.

The answer is “it depends on the details”.
There's no reason in principle why the program being called can't work fine this way, provided the subprocess relinquishes control back (which it might or might not), but launching complex programs via a BAT file is adding an extra layer of complexity so you might want to think about whether you can simplify a bit there.
If the program running the Tcl code doesn't terminate, things get trickier. This is an area where the details are critical; Tcl code can be written to loop indefinitely — it's a programming language so of course it can be told to be annoying if you insist — and the program being controlled could also decide to loop indefinitely of its own accord, which can happen particularly with GUI applications as the looping is where the user is interacting with the GUI. On Windows, many GUI applications run disconnected from the terminal (whether they do this is a compile-time option) and waiting for them to finish can be quite annoying.
It's possible to run multiple subprocesses at once using subprocess.Popen. Be very careful if you do this. It's possible to get into deadlocks (though that depends a lot on what the subprocesses are doing). It's probably easier to just launch each subprocess from its own thread… but then you're dealing with threads and that's also complicated.

How to free Ubuntu memory once python script is stopped from terminal (Ctrl-C)?

I was calling a python script (2.7) from a console (Ubuntu 14.04) using a command: python script_name.py. At some point, I wanted to stop the running script by pressing Ctrl-C. However, when I checked Ubuntu System Monitor, the memory used by the python script was not freed up (I monitored Ubuntu System Monitor before I called the script, during the process, and after I pressed Ctrl-C to stop the script). I tried to free up the memory using a command explained on http://www.upubuntu.com/2013/01/how-to-free-up-unused-memory-in.html , but didn't work (I mean, the memory usage was not changed).
However, if I used pycharm to run and stop the script, the memory was freed up directly once I pressed the Stop button. For some reasons (such as from ssh or just to test from console), I want to run my script from the console (without using pycharm or any other IDEs).
My question is, what is the command, or how to stop running python script and free up directly the memory used by the script, if I run the script from the console?
Many thanks in advance.

Those commands did not work since what you're trying to achieve is not what they do. How did you check the memory being used by your Python script. I use top to see memory and could used by each process (sorted in ascending order by default). You may have checked before the system had time to register that the python process was killed, I've used this a lot and I've never tun into with the OS not getting memory back after a process has been killed with ctrl + c.
Pycharm is probably doing some cleanup when you stop the program from it versus just having to wait for the OS to reclaim memory versys when you SIGTERM a process from a shell

Alternative in python to subprocess

I am trying to write a script which has to make a lot of calls to some bash commands, parse and process the outputs and finally give some output.
I was using subprocess.Popen and subprocess.call
If I understand correct these methods spawn a bah process, run the command, get the output and then kill the process.
Is there a way to have a bash process running in the background continuously and then the python calls could just go directly to that process? This would be something like bash running as a server and python calls going to it.
I feel this would optimize the calls a bit as there is no bash process setup and teardown. Or will it give no performance advantage?

I feel this would optimize the calls a bit as there is no bash process setup and teardown.
subprocess never runs the shell unless you ask it explicitly e.g.,
#!/usr/bin/env python
import subprocess
subprocess.check_call(['ls', '-l'])
This call runs ls program without invoking /bin/sh.
Or will it give no performance advantage?
If your subprocess calls actually use the shell e.g., to specify a pipeline consicely or you use bash process substitution that could be verbose and error-prone to define using subprocess module directly then it is unlikely that invoking bash is a performance bottleneck -- measure it first.
There are Python packages that too allow to specify such commands consicely e.g., plumbum could be used to emulate a shell pipeline.
If you want to use bash as a server process then pexpect is useful for dialog-based interactions with an external process -- though it is unlikely that it affects time performance. fabric allows to run both local and remote commands (ssh).
There are other subprocess wrappers such as sarge which can parse a pipeline specified in a string without invoking the shell e.g., it enables cross-platform support for bash-like syntax (&&, ||, & in command lines) or sh -- a complete subprocess replacement on Unix that provides TTY by default (it seems full-featured but the shell-like piping is less straightforward). You can even use Python-ish BASHwards-looking syntax to run commands with xonsh shell.
Again, it is unlikely that it affects performance in a meaningful way in most cases.
The problem of starting and communicating with external processes in a portable manner is complex -- the interaction between processes, pipes, ttys, signals, threading, async. IO, buffering in various places has rough edges. Introducing a new package may complicate things if you don't know how a specific package solve numerous issues related to running shell commands.

If I understand correct these methods spawn a bah process, run the command, get the output and then kill the process.
subprocess.Popen is a bit more involved. It actually creates an I/O thread to avoid deadlocks. See https://www.python.org/dev/peps/pep-0324/:
A communicate() method, which makes it easy to send stdin data and read stdout and stderr data, without risking deadlocks. Most people are aware of the flow control issues involved with child process communication, but not all have the patience or skills to write a fully correct and deadlock-free select loop. This means that many Python applications contain race conditions. A communicate() method in the standard library solves this problem.
Is there a way to have a bash process running in the background continuously and then the python calls could just go directly to that process?
Sure, you can still use subprocess.Popen and send messages to you subprocess and receive messages back without terminating the subprocess. In the simplest case your messages can be lines.
This allows for request-response style protocols as well as publish-subscribe when the subprocess can keep sending you messages back when an event of interest happens.

How do you terminate the Python shell in Wing IDE?

I'm fairly new to Python and have been using Wing IDE to play around with the features. One of the things that I could find while looking around was how to force terminate the Python shell when executing a command that won't terminate any time soon. An example would be:
import math
math.factorial(1000000)
I know in Visual Studio C++, the command is Ctrl+C, but what exactly is the Python equivalent?

The method used to terminate execution varies between shells. For Wing IDE you use the Restart Shell item on the Options menu.

This depends on your shell. For most shells, it is ctrl-C, or killing the process.
There is no way to do so from within python (unless you are spawning threads or processes) because the thread in question is stuck.

Building python Shell

I have some small python 2.6 scripts built....
Now, I would like run them as seperate processes within a python shell. Each as a seperate process. If one fails to run maybe with its timer, I would like others to continue without killing all scripts.
Should I do this as singleton gui's or combine them into bigger launch pad. My perference would be launch pad type gui....Any ideas?
Its seems that launching scripts out of SciTE, works ok.

Check joblaunch, a shell tool I made for executing interdependent jobs in parallel locally. It has more options.