Is there a way in python to hijack the terminal stdin? Unix only solutions will do fine.
I'm currently writing a small wrapper around top as I want to be able to monitor named processes, e.g. all running python instances. Basically I'm calling pgrep to get process id's and then runs top using the -p option.
Overall this script have worked satisfactorily for a few years now (well with the caveat that top -p only accepts 20 pid's...). However, I now would like adjust the script to update the call to top if new processes matching the name pattern are born. This also works relatively nicely, but... any options set interactively in top gets lost every time I update the pid list but natural causes as I stop and restart top. Therefore I would like to hijack the terminal stdin somehow to be able to backtrack what ever the settings are in affect so I can set them accordingly after updating the pid-list, or even halt updating if neccesary (e.g. if top is awaiting more instructions from the user).
Now perhaps what I'm trying to achieve is just silly and there are better ways to do it, if so I'd highly appreciate enlightenment
(oh. the tag ps were used as the tag top does not exists and I'm to new here to define new tags, after all the two utilities are related)
thanks \p
What you are doing sounds like a bit of a hack. I would just write a Python script using psutil that does exactly what you want. Whatever information you are interested in, psutil should give it to you - and more.
Quick and dirty:
import psutil
import time
while True:
processes = [ p for p in psutil.process_iter() if 'python' in p.name() ]
for p in processes:
# print out whatever information interests you
print(
p.pid,
p.name(),
p.cpu_percent(),
p.io_counters().read_bytes,
p.io_counters().write_bytes
)
time.sleep(10)
Link to Documentation: http://pythonhosted.org/psutil/
Related
I've been using Google Colab for a few weeks now and I've been wondering what the difference is between the two following commands (for example):
!ffmpeg ...
subprocess.Popen(['ffmpeg', ...
I was wondering because I ran into some issues when I started either of the commands above and then tried to stop execution midway. Both of them cancel on KeyboardInterrupt but I noticed that after that the runtime needs a factory reset because it somehow got stuck. Checking ps aux in the Linux console listed a process [ffmpeg] <defunct> which somehow still was running or at least blocking some ressources as it seemed.
I then did some research and came across some similar posts asking questions on how to terminate a subprocess correctly (1, 2, 3). Based on those posts I generally came to the conclusion that using the subprocess.Popen(..) variant obviously provides more flexibility when it comes to handling the subprocess: Defining different stdout procedures or reacting to different returncode etc. But I'm still unsure on what the first command above using the ! as prefix exactly does under the hood.
Using the first command is much easier and requires way less code to start this process. And assuming I don't need a lot of logic handling the process flow it would be a nice way to execute something like ffmpeg - if I were able to terminate it as expected. Even following the answers from the other posts using the 2nd command never got me to a point where I could terminate the process fully once started (even when using shell=False, process.kill() or process.wait() etc.). This got me frustrated, because restarting and re-initializing the Colab instance itself can take several minutes every time.
So, finally, I'd like to understand in more general terms what the difference is and was hoping that someone could enlighten me. Thanks!
! commands are executed by the notebook (or more specifically by the ipython interpreter), and are not valid Python commands. If the code you are writing needs to work outside of the notebook environment, you cannot use ! commands.
As you correctly note, you are unable to interact with the subprocess you launch via !; so it's also less flexible than an explicit subprocess call, though similar in this regard to subprocess.call
Like the documentation mentions, you should generally avoid the bare subprocess.Popen unless you specifically need the detailed flexibility it offers, at the price of having to duplicate the higher-level functionality which subprocess.run et al. already implement. The code to run a command and wait for it to finish is simply
subprocess.check_call(['ffmpeg', ... ])
with variations for capturing its output (check_output) and the more modern run which can easily replace all three of the legacy high-level calls, albeit with some added verbosity.
This Python program enters fdisk. I see the output. fdisk is an interactive program. How do I get the Python program to pass an "m" to the first field and press enter?
import subprocess
a = "dev/sda"
x = subprocess.call(["fdisk", a])
print x
I'd rather not import a new module/library, but I could. I've tried different syntax with subprocess.call() and extra parameters in the above. Nothing seems to work. I get different errors. I've reviewed Python documentation. I want to feed input and press Enter in the subsequent, interactive menu options of fdisk.
Check out the pexpect library (I know you didn't want an extra module, but you want to use the best tool for the job). It's pure Python, with no compiled submodules, so installation is a snap. Basically, it does the same thing in Python as the classic Unix utility expect - spawns child applications, controls them, and responds to expected patterns in their output. It's great for automation, and especially application testing, where you can quickly feed the newest build of a command-line program a series of inputs and guide the interaction based on what output appears.
In case you just don't want another module at all, you can always fall back on the subprocess module's Popen() constructor. It spawns and creates a connection to a child process, allowing you to communicate with it as needed, and in fact pexpect relies a great deal on it. I personally think using pexpect is more intuitive than subprocess.Popen(), but that's just me. YMMV.
How to send string/data to STDIN of a running process in python?
i'd like to create a front end for a CLI program. eg. i want to pass multiple string to this Pascal application:
program spam;
var a,b,c:string;
begin
while e <> "no" do
begin
writeln('what is your name?');
readln(a);
writeln('what is your quest?');
readln(b);
writeln('what is your favorite color?');
readln(c);
print(a,b,c);
end;
end.
how do i pass string to this program from python (using subprocess module in python). thankyou. sorry for my english.
If you want to control another interactive program, it could be worth trying the Pexpect module to do so. It is designed to look for prompt messages and so on, and interact with the program. Note that it doesn't currently work directly on Windows - it does work under Cygwin.
A possible non-Cygwin Windows variant is WinPexpect, which I found via this question. One of the answers on that question suggests the latest version of WinPexpect is at http://sage.math.washington.edu/home/goreckc/sage/wexpect/, but looking at the modification dates I think the BitBucket (the first link) is actually the latest.
As Windows terminals are somewhat different to Unix ones, I don't think there is a direct cross-platform solution. However, the WinPexpect docs say the only difference in the API between it and pexpect is the name of the spawn function. You could probably do something like the following (untested) code to get it to work in both:
try:
import pexpect
spawn = pexpect.spawn
except ImportError:
import winpexpect
spawn = winpexpect.winspawn
# NB. Errors may occur when you run spawn rather than (or as
# well as) when you import it, so you may have to wrap this
# up in a try...except block and handle them appropriately.
child = spawn('command and args')
I want to search for a process and show it,emacs for example,I use
`p = subprocess.Popen('ps -A | grep emacs',shell=True,stdout=subprocess.PIPE)`
to get the process, then how can I wake it up and show it?
in other words,the question shoud be : how python change the state of process?
In short, python has a pty module and look for the solution there.
This question is not that simple as it may look like.
It is simple to change the running state of a process by delivering corresponding signals but it is not simple to manipulate foreground/background properties.
When we talk about manipulating the foreground/background processes, we really talk about 'job control.' In UNIX environment, job control is achieved by coordination of several parts, including kernel, controlling terminal, current shell and almost every process invoked in that shell session. Telling a process to get back to foreground, you have to tell others to shut up and go to background simultaneously. See?
Let's come back to your question. There could be 2 answers to this, one is no way and the other is it could be done but how.
Why 2 answers?
Generally you cannot have job control unless you program for it. You also cannot use a simple pipe to achieve the coordination model which leads to job control mechanism; the reason is essential since you cannot deliver signals through a pipe. That's why the answer is no way, at least no way in a simple pipe implementation.
However, if you have enough patience to program terminal I/O, it still can be done with a lot of labor work. Concept is simple: you cheat your slave program, which is emacs in this example, that it has been attached to a real terminal having a true keyboard and a solid monitor standby, and you prepare your master program, which is the python script, to handle and relay necessary events from its controlling terminal to the slave's pseudo-terminal.
This schema is actually adopted by many terminal emulators. You just need to write another terminal emulator in your case... Wait! Does it have to be done with so much effort, always?
Luckily no.
Your shell manages all the stuff for you in an interactive scenario. You just tell shell to 'fg/bg' the task, quite easy in real life. The designated command combination can be found in shell's manpage. It could look like 'jobs -l | grep emacs' along with 'fg %1'. Nonetheless those combined commands cannot be invoked by a program. It's a different story since a program will start a new shell to interpret its commands and such a new shell cannot control the old running emacs because it doesn't have the privilege. Type it in with your keyboard and read it out on your monitor; that's an interactive scenario.
In an automation scenario, think twice before you employ a job control design because most automation scenarios do not require a job control schema. You need an editor here and a player there, that's all right, but just don't make them to stay "background" and pop to "foreground." They'd better exit when they complete their task.
But if you are unlucky to have to program job control in automation procedures, try to program pseudo-terminal master and slave I/O as well. They look like a sophisticated IPC mechanism and their details are OS-dependent. However this is the standard answer to your question; though annoying, I know.
you can get the output generated by this process, reading the stdout descriptor:
out = p.stdout.read()
Is there a way to generically retrieve process stats using Perl or Python? We could keep it Linux specific.
There are a few problems: I won't know the PID ahead of time, but I can run the process in question from the script itself. For example, I'd have no problem doing:
./myscript.pl some/process/I/want/to/get/stats/for
Basically, I'd like to, at the very least, get the memory consumption of the process, but the more information I can get the better (like run time of the process, average CPU usage of the process, etc.)
Thanks.
Have a look at the Proc::ProcessTable module which returns quite a bit of information on the processes in the system. Call the "fields" method to get a list of details that you can extract from each process.
I recently discovered the above module which has just about replaced the Process module that I had written when writing a Perl kill program for Linux. You can have a look at my script here.
It can be easily extended to extract further information from the ps command. For eg. the 'getbycmd' method returns a list of pid's whose command line invocation matches the passed argument. You can then retrieve a specific process' details by calling 'getdetail' by passing that PID to it like so,
my $psTable = Process->new();
# Get list of process owned by 'root'
for my $pid ( $psTable->getbyuser("root") ) {
$psDetail = $psList->getdetail( $pid );
# Do something with the psDetail..
}
If you are fork()ing the child, you will know it's PID.
From within the parent you can then parse the files in /proc/<PID/ to check the memory and CPU usage, albeit only for as long as the child process is running.
A common misconception is that reading /proc is like reading /home. /proc is designed to give you the same information with one open() that 20 similar syscalls filling some structure could provide. Reading it does not pollute buffers, send innocent programs to paging hell or otherwise contribute to the death of kittens.
Accessing /proc/foo is just telling the kernel "give me information on foo that I can process in a language agnostic way"
If you need more details on what is in /proc/{pid}/ , update your question and I'll post them.