I was trying to work something out with 3-4 python processes that run in parallel. I start/stop the service and interact with it using FastAPI. I wanted to add the functionality to pause the process (from multiprocessing.process) from the FastAPI dashboard and run it again by resuming it in some way. Is there a way to temporarily halt a python process? So far I could only terminate the process altogether. Thanks!
I am working on Unix systems and have a GUI application that in turn spawns couple of other processes. These processes required to run independent of the parent process (GUI application). Basically, when the GUI is crashed or closed, the child processes should keep running.
One approach could be to demonize the processes. Here is an useful answer that runs a process in background through double forking.
What I would like to ask is, if it is possible to have the same result using terminal-multiplexer, like tmux or GNU-Screen. I am not sure how these terminal-multiplexers creates and maintain shell sessions but the basic idea would be to start the GUI application, that uses 'tmux' or 'screen' to creates a shell session and run child processes within the shell session. Would it make the child process independent of parent processes?
Thanks in advance!
It should work if your GUI runs something like this:
tmux new-session -s test -d vim
which creates a detached session named "test", running the "vim" command. The session can then be attached with:
tmux attach-session -t test
I have a linux daemon (based on python module python-daemon) that needs to spawn two processes (consider a producer and a consumer) of the Multiprocessing module to handle some concurrent I/O (the producer reads from an input stream and the consumer uploads the data using python requests).
As per the python docs (https://docs.python.org/2/library/multiprocessing.html), daemonic processes are not allowed to start child processes. How can I handle this? Are there any documents or examples for this approach?
Pls. advise.
Context:
I have tried using threading module. But, due to the GIL problem, the consumer rarely gets a chance to execute. I also looked into tornado and gevent. But, that would require rewriting a lot of the code.
I think there is some confusion here. Document says only if you mark the process that has been created from python as daemon then it cannot create sub process. But your python-daemon is a normal linux daemon.
linux daemon - process running in background. (python daemon library creates such process), these can have subprocess
Only a daemon process created from multiprocessing library cannot create sub-process.
I'm trying to implement a simple scheduler on the machine that I share with my colleague. The idea is to run a process in the background as a server, identified by its pid. I can submit the program I want to run to this server process, say in a different bash terminal, and let the server process schedule the job regarding the availability of the hardware resource. The submit program should be able to lock some content in memory and communicate with the server.
I was trying to use python multiprocessing or subprocess module to do the above thing. But I didn't have a clear idea how this should be done. Any help would be appreciated.
I think cron job or celery would be a better choice for you use case. Just create the celery task and delay its execution according to your needs.
Updated post:
I have a python web application running on a port. It is used to monitor some other processes and one of its features is to allow users to restart his own processes. The restart is done through invoking a bash script, which will proceed to restart those processes and run them in the background.
The problem is, whenever I kill off the python web application after I have used it to restart any user's processes, those processes will take take over the port used by the python web application in a round-robin fashion, so I am unable to restart the python web application due to the port being bounded. As a result, I must kill off the processes involved in the restart until nothing occupies the port the python web application uses.
Everything is ok except for those processes occupying the port. That is really undesirable.
Processes that may be restarted:
redis-server
newrelic-admin run-program (which spawns another web application)
a python worker process
UPDATE (6 June 2013): I have managed to solve this problem. Look at my answer below.
Original Post:
I have a python web application running on a port. This python program has a function that calls a bash script. The bash script spawns a few background processes, then exits.
The problem is, whenever I kill the python program, the background processes spawned by the bash script will take over and occupy that same port.
Specifically the subprocesses are:
a redis server (with daemonize = true in the configuration file)
newrelic-admin run-program (spawns a web application)
a python worker process
Update 2: I've tried running these with nohup. Only the python worker process doesnt attempt to take over the port after I kill the python web application. The redis server and newrelic-admin still do.
I observed this problem when I was using subprocess.call in the python program to run the bash script. I've tried a double fork method in the python program before running the bash script, but it results in the same problem.
How can I prevent any processes spawned from the bash script from taking over the port?
Thank you.
Update: My intention is that, those processes spawned by the bash script should continue running if the python application is killed off. Currently, they do continue running after I kill off the python application. The problem is, when I kill off the python application, the processes spawned by the bash script start to take over the port in a round-robin fashion.
Update 3: Based on the output I see from 'pstree' and 'ps -axf', processes 1 and 2 (the redis server and the web app spawned by newrelic-admin run-program) are not child processes of the python web application. This makes it even weirder that they take over the port that the python web application occupies when I kill it... Anyone knows why?
Just some background on the methods I've tried to solve my above problem, before I go on to the answer proper:
subprocess.call
subprocess.Popen
execve
the double fork method along with one of the above (http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/)
By the way, none of the above worked for me. Whenever I killed off the web application that executes the bash script (which in turns spawns some background processes we shall denote as Q now), the processes in Q will in a round-robin fashion, take over the port occupied by the web application, so I had to kill them one by one before I could restart my web application.
After many days of living with this problem and moving on to other parts of my project, I thought of some random Stack Overflow posts and other articles on the Internet and from my own experience, recalled my experience of ssh'ing into a remote and starting a detached screen session, then logging out, and logging in again some time later to discover the screen session still alive.
So I thought, hey, what the heck, nothing works so far, so I might as well try using screen to see if it can solve my problem. And to my great surprise and joy it does! So I am posting this solution hopefully to help those who are facing the same issue.
In the bash script, I simply started the processes using a named screen process. For instance, for the redis application, I might start it like this:
screen -dmS redisScreenName redis-server redis.conf
So those processes will keep running on those detached screen sessions they were started with. In this case, I did not daemonize the redis process.
To kill the screen process, I used:
screen -S redisScreenName -X quit
However, this does not kill the redis-server. So I had to kill it separately.
Now, in the python web application, I can just use subprocess.call to execute the bash script, which will spawn detached screen sessions (using 'screen -dmS') which run the processes I want to spawn. And when I kill off the python web application, none of the spawned processes take over its port. Everything works smoothly.