Updated post:
I have a python web application running on a port. It is used to monitor some other processes and one of its features is to allow users to restart his own processes. The restart is done through invoking a bash script, which will proceed to restart those processes and run them in the background.
The problem is, whenever I kill off the python web application after I have used it to restart any user's processes, those processes will take take over the port used by the python web application in a round-robin fashion, so I am unable to restart the python web application due to the port being bounded. As a result, I must kill off the processes involved in the restart until nothing occupies the port the python web application uses.
Everything is ok except for those processes occupying the port. That is really undesirable.
Processes that may be restarted:
redis-server
newrelic-admin run-program (which spawns another web application)
a python worker process
UPDATE (6 June 2013): I have managed to solve this problem. Look at my answer below.
Original Post:
I have a python web application running on a port. This python program has a function that calls a bash script. The bash script spawns a few background processes, then exits.
The problem is, whenever I kill the python program, the background processes spawned by the bash script will take over and occupy that same port.
Specifically the subprocesses are:
a redis server (with daemonize = true in the configuration file)
newrelic-admin run-program (spawns a web application)
a python worker process
Update 2: I've tried running these with nohup. Only the python worker process doesnt attempt to take over the port after I kill the python web application. The redis server and newrelic-admin still do.
I observed this problem when I was using subprocess.call in the python program to run the bash script. I've tried a double fork method in the python program before running the bash script, but it results in the same problem.
How can I prevent any processes spawned from the bash script from taking over the port?
Thank you.
Update: My intention is that, those processes spawned by the bash script should continue running if the python application is killed off. Currently, they do continue running after I kill off the python application. The problem is, when I kill off the python application, the processes spawned by the bash script start to take over the port in a round-robin fashion.
Update 3: Based on the output I see from 'pstree' and 'ps -axf', processes 1 and 2 (the redis server and the web app spawned by newrelic-admin run-program) are not child processes of the python web application. This makes it even weirder that they take over the port that the python web application occupies when I kill it... Anyone knows why?
Just some background on the methods I've tried to solve my above problem, before I go on to the answer proper:
subprocess.call
subprocess.Popen
execve
the double fork method along with one of the above (http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/)
By the way, none of the above worked for me. Whenever I killed off the web application that executes the bash script (which in turns spawns some background processes we shall denote as Q now), the processes in Q will in a round-robin fashion, take over the port occupied by the web application, so I had to kill them one by one before I could restart my web application.
After many days of living with this problem and moving on to other parts of my project, I thought of some random Stack Overflow posts and other articles on the Internet and from my own experience, recalled my experience of ssh'ing into a remote and starting a detached screen session, then logging out, and logging in again some time later to discover the screen session still alive.
So I thought, hey, what the heck, nothing works so far, so I might as well try using screen to see if it can solve my problem. And to my great surprise and joy it does! So I am posting this solution hopefully to help those who are facing the same issue.
In the bash script, I simply started the processes using a named screen process. For instance, for the redis application, I might start it like this:
screen -dmS redisScreenName redis-server redis.conf
So those processes will keep running on those detached screen sessions they were started with. In this case, I did not daemonize the redis process.
To kill the screen process, I used:
screen -S redisScreenName -X quit
However, this does not kill the redis-server. So I had to kill it separately.
Now, in the python web application, I can just use subprocess.call to execute the bash script, which will spawn detached screen sessions (using 'screen -dmS') which run the processes I want to spawn. And when I kill off the python web application, none of the spawned processes take over its port. Everything works smoothly.
Related
I am working on Unix systems and have a GUI application that in turn spawns couple of other processes. These processes required to run independent of the parent process (GUI application). Basically, when the GUI is crashed or closed, the child processes should keep running.
One approach could be to demonize the processes. Here is an useful answer that runs a process in background through double forking.
What I would like to ask is, if it is possible to have the same result using terminal-multiplexer, like tmux or GNU-Screen. I am not sure how these terminal-multiplexers creates and maintain shell sessions but the basic idea would be to start the GUI application, that uses 'tmux' or 'screen' to creates a shell session and run child processes within the shell session. Would it make the child process independent of parent processes?
Thanks in advance!
It should work if your GUI runs something like this:
tmux new-session -s test -d vim
which creates a detached session named "test", running the "vim" command. The session can then be attached with:
tmux attach-session -t test
I have been trying to start app in remote hosts using Fabric.
I was running a daemonized Java app, which run perfectly fine when I logged in and run it manually. The app keep running even after I exit the session.
But when I use Fabric run(), my app terminates once the session ends.
Although run(command, pty=False) solved my problem (It is roughly documented here), I still can not see why these are relevant. I am a python newbie, can anyone explain the difference between:
start daemon with ssh logged in and start manually
start daemon using Fabric with pty=True
start daemon using Fabric with pty=False
I'm trying to launch a background process (a Python daemon to be exact) on my iPod (ios 6.1.6 jailbroken), but the Python process gets suspended eventually by the os.
Here are the things I tried to make it run in background:
Launch the process via MobileTerminal. It gets suspended when I hit the home button, same when installing MobileTerminal Backgrounder or configuring the MobileTerminal app to run in background mode with Background Manager.
SSH connect and launch the process with nohup, it also gets suspended. Setting the highest priority with nice also does not work.
Does anyone have some ideas how to achieve this? This must be possible, because processes like the openssh server seem to have permanent background.
I looked on the internet for quite a while, and the only helpful thing I found that might work is to set up a launch daemon, but this makes it not easy to stop / restart the process if I have to restart the device, and I do not know if it works for sure.
Thanks.
I have written a module in Python and want it to run continuously once started and need to stop it when I need to update other modules. I will likely be using monit to restart it, if module has crashed or is otherwise not running.
I was going through different techniques like Daemon, Upstart and many others.
Which is the best way to go so that I use that approach through out my all new modules to keep running them forever?
From your mention of Upstart I will assume that this question is for a service being run on an Ubuntu server.
On an Ubuntu server an upstart job is really the simplest and most convenient option for creating an always on service that starts up at the right time and can be stopped or reloaded with familiar commands.
To create an upstart service you need to add a single file to /etc/init. Called <service-name>.conf. An example script looks like this:
description "My chat server"
author "your#email-address.com"
start on runlevel [2345]
stop on runlevel [!2345]
env AN_ENVIRONMENTAL_VARIABLE=i-want-to-set
respawn
exec /srv/applications/chat.py
This means that everytime the machine is started it will start the chat.py program. If it dies for whatever reason it will restart it. You don't have to worry about double forking or otherwise daemonizing your code. That's handled for you by upstart.
If you want to stop or start your process you can do so with
service chat start
service chat stop
The name chat is automatically found from the name of the .conf file inside /etc/init
I'm only covering the basics of upstart here. There are lots of other features to make it even more useful. All available by running man upstart.
This method is much more convenient, than writing your own daemonization code. A 4-8 line config file for a built in Ubuntu component is much less error prone than making your code safely double fork and then having another process monitor it to make sure it doesn't go away.
Monit is a bit of a red herring. If you want downtime alerts you will need to run a monitoring program on a separate server anyway. Rely on upstart to keep the process always running on a server. Then have a different service that makes sure the server is actually running. Downtime happens for many different reasons. A process running on the same server will tell you precisely nothing if the server itself goes down. You need a separate machine (or a third party provider like pingdom) to alert you about that condition.
You could check out supervisor. What it is capable of is starting a process at system startup, and then keeping it alive until shutdown.
The simplest configuration file would be:
[program:my_script]
command = /home/foo/bar/venv/bin/python /home/foo/bar/scripts/my_script.py
environment = MY_ENV_VAR=FOO, MY_OTHER_ENV_VAR=BAR
autostart = True
autorestart = True
Then you could link it to /etc/supervisord/conf.d, run sudo supervisorctl to enter management console of supervisor, type in reread so that supervisor notices new config entry and update to display new programs on the status list.
To start/restart/stop a program you could execute sudo supervisorctl start/restart/stop my_script.
I used old-style initscript with start-stop-daemon utility.Look at skel in /etc/init.d
I use IDEA 10.5 for my Flask experimentation. Flask has en embedded test server (like Django does)
When I launch my test class, the dev server launches as well on port 5000. All good.
* Running on http://127.0.0.1:5000/
When I click on the "Stop process" button (red square), I get the message saying the process is finished :
Process finished with exit code 143
However the server is still alive (responds to requests) and I can see I still have a python process running.
Obviously this prevents me from relaunching the test straight away, I have to kill the server process first.
How do you manage to get both your program and the server ending at the same time ?
I guess what happens is that you start your flask app which then is forking the development server as a new process. If you stop the app the forked process is still running.
This looks like a problem, that cannot easily be solved within the means of your IDE. You could add something to your main to kill the already running server process, before starting the app again, but that seems ugly.
But why don't you just start your app with app.run(debug=True) as described in flask doc? The server will reload automatically everytime you changed your app so you don't have to stop and restart it manually.
EDIT:
Something a bit quirky just came to my mind: if you just need a comfortable way to kill the server from within the IDE all you have to do is to introduce a syntactical error in one of the places the reloader monitors, save the file and the server will choke on it and die :)
This doesn't happen anymore with newer versions (tested with PyCharm 2.0)