I have a simple question. I have tried to search for a solution but there are no answers which would explain what I need.
The question is:
How do I start a nohup command from Python? Basically the idea is, that I have a Python script which prepares my environment and I need it to launch multiple scripts with nohup commands. How do I start a nohup command like nohup python3 my_script.py & from within a running Python script to have that nohup command running even after I log out?
Thank you
You do not need nohup -- not even in shell, and even less so in Python. It does the following things:
Configures the HUP signal to be ignored (rarely relevant: if a process has no handles on a TTY it isn't going to be notified when that TTY exits regardless; the shell only propagates signals to children in interactive mode, not when running scripts).
If stdout is a terminal, redirects it to nohup.out
If stderr is a terminal, redirects it to wherever stdout was already redirected.
Redirects stdin to /dev/null
That's it. There's no reason to use nohup to do any of those things; they're all trivial to do without it:
</dev/null redirects stdin from /dev/null in shell; stdin=subprocess.DEVNULL does so in Python.
>nohup.out redirects stdout to nohup.out in shell; stdout=open('nohup.out', 'w') does so in Python.
2>&1 makes stderr go to the same place as stdout in shell; stderr=subprocess.STDOUT does so in Python.
Because your process isn't attached to the terminal by virtue of the above redirections, it won't implicitly get a HUP when that terminal closes. If you're worried about a signal being sent to the parent's entire process group, however, you can avoid that by splitting off the child into a separate one:
The subprocess.Popen argument start_new_session=True splits the child process into a separate group from the parent in Python, so a parent sent to the process group of the parent as a whole will not be received by the child.
Adding a preexec_fn with signal.signal(signal.SIGHUP, signal.SIG_IGN) is even more explicit that the child should by default ignore a SIGHUP even if one is received.
Putting this all together might look like (if you really do want logs to go to a file named nohup.out -- I would suggest picking a better name):
import subprocess, signal
subprocess.Popen(['python3', 'my_script.py'],
stdin=subprocess.DEVNULL,
stdout=open('nohup.out', 'w'),
stderr=subprocess.STDOUT,
start_new_session=True,
preexec_fn=(lambda: signal.signal(signal.SIGHUP, signal.SIG_IGN)))
Related
I have Python script bgservice.py and I want it to run all the time, because it is part of the web service I build. How can I make it run continuously even after I logout SSH?
Run nohup python bgservice.py & to get the script to ignore the hangup signal and keep running. Output will be put in nohup.out.
Ideally, you'd run your script with something like supervise so that it can be restarted if (when) it dies.
If you've already started the process, and don't want to kill it and restart under nohup, you can send it to the background, then disown it.
Ctrl+Z (suspend the process)
bg (restart the process in the background
disown %1 (assuming this is job #1, use jobs to determine)
Running a Python Script in the Background
First, you need to add a shebang line in the Python script which looks like the following:
#!/usr/bin/env python3
This path is necessary if you have multiple versions of Python installed and /usr/bin/env will ensure that the first Python interpreter in your $$PATH environment variable is taken. You can also hardcode the path of your Python interpreter (e.g. #!/usr/bin/python3), but this is not flexible and not portable on other machines. Next, you’ll need to set the permissions of the file to allow execution:
chmod +x test.py
Now you can run the script with nohup which ignores the hangup signal. This means that you can close the terminal without stopping the execution. Also, don’t forget to add & so the script runs in the background:
nohup /path/to/test.py &
If you did not add a shebang to the file you can instead run the script with this command:
nohup python /path/to/test.py &
The output will be saved in the nohup.out file, unless you specify the output file like here:
nohup /path/to/test.py > output.log &
nohup python /path/to/test.py > output.log &
If you have redirected the output of the command somewhere else - including /dev/null - that's where it goes instead.
# doesn't create nohup.out
nohup command >/dev/null 2>&1
If you're using nohup, that probably means you want to run the command in the background by putting another & on the end of the whole thing:
# runs in background, still doesn't create nohup.out
nohup command >/dev/null 2>&1 &
You can find the process and its process ID with this command:
ps ax | grep test.py
# or
# list of running processes Python
ps -fA | grep python
ps stands for process status
If you want to stop the execution, you can kill it with the kill command:
kill PID
You could also use GNU screen which just about every Linux/Unix system should have.
If you are on Ubuntu/Debian, its enhanced variant byobu is rather nice too.
You might consider turning your python script into a proper python daemon, as described here.
python-daemon is a good tool that can be used to run python scripts as a background daemon process rather than a forever running script. You will need to modify existing code a bit but its plain and simple.
If you are facing problems with python-daemon, there is another utility supervisor that will do the same for you, but in this case you wont have to write any code (or modify existing) as this is a out of the box solution for daemonizing processes.
Alternate answer: tmux
ssh into the remote machine
type tmux into cmd
start the process you want inside the tmux e.g. python3 main.py
leaving the tmux session by Ctrl+b then d
It is now safe to exit the remote machine. When you come back use tmux attach to re-enter tmux session.
If you want to start multiple sessions, name each session using Ctrl+b then $. then type your session name.
to list all session use tmux list-sessions
to attach a running session use tmux attach-session -t <session-name>.
You can nohup it, but I prefer screen.
Here is a simple solution inside python using a decorator:
import os, time
def daemon(func):
def wrapper(*args, **kwargs):
if os.fork(): return
func(*args, **kwargs)
os._exit(os.EX_OK)
return wrapper
#daemon
def my_func(count=10):
for i in range(0,count):
print('parent pid: %d' % os.getppid())
time.sleep(1)
my_func(count=10)
#still in parent thread
time.sleep(2)
#after 2 seconds the function my_func lives on is own
You can of course replace the content of your bgservice.py file in place of my_func.
Try this:
nohup python -u <your file name>.py >> <your log file>.log &
You can run above command in screen and come out of screen.
Now you can tail logs of your python script by: tail -f <your log file>.log
To kill you script, you can use ps -aux and kill commands.
The zsh shell has an option to make all background processes run with nohup.
In ~/.zshrc add the lines:
setopt nocheckjobs #don't warn about bg processes on exit
setopt nohup #don't kill bg processes on exit
Then you just need to run a process like so: python bgservice.py &, and you no longer need to use the nohup command.
I know not many people use zsh, but it's a really cool shell which I would recommend.
If what you need is that the process should run forever no matter whether you are logged in or not, consider running the process as a daemon.
supervisord is a great out of the box solution that can be used to daemonize any process. It has another controlling utility supervisorctl that can be used to monitor processes that are being run by supervisor.
You don't have to write any extra code or modify existing scripts to make this work. Moreover, verbose documentation makes this process much simpler.
After scratching my head for hours around python-daemon, supervisor is the solution that worked for me in minutes.
Hope this helps someone trying to make python-daemon work
You can also use Yapdi:
Basic usage:
import yapdi
daemon = yapdi.Daemon()
retcode = daemon.daemonize()
# This would run in daemon mode; output is not visible
if retcode == yapdi.OPERATION_SUCCESSFUL:
print('Hello Daemon')
I am trying to use Python to send a SIGINT signal to a process created through subprocess.Popen. I need to run test.bash and have it open in a new window, then wait 4 seconds, send SIGINT (the equivalent of using Ctrl-C) so I am using the following code:
command = "xterm -e bash test.bash"
process = subprocess.Popen(command.split())
time.sleep(4)
os.kill(process.pid, signal.SIGINT)
I have "test.bash" set up to trap the SIGINT signal, and run a cleanup command. This works when I run "test.bash" straight from the command line. But when I send it through xterm -e bash test.bash in Python it instantly terminates the process entirely and does not handle the SIGINT signal at all.
I've tried some things I've read online about killing the process group, but that's not doing anything either. I tried replacing os.kill(process.pid, signal.SIGINT) with process.send_signal(signal.SIGINT) to no avail. Any ideas?
You probably need to add the -hold option to xterm, i.e. xterm -hold -e... to prevent it from closing after the bash script completes since you're using sleep in python rather than the shell.
If you're unable to get the right PID, replacing the second line with this might work:
cmd = "pgrep -f 'xterm -e bash test.bash'"
id = subprocess.check_output(cmd, shell=True)
Then use id instead of process.pid
Running a long and time consuming number crunching process in the shell with a Python script. In the script, to indicate progress, I have inserted occassional print commands like
#!/usr/bin/env python3
#encoding:utf-8
print('Stage 1 completed')
Triggering the script in the shell by
user#hostname:~/WorkingDirectory$chmod 744 myscript.py && nohup ./myscript.py&
It redirects the output to nohup.out, but I cannot see the output until the entire script is done, probably because of stdout buffering. So in this scenario, how do I somehow adjust the buffering parameters to check the progress periodically? Basically, I want zero buffering, so that as soon a print command is issued in the python script, it will appear on nohup.out. Is that possible?
I know it is a rookie question and in addition to the exact solution, any easy to follow reference to the relevant material (which will help me master the buffering aspects of shell without getting into deeper Kernel or hardware level) will be greatly appreciated too.
If it is important, I am using #54~16.04.1-Ubuntu on x86_64
Python is optimised for reading in and printing out lots of data.
So standard input and output of the Python interpreter are buffered by default.
We can override this behavior some ways:
use interpretator python with option -u.
From man python:
-u Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in
binary mode. Note that there is internal buffering in xreadlines(), readlines() and file-object iterators ("for line in
sys.stdin") which is not influenced by this option. To work around this, you will want to use "sys.stdin.readline()" inside a
"while 1:" loop.
Run script in shell:
nohup python -u ./myscript.py&
Or modify shebang line of script to #!/usr/bin/python -u and then run:
nohup ./myscript.py&
use shell command stdbuf for turn off buffering stream
See man stdbuf.
Set unbuffered stream for output:
stdbuf --output=0 nohup ./myscript.py&
Set unbuffered stream for output and errors:
stdbuf -o0 -e0 nohup ./myscript.py&
I have setup a run configuration in Eclipse and need to send SIGINT (Ctrl+C) to the program. There is cleanup code in the program that runs after SIGINT, so pressing Eclipse's "Terminate" buttons won't work (they send SIGKILL I think). Typing CTRL+C into the Console also doesn't work.
How do I send SIGINT to a process running inside an Eclipse Console?
(FWIW I am running a Twisted daemon and need Twisted to shutdown correctly, which only occurs on SIGINT)
If you can determine the process with a utility such as ps, you can use kill to send it a SIGINT. The program will likely be a child process of eclipse.
kill -s INT <pid>
You can send the command via one line:
kill -SIGINT $(ps aux | grep ProgrammName | grep -v grep | awk '{print $2}')
Get the process id and than send the sigint signal
That still seems to be an open issue:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=38016
Just for the sake of completeness: If you came here to find a way to terminate a read line from System.in,
Ctrl + Z worked for me (on Windows).
in some versions, you can do the following.
In the Debug perspective, you can open a view called "Signals"
(Window/Show View/Signals" or Left-Bottom Icon).
You will get a list of all supported signals. Right-click and "Resume
with Signal" will give you the result you need.
I'm making an answer out of a modification of Artur Czajka's comment.
You can use pkill -SIGINT -f ProgramName. Explanation: pkill is similar to killall, -SIGINT states the signal to be used, -f makes it work better in this case (it will look through arguments and stuff instead of just the command name), and ProgramName is the target for pkill.
I have to make graphs from several files with data. I already found a way to run a simple command
xmgrace -batch batch.bfile -nosafe -hardcopy
in which batch.bfile is a text file with grace commands to print the graph I want. I already tried it manually and it works perfectly. To do this with several files I just have to edit one parameter inside batch.bfile and run the same command every time I make a change.
I have already written a python code which edits batch.bfile and goes through all the data files with a for cycle. In each cycle step I want to run the mentioned command directly in the command line.
After searching a bit I found two solutions, one with os.system() and another with subprocess.Popen() and I could only make subprocess.Popen() work without giving any errors by writing:
subprocess.Popen("xmgrace -batch batch.bfile -nosafe -hardcopy", shell=True)
Problem is, this doesn't do anything in practice, i.e., it just isn't the same as running the command directly in the command line. I already tried writing the full directory for the batch.bfile but nothing changed.
I am using Python 2.7 and Mac OS 10.7
Have you checked running xmgrace from the command line using sh? (i.e. invoke /bin/sh, then run xmgrace... which should be the same shell that Popen is using when you set shell=true).
Another solution would be to create a shell script (create a file like myscript.sh, and run chmod +x from the terminal). In the script call xmgrace:
#!/bin/bash
xmgrace -batch batch.bfile -nosafe -hardcopy
You could then test that myscript.sh works, which ought to pick up any environment variables that might be in your profile that might differ from python. If this works, you could call the script from python's subprocess.Popen('myscript.sh'). You can check what the environment variables are set in python for subprocess by running:
import os
os.environ
You may want to check out http://sourceforge.net/projects/graceplot/
When use use Popen, you can capture the application's output to stdout to stderr and print it within your application - this way you can see what is happening:
from subprocess import Popen, PIPE
ps = Popen(reportParameters,bufsize=512, stdout = PIPE, stderr = PIPE)
if ps:
while 1:
stdout = ps.stdout.readline()
stderr = ps.stderr.readline()
exitcode = ps.poll()
if (not stdout and not stderr) and (exitcode is not None):
break
if stdout:
stdout = stdout[:-1]
print stdout
if stderr:
stderr = stderr[:-1]
print stderr