I have a python script in a folder of a remote machine. For executing it, I make a ssh session from my local computer, go to that folder and run.
Commands I use:
ssh remotehost
user#remotehost:~$ cd /my/folder
user#remotehost:~$ python abc.py >> abc.log
Now the problem is ssh session. The script takes lot of time and due to internet issues, ssh session terminates and script doesn't complete. Assume that the remote is always up and running.
Can I run the script without caring of the ssh session termination and do tail -f abc.log with ssh anytime I want ?
You can run the script either in a screen or can run the process in nohup+bg. I always prefer Screen but let me explain both methods.
1. nohup
You can use nohup command to run a process by detaching from terminal like this nohup python /my/folder/abc.py &
This by default creates nohup.out file where all the logs will be stored.
If you want custom file then you can use redirection then it will be nohup python /my/folder/abc.py >> abc.log &
In single command it will be
ssh user#remotehost 'nohup python /my/folder/abc.py >> abc.log &'
nohup wikipedia
2. Screen
From the docs.
Screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells. Each virtual terminal provides the functions of the DEC VT100 terminal and, in addition, several control functions from the ISO 6429 (ECMA 48, ANSI X3.64) and ISO 2022 standards (e.g. insert/delete line and support for multiple character sets). There is a scrollback history buffer for each virtual terminal and a copy-and-paste mechanism that allows the user to move text regions between windows.
When screen is called, it creates a single window with a shell in it (or the specified command) and then gets out of your way so that you can use the program as you normally would. Then, at any time, you can create new (full-screen) windows with other programs in them (including more shells), kill the current window, view a list of the active windows, turn output logging on and off, copy text between windows, view the scrollback history, switch between windows, etc. All windows run their programs completely independent of each other. Programs continue to run when their window is currently not visible and even when the whole screen session is detached from the user's terminal.
Screen Manual
So you can directly run the script in screen using ssh and then you can view the logs whenever needed either by attaching to screen or you can redirect the logs to some file directly or redirect to both file and out put using tee.
Run command in screen and print output in stdout(terminal).
ssh user#remotehost '(screen -dmS ScreenName bash -c "python /my/folder/abc.py; exec bash")'
Run command in screen and redirect output to file.
ssh user#remotehost '(screen -dmS ScreenName bash -c "python /my/folder/abc.py >> abc.log &2>1; exec bash")'
Run command in screen and redirect output to both file as well as to stdout(terminal).
ssh user#remotehost '(screen -dmS ScreenName bash -c "python /my/folder/abc.py &2>1 |tee abc.log; exec bash")'
Note: In all the above commands exec bash is needed else the screen will terminate once the job complets.
Any one of the above commansd should do the job.
In all the above cases you can attach the screen ScreenName using screen -r ScreenName and can see the logs.
I always recommend stderr redirection when redirecting to a file.
some references on using linux screen
10 Screen Command Examples to Manage Linux Terminals
How To Use Linux Screen
This depends what tools are installed on the remote machine you are connecting to. The two major tools I know of are GNU screen and tmux. I use screen so I'll give you some basics of how you would use that one. You can begin normally and enter a screen session before executing your Python script:
ssh remotehost
user#remotehost:~$ cd /my/folder
user#remotehost:~$ screen
user#remotehost:~$ python abc.py >> abc.log
You can then detach from the screen session and the python script will keep running even if your ssh connection is lost.
To detach:
Press Ctrl-A and d
To reattatch:
screen -r
To list screen sessions:
screen -ls
To kill current screen session:
Press Ctrl-d
There is a lot more functionality to screen. This is a good resource for getting started:
https://www.tecmint.com/screen-command-examples-to-manage-linux-terminals/
For more detailed information see the manual:
https://www.gnu.org/software/screen/manual/screen.html
Related
I have Python code that I want to run on a GPU server. Running it is time-consuming and sometimes, I get disconnected from the Internet or server and I have to re-run it. So, I need to let it run and shut down my computer. Is there any way?
You can use screen to maintain a session and achieve the goal that allows you to run the code on the server.
Refer to this: https://www.gnu.org/software/screen/
Some basic commands:
Create session named RunWork
screen -S RunWork
List all sessions:
screen -ls
Open a session
screen -r SessionID
...
If it is a windows server create a bat file.
Run the python script and the shutdown command. You will have to be an admin to shutdown the computer from a script.
bat file
c:\python27\python.exe c:\somescript.py %*
shutdown /s /f /t 0
If you are Linux-based OS then Tmux can help you.
Tmux is a terminal multiplexer. It lets you switch easily between several programs in one terminal, detach them (they keep running in the background) and reattach them to a different terminal.
https://github.com/tmux/tmux/wiki
I've coded a stock trading bot in Python3. I have it hosted on a server (Ubuntu 18.10) that I use iTerm to SSH into. Wondering how to keep the script actively running so that when I exit out of my session it won't kill the active process.
Basically, I want to SSH into my server, start the script then close out and come back into it when the days over to stop the process.
You could use nohup and add & at the end of your command to safely exit you session without killing original process. For example if your script name is script.py:
nohup python3 script.py &
Normally, when running a command using & and exiting the shell afterwards, the shell will terminate the sub-command with the hangup signal (kill -SIGHUP <pid>). This can be prevented using nohup, as it catches the signal and ignores it so that it never reaches the actual application.
You can use screen
sudo apt-get install screen
screen
./run-my-script
Ctrl-A then D to get out of your screen
From there you will be able to close out your ssh terminal. Come back later and run
screen -ls
screen -r $screen_running
The screen running is usually the first 5 digits you see after you've listed all the screens. You can see if you're script is still running or if you've added logging you can see where in the process you are.
Using tmux is a good option. Alternatively you could run the command with an & at the end which lets it run in the background.
https://tmuxcheatsheet.com/
I came here for finding nohup python3 script.py &
Right solution for this thread is screen OR tmux. :)
I'm looking to create a script in python that initiates an SSH session with a server. I know it has to be a simple process i'm just not sure where to start. My ultimate plan is to automate this script to run on startup. i am not even sure python is the best way to go i just know it comes preloaded on raspbain for the pi.
A simple bash script would be better suited to the task. It is possible with python, but there's no obvious reason to make it harder than necessary.
From
write a shell script to ssh to a remote machine and execute commands:
#!/bin/bash
USERNAME=someUser
HOSTS="host1 host2 host3"
SCRIPT="pwd; ls"
for HOSTNAME in ${HOSTS} ; do
ssh -l ${USERNAME} ${HOSTNAME} "${SCRIPT}"
done
From how do i run a script at start up (askubuntu):
You will need root privileges for any the following. To get root, open
a terminal and run the command
sudo su
and the command prompt will change to '#' indicating that the terminal
session has root privileges.
Alternative #1. Add an initscript.
Create a new script in /etc/init.d/myscript.
vi /etc/init.d/myscript
(Obviously it doesn't have to be called "myscript".) In this script,
do whatever you want to do. Perhaps just run the script you mentioned.
#!/bin/sh
/path/to/my/script.sh
Make it executable.
chmod ugo+x /etc/init.d/myscript
Configure the init system to run this script at startup.
update-rc.d myscript defaults
Alternative #2. Add commands to /etc/rc.local
vi /etc/rc.local
with content like the following.
# This script is executed at the end of each multiuser runlevel
/path/to/my/script.sh || exit 1 # Added by me
exit 0
Alternative #3. Add an Upstart job.
Create /etc/init/myjob.conf
vi /etc/init/myjob.conf
with content like the following
description "my job"
start on startup
task
exec /path/to/my/script.sh
Depending on what you do with the ssh connection, if it needs to stay open over the entire up time of the device, you will need to use some more trickery however (ssh connections are autoclosed after a period of inactivity).
I need to execute python script on remote server (access through puTTY), but I don't have a stable Internet connection, and every time I execute the script I get problems after several minutes due to my Internet getting disconnected.
How do I remotely execute the script without being connected to server?
(e.g. I connect to server, run script, and can logout while executing)
You can use a Linux Screen, it opens a background terminal and keeps a shell active even through network disruptions.
Open the screen typing in your terminal $ screen and execute there your script, even if you lose connection it won't kill the process.
Here you will find a well explained How to for this program. I use it for my regular day working on remote.
try this
nohup your_script >/dev/null 2>&1 &
program will be running in background
I've setup an Amazon EC2 server. I have a Python script that is supposed to download large amounts of data from the web onto the server. I can run the script from the terminal through ssh, however very often I loose the ssh connection. When I loose the connection, the script stops.
Is there a method where I tell the script to run from terminal and when I disconnect, the script is still running on the server?
You have a few options.
You can add your script to cron to be run regularly.
You can run your script manually, and detach+background it using nohup.
You can run a tool such as GNU Screen, and detach your terminal and log out, only to continue where you left off later. I use this a lot.
For example:
Log in to your machine, run: screen.
Start your script and either just close your terminal or properly detach your session with: Ctrl+A, D, D.
Disconnect from your terminal.
Reconnect at some later time, and run screen -rD. You should see your stuff just as you left it.
You can also add your script to /etc/rc.d/ to be invoked on book and always be running.
You can also use nohup to make your script run in the background or when you have disconnected from your session:
nohup script.py &
The & at the end of the command explicitly tells nohup to run your script in the background.
If it just a utility you run ad-hoc, not a service daemon of some kind, i would just run it in screen. Than you can disconnect if you want and open the terminal back up later... Or reconnect the terminal if you get disconnected. It should be in your linux distros package manager. Just search for screen
http://www.gnu.org/software/screen/
nohup runs the given command with hangup signals ignored, so that the command can continue running in the background after you log out.
Syntax:
nohup Command [Arg]...
Example:
nohup example.py
nohup rasa run
Also, you can run scripts continuously using the cron command.
For more:
https://ss64.com/bash/nohup.html
https://opensource.com/article/17/11/how-use-cron-linux