I have a very strange issue that I can't seem to figure out.
When I execute a python script containing the following lines while inside a SSH terminal (putty), it works fine. But the moment I run the script via crontab or even nohup python myscript >/dev/null 2>&1& it doesn't seem to execute these commands.
subprocess.call('rsync -avr /path/to/folder/. --include "delta.*" --exclude "*" -e "ssh -o StrictHostKeyChecking=no -i /path/to/key.pem" ec2-user#'+server+':/path/to/folder/', shell=True)
local('ssh -t -o StrictHostKeyChecking=no -i /path/to/key.pem ec2-user#'+server+' "sudo /usr/bin/indexer -c /path/to/sphinx.conf --merge main delta --rotate"')
Basically all the above is doing is syncing a folder with new sphinx search engine updates to a remote server, then the second line runs a remote ssh command to force the search engine to rotate updates into production.
I do have fabric installed (hence the local command) but to avoid having to fab a second file I was hoping a single line of code could allow me to execute sudo commands on a remote server.
Can someone help me out?
I found the answer, for ssh commands in a script run in the background, you need to to have -t -t to force a pseudo terminal.
Reference:
Pseudo-terminal will not be allocated because stdin is not a terminal
Related
I have a modified AWS basicPubSub function to transfer data to the AWS IoT core, I want the script to run at start-up.
I have added this script into, make it executable and updated the init.d
/etc/init.d
chmod 755 LOMAWS.sh
sudo update-rc.d LOMAWS.sh defaults
But the script does not start, how can I make it run from start up?
clear
echo "LOM AWS Script starting"
cd /home/pi/Documents/awsiot/aws-iot-device-sdk-python/samples/basicPubSub
sudo python basicPubSub.py -e "XXXXXXXX-ats.iot.us-east-2.amazonaws.com" -r root_CA.crt -c XXXXXXXX-certificate.pem.crt -k XXXXXXX-private.pem.key
Have you tried UserData?
By default, user data scripts and cloud-init directives run only during the first boot cycle when an instance is launched
I have to run python code on remote python process.
Normally, what I would do is:
ssh admin#localhost -p 61234
which opens a interactive python console and I can execute python code directly.
>>> print('3')
3
>>>
But I want to automate this and pass python code as parameter to ssh.
I tried following options:
ssh admin#localhost -v -p 61234 python logs.py
ssh admin#localhost -v -p 61234 nohup python logs.py
ssh admin#localhost -p 61234 < logs.py
cat logs.py | ssh admin#localhost -p 61234 python -
But all options give following error:
shell request failed on channel 0
logs.py:
#!/usr/bin/env python
# tried with and without first line.
print('3')
netstat -anp | grep 61234
tcp 0 0 127.0.0.1:61234 0.0.0.0:* LISTEN 6/python2.7
Is there a way to do this?
Pretty sure this is overkill, but who knows, maybe you need more than just a simple command in the future.
Paramiko package is what you're looking for. The project is full of demos which demonstrate how and in many different ways.
The function you'll find the most useful is paramiko.client.SSHClient.exec_command
Execute a command on the SSH server. A new Channel is opened and the
requested command is executed. The command’s input and output streams
are returned as Python file-like objects representing stdin, stdout,
and stderr.
Demos Folder
In-depth Testing
Interactive.py is a fully interactive TTY (remote terminal control functions).
PyCharm Professional Edition (Python IDE from JetBrains) has tools for remote development, including SSH remoting, and the ability to run a remote interpreter:
Using ubuntu's 16.04 crontab and #reboot to run python3 script. The script runs properly on reboot as I see the logged output. However, my script's os.system command is not running. It runs fine if ran outside of crontab. My scripts are all executable.
crontab -l output:
SHELL=/bin/bash
#reboot nohup /usr/bin/python3 -u /home/path/scheduler.py >> /path/log.out &
scheduler.py code:
#...(check if web server is running...if not restart)
os.system('nohup /usr/bin/python3 -u /path/webserver/main.py &')
print('this function ran')
When I logged the output of the os.system command , there was no output.
As a side note, I am running python schedule commands to check the general health of a webserver. crontab doesn't seem to be the right tool for this so I just use crontab to start my python scheduler on reboot.
I am using flask as the webserver, and would use gunicorn and systemctrl if I could get it to work... but it didn't so this is my workaround.
The point is that, the command called by os.system is not in default path.
For example, tcpdump is not in /usr/bin/.
So, you can solve the problem by adding the full path of the command.
I was facing the same issue when we try to run python script directly in crontab it just by passes the os.system() commands.
Make launcher.sh:
#!bin/bash
cd /home/pi/
sudo python example.py
Then, make your script executable:
chmod 755 launcher.sh
And at last, add your script to crontab:
crontab -e
and add this line at the end:
#reboot sh /home/pi/launcher.sh
(I set the program to run at each reboot)
I am trying to run a python script on a remote computer via psexec. I am able to connect and run python.exe with the following:
C:\test>psexec \\192.168.X.X -u domain\administrator -p password -i C:\Anaconda\python.exe
The path to python.exe is the path on the remote machine. This opens a python window on the remote machine - all good.
I want to now pass a python script from the host machine to run on the remote. This script is on the host machine in C:\test\test.py. I tried
psexec \\192.168.X.X -u domain\administrator -p password -i "C:\Anaconda\python.exe" -c C:\test\test.py
and get:
C:\Anaconda\python.exe exited on 192.168.X.X with error code 1.
I also tried-c test.py without the full path, and got a similar error. My thought is the remote application cannot find C:\test\test.py. I want to be able to pass the script from the host machine.
Any help is much appreciated. Thanks.
If the .py extension has been associated with the Python installation on the remote machine, you may be able to run your Python script by simply removing the Python executable from the command line:
psexec \\192.168.X.X -u domain\administrator -p password -i -c C:\test\test.py
Please note that I have not tried this as I don't presently have access to a remote machine, so I can't guarantee that it will work.
The line
psexec \\192.168.X.X -u domain\administrator -p password -i "C:\Anaconda\python.exe" -c C:\test\test.py
may be trying to run the command "C:\Anaconda\python.exe" -c C:\test\test.py on the remote machine. In other words, Python may be interpreting the -c switch, rather than PsExec. The Python switch -c specifies some Python code to run, and of course a filename is not valid Python code:
C:\Users\Luke>python -c "print 2 + 2"
4
C:\Users\Luke>python -c C:\test\test.py
File "<string>", line 1
C:\test\test.py
^
SyntaxError: invalid syntax
C:\Users\Luke>echo %ERRORLEVEL%
1
Was able to access a python script on a shared drive from the remote computer and host, and so by copying to the share from the host and reading from the share on the remote machine i had a suitable workaround (the -i switch is not required).
psexec \\remote_machine_name -u domain\user -p pswrd -i C:/Anaconda/python.exe \\server\share\test\test.py
Related: if you are running on windows and writing to a UNC path from a python script i.e test.py above, helpful path formatting help:
python copy files to a network location on Windows without mapping a drive
Through Fabric, I am trying to start a celerycam process using the below nohup command. Unfortunately, nothing is happening. Manually using the same command, I could start the process but not through Fabric. Any advice on how can I solve this?
def start_celerycam():
'''Start celerycam daemon'''
with cd(env.project_dir):
virtualenv('nohup bash -c "python manage.py celerycam --logfile=%scelerycam.log --pidfile=%scelerycam.pid &> %scelerycam.nohup &> %scelerycam.err" &' % (env.celery_log_dir,env.celery_log_dir,env.celery_log_dir,env.celery_log_dir))
I'm using Erich Heine's suggestion to use 'dtach' and it's working pretty well for me:
def runbg(cmd, sockname="dtach"):
return run('dtach -n `mktemp -u /tmp/%s.XXXX` %s' % (sockname, cmd))
This was found here.
As I have experimented, the solution is a combination of two factors:
run process as a daemon: nohup ./command &> /dev/null &
use pty=False for fabric run
So, your function should look like this:
def background_run(command):
command = 'nohup %s &> /dev/null &' % command
run(command, pty=False)
And you can launch it with:
execute(background_run, your_command)
This is an instance of this issue. Background processes will be killed when the command ends. Unfortunately on CentOS 6 doesn't support pty-less sudo commands.
The final entry in the issue mentions using sudo('set -m; service servicename start'). This turns on Job Control and therefore background processes are put in their own process group. As a result they are not terminated when the command ends.
For even more information see this link.
you just need to run
run("(nohup yourcommand >& /dev/null < /dev/null &) && sleep 1")
DTACH is the way to go. It's a software you need to install like a lite version of screen.
This is a better version of the "dtach"-method found above, it will install dtach if necessary. It's to be found here where you can also learn how to get the output of the process which is running in the background:
from fabric.api import run
from fabric.api import sudo
from fabric.contrib.files import exists
def run_bg(cmd, before=None, sockname="dtach", use_sudo=False):
"""Run a command in the background using dtach
:param cmd: The command to run
:param output_file: The file to send all of the output to.
:param before: The command to run before the dtach. E.g. exporting
environment variable
:param sockname: The socket name to use for the temp file
:param use_sudo: Whether or not to use sudo
"""
if not exists("/usr/bin/dtach"):
sudo("apt-get install dtach")
if before:
cmd = "{}; dtach -n `mktemp -u /tmp/{}.XXXX` {}".format(
before, sockname, cmd)
else:
cmd = "dtach -n `mktemp -u /tmp/{}.XXXX` {}".format(sockname, cmd)
if use_sudo:
return sudo(cmd)
else:
return run(cmd)
May this help you, like it helped me to run omxplayer via fabric on a remote rasberry pi!
You can use :
run('nohup /home/ubuntu/spider/bin/python3 /home/ubuntu/spider/Desktop/baidu_index/baidu_index.py > /home/ubuntu/spider/Desktop/baidu_index/baidu_index.py.log 2>&1 &', pty=False)
nohup did not work for me and I did not have tmux or dtach installed on all the boxes I wanted to use this on so I ended up using screen like so:
run("screen -d -m bash -c '{}'".format(command), pty=False)
This tells screen to start a bash shell in a detached terminal that runs your command
You could be running into this issue
Try adding 'pty=False' to the sudo command (I assume virtualenv is calling sudo or run somewhere?)
This worked for me:
sudo('python %s/manage.py celerycam --detach --pidfile=celerycam.pid' % siteDir)
Edit: I had to make sure the pid file was removed first so this was the full code:
# Create new celerycam
sudo('rm celerycam.pid', warn_only=True)
sudo('python %s/manage.py celerycam --detach --pidfile=celerycam.pid' % siteDir)
I was able to circumvent this issue by running nohup ... & over ssh in a separate local shell script. In fabfile.py:
#task
def startup():
local('./do-stuff-in-background.sh {0}'.format(env.host))
and in do-stuff-in-background.sh:
#!/bin/sh
set -e
set -o nounset
HOST=$1
ssh $HOST -T << HERE
nohup df -h 1>>~/df.log 2>>~/df.err &
HERE
Of course, you could also pass in the command and standard output / error log files as arguments to make this script more generally useful.
(In my case, I didn't have admin rights to install dtach, and neither screen -d -m nor pty=False / sleep 1 worked properly for me. YMMV, especially as I have no idea why this works...)