I am passing data from hcitool and hcidump, via sed and awk then piped into python, which then reads from stdin. This works well when run from the command line. It works equally well when put into a shell program and called from the command line.
However, when I call that shell program via cron on startup and run headless, the python program executes, but nothing ever flows from sed | awk into python.
I have read thin wisps of information that stdin may not flow thru headless, but haven't found anything concrete.
What am I missing?
Related
Edit:
The original intent of this question was to find a way to launch an interactive ssh session via a Python script. I'd tried subprocess.call() before and had gotten a Killed response before anything was output onto the terminal. I just assumed this was an issue/limitation with the subprocess module instead of an issue somewhere else.This was found not to be the case when I ran the script on a non-resource limited machine and it worked fine.
This then turned the question into: How can I run an interactive ssh session with whatever resource limitations were preventing it from running?
Shoutout to Charles Duffy who was a huge help in trying to diagnose all of this .
Below is the original question:
Background:
So I have a script that is currently written in bash. It parses the output of a few console functions and then opens up an ssh session based on those parsed outputs.
It currently works fine, but I'd like to expand it's capabilities a bit by adding some flag arguments to it. I've worked with argparse before and thoroughly enjoyed it. I tried to do some flag work in bash, and let's just say it leaves much to be desired.
The Actual Question:
Is it possible to have python to do stuff in a console and then put the user in that console?
Something like using subprocess to run a series of commands onto the currently viewed console? This in contrast to how subprocess normally runs, where it runs commands and then shuts the intermediate console down
Specific Example because I'm not sure if what I'm describing makes sense:
So here's a basic run down of the functionality I was wanting:
Run a python script
Have that script run some console command and parse the output
Run the following command:
ssh -t $correctnode "cd /local_scratch/pbs.$jobid; bash -l"
This command will ssh to the $correctnode, change directory, and then leave a bash window in that node open.
I already know how to do parts 1 and 2. It's part three that I can't figure out. Any help would be appreciated.
Edit: Unlike this question, I am not simply trying to run a command. I'm trying to display a shell that is created by a command. Specifically, I want to display a bash shell created through an ssh command.
Context For Readers
The OP is operating on a very resource-constrained (particularly, it appears, process-constrained) jumphost box, where starting an ssh process as a subprocess of python goes over a relevant limit (on number of processes, perhaps?)
Approach A: Replacing The Python Interpreter With Your Interactive Process
Using the exec*() family of system calls causes your original process to no longer be in memory (unlike the fork()+exec*() combination used to start a subprocess while leaving the parent process running), so it doesn't count against the account's limits.
import argparse
import os
try:
from shlex import quote
except ImportError:
from pipes import quote
parser = argparse.ArgumentParser()
parser.add_argument('node')
parser.add_argument('jobid')
args = parser.parse_args()
remote_cmd_str = 'cd /local_scratch/pbs.%s && exec bash -i' % (quote(args.jobid))
local_cmd = [
'/usr/bin/env', 'ssh', '-tt', node, remote_cmd_str
]
os.execv("/usr/bin/env", local_cmd)
Approach B: Generating Shell Commands From Python
If we use Python to generate a shell command, the shell can invoke that command only after the Python process exited, such that we stay under our externally-enforced process limit.
First, a slightly more robust approach at generating eval-able output:
import argparse
try:
from shlex import quote
except ImportError:
from pipes import quote
parser = argparse.ArgumentParser()
parser.add_argument('node')
parser.add_argument('jobid')
args = parser.parse_args()
remoteCmd = ['cd', '/local_scratch/pbs.%s' % (args.jobid)]
remoteCmdStr = ' '.join(quote(x) for x in remoteCmd) + ' && bash -l'
cmd = ['ssh', '-t', args.correctnode, remoteCmdStr]
print(' '.join(pipes.quote(x) for x in cmd)
To run this from a shell, if the above is named as genSshCmd:
#!/bin/sh
eval "$(genSshCmd "$#")"
Note that there are two separate layers of quoting here: One for the local shell running eval, and the second for the remote shell started by SSH. This is critical -- you don't want a jobid of $(rm -rf ~) to actually invoke rm.
This is in no way a real answer, just an illustration to my comment.
Let's say you have a Python script, test.py:
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('myarg', nargs="*")
args = parser.parse_args()
print("echo Hello world! My arguments are: " + " ".join(args.myarg))
So, you create a bash wrapper around it, test.sh
set -e
$(python test.py $*)
and this is what you get:
$ bash test.sh
Hello world! My arguments are:
$ bash test.sh one two
Hello world! My arguments are: one two
What is going on here:
python script does not execute commands. Instead, it outputs the commands bash script will run (echo in this example). In your case, the last command will be ssh blabla
bash executes the output of the python script (the $(...) part), passing on all its arguments (the $* part)
you can use argparse inside the python script; if anything is wrong with the arguments, the message will be put to stderr and will not be executed by bash; bash script will stop because of set -e flag
I am doing something need to use curl xxx | bash to run something.
I created a python script with sys.stdin.readline() to test like this:
[python3] test.py
import sys
def read_input():
input = sys.stdin.readline().rstrip()
print(input)
read_input()
It works directly run by python3 test.py
test
test
But if I use echo 'python3 test.py' | bash, it will not stop to let me input something.
Any tips?
When you pipe with |, you are redirecting the output from the first command into the input of the second. This means standard input of the second command doesn't connect to the terminal, and therefore cannot read keyboard input. The form curl xxx | bash therefore only functions for non-interactive scripts. This is in no way specific to Python.
You could in principle work around this by saving the input descriptor under another number, but it does get quite complex:
$ ( echo 'exec <&3 3<&- ; echo script starts ; read hello ; echo you entered $hello ; exit' | bash ) 3<&0
script starts
something
you entered something
Here I used () to create a subshell, in which stdin is duplicated on file descriptor 3 using 3<&0, and the script generated in the pipeline both renames that back as stdin with exec <&3 3<&- and exits to prevent further commands from being read from the restored stdin. This has side effects, such as descriptor 3 being open for the echo command.
Since the main reason to use curl address | bash in the first place is to keep the command simple, this is not what you're after. Besides, the pipe prevents you from handling if anything goes wrong during your download; your script could be interrupted anywhere. A traditional download then run isn't that much worse:
curl -O http://somewhere/somefile.py && python somefile.py
In comparison, this saves somefile.py to your filesystem. There are downsides to that, like requiring a writable filesystem and replacing that particular filename. On the upside, if anything goes wrong it stops there and doesn't run a corrupted script, due to &&.
One final possibility if the script you're downloading fits within the command line might be to put it there rather than in a pipe:
python -c "$(curl $url)"
This carries the same weaknesses to interrupted downloads, and additionally places the script contents in a command line which is generally public information (consider ps ax output). But if you're just downloading the script with curl, the information on how to get it likely was too. As this doesn't redirect stdin, it might be the answer to your immediate question.
In general, I recommend not to run any scripts straight off the internet without verification as this curl something | bash command line does. It's way too vulnerable to hijacking, as there's no verification involved for any step. It's better to use a package repository which checks signatures, such as apt.
Another method to get access to a terminal on Linux is via the device /dev/tty. This method is used for instance when ssh asks for a password. It might also be possible to reopen stdout or stderr for input, as in ( exec < /dev/null ; read foo <&2 ; echo $foo ).
I have ten python scripts in the same directory. How to run all of these from command line, that it will work in background?
I use SSH terminal to connect to server CentOS and run Python script as:
python index.py
But when I close client terminal SSH, proccess is died
You can use the & command to make things run in the background, and nohup so it continues on logout, such as
nohup python index.py &
If you want to run multiple things this way, it's probably easiest to just make a script to start them all (with a shell of your choice):
#!/bin/bash
nohup python index1.py &
nohup python index2.py &
...
As long as you don't need to interact with the scripts once they are started (and don't need any stdout printing) this could be pretty easily automated with another python script using the subprocess module:
for script in listofscripts:
#use subprocess.run() for python 3.x (this blocks until each script terminates)
subprocess.call(["python", script], *args) #use popen if you want non - blocking
*args is a link (it's coloring got overwritten by code highliting
also of note: stdout / stderr printing is possible, just more work..
I am running a python script sample_program.py on python via ssh. I log into the machine, and run
python3 sample_program.py &
and log off with the command 'exit'. Unfortunately, the script stops running after a few minutes.
What else could I use to run python scripts remotely and not keep the Terminal open?
nohup
nohup python3 sample_program.py &
is the simplest way (man nohup):
nohup - run a command immune to hangups, with output to a non-tty
and IMHO it is installed everywhere.
at
You can use the at command. The at execute commands at a later time. The at utility shall read commands from standard input and group them together as an at-job, to be executed at a later time.
For more information, options, examples, and others see the [Ubuntu Manpage Repository][1]
Example:
at now +8 hours -f python3 sample_program.py
You can also use convenient shorthands, like tomorrow or noon, as in
echo "tweet fore" | at teatime
Independently of any terminal
ssh root#remoteserver '/root/backup.sh </dev/null >/var/log/root-backup.log 2>&1 &'
You need to close all file descriptors that are connected to the ssh socket, because the ssh session won't close as long as some remote process has the socket open. If you aren't interested in the script's output (presumably because the script itself takes care of writing to a log file), redirect it to /dev/null (but note that this will hide errors such as not being able to start the script).
Using nohup has no useful effect here. nohup arranges for the program it runs not to receive a HUP signal if the program's controlling terminal disappears, but here there is no terminal in the first place, so nothing is going to send a SIGHUP to the process out of the blue. Also, nohup redirects standard output and standard error (but not standard input) to a file, but only if they're connected to a terminal, which, again, they aren't.
You can set a cron job.
For example if now the time is 14:39:00 and today is friday, 30 august, you can add the following cron job (to be executed after 8 hours) in your crontab file using crontab -e command:
39 22 30 8 5 /path/to/python3 /path/to/sample_program.py
Add the shebang to the start of your scripts!
#!/usr/bin/python3
Give it permissions to execute.
chmod +x python3
Execute remotely!
sudo nohup ./python3 >/dev/null 2>&1 &
This way it will run as a background process and detach from the terminal, and you will not be writing an unnecessary nohup.out file.
You DO NOT even need the .py file extension in Linux, nor do you need to use more characters than needed:
{ python3 python3.py }
is just the same with
{ ./python3 }
It just needs the shebang and to be executable.
When I run my python script in the shell terminal, it works
sudo myscript.py --version=22 --base=252 --hosts="{'hostA':[1],'hostB':[22]}"
But when I run in Hudson and Jenkins, using Execute Shell step, somehow, the string --hosts="{'hostA':[1],'hostB':[22]}" is interpreted as
sudo myscript.py --version=22 --base=252 '--hosts="{'hostA':[1],'hostB':[22]}"'
How do we overcome this so that our script would run in Jenkins and Hudson ?
Thank you.
Sincerely
It looks like you're encountering a battle-of-the-quoted-strings type situation due to your use of quotes directly and the fact that Jenkins is shelling out from a generated temp shell script.
I find the best thing to do with Jenkins is to create a bash script that wraps the commands you want to run (and you can also have it do any other environment-related setup you may want to have it do, such as source a config bash script that sets up other env vars).
You can have it accept arguments that may vary from the command line, which can be passed to it from the Jenkins config. So any of the interpolation then happens within the script -- you're just passing strings. (In particular, in this case, you'll have the hosts arg be "{'hostA':[1],'hostB':[22]}", which will be passed to the shell script, and then interpolated, with the double quotes re-included.
So, to that end, say you have a jenkins_run.sh script that runs a command like this:
myscript.py --version=$VERSION --base=$BASE --hosts="$HOSTS"
Where the variables are passed in as arguments and assigned prior to that (you could directly use $0, $1 et al if you want.
I would also be cautious using sudo in conjunction with a Jenkins run, since that could end up prompting for I/O. I would instead recommend setting the permissions on the script such that the using under which Jenkins is running can simply execute the script.