Develop Reference

Output garbled when launching multiple ssh-sessions with pseudo-tty (need remote process to exit when ssh disconnects/is killed)

I have a python script that opens multiple concurrent pseudo-tty ssh sessions to a server. My problem is that the output is garbled:
for i in range(0, 3):
subprocess.Popen(
"ssh -tt -q myserver 'echo 11; echo 22; echo 33; echo 44;'",
shell=True
)
Output:
11
22
33
44
11
22
33
44
11
22
33
44
The output varies. Sometimes it works, but most of the time I get those weird indentations. In reality I want to launch remote python processes (a locust load gen slave), but I've simplified it to just use echo.
Things I've tried:
universal_newlines=True, bufsize=1 (doesnt help)
remove -tt (fixes the output but has the undesired side effect of remote processes not dying right away if python/ssh is terminated)
piping to cat -e to get hidden characters (for debugging):
11^M$
22^M$
33^M$
44^M$
11$
22$
33$
44$
11$
22$
33$
44$
I'm not sure if is even a python issue or just an SSH issue. My guess is that I need to use some sort of line buffering, but I dont know how :-/
I'm on MacOS Mojave, and I've tried both in iTerm2 and Term if that matters.
Edit: I'm not sure it is related, but the problem appears to occur more frequently if I ensure python keeps running until the ssh session has terminated (by adding time.sleep(10) at the end of the script)
edit 2: I tried #FLemaitre 's solution (not using -tt and killing explicitly), and it works in the simple case, but not when spawning locust:
proc = subprocess.Popen(
"ssh servername 'locust --slave --master-port 7777 --no-web -f locustfile.py & read; kill $!'",
shell=True,
stdin=subprocess.PIPE,
)
time.sleep(10)
proc.kill()
proc.wait()
On the remote a bash -c locust --slave ... process is started. It dies when ssh is killed, but locust itself (a child of the above process) does not :-/

I reproduce systematically the issue with the following script:
import subprocess
import time
if __name__ == "__main__":
for i in range(0, 10):
proc = subprocess.Popen(
"ssh -tt -q localhost 'echo 11; echo 22; echo 33; '",
shell=True
)
time.sleep(4)
And I think the issue is not related to Python. These multiple ssh with pseudo-TTY seem to conflict with each other's. Eventually, the terminal used to run this script ends up broken as well (whereas it wasn't sourced):
>cat test2.py
import subprocess
import time
import atexit
... etc ...
I checked the documentation and this -t option seems to do much more than what you are actually trying to achieve. When I remove the second t and the -q options, I sometimes (not often), get a cryptic error message stating that something went wrong (but I no longer manage to reproduce it). I checked with google but without much success. Still, I'm convinced that this option is overkill and I would rather focus on the undying processes. This one issue is well known:
Starting a process over ssh using bash and then killing it on sigint
The second answer is your -tt option, but the best answer suits your example very well and is superior (with -tt you solve the ssh propagation of the termination but do not tackle the same issue between Python and its subprocess). For example:
import subprocess
import time
if __name__ == "__main__":
for i in range(0, 10):
proc = subprocess.Popen(
"ssh localhost 'sleep 90 & read ; kill $!'",
shell=True,
stdin=subprocess.PIPE
)
time.sleep(40)
With this solution, stdin is shared by all actors (python, the python subprocess, the ssh process, the sleep process), and its closure at any point in the chain is detected by the final business process, trigering a graceful shutdown.
Edit with locust:
I gave it a quick try and the issue was that a simple 'kill' is ignored by the slave (looks like an issue on lucust side). It seems to work with a 'kill -9':
import subprocess
import time
if __name__ == "__main__":
for i in range(0, 2):
proc = subprocess.Popen(
"ssh localhost 'python -m locust --slave --no-web -f ~devsup/users/flemaitre/tmp/locust_config.py & read ; kill -9 $!'",
shell=True,
stdin=subprocess.PIPE
)
time.sleep(40)

How to run a background process and do *not* wait?

My goal is simple: kick off rsync and DO NOT WAIT.
Python 2.7.9 on Debian
Sample code:
rsync_cmd = "/usr/bin/rsync -a -e 'ssh -i /home/myuser/.ssh/id_rsa' {0}#{1}:'{2}' {3}".format(remote_user, remote_server, file1, file1)
rsync_cmd2 = "/usr/bin/rsync -a -e 'ssh -i /home/myuser/.ssh/id_rsa' {0}#{1}:'{2}' {3} &".format(remote_user, remote_server, file1, file1)
rsync_path = "/usr/bin/rsync"
rsync_args = shlex.split("-a -e 'ssh -i /home/mysuser/.ssh/id_rsa' {0}#{1}:'{2}' {3}".format(remote_user, remote_server, file1, file1))
#subprocess.call(rsync_cmd, shell=True) # This isn't supposed to work but I tried it
#subprocess.Popen(rsync_cmd, shell=True) # This is supposed to be the solution but not for me
#subprocess.Popen(rsync_cmd2, shell=True) # Adding my own shell "&" to background it, still fails
#subprocess.Popen(rsync_cmd, shell=True, stdin=None, stdout=None, stderr=None, close_fds=True) # This doesn't work
#subprocess.Popen(shlex.split(rsync_cmd)) # This doesn't work
#os.execv(rsync_path, rsync_args) # This doesn't work
#os.spawnv(os.P_NOWAIT, rsync_path, rsync_args) # This doesn't work
#os.system(rsync_cmd2) # This doesn't work
print "DONE"
(I've commented out the execution commands only because I'm actually keeping all of my trials in my code so that I know what I've done and what I haven't done. Obviously, I would run the script with the right line uncommented.)
What happens is this...I can watch the transfer on the server and when it's finished, then I get a "DONE" printed to the screen.
What I'd like to have happen is a "DONE" printed immediately after issuing the rsync command and for the transfer to start.
Seems very straight-forward. I've followed details outlined in other posts, like this one and this one, but something is preventing it from working for me.
Thanks ahead of time.
(I have tried everything I can find in StackExchange and don't feel like this is a duplicate because I still can't get it to work. Something isn't right in my setup and need help.)

Here is verified example for Python REPL:
>>> import subprocess
>>> import sys
>>> p = subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(100)'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT); print('finished')
finished
How to verify that via another terminal window:
$ ps aux | grep python
Output:
user 32820 0.0 0.0 2447684 3972 s003 S+ 10:11PM 0:00.01 /Users/user/venv/bin/python -c import time; time.sleep(100)

Popen() starts a child process—it does not wait for it to exit. You have to call .wait() method explicitly if you want to wait for the child process. In that sense, all subprocesses are background processes.
On the other hand, the child process may inherit various properties/resources from the parent such as open file descriptors, the process group, its control terminal, some signal configuration, etc—it may lead to preventing ancestors processes to exit e.g., Python subprocess .check_call vs .check_output or the child may die prematurely on Ctrl-C (SIGINT signal is sent to the foreground process group) or if the terminal session is closed (SIGHUP).
To disassociate the child process completely, you should make it a daemon. Sometimes something in between could be enough e.g., it is enough to redirect the inherited stdout in a grandchild so that .communicate() in the parent would return when its immediate child exits.

I encountered a similar issue while working with qnx devices and wanted a sub-process that runs independently of the main process and even runs after the main process terminates.
Here's the solution I found that actually works 'creationflags=subprocess.DETACHED_PROCESS':
import subprocess
import time
pid = subprocess.Popen(["python", "path_to_script\turn_ecu_on.py"], creationflags=subprocess.DETACHED_PROCESS)
time.sleep(15)
print("Done")
Link to the doc: https://docs.python.org/3/library/subprocess.html#subprocess.Popen

In Ubuntu the following commands keep working even if python app exits.
url = "https://www.youtube.com/watch?v=t3kcqTE6x4A"
cmd = f"mpv '{url}' && zenity --info --text 'you have watched {url}' &"
os.system(cmd)

automation of processes by python

I'm trying to write a python script that start a process and do some operations atferward.
The commands that I want to automate by script are circled as red in the picture.
The problem is that after performing first command, qemu environment will be run and the other commands should be executed on the qemu environment. So I want to know how can I do these commands by an script in python? Because as I know I can do the first command but I do not know how can I do those commands when I am going to qemu environment.
Could you help me how can I do this process?

First thing that came to mind was pexpect, a quick search on google turned up this blog post automatically-testing-vms-using-pexpect-and-qemu which seems to be pretty much along the lines of what you are doing:
import pexpect
image = "fedora-20.img"
user = "root"
password = "changeme"
# Define the qemu cmd to run
# The important bit is to redirect the serial to stdio
cmd = "qemu-kvm"
cmd += " -m 1024 -serial stdio -net user -net nic"
cmd += " -snapshot -hda %s" % image
cmd += " -watchdog-action poweroff"
# Spawn the qemu process and log to stdout
child = pexpect.spawn(cmd)
child.logfile = sys.stdout
# Now wait for the login
child.expect('(?i)login:')
# And login with the credentials from above
child.sendline(user)
child.expect('(?i)password:')
child.sendline(password)
child.expect('# ')
# Now shutdown the machine and end the process
if child.isalive():
child.sendline('init 0')
child.close()
if child.isalive():
print('Child did not exit gracefully.')
else:
print('Child exited gracefully.')
You could do it with subprocess.Popen also, checking the stdout for the (qemu) lines and writing to stdin. Something roughly like this:
from subprocess import Popen,PIPE
# pass initial command as list of individual args
p = Popen(["./tracecap/temu","-monitor",.....],stdout=PIPE, stdin=PIPE)
# store all the next arguments to pass
args = iter([arg1,arg2,arg3])
# iterate over stdout so we can check where we are
for line in iter(p.stdout.readline,""):
# if (qemu) is at the prompt, enter a command
if line.startswith("(qemu)"):
arg = next(args,"")
# if we have used all args break
if not arg:
break
# else we write the arg with a newline
p.stdin.write(arg+"\n")
print(line)# just use to see the output
Where args contains all the next commands.

Don't forget that Python has batteries included. Take a look of the Suprocess module in the standard lib. There a lot of pitfalls managing processes, and the module take care of them all.
You probably want to start a qemu process and send the next commands writing to its standard input (stdin). Subprocess module will allow you to do it. See that qemu has command line options to connect to stdi: -chardev stdio ,id=id

Python Popen waiting while it shouldn't (bg and output redirected)

When I run directly in a terminal:
echo "useful"; sleep 10 &> /tmp/out.txt & echo "more";
I get both outputs while sleep goes on in the background. I was this same behavious with Popen (python 2.7):
p = Popen('echo "useful"; sleep 10 &> /tmp/out.txt & echo "more";', shell = True, stdout = PIPE, stderr = PIPE)
print p.communicate()
It was my understanding that a background process with redirected stdout and stderr would achieve this, but it does not; I have to wait for sleep. Can someone explain?
I need the other output so changing stdout/stderr arguments in Python is not a solution.
EDIT: I understand now that the behaviour I want (get the output but stop when no more output rather than when completed) is not possible from Python.
However, the behaviour appears more or less automatically when using ssh:
ssh 1.2.3.4 "echo \'useful\'; cd ~/dicp/python; nohup sleep 5 &> /tmp/out.txt & echo \'more\';"
(I can ssh to this address without password). So it's not entirely impossible by working around Python; now I need a way to do it without ssh...

That's because the shell process still has to wait for the background process to finish.
You don't normally realize this is happening because you normally are working in the shell where you backgrounded something. You put a process in the background so you can get control of the shell again and continue to work with it.
In other words, the background process is relative to the shell, not your Python process.

As Martijn Pieters points out, this is not how Python behaves (or is meant to behave). However, because the desired behaviour appears when running the command through ssh with nohup, I found this similar trick:
p = Popen('bash -c "echo \'useful\'; cd ~/dicp/python; nohup sleep 5 &> /tmp/out.txt & echo \'more\';"', shell = True, stdout = PIPE, stderr = PIPE)
print p.communicate()
So if I understand correctly, this starts a new shell (bash -c), which then starts a process not attached to it (nohup). The shell terminates as soon as all other processes complete, but the nohup-process keeps running. Desires behaviour achieved!
Maybe not pretty and probably not efficient, but it works.
EDIT: assuming, of course, that you are using bash. A more general answer is welcome.
EDIT2: actually, if my explanation is correct, I am not sure why nohup does not detach the process even if not using bash -c... Seems like bash -c would be redundant, just detach it from the shell started by Popen, but that does not work.

ssh hangs when command invoked directly, but exits cleanly when run interactive

I need to launch a server on the remote machine and retrieve the port number that the server process is lsitening on. When invoked, the server will listen on a random port and output the port number on stderr.
I want to automate the process of logging on to the remote machine, launching the process, and retrieving the port number. I wrote a Python script called "invokejob.py" that lives on the remote machine to act as a wrapper that invokes the job and then returns the port number, it looks like this:
import re, subprocess
executable = ... # Name of executable
regex = ... # Regex to extract the port number from the output
p = subprocess.Popen(executable,
bufsize=1, # line buffered
stderr=subprocess.PIPE
)
s = p.stderr.readline()
port = re.match(regex).groups()[0]
print port
If I log in interactively, this script works:
$ ssh remotehost.example.com
Last login: Thu Aug 28 17:31:18 2008 from localhost
$ ./invokejob.py
63409
$ exit
logout
Connection to remotehost.example.com closed.
(Note: successful logout, it did not hang).
However, if I try to invoke it from the command-line, it just hangs:
$ ssh remotehost.example.com invokejob.py
Does anybody know why it hangs in the second case, and what I can do to avoid this?
Note that I need to retrieve the output of the program, so I can't just use the ssh "-f" flag or redirect standard output.

s = p.stderr.readline()
I suspect it's the above line. When you invoke a command directly through ssh, you don't get your full pty (assuming Linux), and thus no stderr to read from.
When you log in interactively, stdin, stdout, and stderr are set up for you, and so your script works.

what if you do the following:
ssh <remote host> '<your command> ;<your regexp using awk or something>'
For example
ssh <remote host> '<your program>; ps aux | awk \'/root/ {print $2}\''
This will connect to , execute and then print each PSID for any user root or any process with root in its description.
I have used this method for running all kinds of commands on remote machines. The catch is to wrap the command(s) you wish to execute in single quotation marks (') and to separate each command with a semi-colon (;).