Edit:
The original intent of this question was to find a way to launch an interactive ssh session via a Python script. I'd tried subprocess.call() before and had gotten a Killed response before anything was output onto the terminal. I just assumed this was an issue/limitation with the subprocess module instead of an issue somewhere else.This was found not to be the case when I ran the script on a non-resource limited machine and it worked fine.
This then turned the question into: How can I run an interactive ssh session with whatever resource limitations were preventing it from running?
Shoutout to Charles Duffy who was a huge help in trying to diagnose all of this .
Below is the original question:
Background:
So I have a script that is currently written in bash. It parses the output of a few console functions and then opens up an ssh session based on those parsed outputs.
It currently works fine, but I'd like to expand it's capabilities a bit by adding some flag arguments to it. I've worked with argparse before and thoroughly enjoyed it. I tried to do some flag work in bash, and let's just say it leaves much to be desired.
The Actual Question:
Is it possible to have python to do stuff in a console and then put the user in that console?
Something like using subprocess to run a series of commands onto the currently viewed console? This in contrast to how subprocess normally runs, where it runs commands and then shuts the intermediate console down
Specific Example because I'm not sure if what I'm describing makes sense:
So here's a basic run down of the functionality I was wanting:
Run a python script
Have that script run some console command and parse the output
Run the following command:
ssh -t $correctnode "cd /local_scratch/pbs.$jobid; bash -l"
This command will ssh to the $correctnode, change directory, and then leave a bash window in that node open.
I already know how to do parts 1 and 2. It's part three that I can't figure out. Any help would be appreciated.
Edit: Unlike this question, I am not simply trying to run a command. I'm trying to display a shell that is created by a command. Specifically, I want to display a bash shell created through an ssh command.
Context For Readers
The OP is operating on a very resource-constrained (particularly, it appears, process-constrained) jumphost box, where starting an ssh process as a subprocess of python goes over a relevant limit (on number of processes, perhaps?)
Approach A: Replacing The Python Interpreter With Your Interactive Process
Using the exec*() family of system calls causes your original process to no longer be in memory (unlike the fork()+exec*() combination used to start a subprocess while leaving the parent process running), so it doesn't count against the account's limits.
import argparse
import os
try:
from shlex import quote
except ImportError:
from pipes import quote
parser = argparse.ArgumentParser()
parser.add_argument('node')
parser.add_argument('jobid')
args = parser.parse_args()
remote_cmd_str = 'cd /local_scratch/pbs.%s && exec bash -i' % (quote(args.jobid))
local_cmd = [
'/usr/bin/env', 'ssh', '-tt', node, remote_cmd_str
]
os.execv("/usr/bin/env", local_cmd)
Approach B: Generating Shell Commands From Python
If we use Python to generate a shell command, the shell can invoke that command only after the Python process exited, such that we stay under our externally-enforced process limit.
First, a slightly more robust approach at generating eval-able output:
import argparse
try:
from shlex import quote
except ImportError:
from pipes import quote
parser = argparse.ArgumentParser()
parser.add_argument('node')
parser.add_argument('jobid')
args = parser.parse_args()
remoteCmd = ['cd', '/local_scratch/pbs.%s' % (args.jobid)]
remoteCmdStr = ' '.join(quote(x) for x in remoteCmd) + ' && bash -l'
cmd = ['ssh', '-t', args.correctnode, remoteCmdStr]
print(' '.join(pipes.quote(x) for x in cmd)
To run this from a shell, if the above is named as genSshCmd:
#!/bin/sh
eval "$(genSshCmd "$#")"
Note that there are two separate layers of quoting here: One for the local shell running eval, and the second for the remote shell started by SSH. This is critical -- you don't want a jobid of $(rm -rf ~) to actually invoke rm.
This is in no way a real answer, just an illustration to my comment.
Let's say you have a Python script, test.py:
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('myarg', nargs="*")
args = parser.parse_args()
print("echo Hello world! My arguments are: " + " ".join(args.myarg))
So, you create a bash wrapper around it, test.sh
set -e
$(python test.py $*)
and this is what you get:
$ bash test.sh
Hello world! My arguments are:
$ bash test.sh one two
Hello world! My arguments are: one two
What is going on here:
python script does not execute commands. Instead, it outputs the commands bash script will run (echo in this example). In your case, the last command will be ssh blabla
bash executes the output of the python script (the $(...) part), passing on all its arguments (the $* part)
you can use argparse inside the python script; if anything is wrong with the arguments, the message will be put to stderr and will not be executed by bash; bash script will stop because of set -e flag
Related
Here is a simple GNU parallel command that creates a file called "example_i.txt" inside an existing directory called "example_i". It does this four times, for i from 1 to 4, with one job per core:
parallel -j 4 'cd example_{} && touch example_{}.txt' ::: {1..4}
Not very exciting, I know. The problem appears when I try to run this via python (v3.9) using the subprocess module as follows:
import subprocess
cmd = "parallel -j 4 'cd example_{} && touch example_{}.txt' ::: {1..4}"
subprocess.run(cmd, shell=True)
When doing so I get this error:
/bin/sh: 1: cd: can't cd to example_{1..4}
It looks like using the python subprocess call, bash is not triggering the call correctly as a GNU parallel command. Instead, it is substituting the {1..4} explicitly rather than dividing it up into four jobs.
I also tried this with the less advisable os.system(cmd) syntax and got back the same error.
PS: For context, this question stems from me trying to use UQpy (the RunModel module in particular) for uncertainty quantification of a Fortran code that was handed to me. Although this is not directly related to the question, it is relevant because I would like to know how to get this working using these tools as I am not at liberty to change them.
Following #Mark Setchell's comment, indeed it appears that bash is not used by default on POSIX as can be seen in the documentation for subprocess. This is solved by explicitly telling subprocess to use bash by re-writting my python code snippet as:
import subprocess
cmd = "parallel -j 4 'cd example_{} && touch example_{}.txt' ::: {1..4}"
subprocess.run(cmd, shell=True, executable='/bin/bash')
It should be noted that although the argument executable is here being used in the subprocess.run() call, it is not directly a part of this class. The executable argument is actually part of the subprocess.Popen() class, but it is accessible to subprocess.run() by the **other_popen_kwargs argument.
I'm trying to run a psql command in a Python script, with the subprocess command.
I use a Windows environment and the psql command aims to restore a database located in a remote Linux server.
The snippet is this one :
import os, sys
import subprocess
subprocess.call('psql -h ip_remote_server -p port -U user-d database -n schema --file="C:\Docs\script.sql"')
This does not work and the console tells that the specified file can't be found.
Any help would be greatly appreciated !
Thanks !
Yeah, your problem is definitely your paths. I went through the hassle of installing Python on Windows 10 and created these scripts:
example.bat
#echo off
echo This is a stand-in for your program
echo arg1 = %1
echo arg2 = %2
example.py
import subprocess
subprocess.call("C:\\Users\\bogus\\example.bat example arguments")
Console
C:\Users\bogus>python example.py
This is a stand-in for your program
arg1 = example
arg2 = arguments
As you can see, you do not need to pass shell=True, or split your command into a list.
If you look closely at the documentation for subprocess.call, you will see this (emphasis added):
The arguments shown above are merely some common ones. The full function signature is the same as that of the Popen constructor - this function passes all supplied arguments other than timeout directly through to that interface.
If you look closely the documentation for subprocess.Popen, you will see this (emphasis added):
On Windows, if args is a sequence, it will be converted to a string in a manner described in Converting an argument sequence to a string on Windows. This is because the underlying CreateProcess() operates on strings.
Any advice about splitting your arguments into a list, or passing shell=True, only applies to POSIX, with one exception:
The only time you need to specify shell=True on Windows is when the command you wish to execute is built into the shell (e.g. dir or copy). You do not need shell=True to run a batch file or console-based executable.
I'm attempting to use python to dynamically create bash aliases (like, for example, aliases to log in to a set of servers). I'd love to be able to do something like this:
from subprocess import call
SERVERS = [
("example", "user#example.com"),
#more servers in list
]
for server in SERVERS:
call('alias %s="ssh %s"' % (server[0], server[1]), shell=True)
The problem is that subprocess launches the jobs in a separate shell session, so the program runs fine, but does nothing to the shell session I run it from.
The same problem occurs with python's os.system or attempting to print the commands and pipe them to bash (all of these create the aliases, but in a new shell that is promptly destroyed after the program finishes).
Ultimately, the goal of this is to run this script from .bashrc
How does one do this?
You should write the alias commands to stdout. (eg. just use print).
Then the shell that is calling the Python script can run the alias commands itself.
As commented by #that other guy
eval "$(python yourscript.py)"
in your .bashrc should do it
Lets say I issue a command from the Linux command line. This will cause Linux to create a new Process and lets say that the Process expects to receive the command from the user.
For Example: I will run a python script test.py which will accept a command from the user.
$python test.py
TEST>addController(192.168.56.101)
Controller added
TEST>
The question I have is can I write a script which will go into the command line (TEST>) and issue a command? As far as I know if I write a script to run multiple commands it will wait for the first process to exit before running the next command.
Regards,
Vinay Pai B.H.
You should look into expect. It's a tool that is designed to automate user interaction with commands that need it. The man page explains how to use it.
Seems like there is also pexpect, a Python version of similar functionality.
Assuming the Python script is reading its commands from stdin, you can pass them in with a pipe or a redirection:
$ python test.py <<< 'addController(192.168.56.101)'
$ echo $'addController(192.168.56.101)\nfoo()\nbar()\nbaz()' | python test.py
$ python test.py <<EOF
addController(192.168.56.101)
foo()
bar()
baz()
EOF
If you don't mind waiting for the calls to finish (one at a time) before returning control to your program, you can use the subprocess library. If you want to start something running and not wait for it to finish, you can use the multiprocessing library.
i want to run and control PSFTP from a Python script in order to get log files from a UNIX box onto my Windows machine.
I can start up PSFTP and log in but when i try to run a command remotely such as 'cd' it isn't recognised by PSFTP and is just run in the terminal when i close PSFTP.
The code which i am trying to run is as follows:
import os
os.system("<directory> -l <username> -pw <password>" )
os.system("cd <anotherDirectory>")
i was just wondering if this is actually possible. Or if there is a better way to do this in Python.
Thanks.
You'll need to run PSFTP as a subprocess and speak directly with the process. os.system spawns a separate subshell each time it's invoked so it doesn't work like typing commands sequentially into a command prompt window. Take a look at the documentation for the standard Python subprocess module. You should be able to accomplish your goal from there. Alternatively, there are a few Python SSH packages available such as paramiko and Twisted. If you're already happy with PSFTP, I'd definitely stick with trying to make it work first though.
Subprocess module hint:
# The following line spawns the psftp process and binds its standard input
# to p.stdin and its standard output to p.stdout
p = subprocess.Popen('psftp -l testuser -pw testpass'.split(),
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# Send the 'cd some_directory' command to the process as if a user were
# typing it at the command line
p.stdin.write('cd some_directory\n')
This has sort of been answered in: SFTP in Python? (platform independent)
http://www.lag.net/paramiko/
The advantage to the pure python approach is that you don't always need psftp installed.