Develop Reference

Python getting output from running shell command - gcloud create dataproc cluster

I trying to get the expire-dataproc-tag from running gcloud-dataproc-create-cluster using python
I tried subprocess.Popen, the-issue I think due to it's an ERROR or it taking long time to retrieve the result, I end-up with and empty string
I tried command, and command_1 worked fine, the issue appeares when running command_2
import subprocess
command = "echo hello world"
command_1 = "gcloud compute images list --project {project-id} --no-standard-images"
command_2 = 'gcloud beta dataproc clusters create cluster-name --bucket {bucket} --region europe-west1 --zone europe-west1-b --subnet {subnet} --tags {tag} --project {project-id} --service-account {service-account} --master-machine-type n1-standard-16 --master-boot-disk-size 100 --worker-machine-type n1-standard-1 --worker-boot-disk-size 100 --image {image} --max-idle 2h --metadata enable-oslogin=true --properties {properties} --optional-components=ANACONDA,JUPYTER,ZEPPELIN --enable-component-gateway --single-node --no-address'.split(' ')
process = subprocess.Popen(command_2, stdout=subprocess.PIPE, shell=True)
# process.wait()
try:
print('inside-try')
result, err = process.communicate()
result = result.decode('utf-8')
except Exception as e:
print('The Error', e)
print('the result: ', result)
print("the-error: ", err)
the output is
inside-try
ERROR: (gcloud.beta.dataproc.clusters.create) INVALID_ARGUMENT: Dataproc custom image '{image-name}' has expired. Please rebuild this custom image. To extend the custom image expiration date to '2022-02-11T08:29:58.322549Z', please use this cluster property during cluster creation: 'dataproc:dataproc.custom.image.expiration.token=1.{image-name-properties......}'
the result:
the-error: None
I'm trying to get the ERROR: .... output to the result-variable (to be printed after the result)

You're not capturing stderr from the process.
Try:
process = subprocess.Popen(
command,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True
)
And so err wasn't being set by result, err = process.communicate()
With the above change, err will contain the error message that you're receiving.
I strongly encourage you to consider using Google's SDKs to interact with its services. Not only are these easier to use but, instead of shipping strings in/out of sub-processes, you can ship Python objects.
Here's the documentation for Creating a Dataproc cluster in Python.

Connect to gce instance and run command

I have a simple request. I want to connect to an already existing google compute engine instance, run a command, and close the connection.
I have used the great sample code here for instance creation and deletion.
Additionally, I have a startup script running which works perfectly.
Now I am reading this article to use paramiko to connect to my instance. This may or may not be the best thing to do, so please correct me if I am going down the wrong path.
I have the following sample code:
import paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(
paramiko.AutoAddPolicy())
ssh.connect('35.***.***.**',username='user',password='pass')
stdin, stdout, stderr = ssh.exec_command("sudo su -")
stdin, stdout, stderr = ssh.exec_command("ls -l")
stdout.readlines()
Now - I am not sure which username or password I am supposed to use.
When I run this code, I do not get the list of files and directories in my root as I want, but I do get a list of files and directories in the default user account's home - so it is connecting.
My goal is to connect to a gce instance, run a command, and that is it! For some reason it is trickier than I anticipated. Am I doing something wrong here?

If you are facing a similar use case you can explore gcloud ssh. It worked for me, but I cannot comment if this is best practice or not.
My solution here was something like the following:
import subprocess
def check_for_completion(instance_name = ""):
cmd = "gcloud compute ssh %s --zone=us-east1-b --command=\"sudo -S -i -u root -p '' ls /root/temp/ \""%(instance_name)
try:
res = subprocess.check_output(cmd, shell=True)
items = str(res).split('\n')
return {'response':items,'complete':False}
except:
return {'response':None,'complete':True}

Subprocess on remote server

I am using this code for executing command on remote server.
import subprocess
import sys
COMMAND="ls"
ssh = subprocess.Popen(["ssh", "%s" % HOST, COMMAND],
shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
result = ssh.stdout.readlines()
if result == []:
error = ssh.stderr.readlines()
print >>sys.stderr, "ERROR: %s" % error
else:
print result
When I try to execute this script, I get prompt for password. Is there any way I could avoid it, for example, can I enter password in script somehow? Also, password should be encrypted somehow so that people who have access to the script cannot see it.

Why make it so complicated? Here's what I suggest:
1) Create a ssh config section in your ~/.ssh/config file:
Host myserver
HostName 50.50.50.12 (fill in with your server's ip)
Port xxxx (optional)
User me (your username for server)
2) If you have generated your ssh keypair do it now (with ssh-keygen). Then upload with:
$ ssh-copy-id myserver
3) Now you can use subprocess with ssh. For example, to capture output, I call:
result = subprocess.check_output(['ssh', 'myserver', 'cat', 'somefile'])
Simple, robust, and the only time a password is needed is when you copy the public key to the server.
BTW, you code will probably work just fine as well using these steps.

One way is to create a public key, put it on the server, and do ssh -i /path/to/pub/key user#host or use paramiko like this:
import paramiko
import getpass
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
p = getpass.getpass()
ssh.connect('hostname', username='user', password=p)
stdin, stdout, stderr = ssh.exec_command('ls')
print stdout.readlines()
ssh.close()

You should use pexpect or paramiko to connect to remote machine,then spawn a child ,and then run subprocess to achieve what you want.

Here's what I did when encountering this issue before:
Set up your ssh keys for access to the server.
Set up an alias for the server you're accessing. Below I'll call it remote_server.
Put the following two lines at the end of ~/.bash_profile.
eval $(ssh-agent -s)
ssh-add
Now every time you start your shell, you will be prompted for a passphrase. By entering it, you will authenticate your ssh keys and put them 'in hand' at the start of your bash session. For the remainder of your session you will be able to run commands like
ssh remote_server ls
without being prompted for a passphrase. Here ls will run on the remote server and return the results to you. Likewise your python script should run without password prompt interruption if you execute it from the shell.
You'll also be able to ssh to the server just by typing ssh remote_server without having to enter your username or password every time.
The upside to doing it this way is that you should be doing this anyway to avoid password annoyances and remembering funky server names :) Also you don't have to worry about having passwords saved anywhere in your script. The only potential downside is that if you want to share the python script with others, they'll have to do this configuring as well (which they should anyway).

You don't really need something like pexpect to handle this. SSH keys already provide a very good and secure solution to this sort of issue.
The simplest way to get the results you want would probably be to generate an ssh key and place it in the .ssh folder of your device. I believe github has a pretty good guide to doing that, if you look into it. Once you set up the keys correctly on both systems, you won't actually have to add a single line to your code. When you don't specify a password it will automatically use the key to authenticate you.

While subprocess.Popen might work for wrapping ssh access, this is not the preferred way to do so.
I recommend using paramiko.
import paramiko
ssh_client = paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_client.connect(server, username=user,password=password)
...
ssh_client.close()
And If you want to simulate a terminal, as if a user was typing:
chan=ssh_client.invoke_shell()
def exec_cmd(cmd):
"""Gets ssh command(s), execute them, and returns the output"""
prompt='bash $' # the command line prompt in the ssh terminal
buff=''
chan.send(str(cmd)+'\n')
while not chan.recv_ready():
time.sleep(1)
while not buff.endswith(prompt):
buff+=ssh_client.chan.recv(1024)
return buff[:len(prompt)]
Example usage: exec_cmd('pwd')
If you don't know the prompt in advance, you can set it with:
chan.send('PS1="python-ssh:"\n')

You could use following.
import subprocess
import sys
COMMAND="ls"
ssh = subprocess.Popen("powershell putty.exe user#HOST -pw "password", stdout=PIPE, stdin=PIPE, stderr=STDOUT)
result = ssh.stdout.readlines()
if result == []:
error = ssh.stderr.readlines()
print >>sys.stderr, "ERROR: %s" % error
else:
print result

SSH into a server, run a command and save its output to a variable in Python

I am translating a bash program to Python and have got everything up to the last bit working. Here is the bash code, which simply SSHs into a server, executes a PostgreSQL database dump, and if there was no error, saves it to a file.
output="$(ssh "$SSH_LOCATION" bash <<< "$(printf 'PGPASSWORD=%q pg_dump -U %q %q' "$pg_pass" "$pg_user" "$database")")"
if [[ $? -ne 0 ]]
then
echo "$output"
echo "An error occurred while dumping the database"
exit $?
else
echo "$output" > ~/"$LOCAL_DIR"/"$database"
fi
The variables SSH_LOCATION, LOCAL_DIR, pg_pass, pg_user and database are all defined within the Python program.
Escaping of the variables for the remote shell, especially pg_pass, is very important. This was handled in the bash code by printf '%q'.
What is the best way to translate the above bash code to Python?
Edit: and another caveat, the authentication is handled by public key, not password. Of course, the ssh shell command handles this automatically.

Your python code would look something similar to the following if you want to use paramiko to ssh into the box.
import paramiko
import copy
def ssh_server(server, username=username, password=password, pg_pass, ps_user, database):
client = paramiko.SSHClient()
client.load_system_host_keys()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(server, username, password)
stdin, stdout, stderr = client.exec_command("PGPASSWORD=%s pg_dump -U %s %s" % (pg_pass, $pg_user, $database))
output = copy.deepcopy(stdout.readlines())
error = copy.deepcopy(stderr.readlines())
client.close()
if not error and output:
return stdout.readlines()
else:
print "An error occured\n%s" % error
return False
The following code is just an example. It needs to be tweaked to your needs, of course. Paramiko has support for RSA keys as well as password. If you can ssh passwordless, you don't need to provide a password then and you take the password parameter out.

What is the best way to translate the above bash code to Python?
The answer to such question, more often than not, is: use plumbum.
Borrowing from the docs, which should be enough to easily demonstrate how you can use it (and how awesome it is), you can do something like:
rem = SshMachine("hostname", user = "john", keyfile = "/path/to/idrsa")
output = rem.which("ls")

How do I execute Cassandra CLI commands from a Python script?

I have a python script that I want to use to make remote calls on a server, connect to Cassandra CLI, and execute commands to create keyspaces. One of the attempts that I made was something to this effect:
connect="cassandra-cli -host localhost -port 1960;"
create_keyspace="CREATE KEYSPACE someguy;"
exit="exit;"
final = Popen("{}; {}; {}".format(connect, create_keyspace, exit), shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
stdout, nothing = final.communicate()
Looking through various solutions, I'm not finding what I need. For example, the above code is throwing a "/bin/sh: 1: CREATE: not found", which I think means that it's not executing the CREATE statement on the CLI command line.
Any/all help would be GREATLY appreciated! Thank you!

try this out. I don't have cassandra-cli installed on my machine, so I couldn't test it myself.
from subprocess import check_output
from tempfile import NamedTemporaryFile
CASSANDRA_CMD = 'cassandra-cli -host localhost -port 1960 -f '
def cassandra(commands):
with NamedTemporaryFile() as f:
f.write(';\n'.join(commands))
f.flush()
return check_output(CASSANDRA_CMD + f.name, shell=True)
cassandra(['CREATE KEYSPACE someguy', 'exit'])
As you mentioned in the comment below pycassa a Python client for Cassandra cannot be used since it doesn't seem to support create statements.