Develop Reference

execute which command over ssh in python script

I'm trying to run the command which solsql over SSH in a Python script.
I think the problem is in the ssh command and not the Python part, but maybe it's both.
I tried
subprocess.check_output("ssh root#IP which solsql",
stderr=subprocess.STDOUT, shell=True)
but I get an error.
I tried to run the command manually:
ssh root#{server_IP}" which solsql"
and I get a different output.
On the server I get the real path (/opt/solidDB/soliddb-6.5/bin/solsql)
but over SSH I get this:
which: no solsql in
(/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin)

I think what your looking for is something like paramiko. An example of how to use the library and issue a command to the remote system.
import base64
import paramiko
key = paramiko.RSAKey(data=base64.b64decode(b'AAA...'))
client = paramiko.SSHClient()
client.get_host_keys().add('ssh.example.com', 'ssh-rsa', key)
client.connect('ssh.example.com', username='THE_USER', password='THE_PASSWORD')
stdin, stdout, stderr = client.exec_command('which solsql')
for line in stdout:
print('... ' + line.strip('\n'))
client.close()

When you run a command over SSH, your shell executes a different set of startup files than when you connect interactively to the server. So the fundamental problem is really that the path where this tool is installed is not in your PATH when you connect via ssh from a script.
A common but crude workaround is to force the shell to read in the file with the PATH definition you want; but of course that basically requires you to know at least where the correct PATH is set, so you might as well just figure out where exactly the tool is installed in the first place anyway.
ssh server '. .bashrc; type -all solsql'
(assuming that the PATH is set up in your .bashrc; and ignoring for the time being the difference between executing stuff as yourself and as root. The dot and space before .bashrc are quite significant. Notice also how we use the POSIX command type rather than the brittle which command which should have died a natural but horrible death decades ago).
If you have a good idea of where the tool might be installed, perhaps instead do
subprocess.check_output(['ssh', 'root#' + ip, '''
for path in /opt/solidDB/*/bin /usr/local/bin /usr/bin; do
test -x "$path/solsql" || continue
echo "$path"
exit 0
done
exit 1'''])
Notice how we also avoid the (here, useless) shell=True. Perhaps see also Actual meaning of 'shell=True' in subprocess

First, you need to debug your error.
Use the code like this:
command = "ssh root#IP which solsql"
try:
retult = subprocess.check_output(command,shell=True,stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
print ("Result:", result)
It will output error message to you, and you'll know what to do, for example, ssh could have asked for a password, or didn't find your key, or something else.

Bash script will not run with subprocess in Python

For some reason, no matter how many variations I've tried, I can't seem to execute a bash script I've written. The command words 100% fine in Terminal, but when I try calling it with a subprocess, it returns nothing.
from os import listdir
import subprocess
computer_name = 'homedirectoryname'
moviefolder = '/Users/{}/Documents/Programming/Voicer/Movies'.format(computer_name)
string = 'The lion king'
for i in listdir(moviefolder):
title = i.split('.')
formatted_title = title[0].replace(' ', '\ ')
if string.lower() == title[0].lower():
command = 'vlc {}/{}.{}'.format(moviefolder, formatted_title, title[1])
subprocess.call(["/usr/local/bin",'-i','-c', command], stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)
else:
continue
The bash executable file looks like this:
#/bin/bash
func() {
open -a /Applications/VLC.app/Contents/MacOS/VLC $1
}
Where have I gone wrong?

You should call open directly:
import os
import subprocess
computer_name = 'homedirectoryname'
moviefolder = '/Users/{}/Documents/Programming/Voicer/Movies'.format(computer_name)
string = 'The lion king'
for filename in os.listdir(moviefolder):
title = filename.split('.')
if string.lower() == title[0].lower():
subprocess.call(['open', '-a', '/Applications/VLC.app/Contents/MacOS/VLC', os.path.join(moviefolder, filename)])

Since you are using shell=True, the command must be a string:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. (docs)

Like you even mentioned in a comment, you get /usr/local/bin: is a directory when you properly capture the error from the shell (and take out the erroneous shell=True; or correspondingly refactor the command line to be suitable for this usage, i.e. pass a string instead of a list).
Just to spell this out, you are attempting to run the command /usr/local/bin with some options; but of course, it's not a valid command; so this fails.
The actual script you seem to want to run will declare a function and then exit, which results in the function's definition being lost again, because the subprocess which ran the shell in which this function declaration was executed has now terminated and released all its resources back to the system.
Perhaps you should take more than just a few steps back and explain what you actually want to accomplish; but really, that should be a new, separate question.
Assuming you are actually trying to run vlc, and guessing some other things, too, perhaps you actually want
subprocess.call(['vlc','{}/{}.{}'.format(moviefolder, formatted_title, title[1]),
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
If your PATH is correct, you should not need to specify /usr/local/bin/ explicitly (and if your PATH is wrong, correct it in the code before, instead of hardcoding a directory for the executable you want to call).

/usr/local/bin is a directory. You can't run a directory as if it were a command.
Anyhow, there's no point to having /usr/local/bin anywhere in your command at all. Leave out the shell=True, and explicitly call vlc:
subprocess.call([
'vlc',
'{}/{}.{}'.format(moviefolder, formatted_title, title[1])
])

When shell=True is used in subprocess.call, if the command arguments is a sequence, then the first element of the sequence needs to be the command, and the rest are treated as argument(s) to the shell itself.
So, this should do:
subprocess.call(["/usr/local/bin/{}".format(command), '-i','-c'], shell=True, ...)
Otherwise, you can make the command a string.
Example:
In [20]: subprocess.call(["cat spamegg", "-i", "-c"], shell=True)
foobar

Execute batch file in different directory

I have a a file structure like the following (Windows):
D:\
dir_1\
batch_1.bat
dir_1a\
batch_2.bat
dir_2\
main.py
For the sake of this question, batch_1.bat simply calls batch_2.bat, and looks like:
cd dir_1a
start batch_2.bat %*
Opening batch_1.bat from a command prompt indeed opens batch_2.bat as it's supposed to, and from there on, everything is golden.
Now I want my Python file, D:\dir_2\main.py, to spawn a new process which starts batch_1.bat, which in turn should start batch_2.bat. So I figured the following Python code should work:
import subprocess
subprocess.Popen(['cd "D:/dir_1"', "start batch_1.bat"], shell=True)
This results in "The system cannot find the path specified" being printed to my Python console. (No error is raised, of course.) This is due to the first command. I get the same result even if I cut it down to:
subprocess.Popen(['cd "D:/"'], shell=True)
I also tried starting the batch file directly, like so:
subprocess.Popen("start D:/dir_1/batch_1.bat", shell=True)
For reasons that I don't entirely get, this seems to just open a windows command prompt, in dir_2.
If I forego the start part of this command, then my Python process is going to end up waiting for batch_1 to finish, which I don't want. But it does get a little further:
subprocess.Popen("D:/dir_1/batch_1.bat", shell=True)
This results in batch_1.bat successfully executing... in dir_2, the directory of the Python script, rather than the directory of batch_1.bat, which results in it not being able to find dir_1a\ and hence, batch_2.bat is not executed at all.
I am left highly confused. What am I doing wrong, and what should I be doing instead?

Your question is answered here: Python specify popen working directory via argument
In a nutshell, just pass an optional cwd argument to Popen:
subprocess.Popen(["batch_1.bat"], shell=True, cwd=r'd:\<your path>\dir1')

SSH: A javac command that works in terminal doesn't work when executed over SSH

I'm using Python code to run a Hadoop program on a Linux (Cloudera) machine using SSH.
I'm having some trouble with compiling java files to class files. When I'm executing the command:
javac -cp /usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/* remote_hadoop/javasrc/* from the Linux terminal all the files get compiled successfully.
When I'm executing the same command through my Python SSH client I'm receiving an 'invalid flag' error:
spur.results.RunProcessError: return code: 2
output: b''
stderr output: b'javac: invalid flag: remote_hadoop/javasrc\nUsage: javac \nuse -help for a list of possible options\n'
The python code:
list_of_commands = ["javac", "-cp", r"/usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/*", input_folder + r"/*"]
print ' '.join(list_of_commands)
self.shell.run(list_of_commands)
The command is getting rendered correctly, since what is getting printed is javac -cp /usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/* remote_hadoop/javasrc/*.
UPDATE: It's pretty weird. I can compile one file at a time over ssh, but not all of them. Seems like something happens to the "*" over ssh.

You're passing a list of arguments, not a list of commands. It's not even an accurate list of arguments.
If your underlying tool expects a list of arguments, then pass:
['sh', '-c', 'javac -cp /usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/* remote_hadoop/javasrc/*']
If it expects a list of commands:
['javac -cp /usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/* remote_hadoop/javasrc/*']
If it expects something else -- read the documentation and determine what that something is!
Note that SSH doesn't provide a way to pass a literal argv array when running an arbitrary command; rather, it expects -- at the protocol level -- a string ready for parsing by the remote shell. If your self.shell.run code is doing shell quoting before joining the argument list given, then it would be passing the last argument as the literal string remote_hadoop/javasrc/* -- not expanding it into a list of filenames as a shell would.
Using the sh -c form forces the remote shell to perform expansion on its end, assuming that contents are being given to it in a form which doesn't have remote expansion performed already.

The problem is the way that spur builds the command list into a command string. It takes every command token and encloses it in single quotes (["ls", "*.txt"]) becomes 'ls' '*.txt'). There is no shell expansion of * inside quotes, so the command doesn't work.
You can see the problem in spur's ssh.py on line 323:
def escape_sh(value):
return "'" + value.replace("'", "'\\''") + "'"
I don't use spur, but it looks like it just doesn't allow you to do such things. The problem with "simplifiers" like spur is that if they simplify in a way you don't want, you can't use them.

subprocess not working with change directory in python

I'm doing this simple thing
import subprocess
with cd("/home/myuserid"):
subprocess.call("ls ")
where cd is taken from here and it just does not work (same with any other path):
OSError: [Errno 2] No such file or directory

You have an extra space after ls, which is causing your issue. Remove that and it should work fine.
import subprocess
with cd("/home/myuserid"):
subprocess.call("ls")
When you use subprocess without shell=True, it interprets the entire string you pass as the command to execute. So it looks for a program literally called "ls " when you provide that extra space, which of course doesn't exist.
If you were to use shell=True, it would work fine even with the extra space, because a /bin/sh shell would be used to run the command, and the shell wouldn't care about the extra space. In general it's safer to use the default of shell=False, though, so I'd stick with that.