Develop Reference

Passing variables to a script over ssh using gcloud command -- all variables treated as a single string?

I'm trying to setup a system to run some commands on VM's in google cloud, in my case we want to run a tcpdump at a certain time using the 'at' command. Right now I'm just trying to execute any commands succesfully, when I have to pass arguments along with the command and getting confusing behaviour, which appears to be that the command, and the arguments are executed as a single long command instead of seperate arguements.
I first tried in bash, and thinking my issue was one of quoting, I moved to using python to hopefully make things easier to understand, but I appear to be hitting the same issue and figure I must be doing something wrong.
I have the following functions defined in python, and call them
def execute(cmd):
popen = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
for stdout_line in iter(popen.stdout.readline, ""):
yield stdout_line
popen.stdout.close()
return_code = popen.wait()
if return_code:
raise subprocess.CalledProcessError(return_code, cmd)
def runCapture(project, instance, zone, time, duration):
## Run capture against server
print ("Running capture against Project: " + project + ", Instance: " + instance + ", Zone: " + zone, "at: " + time, "for " + str(duration) + " minutes")
## First connect, schedule capture
## Connect again, schedule upload of capture at capture time + duration time + some overrun.
## gcloud compute ssh --project=${PROJECT} ${INSTANCE} --zone="${ZONE}" --command="...do stuff..." --tunnel-through-iap
## CMD=\${1:-"/usr/sbin/tcpdump -nn -i ens4 -G \$(( ${DURATION}*60 )) -W 1 -w ./\$(uname -n)-%Y-%m-%d_%H.%M.%S.pcap"}
total_time=str(duration*60)
command="/bin/bash -c 'echo \"hello world\"'"
for path in execute(["/usr/bin/gcloud", "compute", "ssh", instance, "--project="+project, "--zone="+zone, "--tunnel-through-iap", "--command=\""+command+"\"", ]):
print(path, end="")
The resulting errors are as follows:
bash: /bin/bash -c 'echo hello: No such file or directory
Traceback (most recent call last):
File "./ingressCapture.py", line 79, in <module>
results = runCapture(project, instance, zone, time, duration)
File "./ingressCapture.py", line 33, in runCapture
for path in execute(["/usr/bin/gcloud", "compute", "ssh", instance, "--project="+project, "--zone="+zone, "--tunnel-through-iap", "--command=\""+command+"\"", ]):
File "./ingressCapture.py", line 17, in execute
raise subprocess.CalledProcessError(return_code, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/gcloud', 'compute', 'ssh', 'tbtst-test3-app-egress-nztw', '--project=devops-tb-sandbox-250222', '--zone=europe-west1-b', '--tunnel-through-iap', '--command="/bin/bash -c \'echo "hello world"\'"']' returned non-zero exit status 127.
It appears to me, that instead of invoking the bash shell and running the echo command, it is instead invoking a command that includes the bash shell and then all the arguments too. I have a bash shell when I login normally via SSH, and can run the commands manually (and they work). Why are the arguments for the command from --command="....." getting called like this and how do I prevent it?

I'm pretty sure your problem is that you have too many quotes.
When you write --command="bash -c 'echo \"Hello World\"'" on the command line, the shell internally marks all the stuff inside the quotes as being in a quoted state and then removes the quotes. The actual argument that ends up going to the program is --command=bash -c 'echo "Hello World"' as a single string in argv (or your language's equivalent).
Try putting import sys ; print(sys.argv[1]) inside a small python script and calling it with ./test.py --command="bash -c 'echo \"Hello World\"'" to see for yourself.
However, in your arglist to subprocess, you're forming this string: --command="/bin/bash -c 'echo "hello world"'", presumably because you thought you needed to match what you'd normally type on the command line. You can see this in the stacktrace (minus the escaped single quotes, since that's syntax highlighting from python). Since python does not perform quote removal, those quotes are going through to the other side of your ssh connection where the login shell is attempting to reparse it as a shell command. The first "word" on the other end of the connection is /bin/bash -c 'echo hello because of those extra quotes so the shell attempts to find a command with that name on the path, and it clearly doesn't exist.
What you need to put into your arglist for subprocess is simply "--command="+command.

Not able to run command via wsl.exe

I'm trying to run this commands from a python script:
def raw(path_avd_py, path_avd, snp_name, out_file):
if OS == 'Windows':
cmd_raw = f"wsl.exe -e sh -c 'python3 {path_avd_py} -a {path_avd}
-s {snp_name} -o {out_file}'"
else:
cmd_raw = f'python3 {path_avd_py} -a {path_avd} -s {snp_name} -o {out_file}'
subprocess.Popen(cmd_raw, shell=True)
time.sleep(25)
return None
def idiffer(i_path, raw_1, raw_2, path, state):
if OS == 'Windows':
cmd_idiff = f"wsl.exe -e sh -c 'python3 {i_path} {raw_1} {raw_2}'"
[...]
file = os.path.join(path, f'{state}.idiff')
with open(file, 'w') as f:
subprocess.Popen(cmd_idiff, stdout=f, text=True)
If im executing cmd_raw with subprocess.run from a python-shell (Powershell), things are working. If im try running this via script, this exception occurs, using different shells:
-e sh: avdecrypt-master\avdecrypt.py: 1: Syntax error: Unterminated quoted string
-e bash: avdecrypt-master\avdecrypt.py: -c: line 0: unexpected EOF while looking for matching `''
avdecrypt-master\avdecrypt.py: -c: line 1: syntax error: unexpected end of file
I already tried os.system, os.run([list]) no change.
Thanks for the help!

For those who have a similar question, I found a solution, which is working for me:
Apparently calling scripts with some argv has to be in one single quotation mark and can be executed via run (in my case important, because the process has to be terminated). This leads to a form like:
cmd = ['wsl.exe', '-e', 'bash', '-c', '-a foo -b bar [...]']
subprocess.run(cmd, shell=True)
Lib shlex is helping here and formatting the strings like subprocess is needing it:
cmd_finished = shlex.split(cmd)
https://docs.python.org/3/library/shlex.html

linux directory permission check and/or dir exist or not

I have a script which is running as a ROOT on linux collecting data from different users. Given a nfs path for each individual user 1) verify the director does not exist 2) verify permission denied
verify_model_path_not_found = '/usr/bin/su ' + userID + ' -c \'/usr/bin/ls ' + u_model_path + ' | /usr/bin/grep \"Permission denied\|No such file or directory\"\''
perm_denied_str = 'Permission denied'
no_file_dir_str = 'No such file or directory'
print("verify_model_path_not_found:", verify_model_path_not_found)
#Run as a root
try:
verify_cmd_out = subprocess.check_output(verify_model_path_not_found, shell=True)
verify_cmd_out = str(verify_cmd_out)
print("verify_cmd_out:", verify_cmd_out, "\n")
except subprocess.CalledProcessError as errorcatch:
print(datetime.datetime.now(), " Error while executing ", verify_model_path_not_found)
print("error code", errorcatch.returncode, errorcatch.output, "\n")
continue
#only add items that are not accessible (perm denied or no file found)
if ((perm_denied_str in verify_cmd_out) or (no_file_dir_str in verify_cmd_out)):
#remaining actions .... send an email to the user ...
Example output Error:
verify_model_path_not_found: /usr/bin/su xxxx -c '/usr/bin/ls /nfs/x/y/z | /usr/bin/grep "Permission denied\|No such file or directory"'
2021-08-10 17:00:31.827186 Error while executing /usr/bin/su xxxx -c '/usr/bin/ls /nfs/x/y/z | /usr/bin/grep "Permission denied\|No such file or directory"'
error code 1 b'' #I know this dir does not exist or perm denied - still getting error
given /nfs/x/y/z, if the user does not have a read access, I would like to get "Permission denied" using grep - "Permission denied" should be the value of verify_cmd_out
given /nfs/x/y/z, if the dir does not exist, I would like to get "No such file or directory" using grep - "No such file or directory" should be the value of verify_cmd_out
once perm denied or no such file is confirmed for the user, certain actions need to place.
The /usr/bin/su xxxx -c ... command is not properly working, any thought or idea how to resolve the issue?

You are examining standard output (file descriptor 1), but error messages (and progress and diagnostics in general) are posted on standard error (file descriptor 2).
Your code is quite clunky anyway. Probably try something along the lines of
import subprocess
def assert_error_stderr(cmd, expected_stderr):
"""
Accept a shell command; verify that it fails with error,
and emits the expected message on standard error.
"""
try:
result = subprocess.run(cmd, check=True, text=True, capture_output=True)
raise ValueError("%r did not raise an error" % cmd)
except CalledProcessError:
assert expected_stderr in result.stderr
def assert_silent_failure(cmd):
"""
Check that a command fails; verify that standard error is empty
"""
try:
result = subprocess.run(cmd, check=True, text=True, capture_output=True)
except CalledProcessError:
assert result.stderr == ''
raise ValueError("%r did not fail", cmd)
assert_silent_failure(['su', '-c' 'test -d ' + u_model_path])
...
but of course using Python when you fundamentally want to test the shell might not make much sense.
#!/bin/sh
! test -d "$1"
Basically never use ls in scripts and generally probably don't rely on a particular error message (it could be localized, or change between versions).
Also, in Python subprocess code, avoid shell=True whenever you can. See also Actual meaning of shell=True in subprocess

Error: function Input_stream::Input_stream(const string&, bool) line 63. Error opening file #HWI-M02942_file1.fasta

I am writing a Python script to perform a BLAST by using the BLAST program DIAMOND automatically. The script executes commands in the terminal of Ubuntu 14.04.
My Python script is:
import subprocess
data_location = "/home/markschuurman/Desktop/Onderzoek_BioCentre/data_course_4/"
input_fasta_file = "#HWI-M02942_file1.fasta"
diamond_temp_dir = "/home/markschuurman/Desktop/DIAMOND_temp_dir/"
diamond_blast_database_location = "/home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/DIAMOND_BLAST_databases/"
diamond_blast_output_file_directory = "/home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/output_files/"
diamond_blast_output_filemame_daa = "matches.daa"
diamond_blast_output_filemame_tsv = "matches.tsv"
max_hits_per_read = "5"
max_evalue = "10"
commands = ["cd " + data_location,
"diamond blastx -d " + diamond_blast_database_location + "tcdb -q " + input_fasta_file + " -a " + diamond_blast_output_file_directory + diamond_blast_output_filemame_daa + " -t " + diamond_temp_dir + " -k " + max_hits_per_read + " -e " + max_evalue,
"diamond view -a " + diamond_blast_output_file_directory + diamond_blast_output_filemame_daa + " -o " + diamond_blast_output_file_directory + diamond_blast_output_filemame_tsv]
for command in commands:
print "Command : " + command
p = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
p_status = p.wait()
print "Command finished"
The script creates the commands to execute after assigning the correct file paths and file names to the variables.
When I try to run this script I get the following error:
/usr/bin/python2.7 /home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/scripts_to_parse_DIAMOND_output/execute_DIAMOND_BLAST.py
Command : cd /home/markschuurman/Desktop/Onderzoek_BioCentre/data_course_4/
Command finished
Command : diamond blastx -d /home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/DIAMOND_BLAST_databases/tcdb -q #HWI-M02942_file1.fasta -a /home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/output_files/matches.daa -t /home/markschuurman/Desktop/DIAMOND_temp_dir/ -k 5 -e 10
Error: function Input_stream::Input_stream(const string&, bool) line 63. Error opening file #HWI-M02942_file1.fasta
Command finished
Command : diamond view -a /home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/output_files/matches.daa -o /home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/output_files/matches.tsv
Error: function Input_stream::Input_stream(const string&, bool) line 75. Error opening file /home/markschuurman/Desktop/Onderzoek_BioCentre/BLAST_with_DIAMOND/output_files/matches.daa
Command finished
I am sure that the commands are correct because, when I execute the commands printed in line 20 separately in terminal there are no errors and the output of the BLAST application is correct.
Why does this error occur while executing the commands in this Python script and not separately in terminal and how to solve this error?

The problem here is that subprocess.Popen() runs the command in the separate process that exits when the command has completed running. The cd command and diamond commands are run in separate processes.
This means that diamond is looking for #HWI-M02942_file1.fasta in the directory where you run the command from.
Your solution to simply use the absolute path here is probably the simplest.

subprocess ssh command fails for some commands but not others (command works in terminal)

As part of a python script, I am hoping to capture the output of a shell command executed via ssh, namely
ssh User#999 screen -list
If I execute the above command directly in terminal, I get the results I need. However, when executing through subprocess.check_output as below, I get a non-zero exit status 1 error.
I am able to execute other commands via ssh and capture the output without problem.
Is there something specific about screen -list that does not like being called in this fashion?
import subprocess
srvr = 'User#999.99.999.9'
print("CMD 1: ===============")
cmd1 = "ssh " + srvr + " ls -l"
print ("COMMAND IS ..... " + cmd1 + "\n")
out1 = subprocess.check_output(cmd1, shell=True)
print(out1 + "\n")
print("CMD 2: ===============")
cmd2 = "ssh " + srvr + " screen -list"
print ("COMMAND IS ..... " + cmd2 + "\n")
out2 = subprocess.check_output(cmd2, shell=True)
print(out2 + "\n")
Error:
subprocess.CalledProcessError: Command '['ssh User#999.99.999.9 screen', '-list']' returned non-zero exit status 1

subprocess.check_output check the exit code of the subprocess; and it raises exception if the exit code is not zero.
If you don't care about exit code, use subprocess.Popen.communicate:
out1, err1 = subprocess.Popen(cmd1,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE).communicate()

That's how subprocess.check_output() is supposed to work. See: http://docs.python.org/2/library/subprocess.html
The command on your server is returning a non zero return code and thus is raising the appropriate Exception CalledProcessError.