Develop Reference

Passing variables to a script over ssh using gcloud command -- all variables treated as a single string?

I'm trying to setup a system to run some commands on VM's in google cloud, in my case we want to run a tcpdump at a certain time using the 'at' command. Right now I'm just trying to execute any commands succesfully, when I have to pass arguments along with the command and getting confusing behaviour, which appears to be that the command, and the arguments are executed as a single long command instead of seperate arguements.
I first tried in bash, and thinking my issue was one of quoting, I moved to using python to hopefully make things easier to understand, but I appear to be hitting the same issue and figure I must be doing something wrong.
I have the following functions defined in python, and call them
def execute(cmd):
popen = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
for stdout_line in iter(popen.stdout.readline, ""):
yield stdout_line
popen.stdout.close()
return_code = popen.wait()
if return_code:
raise subprocess.CalledProcessError(return_code, cmd)
def runCapture(project, instance, zone, time, duration):
## Run capture against server
print ("Running capture against Project: " + project + ", Instance: " + instance + ", Zone: " + zone, "at: " + time, "for " + str(duration) + " minutes")
## First connect, schedule capture
## Connect again, schedule upload of capture at capture time + duration time + some overrun.
## gcloud compute ssh --project=${PROJECT} ${INSTANCE} --zone="${ZONE}" --command="...do stuff..." --tunnel-through-iap
## CMD=\${1:-"/usr/sbin/tcpdump -nn -i ens4 -G \$(( ${DURATION}*60 )) -W 1 -w ./\$(uname -n)-%Y-%m-%d_%H.%M.%S.pcap"}
total_time=str(duration*60)
command="/bin/bash -c 'echo \"hello world\"'"
for path in execute(["/usr/bin/gcloud", "compute", "ssh", instance, "--project="+project, "--zone="+zone, "--tunnel-through-iap", "--command=\""+command+"\"", ]):
print(path, end="")
The resulting errors are as follows:
bash: /bin/bash -c 'echo hello: No such file or directory
Traceback (most recent call last):
File "./ingressCapture.py", line 79, in <module>
results = runCapture(project, instance, zone, time, duration)
File "./ingressCapture.py", line 33, in runCapture
for path in execute(["/usr/bin/gcloud", "compute", "ssh", instance, "--project="+project, "--zone="+zone, "--tunnel-through-iap", "--command=\""+command+"\"", ]):
File "./ingressCapture.py", line 17, in execute
raise subprocess.CalledProcessError(return_code, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/gcloud', 'compute', 'ssh', 'tbtst-test3-app-egress-nztw', '--project=devops-tb-sandbox-250222', '--zone=europe-west1-b', '--tunnel-through-iap', '--command="/bin/bash -c \'echo "hello world"\'"']' returned non-zero exit status 127.
It appears to me, that instead of invoking the bash shell and running the echo command, it is instead invoking a command that includes the bash shell and then all the arguments too. I have a bash shell when I login normally via SSH, and can run the commands manually (and they work). Why are the arguments for the command from --command="....." getting called like this and how do I prevent it?

I'm pretty sure your problem is that you have too many quotes.
When you write --command="bash -c 'echo \"Hello World\"'" on the command line, the shell internally marks all the stuff inside the quotes as being in a quoted state and then removes the quotes. The actual argument that ends up going to the program is --command=bash -c 'echo "Hello World"' as a single string in argv (or your language's equivalent).
Try putting import sys ; print(sys.argv[1]) inside a small python script and calling it with ./test.py --command="bash -c 'echo \"Hello World\"'" to see for yourself.
However, in your arglist to subprocess, you're forming this string: --command="/bin/bash -c 'echo "hello world"'", presumably because you thought you needed to match what you'd normally type on the command line. You can see this in the stacktrace (minus the escaped single quotes, since that's syntax highlighting from python). Since python does not perform quote removal, those quotes are going through to the other side of your ssh connection where the login shell is attempting to reparse it as a shell command. The first "word" on the other end of the connection is /bin/bash -c 'echo hello because of those extra quotes so the shell attempts to find a command with that name on the path, and it clearly doesn't exist.
What you need to put into your arglist for subprocess is simply "--command="+command.

Running PowerShell Script from Python

I'm trying to run a Powershell Script ( check below )
First remark, my Powershell script, when running with Powershell works fine, giving the expected result ( closing all open folders ) , but I have to call this from a Python script, therefore I tried using subprocess.Popen, but I'm having the error :
"io.UnsupportedOperation: fileno"
Tried several different things already, but the solutions suggested aren't working.
I'm trying to call the following Powershell Script :
$shell = New-Object -ComObject Shell.Application
$shell.Windows() | Format-Table LocationName, LocationURL
$window = $shell.Windows()
$window | ForEach-Object { $_.Quit() }
The idea is to close all the open folders, and directly running with Powershell works as expected.
Then I tried to call this script from Python :
p = subprocess.Popen(['powershell.exe', 'C:\\Users\\(correct subfolders)\\TEST.ps1'])
or
p = subprocess.Popen(['C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe', 'C:\\Users\\(correct subfolders)\\TEST.ps1'])
and they don't return errors, a cmd window opens, but nothing happens.
Then I tried the following :
p = subprocess.Popen(['powershell.exe', 'C:\\Users\\(correct subfolders)\\TEST.ps1'], stdout=sys.stdout)
But I have the following error ( check below )
File "<pyshell#15>", line 1, in <module>
p = subprocess.Popen(['powershell.exe', 'C:\\Users\\FernanP\\Desktop\\TEST.ps1'], stdout=sys.stdout)
File "C:\Users\FernanP\AppData\Local\Programs\Python\Python36-32\lib\subprocess.py", line 667, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "C:\Users\FernanP\AppData\Local\Programs\Python\Python36-32\lib\subprocess.py", line 922, in _get_handles
c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
io.UnsupportedOperation: fileno
I looked into similar Questions here, and people said about the Pillow version, but I'm currently using 6.1.0, so it should not be a problem.
Therefore, either an equivalent to the powershell script directly in Python would be enough, or a way to deal with this issue.
Hope I informed sufficiently.
KR

Here I have created my own function to run any powershell script with its parameters
import subprocess # IMPORT FOR SUB PROCESS . RUN METHOD
POWERSHELL_PATH = "powershell.exe" # POWERSHELL EXE PATH
ps_script_path = "C:\\PowershellScripts\\FTP_UPLOAD.PS1" # YOUR POWERSHELL FILE PATH
class Utility: # SHARED CLASS TO USE IN OUR PROJECT
#staticmethod # STATIC METHOD DEFINITION
def run_ftp_upload_powershell_script(script_path, *params): # SCRIPT PATH = POWERSHELL SCRIPT PATH, PARAM = POWERSHELL SCRIPT PARAMETERS ( IF ANY )
commandline_options = [POWERSHELL_PATH, '-ExecutionPolicy', 'Unrestricted', script_path] # ADD POWERSHELL EXE AND EXECUTION POLICY TO COMMAND VARIABLE
for param in params: # LOOP FOR EACH PARAMETER FROM ARRAY
commandline_options.append("'" + param + "'") # APPEND YOUR FOR POWERSHELL SCRIPT
process_result = subprocess.run(commandline_options, stdout = subprocess.PIPE, stderr = subprocess.PIPE, universal_newlines = True) # CALL PROCESS
print(process_result.returncode) # PRINT RETURN CODE OF PROCESS 0 = SUCCESS, NON-ZERO = FAIL
print(process_result.stdout) # PRINT STANDARD OUTPUT FROM POWERSHELL
print(process_result.stderr) # PRINT STANDARD ERROR FROM POWERSHELL ( IF ANY OTHERWISE ITS NULL|NONE )
if process_result.returncode == 0: # COMPARING RESULT
Message = "Success !"
else:
Message = "Error Occurred !"
return Message # RETURN MESSAGE

You can use subprocess.run and need PIPE and shell. The following code worked for me:
import subprocess
result = subprocess.run([r'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe', r'C:\Users\(correct subfolders)\TEST.ps1'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
print(result)
Printing the result can give you the return value like if the command was successfully executed. If you want to extract the result value, you can do,
print(result.stdout.decode('utf-8'))

How do I pass applescript arguments from a python script?

I have an applescript that takes in two parameters on execution.
on run {targetBuddyPhone, targetMessage}
tell application "Messages"
set targetService to 1st service whose service type = iMessage
set targetBuddy to buddy targetBuddyPhone of targetService
send targetMessage to targetBuddy
end tell
end run
I then want this script to execute from within a python script. I know how to execute a applescript from python, but how do I also give it arguments? Here is the python script that I currently have written out.
#!/usr/bin/env python3
import subprocess
def run_applescript(script, *args):
p = subprocess.Popen(['arch', '-i386', 'osascript', '-e', script] +
[unicode(arg).encode('utf8') for arg in args],
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
err = p.wait()
if err:
raise RuntimeError(err, p.stderr.read()[:-1].decode('utf8'))
return p.stdout.read()[:-1].decode('utf8')
The error I receive after trying to execute this code in the terminal is:
Traceback (most recent call last):
File "messageExecuter.py", line 14, in <module>
run_applescript("sendMessage.scpt",1111111111,"hello")
File "messageExecuter.py", line 11, in run_applescript
raise RuntimeError(err, p.stderr.read()[:-1].decode('utf8'))
RuntimeError: (1, u'arch: posix_spawnp: osascript: Bad CPU type in executable')

Clue is in the error message. Delete 'arch', '-i386' from arguments list, as osascript is 64-bit only.

Why does my python script break after compiling?

I'm checking out how much of a performance increase I get after compiling a python script. After research looking into this issue I don't think I will actually see an increase in performance with the script I have written because I found out that once the script is loaded, the execution time doesn't increase. I still would like to know why this is failing to run after compiling as this is my first time trying this. Here is my script
#!/bin/python3
from datetime import datetime
start = datetime.now()
import psutil
BYTES_PER_GB = 1024*1024*1024
# Memory
m = psutil.virtual_memory()
#total = m.total/BYTES_PER_GB
#available = m.available/BYTES_PER_GB
#used = m.used/BYTES_PER_GB
m_free= m.free/BYTES_PER_GB
m_percent = m.percent
# Swap
s = psutil.swap_memory()
s_free = s.free/BYTES_PER_GB
s_percent = s.percent
print(' %.1fG (%.1f%%)  %.1fG (%.1f%%)' % (m_free, m_percent, s_free, s_percent))
print(' %.1fG (%.1f%%)  %.1fG (%.1f%%)' % (m_free, m_percent, s_free, s_percent))
print(datetime.now() - start)
I'm trying to compile with this line
python3 -m py_compile memory
In my print statements I have some special characters from font awesome. Not sure if that would cause a problem but if it doesn't show up correctly in my post then that's what that is.
The output when I try to run the compiled file is
./memorycpython-35.pyc: line 1: $'\026\r\r': command not found
./memorycpython-35.pyc: line 2: �k�W��#s�ddlmZej�ZddlZdZej�Zejeej: command not found
./memorycpython-35.pyc: line 3: syntax error near unexpected token `)'
./memorycpython-35.pyc: line 3: `ej
�Z
e
e je
j Ze�eej�e�dS)�)datetimeNiii#)rZnow�startZpsutilZ
BYTES_PER_GBZvirtual_memory�mZfreeZm_freeZpercentZ m_percentZ
swap_memory�sZs_freeZ s_percent�print�rr�memory<module>s
'
^[[?62;c^[[?62;c
EDIT
To narrow down the problem I wrote the following script
#!/bin/python3
print("Hello World!")
This is the output
./testcpython-35.pyc: line 1: $'\026\r\r': command not found
./testcpython-35.pyc: line 2: syntax error near unexpected token `)'
./testcpython-35.pyc: line 2: `�r�W%�#sed�dS)z
Hello World!N)�print�rr�./test<module>s'
Compiled using
python3 -m py_compile ./test
This creates a file in __pycache__/ called testcpython-35.pyc which I then do chmod +x testcpython-35.pyc and ./testcpython-35.pyc

It appears my issue is the ./testcpython-35.pyc part. When I run python3 testcpython-35.pyc, independent on whether I did chmod +x ./testcpython-35.pyc, the output is scrabbled. As long as I run the compiled program by first specifying what program to run it with, python3, it outputs Hello World! as expected.

Python subprocess Exec format error

Sorry if this question is dumb. I am using python subprocess statement to call a .bat file in Ubuntu (Natty 11.04), however, I got error messages:
Traceback (most recent call last):
File "pfam_picloud.py", line 40, in <module>
a=subprocess.Popen(src2, shell=0)
File "/usr/lib/python2.7/subprocess.py", line 672, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1213, in _execute_child
raise child_exception
run this python file
$python pfam_picloud.py
Python code (pfam_picloud.py)
#!/usr/bin/python
#
met="wTest.dvf"
run="run_pfam.bat"
inp="pfam_input.PFA"
import os
import stat
import shutil
import subprocess
import string
import random
# Generate a random ID for file save
def id_generator(size=6, chars=string.ascii_uppercase + string.digits):
return ''.join(random.choice(chars) for x in range(size))
name_temp=id_generator()
cwd=os.getcwd()
src=cwd
src1=cwd+'/'+name_temp
if not os.path.exists(src1):
os.makedirs(src1)
else:
shutil.rmtree(src1)
os.makedirs(src1)
##
shutil.copy(src+"/"+run,src1)
shutil.copy(src+"/"+met,src1)
shutil.copy(cwd+"/pfam_pi.exe",src1)
shutil.copy(src+"/"+inp,src1)
#
src2=src1+"/run_pfam.bat"
os.chdir(src1)
a=subprocess.Popen(src2, shell=0)
a.wait()
bash file (run_pfam.bat)
#!/bin/sh
./pfam_pi.exe pfam_input.PFA
I can successfully run this bash file in Ubuntu. So I guess, I messed up something in my Python script. Could anyone give me some suggestions? Thanks for any inputs.
EDIT
the file pfam_pi.exe is a Linux executable. I compiled it in Ubuntu. Sorry for the confusion.
update
Well, I got different types of error now.
1. With #!/bin/sh, it said No such file or directory.
2. With /bin/sh, it said exec format error.
3. If I sent everything as arguments a=subprocess.Popen(['./pfam_pi.exe', 'inp', 'src1'], shell=0), it said end of line symbol error

Since feature requests to mark a comment as an answer remain declined, I copy the above solution here.
#Ellioh: Thanks for your comments. I found once I changed the shell=1, problem is solved. – tao.hong

Try running wine (you should have it installed) and pass pfam_pi.exe to it as a parameter. Maybe pfam_pi.exe is not a Linux executable. :-) Certainly, executable file extensions are not meaningful on Linux, but probably it really is a Windows program, otherwise I hardly can imagine it named pfam_pi.exe.
However, if it is a Linux executable, note subprocess.Popen accepts a list of args (the first element is the program itself), not a command line:
>>> import shlex, subprocess
>>> command_line = raw_input()
/bin/vikings -input eggs.txt -output "spam spam.txt" -cmd "echo '$MONEY'"
>>> args = shlex.split(command_line)
>>> print args
['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"]
>>> p = subprocess.Popen(args) # Success!