I'm trying to execute rsync over ssh from a subprocess in a python script to copy images from one server to another. I have a function defined as:
def rsyncBookContent(bookIds, serverEnv):
bookPaths = ""
if len(bookIds) > 1:
bookPaths = "{" + ",".join(("book_"+str(x)) for x in bookIds) + "}"
else:
bookPaths = "book_" + str(bookIds[0])
for host in serverEnv['content.hosts']:
args = ["rsync", "-avz", "--include='*/'", "--include='*.jpg'", "--exclude='*'", "-e", "ssh", options.bookDestDir + "/" + bookPaths, "jill#" + host + ":/home/jill/web/public/static/"]
print "executing " + ' '.join(args)
subprocess.call(args)
What I'm ultimately trying to do is have Python execute this (which works from a bash shell):
rsync -avz --include='*/' --include='*.jpg' --exclude='*' -e ssh /shared/books/{book_482,book_347} jill#10.12.27.20:/home/jill/web/public/static/
And indeed my print statement outputs:
executing rsync -avz --include='*/' --include='*.jpg' --exclude='*' -e ssh /shared/books/{book_482,book_347} jill#10.12.27.20:/home/jill/web/public/static/
But when executed from within this python script, there are two problems:
if len(bookIds) > 1, the list of sub-directories under /shared/books/ is somehow misinterpreted by bash or rsync. The error message is:
rsync: link_stat "/shared/books/{book_482,book_347}" failed: No such file or directory (2))
if len(bookIds) == 1, all files under the source directory are rsynced (not just *.jpg, as is my intention)
Seems as if the subprocess.call function requires some characters to be escaped or something, no?
Figured out my issues. My problems were the result of my misunderstanding of how the subprocess.call function executes and bash's expansion of lists inside curly braces.
When I was issuing the rsync command in a bash shell with subdirectories in curly braces, bash was really expanding that into multiple arguments which were being passed to rsync (/shared/books/book_1 shared/books/book_2, etc.). When passing the same string with curly braces "/shared/books/{book_1, book_2}" to the subprocess.call function, the expansion wasn't happening, since it wasn't going through bash, so my argument to rsync was really "/shared/books/{book_1, book_2}".
Similarly, the single quotes around the file patterns ('*', '*.jpg', etc.) work on the bash command line (only the values inside the single quotes are passed to rsync), but inside subprocess.call, the single quotes are passed to rsync as the file pattern ("'*.jpg'").
New (working) code looks like this:
def rsyncBookContent(bookIds, serverEnv):
bookPaths = []
for b in bookIds:
bookPaths.append(options.bookDestDir + "/book_" + str(b))
args = []
for host in serverEnv['content.hosts']:
# copy all *.jpg files via ssh
args = ["rsync", "-avz", "--include", "*/", "--include", "*.jpg", "--exclude", "*", "-e", "ssh"]
args.extend(bookPaths)
args.append("jill#" + host + ":/home/jill/web/public/static/"])
print "executing " + ' '.join(args)
subprocess.call(args)
Related
I'm trying to setup a system to run some commands on VM's in google cloud, in my case we want to run a tcpdump at a certain time using the 'at' command. Right now I'm just trying to execute any commands succesfully, when I have to pass arguments along with the command and getting confusing behaviour, which appears to be that the command, and the arguments are executed as a single long command instead of seperate arguements.
I first tried in bash, and thinking my issue was one of quoting, I moved to using python to hopefully make things easier to understand, but I appear to be hitting the same issue and figure I must be doing something wrong.
I have the following functions defined in python, and call them
def execute(cmd):
popen = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
for stdout_line in iter(popen.stdout.readline, ""):
yield stdout_line
popen.stdout.close()
return_code = popen.wait()
if return_code:
raise subprocess.CalledProcessError(return_code, cmd)
def runCapture(project, instance, zone, time, duration):
## Run capture against server
print ("Running capture against Project: " + project + ", Instance: " + instance + ", Zone: " + zone, "at: " + time, "for " + str(duration) + " minutes")
## First connect, schedule capture
## Connect again, schedule upload of capture at capture time + duration time + some overrun.
## gcloud compute ssh --project=${PROJECT} ${INSTANCE} --zone="${ZONE}" --command="...do stuff..." --tunnel-through-iap
## CMD=\${1:-"/usr/sbin/tcpdump -nn -i ens4 -G \$(( ${DURATION}*60 )) -W 1 -w ./\$(uname -n)-%Y-%m-%d_%H.%M.%S.pcap"}
total_time=str(duration*60)
command="/bin/bash -c 'echo \"hello world\"'"
for path in execute(["/usr/bin/gcloud", "compute", "ssh", instance, "--project="+project, "--zone="+zone, "--tunnel-through-iap", "--command=\""+command+"\"", ]):
print(path, end="")
The resulting errors are as follows:
bash: /bin/bash -c 'echo hello: No such file or directory
Traceback (most recent call last):
File "./ingressCapture.py", line 79, in <module>
results = runCapture(project, instance, zone, time, duration)
File "./ingressCapture.py", line 33, in runCapture
for path in execute(["/usr/bin/gcloud", "compute", "ssh", instance, "--project="+project, "--zone="+zone, "--tunnel-through-iap", "--command=\""+command+"\"", ]):
File "./ingressCapture.py", line 17, in execute
raise subprocess.CalledProcessError(return_code, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/gcloud', 'compute', 'ssh', 'tbtst-test3-app-egress-nztw', '--project=devops-tb-sandbox-250222', '--zone=europe-west1-b', '--tunnel-through-iap', '--command="/bin/bash -c \'echo "hello world"\'"']' returned non-zero exit status 127.
It appears to me, that instead of invoking the bash shell and running the echo command, it is instead invoking a command that includes the bash shell and then all the arguments too. I have a bash shell when I login normally via SSH, and can run the commands manually (and they work). Why are the arguments for the command from --command="....." getting called like this and how do I prevent it?
I'm pretty sure your problem is that you have too many quotes.
When you write --command="bash -c 'echo \"Hello World\"'" on the command line, the shell internally marks all the stuff inside the quotes as being in a quoted state and then removes the quotes. The actual argument that ends up going to the program is --command=bash -c 'echo "Hello World"' as a single string in argv (or your language's equivalent).
Try putting import sys ; print(sys.argv[1]) inside a small python script and calling it with ./test.py --command="bash -c 'echo \"Hello World\"'" to see for yourself.
However, in your arglist to subprocess, you're forming this string: --command="/bin/bash -c 'echo "hello world"'", presumably because you thought you needed to match what you'd normally type on the command line. You can see this in the stacktrace (minus the escaped single quotes, since that's syntax highlighting from python). Since python does not perform quote removal, those quotes are going through to the other side of your ssh connection where the login shell is attempting to reparse it as a shell command. The first "word" on the other end of the connection is /bin/bash -c 'echo hello because of those extra quotes so the shell attempts to find a command with that name on the path, and it clearly doesn't exist.
What you need to put into your arglist for subprocess is simply "--command="+command.
Python 3.10.6
Windows 10
I have a python function that executes a DXL script using subsystem.run() or os.system() (whichever works best I guess). The problem is that when I run a custom command using python it does not work, but when I paste the same command in the command prompt, it works. I should also clarify that command prompt is not the ms store windows terminal (cannot run ibm doors commands there for some reason). It is the OG prompt
I need to use both python and IBM Doors for the solution.
Here is a summer version of my code (Obviously, the access values are not real):
#staticmethod
def run_dxl_importRTF():
dquotes = chr(0x22) # ASCII --> "
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = dquotes + r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" + dquotes
file_name = "LIBC_String.rtf"
# Based On:
# "C:\Program Files\IBM\Rational\DOORS\9.7\\bin\doors.exe" -dxl "string pModuleName = \"%~1\";string pFilename = \"%~2\";#include <importRTF.dxl>" -f "%TEMP%" -b "doNothing.dxl" -d 11111#11.11.1111.0 -user USER -password PASSWORD
script_arguments = f"{dquotes}string pModuleName=\{dquotes}{module_name}\{dquotes};string pFileName=\{dquotes}{file_name}\{dquotes};#include <{script_path}>{dquotes}"
command = [doors_path, "-dxl", script_arguments, "-f", "%TEMP%", "-b", script_do_nothing_path, '-d', database_config, '-user', user, '-password', password]
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')
return
PYTHON SCRIPT OUTPUT:
COMMAND:
"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" -dxl "string pModuleName=\"TEST_TEMP\";string pFileName=\"LIBC_String.rtf\";#include <importRTF.dxl>" -f %TEMP% -b doNothing.dxl -d 11111#11.11.1111.0 -user USER_TEMP -password PASS_TEMP
STDERR: 'The system cannot find the path specified.\n'
STDOUT:
RETURN CODE: 1
When I run the same command in the command prompt, it works (dxl script is compiled).
I identified the problem which is the script_argument variable. Meaning that, when I try to just enter the IBM Doors server without compiling a DXL script, it works on python and the command prompt.
The python script needs to be dynamic meaning that all of the initial declared variables can change value and have a path string in it. I am also trying to avoid .bat files. They also did not work with dynamic path values
Thanks for your time
I tried:
Changing CurrentDirectory (cwd) to IBM Doors
os.system()
Multiple workarounds
Tried IBM Doors path without double quotes (it doesnt work because of the whitespaces)
.bat files
When calling subprocess.run with a command list and shell=True, python will expand the command list to a string, adding more quoting along the way. The details are OS dependent (on Windows, you always have to expand the list to a command) but you can see the result via the subprocess.list2cmdline() function.
Your problem is these extra escapes. Instead of using a list, build a shell command string that already contains the escaping you want. You can also use ' for quoting strings so that internal " needed for shell quoting can be entered literally.
Putting it all together (and likely messing something up here), you would get
#staticmethod
def run_dxl_importRTF():
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe"
file_name = "LIBC_String.rtf"
script_arguments = (rf'string pModuleName=\"{module_name}\";'
'string pFileName=\"{file_name}\";'
'#include <{script_path}>')
command = (f'"{doors_path}" -dxl "{script_arguments}" -f "%TEMP%"'
' -b "{script_do_nothing_path}" -d {database_config}'
' -user {user} -password {pass}')
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')
I am attempting to run PowerShell script from python to convert .xls files to .xlsb. by looping through a list of file names. I am encountering a PowerShell error "You cannot call a method on a null-valued expression" for command 3 (i.e. cmd3), and I am unsure why (this is my first time with python and running PowerShell script in general). The error is encountered when trying to open the workbook, but when the command is run in PowerShell directly, it seems to work fine.
Code:
import logging, os, shutil, itertools, time, pyxlsb, subprocess
# convert .xls to .xlsb and / transfer new terminology files
for i in itertools.islice(FileList, 0, 6, None):
# define extension
ext = '.xls'
# define file path
psPath = f'{downdir}' + f'\{i}'
# define ps scripts
def run(cmd):
completed = subprocess.run(["powershell", "-Command", cmd], capture_output=True)
return completed
# ps script: open workbook
cmd1 = "$xlExcel12 = 50"
cmd2 = "$Excel = New-Object -Com Excel.Application"
cmd3 = f"$WorkBook = $Excel.Workbooks.Open('{psPath}{ext}')"
cmd4 = f"$WorkBook.SaveAs('{psPath}{ext}',$xlExcel12,[Type]::Missing,
[Type]::Missing,$false,$false,2)"
cmd5 = "$Excel.Quit()"
# ps script: delete.xls files
cmd6 = f"Remove-Item '{psPath}{ext}'"
run(cmd1)
run(cmd2)
run(cmd3)
# change extension
ext = '.xlsb'
run(cmd4)
run(cmd5)
run(cmd6)
# copy .xlsb files to terminology folder
shutil.copy(i + ext, termdir)
Error:
Out[79]: CompletedProcess(args=['powershell', '-Command', "$WorkBook = > > $Excel.Workbooks.Open('C:\Users\Username\Downloads\SEND Terminology.xls')"], returncode=1, stdout=b'', stderr=b"You cannot call a method on a null-valued expression.\r\nAt line:1 char:1\r\n+ $WorkBook = $Excel.Workbooks.Open('C:\Username\User\Downloads\SEND Ter ...\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n + CategoryInfo : InvalidOperation: (:) [], RuntimeException\r\n + FullyQualifiedErrorId : InvokeMethodOnNull\r\n \r\n")
Any input would be helpful.
Thank you!
Problem
As commenter vonPryz correctly stated, the Powershell commands run in separate processes. The memory spaces of processes are isolated from each other, and will be cleared when a process ends.
When you run the commands in separate Powershell processes, the cmd3, cmd4 and cmd5 won't have the variable $Excel available. Powershell defaults to a $null value for non-existing variables, hence the error message "You cannot call a method on a null-valued expression". The same happens for variable $xlExcel12. These variables only exists as long as the processes that created them were running and would only be visible within these processes, even if you managed to create two processes in parallel.
Solution
Commands cmd1..5 need to be run in the same Powershell process, so each command will be able to "see" the variables created by previous commands:
run( cmd1 + ';' + cmd2 + ';' + cmd3 + ';' + cmd4 + ';' + cmd5 )
You will need to change cmd4 to use another variable for the extension that will be used for saving, e. g. extSave
cmd4 = f"$WorkBook.SaveAs('{psPath}{extSave}',$xlExcel12,[Type]::Missing,
[Type]::Missing,$false,$false,2)"
The cmd6 is completely independent, because it does not depend on Powershell variables. It only depends on python variables, which are resolved before the process starts, so it could still be run in a separate process.
I am trying to use subprocess.open to execute python function with parameters(start and end are all string format), but I'm always getting syntax error as
" SyntaxError: invalid syntax"
when I execute the python(python2.7 check.py) , getting the error as below ,
import check ; print check .check_output(06:34:00,06:36:00)
^
SyntaxError: invalid syntax
output:
my code check.py:
def alert(start,end):
print "start:"+str(start)
print "end :"+str(end)
start = "06:34:00"
end = "06:36:00"
cmd = "python2.7 -c 'import check ; print check.check_output('"+str(start)+"','"+str(end)+"')' >> /log/debug.log"
print cmd
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
(output, err) = p.communicate()
p_status = p.wait()
print "output:"+str(output)
I think the problem is my parameter contain colon(:) , How can I use colon in function parameter ? Any suggestions? Thanks for any help.
You're not adding quotes to your command so it gets badly interpreted. However, since it's unclear what your functions are and how you want to call them, here's a working example of what you're trying to achieve.
check.py
def check_output(*args):
print("CALLED: check_output({})".format(", ".join([repr(x) for x in args])))
It just holds one function and prints out the arguments it was called with. For example, if you were to call it from your shell as:
python -c "import check; check.check_output(\"foo\", \"bar\")"
It would print to STDOUT:
CALLED: check_output('foo', 'bar')
So now if you want to call it from another Python script, you can use the subprocess module to the same effect, as long as you properly escape your arguments. For example, if you had another test.py script with:
import subprocess
foo = 'Foo'
bar = 'Bar'
# using 'raw' strings so we don't need to double escape each backslash
cmd = r'python -c "import check; check.check_output(\"' + foo + r'\", \"' + bar + r'\")"'
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
out, err = proc.communicate()
print("OUTPUT: " + out)
and run it, you'll get:
OUTPUT: CALLED: check_output('Foo', 'Bar')
NOTE - when constructing your query string like this your variables must also be escaped like we escape the quotations in the example.
You can also let the subprocess module to do most of your escaping by formulating your command as:
cmd = [
'python',
'-c',
'import check; check.check_output("' + foo + '", "' + bar + '")'
]
And you can make it 'safer', or at least easier to manage using str.format():
cmd = [
'python',
'-c',
'import check; check.check_output("{}", "{}")'.format(foo, bar)
]
Then you don't have to worry about badly escaped characters.
I am having an issue executing remote processes with subprocess.call() and psexec. I am using the following syntax for executing the process remotely with subprocess.call():
def execute(hosts):
''' Using psexec, execute the script on the list of hosts '''
successes = []
wd = r'c:\\'
file = r'c:\\script.exe'
for host in hosts:
res = subprocess.call(shlex.split(r'psexec \\%s -e -s -d -w %s %s ' % (host,wd,file)), stdin=None, stdout=None, stderr=None)
if res == 0:
successes.append(host)
else:
logging.warning("Error executing script on host %s with error code %d" % (host, res))
print shlex.split(r'psexec \\%s -e -s -d -w %s %s ' % (hosts[0],wd,file))
return successes
As you can see, as part of my troubleshooting, I am printing the shlex.split() output to ensure that it is what I want. This print statement gives:
['psexec', '\\HOSTNAME', '-e', '-s', '-d', '-w', 'c:\\', 'c:\\script.exe']
Which is what I would expect. Unfortunately, when I run it, I get an error saying:
PsExec could not start \GN-WRK-02:
The system cannot find the file specified.
Directly after this, I run the psexec command with the exact syntax that the program should be running it with (judging by the shlex.split() output) and it works completely fine. My syntax is:
psexec \\HOSTNAME -e -s -d -w c:\\ c:\\script.exe
Any ideas why this wouldn't be working? If it matters the execute function is being called through multiprocessing's map() function on two or 3 host lists.
Any help would be great! Thanks!
Your \\ double slash in front of the hostname is just one slash; it is doubled to escape the slash.
You can see this in the shlex.split() output:
['psexec', '\\HOSTNAME', '-e, '-s', '-d', '-w', 'c:\\', 'c:\\script.exe']
note that \\ before the hostname is just the two backslashes, just like in the c:\\ filename value.. If you print just that value you see that the backslash at the sart is just one character:
>>> print '\\HOSTNAME'
\HOSTNAME
>>> '\\HOSTNAME'[0]
'\\'
>>> '\\HOSTNAME'[1]
'H'
That's because shlex.split() is a POSIX tool, not a Windows tool, and it interprets the \\ in the raw string as an escape too; if you are using that tool, double the slashes again:
shlex.split(r'psexec \\\\%s -e -s -d -w %s %s ' % (host,wd,file))
An alternative may be to disable POSIX mode, but I am not entirely certain how that would interplay with Windows:
shlex.split(r'psexec \\%s -e -s -d -w %s %s ' % (host,wd,file), posix=False)