Develop Reference

escape ampersand & in string when send as argument python

I have written two python scripts A.py and B.py So B.py gets called in A.py like this:
config_object = {}
with open(config_filename) as data:
config_object = json.load(data, object_pairs_hook=OrderedDict)
command = './scripts/B.py --config-file={} --token-a={} --token-b={}'.format(promote_config_filename, config_object['username'], config_object['password'])
os.system(command)
In here config_object['password'] contains & in it. Say it is something like this S01S0lQb1T3&BRn2^Qt3
Now when this value get passed to B.py it gets password as S01S0lQb1T3 So after & whatever it is getting ignored.
How to solve this?

os.system runs a shell. You can escape arbitrary strings for the shell with shlex.quote() ... but a much superior solution is to use subprocess instead, like the os.system documentation also recommends.
subprocess.run(
['./scripts/B.py',
'--config-file={}'.format(promote_config_filename),
'--token-a={}'.format(config_object['username']),
'--token-b={}'.format(config_object['password'])])
Because there is no shell=True, the strings are now passed to the subprocess verbatim.
Perhaps see also Actual meaning of shell=True in subprocess

#tripleee has good suggestions. In terms of why this is happening, if you are running Linux/Unix at least, the & would start a background process. You can search "linux job control" for more info on that. The shortest (but not best) solution is to wrap your special characters in single or double quotes in the final command.
See this bash for a simple example:
$ echo foo&bar
[1] 20054
foo
Command 'bar' not found, but can be installed with:
sudo apt install bar
[1]+ Done echo foo
$ echo "foo&bar"
foo&bar

Python to call shell script with arguments

I have a shell script I need to call from a Python script. Only thing is that is needs to pass 7 parameters, that are basically variables, and some of those contain spaces.
I see a lot of references to os.system, and to subprocess, but what about the parameters with spaces? I think those will cause an issue.

Edit 2:
subject = "Proposal"
to = "LandonStatis#stackoverflow.com"
from = "Joshua#stackoverflow.com"
REPORT_DIRECTORY = "/home/reports"
file = "/home/files/proposal.txt"
message = "Please accept me as the best answer"
#i didnt include the "flag" because i dont know the flag format
os.system('./home/scripts/send_email.pl "'+subject+'" "'+to+'" "'+from+'" "'+REPORT_DIRECTORY+'" "'+file+'" "'+message+'"')
Edit:
If you want to add a variable, you can do so like this
os.system('ls -al '+str(x))
For example, if your username is "landon statis" with a space,
you can do this: os.system('ls -al "/home/landon statis/Desktop"')
(notice the space in the command above?)
This works for both Windows and Linux.
Use double quotes to prevent word splitting. An argument enclosed in
double quotes presents itself as a single word, even if it contains
whitespace separators.
This is the source, which is a link to Linux Documentation Project. Check them out, it's very useful: Advanced Bash Scripting Guide: Chapter 5 - Quoting

Please do not use os.system or subprocess with shell=True. Both execute a shell and are potentially vulnerable to shell injections. See https://docs.python.org/3/library/subprocess.html#security-considerations for more details.
Best way is to use subprocess and hand it a list of arguments. Example: Let's just use a Shell script that print its arguments. myscript:
#!/bin/sh
for arg in "$#"; do
echo "argument: $arg"
done
Then you can call this script using a list as argument:
subprocess.run(["./myscript", "first", "argument with space", "third"])
Output:
argument: first
argument: argument with space
argument: third

Bash script will not run with subprocess in Python

For some reason, no matter how many variations I've tried, I can't seem to execute a bash script I've written. The command words 100% fine in Terminal, but when I try calling it with a subprocess, it returns nothing.
from os import listdir
import subprocess
computer_name = 'homedirectoryname'
moviefolder = '/Users/{}/Documents/Programming/Voicer/Movies'.format(computer_name)
string = 'The lion king'
for i in listdir(moviefolder):
title = i.split('.')
formatted_title = title[0].replace(' ', '\ ')
if string.lower() == title[0].lower():
command = 'vlc {}/{}.{}'.format(moviefolder, formatted_title, title[1])
subprocess.call(["/usr/local/bin",'-i','-c', command], stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)
else:
continue
The bash executable file looks like this:
#/bin/bash
func() {
open -a /Applications/VLC.app/Contents/MacOS/VLC $1
}
Where have I gone wrong?

You should call open directly:
import os
import subprocess
computer_name = 'homedirectoryname'
moviefolder = '/Users/{}/Documents/Programming/Voicer/Movies'.format(computer_name)
string = 'The lion king'
for filename in os.listdir(moviefolder):
title = filename.split('.')
if string.lower() == title[0].lower():
subprocess.call(['open', '-a', '/Applications/VLC.app/Contents/MacOS/VLC', os.path.join(moviefolder, filename)])

Since you are using shell=True, the command must be a string:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself. (docs)

Like you even mentioned in a comment, you get /usr/local/bin: is a directory when you properly capture the error from the shell (and take out the erroneous shell=True; or correspondingly refactor the command line to be suitable for this usage, i.e. pass a string instead of a list).
Just to spell this out, you are attempting to run the command /usr/local/bin with some options; but of course, it's not a valid command; so this fails.
The actual script you seem to want to run will declare a function and then exit, which results in the function's definition being lost again, because the subprocess which ran the shell in which this function declaration was executed has now terminated and released all its resources back to the system.
Perhaps you should take more than just a few steps back and explain what you actually want to accomplish; but really, that should be a new, separate question.
Assuming you are actually trying to run vlc, and guessing some other things, too, perhaps you actually want
subprocess.call(['vlc','{}/{}.{}'.format(moviefolder, formatted_title, title[1]),
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
If your PATH is correct, you should not need to specify /usr/local/bin/ explicitly (and if your PATH is wrong, correct it in the code before, instead of hardcoding a directory for the executable you want to call).

/usr/local/bin is a directory. You can't run a directory as if it were a command.
Anyhow, there's no point to having /usr/local/bin anywhere in your command at all. Leave out the shell=True, and explicitly call vlc:
subprocess.call([
'vlc',
'{}/{}.{}'.format(moviefolder, formatted_title, title[1])
])

When shell=True is used in subprocess.call, if the command arguments is a sequence, then the first element of the sequence needs to be the command, and the rest are treated as argument(s) to the shell itself.
So, this should do:
subprocess.call(["/usr/local/bin/{}".format(command), '-i','-c'], shell=True, ...)
Otherwise, you can make the command a string.
Example:
In [20]: subprocess.call(["cat spamegg", "-i", "-c"], shell=True)
foobar

Python Subprocess Error in using "cp"

I was trying to use subprocess calls to perform a copy operation (code below):
import subprocess
pr1 = subprocess.call(['cp','-r','./testdir1/*','./testdir2/'], shell = True)
and I got an error saying:
cp: missing file operand
Try `cp --help' for more information.
When I try with shell=False , I get
cp: cannot stat `./testdir1/*': No such file or directory
How do I get around this problem?
I'm using RedHat Linux GNOME Deskop version 2.16.0 and bash shell and Python 2.6
P.S. I read the question posted in Problems with issuing cp command with Popen in Python, and it suggested using shell = True option, which is not working for me as I mentioned :(

When using shell=True, pass a string, not a list to subprocess.call:
subprocess.call('cp -r ./testdir1/* ./testdir2/', shell=True)
The docs say:
On Unix with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself.
So (on Unix), when a list is passed to subprocess.Popen (or subprocess.call), the first element of the list is interpreted as the command, all the other elements in the list are interpreted as arguments for the shell. Since in your case you do not need to pass arguments to the shell, you can just pass a string as the first argument.

This is an old thread now, but I was just having the same problem.
The problem you were having with this call:
subprocess.call(['cp','-r','./testdir1/*','./testdir2/'], shell = False)
was that each of the parameters after the first one are quoted. So to the shell sees the command like this:
cp '-r' './testdir1/*' './testdir2/'
The problem with that is the wildcard character (*). The filesystem looks for a file with the literal name '*' in the testdir1 directory, which of course, is not there.
The solution is to make the call like the selected answer using the shell = True option and none of the parameters quoted.

I know that the option of shell=True may be tempting but it's always inadvisable due to security issues. Instead, you can use a combination of the subprocess and glob modules.
For Python 3.5 or higher:
import subprocess
import glob
subprocess.run(['cp', '-r'] + glob.glob('./testdir1/*') + ['./testdir2/'])
For Python 3.4 or lower:
import subprocess
import glob
subprocess.call(['cp', '-r'] + glob.glob('./testdir1/*') + ['./testdir2/'])

How to escape ssh quoted shell commands in Python

I'm trying to write a function that will issue commands via ssh with Popen and return the output.
def remote(cmd):
escaped = escape(cmd)
return subprocess.Popen(escaped, ...).communicate()[0]
My trouble is how to implement the escape function. Is there a Python module (2.6) that has helpers for this? Google shows there's pipes.quote and re.escape but they seem like they only work for locally run commands. With commands passed to ssh, it seems the escaping needs to be more stringent:
For example on a local machine, this is ok:
echo $(hostname)
When passing to ssh it has to be:
ssh server "echo \$(hostname)"
Also, double quotes can be interpreted in different ways depending on the context. For literal quotes, you need the following:
ssh a4ipe511 "echo \\\"hello\\\""
To do variable exansion, double quotes are also used:
ssh a4ipe511 "echo \"\$(hostname)\""
As you can see, the rules for escaping a command that goes into SSH can get pretty complicated. Since this function will be called by anyone, I'm afraid some complex commands will cause the function to return incorrect output.
Is this something that can be solved with a built-in Python module or do I need to implement the escaping myself?

First:
pipes.quote and re.escape have nothing to do with locally run commands. They simply transform a string; what you do with it after that is your business. So either -- in particular pipes.quote -- is suitable for what you want.
Second:
If you want to run the command echo $(hostname) on a remote host using ssh, you don't need to worry about shell escaping, because subprocess.Popen does not pass your commands into a shell by default. So, for example, this works just fine:
>>> import subprocess
>>> subprocess.call([ 'ssh', 'localhost', 'echo $(hostname)'])
myhost.example.com
0
Double quotes also work as you would expect:
>>> subprocess.call([ 'ssh', 'localhost', 'echo "one two"; echo three'])
one two
three
0
It's not clear to me that you actually have a problem.