I have a shell script I need to call from a Python script. Only thing is that is needs to pass 7 parameters, that are basically variables, and some of those contain spaces.
I see a lot of references to os.system, and to subprocess, but what about the parameters with spaces? I think those will cause an issue.
Edit 2:
subject = "Proposal"
to = "LandonStatis#stackoverflow.com"
from = "Joshua#stackoverflow.com"
REPORT_DIRECTORY = "/home/reports"
file = "/home/files/proposal.txt"
message = "Please accept me as the best answer"
#i didnt include the "flag" because i dont know the flag format
os.system('./home/scripts/send_email.pl "'+subject+'" "'+to+'" "'+from+'" "'+REPORT_DIRECTORY+'" "'+file+'" "'+message+'"')
Edit:
If you want to add a variable, you can do so like this
os.system('ls -al '+str(x))
For example, if your username is "landon statis" with a space,
you can do this: os.system('ls -al "/home/landon statis/Desktop"')
(notice the space in the command above?)
This works for both Windows and Linux.
Use double quotes to prevent word splitting. An argument enclosed in
double quotes presents itself as a single word, even if it contains
whitespace separators.
This is the source, which is a link to Linux Documentation Project. Check them out, it's very useful: Advanced Bash Scripting Guide: Chapter 5 - Quoting
Please do not use os.system or subprocess with shell=True. Both execute a shell and are potentially vulnerable to shell injections. See https://docs.python.org/3/library/subprocess.html#security-considerations for more details.
Best way is to use subprocess and hand it a list of arguments. Example: Let's just use a Shell script that print its arguments. myscript:
#!/bin/sh
for arg in "$#"; do
echo "argument: $arg"
done
Then you can call this script using a list as argument:
subprocess.run(["./myscript", "first", "argument with space", "third"])
Output:
argument: first
argument: argument with space
argument: third
Related
I need to execute the following in bash, but called from a Python script:
<command> --searchBody="{\"query":{\"range":{\"#timestamp\":{\"gte\": \"2020-10-16T00:00:00\",\"lte\": \"2020-10-17T00:00:00\"}}}}"
The double quotes must be escaped except for the ones that bound the --searchBody section
I have the following code in Python
execution = cmd+ ' --searchBody="{\\"query\":{\\"range\":{\\"#timestamp\\":{\\"gte\\": \\"'+startQuery+'\\",\\"lte\\": \\"'+endQuery+'\\"}}}}"'
print(execution)
os.system(execution)
cmd is the rest of the command already pre-populated, startQuery and endQuery are some date strings
The print statement prints the command exactly as it needs to, but when the os.system is run all the backslashes are removed from what is sent to the CLI.
I have tried all manor of escaping with multiple quotes but cannot get it to work. Any ideas?
Thanks
It's better to use subprocess module.
Try something like:
arg = '--searchBody="{\\"query\":{\\"range\":{\\"#timestamp\\":{\\"gte\\": \\"'+startQuery+'\\",\\"lte\\": \\"'+endQuery+'\\"}}}}"'
cmd = 'foobar'
subprocess.call([cmd, arg])
You should carefully read info about shell parameter and shlex module:
https://docs.python.org/3/library/subprocess.html#security-considerations
Using os.system() is like "I don't care about security, and all this crap, just run it"
This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
How to pass all arguments passed to my Bash script to a function of mine? [duplicate]
(7 answers)
Closed 3 years ago.
The problem:
I'm writing a program that performs actions in python based on links, and possibly expanding it to do things beyond that. This program is meant to be quickly used through bash. So, I'm using a custom function to do that.
function youtube() {python3 <youtube program path here> $1}
As for the python file; I'm using sys, os, and re in order to make it function. sys, in order to use both sys.exit() and var = sys.argv[<argNum>], the former in order to exit the program using custom exceptions, like error.searchError() or error.usageError(), and the later for actualling using the arguments from the command itself. os is just for os.system('{}'.format(<your command here>)). And re is for removing the spaces from the second argument, where my problem lies, and replacing them with '+', as per query = re.sub(' ', '+', query).
Now, as for the problem itself. As I mentioned before, the problem lies with the second bash argument, or sys.argv[2]. With sys.argv[0] being the python file, and sys.argv[1] being the option, in this case being -s.
sys.argv[2] is meant to be the actual youtube search query. But, according to whenever I use the command with all three arguments, youtube -s Hi this is a test., I get the following output, as per the custom error I made: No search query provided!. This only happens when python excepts an IndexError, which means that the program is not receiving the second argument from bash or zsh. What is actually supposed to happen, when the second arguments does exist, is:
os.system('open https://www.youtube.com/results?search_query=Hi+this+is+a+test.')
Which opens that link in my default browser. I have tried to add $2 to the custom function, and various ways of entering the second argument through the python source itself, including using a x = input('Search Query: ') statement. But that isn't optimal for what I'm doing.
The code:
The following is the source code for all the nonsense I just typed out.
The custom function:
function youtube() {python3 <python program path here> $1}
For those that have no idea what this means (i.e.; people that don't know much (or anything) about bash coding); The function method creates a bash object, in this case, youtube(). As for the code in the brackets ({}), this uses the function python3, which just pushes the program in argument 0 to the python 3.x interpreter, to open <python program path here>, which is a placeholder for the actual path of the program. As for $1, this is a variable that always equals the text inputted after the function name.
The custom errors:
class error:
def usageError():
usageError = '''Usage: youtube [-s] [<search_query>]
Help: Performs actions related to https://www.youtube.com
Options:
-s Opens https://www.youtube.com/results?search_query=<your query here>'''
print(usageError)
sys.exit()
def searchError():
searchError = 'No search query provided!'
print(searchError)
sys.exit()
Is this irrelevant? I'm not sure, but I'm putting it in anyway! Now, if you don't understand it, the following should explain it.
The error class contains all of the customs errors for this program, ok? Ok, so you get that, but what do these functions do? usageError is raised when argument 1 simply doesn't exist, and prints the usage information to the terminal. Then sys.exit()s the program, basically the equivalent of hitting Alt+f4 in video game. searchError, on the other hand, only happens if argument 2 doesn't exist, meaning there is no search query. It then tells you that you're stupid, and will need to actually enter your query for it to work.
Well, maybe not that exactly, but you get the point.
The Juicy Bits:
option = ''
try: option = sys.argv[1];
except IndexError: raise error.usageError()
if option == '-s':
try:
query = sys.argv[2]
query = re.sub(' ', '+', query)
os.system('open https://www.youtube.com/results?search_query={}'.format(query))
except IndexError: raise error.searchError();
Just to explain; First, the program creates the variable option and then sets it to an empty string. Then, it tries to set option to argument 1, or the option. If argument 1 doesn't exist, it raises the error error.usageError, as explained in The Custom Errors. After that, the program tries to create the variable query, and set it to argument 2, then replace all of the spaces in query with '+' signs. If all of that succeeds to happen, it then loads up the youtube search in your default browser. If not, it raises the error error.searchError().
The Edits
Edit 1. The error was in The Custom Function. Where I should have had an $#, I had an $1. As Jeshurun Roach explains in his answer, $1 only holds the argument 1, and no other arguments. While $# contains all variables.
function youtube() {python3 <python program path here> $#}
$1 refers to the first argument passed into the function. in bash, spaces delimit arguments. so in your example youtube -s Hi this is a test.,
$1 is -s,
$2 is Hi,
$3 is this etc...
What you're looking for is the $# symbol. This value stands for all the arguments.
But just plugging in $# instead of $1 won't fix all your problems. in your python script, each argument will be broken up again by spaces, just like the bash function.
To fix this, you can put quotes around the text after the flag like so: youtube -s 'Hi this is a test.'.
If you call your program like this: youtube -s something cool, then sys.argv[2] is going to be "something".
I'd suggest wrapping your query in quotes. For example youtube -s "something cool".
I have a python script that I'd like to supply with an argument (usually) containing wildcards, referring to a series of files that I'd like to do stuff with. Example here:
#!/usr/bin/env python
import argparse
import glob
parser = argparse.ArgumentParser()
parser.add_argument('-i', action="store", dest="i")
results = parser.parse_args()
print 'argument i is: ', results.i
list_of_matched_files = glob.glob(results.i)
In this case, everything works great if the user adds quotes to the passed argument like so:
./test_script.py -i "foo*.txt"
...but often times the users forget to add quotes to the argument and are stumped when the list only contains the first match because UNIX already expanded the list and argparse only then gets the first list element.
Is there a way (within the script) to prevent UNIX from expanding the list before passing it to python? Or maybe even just to test if the argument doesn't contain quotes and then warn the user?
No. Wildcards are expanded by the shell (Bash, zsh, csh, fish, whatever) before the script even runs, and the script can't do anything about them. Testing whether the argument contains quotes also won't work, as the shell similarly strips the quotes from "foo*.txt" before passing the argument to the script, so all Python sees is foo*.txt.
Its not UNIX that is doing the expansion, it is the shell.
Bash has an option set -o noglob (or -f) which turns off globbing (filename expansion), but that is non-standard.
If you give an end-user access to the command-line then they really should know about quoting. For example, the commonly used find command has a -name parameter which can take glob constructs but they have to be quoted in a similar manner. Your program is no different to any other.
If users can't handle that then maybe you should give them a different interface. You could go to the extreme of writing a GUI or a web/HTML front-end, but that's probably over the top.
Or why not prompt for the filename pattern? You could, for example, use a -p option to indicate prompting, e.g:
import argparse
import glob
parser = argparse.ArgumentParser()
parser.add_argument('-i', action="store", dest="i")
parser.add_argument('-p', action="store_true", default=False)
results = parser.parse_args()
if results.p:
pattern = raw_input("Enter filename pattern: ")
else:
pattern = results.i
list_of_matched_files = glob.glob(pattern)
print list_of_matched_files
(I have assumed Python 2 because of your print statement)
Here the input is not read by the shell but by python, which will not expand glob constructs unless you ask it to.
You can disable the expansion using set -f from the command line. (re-enable with set +f).
As jwodder correctly says though, this happens before the script is run, so the only way I can think of to do this is to wrap it with a shell script that disables expansion temporarily, runs the python script, and re-enables expansion. Preventing UNIX from expanding the list before passing it to python is not possible.
Here is an example for the Bash shell that shows what #Tom Wyllie is talking about:
alias sea='set -f; search_function'
search_function() { perl /home/scripts/search.pl $# ; set +f; }
This defines an alias called "sea" that:
Turns off expansion ("set -f")
Runs the search_function function which is a perl script
Turns expansion back on ("set +f")
The problem with this is that if a user stops execution with ^C or some such then the expansion may not be turned back on leaving the user puzzling why "ls *" is not working. So I'm not necessarily advocating using this. :).
This worked for me:
files = sys.argv[1:]
Even though only one string is on the command line, the shell expands the wildcards and fills sys.argv[] with the list.
Scratching my head... this curl command will work fine from the command line when I copy it from here and paste it in my Windows 7 command line, but I can't get it to execute in my Python 2.7.9 script. Says the system cannot find the specified file. Popen using 'ping' or something like that works just fine, so I'm sure this is a goober typo that I'm just not seeing. I would appreciate a separate set of eyes and any comments as to what is wrong.
proc = subprocess.Popen("curl --ntlm -u : --upload-file c:\\temp\\test.xlsx http://site.domain.com/sites/site/SiteDirectory/folder/test.xlsx")
Have a look at second two paragraphs of the subprocess.Popen documentation if you haven't already:
args should be a sequence of program arguments or else a single string. By default, the program to execute is the first item in args if args is a sequence. If args is a string, the interpretation is platform-dependent and described below. See the shell and executable arguments for additional differences from the default behavior. Unless otherwise stated, it is recommended to pass args as a sequence.
On Unix, if args is a string, the string is interpreted as the name or path of the program to execute. However, this can only be done if not passing arguments to the program. [emphasis mine]
Instead you should pass in a list in which each argument to the program (including the executable name itself) is given as a separate item in the list. This is generally going to be safer in a cross-platform context anyways.
Update: I see now that you're using Windows in which case the advice on UNIX doesn't apply. On Windows though things are even more hairy. The best advice remains to use a list :)
Update 2: Another possible issue (and in fact the OP's issue as reported in the comments on this answer) is that because the full path to the curl executable was not given, it may not be found if the Python interpreter is running in an environment with a different PATH environment variable.
I am a bit confused as to how to get this done.
What I need to do is call an external command, from within a Python script, that takes as input several arguments, and a file name.
Let's call the executable that I am calling "prog", the input file "file", so the command line (in Bash terminal) looks like this:
$ prog --{arg1} {arg2} < {file}
In the above {arg1} is a string, and {arg2} is an integer.
If I use the following:
#!/usr/bin/python
import subprocess as sbp
sbp.call(["prog","--{arg1}","{arg2}","<","{file}"])
The result is an error output from "prog", where it claims that the input is missing {arg2}
The following produces an interesting error:
#!/usr/bin/python
import subprocess as sbp
sbp.call(["prog","--{arg1} {arg2} < {file}"])
all the spaces seem to have been removed from the second string, and equal sign appended at the very end:
command not found --{arg1}{arg2}<{file}=
None of this behavior seems to make any sense to me, and there isn't much that one can go by from the Python man pages found online. Please note that replacing sbp.call with sbp.Popen does not fix the problem.
The issue is that < {file} isn’t actually an argument to the program, but is syntax for the shell to set up redirection. You can tell Python to use the shell, or you can setup the redirection yourself.
from subprocess import *
# have shell interpret redirection
check_call('wc -l < /etc/hosts', shell=True)
# set up redirection in Python
with open('/etc/hosts', 'r') as f:
check_call(['wc', '-l'], stdin=f.fileno())
The advantage of the first method is that it’s faster and easier to type. There are a lot of disadvantages, though: it’s potentially slower since you’re launching a shell; it’s potentially non-portable because it depends on the operating system shell’s syntax; and it can easily break when there are spaces or other special characters in filenames.
So the second method is preferred.