I am trying to execute following command:
result = subprocess.check_output("curl -o '/Users/user/Desktop/workbook.twb' -u xxx:yyy https://bitbucket.xyz.com/rest/api/1.0/projects/xxx/repos/xxx/raw/yyy/test_folder/test.twb", shell=True)
In the above command, I need to replace /Users/user/Desktop/workbook.twb with a string variable e.g. filePath and https://bitbucket.xyz.com/rest/api/1.0/projects/xxx/repos/xxx/raw/yyy/test_folder/test.twb with another variable e.g. repo_path..How can I achieve this?
I tried multiple ways but getting formatting errors in all of them.
Something like that:
param_a = "foo"
param_b = "bar"
query_url = "http://some.host/{a}/{b}.xml".format(a=param_a, b=param_b)
print(query_url) # To understand what's happening.
command = "curl -o '{output_file}' '{query_url}'".format(
output_file="/Users/me/foo-bar",
query_url=query_url
)
print(command)
result = subprocess.check_output(command, shell=True)
Make every step small. When in doubt, print intermediate values.
Related
This question already has answers here:
Passing variables to a subprocess call [duplicate]
(6 answers)
Closed 1 year ago.
I have a python script with a webhost, username and a password variable which i am trying to pass into a shell command in the same python script.
output = subprocess.check_output(['curl -s -G -u username:password -k \"https://webhost/something/something\"'], shell=True, encoding='utf-8')
Can you please help me how i can do this? I have tried multiple things, but none worked.
Thanks
Don't construct a string for the shell to parse; just provide a list. That list can directly contain string-valued variables (or strings constructed from variables).
username = ...
password = ...
url = ...
output = subprocess.check_output([
'curl',
'-s',
'-G',
'-u',
f'{username}:{password}',
'-k',
url
], encoding='utf-8')
Try this,
username = 'abc'
password = 'def'
webhost = '1.2.3.4'
output = subprocess.check_output([f'curl -s -G -u {username}:{password} -k \"https://{webhost}/something/something\"'], shell=True, encoding='utf-8')
It's called an f string. https://docs.python.org/3/tutorial/inputoutput.html
You add an f before the string starts, the you enclose the variables you want to insert in curly braces.
You can pass variables into strings using this syntax.
You can also pass the variables as a list as follow, each argument in the command is a separate item and you can use the f string on the list items you want to parse like this,
username = 'abc'
password = 'def'
webhost = '1.2.3.4'
output = subprocess.check_output(['curl',
'-s',
'-G',
'-u',
f'{username}:{password}',
'-k',
f'\"https://{webhost}/something/something\"'],
encoding = 'utf-8')
Let's say I have this snippet
list_command = 'mongo --host {host} --port {port} ' \
'--username {username} --password {password} --authenticationDatabase {database} < {path}'
def shell_exec(cmd: str):
import subprocess
p = subprocess.call(cmd, shell=True)
return p
Let's say these are the commands I'm trying to run on mongo
use users
show collections
db.base.find().pretty()
If format the string list_command with the appropriate values and pass it to the function with shell=True, it works fine. But I'm trying to avoid it for security purposes.
If I call it with shell=False, I get the following error:
2020-08-31T14:08:49.291+0100 E QUERY [thread1] SyntaxError: missing ; before statement #./mongo/user-01-09-2020:1:4
failed to load: ./mongo/user-01-09-2020
253
Your list_command is a shell command: in particular, it includes input redirection (via < {path}), which is a syntactic feature of the shell. To use it you need shell=True.
If you don’t want to use shell=True, you need to change the way you construct the argument (separate arguments need to be passed as separate items of a list rather than as a single string), and you need to pass the script into the standard input via an explicit pipe, by setting its input parameter:
cmd = ['mongo', '--host', '{host}', '--port', …]
subprocess.run(cmd, input=mongodb_script)
Using input raised the following error: TypeError: init() got an unexpected keyword argument 'input'.
I ended up doing the following:
import subprocess
def shell_exec(cmd: str, stdin=None):
with open(stdin, 'rb') as f:
return subprocess.call(cmd.split(), stdin=f)
I am trying to pass a python variable to a bash command like this:
subscriptionId = "xxxxx"
command = " az account show -s $subscriptionId"
subprocess.check_output(command)
I get there following error:
error : az account show: error: argument --subscription/-s: expected one argument
Assigning a Python variable like subscriptionId = "xxxxx" does not magically place it in your environment, much less pass it to a subprocess. You need to do that interpolation yourself:
command = f"az account show -s {subscriptionId}"
If you really want to use environment variables, add the variable you want and enable shell expansion:
subscriptionId = ...
env = os.environ.copy()
env['subscriptionId'] = subscriptionId
command = "az account show -s ${subscriptionId}"
subprocess.check_output(command, env=env, shell=True)
Alternatively, you can mess with your own process environment:
subscriptionId = ...
os.environ['subscriptionId'] = subscriptionId
command = "az account show -s ${subscriptionId}"
subprocess.check_output(command, shell=True)
These options are, in my opinion, not recommended, since they raise all the security issues that shell=True brings with it, while providing you with no real advantage.
since the variable command is just a string you could simply do this.
subscriptionId = "xxxxx"
command = " az account show -s " + subscriptionId
subprocess.check_output(command)
I am trying to use subprocess.open to execute python function with parameters(start and end are all string format), but I'm always getting syntax error as
" SyntaxError: invalid syntax"
when I execute the python(python2.7 check.py) , getting the error as below ,
import check ; print check .check_output(06:34:00,06:36:00)
^
SyntaxError: invalid syntax
output:
my code check.py:
def alert(start,end):
print "start:"+str(start)
print "end :"+str(end)
start = "06:34:00"
end = "06:36:00"
cmd = "python2.7 -c 'import check ; print check.check_output('"+str(start)+"','"+str(end)+"')' >> /log/debug.log"
print cmd
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
(output, err) = p.communicate()
p_status = p.wait()
print "output:"+str(output)
I think the problem is my parameter contain colon(:) , How can I use colon in function parameter ? Any suggestions? Thanks for any help.
You're not adding quotes to your command so it gets badly interpreted. However, since it's unclear what your functions are and how you want to call them, here's a working example of what you're trying to achieve.
check.py
def check_output(*args):
print("CALLED: check_output({})".format(", ".join([repr(x) for x in args])))
It just holds one function and prints out the arguments it was called with. For example, if you were to call it from your shell as:
python -c "import check; check.check_output(\"foo\", \"bar\")"
It would print to STDOUT:
CALLED: check_output('foo', 'bar')
So now if you want to call it from another Python script, you can use the subprocess module to the same effect, as long as you properly escape your arguments. For example, if you had another test.py script with:
import subprocess
foo = 'Foo'
bar = 'Bar'
# using 'raw' strings so we don't need to double escape each backslash
cmd = r'python -c "import check; check.check_output(\"' + foo + r'\", \"' + bar + r'\")"'
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
out, err = proc.communicate()
print("OUTPUT: " + out)
and run it, you'll get:
OUTPUT: CALLED: check_output('Foo', 'Bar')
NOTE - when constructing your query string like this your variables must also be escaped like we escape the quotations in the example.
You can also let the subprocess module to do most of your escaping by formulating your command as:
cmd = [
'python',
'-c',
'import check; check.check_output("' + foo + '", "' + bar + '")'
]
And you can make it 'safer', or at least easier to manage using str.format():
cmd = [
'python',
'-c',
'import check; check.check_output("{}", "{}")'.format(foo, bar)
]
Then you don't have to worry about badly escaped characters.
I can't seem to get the 'mv' command to work from Python subprocess.Popen with a wildcard.
The code:
def moveFilesByType(source, destination, extension):
params = []
params.append("mv")
params.append(source + "/*." + extension)
params.append(destination + "/")
print params
pipe = subprocess.Popen(params, shell=True, stdout=PIPE)
result, err = pipe.communicate()
return result
The output from print params:
['mv', '/full_path_to_folder_source/*.nib', '/full_path_to_folder_target/']
The paths here are shortened just for readability, but I assure that they are valid. Calling this exact same command from a terminal works but calling in python gives the standard message about improper use of mv:
usage: mv [-f | -i | -n] [-v] source target
mv [-f | -i | -n] [-v] source ... directory
I read that in order for wildcards to work, I would need the parameter shell=True in the Popen call, which is present. Any ideas why this doesn't work? Removing shell=True ends up treating the asterisks as hard literals as expected.
Use a string instead of an array:
params = "mv /full_path_to_folder_source/*.nib /full_path_to_folder_target/"
When you specify arguments via the array form, the argument '/full_path_to_folder_source/*.nib' is passed to mv. You want to force bash to expand the argument, but Popen won't pass each argument through the shell.
You can do it without starting a new process using modules shutil and glob:
import glob
import shutil
def moveFilesByType(source, destination, extension):
for path in glob.glob(source + "/*." + extension):
shutil.move(path, destination)
You shouldn't need to use subprocess for this, check out shutil.copytree