I am trying to pass a python variable to a bash command like this:
subscriptionId = "xxxxx"
command = " az account show -s $subscriptionId"
subprocess.check_output(command)
I get there following error:
error : az account show: error: argument --subscription/-s: expected one argument
Assigning a Python variable like subscriptionId = "xxxxx" does not magically place it in your environment, much less pass it to a subprocess. You need to do that interpolation yourself:
command = f"az account show -s {subscriptionId}"
If you really want to use environment variables, add the variable you want and enable shell expansion:
subscriptionId = ...
env = os.environ.copy()
env['subscriptionId'] = subscriptionId
command = "az account show -s ${subscriptionId}"
subprocess.check_output(command, env=env, shell=True)
Alternatively, you can mess with your own process environment:
subscriptionId = ...
os.environ['subscriptionId'] = subscriptionId
command = "az account show -s ${subscriptionId}"
subprocess.check_output(command, shell=True)
These options are, in my opinion, not recommended, since they raise all the security issues that shell=True brings with it, while providing you with no real advantage.
since the variable command is just a string you could simply do this.
subscriptionId = "xxxxx"
command = " az account show -s " + subscriptionId
subprocess.check_output(command)
Related
Python 3.10.6
Windows 10
I have a python function that executes a DXL script using subsystem.run() or os.system() (whichever works best I guess). The problem is that when I run a custom command using python it does not work, but when I paste the same command in the command prompt, it works. I should also clarify that command prompt is not the ms store windows terminal (cannot run ibm doors commands there for some reason). It is the OG prompt
I need to use both python and IBM Doors for the solution.
Here is a summer version of my code (Obviously, the access values are not real):
#staticmethod
def run_dxl_importRTF():
dquotes = chr(0x22) # ASCII --> "
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = dquotes + r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" + dquotes
file_name = "LIBC_String.rtf"
# Based On:
# "C:\Program Files\IBM\Rational\DOORS\9.7\\bin\doors.exe" -dxl "string pModuleName = \"%~1\";string pFilename = \"%~2\";#include <importRTF.dxl>" -f "%TEMP%" -b "doNothing.dxl" -d 11111#11.11.1111.0 -user USER -password PASSWORD
script_arguments = f"{dquotes}string pModuleName=\{dquotes}{module_name}\{dquotes};string pFileName=\{dquotes}{file_name}\{dquotes};#include <{script_path}>{dquotes}"
command = [doors_path, "-dxl", script_arguments, "-f", "%TEMP%", "-b", script_do_nothing_path, '-d', database_config, '-user', user, '-password', password]
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')
return
PYTHON SCRIPT OUTPUT:
COMMAND:
"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" -dxl "string pModuleName=\"TEST_TEMP\";string pFileName=\"LIBC_String.rtf\";#include <importRTF.dxl>" -f %TEMP% -b doNothing.dxl -d 11111#11.11.1111.0 -user USER_TEMP -password PASS_TEMP
STDERR: 'The system cannot find the path specified.\n'
STDOUT:
RETURN CODE: 1
When I run the same command in the command prompt, it works (dxl script is compiled).
I identified the problem which is the script_argument variable. Meaning that, when I try to just enter the IBM Doors server without compiling a DXL script, it works on python and the command prompt.
The python script needs to be dynamic meaning that all of the initial declared variables can change value and have a path string in it. I am also trying to avoid .bat files. They also did not work with dynamic path values
Thanks for your time
I tried:
Changing CurrentDirectory (cwd) to IBM Doors
os.system()
Multiple workarounds
Tried IBM Doors path without double quotes (it doesnt work because of the whitespaces)
.bat files
When calling subprocess.run with a command list and shell=True, python will expand the command list to a string, adding more quoting along the way. The details are OS dependent (on Windows, you always have to expand the list to a command) but you can see the result via the subprocess.list2cmdline() function.
Your problem is these extra escapes. Instead of using a list, build a shell command string that already contains the escaping you want. You can also use ' for quoting strings so that internal " needed for shell quoting can be entered literally.
Putting it all together (and likely messing something up here), you would get
#staticmethod
def run_dxl_importRTF():
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe"
file_name = "LIBC_String.rtf"
script_arguments = (rf'string pModuleName=\"{module_name}\";'
'string pFileName=\"{file_name}\";'
'#include <{script_path}>')
command = (f'"{doors_path}" -dxl "{script_arguments}" -f "%TEMP%"'
' -b "{script_do_nothing_path}" -d {database_config}'
' -user {user} -password {pass}')
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')
I want to run a few lines in commandline while using input from a GUI
I checked google and what I understand is that it's best to use subprocess.
I'm having a hard time understanding how to input my cli code into subprocess.Popen()
I want to run the following CLI code using the function I created below:
SET PRESTO_PASSWORD=XXXXX
java -jar presto.jar --server https://address.to.server --catalog hive --debug --schema XXXX --user=XXXX --password --execute "SELECT * FROM TABLE;" --output-format CSV_HEADER > test.csv
echo done
I've created the code below but if i run it I only get the output:
output: b''
error: b''
(so I don't even see 'Done' as an output, even though I echo it at the end)
I'm sure that the CLI code is ok, I've ran it manually in CMD and I do get results.
What am I doing wrong? Below the function I use to execute the commands:
def calculate():
def subprocess_cmd(command):
process = subprocess.Popen(command,stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
stdout, stderr = process.communicate()
output = f'output: {stdout}'
output_error = f'error: {stderr}'
return output, output_error
def update_output(output_string):
output_sqlcode.configure(state='normal')
output_sqlcode.insert('end', f'{output_string}\n')
output_sqlcode.configure(state='disabled')
label_output.grid(column=0, row=6, columnspan=4, pady=0)
output_sqlcode.grid(column=0, row=7, columnspan=4, padx = 10, ipadx = 250, ipady=100,pady=10)
usr_name = input_username.get()
usr_pass = input_password.get()
sql_code = input_sqlcode.get()
file_name = 'test'
cli_code = f'SET PRESTO_PASSWORD={usr_pass}; java -jar presto.jar --server {server_address} --catalog hive --debug --schema {schema_name}--user={usr_name} --password --execute "{sql_code};" --output-format CSV_HEADER > {file_name}.csv; echo done'
output, error = subprocess_cmd(cli_code)
update_output(output)
update_output(error)
Update
I think I found the reason why it is not working. I was running the script in a folder which is in the cloud (onedrive). Given that it uses CMD, it gave errors. I moved the folder to a different location but all my locations are UNC drives:
\\xx.xxx.net\xxx\xx\x\xx\xx\07 Personal\xxxx\file.py
Given that all my locations are like that, I need to find a way to run CMD with a UNC location. I tried using Pushd and Popd but that still gave me the UNC location error (probably because i'm trying to run pushd while i'm in UNC working directory?)
cli_code = f'Pushd {unc_location};SET PRESTO_PASSWORD={usr_pass}; java -jar presto.jar --server {server_address} --catalog hive --debug --schema {schema_name}--user={usr_name} --password --execute "{sql_code};" --output-format CSV_HEADER > {file_name}.csv; popd; echo done'
I have the following small powershell script which creates a local account.
create_new_user.ps1 :
param([string]$username)
New-LocalUser -Name $username -Description "Test account" -NoPassword
I would like to pass $username variable from python script as follows;
script.py :
import subprocess, sys
def runscript():
username = "testaccount"
p = subprocess.Popen(['powershell',"-ExecutionPolicy","Unrestricted", './create_new_user.ps1', username],
stdout=sys.stdout)
p.communicate()
runscript()
When i run the script.py, it asks me to input Name for the account and then it creates the account but it does not take the variable from script.py
How can i solve this issue?
Thanks in advance
I am trying to execute following command:
result = subprocess.check_output("curl -o '/Users/user/Desktop/workbook.twb' -u xxx:yyy https://bitbucket.xyz.com/rest/api/1.0/projects/xxx/repos/xxx/raw/yyy/test_folder/test.twb", shell=True)
In the above command, I need to replace /Users/user/Desktop/workbook.twb with a string variable e.g. filePath and https://bitbucket.xyz.com/rest/api/1.0/projects/xxx/repos/xxx/raw/yyy/test_folder/test.twb with another variable e.g. repo_path..How can I achieve this?
I tried multiple ways but getting formatting errors in all of them.
Something like that:
param_a = "foo"
param_b = "bar"
query_url = "http://some.host/{a}/{b}.xml".format(a=param_a, b=param_b)
print(query_url) # To understand what's happening.
command = "curl -o '{output_file}' '{query_url}'".format(
output_file="/Users/me/foo-bar",
query_url=query_url
)
print(command)
result = subprocess.check_output(command, shell=True)
Make every step small. When in doubt, print intermediate values.
I'm calling a bash script which is exporting a few variables, i found a way to get those variables and it's working, once i'm trying to add to args to my bash script it's failing.
Here is part of my python script:
bash_script = "./testBash.sh"
script_execution = Popen(["bash", "-c", "trap 'env' exit; source \"$1\" > /dev/null 2>&1",
"_", bash_script], shell=False, stdout=PIPE)
err_code = script_execution.wait()
variables = script_execution.communicate()[0]
This is my sample Bash script:
export var1="test1"
export var2=$var1/test2
echo "this is firsr var: var1=$var1"
echo "this is the second var: var2=$var2"
Once i'm changing the bash_script = "./testBash.sh" to bash_script = "./testBash.sh test test"
I'm not getting back the exported variables from the bash script into the variables variable in the Python script.
The provided above is a sample, and of course my real scripts are much more bigger.
If you change bash_script = "./testBash.sh" to bash_script = "./testBash.sh test test" then the name of the bash_script changes to "./testBash.sh test test". The 'test test' is not interpreted as separate arguments.
Instead, add the extra arguments to the list being passed to Popen:
bash_script = "./testBash.sh"
script_execution = Popen(
["bash", "-c", "trap 'env' exit; source \"$1\" > /dev/null 2>&1",
"_", bash_script, 'test', 'test'], shell=False, stdout=PIPE)
Then err_code will be 0 (indicating success), instead of 1. It's not clear from your posted code however what you want to happen. The extra test arguments are ignored.
The extra argument are received by the bash script, however. If instead you put
export var1="$2"
in testBash.sh, then the variables (in the Python script) would contain
var1=test
You might also find it more convenient to use
import subprocess
import os
def source(script, update=True):
"""
http://pythonwise.blogspot.fr/2010/04/sourcing-shell-script.html (Miki Tebeka)
http://stackoverflow.com/a/20669683/190597 (unutbu)
"""
proc = subprocess.Popen(
". %s; env -0" % script, stdout=subprocess.PIPE, shell=True)
output = proc.communicate()[0]
env = dict((line.split("=", 1) for line in output.split('\x00') if line))
if update:
os.environ.update(env)
return env
bash_script = "./testBash.sh"
variables = source(bash_script)
print(variables)
which yields the environment variables
{ 'var1': 'test1', 'var2': 'test1/test2', ... }
in a dict.