Develop Reference

Command works on Command Prompt but it does not work when called with subprocess.run() or os.system() in python

Python 3.10.6
Windows 10
I have a python function that executes a DXL script using subsystem.run() or os.system() (whichever works best I guess). The problem is that when I run a custom command using python it does not work, but when I paste the same command in the command prompt, it works. I should also clarify that command prompt is not the ms store windows terminal (cannot run ibm doors commands there for some reason). It is the OG prompt
I need to use both python and IBM Doors for the solution.
Here is a summer version of my code (Obviously, the access values are not real):
#staticmethod
def run_dxl_importRTF():
dquotes = chr(0x22) # ASCII --> "
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = dquotes + r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" + dquotes
file_name = "LIBC_String.rtf"
# Based On:
# "C:\Program Files\IBM\Rational\DOORS\9.7\\bin\doors.exe" -dxl "string pModuleName = \"%~1\";string pFilename = \"%~2\";#include <importRTF.dxl>" -f "%TEMP%" -b "doNothing.dxl" -d 11111#11.11.1111.0 -user USER -password PASSWORD
script_arguments = f"{dquotes}string pModuleName=\{dquotes}{module_name}\{dquotes};string pFileName=\{dquotes}{file_name}\{dquotes};#include <{script_path}>{dquotes}"
command = [doors_path, "-dxl", script_arguments, "-f", "%TEMP%", "-b", script_do_nothing_path, '-d', database_config, '-user', user, '-password', password]
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')
return
PYTHON SCRIPT OUTPUT:
COMMAND:
"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" -dxl "string pModuleName=\"TEST_TEMP\";string pFileName=\"LIBC_String.rtf\";#include <importRTF.dxl>" -f %TEMP% -b doNothing.dxl -d 11111#11.11.1111.0 -user USER_TEMP -password PASS_TEMP
STDERR: 'The system cannot find the path specified.\n'
STDOUT:
RETURN CODE: 1
When I run the same command in the command prompt, it works (dxl script is compiled).
I identified the problem which is the script_argument variable. Meaning that, when I try to just enter the IBM Doors server without compiling a DXL script, it works on python and the command prompt.
The python script needs to be dynamic meaning that all of the initial declared variables can change value and have a path string in it. I am also trying to avoid .bat files. They also did not work with dynamic path values
Thanks for your time
I tried:
Changing CurrentDirectory (cwd) to IBM Doors
os.system()
Multiple workarounds
Tried IBM Doors path without double quotes (it doesnt work because of the whitespaces)
.bat files

When calling subprocess.run with a command list and shell=True, python will expand the command list to a string, adding more quoting along the way. The details are OS dependent (on Windows, you always have to expand the list to a command) but you can see the result via the subprocess.list2cmdline() function.
Your problem is these extra escapes. Instead of using a list, build a shell command string that already contains the escaping you want. You can also use ' for quoting strings so that internal " needed for shell quoting can be entered literally.
Putting it all together (and likely messing something up here), you would get
#staticmethod
def run_dxl_importRTF():
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe"
file_name = "LIBC_String.rtf"
script_arguments = (rf'string pModuleName=\"{module_name}\";'
'string pFileName=\"{file_name}\";'
'#include <{script_path}>')
command = (f'"{doors_path}" -dxl "{script_arguments}" -f "%TEMP%"'
' -b "{script_do_nothing_path}" -d {database_config}'
' -user {user} -password {pass}')
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')

understanding the input to subprocess.Popen()

I want to run a few lines in commandline while using input from a GUI
I checked google and what I understand is that it's best to use subprocess.
I'm having a hard time understanding how to input my cli code into subprocess.Popen()
I want to run the following CLI code using the function I created below:
SET PRESTO_PASSWORD=XXXXX
java -jar presto.jar --server https://address.to.server --catalog hive --debug --schema XXXX --user=XXXX --password --execute "SELECT * FROM TABLE;" --output-format CSV_HEADER > test.csv
echo done
I've created the code below but if i run it I only get the output:
output: b''
error: b''
(so I don't even see 'Done' as an output, even though I echo it at the end)
I'm sure that the CLI code is ok, I've ran it manually in CMD and I do get results.
What am I doing wrong? Below the function I use to execute the commands:
def calculate():
def subprocess_cmd(command):
process = subprocess.Popen(command,stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
stdout, stderr = process.communicate()
output = f'output: {stdout}'
output_error = f'error: {stderr}'
return output, output_error
def update_output(output_string):
output_sqlcode.configure(state='normal')
output_sqlcode.insert('end', f'{output_string}\n')
output_sqlcode.configure(state='disabled')
label_output.grid(column=0, row=6, columnspan=4, pady=0)
output_sqlcode.grid(column=0, row=7, columnspan=4, padx = 10, ipadx = 250, ipady=100,pady=10)
usr_name = input_username.get()
usr_pass = input_password.get()
sql_code = input_sqlcode.get()
file_name = 'test'
cli_code = f'SET PRESTO_PASSWORD={usr_pass}; java -jar presto.jar --server {server_address} --catalog hive --debug --schema {schema_name}--user={usr_name} --password --execute "{sql_code};" --output-format CSV_HEADER > {file_name}.csv; echo done'
output, error = subprocess_cmd(cli_code)
update_output(output)
update_output(error)
Update
I think I found the reason why it is not working. I was running the script in a folder which is in the cloud (onedrive). Given that it uses CMD, it gave errors. I moved the folder to a different location but all my locations are UNC drives:
\\xx.xxx.net\xxx\xx\x\xx\xx\07 Personal\xxxx\file.py
Given that all my locations are like that, I need to find a way to run CMD with a UNC location. I tried using Pushd and Popd but that still gave me the UNC location error (probably because i'm trying to run pushd while i'm in UNC working directory?)
cli_code = f'Pushd {unc_location};SET PRESTO_PASSWORD={usr_pass}; java -jar presto.jar --server {server_address} --catalog hive --debug --schema {schema_name}--user={usr_name} --password --execute "{sql_code};" --output-format CSV_HEADER > {file_name}.csv; popd; echo done'

Getting permission denied error when executing a command via subprocess.Popen but not in shell

I am trying to move a folder into another folder but am getting Permission Denied error when I try to perform this operation in a Python script vs. the move working successfully when I run it in bash or even in Python interactive mode.
cmd = ['sudo', 'mv', '/path1/dir', '/path2']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
if p.returncode != 0:
print(stderr)
I also tried adding shell=True.
p = subprocess.Popen(' '.join(cmd), shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
if p.returncode != 0:
print(stderr)
In both cases, I am getting the following error:
"mv: cannot move '/path1/dir' to '/path2/dir': Permission denied\n"
I invoke my script in the following manner:
sudo python script.py
I tried executing each command in shell as well as Python interactive mode and I don't get any errors. Any idea what is going on over here?

After wasting hours of time debugging as to what was going wrong, I finally figured out what was happening. I was creating /path1 and /path2 using tempfile. Here is a snippet of the code:
class UtilitiesTest(unittest.TestCase):
#staticmethod
def createTestFiles():
dir = tempfile.mkdtemp()
_, file = tempfile.mkstemp(dir=dir)
return dir, file
def test_MoveFileToAnotherLocation(self):
src_dir, src_file = UtilitiesTest.createTestFiles()
dest_dir, dest_file = UtilitiesTest.createTestFiles()
cmd = ['sudo', 'mv', src_dir, dest_dir]
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
if p.returncode != 0:
print(stderr)
Like zwer said in the comments, if I am running this script using sudo, I don't need to add sudo in my mv command. Because I kept getting permission denied errors, I kept thinking that sudo would fix my problem. The actual issue over here was when tempfile.mkstemp() is called, it returns an open file descriptor along with the file path. I didn't pay much attention to the first argument, so when I modified my createTestFiles() to below, everything started working.
#staticmethod
def createTestFiles():
dir = tempfile.mkdtemp()
fd, file = tempfile.mkstemp(dir=dir)
os.close(fd)
return dir, file

Switch user before executing a python script

I am trying to use su linux command in a python script along with getpass and subprocess python standard libraries to switch to a new user before executing my script. I am using this function:
import subprocess
import getpass
def ask_passwd():
pw = getpass.getpass(prompt='Please insert the passwd to use the auto api:') # prompt the user for a passwd
print pw
command = "su new_user"
p = subprocess.Popen(command.split(), stdin=subprocess.PIPE)
p.communicate(pw) # communicate the passwd
def main():
# code here
if __name__ == '__main__':
ask_passwd()
main()
When executing the script the main code works but not the su command. Here what I get back:
Please insert the passwd to use the auto api:
pass
su: must be run from a terminal
Any help? thanks

you may use subprocess.popen. There is an example
import subprocess
sudo_password = 'yourpassword'
command = '-i -u youruser \n id'
command = command.split()
cmd1 = subprocess.Popen(['echo', sudo_password], stdout=subprocess.PIPE)
cmd2 = subprocess.Popen(['sudo', '-S'] + command, stdin=cmd1.stdout, stdout=subprocess.PIPE)
output = cmd2.stdout.read()
print output

Script to capture everything on screen

So I have this python3 script that does a lot of automated testing for me, it takes roughly 20 minutes to run, and some user interaction is required. It also uses paramiko to ssh to a remote host for a separate test.
Eventually, I would like to hand this script over to the rest of my team however, it has one feature missing: evidence collection!
I need to capture everything that appears on the terminal to a file. I have been experimenting with the Linux command 'script'. However, I cannot find an automated method of starting script, and executing the script.
I have a command in /usr/bin/
script log_name;python3.5 /home/centos/scripts/test.py
When I run my command, it just stalls. Any help would be greatly appreciated!
Thanks :)

Is a redirection of the output to a file what you need ?
python3.5 /home/centos/scripts/test.py > output.log 2>&1
Or if you want to keep the output on the terminal AND save it into a file:
python3.5 /home/centos/scripts/test.py 2>&1 | tee output.log

I needed to do this, and ended up with a solution that combined pexpect and ttyrec.
ttyrec produces output files that can be played back with a few different player applications - I use TermTV and IPBT.
If memory serves, I had to use pexpect to launch ttyrec (as well as my test's other commands) because I was using Jenkins to schedule the execution of my test, and pexpect seemed to be the easiest way to get a working interactive shell in a Jenkins job.
In your situation you might be able to get away with using just ttyrec, and skip the pexpect step - try running ttyrec -e command as mentioned in the ttyrec docs.
Finally, on the topic of interactive shells, there's an alternative to pexpect named "empty" that I've had some success with too - see http://empty.sourceforge.net/. If you're running Ubuntu or Debian you can install empty with apt-get install empty-expect

I actually managed to do it in python3, took a lot of work, but here is the python solution:
def record_log(output):
try:
with open(LOG_RUN_OUTPUT, 'a') as file:
file.write(output)
except:
with open(LOG_RUN_OUTPUT, 'w') as file:
file.write(output)
def execute(cmd, store=True):
proc = Popen(cmd.encode("utf8"), shell=True, stdout=PIPE, stderr=PIPE)
output = "\n".join((out.decode()for out in proc.communicate()))
template = '''Command:\n====================\n%s\nResult:\n====================\n%s'''
output = template % (cmd, output)
print(output)
if store:
record_log(output)
return output
# SSH function
def ssh_connect(start_message, host_id, user_name, key, stage_commands):
print(start_message)
try:
ssh.connect(hostname=host_id, username=user_name, key_filename=key, timeout=120)
except:
print("Failed to connect to " + host_id)
for command in stage_commands:
try:
ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(command)
except:
input("Paused, because " + command + " failed to run.\n Please verify and press enter to continue.")
else:
template = '''Command:\n====================\n%s\nResult:\n====================\n%s'''
output = ssh_stderr.read() + ssh_stdout.read()
output = template % (command, output)
record_log(output)
print(output)