Develop Reference

Command works on Command Prompt but it does not work when called with subprocess.run() or os.system() in python

Python 3.10.6
Windows 10
I have a python function that executes a DXL script using subsystem.run() or os.system() (whichever works best I guess). The problem is that when I run a custom command using python it does not work, but when I paste the same command in the command prompt, it works. I should also clarify that command prompt is not the ms store windows terminal (cannot run ibm doors commands there for some reason). It is the OG prompt
I need to use both python and IBM Doors for the solution.
Here is a summer version of my code (Obviously, the access values are not real):
#staticmethod
def run_dxl_importRTF():
dquotes = chr(0x22) # ASCII --> "
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = dquotes + r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" + dquotes
file_name = "LIBC_String.rtf"
# Based On:
# "C:\Program Files\IBM\Rational\DOORS\9.7\\bin\doors.exe" -dxl "string pModuleName = \"%~1\";string pFilename = \"%~2\";#include <importRTF.dxl>" -f "%TEMP%" -b "doNothing.dxl" -d 11111#11.11.1111.0 -user USER -password PASSWORD
script_arguments = f"{dquotes}string pModuleName=\{dquotes}{module_name}\{dquotes};string pFileName=\{dquotes}{file_name}\{dquotes};#include <{script_path}>{dquotes}"
command = [doors_path, "-dxl", script_arguments, "-f", "%TEMP%", "-b", script_do_nothing_path, '-d', database_config, '-user', user, '-password', password]
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')
return
PYTHON SCRIPT OUTPUT:
COMMAND:
"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" -dxl "string pModuleName=\"TEST_TEMP\";string pFileName=\"LIBC_String.rtf\";#include <importRTF.dxl>" -f %TEMP% -b doNothing.dxl -d 11111#11.11.1111.0 -user USER_TEMP -password PASS_TEMP
STDERR: 'The system cannot find the path specified.\n'
STDOUT:
RETURN CODE: 1
When I run the same command in the command prompt, it works (dxl script is compiled).
I identified the problem which is the script_argument variable. Meaning that, when I try to just enter the IBM Doors server without compiling a DXL script, it works on python and the command prompt.
The python script needs to be dynamic meaning that all of the initial declared variables can change value and have a path string in it. I am also trying to avoid .bat files. They also did not work with dynamic path values
Thanks for your time
I tried:
Changing CurrentDirectory (cwd) to IBM Doors
os.system()
Multiple workarounds
Tried IBM Doors path without double quotes (it doesnt work because of the whitespaces)
.bat files

When calling subprocess.run with a command list and shell=True, python will expand the command list to a string, adding more quoting along the way. The details are OS dependent (on Windows, you always have to expand the list to a command) but you can see the result via the subprocess.list2cmdline() function.
Your problem is these extra escapes. Instead of using a list, build a shell command string that already contains the escaping you want. You can also use ' for quoting strings so that internal " needed for shell quoting can be entered literally.
Putting it all together (and likely messing something up here), you would get
#staticmethod
def run_dxl_importRTF():
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe"
file_name = "LIBC_String.rtf"
script_arguments = (rf'string pModuleName=\"{module_name}\";'
'string pFileName=\"{file_name}\";'
'#include <{script_path}>')
command = (f'"{doors_path}" -dxl "{script_arguments}" -f "%TEMP%"'
' -b "{script_do_nothing_path}" -d {database_config}'
' -user {user} -password {pass}')
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')

Writing Python Script from Batch Script is not working for one command

I'm trying to convert batch script into python script.
This is batch script, which is calling Klockwork exe on the project specified building it.
%KwPath%\Kwinject -o kwinjectmp.out msbuild %BaseProjPath%/CodingGuide.vcxproj /t:Rebuild /p:Configuration="Release" /p:Platform="x64" /p:CLToolExe=cl.exe /p:CLToolPath=%VSBinPath%
I have write equivalent python script for it.
args = KwPath + '\\Kwinject.exe sync -o ' + 'kwinjectmp.out' + 'msbuild ' + BaseProject + '\\' + ProjectFolder + '\\' + ProjectName + '/t:Rebuild /p:Configuration="Release" /p:Platform="x64" /p:CLToolExe=cl.exe /p:CLToolPath=' + VSBinPath
print(args)
subprocess.call(args, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
Where I have declared BaseProject, VSBinPath, KwPath correctly. But the exectuion is not happening as it is happening in BatchScript, Basically script is not giving any output/working.

spaces between arguments may be absent because of your usage. try this:
import os
path1 = os.path.join(KwPath, 'Kwinject.exe')
path2 = os.path.join(BaseProject, ProjectFolder, ProjectName)
subprocess.call([
path1,
'sync',
'-o', 'kwinjectmp.out',
'msbuild',
path2,
'/t:Rebuild',
'/p:Configuration="Release"',
'/p:Platform="x64"',
'/p:CLToolExe=cl.exe',
'/p:CLToolPath=' + VSBinPath
],
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT
)

Calling R script from Python does not save log file in version 4

I am calling from python an R script that is very simple called R Code.R:
args <- commandArgs(TRUE)
print(args)
source(args[1])
setwd("<YOUR PATH>")
output <- head(mtcars, n = n)
write.table(output, "output.txt")
using the following script:
import subprocess
pth = "<YOUR PATH>"
subprocess.call(" ".join(["C:/R/R-3.6.0/bin/x64/R.exe", "-f", '"' + pth + '/R Code.R"', "--args",
'"' + pth + '/arguments.txt"',"1>", '"' + pth + '/log.txt"', "2>&1"]))
subprocess.call(" ".join(["C:/R/R-4.0.3/bin/x64/R.exe", "-f", '"' + pth + '/R Code.R"', "--args",
'"' + pth + '/arguments.txt"',"1>", '"' + pth + '/log.txt"', "2>&1"]))
where arguments.txt contain:
n <- 10
The problem is that when I am using R-4.0.3 the log.txt file is not generating and I need to dump a log file because it is automatically looking for it in a posterior process I have.
When I am executing in CMD (Windows) the following command:
C:/R/R-4.0.3/bin/x64/R.exe -f "<YOUR PATH>/R Code.R" --args "<YOUR PATH>/arguments.txt" 1> "<YOUR PATH>/log.txt" 2>&1'
It does work perfectly, it is only when embedded in another software.
Also, I have tried without white space in the name and calling the scripts from root folder without having to specify the path.
Any idea of why it doesn't work for R-4.* or even better, how to solve it?
Thank you!
PD: Thank you, Martin, for your tips and for making me formulate a better question

Rhelp people got this solved, thank you, Duncan Murdoch!
Solution 1:
import os
pth = "<YOUR PATH>"
os.system(" ".join(["C:/R/R-4.0.3/bin/x64/R.exe", "-f", '"' + pth + '/RCode.R"', "--args",
'"' + pth + '/arguments.txt"',"1>", '"' + pth + '/log.txt"']))
Solution 2:
import subprocess
pth = "<YOUR PATH>"
subprocess.call(" ".join(["1>", '"' + pth + '/log.txt"', "2>&1",
"C:/R/R-4.0.3/bin/x64/R.exe", "-f", '"' + pth + '/RCode.R"', "--args",
'"' + pth + '/arguments.txt"']), shell = True)

well, in one case (3.6.0) you use R.exe , in the other (4.0.3) Rscript.exe.
Both R and Rscript have existed for a long time, and they have always had slightly different behavior.
You really should not confuse them with each other (even though, on Windows, I see, they look like the same file .. they will not behave the same).
Ok, now you use R.exe for both.
Just to find out more / see more where the problem may happen, can you try
all of
using a minimal reproducible example, i.e. one we can use directly ourselves, i.e., not using "<YOUR PATH>" (nor setwd(.))
not using file names with a ' ' (space), i.e., e.g., use code.R
calling this from a "terminal"/shell instead of as python subprocess ?
Last but not least: Yes, for R 4.0.0, a completely updated toolset ("brandnew toolchain", e.g. much newer C compiler) was used to build R for windows, "Rtools 4.0" or rtools40: https://cran.r-project.org/bin/windows/Rtools/ . So changes are expected but should typically only have been to the better, not the worse ..

Running vulture from a python script

I'm trying to find a way to run vulture (which finds unused code in python projects) inside a python script.
vulture documentation can be found here:
https://pypi.org/project/vulture/
Does anyone know how to do it?
The only way I know to use vulture is by shell commands.
I tried to tun the shell commands from the script, using module subprocess, something like this:
process = subprocess.run(['vulture', '.'], check=True,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,universal_newlines=True)
which I though would have the same effect as running the shell command "vulture ."
but it doesn't work.
Can anyone help?
Thanks

Vulture dev here.
The Vulture package exposes an API, called scavenge - which it uses internally for running the analysis after parsing command line arguments (here in vulture.main).
It takes in a list of Python files/directories. For each directory, Vulture analyzes all contained *.py files.
To analyze the current directory:
import vulture
v = vulture.Vulture()
v.scavenge(['.'])
If you just want to print the results to stdout, you can call:
v.report()
However, it's also possible to perform custom analysis/filters over Vulture's results. The method vulture.get_unused_code returns a list of vulture.Item objects - which hold the name, type and location of unused code.
For the sake of this answer, I'm just gonna print the name of all unused objects:
for item in v.get_unused_code():
print(item.name)
For more info, see - https://github.com/jendrikseipp/vulture

I see you want to capture the output shown at console:
Below code might help:
import tempfile
import subprocess
def run_command(args):
with tempfile.TemporaryFile() as t:
try:
out = subprocess.check_output(args,shell=True, stderr=t)
t.seek(0)
console_output = '--- Provided Command: --- ' + '\n' + args + '\n' + t.read() + out + '\n'
return_code = 0
except subprocess.CalledProcessError as e:
t.seek(0)
console_output = '--- Provided Command: --- ' + '\n' + args + '\n' + t.read() + e.output + '\n'
return_code = e.returncode
return return_code, console_output
Your expected output will be displayed in console_output
Link:
https://docs.python.org/3/library/subprocess.html

error "sh: 1: read: Illegal option -d" when using here-document in system command run from Python

I'm running a Bash system command from within Python and have encountered this problem when using read to define a here-document in the command:
import os
text = "~|||-:this is text:-|||~"
fileName = "sound.wav"
command =\
"IFS= read -d \'\' text <<EOF\n" +\
text + "\n" +\
"EOF\n" +\
"echo \"${text}\" | text2wave -scale 1 -o " + fileName
os.system(command)
Could you help me figure out how to fix this?
Here is a slightly simplified version:
import os
text = "~|||-:this is text:-|||~"
command =\
"IFS= read -d \'\' text <<EOF\n" +\
text + "\n" +\
"EOF\n" +\
"echo \"${text}\""
os.system(command)
I wanted to make clear in the one above that I'd be using pipes. When I run this, I get the following error:
sh: 1: read: Illegal option -d

There's no reason to do all that in shell. Python can write directly to the standard input of the process that will run text2wave. Here's an example using the subprocess module.
p = subprocess.Popen(["text2wave", "-scale", "1", "-o", filename], stdin=subprocess.PIPE)
p.stdin.write(text + "\n")
p.wait()

A more pythonic way to do this would be using subprocess, either by specifying stdin to check_call or by using Popen.communicate.
Also, -d might not be a valid option in your version of read. What does help [r]ead tell you?