Develop Reference

Command works on Command Prompt but it does not work when called with subprocess.run() or os.system() in python

Python 3.10.6
Windows 10
I have a python function that executes a DXL script using subsystem.run() or os.system() (whichever works best I guess). The problem is that when I run a custom command using python it does not work, but when I paste the same command in the command prompt, it works. I should also clarify that command prompt is not the ms store windows terminal (cannot run ibm doors commands there for some reason). It is the OG prompt
I need to use both python and IBM Doors for the solution.
Here is a summer version of my code (Obviously, the access values are not real):
#staticmethod
def run_dxl_importRTF():
dquotes = chr(0x22) # ASCII --> "
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = dquotes + r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" + dquotes
file_name = "LIBC_String.rtf"
# Based On:
# "C:\Program Files\IBM\Rational\DOORS\9.7\\bin\doors.exe" -dxl "string pModuleName = \"%~1\";string pFilename = \"%~2\";#include <importRTF.dxl>" -f "%TEMP%" -b "doNothing.dxl" -d 11111#11.11.1111.0 -user USER -password PASSWORD
script_arguments = f"{dquotes}string pModuleName=\{dquotes}{module_name}\{dquotes};string pFileName=\{dquotes}{file_name}\{dquotes};#include <{script_path}>{dquotes}"
command = [doors_path, "-dxl", script_arguments, "-f", "%TEMP%", "-b", script_do_nothing_path, '-d', database_config, '-user', user, '-password', password]
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')
return
PYTHON SCRIPT OUTPUT:
COMMAND:
"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" -dxl "string pModuleName=\"TEST_TEMP\";string pFileName=\"LIBC_String.rtf\";#include <importRTF.dxl>" -f %TEMP% -b doNothing.dxl -d 11111#11.11.1111.0 -user USER_TEMP -password PASS_TEMP
STDERR: 'The system cannot find the path specified.\n'
STDOUT:
RETURN CODE: 1
When I run the same command in the command prompt, it works (dxl script is compiled).
I identified the problem which is the script_argument variable. Meaning that, when I try to just enter the IBM Doors server without compiling a DXL script, it works on python and the command prompt.
The python script needs to be dynamic meaning that all of the initial declared variables can change value and have a path string in it. I am also trying to avoid .bat files. They also did not work with dynamic path values
Thanks for your time
I tried:
Changing CurrentDirectory (cwd) to IBM Doors
os.system()
Multiple workarounds
Tried IBM Doors path without double quotes (it doesnt work because of the whitespaces)
.bat files

When calling subprocess.run with a command list and shell=True, python will expand the command list to a string, adding more quoting along the way. The details are OS dependent (on Windows, you always have to expand the list to a command) but you can see the result via the subprocess.list2cmdline() function.
Your problem is these extra escapes. Instead of using a list, build a shell command string that already contains the escaping you want. You can also use ' for quoting strings so that internal " needed for shell quoting can be entered literally.
Putting it all together (and likely messing something up here), you would get
#staticmethod
def run_dxl_importRTF():
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe"
file_name = "LIBC_String.rtf"
script_arguments = (rf'string pModuleName=\"{module_name}\";'
'string pFileName=\"{file_name}\";'
'#include <{script_path}>')
command = (f'"{doors_path}" -dxl "{script_arguments}" -f "%TEMP%"'
' -b "{script_do_nothing_path}" -d {database_config}'
' -user {user} -password {pass}')
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')

Running a C executable inside a python program

I have written a C code where I have converted one file format to another file format. To run my C code, I have taken one command line argument : filestem.
I executed that code using : ./executable_file filestem > outputfile
Where I have got my desired output inside outputfile
Now I want to take that executable and run within a python code.
I am trying like :
import subprocess
import sys
filestem = sys.argv[1];
subprocess.run(['/home/dev/executable_file', filestem , 'outputfile'])
But it is unable to create the outputfile. I think some thing should be added to solve the > issue. But unable to figure out. Please help.

subprocess.run has optional stdout argument, you might give it file handle, so in your case something like
import subprocess
import sys
filestem = sys.argv[1]
with open('outputfile','wb') as f:
subprocess.run(['/home/dev/executable_file', filestem],stdout=f)
should work. I do not have ability to test it so please run it and write if it does work as intended

You have several options:
NOTE - Tested in CentOS 7, using Python 2.7
1. Try pexpect:
"""Usage: executable_file argument ("ex. stack.py -lh")"""
import pexpect
filestem = sys.argv[1]
# Using ls -lh >> outputfile as an example
cmd = "ls {0} >> outputfile".format(filestem)
command_output, exitstatus = pexpect.run("/usr/bin/bash -c '{0}'".format(cmd), withexitstatus=True)
if exitstatus == 0:
print(command_output)
else:
print("Houston, we've had a problem.")
2. Run subprocess with shell=true (Not recommended):
"""Usage: executable_file argument ("ex. stack.py -lh")"""
import sys
import subprocess
filestem = sys.argv[1]
# Using ls -lh >> outputfile as an example
cmd = "ls {0} >> outputfile".format(filestem)
result = subprocess.check_output(shlex.split(cmd), shell=True) # or subprocess.call(cmd, shell=True)
print(result)
It works, but python.org frowns upon this, due to the chance of a shell injection: see "Security Considerations" in the subprocess documentation.
3. If you must use subprocess, run each command separately and take the SDTOUT of the previous command and pipe it into the STDIN of the next command:
p = subprocess.Popen(cmd, stdin=PIPE, stdout=PIPE)
stdout_data, stderr_data = p.communicate()
p = subprocess.Popen(cmd, stdin=stdout_data, stdout=PIPE)
etc...
Good luck with your code!

Python 2.7 keep env variables from a subprocess

I'm calling a bash script which is exporting a few variables, i found a way to get those variables and it's working, once i'm trying to add to args to my bash script it's failing.
Here is part of my python script:
bash_script = "./testBash.sh"
script_execution = Popen(["bash", "-c", "trap 'env' exit; source \"$1\" > /dev/null 2>&1",
"_", bash_script], shell=False, stdout=PIPE)
err_code = script_execution.wait()
variables = script_execution.communicate()[0]
This is my sample Bash script:
export var1="test1"
export var2=$var1/test2
echo "this is firsr var: var1=$var1"
echo "this is the second var: var2=$var2"
Once i'm changing the bash_script = "./testBash.sh" to bash_script = "./testBash.sh test test"
I'm not getting back the exported variables from the bash script into the variables variable in the Python script.
The provided above is a sample, and of course my real scripts are much more bigger.

If you change bash_script = "./testBash.sh" to bash_script = "./testBash.sh test test" then the name of the bash_script changes to "./testBash.sh test test". The 'test test' is not interpreted as separate arguments.
Instead, add the extra arguments to the list being passed to Popen:
bash_script = "./testBash.sh"
script_execution = Popen(
["bash", "-c", "trap 'env' exit; source \"$1\" > /dev/null 2>&1",
"_", bash_script, 'test', 'test'], shell=False, stdout=PIPE)
Then err_code will be 0 (indicating success), instead of 1. It's not clear from your posted code however what you want to happen. The extra test arguments are ignored.
The extra argument are received by the bash script, however. If instead you put
export var1="$2"
in testBash.sh, then the variables (in the Python script) would contain
var1=test
You might also find it more convenient to use
import subprocess
import os
def source(script, update=True):
"""
http://pythonwise.blogspot.fr/2010/04/sourcing-shell-script.html (Miki Tebeka)
http://stackoverflow.com/a/20669683/190597 (unutbu)
"""
proc = subprocess.Popen(
". %s; env -0" % script, stdout=subprocess.PIPE, shell=True)
output = proc.communicate()[0]
env = dict((line.split("=", 1) for line in output.split('\x00') if line))
if update:
os.environ.update(env)
return env
bash_script = "./testBash.sh"
variables = source(bash_script)
print(variables)
which yields the environment variables
{ 'var1': 'test1', 'var2': 'test1/test2', ... }
in a dict.

subprocess.call using cygwin instead of cmd on Windows

I'm programming on Windows 7 and in one of my Python projects I need to call bedtools, which only works with Cygwin on Windows. I'm new to Cygwin, installed the default version + everything needed for bedtools and then used Cygwin to install bedtools by using make as described in the installation instructions.
$ tar -zxvf BEDTools.tar.gz
$ cd BEDTools-<version>
$ make
When I use the Cygwin terminal to call it manually like below, it works without problem and the output file contains the correct result.
bedtools_exe_path intersect -a gene_bed_file -b snp_bed_file -wa -wb > output_file
But when I use subprocess.call in my program it seems to use Windows cmd instead of Cygwin, which doesn't work.
arguments = [bedtools_exe_path, 'intersect', '-a', gene_bed_file, '-b',
snp_bed_file, '-wa', '-wb', '>', output_file]
return_code = suprocess.call(arguments)
Results in no output file and a return code of 3221225781.
arguments = [bedtools_exe_path, 'intersect', '-a', gene_bed_file, '-b',
snp_bed_file, '-wa', '-wb', '>', output_file]
return_code = suprocess.call(arguments, shell=True)
Results in an empty output file and a return code of 3221225781.
cygwin_bash_path = 'D:/Cygwin/bin/bash.exe'
arguments = [cygwin_bash_path, bedtools_exe_path, 'intersect', '-a', gene_bed_file, '-b',
snp_bed_file, '-wa', '-wb', '>', output_file]
return_code = suprocess.call(arguments)
Results in no output file, a return code of 126 and
D:/BEDTools/bin/bedtools.exe: D:/BEDTools/bin/bedtools.exe: cannot execute binary file
arguments = [cygwin_bash_path, bedtools_exe_path, 'intersect', '-a', gene_bed_file, '-b',
snp_bed_file, '-wa', '-wb', '>', output_file]
return_code = suprocess.call(arguments, shell=True)
Results in an empty output file, a return code of 126 and
D:/BEDTools/bin/bedtools.exe: D:/BEDTools/bin/bedtools.exe: cannot execute binary file
Any ideas how I can get it to work?

Imagine you want to run a Linux command from Windows. You could install Linux into a VM and run commands via ssh (Putty/plink on Windows):
#!/usr/bin/env python
import subprocess
cmd = [r'C:\path\to\plink.exe', '-ssh', 'user#vm_host', '/path/to/bedtools']
with open('output', 'wb', 0) as file:
subprocess.check_call(cmd, stdout=file)
Cygwin provides run command that allows to run commands directly:
cmd = [r'C:\cygwin\path\to\run.exe', '-p', '/path/to/', 'bedtools',
'-wait', 'arg1', 'arg2']
Note: Python script is run from Windows in both cases. bedtools is Linux or Cygwin (non-Windows) command here and therefore you should provide POSIX paths.

The following works without a problem. The " does not need to be escaped.
argument = 'sh -c \"' + bedtools_exe_path + ' intersect -a ' + gene_bed_file +
' -b ' + snp_bed_file + ' -wa -wb\"'
with open(output_file, 'w') as file:
subprocess.call(argument, stdout=file)
Using the following works as well:
argument = 'bash -c \"' + bedtools_exe_path + ' intersect -a ' + gene_bed_file +
' -b ' + snp_bed_file + ' -wa -wb\"'
with open(output_file, 'w') as file:
subprocess.call(argument, stdout=file)
With:
bedtools_exe_path = 'D:/BEDTools/bin/bedtools.exe'
gene_bed_file = 'output/gene.csv'
snp_bed_file = 'output/snps.csv'
output_file = 'output/intersect_gene_snp.bed'
Using the path to the cygwin bash.exe (D:/Cygwin/bin/bash.exe) instead of bash or sh does not work.
Thank you, eryksun, Padraic Cunningham and J.F. Sebastian.

subprocess.call

I am new to the subprocess.call function and I have tried different combinations of the same call but it is not working.
I am trying to execute the following command:
cmd = 'sort -k1,1 -k4,4n -k5,5n '+outpath+fnametempout+' > '+outpath+fnameout
print cmd
If I try the call I get an error:
cmd = cmd.split(" ")
print cmd
subprocess.call(cmd)
the error I get is:
sort: stat failed: >: No such file or directory

Doing it this way, you need shell=True to allow the shell redirection to work.
subprocess.call('sort -k1,1 -k4,4n -k5,5n '+outpath+fnametempout,shell=True)
A better way is:
with open(outpath+fnameout,'w') as fout: #context manager is OK since `call` blocks :)
subprocess.call(cmd,stdout=fout)
which avoids spawning a shell all-together and is safe from shell injection type attacks. Here, cmd is a list as in your original, e.g.
cmd = 'sort -k1,1 -k4,4n -k5,5n '+outpath+fnametempout
cmd = cmd.split()
It should also be stated that python has really nice sorting facilities and so I doubt that it is actually necessary to pass the job off to sort via a subprocess.
Finally, rather than using str.split to split the arguments, from a string, it's probably better to use shlex.split as that will properly handle quoted strings.
>>> import shlex
>>> cmd = "foo -b -c 'arg in quotes'"
>>> print cmd.split()
['foo', '-b', '-c', "'arg", 'in', "quotes'"]
>>> print shlex.split(cmd)
['foo', '-b', '-c', 'arg in quotes']

it is not to compecated execute above command in python:
import subprocess
import sys
proc = subprocess.Popen(['sort','-k1','1', '-k4','4n', '-k5','5n', '+outpath+fnametempout+', '>', '+outpath+fnameout'],stdin=subprocess.PIPE)
proc.communicate()

example:
subprocess.call(['ps','aux'])
lines=subprocess.check_output(['ls','-al'])
line_list = lines.split('\n')
or
handle = subprocess.Popen('ls',stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,close_fds=True)
handle.stdout.read()