Develop Reference

Command works on Command Prompt but it does not work when called with subprocess.run() or os.system() in python

Python 3.10.6
Windows 10
I have a python function that executes a DXL script using subsystem.run() or os.system() (whichever works best I guess). The problem is that when I run a custom command using python it does not work, but when I paste the same command in the command prompt, it works. I should also clarify that command prompt is not the ms store windows terminal (cannot run ibm doors commands there for some reason). It is the OG prompt
I need to use both python and IBM Doors for the solution.
Here is a summer version of my code (Obviously, the access values are not real):
#staticmethod
def run_dxl_importRTF():
dquotes = chr(0x22) # ASCII --> "
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = dquotes + r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" + dquotes
file_name = "LIBC_String.rtf"
# Based On:
# "C:\Program Files\IBM\Rational\DOORS\9.7\\bin\doors.exe" -dxl "string pModuleName = \"%~1\";string pFilename = \"%~2\";#include <importRTF.dxl>" -f "%TEMP%" -b "doNothing.dxl" -d 11111#11.11.1111.0 -user USER -password PASSWORD
script_arguments = f"{dquotes}string pModuleName=\{dquotes}{module_name}\{dquotes};string pFileName=\{dquotes}{file_name}\{dquotes};#include <{script_path}>{dquotes}"
command = [doors_path, "-dxl", script_arguments, "-f", "%TEMP%", "-b", script_do_nothing_path, '-d', database_config, '-user', user, '-password', password]
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')
return
PYTHON SCRIPT OUTPUT:
COMMAND:
"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" -dxl "string pModuleName=\"TEST_TEMP\";string pFileName=\"LIBC_String.rtf\";#include <importRTF.dxl>" -f %TEMP% -b doNothing.dxl -d 11111#11.11.1111.0 -user USER_TEMP -password PASS_TEMP
STDERR: 'The system cannot find the path specified.\n'
STDOUT:
RETURN CODE: 1
When I run the same command in the command prompt, it works (dxl script is compiled).
I identified the problem which is the script_argument variable. Meaning that, when I try to just enter the IBM Doors server without compiling a DXL script, it works on python and the command prompt.
The python script needs to be dynamic meaning that all of the initial declared variables can change value and have a path string in it. I am also trying to avoid .bat files. They also did not work with dynamic path values
Thanks for your time
I tried:
Changing CurrentDirectory (cwd) to IBM Doors
os.system()
Multiple workarounds
Tried IBM Doors path without double quotes (it doesnt work because of the whitespaces)
.bat files

When calling subprocess.run with a command list and shell=True, python will expand the command list to a string, adding more quoting along the way. The details are OS dependent (on Windows, you always have to expand the list to a command) but you can see the result via the subprocess.list2cmdline() function.
Your problem is these extra escapes. Instead of using a list, build a shell command string that already contains the escaping you want. You can also use ' for quoting strings so that internal " needed for shell quoting can be entered literally.
Putting it all together (and likely messing something up here), you would get
#staticmethod
def run_dxl_importRTF():
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe"
file_name = "LIBC_String.rtf"
script_arguments = (rf'string pModuleName=\"{module_name}\";'
'string pFileName=\"{file_name}\";'
'#include <{script_path}>')
command = (f'"{doors_path}" -dxl "{script_arguments}" -f "%TEMP%"'
' -b "{script_do_nothing_path}" -d {database_config}'
' -user {user} -password {pass}')
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')

How can I respond to a prompt in os.system?

I'm trying to do something like this:
from subprocess import Popen
p = Popen(["vagrant", "ssh", "vmname", "-c", '"pvcreate -ff /dev/sdb"'])
But it requires user input. Also, that didn't work anyway. They give the error: bash: pvcreate -ff /dev/sdb: command not found, because it's looking for a program pvcreate -ff /dev/sdb, instead of pvcreate with arguments. I also tried this first:
p = Popen(["vagrant", "ssh", "vmname", "-c", "pvcreate", "-ff", "/dev/sdb"])
And have resorted to using this:
os.system("vagrant ssh vmname -c 'pvcreate -ff /dev/sdb'")
However I need to say yes when it prompts me. I've already tried these options as well:
os.system("yes | vagrant ssh vmname -c 'pvcreate -ff /dev/sdb'")
os.system("echo y | vagrant ssh vmname -c 'pvcreate -ff /dev/sdb'")
Is it possible to respond to a prompt using os.system?

I'd suggest using the list form of invocation.
import subprocess
command = ["vagrant", "ssh", "vmname", "-c", "pvcreate -ff /db/sdb"]
output,error = subprocess.Popen(
command, universal_newlines=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE).communicate()
This represents the set of parameters that are going to be passed and eliminates the need to mess around with shell quoting.

create logical volume using python cgi

I am creating python CGI script that accepts lv size from client and then creates and mount the logical volume using nfs.
Here is my code:
#!/usr/bin/python
print "Content-type:text/html"
print ""
import cgi,commands,os,socket,time,getpass
form = cgi.FieldStorage()
st=form.getvalue("st")
mount=form.getvalue('mount')
backup=form.getvalue('backup')
ip=os.environ["REMOTE_ADDR"]
s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
s.bind(("192.168.1.100",4444))
a,b=s.recvfrom(100)
print a
s.sendto(mount,b)
if(backup=='Yes'):
os.system("lvcreate --size "+st+" --thin volume/pooL1")
os.system("lvcreate -V "+st+" --name "+ip+" --thin volume/pooL1")
os.system("mkfs.ext4 /dev/volume/"+ip)
os.system("mkdir /mnt/"+ip)
os.system("mount /dev/volume/"+ip+" /mnt/"+ip+"" )
os.system("lvcreate -s --name snap"+ip+" /dev/volume/"+ip)
os.system("mkdir /media/snap"+ip)
os.system("mount /dev/volume/snap"+ip+" /media/snap"+ip+"")
else:
os.system("lvcreate --size "+st+" --thin volume/pooL1")
os.system("lvcreate -V "+st+" --name "+ip+" --thin volume/pooL1")
os.system("mkfs.ext4 /dev/volume/"+ip)
os.system("mkdir /mnt/"+ip)
os.system("mount /dev/volume/"+ip+" /mnt/"+ip+"/" )
f=open('/etc/fstab','a+')
f.write("/mnt/"+ip+" /dev/volume/"+ip+" ext4 defaults 0 0")
f.close()
f=open('/etc/exports','a+')
f.write("/mnt/"+ip+" "+ip+ "(rw,sync,no_root_squash) \n")
f.close()
os.system("exportfs -a")
s.sendto("now you can use your storage",b)
s.close()
st is storage size.
I have given permissions to apache to create logical volume.The problem is lv is not mounting.Also client is getting internal server error even when lv is created at server.

You need to write sudo before every command like os.system('sudo ...') if you are using rhel version 7.2.
For working sudo user you have to configure /etc/sudoers file and adding extra line #apache ALL=(ALL) NOPASSWD:ALL at any line.

Using subprocess with openssl in a python script

I have thoroughly confused myself with Python subprocess syntax!
I would like to decrypt a string using openssl from within a Python script.
Here is the bash script snippet that works:
readable_code=$(echo "$encrypted_code"| openssl enc -aes-128-cbc -a -d -salt -pass pass:$key)
So in a python script - I understand that to run this same bash command I should use subprocess.
I need to Pipe the echo to the openssl command and as well pass in the encrypted_code and key variables dynamically(its in a loop).
Anyone out there know the correct syntax for this ?
Below's snippet should give the background to what i'm trying to do.
thank-you
import subprocess
key = "my-secret-key"
file = list_of_ips #format ip:long-encrypted-code
with open(file_read) as f:
#read in all connecion requests
content=f.readlines()
#create list that will hold all ips whose decrypted codes have passed test
elements = []
for ip_code in content:
#grab the ip address before the colon
ip = ip_code.split(':', 1)[0]
#grab the encrypted code after the colon
code = ip_code.split(':',1)[1]
#here is where I want to run the bash command and assign to a python variable
decrypted_code = subprocess....using code and key variables
...on it goes....

To emulate the shell command:
$ readable_code=$(echo "$encrypted_code"| openssl enc -aes-128-cbc -a -d -salt -pass "pass:$key")
using subprocess module in Python:
from subprocess import Popen, PIPE
cmd = 'openssl enc -aes-128-cbc -a -d -salt -pass'.split()
p = Popen(cmd + ['pass:' + key], stdin=PIPE, stdout=PIPE)
readable_code = p.communicate(encrypted_code)[0]

I highly recommend you to use Plumbum Python library to write shell scripts.
Particularly it has a convenient way to do piping and redirection.
I don't really understood what exact task you trying to solve, but your code could look approximately like this:
from plubum.cmd import openssl
with open('file') as f:
for ip_code in f:
(openssl['whatever', 'params'] << ip_code)()

python sub-process

I usually execute a Fortran file in Linux (manually) as:
Connect to the server
Go to the specific folder
ifort xxx.for -o xxx && ./xxx (where 'xxx.for' is my Fortran file and 'xxx' is Fortran executable file)
But I need to call my fortran file (xxx.for) from python (I'm a beginner), so I used subprocess with the following command:
cmd = ["ssh", sshConnect, "cd %s;"%(workDir), Fortrancmd %s jobname "%s -o %s" exeFilename "%s && %s ./ %s%s"%(exeFilename)]
But I get an error, and I'm not sure what's wrong. Here's the full code:
import string
import subprocess as subProc
from subprocess import Popen as ProcOpen
from subprocess import PIPE
import numpy
import subprocess
userID = "pear"
serverName = "say4"
workDir = "/home/pear/2/W/fortran/"
Fortrancmd = "ifort"
jobname = "rad.for"
exeFilename = "rad"
sshConnect=userID+"#"+servername
cmd=["ssh", sshConnect, "cd %s;"%(workDir), Fortrancmd %s jobname "%s -o %s" exeFilename "%s && %s ./ %s%s"%(exeFilename)]
**#command to execute fortran files in Linux
**#ifort <filename>.for -o <filename> && ./<filename> (press enter)
**#example:ifort xxx.for -o xxx && ./xxx (press enter)
print cmd
How can I write a python program that performs all 3 steps described above and avoids the error I'm getting?

there are some syntax errors...
original:
cmd=["ssh", sshConnect, "cd %s;"%(workDir), Fortrancmd %s jobname "%s -o %s" exeFilename "%s && %s ./ %s%s"%(exeFilename)]
I think you mean:
cmd = [
"ssh",
sshConnect,
"cd %s;" % (workDir,),
"%s %s -o %s && ./%s" % (Fortrancmd, jobname, exeFilename, exeFilename)
]
A few notes:
a tuple with one element requires a comma at the end of the first argument see (workDir,) to be interpreted as a tuple (vs. simple order-of-operations parens)
it is probably easier to contruct your fortan command with a single string format operation
PS - For readability it is often a good idea to break long lists into multiple lines :)
my advice
I would recommend looking at this stackoverflow thread for ssh instead of using subprocess

For the manual part you may want to look into pexpect or for windows wexpect. These allow you to perform subprocesses and pass input under interactive conditions.
However most of what you're doing sounds like it would work well in a shell script. For simplicity, you could make a shell script on the server side for your server side operations, and then plug in the path in the ssh statement:
ssh user#host "/path/to/script.sh"

one error:
you have an unquoted %s in your list of args, so your string formatting will fail.

Here is a complete example of using the subprocess module to run a remote command via ssh (a simple echo in this case) and grab the results, hope it helps:
>>> import subprocess
>>> proc = subprocess.Popen(("ssh", "remoteuser#host", "echo", "1"), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> stdout, stderr = proc.communicate()
Which in this case returns: ('1\n', '')
Note that to get this to work without requiring a password you will likely have to add your local user's public key to ~remoteuser/.ssh/authorized_keys on the remote machine.

You could use fabric for steps 1 and 2.
This is the basic idea:
from fabric.api import *
env.hosts = ['host']
dir = '/home/...'
def compile(file):
with cd(dir):
run("ifort %s.for -o %s" %(file,file))
run("./%s > stdout.txt" % file)
Create fabfile.py
And you run fab compile:filename

do you have to use python?
ssh user#host "command"