Develop Reference

How does one run `eval $(opam env)` when the command substitution is changes to the environment - all ran from within **python**?

I am aware that one can't easily run eval from python (though I admit I don't fully understand why). I've been trying a bunch of things -- even using opem exec --switch $SWITCH $command_substitution_output to do it but I can't get it to work.
This is my attempt:
def run_eval_opam_env(switch: str,
):
"""
Tries to run through opam exec command arg https://opam.ocaml.org/doc/man/opam-exec.html.
eval $(opam env --switch=switch --set-switch)
note:
- command substituion: replace the command ran here $(cmd) with its output (string).
It turns out, $() is called a command substitution. The command in between $() or backticks (“) is run and the
output replaces $().
ref:
- command sub: https://blog.wplauncher.com/what-is-in-linux/#:~:text=Example%20of%20command%20substitution%20using%20%24()%20in%20Linux%3A&text=Again%2C%20%24()%20is%20a%20command,another%20context%E2%80%9D%20(Source).
- eval $(opam env): https://stackoverflow.com/questions/30155960/what-is-the-use-of-eval-opam-config-env-or-eval-opam-env-and-their-differen?noredirect=1&lq=1
- there is a way with opam exec: https://discuss.ocaml.org/t/is-eval-opam-env-switch-switch-set-switch-equivalent-to-opam-switch-set-switch/10957/4
"""
logging.info(f'{run_eval_opam_env=}')
# - get cmd sub output
try:
cmd_sub_output: str = subprocess.check_output(f'opam env --switch={switch} --set-switch'.split())
except Exception as e:
logging.critical(f'Error: {e=}')
raise e
# - run command to eval
# command: str = f'eval {cmd_sub_output}' # doesn't seem possible https://stackoverflow.com/questions/53950225/python-check-output-call-to-eval-with-arguments-fails
command: list[str] = f'opam exec --switch {switch}'.split()
# command: list[str] = command + cmd_sub_output.split()
command: list[str] = command + [cmd_sub_output]
logging.info(f"-> {command=}")
try:
# res = subprocess.run(command.split(), check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
res = subprocess.run(command, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
logging.info(f'{res.stdout.decode()=}')
logging.info(f'{res.stderr.decode()=}')
except Exception as e:
logging.critical(f'Error: {e=}')
raise e
the output of the command substitution is a bunch of env stuff. How do I excute it in a subprocess or in the opam exec command?
See it's:
b"OPAMSWITCH='ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1'; export OPAMSWITCH;\nOPAM_SWITCH_PREFIX='/lfs/ampere4/0/brando9/.opam/ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1'; export OPAM_SWITCH_PREFIX;\nCAML_LD_LIBRARY_PATH='/lfs/ampere4/0/brando9/.opam/ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1/lib/stublibs:/lfs/ampere4/0/brando9/.opam/ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1/lib/ocaml/stublibs:/lfs/ampere4/0/brando9/.opam/ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1/lib/ocaml'; export CAML_LD_LIBRARY_PATH;\nOCAML_TOPLEVEL_PATH='/lfs/ampere4/0/brando9/.opam/ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1/lib/toplevel'; export OCAML_TOPLEVEL_PATH;\nMANPATH=':/usr/kerberos/man:/usr/share/man:/usr/local/man:/usr/man:/opt/puppetlabs/puppet/share/man:/lfs/ampere4/0/brando9/.opam/ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1/man'; export MANPATH;\nPATH='/lfs/ampere4/0/brando9/.opam/ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1/bin:/dfs/scratch0/brando9/anaconda/envs/iit_synthesis/bin:/lfs/ampere4/0/brando9/.local/bin:/lfs/ampere4/0/brando9/.ruby-build/bin:/lfs/ampere4/0/brando9/.rbenv/shims:/lfs/ampere4/0/brando9/.rbenv/bin:/usr/local/cuda-11.7/bin:/dfs/scratch0/brando9/anaconda/bin:/dfs/scratch0/brando9/anaconda/condabin:/dfs/scratch0/brando9/anaconda/bin:/dfs/scratch0/brando9/.local/bin:/dfs/scratch0/brando9/.ruby-build/bin:/dfs/scratch0/brando9/.rbenv/shims:/dfs/scratch0/brando9/.rbenv/bin:/usr/local/cuda-11.7/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/afs/cs/software/sbin:/afs/cs/software/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/puppetlabs/bin'; export PATH;\n"
but the cmd seems like nonsense:
subprocess.CalledProcessError: Command '['opam', 'exec', '--switch', 'ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1', b"OPAMSWITCH='ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1'; export OPAMSWITCH;\nOPAM_SWITCH_PREFIX='/lfs/ampere4/0/brando9/.opam/ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1'; export OPAM_SWITCH_PREFIX;\nCAML_LD_LIBRARY_PATH='/lfs/ampere4/0/brando9/.opam/ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1/lib/stublibs:/lfs/ampere4/0/brando9/.opam/ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1/lib/ocaml/stublibs:/lfs/ampere4/0/brando9/.opam/ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1/lib/ocaml'; export CAML_LD_LIBRARY_PATH;\nOCAML_TOPLEVEL_PATH='/lfs/ampere4/0/brando9/.opam/ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1/lib/toplevel'; export OCAML_TOPLEVEL_PATH;\nMANPATH=':/usr/kerberos/man:/usr/share/man:/usr/local/man:/usr/man:/opt/puppetlabs/puppet/share/man:/lfs/ampere4/0/brando9/.opam/ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1/man'; export MANPATH;\nPATH='/lfs/ampere4/0/brando9/.opam/ocaml-variants.4.07.1+flambda_coq-serapi.8.11.0+0.11.1/bin:/dfs/scratch0/brando9/anaconda/envs/iit_synthesis/bin:/lfs/ampere4/0/brando9/.local/bin:/lfs/ampere4/0/brando9/.ruby-build/bin:/lfs/ampere4/0/brando9/.rbenv/shims:/lfs/ampere4/0/brando9/.rbenv/bin:/usr/local/cuda-11.7/bin:/dfs/scratch0/brando9/anaconda/bin:/dfs/scratch0/brando9/anaconda/condabin:/dfs/scratch0/brando9/anaconda/bin:/dfs/scratch0/brando9/.local/bin:/dfs/scratch0/brando9/.ruby-build/bin:/dfs/scratch0/brando9/.rbenv/shims:/dfs/scratch0/brando9/.rbenv/bin:/usr/local/cuda-11.7/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/afs/cs/software/sbin:/afs/cs/software/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/puppetlabs/bin'; export PATH;\n"]'
I also tried just doing feval {cmd_sub_output} without success.
related:
https://discuss.ocaml.org/t/is-eval-opam-env-switch-switch-set-switch-equivalent-to-opam-switch-set-switch/10957/6
Python check_output call to eval (with arguments) fails
Running `eval $(something)` commands using Python sub-process
Is there a way to run the bash eval command in python?
What is the use of eval `opam config env` or eval $(opam env) and their difference?
opam exec https://opam.ocaml.org/doc/man/opam-exec.html

Command works on Command Prompt but it does not work when called with subprocess.run() or os.system() in python

Python 3.10.6
Windows 10
I have a python function that executes a DXL script using subsystem.run() or os.system() (whichever works best I guess). The problem is that when I run a custom command using python it does not work, but when I paste the same command in the command prompt, it works. I should also clarify that command prompt is not the ms store windows terminal (cannot run ibm doors commands there for some reason). It is the OG prompt
I need to use both python and IBM Doors for the solution.
Here is a summer version of my code (Obviously, the access values are not real):
#staticmethod
def run_dxl_importRTF():
dquotes = chr(0x22) # ASCII --> "
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = dquotes + r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" + dquotes
file_name = "LIBC_String.rtf"
# Based On:
# "C:\Program Files\IBM\Rational\DOORS\9.7\\bin\doors.exe" -dxl "string pModuleName = \"%~1\";string pFilename = \"%~2\";#include <importRTF.dxl>" -f "%TEMP%" -b "doNothing.dxl" -d 11111#11.11.1111.0 -user USER -password PASSWORD
script_arguments = f"{dquotes}string pModuleName=\{dquotes}{module_name}\{dquotes};string pFileName=\{dquotes}{file_name}\{dquotes};#include <{script_path}>{dquotes}"
command = [doors_path, "-dxl", script_arguments, "-f", "%TEMP%", "-b", script_do_nothing_path, '-d', database_config, '-user', user, '-password', password]
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')
return
PYTHON SCRIPT OUTPUT:
COMMAND:
"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe" -dxl "string pModuleName=\"TEST_TEMP\";string pFileName=\"LIBC_String.rtf\";#include <importRTF.dxl>" -f %TEMP% -b doNothing.dxl -d 11111#11.11.1111.0 -user USER_TEMP -password PASS_TEMP
STDERR: 'The system cannot find the path specified.\n'
STDOUT:
RETURN CODE: 1
When I run the same command in the command prompt, it works (dxl script is compiled).
I identified the problem which is the script_argument variable. Meaning that, when I try to just enter the IBM Doors server without compiling a DXL script, it works on python and the command prompt.
The python script needs to be dynamic meaning that all of the initial declared variables can change value and have a path string in it. I am also trying to avoid .bat files. They also did not work with dynamic path values
Thanks for your time
I tried:
Changing CurrentDirectory (cwd) to IBM Doors
os.system()
Multiple workarounds
Tried IBM Doors path without double quotes (it doesnt work because of the whitespaces)
.bat files

When calling subprocess.run with a command list and shell=True, python will expand the command list to a string, adding more quoting along the way. The details are OS dependent (on Windows, you always have to expand the list to a command) but you can see the result via the subprocess.list2cmdline() function.
Your problem is these extra escapes. Instead of using a list, build a shell command string that already contains the escaping you want. You can also use ' for quoting strings so that internal " needed for shell quoting can be entered literally.
Putting it all together (and likely messing something up here), you would get
#staticmethod
def run_dxl_importRTF():
module_name = "TEST_TEMP"
script_path = "importRTF.dxl"
script_do_nothing_path = "doNothing.dxl"
user = "user"
password = "pass"
database_config = "11111#11.11.1111.0"
doors_path = r"C:\Program Files\IBM\Rational\DOORS\9.7\bin\doors.exe"
file_name = "LIBC_String.rtf"
script_arguments = (rf'string pModuleName=\"{module_name}\";'
'string pFileName=\"{file_name}\";'
'#include <{script_path}>')
command = (f'"{doors_path}" -dxl "{script_arguments}" -f "%TEMP%"'
' -b "{script_do_nothing_path}" -d {database_config}'
' -user {user} -password {pass}')
res = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(f"COMMAND:\n{' '.join(res.args)}")
print(f"STDERR: {repr(res.stderr)}")
print(f'STDOUT: {res.stdout}')
print(f'RETURN CODE: {res.returncode}')

Executing Linux command in background from a python script

I have an ubuntu machine where kubernetes cluster is running. I am basically trying to run a Kubernetes command in background from python3 script but it's not working, please help...
Below is a part of code from a larger script, I am creating cmd4 using a formatted string and then passing cmd4 to os.system as os.system(cmd4). But as soon as I execute the script it starts showing logs in cmdline. I tried this using nohup as well as mentioned below but it starts populating logs in nohup.out.
for podname in pod_names:
if "smf" in podname or "pcf" in podname or "udm" in podname:
containername = 'worker'
else:
containername = 'cppe'
**cmd4 = f"kubectl logs -f -n core {podname} --container={containername} > {podname}_{containername}_log </dev/null &>/dev/null &&"**
cmd5 = f"echo $! >> pid.txt"
os.system(cmd4)
os.system(cmd5)
pid_file = open('pid.txt', 'a+')
pid_file.write("\n")
pid_file.close
=================================
tried with nohup as:
cmd4 = f"nohup kubectl logs -f -n core {podname} --container={containername} > {podname}_{containername}_log </dev/null &>/dev/null &&"
But it gives this o/p
updatedReplicas: 1
deployment.apps/core-pcf configured
nohup: ignoring input and appending output to 'nohup.out'

Have you tried using the subprocess library?
I think the solution for your script will be the following.
import subprocess
for podname in pod_names:
if "smf" in podname or "pcf" in podname or "udm" in podname:
containername = 'worker'
else:
containername = 'cppe'
**cmd4 = f"kubectl logs -f -n core {podname} --container={containername} > {podname}_{containername}_log </dev/null &>/dev/null &&"**
cmd5 = f"echo $! >> pid.txt"
cmd4_bash = subprocess.Popen(cmd4.split(), stdout=subprocess.PIPE)
cmd5_bash = subprocess.Popen(cmd5.split(), stdout=subprocess.PIPE)
pid_file = open('pid.txt', 'a+')
pid_file.write("\n")
pid_file.close

Unable to pass variable to a bash command in python

I am trying to pass a python variable to a bash command like this:
subscriptionId = "xxxxx"
command = " az account show -s $subscriptionId"
subprocess.check_output(command)
I get there following error:
error : az account show: error: argument --subscription/-s: expected one argument

Assigning a Python variable like subscriptionId = "xxxxx" does not magically place it in your environment, much less pass it to a subprocess. You need to do that interpolation yourself:
command = f"az account show -s {subscriptionId}"
If you really want to use environment variables, add the variable you want and enable shell expansion:
subscriptionId = ...
env = os.environ.copy()
env['subscriptionId'] = subscriptionId
command = "az account show -s ${subscriptionId}"
subprocess.check_output(command, env=env, shell=True)
Alternatively, you can mess with your own process environment:
subscriptionId = ...
os.environ['subscriptionId'] = subscriptionId
command = "az account show -s ${subscriptionId}"
subprocess.check_output(command, shell=True)
These options are, in my opinion, not recommended, since they raise all the security issues that shell=True brings with it, while providing you with no real advantage.

since the variable command is just a string you could simply do this.
subscriptionId = "xxxxx"
command = " az account show -s " + subscriptionId
subprocess.check_output(command)

how to umount from python script

i want to umount a mounting point named VirtualDVD.
i want to run the command, "gksudo umount VirtualDVD"
My function is:
def umount(self):
'''unmounts VirtualDVD'''
cmd = 'gksudo umount VirtualDVD'
proc = subprocess.Popen(str(cmd), shell=True, stdout=subprocess.PIPE).stdout.read()
print proc
i try "gksudo umount VirtualDVD" from terminal and everything is ok.
i try "gksudo umount VirtualDVD" from subprocess and it fails...
it pops up the gksudo dialog and i can enter my password, but then it seems that umount fails because the VirtualDVD still is mounted.
why?

I figure it out... i should umount with complete path of mounting point.
i changed the umount function as following and it works...
def umount(self):
'''unmounts VirtualDVD'''
#get virtualdvd folder
home = QtCore.QDir.homePath()
vpath = home + "/VirtualDVD"
cmd = 'gksudo umount ' + vpath
proc = subprocess.Popen(str(cmd), shell=True, stdout=subprocess.PIPE).stdout.read()
print proc