This question already has answers here:
how to "source" file into python script
(8 answers)
Closed 3 years ago.
I am struggling to execute a shell script from a Python program. The actual issue is the script is a load profile script and runs manually as :
. /path/to/file
The program can't be run as sh script as the calling programs are loading some configuration file and so must need to be run as . /path/to/file
Please do guide how can I integrate the same in my Python script? I am using subprocess.Popen command to run the script and as said the only way it works is to run as . /path/to/file and so not giving the right result.
Without knowledge of the precise reason the script needs to be sourced, this is slightly speculative.
The fundamental problem is this: How do I get a source command to take effect outside the shell script?
Let's say your sourced file does something like
export fnord="value"
This cannot (usefully) be run in a subshell (as a normally executed script would) because the environment variable and its value will be lost when the script terminates. The solution is to source (aka .) this snippet from an already running shell; then the value stays in that shell's environment until that shell terminates.
But Python is not a shell, and there is no general way for Python to execute arbitrary shell script code, short of reimplementing the shell in Python. You can reimplement a small subset of the shell's functionality with something like
with open('/path/to/file') as shell_source:
lines = shell_source.readlines()
for line in lines:
if line.strip().startswith('export '):
var, value = line[7:].strip().split('=', 1)
if value.startswith('"'):
value = value.strip('"')
elif value.startswith("'"):
value = value.strip("'")
os.environ[var] = value
with some very strict restrictions (let's not say naïve assumptions) on the allowable shell script syntax in the file. But what if the file contained something else than a series of variable assignments, or the assignment used something other than trivial quoted strings in the values? (Even the export might or might not be there. Its significance is to make the variable visible to subprocesses of the current shell; maybe that is not wanted or required? Also export variable=value is not portable; proper Bourne shell script syntax would use variable=value; export variable or one of the many variations.)
If you know what exactly your Python script needs from the shell script, maybe do something like
r = subprocess.run('. /path/to/file; printf "%s\n" "$somevariable"',
shell=True, capture_output=True, text=True)
os.environ['somevariable'] = r.stdout.split('\n')[-2]
to source the entire script in a subshell, then print to standard output the part you actually need, and capture that from your Python script (and assign it to an environment variable if that's what you eventually need to accomplish).
Related
I have file called . /home/test.sh (the space between the first . and / is intentional) which contains some environmental variables. I need to load this file and run the .py. If I run the command manually first on the Linux server and then run python script it generates the required output. However, I want to call . /home/test.sh from within python to load the profile and run rest of the code. If this profile is not loaded python scripts runs and gives 0 as an output.
The call
subprocess.call('. /home/test.sh',shell=True)
runs fine but the profile is not loaded on the Linux terminal to execute python code and give the desired output.
Can someone help?
Environment variables are not inherited directly by the parent process, which is why your simple approach does not work.
If you are trying to pick up environment variables that have been set in your test.sh, then one thing you could do instead is to use env in a sub-shell to write them to stdout after sourcing the script, and then in Python you can parse these and set them locally.
The code below will work provided that test.sh does not write any output itself. (If it does, then what you could do to work around it would be to echo some separator string afterward sourcing it, and before running the env, and then in the Python code, strip off the separator string and everything before it.)
import subprocess
import os
p = subprocess.Popen(". /home/test.sh; env -0", shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out, _ = p.communicate()
for varspec in out.decode().split("\x00")[:-1]:
pos = varspec.index("=")
name = varspec[:pos]
value = varspec[pos + 1:]
os.environ[name] = value
# just to test whether it works - output of the following should include
# the variables that were set
os.system("env")
It is also worth considering that if all that you want to do is set some environment variables every time before you run any python code, then one option is just to source your test.sh from a shell-script wrapper, and not try to set them inside python at all:
#!/bin/sh
. /home/test.sh
exec "/path/to/your/python/script $#"
Then when you want to run the Python code, you run the wrapper instead.
I have a python code in which at the beginning it takes a string variable let say "element_name" from user and build some sub-folders based on this string and also some output files created by this code move to those folders.
On the other hand, I have a bash script in which some codes should be running in the sub-folders made in python code.
Any help how to introduce those folders in bash? How to pass the "element_name" from python to bash?
In python code "a.py" I tried
first = subprocess.Popen(['/bin/echo', element_name], stdout=subprocess.PIPE)
second = subprocess.Popen(['bash', 'path/to/script', '--args'], stdin=first.stdout)
and then in bash
source a.py
echo $element_name
but it doesn't work.
It's not clear from your question what is in your scripts, but I guess
subprocess.run(['/bin/bash', 'path/to/script', '--args', element_name])
is doing what you intend to do, passing the value of element_name to script as an argument.
I found a way. What I did is to pass the argument in a bash file and import this bash file as a source to my main bash file. Now everything works well.
i'm calling a python script inside my bash script and I was wondering if there is a simple way to set my bash variables within my python script.
Example:
My bash script:
#!/bin/bash
someVar=""
python3 /some/folder/pythonScript.py
My python script:
anotherVar="HelloWorld"
Is there a way I can set my someVar to the value of anotherVar? I was thinking of printing properties in a file inside the python script and then read them from my bash script but maybe there is another way. Also I don't know and don't think it makes any difference but I can name both variable with the same name (someVar/someVar instead of someVar/anotherVar)
No, when you execute python, you start a new process, and every process has access only to their own memory. Imagine what would happen if a process could influence another processes memory! Even for parent/child processes like this, this would be a huge security problem.
You can make python print() something and use that, though:
#!/usr/bin/env python3
print('Hello!')
And in your shell script:
#!/usr/bin/env bash
someVar=$(python3 myscript.py)
echo "$someVar"
There are, of course, many others IPC techniques you could use, such as sockets, pipes, shared memory, etc... But without context, it's difficult to make a specific recommendation.
shlex.quote() in Python 3, or pipes.quote() in Python 2, can be used to generate code which can be evaled by the calling shell. Thus, if the following script:
#!/usr/bin/env python3
import sys, shlex
print('export foobar=%s' % (shlex.quote(sys.argv[1].upper())))
...is named setFoobar and invoked as:
eval "$(setFoobar argOne)"
...then the calling shell will have an environment variable set with the name foobar and the value argOne.
This question already has answers here:
How to get the environment variables of a subprocess after it finishes running?
(5 answers)
Closed 9 years ago.
On Windows there's a 3rd party command line tool I would like to use in my python script. Let's say it's foobar.exe located under C:\Program Files (x86)\foobar. Foobar comes with an additional batch file init_env.bat that will set up the shell environment for foobar.exe to run.
I want to write a python script, that will first call init_env.bat once and then foobar.exe multiple times. However, all mechanisms I know of (subprocess, os.system and backticks) seem to spawn a new process for each execution. Therefore, calling init_env.bat is useless, because it does not change the environment of the process in which the python script runs and thus every subsequent call to foobar.exe fails, because it's environment is not set up.
Is it possible to call init_env.bat from python in a way that allows init_env.bat to alter the environment of the calling scripts process?
Is it possible to call init_env.bat from python in a way that allows
init_env.bat to alter the environment of the calling scripts
process?
Not easily, although, if the init_env.bat is really simple, you could attempt to parse it, and make the changes to os.environ yourself.
Otherwise it's much easier to spawn it in a sub-shell, followed by a call to set to output the new environment variables, and parse the output from that.
The following works for me...
init_env.bat
#echo off
set FOO=foo
set BAR=bar
foobar.bat
#echo off
echo FOO=%FOO%
echo BAR=%BAR%
main.py
import sys, os, subprocess
INIT_ENV_BAT = 'init_env.bat'
FOOBAR_EXE = 'foobar.bat'
def init_env():
vars = subprocess.check_output([INIT_ENV_BAT, '&&', 'set'], shell=True)
for var in vars.splitlines():
k, _, v = map(str.strip, var.strip().partition('='))
if k.startswith('?'):
continue
os.environ[k] = v
def main():
init_env()
subprocess.check_call(FOOBAR_EXE, shell=True)
subprocess.check_call(FOOBAR_EXE, shell=True)
if __name__ == '__main__':
main()
...for which python main.py outputs...
FOO=foo
BAR=bar
FOO=foo
BAR=bar
Note that I'm only using a batch file in place of your foobar.exe because I don't have a .exe file handy which can confirm the environment variables are set.
If you're using a .exe file, you can remove the shell=True clause from the lines subprocess.check_call(FOOBAR_EXE, shell=True).
I a trying to control the volume of mplayer from a python program. The mplayer program gets started from a bash script:
#!/bin/bash
mkfifo /home/administrator/files/mplayer-control.pipe
/usr/bin/mplayer -slave -input file=/home/administrator/files/mplayer-control.pipe /home/administrator/music/file.mp3
Then I have a GUI written in Python that is supposed to be able to control the volume of the instance of mplayer that is being played. I have tried the following:
os.system('echo "set_property volume $musicvol" > /home/administrator/files/mplayer-control.pipe')
That works if i substitute $musicvol with the numeric value instead, but that is unfortunately of no use. I need to be able to pass the variable.
I would also be able to solve it by invoking a bash script from the Python application, but I can not get that to work either:
subprocess.call("/home/administrator/files/setvolume.sh", executable="bash", shell=True)
You don't need to call os.system and invoke a shell to write that line to the FIFO from your Python script- you can just do:
new_volume = 50
with open("/home/administrator/files/mplayer-control.pipe","w") as fp:
fp.write("set_property volume %d\n" % (new_volume,))
It's not clear to me what you expect to happen in your original python, though - is musicvol set in the environment? If instead it's a Python variable that you want to insert into the string that you're passing, the easiest way is to use the string interpolation operator (%) as I've done in the example above.
In your example of using subprocess.call you don't need the executable or shell keyword arguments if setvolume.sh is executable and has a #! line - you could just do:
subprocess.call("/home/administrator/files/setvolume.sh")
However, it's better to just use open and write in Python as above, I think.