I have a file contains set of environment variables .
env_script.env:
export a=hjk
export b=jkjk
export c=kjjhh
export i=jkkl
..........
I want set these environment variables by reading from file .
how can i do this in python
Tried sample code:
pipe = subprocess.Popen([".%s;env", "/home/user/env_script.env"], stdout=subprocess.PIPE, shell=True)
output = pipe.communicate()[0]
env = dict((line.split("=", 1) for line in output.splitlines()))
os.environ.update(env)
Please give some suggestion
There's a great python library python-dotenv that allows you to have your variables exported to your environment from a .env file, or any file you want, which you can keep out of source control (i.e. add to .gitignore):
# to install
pip install -U python-dotenv
# your .env file
export MY_VAR_A=super-secret-value
export MY_VAR_B=other-very-secret-value
...
And you just load it in python when your start like:
# settings.py
from dotenv import load_dotenv
load_dotenv()
Then, you can access any variable later in your code:
from os import environ
my_value_a = environ.get('MY_VALUE_A')
print(my_value_a) # 'super-secret-value'
You don't need to use subprocess.
Read lines and split environment variable name, value and assign it to os.environ:
import os
with open('/home/user/env_script.env') as f:
for line in f:
if 'export' not in line:
continue
if line.startswith('#'):
continue
# Remove leading `export `
# then, split name / value pair
key, value = line.replace('export ', '', 1).strip().split('=', 1)
os.environ[key] = value
or using dict.update and generator expression:
with open('env_script.env') as f:
os.environ.update(
line.replace('export ', '', 1).strip().split('=', 1) for line in f
if 'export' in line
)
Alternatively, you can make a wrapper shell script, which sources the env_script.env, then execute the original python file.
#!/bin/bash
source /home/user/env_script.env
python /path/to/original_script.py
Modern operating systems do not allow a child process to change the environment of its parent. The environment can only be changed for the current process and its descendants. And a Python interpreter is a child of the calling shell.
That's the reason why source is not an external command but is interpreted directly by the shell to allow a change in its environment.
It used to be possible in the good old MS/DOS system with the .COM executable format. A .com executable file had a preamble of 256 (0x100) bytes among which was a pointer to the COMMAND.COM's environment string! So with low level memory functions, and after ensuring not overwriting anything past the environment, a command could change directly its parent environment.
It may still be possible in modern OS, but require cooperation from system. For example Windows can allow a process to get read/write access to the memory of another process, provided the appropriate permissions are set. But this is really a hacky way, and I would not dare doing this in Python.
TL/DR: if your requirement is to change the environment of the calling shell from a Python script, you have misunderstood your requirement.
But what is easy is to start a new shell with a modified environment:
import os
import subprocess
env = os.environ.copy() # get a copy of current environment
# modify the copy of environment at will using for example falsetru's answer
# here is just an example
env['AAA'] = 'BBB'
# and open a subshell with the modified environment
p = subprocess.Popen("/bin/sh", env = env)
p.wait()
Related
I have a python script that calls an application using subprocess. I am calling this application many times, currently I am doing something along the lines of
out, err = subprocess.Popen(f"module load {' '.join(my_module_list)} && ./my_binary", stdout=sp.PIPE, stderr=sp.STDOUT, shell = True).communicate()
to run my program. Ideally I would like to first generate a modified os.environ object that already contains all the paths to the modules I am loading, and then pass it to subprocess.Popen under the env argument. However, since the printenv command doesn't output a python dictionary format, I'm not sure how to access all the modifications that modules load makes to the environment variables. Is there a good, clean way to create the required modified os.environ object?
I'd be tempted to call python in the subprocess and dump from os.environ in it
python -c 'import os; print(os.environ)'
Once you know what you're after, you can pass a dict directly to subprocess's env arg to set custom environmental vars, which could be something like
custom_env = os.environ.copy()
custom_env["foo"] = "bar"
subprocess.Popen(
...
env=custom_env,
)
I have file called . /home/test.sh (the space between the first . and / is intentional) which contains some environmental variables. I need to load this file and run the .py. If I run the command manually first on the Linux server and then run python script it generates the required output. However, I want to call . /home/test.sh from within python to load the profile and run rest of the code. If this profile is not loaded python scripts runs and gives 0 as an output.
The call
subprocess.call('. /home/test.sh',shell=True)
runs fine but the profile is not loaded on the Linux terminal to execute python code and give the desired output.
Can someone help?
Environment variables are not inherited directly by the parent process, which is why your simple approach does not work.
If you are trying to pick up environment variables that have been set in your test.sh, then one thing you could do instead is to use env in a sub-shell to write them to stdout after sourcing the script, and then in Python you can parse these and set them locally.
The code below will work provided that test.sh does not write any output itself. (If it does, then what you could do to work around it would be to echo some separator string afterward sourcing it, and before running the env, and then in the Python code, strip off the separator string and everything before it.)
import subprocess
import os
p = subprocess.Popen(". /home/test.sh; env -0", shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out, _ = p.communicate()
for varspec in out.decode().split("\x00")[:-1]:
pos = varspec.index("=")
name = varspec[:pos]
value = varspec[pos + 1:]
os.environ[name] = value
# just to test whether it works - output of the following should include
# the variables that were set
os.system("env")
It is also worth considering that if all that you want to do is set some environment variables every time before you run any python code, then one option is just to source your test.sh from a shell-script wrapper, and not try to set them inside python at all:
#!/bin/sh
. /home/test.sh
exec "/path/to/your/python/script $#"
Then when you want to run the Python code, you run the wrapper instead.
I am looking to permanently modify the path variable inside windows from a python script. The script is a wrapper to help automate the installation of applications and I want to enable the API to add applications to the path.
So for example I want to install a program called micro which has an install path of C:\Users\USERNAME\Path\to\micro and then add that install path to my path variable so that I can just run micro in my terminal.
I've been made aware of 2 possible solutions, which both do not work:
1. Using os.environ
In python the os module let's you read environment variables, but not actually modify them. so for example:
program_path = "C:\\Users\\USERNAME\\Path\\to\\micro"
new_path = f"{os.environ['PATH']};{program_path}"
os.environ["PATH"] = new_path
This would update the path variable in the python script, but it does not actually modify it on the system which is what I want.
2. setx
I was made aware that it is possible to update your path using the setx command in windows, but for some reason on windows 10 this destroys your path variable.
The idea is that you can call the setx command from python and use it to update the path variable. You should be able to type setx path "%path%;C:\Users\USERNAME\Path\to\micro" and have it update correctly.
So for example, in python code that would be:
program_path = "C:\\Users\\USERNAME\\Path\\to\\micro"
subprocess.Popen(f'setx path "%path%;{program_path}"')
This should take the current path variable and append the program path to it, but instead it just wipes your entire path and replaces it with a literal %path% and then the program path.
So now my path looks like this:
%path%
C:\Users\USERNAME\Path\to\micro
Any ideas on how to get this to work would be appreciated.
Okay, so after a long (and disgusting) amount of research I found a solution. Here is the method I came up with for a cross-platform system of adding to PATH variable:
def add_to_path(program_path:str):
"""Takes in a path to a program and adds it to the system path"""
if os.name == "nt": # Windows systems
import winreg # Allows access to the windows registry
import ctypes # Allows interface with low-level C API's
with winreg.ConnectRegistry(None, winreg.HKEY_CURRENT_USER) as root: # Get the current user registry
with winreg.OpenKey(root, "Environment", 0, winreg.KEY_ALL_ACCESS) as key: # Go to the environment key
existing_path_value = winreg.EnumValue(key, 3)[1] # Grab the current path value
new_path_value = existing_path_value + program_path + ";" # Takes the current path value and appends the new program path
winreg.SetValueEx(key, "PATH", 0, winreg.REG_EXPAND_SZ, new_path_value) # Updated the path with the updated path
# Tell other processes to update their environment
HWND_BROADCAST = 0xFFFF
WM_SETTINGCHANGE = 0x1A
SMTO_ABORTIFHUNG = 0x0002
result = ctypes.c_long()
SendMessageTimeoutW = ctypes.windll.user32.SendMessageTimeoutW
SendMessageTimeoutW(HWND_BROADCAST, WM_SETTINGCHANGE, 0, u"Environment", SMTO_ABORTIFHUNG, 5000, ctypes.byref(result),)
else: # If system is *nix
with open(f"{os.getenv('HOME')}/.bashrc", "a") as bash_file: # Open bashrc file
bash_file.write(f'\nexport PATH="{program_path}:$PATH"\n') # Add program path to Path variable
os.system(f". {os.getenv('HOME')}/.bashrc") # Update bash source
print(f"Added {program_path} to path, please restart shell for changes to take effect")
Neither are pretty, but it does actually work. You do need to restart running shells for it to take effect, but other than that it's perfect.
I have a file with some environment variables that I want to use in a python script
The following works form the command line
$ source myFile.sh
$ python ./myScript.py
and from inside the python script I can access the variables like
import os
os.getenv('myvariable')
How can I source the shell script, then access the variables, from with the python script?
If you are saying backward environment propagation, sorry, you can't. It's a security issue. However, directly source environment from python is definitely valid. But it's more or less a manual process.
import subprocess as sp
SOURCE = 'your_file_path'
proc = sp.Popen(['bash', '-c', 'source {} && env'.format(SOURCE)], stdout=sp.PIPE)
source_env = {tup[0].strip(): tup[1].strip() for tup in map(lambda s: s.strip().split('=', 1), proc.stdout)}
Then you have everything you need in source_env.
If you need to write it back to your local environment (which is not recommended, since source_env keeps you clean):
import os
for k, v in source_env.items():
os.environ[k] = v
Another tiny attention needs to be paid here, is since I called bash here, you should expect the rules are applied here too. So if you want your variable to be seen, you will need to export them.
export VAR1='see me'
VAR2='but not me'
You can not load environmental variables in general from a bash or shell script, it is a different language. You will have to use bash to evaluate the file and then somehow print out the variables and then read them. see Forcing bash to expand variables in a string loaded from a file
I am trying to set a environmental variable permanently. but temporarily it is working.
if i run below program i got the variable path. after close it and open new terminal to find the variable path using the command printenv LD_LIBRARY_PATH nothing will print.
#!/usr/bin/python
import os
import subprocess
def setenv_var():
env_var = "LD_LIBRARY_PATH"
env_path = "/usr/local/lib"`enter code here`
os.environ[env_var] = env_path
process = subprocess.Popen('printenv ' + env_var, stdout=subprocess.PIPE, shell=True)
result = process.communicate()[0]
return result
if __name__ == '__main__':
print setenv_var()
please help me.
Here is what I use to set environment variables:
def setenv_var(env_file, set_this_env=True):
env_var = "LD_LIBRARY_PATH"
env_path = "/usr/local/lib"`enter code here`
# set environments opened later by appending to `source`-d file
with open(env_file, 'a') as f:
f.write(os.linesep + ("%s=%s" % (env_var, env_path)))
if set_this_end:
# set this environment
os.environ[env_var] = env_path
Now you only have to choose where to set it, that is the first argument in the function. I recommend the profile-specific file ~/.profile or if you're using bash which is pretty common ~/.bashrc
You can also set it globally by using a file like /etc/environment but you'll need to have permissions when you run this script (sudo python script.py).
Remember that environments are inherited from the parent process, and you can't have a child set up a parent process' environment.
When you set an environment variable, it only affects the currently running process (and, by extension, any children that are forked after the variable is set). If you are attempting to set an environment variable in your shell and you want that environment variable to always be set for your interactive shells, you need to set it in the startup scripts (eg .login, .bashrc, .profile) for your shell. Commands that you run are (initially) children of the shell from which you run them, so although they inherit the environment of the shell and can change their own environment, they cannot change the environment of your shell.
Whether you do an export from bash or you set your os.environ from Python, these only stay for the session or process's lifetime. If you want to set them permanent you will have to touch and add it to the respective shell's profile file.
For ex. If you are on bash, you could do:
with open("~/.bashrc", "a") as outfile: # 'a' stands for "append"
outfile.write("export LD_LIBRARY_PATH=/usr/local/lib")
Check this out for some insight as to which file to add this depending on the target shell. https://unix.stackexchange.com/questions/117467/how-to-permanently-set-environmental-variables