I am looking for a solution to loading an envvars.sh file within a django application. I would prefer to do this within the settings module and keep a file outside of it. Here is what I currently have:
SITE_ROOT = os.path.dirname(os.path.realpath(__file__))
def source_envvars(fname='envvars.sh'):
"""Equivalent of `$ source {fname}` with lines in the form of `export key=value`"""
envvars = os.path.join(SITE_ROOT, fname)
if not os.path.exists(envvars): return
with open(fname, 'r') as f:
for line in f:
if line.startswith('export '):
line = line.strip()
key = line.split('=')[0].split()[1]
value = line.split('=')[1].strip('"\'')
os.environ[key] = value
if not os.environ.get('DB_HOST'): source_envvars()
# rest of settings.py file...
Are there any downsides of using this approach? I know this doesn't support complicated bash-type exports, but all I have are basic exports of the form:
export DB_HOST=rds.amazon.com/12345
Does the above approach properly instantiate all the variables, or does it seem to be missing something or insecure in some way or other?
Your implementation can only handle fixed formats. A shell script can potentially have conditional statements and loops, etc., none of which can be handled properly without actually sourcing it through a shell.
A more robust method is therefore to use subprocess.check_output to run a shell with . to source envvars.sh and the env command to output the environment variables, the output of which can be then used to create a generator expression to update os.environ with:
os.environ.update(
line.rstrip().split('=', 1)
for line in subprocess.check_output(['sh', '-c', '. envvars.sh; env']).splitlines()
)
Related
I would like to write a pytest case for the behavior of a function that opens a file in case that file does not exist.
I think the question boils down to a different one, namely "How can I be sure a file path does not exist on the file system?"
import pytest
def file_content(file_name):
with open(file_name, "r") as f:
return f.read()
def test_file_content_file_not_found():
file_name_of_inexistent_file = "???"
with pytest.raises(FileNotFoundError):
file_content(file_name_of_inexistent_file)
test_file_content_file_not_found()
"???" denotes the spot I think some great tool out there implements a reasonable and secure way to generate a file name or a mock file system that ensures the failure is guaranteed, but also doesn't need to change the file system.
At the moment, I have a small helper function that generates random strings that are tested if they are existing files and if they are not are returned. This way I can emulate the desired behavior. However, I guess there must be a more standard way to achieve this.
the easiest way is to use the builtin tmp_path fixture to generate a unique, empty directory:
def test_does_not_exist(tmp_path):
with pytest.raises(FileNotFoundError):
file_content(tmp_path.joinpath('dne'))
tmp_path is generated per-test and will start empty, so dne will never exist
if you want a generic, non-pytest solution you can use tempfile.TemporaryDirectory directly:
with tempfile.TemporaryDirectory() as tmpdir:
with pytest.raises(FileNotFoundError):
file_content(os.path.join(tmpdir, 'dne'))
disclaimer: I'm a pytest core dev
Working with scientific data, specifically climate data, I am constantly hard-coding paths to data directories in my Python code. Even if I were to write the most extensible code in the world, the hard-coded file paths prevent it from ever being truly portable. I also feel like having information about the file system of your machine coded in your programs could be security issue.
What solutions are out there for handling the configuration of paths in Python to avoid having to code them out explicitly?
One of the solution rely on using configuration files.
You can store all your path in a json file like so :
{
"base_path" : "/home/bob/base_folder",
"low_temp_area_path" : "/home/bob/base/folder/low_temp"
}
and then in your python code, you could just do :
import json
with open("conf.json") as json_conf :
CONF = json.load(json_conf)
and then you can use your path (or any configuration variable you like) like so :
print "The base path is {}".format(CONF["base_path"])
First off its always good practise to add a main function to go with each class to test that class or functions in the file. Along with this you determine the current working directory. This becomes incredibly important when running python from a cron job or from a directory that is not the current working directory. No JSON files or environment variables are then needed and you will obtain interoperation across Mac, RHEL and Debian distributions.
This is how you do it, and it will work on windows also if you use '\' instead of '/' (if that is even necessary, in your case).
if "__main__" == __name__:
workingDirectory = os.path.realpath(sys.argv[0])
As you can see when you run your command, the working directory is calculated if you provide a full path or relative path, meaning it will work in a cron job automatically.
After that if you want to work with data that is stored in the current directory use:
fileName = os.path.join( workingDirectory, './sub-folder-of-current-directory/filename.csv' )
fp = open( fileName,'r')
or in the case of the above working directory (parallel to your project directory):
fileName = os.path.join( workingDirectory, '../folder-at-same-level-as-my-project/filename.csv' )
fp = open( fileName,'r')
I believe there are many ways around this, but here is what I would do:
Create a JSON config file with all the paths I need defined.
For even more portability, I'd have a default path where I look for this config file but also have a command line input to change it.
In my opinion passing arguments from command line would be best solution. You should take a look at argparse . This allows you to create nice way to handle arguments from the command line. for example:
myDataScript.py /home/userName/datasource1/
I have a file contains set of environment variables .
env_script.env:
export a=hjk
export b=jkjk
export c=kjjhh
export i=jkkl
..........
I want set these environment variables by reading from file .
how can i do this in python
Tried sample code:
pipe = subprocess.Popen([".%s;env", "/home/user/env_script.env"], stdout=subprocess.PIPE, shell=True)
output = pipe.communicate()[0]
env = dict((line.split("=", 1) for line in output.splitlines()))
os.environ.update(env)
Please give some suggestion
There's a great python library python-dotenv that allows you to have your variables exported to your environment from a .env file, or any file you want, which you can keep out of source control (i.e. add to .gitignore):
# to install
pip install -U python-dotenv
# your .env file
export MY_VAR_A=super-secret-value
export MY_VAR_B=other-very-secret-value
...
And you just load it in python when your start like:
# settings.py
from dotenv import load_dotenv
load_dotenv()
Then, you can access any variable later in your code:
from os import environ
my_value_a = environ.get('MY_VALUE_A')
print(my_value_a) # 'super-secret-value'
You don't need to use subprocess.
Read lines and split environment variable name, value and assign it to os.environ:
import os
with open('/home/user/env_script.env') as f:
for line in f:
if 'export' not in line:
continue
if line.startswith('#'):
continue
# Remove leading `export `
# then, split name / value pair
key, value = line.replace('export ', '', 1).strip().split('=', 1)
os.environ[key] = value
or using dict.update and generator expression:
with open('env_script.env') as f:
os.environ.update(
line.replace('export ', '', 1).strip().split('=', 1) for line in f
if 'export' in line
)
Alternatively, you can make a wrapper shell script, which sources the env_script.env, then execute the original python file.
#!/bin/bash
source /home/user/env_script.env
python /path/to/original_script.py
Modern operating systems do not allow a child process to change the environment of its parent. The environment can only be changed for the current process and its descendants. And a Python interpreter is a child of the calling shell.
That's the reason why source is not an external command but is interpreted directly by the shell to allow a change in its environment.
It used to be possible in the good old MS/DOS system with the .COM executable format. A .com executable file had a preamble of 256 (0x100) bytes among which was a pointer to the COMMAND.COM's environment string! So with low level memory functions, and after ensuring not overwriting anything past the environment, a command could change directly its parent environment.
It may still be possible in modern OS, but require cooperation from system. For example Windows can allow a process to get read/write access to the memory of another process, provided the appropriate permissions are set. But this is really a hacky way, and I would not dare doing this in Python.
TL/DR: if your requirement is to change the environment of the calling shell from a Python script, you have misunderstood your requirement.
But what is easy is to start a new shell with a modified environment:
import os
import subprocess
env = os.environ.copy() # get a copy of current environment
# modify the copy of environment at will using for example falsetru's answer
# here is just an example
env['AAA'] = 'BBB'
# and open a subshell with the modified environment
p = subprocess.Popen("/bin/sh", env = env)
p.wait()
Some git commands, git commit for example, invoke a command-line based text editor (such as vim or nano, or other) pre-filled with some values and, after the user saves and exists, do something with the saved file.
How should I proceed to add this functionality to a Python similar command-line program, at Linux?
Please don't stop yourself for giving an answer if it does not use Python, I will be pretty satisfied with a generic abstract answer, or an answer as code in another language.
The solution will depend on what editor you have, which environment variable the editor might possibly be found in and if the editor takes any command line parameters.
This is a simple solution that works on windows without any environment variables or command line arguments to the editor. Modify as is needed.
import subprocess
import os.path
def start_editor(editor,file_name):
if not os.path.isfile(file_name): # If file doesn't exist, create it
with open(file_name,'w'):
pass
command_line=editor+' '+file_name # Add any desired command line args
p = subprocess.Popen(command_line)
p.wait()
file_name='test.txt' # Probably known from elsewhere
editor='notepad.exe' # Read from environment variable if desired
start_editor(editor,file_name)
with open(file_name,'r') as f: # Do something with the file, just an example here
for line in f:
print line
I'm making a python script right now, and I need to use some environment variables which are set in a bash shell script.
The bash script is something like:
#! /bin/sh
#sets some names:
export DISTRO="unified"
#export DISTRO="other"
#number of parallel builds
export BB_NUM_THREADS=2
#set build dir
export BUILDDIR=$PWD
Normally, I would just source this script in bash, then go do my builds. I'm trying to wrap python around the whole process to do some management of the output so I want to remove the manual source ./this_script.sh step.
What I want to do is read this script from python and then use os.environ to set up the variables within it. (I know this will not affect the parent, but only the current running Python instance and that's fine)
So to make my work easier, I'm trying to find out are there any modules which can "parse" the bash script and make use of the environment variables found within? Currently I'm doing this by hand and it's a bit of a pain.
If no such module exists to do exactly what I want, is there a more pythonic (read: easier/shorter) way of manually parsing a file in general, right now I'm doing this:
def parse_bash_script(fn):
with open(fn) as f:
for line in f:
if not line[:1] == '#': #ignore comments
if "export" in line:
line = line.replace(" ","").strip()
var = line[6:line.find("=")]
val = line[line.find("=")+1:len(line)]
if "\"" in val:
val = val[1:-1]
os.environ[var]=val
There is no module to do exactly what you want, but shlex will do a lot of what you want. In particular, it will get the quoting, etc. right without you having to worry about it (which is the hardest part of this), as well as skipping comments, etc. The only thing it won't do is handle the export keywords.
The easy way around that is to preprocess:
with open(fn) as f:
processed = f.read().replace('export ', '')
for line in shlex.split(processed):
var, _, value = line.partition('=')
os.environ[var] = val
It's a bit hackier, but you can also do it a bit less verbosely by post-processing. In particular, shlex will treat export foo="bar spam eggs" as two values: export and foo="bar spam eggs", and you can just skip the ones that == 'export', or where the partition finds nothing, or… For example:
with open(fn) as f:
for line in shlex.split(f.read()):
var, eq, value = line.partition('=')
if eq:
os.environ[var] = val
If you want to get fancier, you can construct a shlex object and (a) drive the parser directly from the file, and (b) control the parsing at a finer-grained level. However, I don't think that's necessary here.
Meanwhile, if you want to handle environment substitution (as the BUILDDIR=$PWD implies), this won't magically take care of that for you. You can make configparser do that for you with its ExtendedInterpolation feature, but then you'll need to trick configparser into handling shlex syntax, at which point… why bother.
You can of course do it manually by writing your own interpolator, but that's hard to get right. You need to know the shell's rules for why $PWD-foo is the same as ${PWD}-foo, but $PWD_foo is the same as ${PWD_foo}, etc.
A better solution at this point—assuming the script is actually safe to run—would be to actually use a shell to do it for you. For example:
with open('script.sh') as f:
script = f.read()
script += b'\nenv'
with subprocess.Popen(['sh'], stdin=subprocess.PIPE, stdout=subprocess.PIPE) as p:
result = p.communicate(script)
for line in result.splitlines():
var, _, value = line.partition('=')
os.environ[var] = value
Of course this will also override things like _=/usr/bin/env, but probably not anything you care about.
def parse_bash_script(fn):
with open(fn) as f:
for line in f:
if not line.startswith('#'): #ignore comments
if "export" in line:
var, _, val = line.partition('=')
var = var.lstrip()
val = val.rstrip()
if val.startswith('"'):
vals = val.rpartition('"')
val = vals[0][1]+vals[2]
os.environ[var]=val
I had the same problem, and based on the advice from abarnert, I decided to implement the solution as a subprocess call to a restricted bash shell, combined with shlex.
import shlex
import subprocess
filename = '/path/to/file.conf'
o, e = subprocess.Popen(
['/bin/bash', '--restricted', '--noprofile', '--init-file',
filename, '-i', '-c', 'declare'],
env={'PATH': ''},
stdout=subprocess.PIPE,
stderr=subprocess.PIPE).communicate()
if e:
raise StandardError('conf error in {}: {}'.format(filename, e))
for token in shlex.split(o):
parts = token.split('=', 1)
if len(parts) == 2:
os.environ[parts[0]] = parts[1]
The advantage to the restricted shell is that it blocks many of the undesirable or malicious side effects that may otherwise happen when executing a shell script. From the bash documentation:
A restricted shell is used to set up an environment more controlled than the standard shell. It behaves identically to bash with the exception that the following are disallowed or not performed:
changing directories with cd
setting or unsetting the values of SHELL, PATH, ENV, or BASH_ENV
specifying command names containing /
specifying a file name containing a / as an argument to the . builtin command
Specifying a filename containing a slash as an argument to the -p option to the hash builtin command
importing function definitions from the shell environment at startup
parsing the value of SHELLOPTS from the shell environment at startup
redirecting output using the >, >|, <>, >&, &>, and >> redirection operators
using the exec builtin command to replace the shell with another command
adding or deleting builtin commands with the -f and -d options to the enable builtin command
Using the enable builtin command to enable disabled shell builtins
specifying the -p option to the command builtin command
turning off restricted mode with set +r or set +o restricted.