Develop Reference

Setting global variables from a dictionary within a function

I am looking to use .yaml to manage several global parameters for a program. I would prefer to manage this from within a function, something like the below. However, it seems globals().update() does not work when included inside a function. Additionally, given the need to load an indeterminate number of variables with unknown names, using the basic global approach is not appropriate. Ideas?
.yaml
test:
- 12
- 13
- 14
- stuff:
john
test2: yo
Python
import os
import yaml
def load_config():
with open(os.path.join(os.getcwd(), {file}), 'r') as reader:
vals = yaml.full_load(reader)
globals().update(vals)
Desired output
load_config()
test
---------------
[12,13,14,{'stuff':'john'}]
test2
---------------
yo
What I get
load_config()
test
---------------
NameError: name 'test' is not defined
test2
---------------
NameError: name 'test2' is not defined
Please note: {file} is for you, the code is not actually written that way. Also note that I understand the use of global is not normally recommended, however it is what is required for the answer of this question.

You had {file} in your code, I've assumed that was intended to just be a string of the actual filename. I certainly hope you weren't looking to .format() and then eval() this code? That would be a very bad and unsafe way to run code.
Just return the dictionary vals itself, and access it as needed:
import os
import yaml
def load_config(fn):
with open(os.path.join(os.getcwd(), fn), 'r') as reader:
# only returning the value, so doing it in one step:
return yaml.full_load(reader)
cfg = load_config('test.yaml')
print(cfg)
print(cfg['test2'])
Output:
{'test': [12, 13, 14, {'stuff': 'john'}], 'test2': 'yo'}
yo
You should definitely never just update globals() with content from an external file. Use of globals() is only for very specific use cases anyway.
Getting the exact desired output is just a matter of formatting the contents of the dictionary:
import os
import yaml
def load_config(fn):
with open(os.path.join(os.getcwd(), fn), 'r') as reader:
return yaml.full_load(reader)
def print_config(d):
for k, v in d.items():
print(f'{k}\n---------------\n{v}\n')
cfg = load_config('test.yaml')
print_config(cfg)
Which gives exactly the output you described.
Note that this is technically superfluous:
os.path.join(os.getcwd(), fn)
By default, file operations are executed on the current working directory, so you'd achieve the same with:
def load_config(fn):
with open(fn, 'r') as reader:
return yaml.full_load(reader)
If you wanted to open the file in the same folder as the script itself, consider this instead:
def load_config(fn):
with open(os.path.join(os.path.dirname(__file__), fn), 'r') as reader:
return yaml.full_load(reader)

Read Value from Config File Python

I have a file .env file contain 5 lines
DB_HOST=http://localhost/
DB_DATABASE=bheng-local
DB_USERNAME=root
DB_PASSWORD=1234567890
UNIX_SOCKET=/tmp/mysql.sock
I want to write python to grab the value of DB_DATABASE
I want this bheng-local
I would have use
import linecache
print linecache.getline('.env', 2)
But some people might change the order of the cofigs, that's why linecache is not my option.
I am not sure how to check for only some strings match but all the entire line, and grab the value after the =.
I'm kind of stuck here :
file = open('.env', "r")
read = file.read()
my_line = ""
for line in read.splitlines():
if line == "DB_DATABASE=":
my_line = line
break
print my_line
Can someone please give me a little push here ?

Have a look at the config parser:
https://docs.python.org/3/library/configparser.html
It's more elegant than a self-made solution

Modify your .env to
[DB]
DB_HOST=http://localhost/
DB_DATABASE=bheng-local
DB_USERNAME=root
DB_PASSWORD=1234567890
UNIX_SOCKET=/tmp/mysql.sock
Python code
#!/usr/local/bin/python
import configparser
config = configparser.ConfigParser()
config.read('test.env')
print config.get('DB','DB_DATABASE')
Output:
bheng-local
Read https://docs.python.org/3/library/configparser.html

This should work for you
#!/usr/local/bin/python
file = open('test.env', "r")
read = file.read()
for line in read.splitlines():
if 'DB_DATABASE=' in line:
print line.split('=',1)[1]

#!/usr/local/bin/python
from configobj import ConfigObj
conf = ConfigObj('test.env')
print conf['DB_DATABASE']

from os import environ, path
from dotenv import load_dotenv
basedir = path.abspath(path.dirname(__file__))
load_dotenv(path.join(basedir, '.env'))
DB_DATABASE = environ.get('DB_DATABASE')
print(DB_DATABASE)
This could be another option

filter a Linux log file using Python

I want to filter a log file to keep all lines matching a certain pattern. I want to do this with Python.
Here's my first attempt:
#!/usr/bin/env python
from sys import argv
script, filename = argv
with open(filename) as f:
for line in f:
try:
e = line.index("some_term_I_want_to_match")
except:
pass
else:
print(line)
How can I improve this to:
save the result to a new file of similar name (i.e., a different extension)
use regex to make it more flexible/powerful.
(I'm just learning Python. This question is as much about learning Python as it is about accomplishing this particular result.)
OK, here's what I came up with so far... But how do you do the equivalent of prepending an r as in the following line
re.compile(r"\s*")
where the string is not a string literal, as in the next line?
re.compile(a_string_variable)
Other than that, I think this updated version does the job:
#!/usr/bin/env python
from sys import argv
import re
import os
import argparse #requires Python 2.7 or above
parser = argparse.ArgumentParser(description='filters a text file on the search phrase')
parser.add_argument('-s','--search', help='search phrase or keyword to match',required=True)
parser.add_argument('-f','--filename', help='input file name',required=True)
parser.add_argument('-v','--verbose', help='display output to the screen too', required=False, action="store_true")
args = parser.parse_args()
keyword = args.search
original_file = args.filename
verbose = args.verbose
base_file, ext = os.path.splitext(original_file)
new_file = base_file + ".filtered" + ext
regex_c = re.compile(keyword)
with open(original_file) as fi:
with open(new_file, 'w') as fo:
for line in fi:
result = regex_c.search(line)
if(result):
fo.write(line)
if(verbose):
print(line)
Can this be easily improved?

Well, you know, you have answered most of your questions yourself already :)
For regular expression matching use re module (the doc has pretty explanatory examples).
You already have made use open() function for opening a file. Use the same function for open files for writing, just provide a corresponding mode parameter ("w" or "a" combined with "+" if you need, see help(open) in the Python interactive shell). That's it.

Is it possible to have an optional with/as statement in python?

Instead of this:
FILE = open(f)
do_something(FILE)
FILE.close()
it's better to use this:
with open(f) as FILE:
do_something(FILE)
What if I have something like this?
if f is not None:
FILE = open(f)
else:
FILE = None
do_something(FILE)
if FILE is not None:
FILE.close()
Where do_something also has an "if FILE is None" clause, and still does something useful in that case - I don't want to just skip do_something if FILE is None.
Is there a sensible way of converting this to with/as form? Or am I just trying to solve the optional file problem in a wrong way?

If you were to just write it like this:
if f is not None:
with open(f) as FILE:
do_something(FILE)
else:
do_something(f)
(file is a builtin btw )
Update
Here is a funky way to do an on-the-fly context with an optional None that won't crash:
from contextlib import contextmanager
none_context = contextmanager(lambda: iter([None]))()
# <contextlib.GeneratorContextManager at 0x1021a0110>
with (open(f) if f is not None else none_context) as FILE:
do_something(FILE)
It creates a context that returns a None value. The with will either produce FILE as a file object, or a None type. But the None type will have a proper __exit__
Update
If you are using Python 3.7 or higher, then you can declare the null context manager for stand-in purposes in a much simpler way:
import contextlib
none_context = contextlib.nullcontext()
You can read more about these here:
https://docs.python.org/3.7/library/contextlib.html#contextlib.nullcontext

Since Python 3.7, you can also do
from contextlib import nullcontext
with (open(file) if file else nullcontext()) as FILE:
# Do something with `FILE`
pass
See the official documentation for more details.

This seems to solve all of your concerns.
if file_name is not None:
with open(file_name) as fh:
do_something(fh)
else:
do_something(None)

something like:
if file: #it checks for None,false values no need of "if file is None"
with open(file) as FILE:
do_something(FILE)
else:
FILE=None

In Python 3.3 and above, you can use contextlib.ExitStack to handle this scenario nicely
with contextlib.ExitStack() as stack:
FILE = stack.enter_context(open(f)) if f else None
do_something(FILE)

Python 3.7 supports contextlib.nullcontext, which can be used to avoid creating your own dummy context manager.
This examples shows how you can conditionally open a file or use the stdout:
import contextlib
import sys
def write_to_file_or_stdout(filepath=None, data):
with (
open(filepath, 'w') if filepath is not None else
contextlib.nullcontext(sys.stdout)
) as file_handle:
file_handle.write(data)
contextlib.nullcontext() can be called without any arguments if the value can be None.

While all of the other answers are excellent, and preferable, note that the with expression may be any expression, so you can do:
with (open(file) if file is not None else None) as FILE:
pass
Note that if the else clause were evaluated, to yield None this would result in an exception, because NoneType does not support the appropriate operations to be used as a context manager.

python search for a string and append to it using regular expression

I need to search for a certain parameter known as jvm_args in a configuration file known as config.ini
**contents of config.ini:
first_paramter=some_value1
second_parameter=some_value2
jvm_args=some_value3**
I need to know how to find this parameter in my file and append something to its value, (i.e append a string to the string some_value3).

If you "just" want to find keys and values in an ini file, I think the configparser module is a better bet than using regexps. The configparser asserts that the file has "sections", though.
Documentation for configparser is here: http://docs.python.org/library/configparser.html - useful examples at the bottom. The configparser can also be used for setting values and writing out a new .ini-file.
Input file:
$ cat /tmp/foo.ini
[some_section]
first_paramter = some_value1
second_parameter = some_value2
jvm_args = some_value3
Code:
#!/usr/bin/python3
import configparser
config = configparser.ConfigParser()
config.read("/tmp/foo.ini")
jvm_args = config.get('some_section', 'jvm_args')
print("jvm_args was: %s" % jvm_args)
config.set('some_section', 'jvm_args', jvm_args + ' some_value4')
with open("/tmp/foo.ini", "w") as fp:
config.write(fp)
Output file:
$ cat /tmp/foo.ini
[some_section]
first_paramter = some_value1
second_parameter = some_value2
jvm_args = some_value3 some_value4

You can use re.sub
import re
import os
file = open('config.ini')
new_file = open('new_config.ini', 'w')
for line in file:
new_file.write(re.sub(r'(jvm_args)\s*=\s*(\w+)', r'\1=\2hello', line))
file.close()
new_file.close()
os.remove('config.ini')
os.rename('new_config.ini', 'config.ini')
also check ConfigParser

As both avasal and tobixen have suggested, you can use the python ConfigParser module to do this. For example, I took this "config.ini" file:
[section]
framter = some_value1
second_parameter = some_value2
jvm_args = some_value3**
and ran this python script:
import ConfigParser
p = ConfigParser.ConfigParser()
p.read("config.ini")
p.set("section", "jvm_args", p.get("section", "jvm_args") + "stuff")
with open("config.ini", "w") as f:
p.write(f)
and the contents of the "config.ini" file after running the script was:
[section]
framter = some_value1
second_parameter = some_value2
jvm_args = some_value3**stuff

without regex you can try:
with open('data1.txt','r') as f:
x,replace=f.read(),'new_entry'
ind=x.index('jvm_args=')+len('jvm_args=')
end=x.find('\n',ind) if x.find('\n',ind)!=-1 else x.rfind('',ind)
x=x.replace(x[ind:end],replace)
with open('data1.txt','w') as f:
f.write(x)