Develop Reference

PyYAML: Custom/default constructor for nodes without tags

I have a YAML file for storing constants. Some of the entries have custom tags such as !HANDLER or !EXPR and these are easily handled by adding constructors to the YAML loader.
However, I also want to have a custom constructor for non-tagged nodes. Reason being, I want to add these non-tagged values to a dictionary for use elsewhere. These values need to be available before parsing finishes hence I can't just let parsing finish and then update the dictionary.
So with a YAML file like
sample_rate: 16000
input_file: !HANDLER
handler_fn: file_loader
handler_input: path/to/file
mode: w
I have a handler constructor
def file_handler_loader(loader, node):
params = loader.construct_mapping(node)
module = __import__('handlers.file_handlers', fromlist=[params.pop('handler_fn')])
func = getattr(module, params.pop('handler_fn'))
handler_input = params.pop('handler_input')
return func(handler_input, **params)
And a function initialize_constants
def _get_loader():
loader = FullLoader
loader.add_constructor('!HANDLER', file_handler_loader)
loader.add_constructor('!EXPR', expression_loader)
return loader
def initialize_constants(path_to_yaml: str) -> None:
try:
with open(path_to_yaml, 'r') as yaml_file:
constants = yaml.load(yaml_file, Loader=_get_loader())
except FileNotFoundError as ex:
LOGGER.error(ex)
exit(-1)
The goal is then to have a constructor for non-tagged entries in the YAML. I haven't been able to figure out though how to add a constructor for non-tagged entries. Ideally, the code would look like below
def default_constructor(loader, node):
param = loader.construct_scalar(node)
constants[node_name] = param
I've also attempted to add an resolver to solve the problem. The code below was tested but didn't work as expected.
loader.add_constructor('!DEF', default_constructor)
loader.add_implicit_resolver('!DEF', re.compile('.*'), first=None)
def default_constructor(loader, node):
# do stuff
In this case what happened was the node contained the value sample_rate and not the 16000 as expected.
Thanks in advance :)

Python: Not able to read properties from property file

I'm trying to read configurations from a property file and store those properties in a variable so that it can be accessed from any other class.
I'm able to read the configuration from the config file and print the same but I'm getting an exception when those variables are accessed from some other class.
my config file
Config.cfg.txt
[Ysl_Leader]
YSL_LEADER=192
Generic class where i will store my properties in a variable.
ConfigReader.py
import configparser
class DockerDetails:
config = configparser.RawConfigParser()
_SECTION = 'Ysl_Leader'
config.read('Config.cfg.txt')
YSL_Leader = config.get('Ysl_Leader', 'YSL_LEADER')
print(YSL_Leader)
Another class where I'm trying to get the get the 'YSL_Leader' value
def logger(request):
print(ConfigReader.DockerDetails.YSL_Leader)
Exception:
File "C:\Users\pvivek\AppData\Local\Programs\Python\Python37-32\lib\configparser.py", line 780, in get
d = self._unify_values(section, vars)
File "C:\Users\pvivek\AppData\Local\Programs\Python\Python37-32\lib\configparser.py", line 1146, in _unify_values
raise NoSectionError(section) from None
configparser.NoSectionError: No section: 'Ysl_Leader'
FYI: I'm not getting any exception when I run ConfigReader.py alone

analyzing your question you try to create an environment file, if it is the case because you are using a class to read the file, you must perform this operation in its constructor (remember to make the reference self) and instantiate to be able to access its values, You can perfectly use a function to perform this reading, remembering that to access the result can be treated as a dictionary
configuration file name = (config.ini)
[DEFAULT]
ANY = ANY
[Ysl_Leader]
YSL_LEADER = 192
[OTHER]
VALUE = value_config
# using classes
class Cenv(object):
"""
[use the constructor to start reading the file]
"""
def __init__(self):
self.config = configparser.ConfigParser()
self.config.read('config.ini')
# using functions
def Fenv():
config = configparser.ConfigParser()
config.read('config.ini')
return config
def main():
# to be able to access it is necessary to instantiate the class
instance = Cenv()
cfg = instance.config
# access the elements using the key (like dictionaries)
print(cfg['Ysl_Leader']['YSL_LEADER'])
print(cfg['OTHER']['VALUE'])
# call function and assign returned values
cfg = Fenv()
print(cfg['Ysl_Leader']['YSL_LEADER'])
print(cfg['OTHER']['VALUE'])
# in the case of django assign values ​​in module settings
if __name__ == '__main__':
main()
you can interpret the result as follows (dictionary)
{
"Ysl_Leader": {
"YSL_LEADER": "192"
},
"OTHER": {
"VALUE": "value_config"
}
}

Python: File being saved as empty when it should be saved with text in it, but only when an unrelated method is run before it

Observe the following Python file:
# configmanager.py
"""
ConfigManager controls the modification and validation of config files.
"""
import os
from ruamel import yaml
from voluptuous import Schema
class ConfigManager():
"""
Controls all interaction with configuration files
"""
def __init__(self):
super().__init__()
self.configvalidator = ConfigValidator()
# The config directory inside users home directory.
# Config files will be stored here.
config_dir = os.path.expanduser('~')+'/.config/MyProject/'
# The default config file
config_file = config_dir+'myproject.conf'
# The default configuration
default_config = {
'key1': {},
'key2': {}
}
def _get_config(self):
"""
Get the config file and return it as python dictionary.
Will create the config directory and default config file if they
do not exist.
"""
# Create config directory if it does not exist
if not os.path.exists(self.config_dir):
os.makedirs(self.config_dir)
# Create default config file if it does not exist
if not os.path.isfile(self.config_file):
config_file = open(self.config_file, 'w')
config_file.write(yaml.dump(self.default_config))
# Open config file, and load from YAML
config_file = open(self.config_file, 'r')
config = yaml.safe_load(config_file)
# Validate config
self.configvalidator.validate(config)
return config
def _save_config(self, config):
"""
Save the config file to disk as YAML
"""
# Open current config file
config_file = open(self.config_file, 'w')
# Validate new config
# THE ERROR IS HERE
# If this runs then the config file gets saved as an empty file.
self.configvalidator.validate(config)
# This shows that the 'config' variable still holds the data
print(config)
# This shows that yaml.dump() is working correctly
print(yaml.dump(config))
config_file.write(yaml.dump(config))
def edit_config(self):
"""
Edits the configuration file
"""
config = self._get_config()
# Perform manipulation on config here
# No actual changes to the config are necessary for the bug to occur
self._save_config(config)
class ConfigValidator():
def __init__(self):
super().__init__()
# Config file schema
# Used for validating the config file with voluptuous
self.schema = Schema({
'key1': {},
'key2': {},
})
def validate(self, config):
"""
Validates the data against the defined schema.
"""
self.schema(config)
app = ConfigManager()
app.edit_config()
-
# ~/.config/MyProject/myproject.conf
key1: {}
key2: {}
Description of my module
This is a module I am working on which is for modifying the config file for my project. It accesses the file in ~/.config/MyProject/myproject.conf, which is saved in YAML format, and stores various pieces of information that are used by my program. I have removed as much of the code as I can, leaving only that necessary for understanding the bug.
ConfigManager
ConfigManager is the class containing methods for manipulating my config file. Here it contains three methods: _get_config(), _save_config(), and edit_config(). When instantised, it will get an instance of ConfigValidator (described below), and assign it to self.configvalidator.
_get_config
_get_config() simply opens the file defined by the class variables, specifically ~/.config/MyProject/myproject.conf, or creates the file with default values if it does not exist. The file is saved in YAML format, so this method loads it into a python object, using ruamel.yaml, validates it using self.configvalidator.validate(config) and returns it for use by other pieces of code.
_save_config
_save_config() is where the error occurs, which is described in detail below. It's purpose is to validate the given data, and if it is valid, save it to disk in YAML format.
edit_config
This a generic function, which, in my program, would make specific changes to my config file, depending on the arguments given. In my example, this function simply gets the config file with self._get_config(), and then saves it using self._save_config, without making any changes.
ConfigValidator
This class is for validating my config file using voluptuous. When instantised, it will create the schema that is to be used, and assign it to self.schema. When the validate method is run, it validates the given data using voluptuous.
The error
Observe the line self.configvalidator.validate(config) in ConfigManager._save_config(). This will validate the given data against the schema, and raise an error if it does not pass validation.
But, in the following line config_file.write(yaml.dump(config)), which simply saves the given data to a file as YAML, it will instead save an empty file to disk. (Note: the file is empty, not deleted)
If I disable the validation, by commenting out self.configvalidator.validate(config), then the file is written correctly as YAML.
If self.configvalidator.validate(config) is run, then the config file is saved as an empty file.
My testing
As can be seen with the line print(config), the data in the variable config does not change after being used for validation, yet when being saved to disk, it would seem that config is an empty variable.
print(yaml.dump(config)) shows that ruamel.yaml does work correctly.
If I change edit_config to give invalid data to _save_config, then self.configvalidator.validate(config) will raise an error, as expected. self.configvalidator.validate(config) is running correctly.
End
If self.configvalidator.validate(config) is run, then config_file.write(yaml.dump(config)) saves the config file as an empty file, despite the data in the variable config not changing.
If self.configvalidator.validate(config) is not run, then config_file.write(yaml.dump(config)) saves the file correctly.
That is my error, and it makes absolutely no sense to me.
If your keen to help, then configmanager.py should run correctly (with the error) on your machine, as long as it has access to ruamel.yaml and voluptuous. It will create ~/.config/MyProject/myproject.conf, then save it as empty. Save my example myproject.conf to see how it is then saved as empty when configmanager.py is run. If configmanager.py is run again, when myproject.conf is empty, then a validation error will be raised in _get_config, as expected.
I am so confused by this bug, so if you have any insight it would be greatly appreciated.
Cheers

embedding resources in python scripts

I'd like to figure out how to embed binary content in a python script. For instance, I don't want to have any external files around (images, sound, ... ), I want all this content living inside of my python scripts.
Little example to clarify, let's say I got this small snippet:
from StringIO import StringIO
from PIL import Image, ImageFilter
embedded_resource = StringIO(open("Lenna.png", "rb").read())
im = Image.open(embedded_resource)
im.show()
im_sharp = im.filter(ImageFilter.SHARPEN)
im_sharp.show()
As you can see, the example is reading the external file 'Lenna.png'
Question
How to proceed to embed "Lenna.png" as a resource (variable) into my python script. What's the fastest way to achieve this simple task using python?

You might find the following class rather useful for embedding resources in your program. To use it, call the package method with paths to the files that you want to embed. The class will print out a DATA attribute that should be used to replace the one already found in the class. If you want to add files to your pre-built data, use the add method instead. To use the class in your program, make calls to the load method using context manager syntax. The returned value is a Path object that can be used as a filename argument to other functions or for the purpose of directly loading the reconstituted file. See this SMTP Client for example usage.
import base64
import contextlib
import pathlib
import pickle
import pickletools
import sys
import zlib
class Resource:
"""Manager for resources that would normally be held externally."""
WIDTH = 76
__CACHE = None
DATA = b''
#classmethod
def package(cls, *paths):
"""Creates a resource string to be copied into the class."""
cls.__generate_data(paths, {})
#classmethod
def add(cls, *paths):
"""Include paths in the pre-generated DATA block up above."""
cls.__preload()
cls.__generate_data(paths, cls.__CACHE.copy())
#classmethod
def __generate_data(cls, paths, buffer):
"""Load paths into buffer and output DATA code for the class."""
for path in map(pathlib.Path, paths):
if not path.is_file():
raise ValueError('{!r} is not a file'.format(path))
key = path.name
if key in buffer:
raise KeyError('{!r} has already been included'.format(key))
with path.open('rb') as file:
buffer[key] = file.read()
pickled = pickle.dumps(buffer, pickle.HIGHEST_PROTOCOL)
optimized = pickletools.optimize(pickled)
compressed = zlib.compress(optimized, zlib.Z_BEST_COMPRESSION)
encoded = base64.b85encode(compressed)
cls.__print(" DATA = b'''")
for offset in range(0, len(encoded), cls.WIDTH):
cls.__print("\\\n" + encoded[
slice(offset, offset + cls.WIDTH)].decode('ascii'))
cls.__print("'''")
#staticmethod
def __print(line):
"""Provides alternative printing interface for simplicity."""
sys.stdout.write(line)
sys.stdout.flush()
#classmethod
#contextlib.contextmanager
def load(cls, name, delete=True):
"""Dynamically loads resources and makes them usable while needed."""
cls.__preload()
if name not in cls.__CACHE:
raise KeyError('{!r} cannot be found'.format(name))
path = pathlib.Path(name)
with path.open('wb') as file:
file.write(cls.__CACHE[name])
yield path
if delete:
path.unlink()
#classmethod
def __preload(cls):
"""Warm up the cache if it does not exist in a ready state yet."""
if cls.__CACHE is None:
decoded = base64.b85decode(cls.DATA)
decompressed = zlib.decompress(decoded)
cls.__CACHE = pickle.loads(decompressed)
def __init__(self):
"""Creates an error explaining class was used improperly."""
raise NotImplementedError('class was not designed for instantiation')

The best way to go about this is converting your picture into a python string, and have it in a separate file called something like resources.py, then you simply parse it.
If you are looking to embed the whole thing inside a single binary, then you're looking at something like py2exe. Here is an example embedding external files
In the first scenario, you could even use base64 to (de)code the picture, something like this:
import base64
file = open('yourImage.png');
encoded = base64.b64encode(file.read())
data = base64.b64decode(encoded) # Don't forget to file.close() !

Reading YAML config file in python and using variables

Say I have a yaml config file such as:
test1:
minVolt: -1
maxVolt: 1
test2:
curr: 5
volt: 5
I can read the file into python using:
import yaml
with open("config.yaml", "r") as f:
config = yaml.load(f)
Then I can access the variables with
config['test1']['minVolt']
Style-wise, what is the best way to use variables from the config file? I will be using the variables in multiple modules. If I simply access the variables as shown above, if something is renamed, I will need to rename every instance of the variable.
Just wondering what the best or common practices for using variables from a config file in different modules.

You can do this:
class Test1Class:
def __init__(self, raw):
self.minVolt = raw['minVolt']
self.maxVolt = raw['maxVolt']
class Test2Class:
def __init__(self, raw):
self.curr = raw['curr']
self.volt = raw['volt']
class Config:
def __init__(self, raw):
self.test1 = Test1Class(raw['test1'])
self.test2 = Test2Class(raw['test2'])
config = Config(yaml.safe_load("""
test1:
minVolt: -1
maxVolt: 1
test2:
curr: 5
volt: 5
"""))
And then access your values with:
config.test1.minVolt
When you rename the values in the YAML file, you only need to change the classes at one place.
Note: PyYaml also allows you to directly deserialize YAML to custom classes. However, for that to work, you'd need to add tags to your YAML file so that PyYaml knows which classes to deserialize to. I expect that you do not want to make your YAML input more complex.

See Munch, Load YAML as nested objects instead of dictionary in Python
import yaml
from munch import munchify
c = munchify(f)yaml.safe_load(…))
print(c.test1.minVolt)
# -1
# Or
f = open(…)
c = Munch.fromYAML(f)