I'm using ConfigObj in python with Template-style interpolation. Unwrapping my config dictionary via ** doesn't seem to do interpolation. Is this a feature or a bug? Any nice workarounds?
$ cat my.conf
foo = /test
bar = $foo/directory
>>> import configobj
>>> config = configobj.ConfigObj('my.conf', interpolation='Template')
>>> config['bar']
'/test/directory'
>>> '{bar}'.format(**config)
'$foo/directory'
I'd expect the second line to be /test/directory. Why doesn't interpolation work with **kwargs?
When unpacking the keyword argument, then a new object is created: of type dict. This dictionary contains the the raw-values of the configuration (no interpolation)
Demonstration:
>>> id(config)
31143152
>>> def showKeywordArgs(**kwargs):
... print(kwargs, type(kwargs), id(kwargs))
...
>>> showKeywordArgs(**config)
({'foo': '/test', 'bar': '$foo/directory'}, <type 'dict'>, 35738944)
To resolve your problem you could create an expanded version of your configuration like this:
>>> expandedConfig = {k: config[k] for k in config}
>>> '{bar}'.format(**expandedConfig)
'/test/directory'
Another more elegant way is to simply avoid unpacking: This can be achieved by using the function string.Formatter.vformat:
import string
fmt = string.Formatter()
fmt.vformat("{bar}", None, config)
I have had a similar problem.
A workaround is to use configobj's function ".dict()". This works because configobj returns a real dictionary, which Python knows how to unpack.
Your example becomes:
>>> import configobj
>>> config = configobj.ConfigObj('my.conf', interpolation='Template')
>>> config['bar']
'/test/directory'
>>> '{bar}'.format(**config.dict())
'/test/directory'
Related
I am a big fan of using dictionaries to format strings. It helps me read the string format I am using as well as let me take advantage of existing dictionaries. For example:
class MyClass:
def __init__(self):
self.title = 'Title'
a = MyClass()
print 'The title is %(title)s' % a.__dict__
path = '/path/to/a/file'
print 'You put your file here: %(path)s' % locals()
However I cannot figure out the python 3.x syntax for doing the same (or if that is even possible). I would like to do the following
# Fails, KeyError 'latitude'
geopoint = {'latitude':41.123,'longitude':71.091}
print '{latitude} {longitude}'.format(geopoint)
# Succeeds
print '{latitude} {longitude}'.format(latitude=41.123,longitude=71.091)
Is this good for you?
geopoint = {'latitude':41.123,'longitude':71.091}
print('{latitude} {longitude}'.format(**geopoint))
To unpack a dictionary into keyword arguments, use **. Also,, new-style formatting supports referring to attributes of objects and items of mappings:
'{0[latitude]} {0[longitude]}'.format(geopoint)
'The title is {0.title}s'.format(a) # the a from your first example
As Python 3.0 and 3.1 are EOL'ed and no one uses them, you can and should use str.format_map(mapping) (Python 3.2+):
Similar to str.format(**mapping), except that mapping is used directly and not copied to a dict. This is useful if for example mapping is a dict subclass.
What this means is that you can use for example a defaultdict that would set (and return) a default value for keys that are missing:
>>> from collections import defaultdict
>>> vals = defaultdict(lambda: '<unset>', {'bar': 'baz'})
>>> 'foo is {foo} and bar is {bar}'.format_map(vals)
'foo is <unset> and bar is baz'
Even if the mapping provided is a dict, not a subclass, this would probably still be slightly faster.
The difference is not big though, given
>>> d = dict(foo='x', bar='y', baz='z')
then
>>> 'foo is {foo}, bar is {bar} and baz is {baz}'.format_map(d)
is about 10 ns (2 %) faster than
>>> 'foo is {foo}, bar is {bar} and baz is {baz}'.format(**d)
on my Python 3.4.3. The difference would probably be larger as more keys are in the dictionary, and
Note that the format language is much more flexible than that though; they can contain indexed expressions, attribute accesses and so on, so you can format a whole object, or 2 of them:
>>> p1 = {'latitude':41.123,'longitude':71.091}
>>> p2 = {'latitude':56.456,'longitude':23.456}
>>> '{0[latitude]} {0[longitude]} - {1[latitude]} {1[longitude]}'.format(p1, p2)
'41.123 71.091 - 56.456 23.456'
Starting from 3.6 you can use the interpolated strings too:
>>> f'lat:{p1["latitude"]} lng:{p1["longitude"]}'
'lat:41.123 lng:71.091'
You just need to remember to use the other quote characters within the nested quotes. Another upside of this approach is that it is much faster than calling a formatting method.
print("{latitude} {longitude}".format(**geopoint))
Since the question is specific to Python 3, here's using the new f-string syntax, available since Python 3.6:
>>> geopoint = {'latitude':41.123,'longitude':71.091}
>>> print(f'{geopoint["latitude"]} {geopoint["longitude"]}')
41.123 71.091
Note the outer single quotes and inner double quotes (you could also do it the other way around).
The Python 2 syntax works in Python 3 as well:
>>> class MyClass:
... def __init__(self):
... self.title = 'Title'
...
>>> a = MyClass()
>>> print('The title is %(title)s' % a.__dict__)
The title is Title
>>>
>>> path = '/path/to/a/file'
>>> print('You put your file here: %(path)s' % locals())
You put your file here: /path/to/a/file
geopoint = {'latitude':41.123,'longitude':71.091}
# working examples.
print(f'{geopoint["latitude"]} {geopoint["longitude"]}') # from above answer
print('{geopoint[latitude]} {geopoint[longitude]}'.format(geopoint=geopoint)) # alternate for format method (including dict name in string).
print('%(latitude)s %(longitude)s'%geopoint) # thanks #tcll
Use format_map to do what you want
print('{latitude} {longitude}'.format_map(geopoint))
This has the advantage that
the dictionary does not have to be blown up into parameters (compared to **geopoint) and that
the format string only has access to the provided map and not the entire scope of variables (compared to F-strings).
Most answers formatted only the values of the dict.
If you want to also format the key into the string you can use dict.items():
geopoint = {'latitude':41.123,'longitude':71.091}
print("{} {}".format(*geopoint.items()))
Output:
('latitude', 41.123) ('longitude', 71.091)
If you want to format in an arbitry way, that is, not showing the key-values like tuples:
from functools import reduce
print("{} is {} and {} is {}".format(*reduce((lambda x, y: x + y), [list(item) for item in geopoint.items()])))
Output:
latitude is 41.123 and longitude is 71.091
I mean a dictionary where of you could get the value by key or the key by value depending on what you need.
You could use bidict package which provides a bidirectional map. The syntax looks as follows (taken from the documentation):
>>> from bidict import bidict
>>> element_by_symbol = bidict(H='hydrogen')
>>> element_by_symbol
bidict({'H': 'hydrogen'})
>>> element_by_symbol['H']
'hydrogen'
>>> element_by_symbol.inv
bidict({'hydrogen': 'H'})
>>> element_by_symbol.inv['hydrogen']
'H'
>>> element_by_symbol.inv.inv is element_by_symbol
True
Or you can implement it yourself, for example using one of the solutions provided here.
Let's say I have an args namespace after parsing my command line with argparse. Now, I want to use this to create some objects like this:
foo = Foo(bar=args.bar)
Unfortunately, I have the restriction that if a keyword argument is set, it must not be None. Now, I need to check if args.bar is set and act accordingly:
if args.bar:
foo = Foo(bar=args.bar)
else:
foo = Foo()
This is unwieldy and doesn't scale for more arguments. What I'd like to have, is something like this:
foo = Foo(**args.__dict__)
but this still suffers from my initial problem and additionally doesn't work for keys that are not keyword arguments of the __init__ method. Is there a good way to achieve these things?
You could try something like this:
>>> defined_args = {k:v for k,v in args._get_kwargs() if v is not None}
>>> foo = Foo(**defined_args)
For example:
>>> import argparse
>>> args = argparse.Namespace(key1=None,key2='value')
>>> {k:v for k,v in args._get_kwargs() if v is not None}
{'key2': 'value'}
Note, however, that _get_kwargs() is not part of the public API so may or may not be available in future releases/versions.
I think you can use vars():
args = parser.parse_args()
Foo(**vars(args))
vars([object]) returns the namespace as a dictionary
I am a big fan of using dictionaries to format strings. It helps me read the string format I am using as well as let me take advantage of existing dictionaries. For example:
class MyClass:
def __init__(self):
self.title = 'Title'
a = MyClass()
print 'The title is %(title)s' % a.__dict__
path = '/path/to/a/file'
print 'You put your file here: %(path)s' % locals()
However I cannot figure out the python 3.x syntax for doing the same (or if that is even possible). I would like to do the following
# Fails, KeyError 'latitude'
geopoint = {'latitude':41.123,'longitude':71.091}
print '{latitude} {longitude}'.format(geopoint)
# Succeeds
print '{latitude} {longitude}'.format(latitude=41.123,longitude=71.091)
Is this good for you?
geopoint = {'latitude':41.123,'longitude':71.091}
print('{latitude} {longitude}'.format(**geopoint))
To unpack a dictionary into keyword arguments, use **. Also,, new-style formatting supports referring to attributes of objects and items of mappings:
'{0[latitude]} {0[longitude]}'.format(geopoint)
'The title is {0.title}s'.format(a) # the a from your first example
As Python 3.0 and 3.1 are EOL'ed and no one uses them, you can and should use str.format_map(mapping) (Python 3.2+):
Similar to str.format(**mapping), except that mapping is used directly and not copied to a dict. This is useful if for example mapping is a dict subclass.
What this means is that you can use for example a defaultdict that would set (and return) a default value for keys that are missing:
>>> from collections import defaultdict
>>> vals = defaultdict(lambda: '<unset>', {'bar': 'baz'})
>>> 'foo is {foo} and bar is {bar}'.format_map(vals)
'foo is <unset> and bar is baz'
Even if the mapping provided is a dict, not a subclass, this would probably still be slightly faster.
The difference is not big though, given
>>> d = dict(foo='x', bar='y', baz='z')
then
>>> 'foo is {foo}, bar is {bar} and baz is {baz}'.format_map(d)
is about 10 ns (2 %) faster than
>>> 'foo is {foo}, bar is {bar} and baz is {baz}'.format(**d)
on my Python 3.4.3. The difference would probably be larger as more keys are in the dictionary, and
Note that the format language is much more flexible than that though; they can contain indexed expressions, attribute accesses and so on, so you can format a whole object, or 2 of them:
>>> p1 = {'latitude':41.123,'longitude':71.091}
>>> p2 = {'latitude':56.456,'longitude':23.456}
>>> '{0[latitude]} {0[longitude]} - {1[latitude]} {1[longitude]}'.format(p1, p2)
'41.123 71.091 - 56.456 23.456'
Starting from 3.6 you can use the interpolated strings too:
>>> f'lat:{p1["latitude"]} lng:{p1["longitude"]}'
'lat:41.123 lng:71.091'
You just need to remember to use the other quote characters within the nested quotes. Another upside of this approach is that it is much faster than calling a formatting method.
print("{latitude} {longitude}".format(**geopoint))
Since the question is specific to Python 3, here's using the new f-string syntax, available since Python 3.6:
>>> geopoint = {'latitude':41.123,'longitude':71.091}
>>> print(f'{geopoint["latitude"]} {geopoint["longitude"]}')
41.123 71.091
Note the outer single quotes and inner double quotes (you could also do it the other way around).
The Python 2 syntax works in Python 3 as well:
>>> class MyClass:
... def __init__(self):
... self.title = 'Title'
...
>>> a = MyClass()
>>> print('The title is %(title)s' % a.__dict__)
The title is Title
>>>
>>> path = '/path/to/a/file'
>>> print('You put your file here: %(path)s' % locals())
You put your file here: /path/to/a/file
geopoint = {'latitude':41.123,'longitude':71.091}
# working examples.
print(f'{geopoint["latitude"]} {geopoint["longitude"]}') # from above answer
print('{geopoint[latitude]} {geopoint[longitude]}'.format(geopoint=geopoint)) # alternate for format method (including dict name in string).
print('%(latitude)s %(longitude)s'%geopoint) # thanks #tcll
Use format_map to do what you want
print('{latitude} {longitude}'.format_map(geopoint))
This has the advantage that
the dictionary does not have to be blown up into parameters (compared to **geopoint) and that
the format string only has access to the provided map and not the entire scope of variables (compared to F-strings).
Most answers formatted only the values of the dict.
If you want to also format the key into the string you can use dict.items():
geopoint = {'latitude':41.123,'longitude':71.091}
print("{} {}".format(*geopoint.items()))
Output:
('latitude', 41.123) ('longitude', 71.091)
If you want to format in an arbitry way, that is, not showing the key-values like tuples:
from functools import reduce
print("{} is {} and {} is {}".format(*reduce((lambda x, y: x + y), [list(item) for item in geopoint.items()])))
Output:
latitude is 41.123 and longitude is 71.091
How do I serialize a Python dictionary into a string, and then back to a dictionary? The dictionary will have lists and other dictionaries inside it.
It depends on what you're wanting to use it for. If you're just trying to save it, you should use pickle (or, if you’re using CPython 2.x, cPickle, which is faster).
>>> import pickle
>>> pickle.dumps({'foo': 'bar'})
b'\x80\x03}q\x00X\x03\x00\x00\x00fooq\x01X\x03\x00\x00\x00barq\x02s.'
>>> pickle.loads(_)
{'foo': 'bar'}
If you want it to be readable, you could use json:
>>> import json
>>> json.dumps({'foo': 'bar'})
'{"foo": "bar"}'
>>> json.loads(_)
{'foo': 'bar'}
json is, however, very limited in what it will support, while pickle can be used for arbitrary objects (if it doesn't work automatically, the class can define __getstate__ to specify precisely how it should be pickled).
>>> pickle.dumps(object())
b'\x80\x03cbuiltins\nobject\nq\x00)\x81q\x01.'
>>> json.dumps(object())
Traceback (most recent call last):
...
TypeError: <object object at 0x7fa0348230c0> is not JSON serializable
Pickle is great but I think it's worth mentioning literal_eval from the ast module for an even lighter weight solution if you're only serializing basic python types. It's basically a "safe" version of the notorious eval function that only allows evaluation of basic python types as opposed to any valid python code.
Example:
>>> d = {}
>>> d[0] = range(10)
>>> d['1'] = {}
>>> d['1'][0] = range(10)
>>> d['1'][1] = 'hello'
>>> data_string = str(d)
>>> print data_string
{0: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], '1': {0: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], 1: 'hello'}}
>>> from ast import literal_eval
>>> d == literal_eval(data_string)
True
One benefit is that the serialized data is just python code, so it's very human friendly. Compare it to what you would get with pickle.dumps:
>>> import pickle
>>> print pickle.dumps(d)
(dp0
I0
(lp1
I0
aI1
aI2
aI3
aI4
aI5
aI6
aI7
aI8
aI9
asS'1'
p2
(dp3
I0
(lp4
I0
aI1
aI2
aI3
aI4
aI5
aI6
aI7
aI8
aI9
asI1
S'hello'
p5
ss.
The downside is that as soon as the the data includes a type that is not supported by literal_ast you'll have to transition to something else like pickling.
Use Python's json module, or simplejson if you don't have python 2.6 or higher.
If you fully trust the string and don't care about python injection attacks then this is very simple solution:
d = { 'method' : "eval", 'safe' : False, 'guarantees' : None }
s = str(d)
d2 = eval(s)
for k in d2:
print k+"="+d2[k]
If you're more safety conscious then ast.literal_eval is a better bet.
One thing json cannot do is dict indexed with numerals. The following snippet
import json
dictionary = dict({0:0, 1:5, 2:10})
serialized = json.dumps(dictionary)
unpacked = json.loads(serialized)
print(unpacked[0])
will throw
KeyError: 0
Because keys are converted to strings. cPickle preserves the numeric type and the unpacked dict can be used right away.
pyyaml should also be mentioned here. It is both human readable and can serialize any python object.
pyyaml is hosted here:
https://pypi.org/project/PyYAML
While not strictly serialization, json may be reasonable approach here. That will handled nested dicts and lists, and data as long as your data is "simple": strings, and basic numeric types.
A new alternative to JSON or YaML is NestedText. It supports strings that are nested in lists and dictionaries to any depth. It conveys nesting through the use of indenting, and so has no need for either quoting or escaping. As such, the result tends to be very readable. The result looks like YaML, but without all the special cases. It is especially appropriate for serializing code snippets. For example, here is an a single test case extracted from a much larger set that was serialized with NestedText:
base tests:
-
args: --quiet --config test7 files -N configs/subdir
expected:
> Archive: test7-\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\d
> «TESTS»/configs/subdir/
> «TESTS»/configs/subdir/file
Be aware, that integers, floats, and bools are converted to strings.
If you are trying to only serialize then pprint may also be a good option. It requires the object to be serialized and a file stream.
Here's some code:
from pprint import pprint
my_dict = {1:'a',2:'b'}
with open('test_results.txt','wb') as f:
pprint(my_dict,f)
I am not sure if we can deserialize easily. I was using json to serialize and deserialze earlier which works correctly in most cases.
f.write(json.dumps(my_dict, sort_keys = True, indent = 2, ensure_ascii=True))
However, in one particular case, there were some errors writing non-unicode data to json.