Using Python's string.Template class - how might I utilize the ${} for fields in a dictionary that contain spaces?
E.g.
t = string.Template("hello ${some field}")
d = { "some field": "world" }
print( t.substitute(d) ) # Returns "invalid placeholder in string"
Edit: Here's the closest I could get, with the caveat being that all variables need to be wrapped in a brackets (otherwise all space separated words would be matched).
class MyTemplate(string.Template):
delimiter = '$'
idpattern = '[_a-z][\s_a-z0-9]*'
t = MyTemplate("${foo foo} world ${bar}")
s = t.substitute({ "foo foo": "hello", "bar": "goodbye" })
# hello world goodbye
Just in case this might be helpful to somebody else. In python 3 you can use format_map:
t = "hello {some field}"
d = { "some field": "world" }
print( t.format_map(d) )
# hello world
From Doc it says that we could use Template option
https://docs.python.org/dev/library/string.html#template-strings
import string
class MyTemplate(string.Template):
delimiter = '%'
idpattern = '[a-z]+ [a-z]+'
t = MyTemplate('%% %with_underscore %notunderscored')
d = { 'with_underscore':'replaced',
'notunderscored':'not replaced',
}
print t.safe_substitute(d)
I am doing a task in python (learning phase) wherein i have a text file with list of ip's eg:
10.8.9.0
10.7.8.7
10.4.5.6 and so on. Each on one line , one below another.
I have to read its contents and create its json as [{"ip":"10.8.9.0"},{"ip":"10.7.8.7"}..]
Code:
with open("filename.txt") as file:
content = [x.strip('\n') for x in file.readlines()]
print content
print "content",type(content)
content_json=json.dumps(content)
print content_json
print type(content_json)
The output of content is ['ip adrress1','ip address2'] which is a list.
When i dump the list in content_json the type shown is "Str" .
However i need it as json
My concern is - my further task is to validate ip and add a item in existing json stating {"status":"valid/invalid"}.
I dnt know how to do that as the type of my json is showing str.
Kindly let me knw how to proceed and add status for every ip in existing json.
Also i wish to know why is the type of the json i dumped my list with is being showed as str.
The desired output should be
[
{
"ip":"10.8.9.0",
"status":"valid"
},
{
"ip":"10.7.8.A",
"status":"invalid"
}, ..so on
]
First thing: The result is a list because you're building a list with
[x.strip('\n') for x in file.readlines()]. In case you're not sure that means: Take every line x in file, remove the \n character from it and then build a list of those results. You want something like [{"ip":x.strip('\n')} for x in file.readlines()].
Now, the function json.dumps takes a Python object and attempts to create a JSON representation of it. That representation is serialized as a string so if you ask for the type of content_json that's what you'll get.
You have to make the distinction between a python list/dictionary and a JSON string.
This
>>> with open('input.txt') as inp:
... result = [dict(ip=ip.strip()) for ip in inp]
...
>>> result
[{'ip': '10.8.9.0'}, {'ip': '10.7.8.7'}, {'ip': '10.4.5.6'}]
will give you a list of dictionaries that is easy to mutate. When you are done with it, you can dump it as a JSON string:
>>> result[1]['status'] = 'valid'
>>> result
[{'ip': '10.8.9.0'}, {'status': 'valid', 'ip': '10.7.8.7'}, {'ip': '10.4.5.6'}]
>>> json.dumps(result)
'[{"ip": "10.8.9.0"}, {"status": "valid", "ip": "10.7.8.7"}, {"ip": "10.4.5.6"}]'
You should supply key:value properly for the dump. Putting just the value alone would store it as String
Refer this :
https://docs.python.org/2/library/json.html
Maybe something like this?
import json
import socket
result = list()
with open("filename.txt") as file:
for line in file:
ip = line.strip()
try:
socket.inet_aton(ip)
result.append({"ip": line.strip(), "status": "valid"})
except socket.error:
result.append({"ip": line.strip(), "status": "invalid"})
print(json.dumps(result))
Finally, I got a fix:
import os
import sys
import json
from IPy import IP
filepath="E:/Work/"
filename="data.txt"
result = list()
with open(os.path.join(filepath+filename)) as file:
for line in file:
ip = line.strip()
if ip.startswith("0"):
result.append({"ip": line.strip(), "status": "invalid"})
else:
try:
ip_add=IP(ip)
result.append({"ip": line.strip(), "status": "Valid"})
except ValueError:
result.append({"ip": line.strip(), "status": "invalid"})
print(json.dumps(result))
What is the best way to convert a log string back to the LogRecord that caused that string to be generated in the first place, provided that I have the formatter string.
I know I can use regex for this, but I was wondering if there's a better way to achieve this.
Formatter:
%(asctime)s--%(name)s--%(levelname)s--%(funcName)s:%(lineno)s---%(message)s
Sample:
2014-07-28 16:46:39,221--sys.log--DEBUG--hello:61---hello world
Regex:
^(?P<asctime>.*?)--(?P<name>.*?)--(?P<levelname>.*?)--(?P<funcName>.*?):(?P<lineno>.*?)---(?P<message>.*?)$
Regex example:
import re
pattern = re.compile('^(?P<asctime>.*?)--(?P<name>.*?)--(?P<levelname>.*?)--(?P<funcName>.*?):(?P<lineno>.*?)---(?P<message>.*?)$')
print pattern.match('2014-07-28 16:46:39,221--sys.log--DEBUG--hello:61---hello world').groupdict()
Output:
{'name': 'sys.log', 'funcName': 'hello', 'lineno': '61', 'asctime': '2014-07-2816:46:39,221', 'message': 'hello world', 'levelname': 'DEBUG'}
References:
https://docs.python.org/2/library/logging.html
https://docs.python.org/2/howto/logging.html
For this example, just split at the double-dashes:
sample = '2014-07-28 16:46:39,221--sys.log--DEBUG--hello:61---hello world'
fields = ('asctime', 'name', 'levelname', 'funcName', 'message')
values = { k: v for k, v in zip(fields, sample.split('--', len(fields) - 1)) }
# and do some mending
values['funcName'], values['lineno'] = values['funcName'].split(':')
values['message'] = values['message'][1:]
>>> values
{'asctime': '2014-07-28 16:46:39,221',
'funcName': 'hello',
'levelname': 'DEBUG',
'lineno': '61',
'message': 'hello world',
'name': 'sys.log'}
Is there a way to define a XPath type query for nested python dictionaries.
Something like this:
foo = {
'spam':'eggs',
'morefoo': {
'bar':'soap',
'morebar': {'bacon' : 'foobar'}
}
}
print( foo.select("/morefoo/morebar") )
>> {'bacon' : 'foobar'}
I also needed to select nested lists ;)
This can be done easily with #jellybean's solution:
def xpath_get(mydict, path):
elem = mydict
try:
for x in path.strip("/").split("/"):
try:
x = int(x)
elem = elem[x]
except ValueError:
elem = elem.get(x)
except:
pass
return elem
foo = {
'spam':'eggs',
'morefoo': [{
'bar':'soap',
'morebar': {
'bacon' : {
'bla':'balbla'
}
}
},
'bla'
]
}
print xpath_get(foo, "/morefoo/0/morebar/bacon")
[EDIT 2016] This question and the accepted answer are ancient. The newer answers may do the job better than the original answer. However I did not test them so I won't change the accepted answer.
One of the best libraries I've been able to identify, which, in addition, is very actively developed, is an extracted project from boto: JMESPath. It has a very powerful syntax of doing things that would normally take pages of code to express.
Here are some examples:
search('foo | bar', {"foo": {"bar": "baz"}}) -> "baz"
search('foo[*].bar | [0]', {
"foo": [{"bar": ["first1", "second1"]},
{"bar": ["first2", "second2"]}]}) -> ["first1", "second1"]
search('foo | [0]', {"foo": [0, 1, 2]}) -> [0]
There is an easier way to do this now.
http://github.com/akesterson/dpath-python
$ easy_install dpath
>>> dpath.util.search(YOUR_DICTIONARY, "morefoo/morebar")
... done. Or if you don't like getting your results back in a view (merged dictionary that retains the paths), yield them instead:
$ easy_install dpath
>>> for (path, value) in dpath.util.search(YOUR_DICTIONARY, "morefoo/morebar", yielded=True)
... and done. 'value' will hold {'bacon': 'foobar'} in that case.
Not exactly beautiful, but you might use sth like
def xpath_get(mydict, path):
elem = mydict
try:
for x in path.strip("/").split("/"):
elem = elem.get(x)
except:
pass
return elem
This doesn't support xpath stuff like indices, of course ... not to mention the / key trap unutbu indicated.
There is the newer jsonpath-rw library supporting a JSONPATH syntax but for python dictionaries and arrays, as you wished.
So your 1st example becomes:
from jsonpath_rw import parse
print( parse('$.morefoo.morebar').find(foo) )
And the 2nd:
print( parse("$.morefoo[0].morebar.bacon").find(foo) )
PS: An alternative simpler library also supporting dictionaries is python-json-pointer with a more XPath-like syntax.
dict > jmespath
You can use JMESPath which is a query language for JSON, and which has a python implementation.
import jmespath # pip install jmespath
data = {'root': {'section': {'item1': 'value1', 'item2': 'value2'}}}
jmespath.search('root.section.item2', data)
Out[42]: 'value2'
The jmespath query syntax and live examples: http://jmespath.org/tutorial.html
dict > xml > xpath
Another option would be converting your dictionaries to XML using something like dicttoxml and then use regular XPath expressions e.g. via lxml or whatever other library you prefer.
from dicttoxml import dicttoxml # pip install dicttoxml
from lxml import etree # pip install lxml
data = {'root': {'section': {'item1': 'value1', 'item2': 'value2'}}}
xml_data = dicttoxml(data, attr_type=False)
Out[43]: b'<?xml version="1.0" encoding="UTF-8" ?><root><root><section><item1>value1</item1><item2>value2</item2></section></root></root>'
tree = etree.fromstring(xml_data)
tree.xpath('//item2/text()')
Out[44]: ['value2']
Json Pointer
Yet another option is Json Pointer which is an IETF spec that has a python implementation:
https://github.com/stefankoegl/python-json-pointer
From the jsonpointer-python tutorial:
from jsonpointer import resolve_pointer
obj = {"foo": {"anArray": [ {"prop": 44}], "another prop": {"baz": "A string" }}}
resolve_pointer(obj, '') == obj
# True
resolve_pointer(obj, '/foo/another%20prop/baz') == obj['foo']['another prop']['baz']
# True
>>> resolve_pointer(obj, '/foo/anArray/0') == obj['foo']['anArray'][0]
# True
If terseness is your fancy:
def xpath(root, path, sch='/'):
return reduce(lambda acc, nxt: acc[nxt],
[int(x) if x.isdigit() else x for x in path.split(sch)],
root)
Of course, if you only have dicts, then it's simpler:
def xpath(root, path, sch='/'):
return reduce(lambda acc, nxt: acc[nxt],
path.split(sch),
root)
Good luck finding any errors in your path spec tho ;-)
Another alternative (besides that suggested by jellybean) is this:
def querydict(d, q):
keys = q.split('/')
nd = d
for k in keys:
if k == '':
continue
if k in nd:
nd = nd[k]
else:
return None
return nd
foo = {
'spam':'eggs',
'morefoo': {
'bar':'soap',
'morebar': {'bacon' : 'foobar'}
}
}
print querydict(foo, "/morefoo/morebar")
More work would have to be put into how the XPath-like selector would work.
'/' is a valid dictionary key, so how would
foo={'/':{'/':'eggs'},'//':'ham'}
be handled?
foo.select("///")
would be ambiguous.
Is there any reason for you to the query it the way like the XPath pattern? As the commenter to your question suggested, it just a dictionary, so you can access the elements in a nest manner. Also, considering that data is in the form of JSON, you can use simplejson module to load it and access the elements too.
There is this project JSONPATH, which is trying to help people do opposite of what you intend to do (given an XPATH, how to make it easily accessible via python objects), which seems more useful.
def Dict(var, *arg, **kwarg):
""" Return the value of an (imbricated) dictionnary, if all fields exist else return "" unless "default=new_value" specified as end argument
Avoid TypeError: argument of type 'NoneType' is not iterable
Ex: Dict(variable_dict, 'field1', 'field2', default = 0)
"""
for key in arg:
if isinstance(var, dict) and key and key in var: var = var[key]
else: return kwarg['default'] if kwarg and 'default' in kwarg else "" # Allow Dict(var, tvdbid).isdigit() for example
return kwarg['default'] if var in (None, '', 'N/A', 'null') and kwarg and 'default' in kwarg else "" if var in (None, '', 'N/A', 'null') else var
foo = {
'spam':'eggs',
'morefoo': {
'bar':'soap',
'morebar': {'bacon' : 'foobar'}
}
}
print Dict(foo, 'morefoo', 'morebar')
print Dict(foo, 'morefoo', 'morebar', default=None)
Have a SaveDict(value, var, *arg) function that can even append to lists in dict...
I reference form this link..
Following code is for json xpath base parse implemented in python :
import json
import xmltodict
# Parse the json string
class jsonprase(object):
def __init__(self, json_value):
try:
self.json_value = json.loads(json_value)
except Exception :
raise ValueError('must be a json str value')
def find_json_node_by_xpath(self, xpath):
elem = self.json_value
nodes = xpath.strip("/").split("/")
for x in range(len(nodes)):
try:
elem = elem.get(nodes[x])
except AttributeError:
elem = [y.get(nodes[x]) for y in elem]
return elem
def datalength(self, xpath="/"):
return len(self.find_json_node_by_xpath(xpath))
#property
def json_to_xml(self):
try:
root = {"root": self.json_value}
xml = xmltodict.unparse(root, pretty=True)
except ArithmeticError :
pyapilog().error(e)
return xml
Test Json :
{
"responseHeader": {
"zkConnected": true,
"status": 0,
"QTime": 2675,
"params": {
"q": "TxnInitTime:[2021-11-01T00:00:00Z TO 2021-11-30T23:59:59Z] AND Status:6",
"stats": "on",
"stats.facet": "CountryCode",
"rows": "0",
"wt": "json",
"stats.field": "ItemPrice"
}
},
"response": {
"numFound": 15162439,
"start": 0,
"maxScore": 1.8660598,
"docs": []
}
}
Test Code to read the values from above input json.
numFound = jsonprase(ABOVE_INPUT_JSON).find_json_node_by_xpath('/response/numFound')
print(numFound)