I am parsing dictionaries into nested Json (Python 2.7). The problem I have that sometimes there is no value for the variable so it takes None, it depends if the variable came or not. It would be ok, but I want to avoid escape characters before quotations. Before I get nested Json I do json.loads() to remove string escapes.
The problem is to return None if there is no dict.
Example code:
import json
data1 = '{"foo":"bar"}'
data2 = None
sensor = {'header':'Info',
'data1': json.loads(data1),
'data2': json.loads(data2)}
output = json.dumps(sensor)
print(output)
Expected outcome:
{"data2": null, "data1": {"foo": "bar"}, "header": "Info"}
Received error:
Traceback (most recent call last):
File "\jsonDecoding.py", line 7, in <module>
'data2': json.loads(data2)}
File "\Python\Python35\lib\json\__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'NoneType'
Tried solution:
class Decoder(json.JSONDecoder):
def default(self, obj):
if obj is None::
return None
else: return super(Decoder, self).default(obj)
data1 = '{"foo":"bar"}'
data2 = None
sensor = {'header':'Info',
'data1': json.loads(data1),
'data2': json.loads(data2, cls=Decoder)}
output = json.dumps(sensor)
I thought implementing Decoder.default() should solve the problem but it calls the class but does not call the default method.
There are plenty of talks about None in place of key or value but I did not find in place of whole object
I think you make the problem harder than it is. We can construct a None-safe function:
def nonesafe_loads(obj):
if obj is not None:
return json.loads(obj)
This will return the json.loads(obj) in case obj is not None, and otherwise it will return None. Or we can construct a "best effort" JSON loads, that aims to json.loads the string, or in case that fails, returns the obj itself:
def besteffort_loads(obj):
try:
return json.loads(obj)
except (TypeError, ValueError):
return obj
Now we can just use our nonesafe_loads in the program:
data1 = '{"foo":"bar"}'
data2 = None
sensor = {'header':'Info',
'data1': nonesafe_loads(data1),
'data2': nonesafe_loads(data2)}
output = json.dumps(sensor)
Related
I have a piece of code that fetches data from the ticketmaster API using a function I've named get_event_info. The first revision of the code worked as desired, subsequently I modified the original function to make use of header based authentication instead of URL based. I also added a few lines to the function which were intended to validate the response status code. After making these changes the code began producing the following TypeError:
Traceback (most recent call last):
File "ticketmaster_only_w_headers.py", line 146, in <module>
for event in ticket_search["_embedded"]["events"].items():
TypeError: 'NoneType' object is not subscriptable
I've read quite a bit about this type of error but I'm still unable to determine why my code is producing it in this instance. I would really appreciate an explanation on why my code is producing this error and what troubleshooting methods I should have used to uncover the source error. I'm fairly comfortable with programming but certainly no expert so the simpler the language used the better.
(Function Definition)
def get_event_info(search):
if search in CACHE_DICTION:
d = CACHE_DICTION[search]
else:
api_url = '{0}events/'.format(api_url_base)
payload = {"keyword": search, "apikey": api_token,
"format": "json", "dmaId": "366", "size": 200, "radius": "2"}
response = requests.get(api_url, headers=headers, params=payload)
if response.status_code == 200:
d = json.loads(response.text)
CACHE_DICTION[search] = d
f = open(CACHE_FNAME, 'w')
f.write(json.dumps(CACHE_DICTION))
f.close()
else:
d = None
return d
(Code snippet that triggers the error)
ticket_search = get_event_info("")
for event in ticket_search["_embedded"]["events"]:
a = event["id"]
b = event["name"]
if "dateTime" in event["dates"]["start"]:
c = event["dates"]["start"]["dateTime"].replace(
"T", " ").replace("Z", "")
else:
c = "NONE"
if "end" in event["dates"] and "dateTime" in event["dates"]["end"]:
j = event["dates"]["end"]["dateTime"].replace(
"T", " ").replace("Z", "")
else:
j = "NONE"
(Code that creates, opens, and writes to the cache used in the above code)
CACHE_FNAME = "ticketmaster_cache.json"
try:
cache_file = open(CACHE_FNAME, "r")
cache_contents = cache_file.read()
CACHE_DICTION = json.loads(cache_contents)
cache_file.close()
except:
CACHE_DICTION = {}
The previous revision of the get_event_info function shown below which does not produce any TypeError:
def get_event_info(search, ticketmaster_key = ticketmaster_key):
if search in CACHE_DICTION:
d = CACHE_DICTION[search]
else:
data = requests.get("https://app.ticketmaster.com/discovery/v2/events",
params = {"keyword": search, "apikey": ticketmaster_key,
"format":"json", "dmaId": "366", "size": 200, "radius": "2"})
print(data.url)
d = json.loads(data.text)
CACHE_DICTION[search] = d
f = open(CACHE_FNAME, 'w')
f.write(json.dumps(CACHE_DICTION))
f.close()
return d
Traceback & Error message I see when I run the latest revision of the code:
Traceback (most recent call last):
File "ticketmaster_only_w_headers.py", line 146, in <module>
for event in ticket_search["_embedded"]["events"]:
TypeError: 'NoneType' object is not subscriptable
Whenever you have a function that can explicitly return None, you should always check the return value first:
def func(a):
if a == 1:
return list(range(10)) # could return a list
else:
return None # or it could return None
a = 10
f = func(a)
f[1]
# raises TypeError: NoneType is not subscriptable
# check for NoneType first
if f is not None:
print(f[1])
# otherwise, kick out different result
else:
print('Got "None" for f!')
# Got "None" for f!
Your ticket_search is returned as None, but because your for loop is trying to do a key-lookup, it's failing, because None doesn't support that operation. Your logic, following from the above, should look like:
if ticket_search is not None:
for event in ticket_search["_embedded"]["events"]:
a = event["id"]
else:
raise TypeError
# or do something else
Well, the interpreter is explicitly telling you that you are trying to evaluate something like a[i], where a is None (instead of the intended type, like a list or a dict). In your case, it is either ticket_search itself, or ticket_search["_embedded"].
In any case, if you can rerun your code at all, putting a print(ticket_search) under ticket_search = get_event_info("") should make everything clear.
So for testing purposes, I am trying to create a python ActiveResource object from a json file (I want the object to have attributes from the json file). More specifically I am using the ShopifyResource from (https://github.com/Shopify/shopify_python_api), which extends the ActiveResource object.
I looked through the source code and found some functions that I thought would be of some use:
(https://github.com/Shopify/shopify_python_api/blob/master/shopify/base.py)
from pyactiveresource.activeresource import ActiveResource
import shopify.mixins as mixins
class ShopifyResource(ActiveResource, mixins.Countable):
_format = formats.JSONFormat
def _load_attributes_from_response(self, response):
if response.body.strip():
self._update(self.__class__.format.decode(response.body))
where _update is from ActiveResource (https://github.com/Shopify/pyactiveresource/blob/master/pyactiveresource/activeresource.py)
def _update(self, attributes):
"""Update the object with the given attributes.
Args:
attributes: A dictionary of attributes.
Returns:
None
"""
if not isinstance(attributes, dict):
return
for key, value in six.iteritems(attributes):
if isinstance(value, dict):
klass = self._find_class_for(key)
attr = klass(value)
elif isinstance(value, list):
klass = None
attr = []
for child in value:
if isinstance(child, dict):
if klass is None:
klass = self._find_class_for_collection(key)
attr.append(klass(child))
else:
attr.append(child)
else:
attr = value
# Store the actual value in the attributes dictionary
self.attributes[key] = attr
So then I tried to do the following:
order = Order()
with open("file.json")) as json_file:
x = json.loads(json_file.read())
order._update(x)
Where Order extends ShopifyResource (which extends ActiveResource). If am not mistaken x should be a dictionary, which is an approriate parameter for the _update() function.
Yet I get the following output:
raceback (most recent call last):
File "/home/vineet/Documents/project/tests/test_sync.py", line 137, in testSaveOrder1
self.getOrder()
File "/home/vineet/Documents/project/tests/tests/test_sync.py", line 113, in getOrder
order._update(x)
File "/home/vineet/Documents/project/venv/lib/python3.6/site-packages/pyactiveresource/activeresource.py", line 962, in _update
attr.append(klass(child))
File "/home/vineet/Documents/project/venv/lib/python3.6/site-packages/shopify/base.py", line 126, in __init__
prefix_options, attributes = self.__class__._split_options(attributes)
File "/home/vineet/Documents/project/venv/lib/python3.6/site-packages/pyactiveresource/activeresource.py", line 466, in _split_options
if key in cls._prefix_parameters():
File "/home/vineet/Documents/project/venv/lib/python3.6/site-packages/pyactiveresource/activeresource.py", line 720, in _prefix_parameters
for match in template.pattern.finditer(path):
TypeError: cannot use a string pattern on a bytes-like object
I even tried the following:
order._update(order._format.decode(json_file.read()))
But that didn't work since 'str' object has no attribute 'decode'.
It seems You are worried if x has correct format. Print it, and check.
Btw: And use
x = json.load(json_file)
instead of
x = json.loads(json_file.read())
I'm running the following:
for server in server_list:
for item in required_fields:
print item, eval(item)
There is a possibility that some keys may not exist, but worse it's represented on a parent key not the one I'm scanning for.
So I'm scanning the json for the following key:
server['server_management']['server_total_cost_of_ownership']['description']
Which doesn't exist but it's actually the parent that is null:
server['server_management']['server_total_cost_of_ownership']
How do I write my code to account for this? It's not giving a key error. Right now I get the following traceback:
Traceback (most recent call last):
File "C:/projects/blah/scripts/test.py", line 29, in <module>
print item, eval(item)
File "<string>", line 1, in <module>
TypeError: 'NoneType' object has no attribute '__getitem__'
Full code:
import csv
import json
import os
import requests
import sys
required_fields = ["server['server_name']","server['server_info']['asset_type']['display_name']",
"server['asset_status']['display_name']", "server['record_owner']['group_name']",
"server['server_management']['server_total_cost_of_ownership']['description']",
"server['server_management']['primary_business_owner']['name']",
"server['environment']['display_name']", "server['is_virtual']",
"server['managed_by']['display_name']", "server['server_info']['billable_ibm']",
"server['server_info']['billing_sub_type']['display_name']",
"server['server_info']['serial_number']", "server['location']['display_name']",
"server['inception_date']", "server['server_info']['decommission_date']" ]
# Query API for all servers
def get_servers_info():
servers_info = requests.get('url')
return servers_info.json()
def get_server_info(sid):
server_info = requests.get('url')
return server_info.json()
server_list = get_servers_info()
for server in server_list:
for item in required_fields:
print item, eval(item)
In fact you should avoid eval. After the json load since you know the key name, you can use a list to go deeper in the tree.
server['server_management']['primary_business_owner']['name']" => ["server_management', 'primary_business_owner', 'name']
Here a snippet for a json validation against a list of required fields.
data={
"d": {
"p":{
"r":[
"test"
]
}
},
"a": 3
}
def _get_attr(dict_, attrs):
try:
src = attrs[:]
root = attrs.pop(0)
node = dict_[root]
null = object()
for i, attr in enumerate(attrs[:]):
try:
node = node.get(attr, null)
except AttributeError:
node = null
if node is null:
# i+2 pop and last element
raise ValueError("%s not present (level %s)" % (attr, '->'.join(src[: i+2])))
return node
except KeyError:
raise ValueError("%s not present" % root)
# assume list of required field
reqs = [
["d", "p", "r"],
["d"],
["k"],
["d", "p", "r", "e"],
]
for req in reqs:
try:
_get_attr(data, req)
except ValueError as E:
print(E)
# prints
# k not present
# e not present (level d->p->r->e)
Ignoring the context of the code and not understanding the use of eval here, the way to do this is to use .get() and seed it with reasonable defaults.
For example:
server['server_management']['server_total_cost_of_ownership']['description']
Can be:
server.get('server_management', {}).get('server_total_cost_of_ownership', {}).get('description', '')
Then if any of the keys do not exist you will always get back an empty description ''.
Your problem here is totally unrelated to using eval[1]. The exception you get is the same as if the code would have been there directly. What you are running (via eval) is:
a = server['server_management']
b = a['server_total_cost_of_ownership']
c = b['description']
Yet, b is None, so resolving it to c will fail. Like a KeyError, you can also catch a TypeError:
for server in server_list:
for item in required_fields:
try:
print item, eval(item)
except TypeError:
print("Guess you're lucky you didn't include a fork bomb in your own code to eval.")
You may of course alternatively pass, print the offending item, open a browser to some page or do whatever error handling is appropriate given your input data.
[1] While not bickering around, I've made a new answer that works without eval. You can use precisely the same error handling:
for server in server_list:
for item in required_fields:
value = server
for key in parse_fields(field):
try:
value = value[key]
except TypeError:
print("Remember Kiddo: Eval is Evil!")
break
else: # for: else: triggers only if no break was issued
print item, value
I am using the python package pymongo to retrieve data from a mongodb database.
>>> r = collection.find() # returns an object of class 'Cursor'
Then I convert to a list
>>> l = list(r) # returns a 'list' of 'dict'
here is what print(l) returns:
>>> [{u'date': datetime.datetime(2009, 11, 10, 10, 45), u'_id': 1, u'name': u'name1', u'value': 11},{u'date': datetime.datetime(2013, 11, 10, 10, 45), u'_id': 2, u'name': u'name2', u'value': 22}]
Now I need to convert to JSON so that I can manipulate it.
>>> json.dumps(l)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python2.7/json/encoder.py", line 178, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: datetime.datetime(2009, 11, 12, 11, 14) is not JSON serializable
I have also tried to follow http://api.mongodb.org/python/1.7/api/pymongo/json_util.html without success:
Edit: the recent version of the link is http://api.mongodb.org/python/current/api/bson/json_util.html
>>> json.dumps(l, default=json_util.default)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'json_util' is not defined
Note: precisely I need to push this result to R using the R package rPython and its function rPython::python.get("l")
Side Question: What is the u (u'Date', u'name', etc..) before each field in the list of dict?
The pymongo documentation you pointed is obsolete. If you're using version 1.7 I recommend updating. With a more recent version you can do this:
from bson.json_util import dumps
dumps(l)
https://pymongo.readthedocs.io/en/stable/api/bson/json_util.html
Side answer: u'name', u'date', u'_id' etc are the names of the fields of the document on the database.
from bson import json_util
json.dumps(result,default=json_util.default)
In my situation, this error is due to mongo DB id object in flask
all you have to do is convert id (NOTE: if you need id convert it else you can pop it too)
I'm sharing my solution which I figured out hope this helps someone
from flask import jsonify
def get_data(self,data):
data['_id'] = str(data['_id'])
return data
app = Flask(__name__)
#app.route('/')
def apimethod():
temp = [self.get_data(i) for i in self.collection.find()]
return jsonify(temp)
also dumps from pymongo don't help alot
from bson.json_util import dumps,loads
because it is returning a string instead of dict which was expected in my situation to create API and I have to load again if I did dumps.
This thread helped me - thank you.
Wanted to share my final solution to get the JSON back into a JSON/Dictionary Object: (Based on your example)
from bson.json_util import dumps, loads
r = collection.find()
l = list(r) # Converts object to list
d = dumps(l) # Converts to String
dict_needed = loads(d[0]) # Serializes String and creates dictionary
Now you have the JSON in a dictionary object and can edit as needed.
I was facing the same issue, I wrote a code that converts document to dictionary. You can use that for reference. Pass the object obtained by find_one() into documentToJson() method and the results of find() into convertDocumentsToJson. There is type in the name Json, instead the code converts to Dict rather than json.
from bson.json_util import dumps
class UtilService:
def __init__(self):
pass
#staticmethod
def pinCodeParser(path):
location = {}
f = open(path)
for line in f:
words = line.split()
location[words[1]] = (words[-3],words[-2])
return location
#staticmethod
def listHelper(str):
s = []
str = str.split(',')
for e in str:
s.append(e.replace("[","").replace("]",""))
return s
#staticmethod
def parseList(str):
if ',' in str:
return UtilService.listHelper(str)
return str
#staticmethod
def trimStr(str):
return str.replace('"','')
#staticmethod
def documentToJson(document):
document = eval(dumps(document))
mp = {}
for key, value in document.iteritems():
if "_id" in key:
mp["id"] = str(value["$oid"])
else:
mp[ UtilService.trimStr(key) ] = UtilService.parseList( value )
return mp
#staticmethod
def convertDocumentsToJson(documents):
result = []
for document in documents:
result.append(UtilService.documentToJson(document))
return result
object = {'score-set': [('SomeString', 1.0)], 'n': 10, 'num-found': 1, 'start': 0}
type(object) is dict.
When I do this in the command line
json.dump(object,f)
where f is an writable open file. I get the dump in the file perfectly.
But inside a program in a context like this:
def JSONresponse(object,request,jsonIndent=None):
r=HttpResponse(mimetype="application/json")
callback1=request.GET["jsoncallback"] if "jsoncallback" in request.GET else None
callback2=request.GET["callback"] if "callback" in request.GET else None
callback = callback1 or callback2
if callback and len(callback) > 1:
r.write(callback + "(")
json.dump(object,r,indent=jsonIndent)
r.write(");")
else:
json.dump(object,r,indent=jsonIndent)
r.write("\n")
return r
I get the following exception on the json.dump line.
Exception Value: 1.0 is not JSON serializable
Any hints would be much appreciated,
Thanks a lot,
I'm guessing your float isn't really a float, but acts like it in certain ways. Try converting it to a float before serializing it:
object['score-set'] = [(a, float(b)) for (a, b) in object['score-set']]