Develop Reference

how to deserialize JSON with keys that are not valid Python names in marshmallow

I'm getting a JSON object with a key "sera:blah"
How would I deserialize that object into a python data type using the marshmallow library as that colon is an invalid property name?
Edit:
So classes in python cannot accept a colon in the porperty name. It's invalid syntax.
Edit2:
Ideally I would like to have a workaround within marshmallow.

I see 2 routes you can take with this
Try to deserialize it with JSON.loads first, and iterate through each property and replace all the malformed keys before feeding it to marshmallow, or
Use the JSON.JSONDecoder class and roll your own object_hook function. Then call the .decode() function before feeding it to marshmallow.
I've expanded on the latter (which I think is more appropriate)
from json import loads, JSONDecoder
s = """{
"obj1": 123,
"list": [
{"example2": 42},
{"sera:blah": false},
{"object:3": {"nest:ed": "obj"}}
]
}"""
data = loads(s)
print(data)
def obj_transform(obj):
for key in obj.keys(): # Iterate through obj
if ':' in key:
obj[key.replace(':', '_')] = obj.pop(key)
return obj
decoder = JSONDecoder(object_hook=obj_transform)
print(decoder.decode(s))
The result of this prints:
{'obj1': 123, 'list': [{'example2': 42}, {'sera:blah': False}, {'object:3': {'nest:ed': 'obj'}}]}
{'obj1': 123, 'list': [{'example2': 42}, {'sera_blah': False}, {'object_3': {'nest_ed': 'obj'}}]}
Which seems like what you are looking for, to sanitize your input to marshmallow.

Marshmallow handles this with the data_key attribute.
class MySchema(ma.Schema):
sara_blah = ma.fields.String(data_key="sara_blah")
(This is marshmallow 3 syntax. Marshmallow 2 used load_from and dump_to.)

How to serialize objects which members are not serializable directly, but their str() representation is?

I receive dicts such as (without knowing the exact structure in advance)
{
'a': 1,
'id': UUID('6b3acb30-08bf-400c-bc64-bf70489e388c'),
}
This dict is not directly serializable, but when casting the value of id to an str - it is:
import json
import uuid
print(json.dumps({
'a': 1,
'id': str(uuid.UUID('6b3acb30-08bf-400c-bc64-bf70489e388c')),
}))
# outputs {"a": 1, "id": "6b3acb30-08bf-400c-bc64-bf70489e388c"}
In the general case where I have elements which need to be casted to an str before being serializable, is there a generic (pythonic) way to make the transformation automatically?

The best option is to override JSONEncoder.default method:
class MyJSONEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, uuid.UUID):
return str(o)
return super().default(o)
print( MyJSONEncoder().encode(data) )
If you want to stringify everything that the default encoder cannot handle, you may use the following trick, although I would recommend to take control over the types you want to support.
class MyJSONEncoder(json.JSONEncoder):
def default(self, o):
try:
return super().default(o)
except TypeError:
return str(o)
print( json.dumps(data, cls=MyJSONEncoder) )
DOCS: https://docs.python.org/3/library/json.html#json.JSONEncoder.default

No there is not, but you can check for the data items individually and convert when needed. This way you do not need to know the data structure in advance. Consider the following:
import json
import uuid
data = {
'a': 1,
'id': UUID('6b3acb30-08bf-400c-bc64-bf70489e388c')
}
for k, v in data.items():
try:
json.dumps(v)
except TypeError:
data[k] = str(uuid.v)
print(json.dumps(data))
# outputs {"a": 1, "id": "6b3acb30-08bf-400c-bc64-bf70489e388c"}

Django "is not JSON serializable" - ajax, views.py - How to implement?

I have read the documentation, but I am not exactly sure how to implement serializer.serialize for JSON objects in my view.py. If anyone can help me understand this a little better. I have the following code in my view.py:
#user_passes_test(lambda u: u.is_superuser)
def ProjDetails(request):
proj_id = request.GET['proj_id']
proj = Proj.objects.filter(id=proj_id)
role_list = ProjRole.objects.filter(proj=proj)
proj = {
"proj": proj,
"roles": []
}
for r in role_list:
proj['roles'].append(r.id)
return HttpResponse(json.dumps(proj), content_type='application/json; charset=UTF-8')
I am trying to call this with .ajax (I am still working on the ajax, so it probably is not right):
$('#proj_list #sel_proj').click(function(){
$('div.sel').removeClass("sel");
$(this).addClass("sel");
var project_id = $(this).data('id');
$.ajax({
url:'../../proj_details',
data: {proj_id: proj_id},
// dataType: 'html',
success: function(data){
$('#proj_display').html(data)
},
error: function () {
alert("Failed to find the project!")
}
});
Once I get the JSON call to work, then I will focus more on the ajax.
Biggest problem, I am getting a 500 http error with:
TypeError at ../proj_details
[<Project: Example>] is not JSON serializable
I am using Django 1.7, but I even added SESSION_SERIALIZER = 'django.contrib.sessions.serializers.PickleSerializer' to my settings.py without any luck. So I imported serializers from the django.core and tried to use serializer.serialize, but I am not understanding how to implement it I guess because my errors just keep getting worse. I have seen other posts with the same error, but still not understanding for my particular requirements.
+++++++++++++++ EDIT +++++++++++++++++++
So the only way I have been able to get this to work without multiple errors, circular errors, multiple argument errors, etc, is the following:
def ProjDetails(request):
def date_handler(obj):
return obj.strftime("%B %d, %Y") if hasattr(obj, 'strftime') else obj
proj_id = request.GET['proj_id']
proj = Proj.objects.get(id=proj_id)
corp = Corp.objects.get(id=proj.corp.id)
role_list = ProjRole.objects.filter(proj=proj).all()
proj = {
"proj": {
'title': proj.title,
'id': proj.id,
'date': proj.date,
'description': proj.description
}
"roles": [],
"company": {
'name': corp.name,
'pic': unicode(corp.pic),
}
}
for r in role_list:
proj['roles'].append(r.name)
return HttpResponse(json.dumps(proj, default=date_handler), content_type='application/json; charset=UTF-8')
The only thing I don't like about this is I actually have to manually pull what attributes I want from the model into the dictionary, instead of all the attributes being pulled from the model and then I can choose which ones I want to use in my templates. I would rather not have to pull everything like my example above. The 'roles' = [] is giving me some hiccups too because I can't seem to get it to work when there are multiple roles for a proj object.
I like Eugene's method because it would be cleaner, but I can't seem to get it to work with the corp model. The proj tables have a corp_id, yet I keep getting corp_id is not an attribute when I attempt it with using .value().get() for the proj object. I don't understand how to implement grzgrzgrz3's answer either. I usually work more with JS, HTML, and CSS, and I am new to Django/python for web development.
So any suggestions to make this more efficient would be great. Thank!!

Django model's instance can't be serialized, you should use values() method to retrieve dict instead of class instance. Also, you can use only() method to retrieve only id field for roles:
proj = Proj.objects.filter(id=proj_id).values().get()
role_list = ProjRole.objects.only("id").filter(proj__id=proj_id)
proj = {
"proj": proj,
"roles": role_list
}

Write custom HttpResponse and handle there all not serializable python/django objects.
class HttpJsonResponse(HttpResponse):
content_type="application/json"
def __init__(self,data):
def json_serial(obj):
"""JSON serializer for objects not serializable by default json code"""
if isinstance(obj, datetime.date):
serial = obj.isoformat()
return serial
json_data = json.dumps(data, indent=4, default=json_serial)
super(HttpJsonResponse, self).__init__(json_data, self.content_type)
In the example function json_serial converting datetime.date object into string object which is serializable.
*UPDATE
You can mix both answers.
def ProjDetails(request):
proj_id = request.GET['proj_id']
proj = Proj.objects.filter(id=proj_id).values().get()
corp = Corp.objects.filter(id=proj.corp.id).values().get()
role_list = ProjRole.objects.filter(proj=proj).values().all()
proj = {
"proj": proj,
"roles": role_list,
"company": corp
}
return HttpJsonResponse(proj)
Make sure you are importing datetime module.
import datetime
instead datetime class
import datetime.datetime

My answer, as described up above. This is what worked for me.
def ProjDetails(request):
def date_handler(obj):
return obj.strftime("%B %d, %Y") if hasattr(obj, 'strftime') else obj
proj_id = request.GET['proj_id']
proj = Proj.objects.get(id=proj_id)
corp = Corp.objects.get(id=proj.corp.id)
role_list = ProjRole.objects.filter(proj=proj).all()
proj = {
"proj": {
'title': proj.title,
'id': proj.id,
'date': proj.date,
'description': proj.description
}
"roles": [],
"company": {
'name': corp.name,
'pic': unicode(corp.pic),
}
}
for r in role_list:
proj['roles'].append(r.name)
return HttpResponse(json.dumps(proj, default=date_handler), content_type='application/json; charset=UTF-8')

Python - Parsing JSON Data Set

I am trying to parse a JSON data set that looks something like this:
{"data":[
{
"Rest":0,
"Status":"The campaign is moved to the archive",
"IsActive":"No",
"StatusArchive":"Yes",
"Login":"some_login",
"ContextStrategyName":"Default",
"CampaignID":1111111,
"StatusShow":"No",
"StartDate":"2013-01-20",
"Sum":0,
"StatusModerate":"Yes",
"Clicks":0,
"Shows":0,
"ManagerName":"XYZ",
"StatusActivating":"Yes",
"StrategyName":"HighestPosition",
"SumAvailableForTransfer":0,
"AgencyName":null,
"Name":"Campaign_01"
},
{
"Rest":82.6200000000008,
"Status":"Impressions will begin tomorrow at 10:00",
"IsActive":"Yes",
"StatusArchive":"No",
"Login":"some_login",
"ContextStrategyName":"Default",
"CampaignID":2222222,
"StatusShow":"Yes",
"StartDate":"2013-01-28",
"Sum":15998,"StatusModerate":"Yes",
"Clicks":7571,
"Shows":5535646,
"ManagerName":"XYZ",
"StatusActivating":"Yes",
"StrategyName":"HighestPosition",
"SumAvailableForTransfer":0,
"AgencyName":null,
"Name":"Campaign_02"
}
]
}
Lets assume that there can be many of these data sets.
I would like to iterate through each one of them and grab the "Name" and the "Campaign ID" parameter.
So far my code looks something like this:
decoded_response = response.read().decode("UTF-8")
data = json.loads(decoded.response)
for item in data[0]:
for x in data[0][item] ...
-> need a get name procedure
-> need a get campaign_id procedure
Probably quite straight forward! I am not good with lists/dictionaries :(

Access dictionaries with d[dict_key] or d.get(dict_key, default) (to provide default value):
jsonResponse=json.loads(decoded_response)
jsonData = jsonResponse["data"]
for item in jsonData:
name = item.get("Name")
campaignID = item.get("CampaignID")
I suggest you read something about dictionaries.

Accessing json type data without knowing layout of data?

I have a file with JSON data I am loading using json.load.
Suppose I want to put a variable in the json data, which references another data field. How can I process this reference in python?
eg:
{
"dictionary" : {
"list_1" : [
"item_1"
],
"list_2" : [
"$dictionary.list_1"
]
}
}
when I come across $, I then want list_2 to grab the data from: dictionary.list_1
and extend list_2, as if I had written in my python code:
jsonData["dictionary"]["list_2"].extend(jsonData["dictionary"]["list_1"])

As far as I know, there is nothing in the JSON standard for doing references. My first suggestion would be to use YAML which does have references in the form of Node Anchors. Python has a good implementation of YAML which supports those.
That being said, if you're set on using JSON, you'll have to roll your own implementation.
One possible example(though this doesn't extend the current array by the referenced array because that's ambiguous in the case of dicts, it replaces the reference by the value it refers to) is below. Note that it doesn't handle malformed references you'll have to add the error-checking yourself or guarantee that there aren't malformed references. If you want to change it to extend instead of replacing, you can, but you know your use-case better than I so you'll be able to specify it that way. This is meant to give you a starting point.
def resolve_references(structure, sub_structure=None):
if sub_structure is None:
return resolve_references(structure, structure)
if isinstance(sub_structure, list):
tmp = []
for item in sub_structure:
tmp.append(resolve_references(structure, item))
return tmp
if isinstance(sub_structure, dict):
tmp = {}
for key,value in sub_structure.items():
tmp[key] = resolve_references(structure, value)
return tmp
if isinstance(sub_structure, str) or isinstance(sub_structure, unicode):
if sub_structure[0] != "$":
return sub_structure
keys = sub_structure[1:].split(".")
def get_value(obj, key):
if isinstance(obj, dict):
return obj[key]
if isinstance(obj, list):
return obj[int(key)]
return value
value = get_value(structure, keys[0])
for key in keys[1:]:
value = get_value(value, key)
return value
return sub_structure
Example usage:
>>> import json
>>> json_str = """
... {
... "dictionary" : {
... "list_1" : [
... "item_1"
... ],
...
... "list_2" : "$dictionary.list_1"
... }
... }
... """
>>> obj = json.loads(json_str)
>>> resolve_references(obj)
{u'dictionary': {u'list_2': [u'item_1'], u'list_1': [u'item_1']}}