Develop Reference

How can I best convert an API JSON object to a single row for SQL server?

I have a script setup to pull a JSON from an API and I need to convert objects into different columns for a single row layout for a SQL server. See the example below for the body raw layout of an example object:
"answers": {
"agent_star_rating": {
"question_id": 145,
"question_text": "How satisfied are you with the service you received from {{ employee.first_name }} today?",
"comment": "John was exceptionally friendly and knowledgeable.",
"selected_options": {
"1072": {
"option_id": 1072,
"option_text": "5",
"integer_value": 5
}
}
},
In said example I need the output for all parts of agent_star_rating to be individual columns so all data spits out 1 row for the entire survey on our SQL server. I have tried mapping several keys like so:
agent_star_rating = [list(response['answers']['agent_star_rating']['selected_options'].values())[0]['integer_value']]
agent_question = (response['answers']['agent_star_rating']['question_text'])
agent_comment = (response['answers']['agent_star_rating']['comment'])
response['agent_question'] = agent_question
response['agent_comment'] = agent_comment
response['agent_star_rating'] = agent_star_rating
I get the expected result until we reach a point where some surveys have skipped a field like ['question text'] and we'll get a missing key error. This happens over the course of other objects and I am failing to come up with a solution for these missing keys. If there is a better way to format the output as I've described beyond the keys method I've used I'd also love to hear ideas! I'm fresh to learning python/pandas so pardon any improper terminology!

I would do something like this:
# values that you always capture
row = ['value1', 'value2', ...]
gottem_attrs = {'question_id': '' ,
'question_text': '',
'comment': '',
'selected_options': ''}
# find and save the values that response have
for attr in list(response['agent_star_rating']):
gottem_attrs[attr] = response['agent_star_rating'][attr]
# then you have your final row
final_row = row + gottem_attrs.values()
If the response have a value in his attribute, this code will save it. Else, it will save a empty string for that value.

Accessing Json object values with the Help of a dictionary

I've a huge json file which has a lot of nested key value pairs. So I thought I should save the keys as dictionary values and use that dictionary values as keys to access the values from the json file. Say for example:
json_obj = {
"calendar" :{
"day": {
"activities" : {
"morning:"walk"
}
}
}
}
so I thought to access the key morning value, instead of writing
json_obj['calendar']['day']['activities']['morning']
I should keep a dictionary which will contain the query parameters like
query_parameters = {
0 :[['calendar'],['day'],['activities'],['morning']]
}
And use this dictionary to query from the json object.
But
Here is my question?
Can I use this dictionary to write query or to access values from my json_obj without using any loops ?
say something like json_obj[query_parameters[0]] # this is not right syntax I know
Or do you have any suggestions when accessing these long key value pair from an object?

You can write a function like this
This function will return value if exist otherwise returns None
def fun(query, data):
if not query:
return data
if query[0][0] in data:
return fun(query[1:], data[query[0][0]])
print(fun(query_parameters[0], json_obj)) # walk

my loop is only printing the second part of the dictionary , i'm using json

import json
data ='''
{
"names": {"first_boy" : "khaled"},
"names": {"second_boy" : "waseem"}
}
'''
info = json.loads(data)
for line in info:
print(info["names"])
I expected it to print the first_boy and the second_boy dictionary ,but it's printing
{'second_boy': 'waseem'}

Dicts in python can only support one of the same key. Similarly, most implementations of JSON do not allow duplicate keys. The way python handles this, when using json.loads() (or anything else that constructs a dict) is to simply use the most recent definition of any given key.
In this case, {"second_boy":"waseem"} overwrites {"first_boy":"khaled"}.

The problem here is that the key "names" exists 2 times.
Maybe you can do this:
import json
data ='''
{
"names": {"first_boy" : "khaled",
"second_boy" : "waseem"}
}
'''
info = json.loads(data)
for key, value in info['names'].items():
print(key, value)

Transform string into access function to access a series in a dictionary

I'm trying to pull data from a dictionary ('data') in which several series are provided:
For instance, equity is extracted with:
data['Financials']['Balance_Sheet']['equity']
As I'm having several functions each calling one different series (e.g. equity, debt, goodwill, etc...), I would like to be able to define the "access" for each of those by defining a string such as:
Goodwill -> data['Financials']['Balance_Sheet']['Goodwill']
Equity->data['Financials']['Balance_Sheet']['Equity']
My idea is to do something like that:
Data_pulled= ACCESS('data['Financials']['Balance_Sheet']['Goodwill']')
What is the ACCESS function required to transform the string into a acccess function?
Hope this is clear! Thanks a lot for your help guys - much appreciated! :)
Max

I question what you're trying to accomplish here. A better answer is probably to write a accessor function that can safely get the field you want without having to type the whole thing out every time.
Consider the following code:
def ACCESS(*strings):
def ACCESS_LAMBDA(dic):
result = dic
for key in strings:
result = result[key]
return result
return ACCESS_LAMBDA
dic = { 'aa': { 'bb': { 'cc': 42 } } }
ABC_ACCESS = ACCESS('aa','bb','cc')
print ABC_ACCESS(dic)
This is called a closure, where you can define a function at runtime. Here you'd create pull_goodwill = ACCESS('Financials','Balance_Sheet','Goodwill') then get the value with Data_pulled = pull_goodwill(data)
This doesn't exactly answer your question, and the star-arguments and Lambda-returned-function are pretty advanced things. But, don't just "call eval()" that's a pretty insecure coding habit to get into. eval() has its uses... But, think about what you're trying to do, and see if there is a simple abstraction that you can program to access the data you want, rather than relying on the python parser to fetch a value from a dict.
edit:
Link to information about closures in python
edit2:
In order to not have to pass a dictionary to the returned lambda-function, you can pass it into the function constructor. Here's what that would look like, note the change to ACCESS's definition now includes dic and that the ACCESS_LAMBDA definition now takes no arguments.
def ACCESS(dic, *strings):
def ACCESS_LAMBDA():
result = dic
for key in strings:
result = result[key]
return result
return ACCESS_LAMBDA
dic = { 'a': { 'b': { 'c': 42 } } }
ABC_ACCESS = ACCESS(dic, 'a','b','c')
print ABC_ACCESS()
(Note here, that if dict is modified, then the ABC_ACCESS value will change. this is because python passes by reference, if you want a constant value you'd need to make a copy of dic.)

Python : convert unicode string to raw object/text

I've got a set of key, value pairs dictionary in my Django application. The value in the dictionary is a string type.
{u'question': u'forms.CharField(max_length=512)'}
I need to convert this "value" string to an actual object, and get something like this.
properties = {
'question' : forms.CharField(max_lenth=512)
}
Notice that values in the second dictionary are actual Django form fields and NOT strings. I need to do this manipulation to create dynamic forms. The second dictionary is to be passed to "type" built-in function. Sample code can be found on this page. http://dougalmatthews.com/articles/2009/dec/16/nicer-dynamic-forms-django/ .

If you modify your representation a bit:
fields = {u'question': u'{"field": "django.forms.CharField", "params": {"max_length": 512}}'}
then you can use the following:
from django.utils import importlib, simplejson
def get_field(fname):
module, name = fname.rsplit('.', 1)
return getattr(importlib.import_module(module), name)
print dict((k.encode('ascii', 'ignore'), get_field(v['field'])(**v['params']))
for k, v in ((k, simplejson.loads(v)) for k, v in fields.iteritems()))

Following your code, I suggest to separate field name from field attrs:
my_fields = {u'question': {'name': 'CharField', 'attrs': {'max_length': 512} }}
and then something like:
properties = {}
for field_name, field_def in my_fields.items():
properties[field_name] = getattr(forms, field_def['name'])(**field_def['attrs'])

EDIT: Based on clarifying comments by the OP, this isn't an appropriate solution.
I don't know the constraints you are under, but perhaps the following representation will work better:
{u'question': lambda: forms.CharField(max_length=512)}
You can then "realise" the fields thus:
dict((k, v()) for (k, v) in props.iteritems())