Develop Reference

List of dictionaries using keys as headers & rows are the values of any data type

** I'm currently tasked with creating a CSV file from a list of dictionaries where the headers are the dict keys and the rows the dict values but values should be any data type.**
store_value = [{'Key': 649.0, 'Folder': '/ANZ/', 'End Date': 44907.0, 'Depends On': '', 'Start Date Constraint': '', 'Description': '', 'Resolution / Update': '', 'Comments': 'Comment', 'Staging Link': 'https://', 'Retailer URL': '', 'Item Number': ''}, {'Key': 651.0, 'Folder': '/ANZ/', 'End Date': 44917.0, 'Depends On': '', 'Start Date Constraint': '', 'Description': ' https:', 'Resolution / Update': '', 'Comments': 'Approved', 'Staging Link': '', 'Retailer URL': '', 'Item Number': ''}]
f = open("test.csv", "a")
f.write(','.join(list(store_value[0].keys()))) # write headers
f.write('\n')
for i in store_value:
print(type(i))
f.write(','.join(list(i.values()))) # write values
f.write('\n')
f.close()
TypeError: sequence item 0: expected str instance, float found

You cannot join() float or int.
Change this line:
f.write(','.join(list(i.values())))
...to...
f.write(','.join(map(str, i.values())))
Having said that, your entire program can be simplified thus:
import csv
store_value = [
{'Key': 649.0, 'Folder': '/ANZ/', 'End Date': 44907.0, 'Depends On': '', 'Start Date Constraint': '', 'Description': '', 'Resolution / Update': '', 'Comments': 'Comment', 'Staging Link': 'https://', 'Retailer URL': '', 'Item Number': ''},
{'Key': 651.0, 'Folder': '/ANZ/', 'End Date': 44917.0, 'Depends On': '', 'Start Date Constraint': '', 'Description': ' https:', 'Resolution / Update': '', 'Comments': 'Approved', 'Staging Link': '', 'Retailer URL': '', 'Item Number': ''}
]
with open('test.csv', 'w', newline='') as f:
writer = csv.DictWriter(f, fieldnames=store_value[0])
writer.writeheader()
writer.writerows(store_value)

You are getting this error
TypeError: sequence item 0: expected str instance, float found
because .join() method only works with strings. To solve this you can turn int into str. using map(str, list(i.values()))
Your final code should be:
store_value = [{'Key': 649.0, 'Folder': '/ANZ/', 'End Date': 44907.0, 'Depends On': '', 'Start Date Constraint': '', 'Description': '', 'Resolution / Update': '', 'Comments': 'Comment', 'Staging Link': 'https://', 'Retailer URL': '', 'Item Number': ''}, {'Key': 651.0, 'Folder': '/ANZ/', 'End Date': 44917.0, 'Depends On': '', 'Start Date Constraint': '', 'Description': ' https:', 'Resolution / Update': '', 'Comments': 'Approved', 'Staging Link': '', 'Retailer URL': '', 'Item Number': ''}]
f = open("test.csv", "a")
f.write(','.join(list(store_value[0].keys()))) # write headers
f.write('\n')
for i in store_value:
print(type(i))
f.write(','.join(map(str, list(i.values())))) # write values
f.write('\n')
f.close()

How to convert a record or list into JSON format using python?

I am trying to convert a list into JSON using Python. The result from a query looks something like this :
Output:
[<Record r=<Relationship id=106 nodes=(<Node id=71 labels=frozenset({'TBox'}) properties={'identifier': '', 'ontology_level': 'lower', 'neo4jImportId': '105', 'html_info': '', 'namespace': 'car', 'admin': '', 'description': 'remove', 'sing': '', 'pl': '', 'title': 'BMProcessor', 'version': 'v6.0'}>, <Node id=59 labels=frozenset({'TBox'}) properties={'identifier': '', 'ontology_level': 'lower', 'neo4jImportId': '93', 'html_info': '', 'namespace': 'car', 'admin': '', 'description': 'A DataProcessor which represents a ML algorithm', 'sing': '', 'pl': '', 'title': 'LearningProcessor', 'version': 'v6.0'}>) type='subclass_of' properties={}> b=<Node id=59 labels=frozenset({'TBox'}) properties={'identifier': '', 'ontology_level': 'lower', 'neo4jImportId': '93', 'html_info': '', 'namespace': 'car', 'admin': '', 'description': 'A DataProcessor which represents a ML algorithm', 'sing': '', 'pl': '', 'title': 'LearningProcessor', 'version': 'v6.0'}> n=<Node id=71 labels=frozenset({'TBox'}) properties={'identifier': '', 'ontology_level': 'lower', 'neo4jImportId': '105', 'html_info': '', 'namespace': 'car', 'admin': '', 'description': 'remove', 'sing': '', 'pl': '', 'title': 'BMProcessor', 'version': 'v6.0'}>>]
Function :
def runQuery(query):
pprint.pprint(connection.execute(query))
When I perform a simple json.dumps() it return with TypeError: Object of type is not JSON serializable
I want to print a JSON format out of this. How can I do so?

You can get the result.data() which is a dictionary.
You can also iterate over the records in the cursor and convert one by one, and extract the fields you want, which is what the linked example above does - pass the cursor and use the blob['field'] syntax
def serialize_genre(genre):
return {
'id': genre['id'],
'name': genre['name'],
}

Adding item to every dictionary in a list of nested dictionaries

I have a list where there are nested dictionaries, such as the following structure:
[
{'family': '',
'given': '',
'affiliation': [{'': ''}]},
{'family': '',
'given': '',
'affiliation': [{'': ''}]},
{'family': '',
'given name': '',
'localId': '',
'affiliation': [{'': ''}]},
{'family': '',
'given name': '',
'affiliation': [{'': ''}]}
]
I want to add an id dictionary to each of the nested dictionary in this such that the structure becomes:
[
{'id':'',
'family': '',
'given': '',
'affiliation': [{'': ''}]},
{'id':'',
'family': '',
'given': '',
'affiliation': [{'': ''}]},
{'id':'',
'family': '',
'given name': '',
'localId': '',
'affiliation': [{'': ''}]},
{'id':'',
'family': '',
'given name': '',
'affiliation': [{'': ''}]}
]
I tried the following code, where tmp is my list name:
tmp.append({'id':df.id[0]})
which results in:
[
{'family': '',
'given': '',
'affiliation': [{'': ''}]},
{'family': '',
'given': '',
'affiliation': [{'': ''}]},
{'family': '',
'given name': '',
'localId': '',
'affiliation': [{'': ''}]},
{'family': '',
'given name': '',
'affiliation': [{'': ''}]},
{'id': '776'}
]
Can anyone suggest how to solve this problem so that i am able to add id in each dictionary?
I would prefer a one line code instead of loop so that i can apply to each cell of a dataframe where I have these nested dictionaries.

Just loop the list items and add the new field
l = [
{'family': '',
'given': '',
'affiliation': [{'': ''}]},
{'family': '',
'given': '',
'affiliation': [{'': ''}]},
{'family': '',
'given name': '',
'localId': '',
'affiliation': [{'': ''}]},
{'family': '',
'given name': '',
'affiliation': [{'': ''}]}
]
for e in l:
e['id'] = ''

I think you have to loop through your list of dictionaries and add 'id' attribute each one of them
for i in range(len(tmp)):
tmp[i]['id'] = ''

Try this:
[i.update({'id':''}) for i in tmp]

Removing duplicate entries?

I need to compare values from different rows. Each row is a dictionary, and I need to compare the values in adjacent rows for the key 'flag'. How would I do this? Simply saying:
for row in range(1,len(myjson))::
if row['flag'] == (row-1)['flag']:
print yes
returns a TypeError: 'int' object is not subscriptable
Even though range returns a list of ints...
RESPONSE TO COMMENTS:
List of rows is a list of dictionaries. Originally, I import a tab-delimited file and read it in using the csv.dict module such that it is a list of dictionaries with the keys corresponding to the variable names.
Code: (where myjson is a list of dictionaries)
for row in myjson:
print row
Output:
{'website': '', 'phone': '', 'flag': 0, 'name': 'Diane Grant Albrecht M.S.', 'email': ''}
{'website': 'www.got.com', 'phone': '111-222-3333', 'flag': 1, 'name': 'Lannister G. Cersei M.A.T., CEP', 'email': 'cersei#got.com'}
{'website': '', 'phone': '', 'flag': 2, 'name': 'Argle D. Bargle Ed.M.', 'email': ''}
{'website': 'www.daManWithThePlan.com', 'phone': '000-000-1111', 'flag': 3, 'name': 'Sam D. Man Ed.M.', 'email': 'dman123#gmail.com'}
{'website': '', 'phone': '', 'flag': 3, 'name': 'Sam D. Man Ed.M.', 'email': ''}
{'website': 'www.daManWithThePlan.com', 'phone': '111-222-333', 'flag': 3, 'name': 'Sam D. Man Ed.M.', 'email': 'dman123#gmail.com'}
{'website': '', 'phone': '', 'flag': 4, 'name': 'D G Bamf M.S.', 'email': ''}
{'website': '', 'phone': '', 'flag': 5, 'name': 'Amy Tramy Lamy Ph.D.', 'email': ''}
Also:
type(myjson)
<type 'list'>

For comparing adjacent items you can use zip:
Example:
>>> lis = [1,1,2,3,4,4,5,6,7,7]
for x,y in zip(lis, lis[1:]):
if x == y :
print x,y,'are equal'
...
1 1 are equal
4 4 are equal
7 7 are equal
For your list of dictionaries, you can do something like :
from itertools import izip
it1 = iter(list_of_dicts)
it2 = iter(list_of_dicts)
next(it2)
for x,y in izip(it1, it2):
if x['flag'] == y['flag']
print yes
Update:
For more than 2 adjacent items you can use itertools.groupby:
>>> lis = [1,1,1,1,1,2,2,3,4]
for k,group in groupby(lis):
print list(group)
[1, 1, 1, 1, 1]
[2, 2]
[3]
[4]
For your code it would be :
>>> for k, group in groupby(dic, key = lambda x : x['flag']):
... print list(group)
...
[{'website': '', 'phone': '', 'flag': 0, 'name': 'Diane Grant Albrecht M.S.', 'email': ''}]
[{'website': 'www.got.com', 'phone': '111-222-3333', 'flag': 1, 'name': 'Lannister G. Cersei M.A.T., CEP', 'email': 'cersei#got.com'}]
[{'website': '', 'phone': '', 'flag': 2, 'name': 'Argle D. Bargle Ed.M.', 'email': ''}]
[{'website': 'www.daManWithThePlan.com', 'phone': '000-000-1111', 'flag': 3, 'name': 'Sam D. Man Ed.M.', 'email': 'dman123#gmail.com'}, {'website': '', 'phone': '', 'flag': 3, 'name': 'Sam D. Man Ed.M.', 'email': ''}, {'website': 'www.daManWithThePlan.com', 'phone': '111-222-333', 'flag': 3, 'name': 'Sam D. Man Ed.M.', 'email': 'dman123#gmail.com'}]
[{'website': '', 'phone': '', 'flag': 4, 'name': 'D G Bamf M.S.', 'email': ''}]
[{'website': '', 'phone': '', 'flag': 5, 'name': 'Amy Tramy Lamy Ph.D.', 'email': ''}]

Your exception indicates that list_of_rows is not what you think it is.
To look at other, adjacent rows, provided list_of_rows is indeed a list, I'd use enumerate() to include the current index and then use that index to load next and previous rows:
for i, row in enumerate(list_of_rows):
previous = list_of_rows[i - 1] if i else None
next = list_of_rows[i + 1] if i + 1 < len(list_of_rows) else None

Looks like you want to access list elements in batches:
http://code.activestate.com/recipes/303279/

You could try this
pre_item = list_of_rows[0]['flag']
for row in list_of_rows[1:]:
if row['flag'] == pre_item :
print yes
pre_item = row['flag']

list_of_rows = [ { 'a': 'foo',
'flag': 'bar' },
{ 'a': 'blo',
'flag': 'bar' } ]
for row, successor_row in zip(list_of_rows, list_of_rows[1:]):
if row['flag'] == successor_row['flag']:
print "yes"

It's simple. If you need to remove those dicts that have the same value for key "flag", as the title of your post suggests (it is somewhat misleading because your dictionaries are not strictly speaking duplicates), you can simply loop over the whole list of dictionaries, keeping track of flags in a separate list, if an item has a flag which is already in the list of flags simply don't add it, it would look something like:
def filterDicts(listOfDicts):
result = []
flags = []
for di in listOfDicts:
if di["flag"] not in flags:
result.append(di)
flags.append(di["flag"])
return result
When called with value of list of dictionaries that you have provided, it returns list with 5 items, each has an unique value of flag.

Flagging Entries with the Same Names?

I'm working with data where people have entered their names and some contact information. However, since they were unable to enter multiple entries for some of the fields, some people entered their names multiple times, resulting in 'duplicate' entries...
I'm trying to mark duplicate entries by the same user using a variable 'flag'.
For each row, what I want to happen is that if the name entry in the row is NOT the same as the name entry in the next row, the flag entry should increase by one.
How do I do this?
This is the code I currently have:
# FLAG 2
import csv
myjson = []
with(open("ieca_first_col_fake_text.txt", "rU")) as f:
sheet = csv.DictReader(f,delimiter="\t")
sheet.fieldnames.append('flag')
print sheet.fieldnames
for row in sheet:
myjson.append(row)
flag_counter = 0
myjson[0]['flag'] = flag_counter
for i in range(len(myjson)-1):
if myjson[i]['name'] != myjson[i+1]['name']:
myjson[i+1]['flag'] = flag_counter + 1
else:
myjson[i]['flag'] = flag_counter
for i in range(len(myjson)):
print myjson[i]
This is example data:
name phone email website area degree
Diane Grant Albrecht M.S.
Lannister G. Cersei M.A.T., CEP 111-222-3333 cersei#got.com www.got.com
Argle D. Bargle Ed.M.
Sam D. Man Ed.M. 000-000-1111 dman123#gmail.com www.daManWithThePlan.com
Sam D. Man Ed.M.
Sam D. Man Ed.M. 111-222-333 dman123#gmail.com www.daManWithThePlan.com
D G Bamf M.S.
Amy Tramy Lamy Ph.D.
And this is the output that results from operating on the example data:
['name', 'phone', 'email', 'website', 'flag']
{'website': '', 'phone': '', 'flag': 0, 'name': 'Diane Grant Albrecht M.S.', 'email': ''}
{'website': 'www.got.com', 'phone': '111-222-3333', 'flag': 1, 'name': 'Lannister G. Cersei M.A.T., CEP', 'email': 'cersei#got.com'}
{'website': '', 'phone': '', 'flag': 1, 'name': 'Argle D. Bargle Ed.M.', 'email': ''}
{'website': 'www.daManWithThePlan.com', 'phone': '000-000-1111', 'flag': 0, 'name': 'Sam D. Man Ed.M.', 'email': 'dman123#gmail.com'}
{'website': None, 'phone': '', 'flag': 0, 'name': 'Sam D. Man Ed.M.', 'email': None}
{'website': 'www.daManWithThePlan.com', 'phone': '111-222-333', 'flag': None, 'name': 'Sam D. Man Ed.M.', 'email': ' dman123#gmail.com'}
{'website': '', 'phone': '', 'flag': 1, 'name': 'D G Bamf M.S.', 'email': ''}
{'website': '', 'phone': '', 'flag': 1, 'name': 'Amy Tramy Lamy Ph.D.', 'email': ''}
Note that the flags do not correspond to the desired pattern.
And here is an ideal output (notice the difference in flag entries):
['name', 'phone', 'email', 'website', 'flag']
{'website': '', 'phone': '', 'flag': 0, 'name': 'Diane Grant Albrecht M.S.', 'email': ''}
{'website': 'www.got.com', 'phone': '111-222-3333', 'flag': 1, 'name': 'Lannister G. Cersei M.A.T., CEP', 'email': 'cersei#got.com'}
{'website': '', 'phone': '', 'flag': 2, 'name': 'Argle D. Bargle Ed.M.', 'email': ''}
{'website': 'www.daManWithThePlan.com', 'phone': '000-000-1111', 'flag': 3, 'name': 'Sam D. Man Ed.M.', 'email': 'dman123#gmail.com'}
{'website': None, 'phone': '', 'flag': 3, 'name': 'Sam D. Man Ed.M.', 'email': None}
{'website': 'www.daManWithThePlan.com', 'phone': '111-222-333', 'flag': 3, 'name': 'Sam D. Man Ed.M.', 'email': ' dman123#gmail.com'}
{'website': '', 'phone': '', 'flag': 4, 'name': 'D G Bamf M.S.', 'email': ''}
{'website': '', 'phone': '', 'flag': 5, 'name': 'Amy Tramy Lamy Ph.D.', 'email': ''}

EDIT:
Ths loop workes for me (output as expected):
for i in range(len(myjson)-1):
if myjson[i]['name'] != myjson[i+1]['name']:
print "not same" ,myjson[i]['name'] ,' ', myjson[i+1]['name']
flag_counter = flag_counter + 1
myjson[i+1]['flag'] = flag_counter
else:
print 'equal', myjson[i]['name'] ,' ', myjson[i+1]['name']
myjson[i]['flag'] = flag_counter
Note that I had to format the csv file by hand (tabs weren't tabs, but spaces). Make sure it is correct in your file. The names have to be exactly correct, no additional spaces allows
But I am not sure if this is the only bug, as there are many dangerous 'off-by-one' traps. If it still doesn't work, just update your output and code and we will see!