How can I store data in a dictionary without overriding it?

How can I store data in a dictionary without overriding it? - python

How can I store data in a dictionary in Python without overriding the existing one?
For example:
output = {}
name = raw_input("Enter name")
age = input("Enter your age")
course = raw_input("Enter course")
school = raw_input("Enter school")
output['name'] = name
output['age'] = age
output['course'] = course
output['school'] = school
The output is this.
{
"name": "Student 1",
"age": 25,
"course": "BSCS",
"school": "School 1"
}
Then, if I add another field, it overrides the existing data.
How can I store it just like this:
{
"students": [
{
"name": "Student1",
"age": 25,
"course": "BSIT",
"school": "School 1"
},
{
"name": "Student2",
"age": 26,
"course": "BSCS",
"school": "School 2"
},
{
"name": "Student3",
"age": 27,
"course": "BSCE",
"school": "School 3"
}
]
}

A key is unique, so if you want to store multiple values in one key, make the value a list or another dict, tuple, custom Object etc..
E.g.
my_dict = {}
my_dict["students"] = []
my_dict["students"].append( new_dict )
I would consider making a class or using a tuple to store the students data inside the list, however, if you want the JSON like format, you may use other dictionaries like:
new_dict = {"name": "Max", "age":12}
my_dict["students"].append( new_dict )
In case of an object you'd make something like:
class Student(object):
__init__(self, name, age):
self.name = name
self.age = age
So now you could do something like:
my_dict.append( Student("max", 12) )

You can also solve it using the inbuilt collections module and a class called defaultdict in it.
import collections as cl
output = cl.defaultdict(list)
for i in range(n):
name, age, course, school = map(str, raw_input().split())
age, key, value = int(age), "student" + str(i + 1), dict()
value["name"], value["age"], value["course"], value["school"] = name, age, course, school
output[key] = value
As far as the documentation says
This module implements specialized container datatypes providing alternatives to Python’s general purpose built-in containers, dict, list, set, and tuple.
Python Documentation

Related

python getting json values from list

I have some json data similar to this...
{
"people": [
{
"name": "billy",
"age": "12"
...
...
},
{
"name": "karl",
"age": "31"
...
...
},
...
...
]
}
At the moment I can do this to get a entry from the people list...
wantedPerson = "karl"
for person in people:
if person['name'] == wantedPerson:
* I have the persons entry *
break
Is there a better way of doing this? Something similar to how we can .get('key') ?
Thanks,
Chris

Assuming you load that json data using the standard library for it, you're fairly close to optimal, perhaps you were looking for something like this:
from json import loads
text = '{"people": [{"name": "billy", "age": "12"}, {"name": "karl", "age": "31"}]}'
data = loads(text)
people = [p for p in data['people'] if p['name'] == 'karl']
If you frequently need to access this data, you might just do something like this:
all_people = {p['name']: p for p in data['people']}
print(all_people['karl'])
That is, all_people becomes a dictionary that uses the name as a key, so you can access any person in it quickly by accessing them by name. This assumes however that there are no duplicate names in your data.

First, there's no problem with your current 'naive' approach - it's clear and efficient since you can't find the value you're looking for without scanning the list.
It seems that you refer to better as shorter, so if you want a one-liner solution, consider the following:
next((person for person in people if person.name == wantedPerson), None)
It gets the first person in the list that has the required name or None if no such person was found.

similarly
ps = {
"people": [
{
"name": "billy",
"age": "12"
},
{
"name": "karl",
"age": "31"
},
]
}
print([x for x in ps['people'] if 'karl' in x.values()])
For possible alternatives or details see e.g. # Get key by value in dictionary

Save List of object with heterogenous keys in csv file in python

I need to store list of objects whose number of keys are not fixed in CSV file.
Example :
suppose I have a object
obj1 = {
"name": "abc",
"class" : "10th"
}
obje2 = {
"name": "abcd",
"class" : "11th",
"div" : "A"
}
So There will be common elements, but I need to store all the elements
My output would be like this :

Sure thing. You'll just have to gather up all of the keys first – that's done with set unions – and then feed the data to a csv.DictWriter.
import csv
import sys
data = [
{"name": "abc", "class": "10th"},
{"name": "abcd", "class": "11th", "div": "A"},
{"name": "qweabcd", "class": "11th", "diy": "Q"},
]
all_keys = set()
for obj in data:
all_keys |= set(obj)
writer = csv.DictWriter(sys.stdout, sorted(all_keys))
writer.writeheader()
for obj in data:
writer.writerow(obj)
This outputs
class,div,diy,name
10th,,,abc
11th,A,,abcd
11th,,Q,qweabcd
Since sets don't have an intrinsic order, I'm using sorted to sort the keys alphabetically.
If you need a specific order, that can be done with a custom key function to sorted.
key_order = ["name", "class"]
def get_key_sort_value(key):
if key in key_order:
return (0, key_order.index(key))
return (1, key)
writer = csv.DictWriter(sys.stdout, sorted(all_keys, key=get_key_sort_value))
# ...
will sort name and class first, followed by other keys in alphabetical order.

Another solution, with pandas:
import pandas as pd
obj1 = {"name": "abc", "class": "10th"}
obj2 = {"name": "abcd", "class": "11th", "div": "A"}
df = pd.DataFrame([obj1, obj2])
print(df)
df.to_csv("data.csv", index=False)
Prints:
name class div
0 abc 10th NaN
1 abcd 11th A
and saves data.csv (screenshot from LibreOffice):

Python: How to filter dictionary?

I want to achieve the following, but using dictionary.
name_selected = "Jason"
if name_selected == "John":
age = 10
gender = "Male"
elif name_selected == "Jason":
age = 20
gender = "Male"
print(age, gender)
data = {"name": ["John", "Jason"],
"age": [10, 20],
"gender": ["Male", "Male"]}

That's a poor data organization. It would be better if you made the names the dictionary keys:
data = {
"John": { "age": 10, "gender": "Male"},
"Jason": { "age": 20, "gender": "Male"}
}
age = data[name_selected]["age"]
gender = data[name_selected]["gender"]
But if you're stuck with your data structure, you can use index() to get the index in the name element, then use that to get the corresponding values in the other elements.
try:
index = data['name'].index(name_selected)
age = data['age'][index]
gender = data['gender'][index]
except:
print(f'{name_selected} not found')

Since you're testing name_selected == ..., the possible values of name_selected are the dictionaries you want. Dictionaries are the structure that replaces if-elif lookup on a variable.
Given that, you can store the remaining data in any number of ways. Here's a tuple that you can unpack:
data = {'John': (10, 'Male'),
'Jason': (20, 'Male')}
Then you can do
age, gender = data[name_selected]
You can extend the idea with another nested dictionary:
data = {'John': {
'age': 10, 'gender': 'Male'},
'Jason': {
'age': 20, 'gender': 'Male'}
}
Now it might be better not to unpack:
person = data[name_selected]
# use person ['age'] and person['gender']
A more application-specific solution would be to make the values custom objects (like a collections.namedtuple) that would let you access the data for an individual as attributes.

How can I parse nested JSON to CSV

I have a new project where I obtain JSON data back from a REST API - I'm trying to parse this data to csv pipe delimited to import to our legacy software
I can't seem to get all the value pairs parsed properly - this is my first exposure to JSON and I've tried so many things but only getting a little right at a time
I have used Python and can get some items that I need but not the whole JSON tree - it comes across as a list and has some dictionaries and lists in it as well
I know my code is incomplete and just looking for someone to point me in the right direction on what tools in python can get the job done
import json
import csv
with open('tenants.json') as access_json:
read_content = json.load(access_json)
for rm_access in read_content:
rm_data = rm_access
print(rm_data)
contacts_data = rm_data['Contacts']
leases_data = rm_data['Leases']
udfs_data = rm_data['UserDefinedValues']
for contacts_access in contacts_data:
rm_contacts = contacts_access
UPDATED:
import pandas as pd
with open('tenants.json') as access_json:
read_content = json.load(access_json)
for rm_access in read_content:
rm_data = rm_access
pd.set_option('display.max_rows', 10000)
pd.set_option('display.max_columns', 150)
TenantID = []
TenantDisplayID = []
Name = []
FirstName = []
LastName = []
WebMessage = []
Comment = []
RentDueDay = []
RentPeriod = []
FirstContact = []
PropertyID = []
PostingStartDate = []
CreateDate = []
CreateUserID = []
UpdateDate = []
UpdateUserID = []
Contacts = []
for rm_access in read_content:
rm_data = rm_access
TenantID.append(rm_data["TenantID"])
TenantDisplayID.append(rm_data["TenantDisplayID"])
Name.append(rm_data["Name"])
FirstName.append(rm_data["FirstName"])
LastName.append(rm_data["LastName"])
WebMessage.append(rm_data["WebMessage"])
Comment.append(rm_data["Comment"])
RentDueDay.append(rm_data["RentDueDay"])
RentPeriod.append(rm_data["RentPeriod"])
# FirstContact.append(rm_data["FirstContact"])
PropertyID.append(rm_data["PropertyID"])
PostingStartDate.append(rm_data["PostingStartDate"])
CreateDate.append(rm_data["CreateDate"])
CreateUserID.append(rm_data["CreateUserID"])
UpdateUserID.append(rm_data["UpdateUserID"])
Contacts.append(rm_data["Contacts"])
df = pd.DataFrame({"TenantID":TenantID,"TenantDisplayID":TenantDisplayID, "Name"
: Name,"FirstName":FirstName, "LastName": LastName,"WebMessage": WebMessage,"Com
ment": Comment, "RentDueDay": RentDueDay, "RentPeriod": RentPeriod, "PropertyID"
: PropertyID, "PostingStartDate": PostingStartDate,"CreateDate": CreateDate, "Cr
eateUserID": CreateUserID,"UpdateUserID": UpdateUserID,"Contacts": Contacts})
print(df)
Here is sample of the file
[
{
"TenantID": 115,
"TenantDisplayID": 115,
"Name": "Jane Doe",
"FirstName": "Jane",
"LastName": "Doe",
"WebMessage": "",
"Comment": "",
"RentDueDay": 1,
"RentPeriod": "Monthly",
"FirstContact": "2015-11-01T15:30:00",
"PropertyID": 17,
"PostingStartDate": "2010-10-01T00:00:00",
"CreateDate": "2014-04-16T13:35:37",
"CreateUserID": 1,
"UpdateDate": "2017-03-22T11:31:48",
"UpdateUserID": 1,
"Contacts": [
{
"ContactID": 128,
"FirstName": "Jane",
"LastName": "Doe",
"MiddleName": "",
"IsPrimary": true,
"DateOfBirth": "1975-02-27T00:00:00",
"FederalTaxID": "111-11-1111",
"Comment": "",
"Email": "jane.doe#mail.com",
"License": "ZZT4532",
"Vehicle": "BMW 3 Series",
"IsShowOnBill": true,
"Employer": "REW",
"ApplicantType": "Applicant",
"CreateDate": "2014-04-16T13:35:37",
"CreateUserID": 1,
"UpdateDate": "2017-03-22T11:31:48",
"AnnualIncome": 0.0,
"UpdateUserID": 1,
"ParentID": 115,
"ParentType": "Tenant",
"PhoneNumbers": [
{
"PhoneNumberID": 286,
"PhoneNumberTypeID": 2,
"PhoneNumber": "703-555-5610",
"Extension": "",
"StrippedPhoneNumber": "7035555610",
"IsPrimary": true,
"ParentID": 128,
"ParentType": "Contact"
}
]
}
],
"UserDefinedValues": [
{
"UserDefinedValueID": 1,
"UserDefinedFieldID": 4,
"ParentID": 115,
"Name": "Emerg Contact Name",
"Value": "Terry Harper",
"UpdateDate": "2016-01-22T15:41:53",
"FieldType": "Text",
"UpdateUserID": 2,
"CreateUserID": 2
},
{
"UserDefinedValueID": 174,
"UserDefinedFieldID": 5,
"ParentID": 115,
"Name": "Emerg Contact Phone",
"Value": "703-555-3568",
"UpdateDate": "2016-01-22T15:42:03",
"FieldType": "Text",
"UpdateUserID": 2,
"CreateUserID": 2
}
],
"Leases": [
{
"LeaseID": 115,
"TenantID": 115,
"UnitID": 181,
"PropertyID": 17,
"MoveInDate": "2010-10-01T00:00:00",
"SortOrder": 1,
"CreateDate": "2014-04-16T13:35:37",
"UpdateDate": "2017-03-22T11:31:48",
"CreateUserID": 1,
"UpdateUserID": 1
}
],
"Addresses": [
{
"AddressID": 286,
"AddressTypeID": 1,
"Address": "14393 Montgomery Road Lot #102\r\nCincinnati, OH 45122",
"Street": "14393 Montgomery Road Lot #102",
"City": "Cincinnati",
"State": "OH",
"PostalCode": "45122",
"IsPrimary": true,
"ParentID": 115,
"ParentType": "Tenant"
}
],
"OpenReceivables": [],
"Status": "Current"
},
Not all tenants will have all elements which is also tricky
I need the data from the top where there is TenantID, TenantDisplayID, etc
I also need the data from the Contacts, PhoneNumbers, Leases, etc values
Each line should be static so if it doesn't have certain tags then I'd like a Null or None so it would look like
TentantID|TenantDisplayID|FirstName….etc so each line has same number of fields

Something like this should work:
import pandas as pd
pd.set_option('display.max_rows', 10000)
pd.set_option('display.max_columns', 100000)
TenantID = []
TenantDisplayID = []
Name = []
FirstName = []
LastName = []
WebMessage = []
Comment = []
RentDueDay = []
RentPeriod = []
FirstContact = []
PropertyID = []
PostingStartDate = []
CreateDate = []
CreateUserID = []
UpdateDate = []
UpdateUserID = []
Contacts = []
for rm_access in read_content:
rm_data = rm_access
print(rm_data)
TenantID.append(rm_data["TenantID"])
TenantDisplayID.append(rm_data["TenantDisplayID"])
Name.append(rm_data["Name"])
FirstName.append(rm_data["FirstName"])
LastName.append(rm_data["LastName"])
WebMessage.append(rm_data["WebMessage"])
Comment.append(rm_data["Comment"])
RentDueDay.append(rm_data["RentDueDay"])
RentPeriod.append(rm_data["RentPeriod"])
FirstContact.append(rm_data["FirstContact"])
PropertyID.append(rm_data["PropertyID"])
PostingStartDate.append(rm_data["PostingStartDate"])
CreateDate.append(rm_data["CreateDate"])
CreateUserID.append(rm_data["CreateUserID"])
UpdateUserID.append(rm_data["UpdateUserID"])
Contacts.append(rm_data["Contacts"])
df = pd.DataFrame({"TenantID":TenantID,"TenantDisplayID":TenantDisplayID, "Name": Name,
"FirstName":FirstName, "LastName": LastName,"WebMessage": WebMessage,
"Comment": Comment, "RentDueDay": RentDueDay, "RentPeriod": RentPeriod,
"FirstContact": FirstContact, "PropertyID": PropertyID, "PostingStartDate": PostingStartDate,
"CreateDate": CreateDate, "CreateUserID": CreateUserID,"UpdateUserID": UpdateUserID,
"Contacts": Contacts})
print(df)

The General Problem
The problem with this task (and other similar ones) is not just how to create an algorithm - I am sure you will theoretically be able to solve this with a (not so) nice amount of nested for-loops. The problem is to organise the code in a way that you don't get a headache - i.e. in a way that you can fix bugs easily, that you can write unittests, that you can understand the code easily from reading it (in six months from now) and that you can easily change your code in case you need to do so.
I do not know anybody who does not make mistakes when wrapping their head around a deeply nested structure. And chasing for bugs in a code which is heavily nested because it mirrors the nested structure of the data, can be quite frustrating.
The Quick (and most probably: Best) Solution
Rely on packages that are made for your exact usecase, such as
https://github.com/cwacek/python-jsonschema-objects
In case you have a formal definition of the API schema, you could use packages for that. If, for instance, your API has a Swagger schema definition, you cann use swagger-py (https://github.com/digium/swagger-py) to get your JSON response into Python objects.
The Principle Solution: Object Oriented Programming and Recursion
Even if there might be some libraries for your concrete use case, I would like to explain the principle of how to deal with "that kind" of tasks:
A good way to organise code for this kind of problem is using Object Oriented Programming. The nesting hassle can be laid out much clearer by making use of the principle of recursion. This also makes it easier to chabge the code, in case the JSON schema of your API response changes for any reasons (an update of the API, for instance). In your case I would suggest you create something like the following:
class JsonObject:
"""Parent Class for any Object that will be retrieved from the JSON
and potentially has nested JsonObjects inside.
This class takes care of parsing the json into python Objects and deals
with the recursion into the nested structures."""
primitives = []
json_objects = {
# For each class, this dict defines all the "embedded" classes which
# live directly "under" that class in the nested JSON. It will have the
# following structure:
# attribute_name : class
# In your case the JSON schema does not have any "single" objects
# in the nesting strcuture, but only lists of nested objects. I
# still , to demonstrate how you would do it in case, there would be
# single "embedded"
}
json_object_lists = {
# For each class, this dict defines all the "embedded" subclasses which
# are provided in a list "under" that class in the nested JSON.
# It will have the following structure:
# attribute_name : class
}
#classmethod
def from_dict(cls, d: dict) -> "JsonObject":
instance = cls()
for attribute in cls.primitives:
# Here we just parse all the primitives
instance.attribute = getattr(d, attribute, None)
for attribute, klass in cls.json_object_lists.items():
# Here we parse all lists of embedded JSON Objects
nested_objects = []
l = getattr(d, attribute, [])
for nested_dict in l:
nested_objects += klass.from_dict(nested_dict)
setattr(instance, attribute, nested_objects)
for attribute, klass in cls.json_objects.items():
# Here we parse all "single" embedded JSON Objects
setattr(
instance,
attribute,
klass.from_dict(getattr(d, attribute, None)
)
def to_csv(self) -> str:
pass
Since you didn't explain how exactly you want to create a csv from the JSON, I didn't implement that method and left this to you. It is also not necessary to explain the overall approach.
Now we have the general Parent class all our specific will inherit from, so that we can apply recursion to our problem. Now we only need to define these concrete structures, according to the JSON schema we want to parse. I got the following from your sample, but you can easily change the things you need to:
class Address(JsonObject):
primitives = [
"AddressID",
"AddressTypeID",
"Address",
"Street",
"City",
"State",
"PostalCode",
"IsPrimary",
"ParentID",
"ParentType",
]
json_objects = {}
json_object_lists = {}
class Lease(JsonObject):
primitives = [
"LeaseID",
"TenantID",
"UnitID",
"PropertyID",
"MoveInDate",
"SortOrder",
"CreateDate",
"UpdateDate",
"CreateUserID",
"UpdateUserID",
]
json_objects = {}
json_object_lists = {}
class UserDefinedValue(JsonObject):
primitives = [
"UserDefinedValueID",
"UserDefinedFieldID",
"ParentID",
"Name",
"Value",
"UpdateDate",
"FieldType",
"UpdateUserID",
"CreateUserID",
]
json_objects = {}
json_object_lists = {}
class PhoneNumber(JsonObject):
primitives = [
"PhoneNumberID",
"PhoneNumberTypeID",
"PhoneNumber",
"Extension",
"StrippedPhoneNumber",
"IsPrimary",
"ParentID",
"ParentType",
]
json_objects = {}
json_object_lists = {}
class Contact(JsonObject):
primitives = [
"ContactID",
"FirstName",
"LastName",
"MiddleName",
"IsPrimary",
"DateOfBirth",
"FederalTaxID",
"Comment",
"Email",
"License",
"Vehicle",
"IsShowOnBill",
"Employer",
"ApplicantType",
"CreateDate",
"CreateUserID",
"UpdateDate",
"AnnualIncome",
"UpdateUserID",
"ParentID",
"ParentType",
]
json_objects = {}
json_object_lists = {
"PhoneNumbers": PhoneNumber,
}
class Tenant(JsonObject):
primitives = [
"TenantID",
"TenantDisplayID",
"Name",
"FirstName",
"LastName",
"WebMessage",
"Comment",
"RentDueDay",
"RentPeriod",
"FirstContact",
"PropertyID",
"PostingStartDate",
"CreateDate",
"CreateUserID",
"UpdateDate",
"UpdateUserID",
"OpenReceivables", # Maybe this is also a nested Object? Not clear from your sample.
"Status",
]
json_object_lists = {
"Contacts": Contact,
"UserDefinedValues": UserDefinedValue,
"Leases": Lease,
"Addresses": Address,
}
json_objects = {}
You might imagine the "beauty" (at least: order) of that approach, which lies in the following: With this structure, we could tackle any level of nesting in the JSON response of your API without additional headache - our code would not deepen its indentation level, because we have separated the nasty nesting into the recursive definition of JsonObjects from_json method. That is why it is much easier now to identify bugs or apply changes to our code.
To finally parse the JSON now into our Objects, you would do something like the following:
import typing
import json
def tenants_from_json(json_string: str) -> typing.Iterable["Tenant"]:
tenants = [
Tenant.from_dict(tenant_dict)
for tenant_dict in json.loads(json_string)
]
return tenants
Important Final Side Note: This is just the basic Principle
My code example is just a very brief introduction into the idea of using objects and recursion to deal with an overwhelming (and nasty) nesting of a structure. The code has some flaws. For instance one should avoid define mutable class variables. And of course the whole code should validate the data it gets from the API. You also might want to add the type of each attribute and represent that correctly in the Python objects (Your sample has integers, datetimes and strings, for instance).
I really only wanted to show you the very principle of Object Oriented Programming here.
I didn't take the time to test my code. So there are probably bugs left. Again, I just wanted to demonstrate the principle.

How should I add a field containing a list of dictionaries in Marshmallow Python?

In Marshmallow in order to have a list field you can use:
include_in = fields.List(cls_or_instance=fields.Str(),
default=['sample1', 'sample2'])
This is OK, but I have a new requirement to have a list of dictionaries in a field. A sample payload:
[{
"name": "Ali",
"age": 20
},
{
"name": "Hasan",
"age": 32
}]
This payload is part of the bigger schema, so now the question is how should I add and validate such a field?
EDIT-1:
I went a step further and could find out that there is a Dict field type in Marshmallow so until now I have the below code sample:
fields.List(fields.Dict(
keys=fields.String(validate=OneOf(('name', 'age'))),
values=fields.String(required=True)
))
Now new problem arise and I cannot set different data types for fields in the dictionary (name and age). I'd be happy if someone could shed some light on this.

If the items in the list have the same shape, you can use a nested field within fields.List, like so:
class PersonSchema(Schema):
name = fields.Str()
age = fields.Int()
class RootSchema(Schema):
people = fields.List(fields.Nested(PersonSchema))

Another approach for validating list of dictionaries in a field using one schema class.
from marshmallow import Schema, ValidationError
class PeopleSchema(Schema):
name = fields.Str(required=True)
age = fields.Int(required=True)
people = [{
"name": "Ali",
"age": 20
},
{
"name": "Hasan",
"age": 32
},
{
"name": "Ali",
"age": "twenty" # Error: Not an int
}
]
def validate_people():
try:
validated_data = PeopleSchema(many=True).load(people)
except ValidationError as err:
print(err.messages)
validate_people()
Output:
{2: {'age': ['Not a valid integer.']}}

Develop Reference

Python is a programming language that lets you work quickly and integrate systems more effectively.

How can I store data in a dictionary without overriding it? - python

Related

python getting json values from list

Save List of object with heterogenous keys in csv file in python

Python: How to filter dictionary?

How can I parse nested JSON to CSV

How should I add a field containing a list of dictionaries in Marshmallow Python?

Categories

Resources