Get List of Distinct From Nested List - python

I'm brand new to Python coming over from a javascript/nodeJS upbringing. I have a python dictionary with a nested list that I'm having lots of trouble accessing. I need to pull out the nested list so I can check for unique e-mails. Here is what my GET request pulls back:
{"data": [{"login_date": "2014-04-17T19:14:29+08:00", "email": "walter.elwyn#yahoo.com"}, {"login_date": "2014-04-22T09:31:56+04:00", "email": "helyn67#cruickshankmckenzie.com"}]
The actual data dictionary that gets returned is much longer but you get the point... so I have two questions:
how do I access a specific point by e-mail in the content dictionary
how do I loop over the dictionary for unique emails?
Here is my code:
from flask import Flask
import requests
import urllib2
app = Flask(__name__)
#app.route('/')
def hello_world():
content = urllib2.urlopen('https://9g9xhayrh5.execute-api.us-west-2.amazonaws.com/test/data').read()
print content
return 'check console'
if __name__ == '__main__':
app.run()


If you have a structure like this:
>>> response = {"data": [{"login_date": "2014-04-17T19:14:29+08:00", "email": "walter.elwyn#yahoo.com"}, {"login_date": "2014-04-22T09:31:56+04:00", "email": "helyn67#cruickshankmckenzie.com"}]}
Then what you have is a dict with key data
>>> response['data']
[{'login_date': '2014-04-17T19:14:29+08:00', 'email': 'walter.elwyn#yahoo.com'}, {'login_date': '2014-04-22T09:31:56+04:00', 'email': 'helyn67#cruickshankmckenzie.com'}]
You can get a list of emails with a list comprehension:
>>> [user['email'] for user in response['data']]
['walter.elwyn#yahoo.com', 'helyn67#cruickshankmckenzie.com']
How do I loop over the dictionary for unique emails?
To make that list unique, you can use a set comprehension:
>>> {user['email'] for user in response['data']}
set(['helyn67#cruickshankmckenzie.com', 'walter.elwyn#yahoo.com'])
How do I access a specific point by e-mail in the content dictionary
If you want to filter for a given email, you can use filter()
>>> filter(lambda user: user['email'] == 'walter.elwyn#yahoo.com', response['data'])
[{'login_date': '2014-04-17T19:14:29+08:00', 'email': 'walter.elwyn#yahoo.com'}]
If I do content['data] in my case, I get an internal server error
That's because to do content['data'], content needs to be a dict-like object. Let's use requests lib to do what you want:
>>> import requests
>>> content = requests.get('https://9g9xhayrh5.execute-api.us-west-2.amazonaws.com/test/data')
>>> response = content.json() # reads the data as JSON, into a dict-like object
>>> response.keys()
[u'data']
Now you can use response['data']

Related

How to print attributes of a json file

I would appreciate some help: how could I print just the country from the info obtained via this API call? Thanks!
import requests
import json
url = "https://randomuser.me/api/"
data = requests.get(url).json()
print(data)

You should play a little more with the json in order to learn how to use it, a helpful way to understand them is to go layer by layer printing the keys dict.keys() to see where you should go next if you dont have a documentation
in this particular case it returns a dictionary with the following first layer structure:
{
"results": [ ... ]
"info": { ... }
}
where results contains a single dictionary inside, therefore we can take
data['results'][0] to wok with
there is 'location', and there is a 'country', you can access this in that order to print the country:
print(data['results'][0]['location']['country'])

"KeyError: 'added' "when importing different lists of json

I'm getting a "KeyError: 'added'" Error when i try to import one of my three lists inside a json file. The list i'm trying to import is the one called "added". I had this working when it was just one list without the name "added" on top but now it seems like i can't access the list anymore. I want to individually import them basically.
This is my code to get the Json parts before i import them into the database:
import requests
import json
from users.models import Facility, FacilityAddress, FacilityInspectionInfo, FacilityComplaints
from django.core.management.base import BaseCommand
IMPORT_URL = 'https://url/imports.json'
class Command(BaseCommand):
def import_facility_from_file(self, data):
UUID = data.get('UUID', None)
Name = data.get('Name', None)
PrimaryAddress = data["AddressInfo"]["PrimaryAddress"]
""" This is what happens after the database entry related code """
def handle(self, *args, **options):
"""
Call the function to import data from json url
"""
headers = {'Content-Type': 'application/json'}
response = requests.get(
url=IMPORT_URL,
headers=headers,
)
response.raise_for_status()
data = response.json()
for key, data_object in data.items():
self.import_facility_from_file(data_object)
The new version of the Json File that im trying to use but thats causing the error:
{
"added":
{"125hk24h5kjh43k5":
{
"UUID":"125hk24h5kjh43k5",
"Name":"Test Facility 1",
"AddressInfo":
{"PrimaryAddress":"1234 Drive RD"},
"ImporterLastModifiedTimestamp":1643721420}},
"deleted":["235hk24h5kjh43k5,235hk345789h43k5"],
"modified":{"995hk24h5kjh43k5":
{
"UUID":"995hk24h5kjh43k5",
"Name":"Test Facility 2",
"AddressInfo":
{"PrimaryAddress":"2345 Test RD"},
"ImporterLastModifiedTimestamp":1643721420}
}
}
The old version of the json file that worked perfectly with the code i intially wrote:
{"00016ed7be4872a19d6e16afc98a7389b2bb324a2":
{"UUID":"00016ed7be4872a19d6e1ed6f36b647f3eb41cadedd2130b103a5851caebc26fbbbf24c2f1a64d2cf34ac4e03aaa30309816f58c397e6afc98a7389b2bb324a2","Name":"Test Facility","IssuedNumber":"123456","Licensee":"Test Licensee","Email":"test#example.com","AdministratorName":"Test Name","TelephoneNumber":"(123) 456-7890324879","ImporterLastModifiedTimestamp":"1362985200",
"AddressInfo":{"PrimaryAddress":"123 Fake Road","SecondaryAddress":"","City":"Testcity","RegionOrState":"TX","PostalCode":"12345","Geolocation":"00.0000,-00.0000"},"Capacity":100,"MostRecentLicenseTimestamp":1575180000,"ClosedTimestamp":0,
"InspectionInfo":{"ComplaintRelatedVisits":0,"InspectionRelatedVisits":0,"NumberOfVisits":0,"LastVisitTimestamp":0},
"Complaints":{"ComplaintsTypeA":0,"ComplaintsTypeB":0,"SubstantiatedAllegations":0,"TotalAllegations":0}},
"00016ed7be4872a15435435435b2bb324a2":
{"UUID":"000c93dcb7a0b3d5783bb330892aff6abdb9fb57a7d3701c2d903f3640877579f3173ecd8a80532f6c3d53dbacde78a6a54ae42fef321a5793f5a01934f8de7a","Name":"Test Facility 2","IssuedNumber":"123456","Licensee":"Test Licensee","Email":"test#example.com","AdministratorName":"Test Name","TelephoneNumber":"(123) 456-7890324879","ImporterLastModifiedTimestamp":"1362985200",
"AddressInfo":{"PrimaryAddress":"123 Fake Road","SecondaryAddress":"","City":"Testcity","RegionOrState":"TX","PostalCode":"12345","Geolocation":"00.0000,-00.0000"},"Capacity":100,"MostRecentLicenseTimestamp":1575180000,"ClosedTimestamp":0,
"InspectionInfo":{"ComplaintRelatedVisits":0,"InspectionRelatedVisits":0,"NumberOfVisits":0,"LastVisitTimestamp":0},
"Complaints":{"ComplaintsTypeA":0,"ComplaintsTypeB":0,"SubstantiatedAllegations":0,"TotalAllegations":0}},
"00234324324343243afc98a7389b2bb324a2":
{"UUID":"fffd4dec10054e6e1deb2a2266a7c6bb0136ba46222e734ceed5855651f735cfbe0bb66cfaf27c3d175ae261a8f6df0c36b5390d15c70b07d67e35e1081aaf6d","Name":"Test Facility 3","IssuedNumber":"123456","Licensee":"Test Licensee","Email":"test#example.com","AdministratorName":"Test Name","TelephoneNumber":"(123) 456-7890324879","ImporterLastModifiedTimestamp":"1362985200",
"AddressInfo":{"PrimaryAddress":"123 Fake Road","SecondaryAddress":"","City":"Testcity","RegionOrState":"TX","PostalCode":"12345","Geolocation":"00.0000,-00.0000"},"Capacity":100,"MostRecentLicenseTimestamp":1575180000,"ClosedTimestamp":0,
"InspectionInfo":{"ComplaintRelatedVisits":0,"InspectionRelatedVisits":0,"NumberOfVisits":0,"LastVisitTimestamp":0},
"Complaints":{"ComplaintsTypeA":0,"ComplaintsTypeB":0,"SubstantiatedAllegations":0,"TotalAllegations":0}}}
So i tried it like this to get the UUID and other information from the modified json file:
UUID = data["added"]["UUID"]
but i'm getting this error:
KeyError: 'added'

It sounds like the json format has changed underneath you and you need to adapt.
How about processing all the items in both the "added" and "modified" sections?
for key, data_object in data.items():
if key in ["added", "modified"]:
for key, data in data_object.items():
self.import_facility_from_file(data)

AttributeError: 'list' object has no attribute 'get' because you've list data type in between your dictionary so you can add check for type like this
for key, data_object in data.items():
if type(data_object) == dict:
for data in data_object.values():
self.import_facility_from_file(data)

Extract value from uncorrectly parsed dict from json output

I am processing kafka streams in a python flask server. I read the responses with json and need to extract the udid values from the stream. I read each response with request.json and save it in a dictionary. When i try to parse it fails. the dict contains the following values
dict_items([('data', {'SDKVersion': '7.1.2', 'appVersion': '6.5.5', 'dateTime': '2019-08-05 15:01:28+0200', 'device': 'iPhone', 'id': '3971',....})])
parsing the dict the normal way doesnt work ie event_data['status'] gives error.Perhaps it is because its not a pure dict....?
#app.route('/data/idApp/5710/event/start', methods=['POST'])
def give_greeting():
print("Hola")
event_data = request.json
print(event_data.items())
print(event_data['status'])
#print(event_data['udid'])
#print(event_data['Additional'])
return 'Hello, {0}!'.format(event_data)
The values contained in event data are the following
dict_items([('data', {'SDKVersion': '7.1.2', 'appVersion': '6.5.5', 'dateTime': '2019-08-05 15:01:28+0200', 'device': 'iPhone', 'id': '3971',....})])
The expected result would be this result
print(event_data['status'])->start
print(event_data['udid'])->BAEB347B-9110-4CC8-BF99-FA4039C4599B
print(event_data['SDKVersion'])->7.1.2
etc
the output of
print(event_data.keys()) is dict_keys(['data'])

The data you are expecting is wrapped in an additional data property.
You only need to do one extra step to access this data.
data_dict = request.json
event_data = data_dict['data']
Now you should be able to access the information you want with
event_data['SDKVersion']
...
as you have described above.
As #jonrsharpe stated, this is not an issue with the parsing. The parsing either fails or succeeds, but you will never get a "broken" object (be it dict, list, ...) from parsing JSON.

Get all usernames stored in TinyDB database

I'm new to TinyDB, and new to Python.
I have various usernames stored in a TinyDB database, they also have other information stored (age, email addresses etc), however, I wish to return all the usernames only.
{"_default": {"1": {"Username": "John", "Age": "30"}, "2": {"Username":
"Andrew", "Age":"40", "Email": "example#example.com"}}}
My GUI would have the button "Show all usernames".
I can return information about specific users, and I can get all the information stored in the database (db.all()), however I cannot just seem to get all the usernames from the entire database.
Is there a way to do this?
Or am I looking at this problem the wrong way.
Many thanks!

The database itself is iterable, so perhaps this would be more elegant and would avoid having to open the JSON file directly:
db = TinyDB('database_name.json')
usernames = [r['Username'] for r in db]
Gives:
['John', 'Andrew']

Update: I found a (slightly convoluted) workaround.
It involves reading the file as a JSON file and then looping through the dictionary IDs, stopping the loop upon a key error.
with open("database_name.json", "r") as read_file:
data=json.load(read_file)
try:
current_number = 1
while current_number <=100000000000:
current_number = str(current_number)
print(data['_default'][current_number]['Username'])
current_number = int(current_number)
current_number += 1
except:
KeyError

How can i make an API wrapper for a HTTP service that uses json?

I want to make a HTTP wrapper for twitch.tv in python. How would I do this? I know how to use HTTP GET and that's about it. I would like it to work like this:
import twichtvwrapper
twich = twichtvwrapper(useragent, username, password).
channel = twich.channel(channelname)
then all the json properties would go in here like:
print(channel.game) #this would say the last played game
print(channel.displayname) #this would print the display name of the channel
print(cahnnel.link.chat) #this will return the chat link
etc.
How would I go about making this wrapper?

You can convert between Python dicts and JSON strings with the json module of the Standard Library.
import json
# package a python dict as json
dict0 = {'spam': 'data', 'eggs': 'more data'}
pack = json.dumps(dict0)
# turn it back to a dict
dict1 = json.loads(pack)
>>> print dict1['spam']
data
So, if your program gets a JSON response from a HTTP request, just convert it to a dict and use regular Python dict methods to do the rest.

Categories

Resources