OK, I'm a newbie and I think I'm doing everything I should be, but I am still getting a KeyError: venues. (I also tried using "venue" instead and I am not at my maximum quota for the day at FourSquare)... I am using a Jupyter Notebook to do this
Using this code:
VERSION = '20200418'
RADIUS = 1000
LIMIT = 2
**url = 'https://api.foursquare.com/v2/venues/explore?client_id={}&client_secret={}&ll={},{}&v={}&radius={}&limit={}'.format(CLIENT_ID, CLIENT_SECRET, latitude, longitude, VERSION, RADIUS, LIMIT)
url
results = requests.get(url).json()**
I get 2 results (shown at end of this post)
When I try to take those results and put them into a dataframe, i get "KeyError: venues"
# assign relevant part of JSON to venues
venues = results['response']['venues']
# tranform venues into a dataframe
dataframe = json_normalize(venues)
dataframe.head()
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-29-5acf500bf9ad> in <module>
1 # assign relevant part of JSON to venues
----> 2 venues = results['response']['venues']
3
4 # tranform venues into a dataframe
5 dataframe = json_normalize(venues)
KeyError: 'venues'
I'm not really sure where I am going wrong... This has worked for me with other locations... But then again, like I said, I'm new at this... (I haven't maxed out my queries, and I've tried using "venue" instead)... Thank you
FourSquareResults:
{'meta': {'code': 200, 'requestId': '5ec42de01a4b0a001baa10ff'},
'response': {'suggestedFilters': {'header': 'Tap to show:',
'filters': [{'name': 'Open now', 'key': 'openNow'}]},
'warning': {'text': "There aren't a lot of results near you. Try something more general, reset your filters, or expand the search area."},
'headerLocation': 'Cranford',
'headerFullLocation': 'Cranford',
'headerLocationGranularity': 'city',
'totalResults': 20,
'suggestedBounds': {'ne': {'lat': 40.67401708586377,
'lng': -74.29300815204098},
'sw': {'lat': 40.65601706786374, 'lng': -74.31669390523408}},
'groups': [{'type': 'Recommended Places',
'name': 'recommended',
'items': [{'reasons': {'count': 0,
'items': [{'summary': 'This spot is popular',
'type': 'general',
'reasonName': 'globalInteractionReason'}]},
'venue': {'id': '4c13c8d2b7b9c928d127aa37',
'name': 'Cranford Canoe Club',
'location': {'address': '250 Springfield Ave',
'crossStreet': 'Orange Avenue',
'lat': 40.66022488705574,
'lng': -74.3061084180977,
'labeledLatLngs': [{'label': 'display',
'lat': 40.66022488705574,
'lng': -74.3061084180977},
{'label': 'entrance', 'lat': 40.660264, 'lng': -74.306191}],
'distance': 543,
'postalCode': '07016',
'cc': 'US',
'city': 'Cranford',
'state': 'NJ',
'country': 'United States',
'formattedAddress': ['250 Springfield Ave (Orange Avenue)',
'Cranford, NJ 07016',
'United States']},
'categories': [{'id': '4f4528bc4b90abdf24c9de85',
'name': 'Athletics & Sports',
'pluralName': 'Athletics & Sports',
'shortName': 'Athletics & Sports',
'icon': {'prefix': 'https://ss3.4sqi.net/img/categories_v2/shops/sports_outdoors_',
'suffix': '.png'},
'primary': True}],
'photos': {'count': 0, 'groups': []},
'venuePage': {'id': '60380091'}},
'referralId': 'e-0-4c13c8d2b7b9c928d127aa37-0'},
{'reasons': {'count': 0,
'items': [{'summary': 'This spot is popular',
'type': 'general',
'reasonName': 'globalInteractionReason'}]},
'venue': {'id': '4d965995e07ea35d07e2bd02',
'name': 'Mizu Sushi',
'location': {'address': '103 Union Ave.',
'lat': 40.65664427772896,
'lng': -74.30343966195308,
'labeledLatLngs': [{'label': 'display',
'lat': 40.65664427772896,
'lng': -74.30343966195308}],
'distance': 939,
'postalCode': '07016',
'cc': 'US',
'city': 'Cranford',
'state': 'NJ',
'country': 'United States',
'formattedAddress': ['103 Union Ave.',
'Cranford, NJ 07016',
'United States']},
'categories': [{'id': '4bf58dd8d48988d1d2941735',
'name': 'Sushi Restaurant',
'pluralName': 'Sushi Restaurants',
'shortName': 'Sushi',
'icon': {'prefix': 'https://ss3.4sqi.net/img/categories_v2/food/sushi_',
'suffix': '.png'},
'primary': True}],
'photos': {'count': 0, 'groups': []}},
'referralId': 'e-0-4d965995e07ea35d07e2bd02-1'}]}]}}
Look more closely at response that you're getting - there's no "venues" key there. Closest one that I see is "groups" list, which has "items" list in it, and individual items have "venue" key in them.
Related
I have a dictionary of OSM data that I queried using overpy. I would like to get this data into a DF With each tag as its own column. I am having trouble with the tags as they are nested within the dictionary and cannot reference the tags by name as they change from feature to feature
Example of dictionary:
{'version': 0.6,
'generator': 'Overpass API 0.7.57 93a4d346',
'osm3s': {'timestamp_osm_base': '2022-05-18T13:41:10Z',
'timestamp_areas_base': '2022-05-18T13:27:04Z',
'copyright': 'The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.'},
'elements': [{'type': 'node',
'id': 33827805,
'lat': 46.2519258,
'lon': -60.5281258,
'tags': {'ele': '240', 'name': "Kelly's Mountain", 'natural': 'peak'}},
{'type': 'node',
'id': 244245796,
'lat': 56.008075,
'lon': -130.003485,
'tags': {'alt_name': 'Monument 8',
'ele': '1570',
'historic': 'boundary_stone',
'man_made': 'survey_point',
'name': 'Mount Welker',
'name:fr': 'Borne frontière 8',
'natural': 'peak',
'operator': 'International Boundary Commission',
'ref': 'MON 8',
'source': 'CA-US International Boundary Commission;NRCan-CanVec-10.0',
'start_date': '1905'}},
{'type': 'node',
'id': 244245903,
'lat': 56.2650369,
'lon': -130.615449,
'tags': {'alt_name': 'Monument 27',
'ele': '1667',
'historic': 'boundary_stone',
'man_made': 'survey_point',
'name': 'Mount Middleton',
'name:fr': 'Borne frontière 27',
'natural': 'peak',
'operator': 'International Boundary Commission',
'ref': 'MON 27',
'source': 'CA-US International Boundary Commission',
'start_date': '1920'}},
{'type': 'node',
'id': 244245911,
'lat': 56.3668442,
'lon': -130.78198,
'tags': {'alt_name': 'Monument 40',
'ele': '1828',
'historic': 'boundary_stone',
'man_made': 'survey_point',
'name': 'Mount Stoeckl',
'name:fr': 'Borne frontière 40',
'natural': 'peak',
'operator': 'International Boundary Commission',
'ref': 'MON 40',
'source': 'CA-US International Boundary Commission',
'start_date': '1905'}},
{'type': 'node',
'id': 244245929,
'lat': 56.4060047,
'lon': -131.087348,
'tags': {'alt_name': 'Unmarked Boundary Point 47',
'ele': '1840',
'man_made': 'survey_point',
'name': 'Mount Lewis Cass',
'name:fr': 'Point frontalier non-marqué 47',
'natural': 'peak',
'operator': 'International Boundary Commission',
'ref': 'BP 47',
'source': 'CA-US International Boundary Commission',
'start_date': '1907'}},
...]}
My desired end result would be a dataframe as such but with a column for all tag types. I.e. a column titled 'ele' with the first row value being 240 and the second with a value of 1570
I have this list that contains dictionaries as its element
dict_1 = [{'id': '0eb7df70-f319-4562-ab2a-9e641e978b3b', 'first_name': 'Rahx', 'surname': 'Smith ', 'devices': {'os': 'Apple iPhone', 'mac_address': 'f4:af:e7:b7:ab:22', 'manufacturer': 'Apple'}, 'lat': 54.33166199876629, 'lng': -6.277842272724769, 'seenTime': 1582814754000},
{'id': 'a0bb8d38-0d27-4d7f-acc0-1e850a706b6c', 'first_name': 'Lucy', 'surname': 'Pye', 'devices': {'os': 'Apple iPhone', 'mac_address': 'f8:87:f1:72:4c:4d', 'manufacturer': 'Apple'}, 'lat': 54.33166199876629, 'lng': -6.277842272724769, 'seenTime': 1582814754000},
{'id': '0eb7df70-f319-4562-ab2a-9e641e978b3b', 'first_name': 'xyx', 'surname': 'dcsdd', 'devices': {'os': 'NOKIA Phone', 'mac_address': '78:28:ca:a8:56:b9', 'manufacturer': 'NOKIA'}, 'lat': 54.33166199876629, 'lng': -6.277842272724769, 'seenTime': 1582814754000},
{'id': 'a0bb8d38-0d27-4d7f-acc0-1e850a706b6c', 'first_name': 'ddwdw', 'surname': 'sdsds', 'devices': {'os': 'MI Phone', 'mac_address': 'dc:08:0f:3f:57:0c', 'manufacturer': 'MI'}, 'lat': 54.33218267030654, 'lng': -6.27796001203896, 'seenTime': 1582814693000}]
and I want output like this from dict_1 variable
{
"f77df8c2-b19d-4341-9021-7beab4b9ebcd":{
"first_name":"anonymous",
"surname":"anonymous",
"lat":57.14913102,
"lng":-2.09987143,
"devices": {'os': 'MI Phone', 'mac_address': 'dc:08:0f:3f:57:0c', 'manufacturer': 'MI'},
"seenTime": 1582814693000
},
"7beab4b9ebcd-b19d-9021-f77df8c2-4341":{
etc.
},
etc.
}
help me to know what should I do in this case.
Try this.
dict_1 = {x.pop('id'): x for x in dict_1}
I think this could do the job :
dict_2 = {}
for d in dict_1 :
id = d.pop('id')
dict_2[id] = d
I want to be able to GET information from API 1 and match it with API 2 and be able to update API 2's information with API 1. I am trying to figure out the most efficient/automated way to accomplish this as it also needs to be updated at a interval of every 10 minutes
I can query and get the results from API 1 this is my code and what my code looks like.
import json
import requests
myToken = '52c32f6588004cb3ab33b0ff320b8e4f'
myUrl = 'https://api1.com/api/v1/devices.json'
head = {'Authorization': 'Token {}'.format(myToken)}
response = requests.get(myUrl, headers=head)
r = json.loads(response.content)
r
The payload looks like this from API 1
{ "device" : {
"id": 153,
"battery_status" : 61,
"serial_no": "5QBYGKUI05",
"location_lat": "-45.948917",
"location_lng": "29.832179",
"location_address": "800 Laurel Rd, Lansdale, PA 192522,USA"}
}
I want to be able to take this information and match by "serial_no" and update all the other pieces of information for the corresponding device in API 2
I query the data for API 2 and this is what my code looks like
params = {
"location":'cf6707e3-f0ae-4040-a184-737b21a4bbd1',
"dateAdded":'ge:11/23/2020'}
url = requests.get('https://api2.com/api/assets',auth=('api2', '123456'), params=params)
r = json.loads(url.content)
r['items']
The JSON payload looks like this
[{'id': '064ca857-3783-460e-a7a2-245e054dcbe3',
'name': 'Apple Laptop 1',
'model': {'id': '50f5993e-2abf-49c8-86e0-8743dd58db6f',
'name': 'MacBook Pro'},
'manufacturer': {'id': 'f56244e2-76e3-46da-97dd-f72f92ca0779',
'name': 'APPLE'},
'room': {'id': '700ff2dc-0118-46c6-936a-01f0fa88c620',
'name': 'Storage Room 1',
'thirdPartyId': ''},
'location': {'id': 'cf6707e3-f0ae-4040-a184-737b21a4bbd1',
'name': 'Iron Mountain',
'thirdPartyId': ''},
'position': 'NonMounted',
'containerAsset': {'id': '00000000-0000-0000-0000-000000000000',
'name': None},
'baseAsset': {'id': '064ca857-3783-460e-a7a2-245e054dcbe3',
'name': 'Apple Laptop 1'},
'description': None,
'status': {'id': 'df9906d8-2856-45e3-9cba-bd7a1ac4971f',
'name': 'Production'},
'serialNumber': '5QBYGKUI06',
'tagNumber': None,
'alternateTagNumber': None,
'verificationStatus': {'id': 'cb3560a9-eef5-47b9-b033-394d3a09db18',
'name': 'Verified'},
'requiresRFID': False,
'requiresHangTag': False,
'bottomPosition': 0.0,
'leftPosition': 0.0,
'rackPosition': 'Front',
'labelX': None,
'labelY': None,
'verifyNameInRear': False,
'verifySerialNumberInRear': False,
'verifyBarcodeInRear': False,
'isNonDataCenter': False,
'rotate': False,
'customer': {'id': '00000000-0000-0000-0000-000000000000', 'name': None},
'thirdPartyId': '',
'temperature': None,
'dateLastScanned': None,
'placement': 'Floor',
'lastScannedLabelX': None,
'lastScannedLabelY': None,
'userDefinedValues': [{'userDefinedKeyId': '79e77a1e-4030-4308-a8ff-9caf40c04fbd',
'userDefinedKeyName': 'Longitude ',
'value': '-75.208917'},
{'userDefinedKeyId': '72c8056e-9b7d-40ac-9270-9f5929097e82',
'userDefinedKeyName': 'Address',
'value': '800 Laurel Rd, New York ,NY 19050, USA'},
{'userDefinedKeyId': '31aeeb91-daef-4364-8dd6-b0e3436d6a51',
'userDefinedKeyName': 'Battery Level',
'value': '67'},
{'userDefinedKeyId': '22b7ce4f-7d3d-4282-9ecb-e8ec2238acf2',
'userDefinedKeyName': 'Latitude',
'value': '35.932179'}]}
The documentation provided by API 2 tells me they only support PUT for updates as of right now but I would also want to know how I would do this using PATCH as it will be available in the future. So the data payload that I need to successful PUT is this
payload = {'id': '064ca857-3783-460e-a7a2-245e054dcbe3',
'name': 'Apple Laptop 1',
'model': {'id': '50f5993e-2abf-49c8-86e0-8743dd58db6f',
'name': 'MacBook Pro'},
'manufacturer': {'id': 'f56244e2-76e3-46da-97dd-f72f92ca0779',
'name': 'APPLE'},
'room': {'id': '700ff2dc-0118-46c6-936a-01f0fa88c620',
'name': 'Storage Room 1',
'thirdPartyId': ''},
'status': {'id': 'df9906d8-2856-45e3-9cba-bd7a1ac4971f',
'name': 'Production'},
'serialNumber': '5QBYGKUI06',
'verificationStatus': {'id': 'cb3560a9-eef5-47b9-b033-394d3a09db18',
'name': 'Verified'},
'requiresRFID': 'False',
'requiresHangTag': 'False',
'userDefinedValues': [{'userDefinedKeyId': '79e77a1e-4030-4308-a8ff-9caf40c04fbd',
'userDefinedKeyName': 'Longitude ',
'value': '-75.248920'},
{'userDefinedKeyId': '72c8056e-9b7d-40ac-9270-9f5929097e82',
'userDefinedKeyName': 'Address',
'value': '801 Laurel Rd, New York, Ny 192250, USA'},
{'userDefinedKeyId': '31aeeb91-daef-4364-8dd6-b0e3436d6a51',
'userDefinedKeyName': 'Battery Level',
'value': '67'},
{'userDefinedKeyId': '22b7ce4f-7d3d-4282-9ecb-e8ec2238acf2',
'userDefinedKeyName': 'Latitude',
'value': '29.782177'}]}
So apart of this is figuring out how I can query the json data portions that I need for the update
I am able to update the information using this line
requests.put('https://api2.com/api/assets/064ca857-3783-460e-a7a2-245e054dcbe3',auth=('API2', '123456'), data=json.dumps(payload))
but I need for it to dynamically update so I don't think the hard coded id parameter in the line will be efficient in a automation/efficiency standpoint. If anybody has any ideas, resources to point me in the right direction to know more about this process (I don't really know what it is even called) would be greatly appreciated.
Not entirely sure what you are trying to do here, but if you want to pull information nested in the responses you can do this.
Serial number from API 1
r['device']['serial_no']
Serial number for API 2
either r[0]['serialNumber'] or r['items'][0]['serialNumber'] depending on what you are showing
To modify the payload serial number, for example
payload['serialNumber'] = '123456abcdef'
{'meta': {'code': 200, 'requestId': '5e7c703bb9a389001b7d1e8c'},
'response': {'suggestedFilters': {'header': 'Tap to show:',
'filters': [{'name': 'Open now', 'key': 'openNow'}]},
'headerLocation': 'Lagos',
'headerFullLocation': 'Lagos',
'headerLocationGranularity': 'city',
'totalResults': 39,
'suggestedBounds': {'ne': {'lat': 6.655478745000045,
'lng': 3.355524537252914},
'sw': {'lat': 6.565478654999954, 'lng': 3.2650912627470863}},
'groups': [{'type': 'Recommended Places',
'name': 'recommended',
'items': [{'reasons': {'count': 0,
'items': [{'summary': 'This spot is popular',
'type': 'general',
'reasonName': 'globalInteractionReason'}]},
'venue': {'id': '502806dce4b0f23b021f3b77',
'name': 'KFC',
'location': {'lat': 6.604589745106469,
'lng': 3.3089358809010045,
'labeledLatLngs': [{'label': 'display',
'lat': 6.604589745106469,
'lng': 3.3089358809010045}],
'distance': 672,
'cc': 'NG',
'city': 'Egbeda',
'state': 'Lagos',
'country': 'Nigeria',
'formattedAddress': ['Egbeda', 'Lagos', 'Nigeria']},
'categories': [{'id': '4bf58dd8d48988d16e941735',
'name': 'Fast Food Restaurant',
'pluralName': 'Fast Food Restaurants',
'shortName': 'Fast Food',
'icon': {'prefix': 'https://ss3.4sqi.net/img/categories_v2/food/fastfood_',
'suffix': '.png'},
'primary': True}],
'photos': {'count': 0, 'groups': []}},
'referralId': 'e-0-502806dce4b0f23b021f3b77-0'},
That is a part of my file about called 'results'
I then
def getCAT(row):
try:
categories_list=row['categories']
except:
categories_list=row['venue.categories']
if len(categories_list)==0:
return None
else:
return categories_list[0]['name']
venues=results['response']['groups'][0]['items']
nearby_venues=pd.json_normalize(venues)
filtered_cols=['venue.name', 'venue.catergories', 'venue.location.lat', 'venue.location.lng']
nearby_venues= nearby_venues.loc[: , filtered_cols]
nearby_venues['venue.categories']=nearby_venues.apply(getCAT, axis=1)
nearby_venues.columns=[col.split(".")[-1] for col in nearby_venues.columns]
nearby_venues.head()
I get KeyError: 'Passing list-likes to .loc or [] with any missing labels is no longer supported on json file.
if I comment out that part, it runs well but with limited result. What am I doing wrong?
pandas.DataFrame.loc
property DataFrame.loc
Access a group of rows and columns by label(s) or a boolean array.
Try to remove the venue. from the line iltered_cols=['venue.name', 'venue.catergories', 'venue.location.lat', 'venue.location.lng']
I have a dataset (that pull its data from a dict) that I am attempting to clean and republish. Within this data set, there is a field with a sublist that I would like to extract specific data from.
Here's the data:
[{'id': 'oH58h122Jpv47pqXhL9p_Q', 'alias': 'original-pizza-brooklyn-4', 'name': 'Original Pizza', 'image_url': 'https://s3-media1.fl.yelpcdn.com/bphoto/HVT0Vr_Vh52R_niODyPzCQ/o.jpg', 'is_closed': False, 'url': 'https://www.yelp.com/biz/original-pizza-brooklyn-4?adjust_creative=IelPnWlrTpzPtN2YRie19A&utm_campaign=yelp_api_v3&utm_medium=api_v3_business_search&utm_source=IelPnWlrTpzPtN2YRie19A', 'review_count': 102, 'categories': [{'alias': 'pizza', 'title': 'Pizza'}], 'rating': 4.0, 'coordinates': {'latitude': 40.63781, 'longitude': -73.8963799}, 'transactions': [], 'price': '$', 'location': {'address1': '9514 Ave L', 'address2': '', 'address3': '', 'city': 'Brooklyn', 'zip_code': '11236', 'country': 'US', 'state': 'NY', 'display_address': ['9514 Ave L', 'Brooklyn, NY 11236']}, 'phone': '+17185313559', 'display_phone': '(718) 531-3559', 'distance': 319.98144420799355},
Here's how the data is presented within the csv/spreadsheet:
location
{'address1': '9514 Ave L', 'address2': '', 'address3': '', 'city': 'Brooklyn', 'zip_code': '11236', 'country': 'US', 'state': 'NY', 'display_address': ['9514 Ave L', 'Brooklyn, NY 11236']}
Is there a way to pull location.city for example?
The below code simply adds a few fields and exports it to a csv.
def data_set(data):
df = pd.DataFrame(data)
df['zip'] = get_zip()
df['region'] = get_region()
newdf = df.filter(['name', 'phone', 'location', 'zip', 'region', 'coordinates', 'rating', 'review_count',
'categories', 'url'], axis=1)
if not os.path.isfile('yelp_data.csv'):
newdf.to_csv('data.csv', header='column_names')
else: # else it exists so append without writing the header
newdf.to_csv('data.csv', mode='a', header=False)
If that doesn't make sense, please let me know. Thanks in advance!