Related
Looking into translating the following nested dictionary which is an API pull from Yelp into a pandas dataframe to run visualization on:
Top 50 Pizzerias in Chicago
{'businesses': [{'alias': 'pequods-pizzeria-chicago',
'categories': [{'alias': 'pizza', 'title': 'Pizza'}],
'coordinates': {'latitude': 41.92187, 'longitude': -87.664486},
'display_phone': '(773) 327-1512',
'distance': 2158.7084581522413,
'id': 'DXwSYgiXqIVNdO9dazel6w',
'image_url': 'https://s3-media1.fl.yelpcdn.com/bphoto/8QJUNblfCI0EDhOjuIWJ4A/o.jpg',
'is_closed': False,
'location': {'address1': '2207 N Clybourn Ave',
'address2': '',
'address3': '',
'city': 'Chicago',
'country': 'US',
'display_address': ['2207 N Clybourn Ave',
'Chicago, IL 60614'],
'state': 'IL',
'zip_code': '60614'},
'name': "Pequod's Pizzeria",
'phone': '+17733271512',
'price': '$$',
'rating': 4.0,
'review_count': 6586,
'transactions': ['restaurant_reservation', 'delivery'],
'url': 'https://www.yelp.com/biz/pequods-pizzeria-chicago?adjust_creative=wt2WY5Ii_urZB8YeHggW2g&utm_campaign=yelp_api_v3&utm_medium=api_v3_business_search&utm_source=wt2WY5Ii_urZB8YeHggW2g'},
{'alias': 'lou-malnatis-pizzeria-chicago',
'categories': [{'alias': 'pizza', 'title': 'Pizza'},
{'alias': 'italian', 'title': 'Italian'},
{'alias': 'sandwiches', 'title': 'Sandwiches'}],
'coordinates': {'latitude': 41.890357,
'longitude': -87.633704},
'display_phone': '(312) 828-9800',
'distance': 4000.9990531720227,
'id': '8vFJH_paXsMocmEO_KAa3w',
'image_url': 'https://s3-media3.fl.yelpcdn.com/bphoto/9FiL-9Pbytyg6usOE02lYg/o.jpg',
'is_closed': False,
'location': {'address1': '439 N Wells St',
'address2': '',
'address3': '',
'city': 'Chicago',
'country': 'US',
'display_address': ['439 N Wells St',
'Chicago, IL 60654'],
'state': 'IL',
'zip_code': '60654'},
'name': "Lou Malnati's Pizzeria",
'phone': '+13128289800',
'price': '$$',
'rating': 4.0,
'review_count': 6368,
'transactions': ['pickup', 'delivery'],
'url': 'https://www.yelp.com/biz/lou-malnatis-pizzeria-chicago?adjust_creative=wt2WY5Ii_urZB8YeHggW2g&utm_campaign=yelp_api_v3&utm_medium=api_v3_business_search&utm_source=wt2WY5Ii_urZB8YeHggW2g'},
....]
I've tried the below and iterations of it but haven't had any luck.
df = pd.DataFrame.from_dict(topresponse)
Im really new to coding so any advice would be helpful
response["businesses"] is a list of records, so:
df = pd.DataFrame.from_records(response["businesses"])
With a list of 150+ Neighborhoods , I am using Foursquare API to retrieve nearby venues at 500m radius of a given Neighbourhood. Each Neighbourhood is expected to return 10-20 nearby venues.
Refer to snippet of json result as returned by Foursquare.
With results['response']['groups'][0]['items'], I able to retrieve the nearby venues information and make it a Table as below. However results['response']['groups'][0]['items'] does not have the Neighbourhood ( under headerFullLocation in json) of associated venues.
Q: How can I link the Neighbourhood(headerFullLocation) to its associated nearby venue and add it as a column to table below? Thanks for the advice.
{'suggestedFilters': {'header': 'Tap to show:',
'filters': [{'name': 'Open now', 'key': 'openNow'}]},
'headerLocation': 'Alexandra Park',
'headerFullLocation': 'Alexandra Park, Toronto',**
'headerLocationGranularity': 'neighborhood',
'totalResults': 138,
'suggestedBounds': {'ne': {'lat': 43.6545000045, 'lng': -79.39379244047241},
'sw': {'lat': 43.645499995499996, 'lng': -79.4062075595276}},
'groups': [{'type': 'Recommended Places',
'name': 'recommended',
'items': [{'reasons': {'count': 0,
'items': [{'summary': 'This spot is popular',
'type': 'general',
'reasonName': 'globalInteractionReason'}]},
'venue': {'id': '5644dbaa498e7f7534154326',
'**name': 'Maker Pizza',**
'contact': {},
'location': {'address': '59 Cameron St',
'lat': 43.6504011331197,
'lng': -79.39804047841302,
'labeledLatLngs': [{'label': 'display',
'lat': 43.6504011331197,
'lng': -79.39804047841302}],
'distance': 164,
'postalCode': 'M5T 2H1',
'cc': 'CA',
'city': 'Toronto',
'state': 'ON',
'country': 'Canada',
'formattedAddress': ['59 Cameron St', 'Toronto ON M5T 2H1', 'Canada']},
'categories': [{'id': '4bf58dd8d48988d1ca941735',
'name': 'Pizza Place',
'pluralName': 'Pizza Places',
'shortName': 'Pizza',
'icon': {'prefix': 'https://ss3.4sqi.net/img/categories_v2/food/pizza_',
'suffix': '.png'},
'primary': True}],
'verified': False,
'stats': {'tipCount': 0,
'usersCount': 0,
'checkinsCount': 0,
'visitsCount': 0},
'beenHere': {'count': 0,
'lastCheckinExpiredAt': 0,
'marked': False,
'unconfirmedCount': 0},
'photos': {'count': 0, 'groups': []},
'hereNow': {'count': 0, 'summary': 'Nobody here', 'groups': []}},
Why don't you just do venues['Neighbourhood'] = response['headerFullLocation']. I am assuming, you send a separate request for each neigbhourhood and plan to concatenate multiple venue dataframes in the end.
So I am trying to do a project for work where I create a matrix where the x-axis is a list of stores and the y axis is another list of stores and the values are the distances returned see the link below for an example of what im looking for
Example Output
This is the code I am running with dummy addresses for lst_stores1 and lst_stores2
INPUT:
import googlemaps
lst_store1 = ['777 Brockton Avenue, Abington MA 2351',
'30 Memorial Drive, Avon MA',
'250 Hartford Avenue, Bellingham MA',
'700 Oak Street, Brockton MA',
'591 Memorial Dr, Chicopee MA']
lst_store2 = ['55 Brooksby Village Way, Danvers MA',
'137 Teaticket Hwy, East Falmouth MA',
'42 Fairhaven Commons Way, Fairhaven MA',
'374 William S Canning Blvd, Fall River MA',
'121 Worcester Rd, Framingham MA']
my_dist = gmaps.distance_matrix(lst_store1,lst_store2)
print(my_dist)
OUTPUT:
'destination_addresses': ['55 Brooksby Village Dr, Danvers, MA 01923, USA', '137 Teaticket Hwy, Teaticket, MA 02536, USA', '42 Fairhaven Commons Way, Fairhaven, MA 02719, USA', '374 William S Canning Blvd, Fall River, MA 02721, USA', '121 Worcester Rd, Framingham, MA 01701, USA'], 'origin_addresses': ['777 Brockton Ave, Abington, MA 02351, USA', '30 Memorial Dr, Avon, MA 02322, USA', '250 Hartford Ave, Bellingham, MA 02019, USA', '700 Oak St, Brockton, MA 02301, USA', '591 Memorial Dr, Chicopee, MA 01020, USA'], 'rows': [{'elements': [{'distance': {'text': '65.0 km', 'value': 65015}, 'duration': {'text': '1 hour 4 mins', 'value': 3860}, 'status': 'OK'}, {'distance': {'text': '89.0 km', 'value': 89014}, 'duration': {'text': '1 hour 14 mins', 'value': 4437}, 'status': 'OK'}, {'distance': {'text': '72.4 km', 'value': 72367}, 'duration': {'text': '56 mins', 'value': 3339}, 'status': 'OK'}, {'distance': {'text': '63.4 km', 'value': 63418}, 'duration': {'text': '51 mins', 'value': 3034}, 'status': 'OK'}, {'distance': {'text': '58.7 km', 'value': 58690}, 'duration': {'text': '50 mins', 'value': 2998}, 'status': 'OK'}]}, {'elements': [{'distance': {'text': '62.6 km', 'value': 62649}, 'duration': {'text': '53 mins', 'value': 3189}, 'status': 'OK'}, {'distance': {'text': '96.8 km', 'value': 96832}, 'duration': {'text': '1 hour 5 mins', 'value': 3889}, 'status': 'OK'}, {'distance': {'text': '70.4 km', 'value': 70413}, 'duration': {'text': '46 mins', 'value': 2788}, 'status': 'OK'}, {'distance': {'text': '61.5 km', 'value': 61463}, 'duration': {'text': '41 mins', 'value': 2483}, 'status': 'OK'}, {'distance': {'text': '50.5 km', 'value': 50512}, 'duration': {'text': '38 mins', 'value': 2273}, 'status': 'OK'}]}, {'elements': [{'distance': {'text': '95.3 km', 'value': 95321}, 'duration': {'text': '1 hour 2 mins', 'value': 3702}, 'status': 'OK'}, {'distance': {'text': '115 km', 'value': 115239}, 'duration': {'text': '1 hour 14 mins', 'value': 4436}, 'status': 'OK'}, {'distance': {'text': '90.2 km', 'value': 90161}, 'duration': {'text': '57 mins', 'value': 3427}, 'status': 'OK'}, {'distance': {'text': '81.2 km', 'value': 81211}, 'duration': {'text': '52 mins', 'value': 3122}, 'status': 'OK'}, {'distance': {'text': '39.8 km', 'value': 39785}, 'duration': {'text': '29 mins', 'value': 1710}, 'status': 'OK'}]}, {'elements': [{'distance': {'text': '65.5 km', 'value': 65521}, 'duration': {'text': '55 mins', 'value': 3323}, 'status': 'OK'}, {'distance': {'text': '91.4 km', 'value': 91450}, 'duration': {'text': '1 hour 2 mins', 'value': 3726}, 'status': 'OK'}, {'distance': {'text': '65.0 km', 'value': 65031}, 'duration': {'text': '44 mins', 'value': 2625}, 'status': 'OK'}, {'distance': {'text': '56.1 km', 'value': 56081}, 'duration': {'text': '39 mins', 'value': 2320}, 'status': 'OK'}, {'distance': {'text': '53.4 km', 'value': 53385}, 'duration': {'text': '40 mins', 'value': 2407}, 'status': 'OK'}]}, {'elements': [{'distance': {'text': '168 km', 'value': 167923}, 'duration': {'text': '1 hour 43 mins', 'value': 6187}, 'status': 'OK'}, {'distance': {'text': '227 km', 'value': 227118}, 'duration': {'text': '2 hours 17 mins', 'value': 8217}, 'status': 'OK'}, {'distance': {'text': '183 km', 'value': 183401}, 'duration': {'text': '1 hour 54 mins', 'value': 6818}, 'status': 'OK'}, {'distance': {'text': '163 km', 'value': 163452}, 'duration': {'text': '1 hour 43 mins', 'value': 6170}, 'status': 'OK'}, {'distance': {'text': '112 km', 'value': 112386}, 'duration': {'text': '1 hour 10 mins', 'value': 4196}, 'status': 'OK'}]}], 'status': 'OK'}
I'm having trouble deciphering the output it's returning and figuring out how to convert it into the format I described above.
P.S. I have imported the appropriate libraries and set up my API key etc it is being referenced in a different cell in jupyter
For what I understand, you want to get the corresponding distance of a store from x to each store in y. What I done to achieve this is to put a for loop to each store in both list and sending the distance matrix request for each combination. This is so,I can put the result of each combination to the array. I then use a for loop to with the range of length to lookup the index on my result array so that I can get each result from the array. Here is a sample code. Here is the breakdown of my code:
First, I used these lines from your code:
import googlemaps
lst_store1 = ['777 Brockton Avenue, Abington MA 2351',
'30 Memorial Drive, Avon MA',
'250 Hartford Avenue, Bellingham MA',
'700 Oak Street, Brockton MA',
'591 Memorial Dr, Chicopee MA']
lst_store2 = ['55 Brooksby Village Way, Danvers MA',
'137 Teaticket Hwy, East Falmouth MA',
'42 Fairhaven Commons Way, Fairhaven MA',
'374 William S Canning Blvd, Fall River MA',
'121 Worcester Rd, Framingham MA']
Then I added the line where I put the API Key. Note that To properly use the Distance Matrix API, you need to have an API Key.
gmaps = googlemaps.Client(key='YOUR_API_KEY_HERE')
Then declare an empty array where I will put my results:
my_result= []
Then use a for loop for lst_store1 array and put a for loop inside it for lst_store2 array. Then append the result of the distance matrix between each combination of your array. Here are the code:
for x in lst_store1:
for y in lst_store2:
my_result.append(gmaps.distance_matrix(x,y))
You will get something like these combination:
orig: 777 Brockton Avenue, Abington MA 2351 dest: 55 Brooksby Village Way, Danvers MA
orig: 777 Brockton Avenue, Abington MA 2351 dest: 137 Teaticket Hwy, East Falmouth MA
orig: 777 Brockton Avenue, Abington MA 2351 dest:42 Fairhaven Commons Way, Fairhaven MA
orig: 777 Brockton Avenue, Abington MA 2351 dest:374 William S Canning Blvd, Fall River MA
orig: 777 Brockton Avenue, Abington MA 2351 dest: 121 Worcester Rd, Framingham MA
orig: 30 Memorial Drive, Avon MA dest: 55 Brooksby Village Way, Danvers MA
orig: 30 Memorial Drive, Avon MA 2351 dest: 137 Teaticket Hwy, East Falmouth MA
... and so on. You'll have 25 combinations from your list.
Then I put a for loop using range of length so that we can get the index. You can also edit the format on how you print the output of the array.
for z in range(len(my_result)):
print("{}- from:{} to:{} distance:{}".format(z+1, my_result[z]['origin_addresses'],my_result[z]['destination_addresses'], my_result[z]['rows'][0]['elements'][0]['distance']['value'] ))
This will be the sample output of the print from the for loop which will be easier for you to sort:
1- from:['777 Brockton Ave, Abington, MA 02351, USA'] to:['55 Brooksby Village Dr, Danvers, MA 01923, USA'] distance:65015
2- from:['777 Brockton Ave, Abington, MA 02351, USA'] to:['137 Teaticket Hwy, Teaticket, MA 02536, USA'] distance:88198
3- from:['777 Brockton Ave, Abington, MA 02351, USA'] to:['42 Fairhaven Commons Way, Fairhaven, MA 02719, USA'] distance:72367
4- from:['777 Brockton Ave, Abington, MA 02351, USA'] to:['374 William S Canning Blvd, Fall River, MA 02721, USA'] distance:63418
5- from:['777 Brockton Ave, Abington, MA 02351, USA'] to:['121 Worcester Rd, Framingham, MA 01701, USA'] distance:58690
6- from:['30 Memorial Dr, Avon, MA 02322, USA'] to:['55 Brooksby Village Dr, Danvers, MA 01923, USA'] distance:62649
7- from:['30 Memorial Dr, Avon, MA 02322, USA'] to:['137 Teaticket Hwy, Teaticket, MA 02536, USA'] distance:96832
Hope this helps!
I have a a list of dicts that looks like this:
source_dict = [{'ppl': 10, 'items': 15, 'airport': 'lax', 'city': 'Los Angeles', 'timestamp': 1, 'region': 'North America', 'country': 'United States'},
{'ppl': 20, 'items': 32, 'airport': 'JFK', 'city': 'New York', 'timestamp': 2, 'region': 'North America', 'country': 'United States'},
{'ppl': 50, 'items': 20, 'airport': 'ABC', 'city': 'London', 'timestamp': 1, 'region': 'Europe', 'country': 'United Kingdom'}... ]
#Gets the list of countries in the dict
countries = list(set(stats['country'] for stats in source_dict))
I know I can use an a collection:
counter = collections.Counter()
for d in source_dict:
counter.update(d)
But, want to group by country and get totals for only certain keys not all of them.
So the result should be
{'Country': 'United States', 'p95': 30, 'items':37},
{'Country': 'England', 'ppl': 50, 'items':20},...
Im not sure how to incorporate multiple keys into a counter. To produce that result
This is one approach using collections.defaultdict & collections.Counter.
Ex:
from collections import defaultdict, Counter
source_dict = [{'ppl': 10, 'items': 15, 'airport': 'lax', 'city': 'Los Angeles', 'timestamp': 1, 'region': 'North America', 'country': 'United States'},
{'ppl': 20, 'items': 32, 'airport': 'JFK', 'city': 'New York', 'timestamp': 2, 'region': 'North America', 'country': 'United States'},
{'ppl': 50, 'items': 20, 'airport': 'ABC', 'city': 'London', 'timestamp': 1, 'region': 'Europe', 'country': 'United Kingdom'} ]
result = defaultdict(Counter)
for stats in source_dict:
result[stats['country']].update(Counter({'ppl': stats['ppl'], "items": stats['items']}))
#result = [{'Country': k, **v} for k, v in result.items()] #Required output
print(result)
Output:
defaultdict(<class 'collections.Counter'>,
{'United Kingdom': Counter({'ppl': 50, 'items': 20}),
'United States': Counter({'items': 47, 'ppl': 30})})
In pandas you can do this:
import io
import pandas as pd
dff=io.StringIO("""ppl,items,airport,city,timestamp,region,country
10,15,lax,Los Angeles,1,North America,United States
20,32,JFK,New York,2,North America,United States
50,20,ABC,London,1,Europe,United Kingdom""")
df3=pd.read_csv(dff)
df3
ppl items airport city timestamp region country
0 10 15 lax Los Angeles 1 North America United States
1 20 32 JFK New York 2 North America United States
2 50 20 ABC London 1 Europe United Kingdom
df3.groupby('region').agg({'ppl':'sum', 'items':'sum'})
# ppl items
#region
#Europe 50 20
#North America 30 47
I need assistance handling the dict file type that is returned from the google maps api.
Currently, the results are handing me a dict of the resulting data (starting addresses, ending addresses, travel times, distances etc) which I cannot process. I can extract the start and end addresses simply, but the bulk data is proving difficult to extract, and I think it is because of its structure.
The sample of the code I have is as follows;
import googlemaps
import csv
import pandas as pd
postcodes = pd.read_csv("SW.csv", sep=',', usecols=['postcode'], squeeze=True)
infile1 = open('SW.csv', 'r')
reader1 = csv.reader(infile1)
Location1 = postcodes[0:10]
Location2 = 'SW1A 2HQ'
my_distance = gmaps.distance_matrix(Location1, Location2, mode='bicycling', language=None, avoid=None, units='metric',
departure_time='2475925955', arrival_time=None,
transit_routing_preference=None)
print(my_distance)
Which generates the following output;
{'origin_addresses': ['Cossar Mews, Brixton, London SW2 2TR, UK',
'Bushnell Rd, London SW17 8QP, UK', 'Maltings Pl, Fulham, London SW6
2BX, UK', 'Knightsbridge, London SW7 1BJ, UK', 'Chelsea, London SW3
3EE, UK', 'Hester Rd, London SW11 4AJ, UK', 'Brixton, London SW2 1HZ,
UK', 'Randall Cl, London SW11 3TG, UK', 'Sloane St, London SW1X 9SF,
UK', 'Binfield Rd, London SW4 6TA, UK'], 'rows': [{'elements':
[{'duration': {'text': '28 mins', 'value': 1657}, 'status': 'OK',
'distance': {'text': '7.5 km', 'value': 7507}}]}, {'elements':
[{'duration': {'text': '31 mins', 'value': 1850}, 'status': 'OK',
'distance': {'text': '9.2 km', 'value': 9176}}]}, {'elements':
[{'duration': {'text': '27 mins', 'value': 1620}, 'status': 'OK',
'distance': {'text': '7.0 km', 'value': 7038}}]}, {'elements':
[{'duration': {'text': '16 mins', 'value': 953}, 'status': 'OK',
'distance': {'text': '4.0 km', 'value': 4038}}]}, {'elements':
[{'duration': {'text': '15 mins', 'value': 899}, 'status': 'OK',
'distance': {'text': '3.4 km', 'value': 3366}}]}, {'elements':
[{'duration': {'text': '21 mins', 'value': 1260}, 'status': 'OK',
'distance': {'text': '5.3 km', 'value': 5265}}]}, {'elements':
[{'duration': {'text': '28 mins', 'value': 1682}, 'status': 'OK',
'distance': {'text': '7.5 km', 'value': 7502}}]}, {'elements':
[{'duration': {'text': '23 mins', 'value': 1368}, 'status': 'OK',
'distance': {'text': '5.9 km', 'value': 5876}}]}, {'elements':
[{'duration': {'text': '14 mins', 'value': 839}, 'status': 'OK',
'distance': {'text': '3.3 km', 'value': 3341}}]}, {'elements':
[{'duration': {'text': '16 mins', 'value': 982}, 'status': 'OK',
'distance': {'text': '4.3 km', 'value': 4294}}]}],
'destination_addresses': ['Horse Guards Rd, London SW1A 2HQ, UK'],
'status': 'OK'}
I am then using the following code to extract it;
origin = my_distance['origin_addresses']
dest = my_distance['destination_addresses']
dist = my_distance['rows']
I have tried the df_from_list and many others to try and process the dist data. The end goal is to have a matrix with the origin addresses on each row, the destination addresses forming columns, with distance and time as data variables within these columns.
Something similar to this
| DEST 1 | DEST 2 |
| TIME | DIST | TIME | DIST |
START 1 | X | Y | Z | T |
START 2 | A | B | C | T |
Please can someone help me process the my_distance output (shown above) into an architecture similar to that shown above.
Thanks!
This basicly creates a dictionary with the starts and the destination adresses.
The destination adresses have a list of tupels as values. The first element in the tuple is the duration and the second the distance
e.g. (45, 7.0)#45=45min and 7.0 = 7km. Then I create the dataframe with pandas.DataFrame.from_dict()
import pandas as pd
dct = {d_adresses:[] for d_adresses in data['destination_addresses']}
dct['starts'] = []
for i in range(len(data['origin_addresses'])):
duration=int(data['rows'][i]['elements'][0]['duration']['text'].split(' ')[0])
distance=float(data['rows'][i]['elements'][0]['distance']['text'].split(' ')[0])
for key in dct:
if key != 'starts':
dct[key].append((duration, distance))
dct['starts'].append(data['origin_addresses'][i])
df = pd.DataFrame.from_dict(dct)
df.set_index('starts', inplace=True)
I create an empty dataframe before running gmaps.distance_matrix, and place the dictionary keys into the dataframe. Similar to the above solution:
traffic = pd.DataFrame({'time': [], 'origins': [], 'destinations': [], 'destination_addresses': [], 'origin_addresses': [], 'rows': [], 'status': []})
for origin in origins:
for destination in destinations:
traffic = traffic.append({'time': [00:00], 'origins': [origin], 'destinations': [destination]}, ignore_index=True)
if origin != destination:
if cityname == cityname:
# Get travel distance and time for a matrix of origins and destinations
traffic_result = gmaps.distance_matrix((origin), (destination),
mode="driving", language=None, avoid=None, units="metric",
departure_time=00:00, arrival_time=None, transit_mode=None,
transit_routing_preference=None, traffic_model=None, region=None)
for key in traffic_result.keys():
for value in traffic_result[key]:
print(key, value)
traffic = traffic.append({key: [value]}, ignore_index=True)