I'm working with the songkick api found here and am very close to finishing a program I'm working on to pull info about a few specific artists upcoming shows. I have inputted an array of metro_areas that I want to track and output ONLY shows within THESE cities and their accompanying ID's. Other info I'm pulling is date of show, artist name, venue name. Basically right now my program is able to pull every show from the list of artist_ids I've inputted iterating through the id's in the request url for the range of dates in my parameters. My current output looks something like this:
['Date', 'Artist Name', 'Venue Name', 'City', 'metroArea ID']
['FEB - 7', 'Rosalia', 'NOTO', 'Philadelphia, PA, US', 5202]
['FEB - 8', 'Rosalia', 'Audio', 'San Francisco, CA, US', 26330]
['FEB - 8', 'Kid Cudi', 'Shady Park', 'Tempe, AZ, US', 23068]
['FEB - 8', 'Kid Cudi', 'Madison Square Garden', 'New York City, NY, US', 7644]
I want the output to be like this:
['FEB - 8', 'Rosalia', 'Audio', 'San Francisco, CA, US', 26330]
['FEB - 8', 'Kid Cudi', 'Madison Square Garden', 'New York City, NY, US', 7644]
Based on this array I've defined at the beginning of my program to match with the metro_areas in the songkick object array.
metro_areas = [
('Los Angeles', '17835'),
('San Francisco', '26330'),
('New York City', '7644'),
('Seattle', '2846'),
('Nashville', '11104')
]
Here is the json object array that I pull for each artist_id:
{
"resultsPage": {
"results": {
"event": [
{
"id":11129128,
"type":"Concert",
"uri":"http://www.songkick.com/concerts/11129128-wild-flag-at-fillmore?utm_source=PARTNER_ID&utm_medium=partner",
"displayName":"Wild Flag at The Fillmore (April 18, 2012)",
"start": {
"time":"20:00:00",
"date":"2012-04-18",
"datetime":"2012-04-18T20:00:00-0800"
},
"performance": [
{
"artist": {
"id":29835,
"uri":"http://www.songkick.com/artists/29835-wild-flag?utm_source=PARTNER_ID&utm_medium=partner",
"displayName":"Wild Flag",
"identifier": []
},
"id":21579303,
"displayName":"Wild Flag",
"billingIndex":1,
"billing":"headline"
}
],
"location": {
"city":"San Francisco, CA, US",
"lng":-122.4332937,
"lat":37.7842398
},
"venue": {
"id":6239,
"displayName":"The Fillmore",
"uri":"http://www.songkick.com/venues/6239-fillmore?utm_source=PARTNER_ID&utm_medium=partner",
"lng":-122.4332937,
"lat":37.7842398,
"metroArea": {
"id":26330,
"uri":"http://www.songkick.com/metro-areas/26330-us-sf-bay-area?utm_source=PARTNER_ID&utm_medium=partner",
"displayName":"SF Bay Area",
"country": { "displayName":"US" },
"state": { "displayName":"CA" }
}
},
"status":"ok",
"popularity":0.012763
}, ....
]
},
"totalEntries":24,
"perPage":50,
"page":1,
"status":"ok"
}
}
Some more code to see how I'm getting to my output from the JSON in the songkick requests.
metro_areas = [
('Los Angeles','17835'),
('San Francisco', '26330'),
('New York City','7644'),
('Seattle','2846'),
('Nashville','11104')
]
# artists we want to track
artist_ids = [
('Rosalia', '4610868'), ('EARTHGANG', '5720759'), ('Kid Cudi', '8630279'), ('Kanye West', '5566863'),
('Ludacris', '398291'), ('Hayley Williams', '10087966')
]
# Fetch existing events in each metro area
for artist_id in artist_ids:
params = {
'apikey': 'API_KEY',
'min_date': '2020-02-01',
'max_date': '2020-02-08',
# 'type': 'Concert'
}
r = requests.get('https://api.songkick.com/api/3.0/artists/' + artist_id[1] + '/calendar.json', params=params)
response = r.json()
shows = response['resultsPage']['results']
for show in shows:
try:
shows = shows['event']
formatted_shows = [{
'artistID': [perf['artist']['id'] for perf in s['performance']],
'date': s['start']['date'],
'name': [perf['artist']['displayName'] for perf in s['performance']],
'metroArea': s['venue']['metroArea']['id'],
'city': s['location']['city'],
'venue': s['venue']['displayName']
}
for s in shows if len(s['performance']) > 0
]
for sub in formatted_shows:
if sub['artistID'] == artist_id[1]:
sub['name'] = artist_id[0]
new_show = artist_id[1]
new_show_name = artist_id[0]
new_date = sub['date']
new_date_time = new_date = datetime.strptime(new_date, '%Y-%m-%d')
date_time_fin = new_date_time.strftime('%b - %-d').upper()
formatted_show_final = [date_time_fin, new_show_name, sub['venue'], sub['city'], sub['metroArea']
print(formatted_show_final)
Long story short, I need to find a way to iterate through each of my listed metro_areas id's (LA, SF, NYC, Seattle, Nashville) only and only output the shows that match with 'metroArea': s['venue']['metroArea']['id'] for each request iteration.
If I understood well the question, add inside the second for loop: if sub['metroArea'] in [area[1] for area in metro_areas]:
for show in shows:
try:
shows = shows['event']
formatted_shows = [{
'artistID': [perf['artist']['id'] for perf in s['performance']],
'date': s['start']['date'],
'name': [perf['artist']['displayName'] for perf in s['performance']],
'metroArea': s['venue']['metroArea']['id'],
'city': s['location']['city'],
'venue': s['venue']['displayName']
}
for s in shows if len(s['performance']) > 0
]
for sub in formatted_shows:
#Modified here to apply str() function to transform #sub['metroArea'] to string instead of int value
if str(sub['metroArea']) in [area[1] for area in metro_areas]:
if sub['artistID'] == artist_id[1]:
sub['name'] = artist_id[0]
new_show = artist_id[1]
new_show_name = artist_id[0]
new_date = sub['date']
new_date_time = new_date = datetime.strptime(new_date, '%Y-%m-%d')
date_time_fin = new_date_time.strftime('%b - %-d').upper()
formatted_show_final = [date_time_fin, new_show_name, sub['venue'], sub['city'], sub['metroArea']]
print(formatted_show_final)
Related
Conveting csv data to json and adding new filed names "parsed_address" and based on address type value is going to place in parsed_address field.
I am able to create parsed_address field and placing address column under it but address field should place based on address_type. I am getting all the address for particualr id in same address_type and repeating in next address_type as well.
Input Data
"source_id"|"first_name"|"last_name"|"address_type"|"address_line_1"|"city"
"41614335"|Reinaldo|Tonkoski Jr.|Primary|Deh 211 Box 2222|Brookings|
"41614335"|Reinaldo|Tonkoski Jr.|home|"2409 10th St Apt 123"|Brookings
"07605348"|E|Christodoulou|Primary|"4D Ag Lavras st"|Kifissia
"07605348"|E|Christodoulou|home|"131 N Hamilton Dr Apt 308"|Beverly Hills
Output Getting
[
{
"source_id":7605348,
"first_name":"E",
"last_name":"Christodoulou",
"parsed_address":{
"address_type":"Primary",
"address":[
{
"address_line_1":"Deh 211 Box 2222",
"city":"Brookings"
},
{
"address_line_1":"4D Ag Lavras st",
"city":"Kifissia"
}
]
}
},
{
"source_id":41614335,
"first_name":"Reinaldo",
"last_name":"Tonkoski Jr.",
"parsed_address":{
"address_type":"home",
"address":[
{
"address_line_1":"2409 10th St Apt 123",
"city":"Brookings"
},
{
"address_line_1":"131 N Hamilton Dr Apt 308",
"city":"Beverly Hills"
}
]
}
}
]
Expected Output
[
{
"source_id":7605348,
"first_name":"E",
"last_name":"Christodoulou",
"parsed_address":{
"address_type":"Primary",
"address":
{
"address_line_1":"Deh 211 Box 2222",
"city":"Brookings"
}
"address_type":"home",
"address" :
{
"address_line_1":"4D Ag Lavras st",
"city":"Kifissia"
}
TL;DR
from ast import literal_eval
from io import StringIO
from pprint import pprint
inputs = StringIO(""""source_id"|"first_name"|"last_name"|"address_type"|"address_line_1"|"city"
"41614335"|Reinaldo|Tonkoski Jr.|Primary|Deh 211 Box 2222|Brookings
"41614335"|Reinaldo|Tonkoski Jr.|home|"2409 10th St Apt 123"|Brookings
"07605348"|E|Christodoulou|Primary|"4D Ag Lavras st"|Kifissia
"07605348"|E|Christodoulou|home|"131 N Hamilton Dr Apt 308"|Beverly Hills""")
df = pd.read_csv(inputs, sep='|')
# Format the parsed_address to the dictionary format you want.
df['parsed_address'] = [{'address_type':row['address_type'], 'address':{'address_line_1':row['address_line_1'], 'city':row['city']}}
for _, row in df.iterrows()]
lol = []
# Group by the source_id
for _, group_rows in df.groupby('source_id'):
# Combine all parse_address for each group
parsed_address = list(group_rows['parsed_address'])
# Keeps the values of source_id, first and last name.
group_rows = group_rows.iloc[0][['source_id', 'first_name', 'last_name']]
# Set the parsed_address to the list you've combined.
group_rows['parsed_address'] = parsed_address
lol.append(group_rows)
js = literal_eval(pd.DataFrame(lol).to_json(orient='records'))
pprint(js)
[out]:
[{'first_name': 'E',
'last_name': 'Christodoulou',
'parsed_address': [{'address': {'address_line_1': '4D Ag Lavras st',
'city': 'Kifissia'},
'address_type': 'Primary'},
{'address': {'address_line_1': '131 N Hamilton Dr Apt 308',
'city': 'Beverly Hills'},
'address_type': 'home'}],
'source_id': 7605348},
{'first_name': 'Reinaldo',
'last_name': 'Tonkoski Jr.',
'parsed_address': [{'address': {'address_line_1': 'Deh 211 Box 2222',
'city': 'Brookings'},
'address_type': 'Primary'},
{'address': {'address_line_1': '2409 10th St Apt 123',
'city': 'Brookings'},
'address_type': 'home'}],
'source_id': 41614335}]
I have a code that should write information to excel using selenium. I have 1 list with some information. I need to write all this to excel, and i have solution. But, when i tried to use it i got 'DataFrame' object is not callable. How can i solve it?
All this code into iteration:
for schools in List: #in the List i have data from excel file with Name of schools
data = pd.DataFrame()
data({
"School Name":School_list_result[0::17],
"Principal":School_list_result[1::17],
"Principal's E-mail":School_list_result[2::17],
"Type":School_list_result[8::17],
"Grade Span": School_list_result[3::17],
"Address":School_list_result[4::17],
"Phone":School_list_result[14::17],
"Website":School_list_result[13::17],
"Associations/Communities":School_list_result[5::17],
"GreatSchools Summary Rating":School_list_result[6::17],
"U.S.News Rankings":School_list_result[12::17],
"Total # Students":School_list_result[15::17],
"Full-Time Teachers":School_list_result[16::17],
"Student/Teacher Ratio":School_list_result[17::17],
"Charter":School_list_result[9::17],
"Enrollment by Race/Ethnicity": School_list_result[7::17],
"Enrollment by Gender":School_list_result[10::17],
"Enrollment by Grade":School_list_result[11::17],
})
data.to_excel("D:\Schools.xlsx")
In School_list_result i have this data:
'Cape Elizabeth High School',
'Mr. Jeffrey Shedd',
'No data.',
'9-12',
'345 Ocean House Road, Cape Elizabeth, ME 04107',
'Cape Elizabeth Public Schools',
'8/10',
'White\n91%\nAsian\n3%\nTwo or more races\n3%\nHispanic\n3%\nBlack\n1%',
'Regular school',
'No',
' Male Female\n Students 281 252',
' 9 10 11 12\n Students 139 135 117 142',
'#5,667 in National Rankings',
'https://cehs.cape.k12.me.us/',
'Tel: (207)799-3309',
'516 students',
'47 teachers',
'11:1',
Please follow the syntax about how to create a dataframe
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html
So your code should be modified as:
for schools in List: #in the List i have data from excel file with Name of schools
data = pd.DataFrame(data={
"School Name": School_list_result[0::17],
"Principal": School_list_result[1::17],
"Principal's E-mail": School_list_result[2::17],
"Type": School_list_result[8::17],
"Grade Span": School_list_result[3::17],
"Address": School_list_result[4::17],
"Phone": School_list_result[14::17],
"Website": School_list_result[13::17],
"Associations/Communities": School_list_result[5::17],
"GreatSchools Summary Rating": School_list_result[6::17],
"U.S.News Rankings": School_list_result[12::17],
"Total # Students": School_list_result[15::17],
"Full-Time Teachers": School_list_result[16::17],
"Student/Teacher Ratio": School_list_result[17::17],
"Charter": School_list_result[9::17],
"Enrollment by Race/Ethnicity": School_list_result[7::17],
"Enrollment by Gender": School_list_result[10::17],
"Enrollment by Grade": School_list_result[11::17],
})
Do you want to add in an existing xlsx file?
First, create the dictionary and then call the DataFrame method, like this:
r = {"column1":["data"], "column2":["data"]}
data = pd.DataFrame(r)
I have the following code in index.html which grabs a dictionary list and prints out keys and values into a table
$(function() {
$('a#search').bind('click', function() {
$.getJSON('/_search', {
a: $('input[name="a"]').val()
}, function(data) {
var tableData = '<table>'
$.each(data.result, function(key, value){
tableData += '<tr><td>' + ' ' + key + ' ' + '</td>';
alert(key)
$.each(value, function(val){
alert(value[val])
tableData += '<td>' + value[val] + '</td>';
});
tableData += '</tr>';
});
tableData += '</table>';
$('#table').html(tableData);
});
What it is grabbing is a dictionary list from search.py
result = defaultdict(list)
return jsonify(result=result)
result contains the following
defaultdict(<class 'list'>, {'Developer': ['Office Koukan', 'Jorudan', 'Beam Software'], 'Publisher': ['Shouei', 'VAP', 'Hi Tech Expressions'], 'ReleaseDate': ['March 18, 1994', 'November 18, 1994', 'October 1, 1993'], 'Title': ['Idea no Hi', 'Pachinko Hi Hisshouhou', 'hunThe Hunt for Red October']})
However my output is as follows
Developer Publisher ReleaseDate Title
Office Koukan Jorudan Beam Software
Shouei VAP Hi Tech Expressions
March 18, 1994 November 18, 1994 October 1, 1993
Idea no Hi Pachinko Hi Hisshouhou hunThe Hunt for Red October
When the output should be
Developer Publisher ReleaseDate Title
Office Shouei ... ...
Koukan VAP ... ...
Jorudan ... ... ...
Beam Software ... ... ...
Any idea what I might be doing wrong?
This is a sample of what I was referencing (note I took out the part that gets data and hard coded that, pushed the result to a div.
Note that a simple array processes differently than the array of objects using $.each() regarding the index vs they key object name (like the header values you have)
// sample data to use (instead of json call)
var mything = {
'Developer': ['Office Koukan', 'Jorudan', 'Beam Software'],
'Publisher': ['Shouei', 'VAP', 'Hi Tech Expressions'],
'ReleaseDate': ['March 18, 1994', 'November 18, 1994', 'October 1, 1993'],
'Title': ['Idea no Hi', 'Pachinko Hi Hisshouhou', 'hunThe Hunt for Red October']
};
$(function() {
var data = mything;// sample data
// create a table to append rows to
var tableData = $('<table>');
// append a header row
tableData.append('<tr/>');
// reference that header row to append header td to
var header = tableData.find('tr').eq(0);
$.each(data, function(headthing, value) {
header.append('<td>' + headthing + '</td>');
$.each(value, function(idx, myval) {
if (!tableData.find('tr').eq(idx + 1).length) {
tableData.append('<tr/>'); // new row if not one
}
// put the columns in the row for each data
tableData.find('tr').eq(idx + 1).append('<td>' + myval + '</td>');
});
});
// put the table somewhere
$('#results').html(tableData);
});
Here is a fiddle showing it https://jsfiddle.net/MarkSchultheiss/ro5egfu3/1/
What is a good python API I can use to get the population of a city? I have tried using geocoder, but it is not working - not sure why.
geocoder.population('San Francisco, California')
returns
'module' object has no attribute 'population'
Why is this happening, and how can I fix it?
Alternatively, is there a different python api I can use for this?
Certainly you can get the population of a city using geocoder and Google,
but it requires an API key.
Here are two quite different alternative solutions:
OpenDataSoft
The first solution uses the OpenDataSoft API and basic Python 3.
The country needs to be specified via a two-letter country code, see examples below.
import requests
import json
def get_city_opendata(city, country):
tmp = 'https://public.opendatasoft.com/api/records/1.0/search/?dataset=worldcitiespop&q=%s&sort=population&facet=country&refine.country=%s'
cmd = tmp % (city, country)
res = requests.get(cmd)
dct = json.loads(res.content)
out = dct['records'][0]['fields']
return out
get_city_opendata('Berlin', 'de')
#{'city': 'berlin',
# 'country': 'de',
# 'region': '16',
# 'geopoint': [52.516667, 13.4],
# 'longitude': 13.4,
# 'latitude': 52.516667,
# 'accentcity': 'Berlin',
# 'population': 3398362}
get_city_opendata('San Francisco', 'us')
#{'city': 'san francisco',
# 'country': 'us',
# 'region': 'CA',
# 'geopoint': [37.775, -122.4183333],
# 'longitude': -122.4183333,
# 'latitude': 37.775,
# 'accentcity': 'San Francisco',
# 'population': 732072}
WikiData
The second solution uses the WikiData API and the qwikidata package.
Here, the country is given by its English name (or a part of it), see examples below.
I'm sure the SPARQL command can be written much more efficiently and elegantly (feel free to edit), but it does the job.
import qwikidata
import qwikidata.sparql
def get_city_wikidata(city, country):
query = """
SELECT ?city ?cityLabel ?country ?countryLabel ?population
WHERE
{
?city rdfs:label '%s'#en.
?city wdt:P1082 ?population.
?city wdt:P17 ?country.
?city rdfs:label ?cityLabel.
?country rdfs:label ?countryLabel.
FILTER(LANG(?cityLabel) = "en").
FILTER(LANG(?countryLabel) = "en").
FILTER(CONTAINS(?countryLabel, "%s")).
}
""" % (city, country)
res = qwikidata.sparql.return_sparql_query_results(query)
out = res['results']['bindings'][0]
return out
get_city_wikidata('Berlin', 'Germany')
#{'city': {'type': 'uri', 'value': 'http://www.wikidata.org/entity/Q64'},
# 'population': {'datatype': 'http://www.w3.org/2001/XMLSchema#decimal',
# 'type': 'literal',
# 'value': '3613495'},
# 'country': {'type': 'uri', 'value': 'http://www.wikidata.org/entity/Q183'},
# 'cityLabel': {'xml:lang': 'en', 'type': 'literal', 'value': 'Berlin'},
# 'countryLabel': {'xml:lang': 'en', 'type': 'literal', 'value': 'Germany'}}
get_city_wikidata('San Francisco', 'America')
#{'city': {'type': 'uri', 'value': 'http://www.wikidata.org/entity/Q62'},
# 'population': {'datatype': 'http://www.w3.org/2001/XMLSchema#decimal',
# 'type': 'literal',
# 'value': '805235'},
# 'country': {'type': 'uri', 'value': 'http://www.wikidata.org/entity/Q30'},
# 'cityLabel': {'xml:lang': 'en', 'type': 'literal', 'value': 'San Francisco'},
# 'countryLabel': {'xml:lang': 'en',
# 'type': 'literal',
# 'value': 'United States of America'}}
Both approaches return dictionaries from which you can extract the infos you need using basic Python.
Hope that helps!
from urllib.request import urlopen
import json
import pycountry
import requests
from geopy.geocoders import Nominatim
def get_city_opendata(city, country):
tmp = 'https://public.opendatasoft.com/api/records/1.0/search/?dataset=worldcitiespop&q=%s&sort=population&facet=country&refine.country=%s'
cmd = tmp % (city, country)
res = requests.get(cmd)
dct = json.loads(res.content)
out = dct['records'][0]['fields']
return out
def getcode(cc):
countries = {}
for country in pycountry.countries:
countries[country.name] = country.alpha_2
codes = countries.get(cc)
return codes
def getplace(lat, lon):
key = "PUT YOUR OWN GOOGLE API KEY HERE" #PUT YOUR OWN GOOGLE API KEY HERE
url = "https://maps.googleapis.com/maps/api/geocode/json?"
url += "latlng=%s,%s&sensor=false&key=%s" % (lat, lon, key)
v = urlopen(url).read()
j = json.loads(v)
components = j['results'][0]['address_components']
country = town = None
for c in components:
if "country" in c['types']:
country = c['long_name']
if "postal_town" in c['types']:
town = c['long_name']
return town, country
address= input('Input an address or town name\t')
geolocator = Nominatim(user_agent="Your_Name")
location = geolocator.geocode(address)
locationLat = location.latitude
locationLon = location.longitude
towncountry = getplace(location.latitude, location.longitude)
mycity = towncountry[0]
mycountry = towncountry[1]
print(towncountry)
print(mycountry)
print(mycity)
mycccode = getcode(mycountry)
mycccode = mycccode.lower()
print(mycccode)
populationdict = get_city_opendata(address, mycccode)
population = populationdict.get('population')
print('population',population)
print(location.address)
print((location.latitude, location.longitude))
I am very grateful for the previous answers. I had to solve this issue too. My code above follows on from David's answer above, where he recommends the OpenDataSoft API. Apparently the Google API at this time doesn't provide population results.
The code which I used below is able to get population of a city, OpenDataSoft doesn't always return town populations.
My code combines code from a few answers to different questions that I found on stackoverflow.
You will need to get a google maps developer api key, and do relevant pip installs.
Firstly this code gets the long,lat coordinates of any place name
based on user input
Then it uses those to get the country name off google maps
Then it uses the country name to get the abbreviated 2
letters for the country
Then it sends the place name and the abbreviated 2 letters to get the population from the OpenDataSoft
I am trying to get the country name from the latitude and longitude points from my pandas dataframe.
Currently I have used geolocator.reverse(latitude,longitude) to get the full address of the geographic location. But there is no option to retrieve the country name from the full address as it returns a list.
Method used:
def get_country(row):
pos = str(row['StartLat']) + ', ' + str(row['StartLong'])
locations = geolocator.reverse(pos)
return locations
Call to get_country by passing the dataframe:
df4['country'] = df4.apply(lambda row: get_country(row), axis = 1)
Current output:
StartLat StartLong Address
52.509669 13.376294 Potsdamer Platz, Mitte, Berlin, Deutschland, Europe
Just wondering whether there is some Python library to retrieve the country when we pass the geographic points.
Any help would be appreciated.
In your get_country function, your return value location will have an attribute raw, which is a dict that looks like this:
{
'address': {
'attraction': 'Potsdamer Platz',
'city': 'Berlin',
'city_district': 'Mitte',
'country': 'Deutschland',
'country_code': 'de',
'postcode': '10117',
'road': 'Potsdamer Platz',
'state': 'Berlin'
},
'boundingbox': ['52.5093982', '52.5095982', '13.3764983', '13.3766983'],
'display_name': 'Potsdamer Platz, Mitte, Berlin, 10117, Deutschland',
... and so one ...
}
so location.raw['address']['country'] gives 'Deutschland'
If I read your question correctly, a possible solution could be:
def get_country(row):
pos = str(row['StartLat']) + ', ' + str(row['StartLong'])
locations = geolocator.reverse(pos)
return location.raw['address']['country']
EDIT: The format of the location.raw object will differ depending on which geolocator service you are using. My example uses geopy.geocoders.Nominatim, from the example on geopy's documentation site, so your results might differ.
My code,hopefully that helps:
from geopy.geocoders import Nominatim
nm = Nominatim()
place, (lat, lng) = nm.geocode("3995 23rd st, San Francisco,CA 94114")
print('Country' + ": " + place.split()[-1])
I'm not sure what service you're using with geopy, but as a small plug which I'm probably biased towards, this I think could be a simpler solution for you.
https://github.com/Ziptastic/ziptastic-python
from ziptastic import Ziptastic
# Set API key.
api = Ziptastic('<your api key>')
result = api.get_from_coordinates('42.9934', '-84.1595')
Which will return a list of dictionaries like so:
[
{
"city": "Owosso",
"geohash": "dpshsfsytw8k",
"country": "US",
"county": "Shiawassee",
"state": "Michigan",
"state_short": "MI",
"postal_code": "48867",
"latitude": 42.9934,
"longitude": -84.1595,
"timezone": "America/Detroit"
}
]