I am trying to gather weather data from the national weather service and read it into a python script. They offer a JSON return, but they also offer another return which isn't formatted JSON but has more variables (which I want). This set of data looks like it is formatted as a python dictionary. It looks like this:
stations={
KAPC:
{
'id':'KAPC',
'stnid':'92',
'name':'Napa, Napa County Airport',
'elev':'33',
'latitude':'38.20750',
'longitude':'-122.27944',
'distance':'',
'provider':'NWS/FAA',
'link':'http://www.wrh.noaa.gov/mesowest/getobext.php?sid=KAPC',
'Date':'24 Feb 8:54 am',
'Temp':'39',
'TempC':'4',
'Dewp':'29',
'Relh':'67',
'Wind':'NE#6',
'Direction':'50°',
'Winds':'6',
'WindChill':'35',
'Windd':'50',
'SLP':'1027.1',
'Altimeter':'30.36',
'Weather':'',
'Visibility':'10.00',
'Wx':'',
'Clouds':'CLR',
[...]
So, to me, it looks like its got a defined variable stations equal to a dictionary of dictionaries containing the stations and their variables. My question is how do I access this data. Right now I am trying:
import urllib
response = urrllib.urlopen(url)
r = response.read()
If I try to use the JSON module, it clearly fails because this isn't json. And if I just try to read the file, it comes back with a long string of characters. Any suggestions on how to extract this data? If possible, I would just like to get the dictionary as it exists in the url return, ie stations={...} Thanks!
See, As far I infer from the question, I assume that you have data in the form of text which in not a valid JSON data, So given we have a text like: line = "stations={'KAPC':{'id':'KAPC', 'stnid':'92', 'name':'Napa, Napa County Airport'}}" (say), then we can extract the dictionary by splitting it at the = symbol and then use the eval() method which initializes the dictionary variable with the required data.
dictionary_text = line.split("=")[1]
python_dictionary = eval(dictionary_text)
print python_dictionary
>>> {'KAPC': {'id': 'KAPC', 'name': 'Napa, Napa County Airport', 'stnid': '92'}}
The python_dictionary now behaves like a Python Dictionary with key, value pairs , and you can access any attribute using python_dictionary["KAPC"]["id"]
Related
I am making a program that consists of scraping data from a job page, and I get to this data
{"job":{"ciphertext":"~01142b81f148312a7c","rid":225177647,"uid":"1416152499115024384","type":2,"access":4,"title":"Need app developers to handle our app upgrades","status":1,"category":{"name":"Mobile Development","urlSlug":"mobile-development"
,"contractorTier":2,"description":"We have an app currently built, we are looking for someone to \n\n1) Manage the app for bugs etc \n2) Provide feature upgrades \n3) Overall Management and optimization \n\nPlease get in touch and i will share more details. ","questions":null,"qualifications":{"type":0,"location":null,"minOdeskHours":0,"groupRecno":0,"shouldHavePortfolio":false,"tests":null,"minHoursWeek":40,"group":null,"prefEnglishSkill":0,"minJobSuccessScore":0,"risingTalent":true,"locationCheckRequired":false,"countries":null,"regions":null,"states":null,"timezones":null,"localMarket":false,"onSiteType":null,"locations":null,"localDescription":null,"localFlexibilityDescription":null,"earnings":null,"languages":null
],"clientActivity":{"lastBuyerActivity":null,"totalApplicants":0,"totalHired":0,"totalInvitedToInterview":0,"unansweredInvites":0,"invitationsSent":0
,"buyer":{"isPaymentMethodVerified":false,"location":{"offsetFromUtcMillis":14400000,"countryTimezone":"United Arab Emirates (UTC+04:00)","city":"Dubai","country":"United Arab Emirates"
,"stats":{"totalAssignments":31,"activeAssignmentsCount":3,"feedbackCount":27,"score":4.9258937139,"totalJobsWithHires":30,"hoursCount":7.16666667,"totalCharges":{"currencyCode":"USD","amount":19695.83
,"jobs":{"postedCount":59,"openCount":2
,"avgHourlyJobsRate":{"amount":19.999534874418824
But the problem is that the only data I need is:
-Title
-Description
-Customer activity (lastBuyerActivity, totalApplicants, totalHired, totalInvitedToInterview, unansweredInvites, invitationsSent)
-Buyer (isPaymentMethodVerified, location (Country))
-stats (All items)
-jobs (all items)
-avgHourlyJobsRate
These sort of data are JSON type data. Python understands these sort of data through dictionary data type.
Suppose you have your data stored in a string. You can use di = exec(myData) to convert the string to dictionary. Then you can access the structured data like: di["job"] which return's the job section of the data.
di = exec(myData)
print(`di["job"]`)
However this is just a hack and it is not recommended because it's a
bit messy and unpythonic.
The appropriate way is to use JSON library to convert the data to dictionary. Take a look at the code snippet below to get an idea of what is the appropriate way:
import json
myData = "Put your data Here"
res = json.loads(myData)
print(res["jobs"])
convert the data to dictionary using json.loads
then you can easily use the dictionary keys that your want to lookup or filter the data.
This seems to be a dictionary so you can extract something from it by doing: dictionary["job"]["uid"] for example. If it is a Json file convert the data to a Python dictionary
I'm a beginner programmer trying to work with the Yelp API and I've been able to pull the information I need but I can't figure out how to only export a single part of the address into my csv file. This is the code I am working with:
**Convert the JSON string to a dictionary*
business_data = response.json()
c = csv.writer(open('testing.csv', 'a'), lineterminator ='\n')
for biz in business_data['businesses']:
c.writerow([biz['name'], biz['location'], biz['phone'], biz['url']])
In the last line of code, in the for loop, I want to be able to target a specific element of the 'location' like this:
#c.writerow([biz['name'], biz['address1'], biz['city'], biz['state'], biz['zip_code'], biz['phone'], biz['url']])
on the Yelp website it shows that I can target these specifics but I just can't seem to figure out how with c.writerow()
Yelp shows that I can target these like this:
businesses[x].location.address1
businesses[x].location.address2
businesses[x].location.city
From the response, biz['location'] is a python dictionary meaning it consists of key, value pairs.
You can validate this by printing type(biz['location']). To answer your question, all you need is to call the dict key & write the value into the file.
c.writerow([biz['name'], biz['location']['address1'], biz['location']['city'], biz['location']['state'], biz['location']['zip_code'] biz['phone'], biz['url']])
I want to access the place value in the dictionary resent in the URL
https://earthquake.usgs.gov/fdsnws/event/1/query?format=geojson&starttime=2016-10-01&endtime=2016-10-02
I wrote the below code but it is throwing an error key 0. I want to access some of the values like place, title, geometry from the URL and write it to CSV file.
import urllib2
import json
url = 'https://earthquake.usgs.gov/fdsnws/event/1/query?format=geojson&starttime=2016-10-01&endtime=2016-10-02'
# download the json string
json_string = urllib2.urlopen(url).read()
# de-serialize the string so that we can work with it
j = json.loads(json_string)
names = [d['properties'] for d in j[0]['type']]
print names
I am new to python.
j is a dictionary that has keys ['type', 'metadata', 'features', 'bbox'], but not 0. You are probably looking for j['features'], not j[0]['type'], but then the value of names is a list of dictionaries, not names. I hypothesize that the JSON API of the site has changed since you (or whoever wrote the code) last used it.
I'm using the Google Visualization Library for python (gviz) to generate chart objects. This works great for generating JSON that can be read by the Google Charts using the DataTable.ToJSon method. What I'm trying to do now, however, is add multiple Google Chart data tables to one JSON dictionary. In other words, what I'm making now is this:
Chart_Data = JSON_Chart_Data_1
and what I want to make is this:
Chart_Data = {'Chart_1' : JSON_Chart_Data_1,
'Chart_2' : JSON_Chart_Data_2,}
Where Chart_Data is converted into a JSON string in both cases.
I'm pretty sure I can do this by converting the JSON string from gviz back into a python dictionary, compile the strings in a container dictionary as necessary, and then convert that container dictionary back into JSON, but that doesn't seem like a very elegant way to do it. Is there a better way? What I'm picturing is a .ToPythonObject method equivalent to .ToJSon, but there doesn't appear to be one in the library.
Thanks a lot,
Alex
I ended up doing my original, inelegant, solution to the problem with this function:
def CombineJson(JSON_List):
#Jsonlist should be a list of tuples, with the dictionary key and the json file to go with that.
#eg: [('json1', 'some json string'), ('json2', 'some other json string')]
Python_Dict = {}
for tup in JSON_List:
parsed_json = json.loads(tup[1])
Python_Dict[tup[0]] = parsed_json
BigJson = json.dumps(Python_Dict)
return BigJson
Thanks guys,
Alex
My code:
from urllib import request
import json
lat = 31.33 ; long = -84.52
webpage = "http://www.mapquestapi.com/geocoding/v1/reverse?key=MY_KEY&callback=renderReverse&location={},{}".format(lat, long)
response = request.urlopen(webpage)
json_data = response.read().decode(response.info().get_param('charset') or 'utf-8')
data = json.loads(json_data)
print(data)
This gives me following error:
ValueError: Expecting value: line 1 column 1 (char 0)
I am trying to read county and state from MapQuest reverse geocoding api. The response looks like this:
renderReverse({"info":{"statuscode":0,"copyright":{"text":"\u00A9 2015 MapQuest, Inc.","imageUrl":"http://api.mqcdn.com/res/mqlogo.gif","imageAltText":"\u00A9 2015 MapQuest, Inc."},"messages":[]},"options":{"maxResults":1,"thumbMaps":true,"ignoreLatLngInput":false},"results":[{"providedLocation":{"latLng":{"lat":32.841516,"lng":-83.660992}},"locations":[{"street":"562 Patterson St","adminArea6":"","adminArea6Type":"Neighborhood","adminArea5":"Macon","adminArea5Type":"City","adminArea4":"Bibb","adminArea4Type":"County","adminArea3":"GA","adminArea3Type":"State","adminArea1":"US","adminArea1Type":"Country","postalCode":"31204-3508","geocodeQualityCode":"P1AAA","geocodeQuality":"POINT","dragPoint":false,"sideOfStreet":"L","linkId":"0","unknownInput":"","type":"s","latLng":{"lat":32.84117,"lng":-83.660973},"displayLatLng":{"lat":32.84117,"lng":-83.660973},"mapUrl":"http://www.mapquestapi.com/staticmap/v4/getmap?key=MY_KEY&type=map&size=225,160&pois=purple-1,32.84117,-83.660973,0,0,|¢er=32.84117,-83.660973&zoom=15&rand=-189494136"}]}]})
How do I convert this string to a dict from which I can query using a key? Any help would be appreciated. Thanks!
First, get rid of the callback parameter in your URL since that's what is causing the response to be wrapped in renderReverse()
webpage = "http://www.mapquestapi.com/geocoding/v1/reverse?key=MY_KEY&location={},{}".format(lat, long)
That will give you valid json which should work with the json.loads function you call. At this point you can interact with data like a dictionary, getting the county and state names by their keys. The way mapquests structures their json is pretty strange, so it looks like you may have to do some string matching to get the correct key name. In this case, 'adminArea4Type' is set to 'County' so you want to access the 'adminArea4' key to return the county name.
data['results'][0]['locations'][0]['adminArea4']