I am trying to convert my json data from OpenWeatherMap but it keeps giving me error. when i tried other json file on other tutorial site, it was working perfectly fine.
Json format:
{
"cod": "200",
"message": 0,
"cnt": 40,
"list": [
{
"dt": 1586250000,
"main": {
"temp": 303.36,
"feels_like": 306.76,
"temp_min": 303.36,
"temp_max": 303.95,
"pressure": 1006,
"sea_level": 1006,
"grnd_level": 1004,
"humidity": 61,
"temp_kf": -0.59
},
"weather": [{
"id": 500,
"main": "Rain",
"description": "light rain",
"icon": "10d"
}],
"clouds": {
"all": 97
},
"wind": {
"speed": 1.74,
"deg": 38
},
"rain": {
"3h": 0.29
},
"sys": {
"pod": "d"
},
"dt_txt": "2020-04-07 09:00:00"
}, {
"dt": 1586260800,
"main": {
"temp": 300.42,
"feels_like": 303.73,
"temp_min": 300.42,
"temp_max": 300.86,
"pressure": 1008,
"sea_level": 1008,
"grnd_level": 1006,
"humidity": 76,
"temp_kf": -0.44
},
"weather": [{
"id": 500,
"main": "Rain",
"description": "light rain",
"icon": "10n"
}],
"clouds": {
"all": 83
},
"wind": {
"speed": 2.5,
"deg": 52
},
"rain": {
"3h": 0.53
},
"sys": {
"pod": "n"
},
"dt_txt": "2020-04-07 12:00:00"
}, {
"dt": 1586271600,
"main": {
"temp": 299.85,
"feels_like": 303.12,
"temp_min": 299.85,
"temp_max": 300.15,
"pressure": 1010,
"sea_level": 1010,
"grnd_level": 1008,
"humidity": 80,
"temp_kf": -0.3
},
"weather": [{
"id": 500,
"main": "Rain",
"description": "light rain",
"icon": "10n"
}],
"clouds": {
"all": 62
},
"wind": {
"speed": 2.78,
"deg": 32
},
"rain": {
"3h": 0.16
},
"sys": {
"pod": "n"
},
"dt_txt": "2020-04-07 15:00:00"
}
],
"city": {
"id": 1880252,
"name": "Singapore",
"coord": {
"lat": 1.2897,
"lon": 103.8501
},
"country": "SG",
"population": 3547809,
"timezone": 28800,
"sunrise": 1586214152,
"sunset": 1586257828
}
}
I have created a function with python to read the JSON with api call and convert it to a dataframe using pandas
def _weather():
url = 'http://api.openweathermap.org/data/2.5/forecast?q=Singapore,{API}'
res = requests.get(url)
data = res.json()
return data
after reading it as json , i tried to convert it to a dataframe but am receiving error
#df = pd.dataFrame(_weather)
df = pd.read_json(_weather)
Any steps did i miss out ?
The expected dataframe is as shown in the picture,
I am trying to display 16 row data as i am extracting data for 16day/daily forecast.
Try this and understand more on json_normalize,
def _weather_pd(url):
df1 = json_normalize(url['list'], 'weather')
df2 = json_normalize(url['list'])
df = df2.drop('weather', axis=1).join(df1)
return df
just another variant on the question, I prefer glom or jmespath when dealing with nested data, especially json :
the path to the weather section is : list key -> 'list container' -> weather key -> 'list container'
the lists are identified with [] symbol, while the dictionary keys are prefixed with a dot(.)
import jmespath
expression = jmespath.compile('list[].weather[]')
res = expression.search(data)
pd.DataFrame(res)
id main description icon
0 500 Rain light rain 10d
1 500 Rain light rain 10n
2 500 Rain light rain 10n
Related
I am trying to retrieve the temperature of a city using the openweathermap api however I am not able to retrieve it. I am able to retrieve the City name but not temperature and other bits I would like to retrieve. I know its probably to do with how I am trying to retrieve it but I can't figure it out.
Here is my code for trying to get the name and temperature (name works but temperature does not)
def format_response(weather):
try:
name = weather['city']['name']
temperature = weather['main']['temp']
final_str = f'City: {name} \n Temperature {temperature}'
except:
final_str = 'There was a problem retrieving that information'
return final_str
And here is the JSON given for expected API response on the openweathermap website which matches the output in my console.
{
"cod": "200",
"message": 0,
"cnt": 40,
"list": [
{
"dt": 1647345600,
"main": {
"temp": 286.88,
"feels_like": 285.93,
"temp_min": 286.74,
"temp_max": 286.88,
"pressure": 1021,
"sea_level": 1021,
"grnd_level": 1018,
"humidity": 62,
"temp_kf": 0.14
},
"weather": [
{
"id": 804,
"main": "Clouds",
"description": "overcast clouds",
"icon": "04d"
}
],
"clouds": {
"all": 85
},
"wind": {
"speed": 3.25,
"deg": 134,
"gust": 4.45
},
"visibility": 10000,
"pop": 0,
"sys": {
"pod": "d"
},
"dt_txt": "2022-03-15 12:00:00"
},
{
"dt": 1647356400,
"main": {
"temp": 286.71,
"feels_like": 285.77,
"temp_min": 286.38,
"temp_max": 286.71,
"pressure": 1021,
"sea_level": 1021,
"grnd_level": 1017,
"humidity": 63,
"temp_kf": 0.33
},
"weather": [
{
"id": 804,
"main": "Clouds",
"description": "overcast clouds",
"icon": "04d"
}
],
"clouds": {
"all": 90
},
"wind": {
"speed": 3.34,
"deg": 172,
"gust": 4.03
},
"visibility": 10000,
"pop": 0,
"sys": {
"pod": "d"
},
"dt_txt": "2022-03-15 15:00:00"
},
...
],
"city": {
"id": 2643743,
"name": "London",
"coord": {
"lat": 51.5073,
"lon": -0.1277
},
"country": "GB",
"population": 1000000,
"timezone": 0,
"sunrise": 1647324903,
"sunset": 1647367441
}
}
Given the JSON you've provided, you need to loop over the "list" property and then access the ["main"]["temp"] field of each item:
def get_temps(weather_data):
for e in weather_data["list"]:
print(e["main"]["temp"])
# this is your JSON clipped to 2 items in the
# 'list' property
js = {
"cod": "200",
"message": 0,
"cnt": 40,
"list": [
{
"dt": 1647345600,
"main": {
"temp": 286.88,
"feels_like": 285.93,
"temp_min": 286.74,
"temp_max": 286.88,
"pressure": 1021,
"sea_level": 1021,
"grnd_level": 1018,
"humidity": 62,
"temp_kf": 0.14
},
"weather": [
{
"id": 804,
"main": "Clouds",
"description": "overcast clouds",
"icon": "04d"
}
],
"clouds": {
"all": 85
},
"wind": {
"speed": 3.25,
"deg": 134,
"gust": 4.45
},
"visibility": 10000,
"pop": 0,
"sys": {
"pod": "d"
},
"dt_txt": "2022-03-15 12:00:00"
},
{
"dt": 1647356400,
"main": {
"temp": 286.71,
"feels_like": 285.77,
"temp_min": 286.38,
"temp_max": 286.71,
"pressure": 1021,
"sea_level": 1021,
"grnd_level": 1017,
"humidity": 63,
"temp_kf": 0.33
},
"weather": [
{
"id": 804,
"main": "Clouds",
"description": "overcast clouds",
"icon": "04d"
}
],
"clouds": {
"all": 90
},
"wind": {
"speed": 3.34,
"deg": 172,
"gust": 4.03
},
"visibility": 10000,
"pop": 0,
"sys": {
"pod": "d"
},
"dt_txt": "2022-03-15 15:00:00"
}
],
"city": {
"id": 2643743,
"name": "London",
"coord": {
"lat": 51.5073,
"lon": -0.1277
},
"country": "GB",
"population": 1000000,
"timezone": 0,
"sunrise": 1647324903,
"sunset": 1647367441
}
}
get_temps(js)
And I get this:
286.88
286.71
If you only care about, say, getting the most recent temperature reading, you can just read the first item of the list.
# I don't know if this API can return an empty list, but
# this also makes sure you don't access an empty list
def get_latest_temp(weather_data):
if weather_data["list"]:
return weather_data["list"][0]["main"]["temp"]
else:
return None
I've JSON file returned from API. And i need to access one variable but it's inside JSON inside list. I've tried some things but none worked. I'm using python and json module.
{
"list": [
{
"dt": 1636318800,
"main": {
"temp": 281.07,
"feels_like": 277.81,
"temp_min": 280.86,
"temp_max": 281.07,
"pressure": 1014,
"sea_level": 1014,
"grnd_level": 987,
"humidity": 72,
"temp_kf": 0.21
},
"weather": [
{
"id": 804,
"main": "Clouds",
"description": "overcast clouds",
"icon": "04n"
}
],
"clouds": {
"all": 97
},
"wind": {
"speed": 5.85,
"deg": 242,
"gust": 10.34
},
"visibility": 10000,
"pop": 0.17,
"sys": {
"pod": "n"
},
"dt_txt": "2021-11-07 21:00:00"
}]
}
with open("test.json") as json_file:
data = json.load(json_file)
for p in data[list]["main"]:
temps = p["temp_min"]
If you want to access the same variable for each element in the list you need to use
for p in data[list]:
temps = p["main"]["temp_min"]
# do something with temps var...
If you just want the variable from one specific item in the list you should use
temps = data[list][0]["main"]["temp_min"]
I have a dictionary like this:
dici = {
"city": {
"coord": {
"lat": 123123123,
"lon": 123123
},
"country": "asdsd",
"id": 2735943,
"name": "qweqwe",
"population": 249633,
"sunrise": 1617689349,
"sunset": 1617735844,
"timezone": 3600
},
"cnt": 40,
"cod": "200",
"list": [
{
"clouds": {
"all": 0
},
"dt": 1617721200,
"dt_txt": "2021-04-06 15:00:00",
"main": {
"feels_like": 17.87,
"grnd_level": 1002,
"humidity": 65,
"pressure": 1014,
"sea_level": 1014,
"temp": 18.29,
"temp_kf": 0.25,
"temp_max": 18.29,
"temp_min": 18.04
},
"pop": 0,
"sys": {
"pod": "d"
},
"visibility": 10000,
"weather": [
{
"description": "clear sky",
"icon": "01d",
"id": 800,
"main": "Clear"
}
],
"wind": {
"deg": 299,
"speed": 3.36
}
},
{
"clouds": {
"all": 16
},
"dt": 1617732000,
"dt_txt": "2021-04-06 18:00:00",
"main": {
"feels_like": 15.78,
"grnd_level": 1001,
"humidity": 63,
"pressure": 1013,
"sea_level": 1013,
"temp": 16.44,
"temp_kf": 0.58,
"temp_max": 16.44,
"temp_min": 15.86
},
"pop": 0,
"sys": {
"pod": "d"
},
"visibility": 10000,
"weather": [
{
"description": "few clouds",
"icon": "02d",
"id": 801,
"main": "Clouds"
}
],
"wind": {
"deg": 295,
"speed": 2.21
}
}
]
}`
And I would like to extract the entries of the array list when for example temp_max is greater than 16. What I have been doing is the following:
unwind = db.forecast.aggregate([
{"$unwind": "$list"},
{"$project":{"list.dt_txt":1,
"list.main.temp":1,
"list.main.temp_max":1,
"list.main.temp_min":1}},
{"$match":{"list.main.temp_max":{"$gt":10}}
}
])
a = db.forecast.aggregate([
{"$match":
{"list.main[*].temp_max": {"$gt": 10}}}
])
b = db.forecast.find({"list.main.temp_max":{"$gt":16}})
But all of them get errors in the syntax, I can't seem to find the right syntax to access the temp_max key. Anyone can help me? Thanks!
You were close with the first query; this should work:
unwind = db.forecast.aggregate([
{"$unwind": "$list"},
{"$match": {"list.main.temp_max": {"$gt": 16}}},
{"$project": {"list.dt_txt": 1, "list.main.temp": 1,
"list.main.temp_max": 1, "list.main.temp_min": 1, '_id': 0}}
])
Alternatively a find would work but only matches on the first list item, so the aggregate query is likely better.
b = db.forecast.find({"list.main.temp_max": {"$gt": 16}}, {'list.main.$': 1, '_id': 0})
My IBM cloudant is connected to python.
I need to get nested JSON data from cloudant to python environment. My aim is to take the necessary data and create a new dataframe.
Here it is nested JSON
{
"_id": "1",
"topic": "",
"payload": {
"devices": [
{
"_id": "70",
"last_date": 1499503900,
"modules": [
{
"_id": "0",
"last_seen": 1497151699,
"dashboard_data": {
"time_utc": 1497150981,
"Temperature": 41.8,
"Humidity": 24,
"date_max_temp": 1497150519,
"date_min_temp": 1497146892,
"min_temp": 13.9,
"max_temp": 45.6
},
"data_type": [
"Temperature",
"Humidity"
],
"module_name": "X",
"last_setup": 1487170865,
"battery_vp": 3474,
"battery_percent": 0,
"rf_status": 79,
"firmware": 44
},
{
"_id": "03",
"last_message": 1499503897,
"last_seen": 1499503897,
"dashboard_data": {
"time_utc": 1499503795,
"Temperature": 19.9,
"temp_trend": "stable",
"Humidity": 49,
"CO2": 390,
"date_max_temp": 1499468625,
"date_min_temp": 1499490158,
"min_temp": 19.4,
"max_temp": 21.2
},
"data_type": [
"Temperature",
"CO2",
"Humidity"
],
"module_name": "Y",
"last_setup": 1487173334,
"battery_vp": 5484,
"battery_percent": 71,
"rf_status": 70,
"firmware": 44
},
{
"_id": "03",
"last_message": 1499503897,
"last_seen": 1499503852,
"dashboard_data": {
"time_utc": 1499503801,
"Temperature": 19.7,
"temp_trend": "stable",
"Humidity": 51,
"CO2": 414,
"date_max_temp": 1499468631,
"date_min_temp": 1499489242,
"min_temp": 18.5,
"max_temp": 20.7
},
"data_type": [
"Temperature",
"CO2",
"Humidity"
],
"module_name": "M",
"last_setup": 1487173386,
"battery_vp": 5518,
"battery_percent": 73,
"rf_status": 83,
"firmware": 44
},
{
"_id": "03",
"last_message": 1499503897,
"last_seen": 1499503872,
"dashboard_data": {
"time_utc": 1499503820,
"Temperature": 22,
"temp_trend": "stable",
"Humidity": 49,
"CO2": 419,
"date_max_temp": 1499468599,
"date_min_temp": 1499495002,
"min_temp": 22,
"max_temp": 22.9
},
"data_type": [
"Temperature",
"CO2",
"Humidity"
],
"module_name": "N",
"last_setup": 1487173462,
"battery_vp": 5494,
"battery_percent": 72,
"rf_status": 77,
"firmware": 44
}
],
"place": {
"country": "GB",
},
"type": "NAMain",
"read_only": "true",
"dashboard_data": {
"AbsolutePressure": 1010.5,
"Noise": 45,
"Temperature": 20.5,
"temp_trend": "stable",
"Humidity": 52,
"Pressure": 1010.5,
"pressure_trend": "up",
"CO2": 406,
"date_max_temp": 1499468648,
"date_min_temp": 1499500180,
"min_temp": 20.4,
"max_temp": 22.1
},
"data_type": [
"Temperature",
"CO2",
"Humidity",
"Noise",
"Pressure"
],
"co2_calibrating": "false",
"date_setup": 1487170943,
"last_setup": 1487170943,
"module_name": "K",
"firmware": 124,
"last_upgrade": 1487170889,
"wifi_status": 56
}
]
}
}
I have almost 50000 json data like this.
My aim is to creat a new pandas dataframe, which includes last_date in the beginning and Temperature for each _id.
I tried this:
import cloudant
import pandas as pd
import json
database2 = client['net']
for item in database2:
print(item['payload']['devices'][0]['last_date'])
But it gives me
KeyError: 'payload'
and secondly this:
wanted = ['las_date', 'module_name']
for item in database2:
if not 'las_date' in item:
continue
data = item['las_date']
for key in wanted:
if key in details:
print(key, ':', json.dumps(details[key], indent=4))
But I got this:
HTTPError: 429 Client Error: Too Many Requests
After correcting the json data, I tried
item['payload']['devices'][0]['last_date']
and I get output as:
1499503900
The error you see:
HTTPError: 429 Client Error: Too Many Requests
is a consequence of Cloudant being rate limited on IBM Cloud. The free version of Cloudant ("Lite") allows for 100 lookups, 10 writes and 5 queries per second after which you'll get the error above.
You can increase the limits (at a cost) from the Cloudant dashboard. The Cloudant Python library you use has the ability to retry-backoff on 429 errors which may relieve your issue. See
http://python-cloudant.readthedocs.io/en/latest/getting_started.html?highlight=429
I'm still new to Python while exploring, learning, and today I'm working with JSON. How do I parse and print 'dt_txt' key and corresponding value each time it appears in my dictionary? Here is my code.
import requests, pytemperature, json
r = requests.get('http://samples.openweathermap.org/data/2.5/forecast?
lat=35&lon=139&appid=b1b15e88fa797225412429c1c50c122a1')
dict = r.json()
for key, value in dict.items():
if 'dt_txt' in str(key):
print(key)
Here is JSON snapshot or full content in link above.
{
"cod": "200",
"message": 0.179,
"cnt": 40,
"list": [{
"dt": 1509202800,
"main": {
"temp": 297.18,
"temp_min": 291.573,
"temp_max": 297.18,
"pressure": 1027.02,
"sea_level": 1029.75,
"grnd_level": 1027.02,
"humidity": 68,
"temp_kf": 5.6
},
"weather": [{
"id": 500,
"main": "Rain",
"description": "light rain",
"icon": "10d"
}
],
"clouds": {
"all": 88
},
"wind": {
"speed": 1.61,
"deg": 99.0033
},
"rain": {
"3h": 0.09
},
"sys": {
"pod": "d"
},
"dt_txt": "2017-10-28 15:00:00"
}, {
"dt": 1509213600,
"main": {
"temp": 297.32,
"temp_min": 293.116,
"temp_max": 297.32,
"pressure": 1024.56,
"sea_level": 1027.16,
"grnd_level": 1024.56,
"humidity": 76,
"temp_kf": 4.2
},
"weather": [{
"id": 500,
"main": "Rain",
"description": "light rain",
"icon": "10d"
}
],
"clouds": {
"all": 48
},
"wind": {
"speed": 1.96,
"deg": 173.002
},
"rain": {
"3h": 0.41
},
"sys": {
"pod": "d"
},
"dt_txt": "2017-10-28 18:00:00"
Side note: In the end I am trying to print the date, temp_min, temp_max, main, and description. I will be converting the temp from kelvin to fahrenheit then using gmail to text message me each day the new forecast. Thank you in advance for any help!
You can all each dict keys by your_dict['your_value'], try this:
import requests, json
r = requests.get('http://samples.openweathermap.org/data/2.5/forecast?lat=35&lon=139&appid=b1b15e88fa797225412429c1c50c122a1')
dict = r.json()
select_data = dict['list']
for box in select_data:
if 'dt_txt' in box:
print(box['dt_txt'])
else:
print('not found')