Related
I have an example of annotation file
{'text': "BELGIE BELGIQUE BELGIEN\nIDENTITEITSKAART CARTE D'IDENTITE PERSONALAUSWEIS\nBELGIUM\nIDENTITY CARD\nNaam / Name\nDermrive\nVoornamen / Given names\nBrando Jerom L\nGeslacht / Nationaliteit /\nGeboortedatum /\nSex\nNationality\nDate of birth\nM/M\nBEL\n19 05 1982\nRijksregisternr. 7 National Register Nº\n85.08.23-562.77\nKaartnr. / Card Nº\n752-0465474-34\nVervalt op / Expires on\n23 07 2025\n", 'spans': [{'start': 24, 'end': 40, 'token_start': 16, 'token_end': 16, 'label': 'CardType'}, {'start': 41, 'end': 57, 'token_start': 16, 'token_end': 16, 'label': 'CardType'}, {'start': 58, 'end': 73, 'token_start': 15, 'token_end': 15, 'label': 'CardType'}, {'start': 108, 'end': 116, 'token_start': 8, 'token_end': 8, 'label': 'LastName'}, {'start': 141, 'end': 155, 'token_start': 14, 'token_end': 14, 'label': 'FirstName'}, {'start': 229, 'end': 232, 'token_start': 3, 'token_end': 3, 'label': 'Gender_nid'}, {'start': 233, 'end': 236, 'token_start': 3, 'token_end': 3, 'label': 'Nationality_nid'}, {'start': 237, 'end': 247, 'token_start': 10, 'token_end': 10, 'label': 'DateOfBirth_nid'}, {'start': 288, 'end': 303, 'token_start': 15, 'token_end': 15, 'label': 'Ssn'}, {'start': 323, 'end': 337, 'token_start': 14, 'token_end': 14, 'label': 'CardNumber'}, {'start': 362, 'end': 372, 'token_start': 10, 'token_end': 10, 'label': 'ValidUntil_nid'}]}
So when a i have a start and end position of "LastName"entity, in the example is "Dermrive", when i produce another, shorter or longer LastName for example "Brad", i need to change all the rest by difference of this words, so that other labels stays in the correct postition. Its works perfecly with one entity, but when i try to change all of them, the output is messy and labels are not correct anymore.
def replace_text_by_index_and_type(self, new_text, type):
label_position = self.search_label_position_in_spans(self.annotation['spans'], type.value)
label = self.annotation['spans'][label_position]
begin_new_string = self.annotation["text"][:label["start"]]
end_new_string = self.annotation["text"][label["end"]:]
new_string = begin_new_string + new_text + end_new_string
for to_change_ent in self.annotation['spans'][label_position+1:]:
diff = len(new_text) - (label["end"] - label["start"])
self.annotation['spans'][label_position]["end"] = self.annotation['spans'][label_position]["end"] + diff
#print(f"Diff between original {to_change_ent} and new_string: {diff}")
to_change_ent["start"] += diff
to_change_ent["end"] += diff
return new_string
I start to change all entities from the second one, to keep the start position of first one. And add diff to ending position of first entity, as a results the firstname and lastname are correct, but other entities are shifted to mess.
What is the difference between studentsDict.values() and studentsDict[key].values in the following code?
studentsDict = {'Ayush': {'maths': 24, 'english': 19, 'hindi': 97, 'bio': 20, 'science': 0}, 'Pankaj': {'maths': 52, 'english': 76, 'hindi': 68, 'bio': 97, 'science': 66}, 'Raj': {'maths': 85, 'english': 79, 'hindi': 51, 'bio': 36, 'science': 75}, 'iC5z4DK': {'maths': 24, 'english': 92, 'hindi': 31, 'bio': 29, 'science': 91}, 'Zf1WSV6': {'maths': 81, 'english': 58, 'hindi': 85, 'bio': 31, 'science': 7}}
for key in studentsDict.keys():
for marks in studentsDict[key].values():
if marks < 33:
print(key, "FAILED")
break
studentsDict.keys() gives you each of the keys in the outer dict: "Ayush", "Pankaj", "Raj", "iC5z4DK" and "Zf1WSV6".
studentsDict[key].values() gives you the values for the entry in studentsDict corresponding to key. For example, if key is "Ayush", you would get 24, 19, 97, 20, and 0.
I'm getting weather forecasting data from weatherstack API.
params = {
'access_key': 'enter_key_here',
'query': 'Montreal',
'forecast_days': '2', #two days forecast
'hourly': '1' #API to return weather data split hourly
}
api_result = requests.get('https://api.weatherstack.com/forecast', params)
api_response = api_result.json()
api_response
The output looks like this:
{'request': {'type': 'City',
'query': 'Montreal, Canada',
'language': 'en',
'unit': 'm'},
'location': {'name': 'Montreal',
'country': 'Canada',
'region': 'Quebec',
'lat': '45.500',
'lon': '-73.583',
'timezone_id': 'America/Toronto',
'localtime': '2021-05-11 13:08',
'localtime_epoch': 1620738480,
'utc_offset': '-4.0'},
'current': {'observation_time': '05:08 PM',
'temperature': 11,
'weather_code': 122,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0004_black_low_cloud.png'],
'weather_descriptions': ['Overcast'],
'wind_speed': 31,
'wind_degree': 230,
'wind_dir': 'SW',
'pressure': 1012,
'precip': 1,
'humidity': 58,
'cloudcover': 100,
'feelslike': 8,
'uv_index': 2,
'visibility': 14,
'is_day': 'yes'},
'forecast': {'2021-05-11': {'date': '2021-05-11',
'date_epoch': 1620691200,
'astro': {'sunrise': '05:28 AM',
'sunset': '08:14 PM',
'moonrise': '05:35 AM',
'moonset': '08:14 PM',
'moon_phase': 'Waxing Crescent',
'moon_illumination': 7},
'mintemp': 6,
'maxtemp': 13,
'avgtemp': 8,
'totalsnow': 0,
'sunhour': 12.4,
'uv_index': 6,
'hourly': [{'time': '0',
'temperature': 9,
'wind_speed': 16,
'wind_degree': 240,
'wind_dir': 'WSW',
'weather_code': 119,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0003_white_cloud.png'],
'weather_descriptions': ['Cloudy'],
'precip': 0.1,
'humidity': 74,
'visibility': 10,
'pressure': 1012,
'cloudcover': 75,
'heatindex': 9,
'dewpoint': 5,
'windchill': 7,
'windgust': 23,
'feelslike': 7,
'chanceofrain': 43,
'chanceofremdry': 31,
'chanceofwindy': 0,
'chanceofovercast': 86,
'chanceofsunshine': 5,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 1},
{'time': '300',
'temperature': 8,
'wind_speed': 15,
'wind_degree': 247,
'wind_dir': 'WSW',
'weather_code': 122,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0004_black_low_cloud.png'],
'weather_descriptions': ['Overcast'],
'precip': 0,
'humidity': 83,
'visibility': 10,
'pressure': 1012,
'cloudcover': 88,
'heatindex': 8,
'dewpoint': 6,
'windchill': 6,
'windgust': 21,
'feelslike': 6,
'chanceofrain': 0,
'chanceofremdry': 89,
'chanceofwindy': 0,
'chanceofovercast': 88,
'chanceofsunshine': 15,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 1},
{'time': '600',
'temperature': 7,
'wind_speed': 17,
'wind_degree': 239,
'wind_dir': 'WSW',
'weather_code': 122,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0004_black_low_cloud.png'],
'weather_descriptions': ['Overcast'],
'precip': 0,
'humidity': 90,
'visibility': 10,
'pressure': 1012,
'cloudcover': 100,
'heatindex': 7,
'dewpoint': 6,
'windchill': 4,
'windgust': 24,
'feelslike': 4,
'chanceofrain': 0,
'chanceofremdry': 84,
'chanceofwindy': 0,
'chanceofovercast': 90,
'chanceofsunshine': 15,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 2},
{'time': '900',
'temperature': 8,
'wind_speed': 21,
'wind_degree': 248,
'wind_dir': 'WSW',
'weather_code': 296,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0017_cloudy_with_light_rain.png'],
'weather_descriptions': ['Light rain'],
'precip': 0.5,
'humidity': 88,
'visibility': 10,
'pressure': 1012,
'cloudcover': 100,
'heatindex': 8,
'dewpoint': 6,
'windchill': 5,
'windgust': 27,
'feelslike': 5,
'chanceofrain': 28,
'chanceofremdry': 57,
'chanceofwindy': 0,
'chanceofovercast': 83,
'chanceofsunshine': 11,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 2},
{'time': '1200',
'temperature': 10,
'wind_speed': 24,
'wind_degree': 259,
'wind_dir': 'WSW',
'weather_code': 293,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0017_cloudy_with_light_rain.png'],
'weather_descriptions': ['Patchy light rain'],
'precip': 1.3,
'humidity': 71,
'visibility': 9,
'pressure': 1012,
'cloudcover': 85,
'heatindex': 10,
'dewpoint': 4,
'windchill': 7,
'windgust': 28,
'feelslike': 7,
'chanceofrain': 82,
'chanceofremdry': 0,
'chanceofwindy': 0,
'chanceofovercast': 87,
'chanceofsunshine': 0,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 2},
{'time': '1500',
'temperature': 12,
'wind_speed': 23,
'wind_degree': 273,
'wind_dir': 'W',
'weather_code': 176,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0009_light_rain_showers.png'],
'weather_descriptions': ['Patchy rain possible'],
'precip': 0.7,
'humidity': 49,
'visibility': 9,
'pressure': 1012,
'cloudcover': 66,
'heatindex': 12,
'dewpoint': 2,
'windchill': 10,
'windgust': 27,
'feelslike': 10,
'chanceofrain': 79,
'chanceofremdry': 0,
'chanceofwindy': 0,
'chanceofovercast': 89,
'chanceofsunshine': 0,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 3},
{'time': '1800',
'temperature': 11,
'wind_speed': 21,
'wind_degree': 277,
'wind_dir': 'W',
'weather_code': 116,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0002_sunny_intervals.png'],
'weather_descriptions': ['Partly cloudy'],
'precip': 0.1,
'humidity': 54,
'visibility': 9,
'pressure': 1011,
'cloudcover': 73,
'heatindex': 11,
'dewpoint': 2,
'windchill': 9,
'windgust': 26,
'feelslike': 9,
'chanceofrain': 56,
'chanceofremdry': 30,
'chanceofwindy': 0,
'chanceofovercast': 68,
'chanceofsunshine': 27,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 4},
{'time': '2100',
'temperature': 8,
'wind_speed': 17,
'wind_degree': 273,
'wind_dir': 'W',
'weather_code': 176,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0025_light_rain_showers_night.png'],
'weather_descriptions': ['Patchy rain possible'],
'precip': 0,
'humidity': 75,
'visibility': 10,
'pressure': 1013,
'cloudcover': 61,
'heatindex': 8,
'dewpoint': 3,
'windchill': 5,
'windgust': 24,
'feelslike': 5,
'chanceofrain': 22,
'chanceofremdry': 60,
'chanceofwindy': 0,
'chanceofovercast': 59,
'chanceofsunshine': 53,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 1}]},
'2021-05-12': {'date': '2021-05-12',
'date_epoch': 1620777600,
'astro': {'sunrise': '05:27 AM',
'sunset': '08:15 PM',
'moonrise': '05:59 AM',
'moonset': '09:19 PM',
'moon_phase': 'Waxing Crescent',
'moon_illumination': 14},
'mintemp': 5,
'maxtemp': 15,
'avgtemp': 10,
'totalsnow': 0,
'sunhour': 11.4,
'uv_index': 4,
'hourly': [{'time': '0',
'temperature': 6,
'wind_speed': 16,
'wind_degree': 266,
'wind_dir': 'W',
'weather_code': 122,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0004_black_low_cloud.png'],
'weather_descriptions': ['Overcast'],
'precip': 0.1,
'humidity': 85,
'visibility': 10,
'pressure': 1013,
'cloudcover': 100,
'heatindex': 6,
'dewpoint': 4,
'windchill': 3,
'windgust': 21,
'feelslike': 3,
'chanceofrain': 44,
'chanceofremdry': 29,
'chanceofwindy': 0,
'chanceofovercast': 92,
'chanceofsunshine': 4,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 1},
{'time': '300',
'temperature': 6,
'wind_speed': 13,
'wind_degree': 270,
'wind_dir': 'W',
'weather_code': 122,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0004_black_low_cloud.png'],
'weather_descriptions': ['Overcast'],
'precip': 0,
'humidity': 85,
'visibility': 10,
'pressure': 1013,
'cloudcover': 98,
'heatindex': 6,
'dewpoint': 4,
'windchill': 3,
'windgust': 18,
'feelslike': 3,
'chanceofrain': 0,
'chanceofremdry': 88,
'chanceofwindy': 0,
'chanceofovercast': 90,
'chanceofsunshine': 13,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 1},
{'time': '600',
'temperature': 7,
'wind_speed': 11,
'wind_degree': 293,
'wind_dir': 'WNW',
'weather_code': 122,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0004_black_low_cloud.png'],
'weather_descriptions': ['Overcast'],
'precip': 0,
'humidity': 81,
'visibility': 10,
'pressure': 1016,
'cloudcover': 94,
'heatindex': 7,
'dewpoint': 3,
'windchill': 4,
'windgust': 16,
'feelslike': 4,
'chanceofrain': 0,
'chanceofremdry': 91,
'chanceofwindy': 0,
'chanceofovercast': 88,
'chanceofsunshine': 15,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 2},
{'time': '900',
'temperature': 10,
'wind_speed': 13,
'wind_degree': 324,
'wind_dir': 'NW',
'weather_code': 176,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0009_light_rain_showers.png'],
'weather_descriptions': ['Patchy rain possible'],
'precip': 0,
'humidity': 60,
'visibility': 10,
'pressure': 1018,
'cloudcover': 98,
'heatindex': 10,
'dewpoint': 3,
'windchill': 8,
'windgust': 15,
'feelslike': 8,
'chanceofrain': 23,
'chanceofremdry': 61,
'chanceofwindy': 0,
'chanceofovercast': 93,
'chanceofsunshine': 7,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 3},
{'time': '1200',
'temperature': 14,
'wind_speed': 14,
'wind_degree': 344,
'wind_dir': 'NNW',
'weather_code': 176,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0009_light_rain_showers.png'],
'weather_descriptions': ['Patchy rain possible'],
'precip': 0.1,
'humidity': 42,
'visibility': 10,
'pressure': 1018,
'cloudcover': 95,
'heatindex': 14,
'dewpoint': 1,
'windchill': 13,
'windgust': 16,
'feelslike': 13,
'chanceofrain': 66,
'chanceofremdry': 0,
'chanceofwindy': 0,
'chanceofovercast': 90,
'chanceofsunshine': 0,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 3},
{'time': '1500',
'temperature': 15,
'wind_speed': 10,
'wind_degree': 332,
'wind_dir': 'NNW',
'weather_code': 116,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0002_sunny_intervals.png'],
'weather_descriptions': ['Partly cloudy'],
'precip': 0.1,
'humidity': 39,
'visibility': 10,
'pressure': 1018,
'cloudcover': 72,
'heatindex': 15,
'dewpoint': 1,
'windchill': 14,
'windgust': 12,
'feelslike': 14,
'chanceofrain': 41,
'chanceofremdry': 28,
'chanceofwindy': 0,
'chanceofovercast': 72,
'chanceofsunshine': 23,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 4},
{'time': '1800',
'temperature': 14,
'wind_speed': 6,
'wind_degree': 271,
'wind_dir': 'W',
'weather_code': 116,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0002_sunny_intervals.png'],
'weather_descriptions': ['Partly cloudy'],
'precip': 0,
'humidity': 44,
'visibility': 10,
'pressure': 1019,
'cloudcover': 44,
'heatindex': 14,
'dewpoint': 2,
'windchill': 14,
'windgust': 9,
'feelslike': 14,
'chanceofrain': 0,
'chanceofremdry': 85,
'chanceofwindy': 0,
'chanceofovercast': 37,
'chanceofsunshine': 75,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 4},
{'time': '2100',
'temperature': 12,
'wind_speed': 6,
'wind_degree': 239,
'wind_dir': 'WSW',
'weather_code': 116,
'weather_icons': ['https://assets.weatherstack.com/images/wsymbols01_png_64/wsymbol_0004_black_low_cloud.png'],
'weather_descriptions': ['Partly cloudy'],
'precip': 0,
'humidity': 56,
'visibility': 10,
'pressure': 1020,
'cloudcover': 41,
'heatindex': 12,
'dewpoint': 3,
'windchill': 12,
'windgust': 10,
'feelslike': 12,
'chanceofrain': 0,
'chanceofremdry': 84,
'chanceofwindy': 0,
'chanceofovercast': 37,
'chanceofsunshine': 82,
'chanceoffrost': 0,
'chanceofhightemp': 0,
'chanceoffog': 0,
'chanceofsnow': 0,
'chanceofthunder': 0,
'uv_index': 1}]}}}
How do I iterate through this output and create a new data frame to look like this:
(And get only the date time hour, and rain values)
Date
precip at time 0
precip at time 300
...
2021-05-11
0.1
0
...
2021-05-12
0.1
0
...
...
...
...
...
Also, I want to convert the date column to Month Day ,Year. ie. 2021-05-11 -> May 05, 2021. I tried using
df['Date'] = datetime.datetime.strftime('%b %d,%Y')
then
df['Date'] = pandas.to_datetime(df['Date'])
But I get this error:
AttributeError: type object 'datetime.datetime' has no attribute 'datetime'
Create dataframe
This will create a dataframe with the required data and column names.
import pandas as pd
### code to get data
forecast={}
for date in api_response['forecast']:
precip = {f"precip at time {api_response['forecast'][date]['hourly'][hour]['time']}":
api_response['forecast'][date]['hourly'][hour]['precip'] for hour in range(0, 8)}
forecast[date] = precip
df = pd.DataFrame.from_dict(forecast, orient='index', dtype=float).reset_index()
df.rename(columns={'index':'Date'}, inplace=True)
print(df)
Create dataframe and format headers
If you wanted the times in HH:MM format make these changes/additions to the code.
import json
import pandas as pd
### code to get data
forecast={}
for date in api_response['forecast']:
precip = {api_response['forecast'][date]['hourly'][hour]['time']:
api_response['forecast'][date]['hourly'][hour]['precip'] for hour in range(0, 8)}
forecast[date] = precip
df = pd.DataFrame.from_dict(forecast, orient='index', dtype=float).reset_index()
cols = ['Date' if col=='index' else f'precip at time {col.zfill(4)[:2]+":"+col.zfill(4)[2:]}' for col in df.columns]
df.columns = cols
print(df)
Change Date column format
To change the format of the Date column we first need to convert it to a datetime, which we can do using pd.to_datetime.
We can then chain strftime to get the required format, May 05, 2021.
For a full list of the directives you can use with strftime see here.
df['Date'] = pd.to_datetime(df['Date']).dt.strftime('%B %d, %Y')
This question already has answers here:
Get country name from Country code in python?
(3 answers)
Closed 4 years ago.
I'm interested in getting a mapping of country codes to international phone number prefixes, like so:
{'US': '+1', 'GB': '+44', 'DE': '+49', ...}
One library that probably contains this information is python-phonenumbers. However, after a quick perusal of the source code I wasn't able to find where this information is stored. For example, the shortdata/region_DE.py module looks like this:
"""Auto-generated file, do not edit by hand. DE metadata"""
from ..phonemetadata import NumberFormat, PhoneNumberDesc, PhoneMetadata
PHONE_METADATA_DE = PhoneMetadata(id='DE', country_code=None, international_prefix=None,
general_desc=PhoneNumberDesc(national_number_pattern='1\\d{2,5}', possible_length=(3, 6)),
toll_free=PhoneNumberDesc(national_number_pattern='116\\d{3}', example_number='116000', possible_length=(6,)),
emergency=PhoneNumberDesc(national_number_pattern='11[02]', example_number='112', possible_length=(3,)),
short_code=PhoneNumberDesc(national_number_pattern='11(?:[025]|6(?:00[06]|1(?:1[17]|23)))', example_number='115', possible_length=(3, 6)),
short_data=True)
It seems like the country_code and international_prefix fields are None. How can I get such a mapping (possibly with a different library)?
You can get the mapping you want using pycountry and phonenumbers, along with a simple dictionary comprehension:
import phonenumbers as pn
import pycountry
dct = {c.alpha_2: pn.country_code_for_region(c.alpha_2) for c in pycountry.countries}
print(dct)
Output:
{'SK': 421, 'KI': 686, 'LV': 371, 'GH': 233, 'JP': 81, 'SA': 966, 'TD': 235, 'SX': 1, 'CY': 357, 'CH': 41, 'EG': 20, 'PA': 507, 'KP': 850, 'CO': 57, 'GW': 245, 'KG': 996, 'AW': 297, 'FM': 691, 'SB': 677, 'HR': 385, 'PY': 595, 'BG': 359, 'IQ': 964, 'ID': 62, 'GQ': 240, 'CA': 1, 'CG': 242, 'MO': 853, 'SL': 232, 'LA': 856, 'OM': 968, 'MP': 1, 'DK': 45, 'FI': 358, 'DO': 1, 'BM': 1, 'GN': 224, 'NE': 227, 'ER': 291, 'DE': 49, 'UM': 0, 'CM': 237, 'PR': 1, 'RO': 40, 'AZ': 994, 'DZ': 213, 'BW': 267, 'MK': 389, 'HN': 504, 'IS': 354, 'SJ': 47, 'ME': 382, 'NR': 674, 'AD': 376, 'BY': 375, 'RE': 262, 'PG': 675, 'SO': 252, 'NO': 47, 'CC': 61, 'EE': 372, 'BN': 673, 'AU': 61, 'HM': 0, 'ML': 223, 'BD': 880, 'GE': 995, 'US': 1, 'UY': 598, 'SM': 378, 'NG': 234, 'BE': 32, 'KY': 1, 'AR': 54, 'CR': 506, 'VA': 39, 'YE': 967, 'TR': 90, 'CV': 238, 'DM': 1, 'ZM': 260, 'BR': 55, 'MG': 261, 'BL': 590, 'FJ': 679, 'SH': 290, 'KN': 1, 'ZA': 27, 'CF': 236, 'ZW': 263, 'PL': 48, 'SV': 503, 'QA': 974, 'MN': 976, 'SE': 46, 'JE': 44, 'PS': 970, 'MZ': 258, 'TK': 690, 'PM': 508, 'CW': 599, 'HK': 852, 'LB': 961, 'SY': 963, 'LC': 1, 'IE': 353, 'RW': 250, 'NL': 31, 'MA': 212, 'GM': 220, 'IR': 98, 'AT': 43, 'SZ': 268, 'GT': 502, 'MT': 356, 'BQ': 599, 'MX': 52, 'NC': 687, 'CK': 682, 'SI': 386, 'VE': 58, 'IM': 44, 'AM': 374, 'SD': 249, 'LY': 218, 'LI': 423, 'TN': 216, 'UG': 256, 'RU': 7, 'DJ': 253, 'IL': 972, 'TM': 993, 'BF': 226, 'GF': 594, 'TO': 676, 'GI': 350, 'MH': 692, 'UZ': 998, 'PF': 689, 'KZ': 7, 'GA': 241, 'PE': 51, 'TV': 688, 'BT': 975, 'MQ': 596, 'MF': 590, 'AF': 93, 'IN': 91, 'AX': 358, 'BH': 973, 'JM': 1, 'MY': 60, 'BO': 591, 'AI': 1, 'SR': 597, 'ET': 251, 'ES': 34, 'TF': 0, 'GU': 1, 'BJ': 229, 'SS': 211, 'KE': 254, 'BZ': 501, 'IO': 246, 'MU': 230, 'CL': 56, 'MD': 373, 'LU': 352, 'TJ': 992, 'EC': 593, 'VG': 1, 'NZ': 64, 'VU': 678, 'FO': 298, 'LR': 231, 'AL': 355, 'GB': 44, 'AS': 1, 'IT': 39, 'TC': 1, 'TW': 886, 'BI': 257, 'HU': 36, 'TL': 670, 'GG': 44, 'PN': 0, 'SG': 65, 'LS': 266, 'KH': 855, 'FR': 33, 'BV': 0, 'CX': 61, 'AE': 971, 'LT': 370, 'PT': 351, 'KR': 82, 'BB': 1, 'TG': 228, 'AQ': 0, 'EH': 212, 'AG': 1, 'VN': 84, 'CI': 225, 'BS': 1, 'GL': 299, 'MW': 265, 'NU': 683, 'NF': 672, 'LK': 94, 'MS': 1, 'GP': 590, 'NP': 977, 'PW': 680, 'PK': 92, 'WF': 681, 'BA': 387, 'KM': 269, 'JO': 962, 'CU': 53, 'GR': 30, 'YT': 262, 'RS': 381, 'NA': 264, 'ST': 239, 'SC': 248, 'CN': 86, 'CD': 243, 'GS': 0, 'KW': 965, 'MM': 95, 'AO': 244, 'MV': 960, 'UA': 380, 'TT': 1, 'FK': 500, 'WS': 685, 'CZ': 420, 'PH': 63, 'VI': 1, 'TZ': 255, 'MR': 222, 'MC': 377, 'SN': 221, 'HT': 509, 'VC': 1, 'NI': 505, 'GD': 1, 'GY': 592, 'TH': 66}
I have just found a python library that must be perfect for your problem.
It's called PhoneISO3166.
This is the github link: GitHub phoneiso3166
I am querying a whole-house power monitor (Neurio) that returns data in JSON format.
When I enter the URL in a Chrome browser, http://192.168.1.87/current-sample, I get properly formatted JSON data as follows:
{"sensorId":"0x0000C47F51019B7D","timestamp":"2016-12-24T14:56:08Z","channels":[{"type":"PHASE_A_CONSUMPTION","ch":1,"eImp_Ws":55552784178,"eExp_Ws":23,"p_W":3188,"q_VAR":321,"v_V":121.753},{"type":"PHASE_B_CONSUMPTION","ch":2,"eImp_Ws":62493402411,"eExp_Ws":23,"p_W":3499,"q_VAR":263,"v_V":120.334},{"type":"CONSUMPTION","ch":3,"eImp_Ws":118046186640,"eExp_Ws":41,"p_W":6687,"q_VAR":584,"v_V":121.044}],"cts":[{"ct":1,"p_W":3188,"q_VAR":321,"v_V":121.753},{"ct":2,"p_W":3499,"q_VAR":263,"v_V":120.334},{"ct":3,"p_W":0,"q_VAR":0,"v_V":0.000},{"ct":4,"p_W":0,"q_VAR":0,"v_V":121.747}]}
which parses correctly in JSONLint.
When I attempt to pull the same data in Python using the following line of code:
pvdata = requests.get('http://'+neurioip+'/current-sample').json()
The returned data includes invalid characters as in the following example. (Data retrieved by simply printing pvdata)
{u'channels': [{u'eExp_Ws': 23, u'v_V': 122.434, u'ch': 1, u'eImp_Ws': 55554346060, u'q_VAR': 305, u'p_W': 1489, u'type': u'PHASE_A_CONSUMPTION'}, {u'eExp_Ws': 23, u'v_V': 120.981, u'ch': 2, u'eImp_Ws': 62495160471, u'q_VAR': 237, u'p_W': 1872, u'type': u'PHASE_B_CONSUMPTION'}, {u'eExp_Ws': 41, u'v_V': 121.708, u'ch': 3, u'eImp_Ws': 118049506582, u'q_VAR': 542, u'p_W': 3360, u'type': u'CONSUMPTION'}], u'sensorId': u'0x0000C47F51019B7D', u'cts': [{u'p_W': 1489, u'q_VAR': 305, u'v_V': 122.434, u'ct': 1}, {u'p_W': 1872, u'q_VAR': 237, u'v_V': 120.981, u'ct': 2}, {u'p_W': 0, u'q_VAR': 0, u'v_V': 0.0, u'ct': 3}, {u'p_W': 0, u'q_VAR': 0, u'v_V': 122.432, u'ct': 4}], u'timestamp': u'2016-12-24T15:03:42Z'}
Data retreived using the following code:
for keys,values in pvdata.items():
print(keys)
print(values)
channels
[{u'eExp_Ws': 23, u'v_V': 122.843, u'ch': 1, u'eImp_Ws': 55555370977, u'q_VAR': 14, u'p_W': 230, u'type': u'PHASE_A_CONSUMPTION'}, {u'eExp_Ws': 23, u'v_V': 121.088, u'ch': 2, u'eImp_Ws': 62496733790, u'q_VAR': -3, u'p_W': 661, u'type': u'PHASE_B_CONSUMPTION'}, {u'eExp_Ws': 41, u'v_V': 121.965, u'ch': 3, u'eImp_Ws': 118052104817, u'q_VAR': 12, u'p_W': 890, u'type': u'CONSUMPTION'}]
sensorId
0x0000C47F51019B7D
cts
[{u'p_W': 230, u'q_VAR': 14, u'v_V': 122.843, u'ct': 1}, {u'p_W': 661, u'q_VAR': -3, u'v_V': 121.088, u'ct': 2}, {u'p_W': 0, u'q_VAR': 0, u'v_V': 0.0, u'ct': 3}, {u'p_W': 0, u'q_VAR': 0, u'v_V': 122.84, u'ct': 4}]
timestamp
2016-12-24T15:25:16Z
The "u" characters makes this unparsable in JSONlint or in subsequent lines of code in my program.
I've looked at default character encoding in my Python environment but that doesn't seem to lead anywhere. I'm looking for other ideas to investigate.
.json() is what parses the JSON. It's done. You have a python dictionary which you can use normally now (as you've already seen by iterating through it).
It is unparsable since the string representation of a dict simply isn't JSON; you need to remarshall the dict to JSON if that's what you need:
In [36]: d = {u'channels': [{u'eExp_Ws': 23, u'v_V': 122.434, u'ch': 1, u'eImp_Ws': 55554346060, u'q_VAR': 305, u'p_W': 1489, u'type': u'PHASE_A_CONSUMPTION'}, {u'eExp_Ws': 23, u'v_V': 120.981, u'ch': 2, u'eImp_Ws': 62495160471, u'q_VAR': 237, u'p_W': 1872, u'type': u'PHASE_B_CONSUMPTION'}, {u'eExp_Ws': 41, u'v_V': 121.708, u'ch': 3, u'eImp_Ws': 118049506582, u'q_VAR': 542, u'p_W': 3360, u'type': u'CONSUMPTION'}], u'sensorId': u'0x0000C47F51019B7D', u'cts': [{u'p_W': 1489, u'q_VAR': 305, u'v_V': 122.434, u'ct': 1}, {u'p_W': 1872, u'q_VAR': 237, u'v_V': 120.981, u'ct': 2}, {u'p_W': 0, u'q_VAR': 0, u'v_V': 0.0, u'ct': 3}, {u'p_W': 0, u'q_VAR': 0, u'v_V': 122.432, u'ct': 4}], u'timestamp': u'2016-12-24T15:03:42Z'}
In [38]: json.dumps(d)
Out[38]: '{"sensorId": "0x0000C47F51019B7D", "timestamp": "2016-12-24T15:03:42Z", "cts": [{"q_VAR": 305, "ct": 1, "v_V": 122.434, "p_W": 1489}, {"q_VAR": 237, "ct": 2, "v_V": 120.981, "p_W": 1872}, {"q_VAR": 0, "ct": 3, "v_V": 0.0, "p_W": 0}, {"q_VAR": 0, "ct": 4, "v_V": 122.432, "p_W": 0}], "channels": [{"q_VAR": 305, "type": "PHASE_A_CONSUMPTION", "p_W": 1489, "v_V": 122.434, "ch": 1, "eImp_Ws": 55554346060, "eExp_Ws": 23}, {"q_VAR": 237, "type": "PHASE_B_CONSUMPTION", "p_W":
1872, "v_V": 120.981, "ch": 2, "eImp_Ws": 62495160471, "eExp_Ws": 23}, {"q_VAR": 542, "type": "CONSUMPTION", "p_W": 3360, "v_V": 121.708, "ch": 3, "eImp_Ws": 118049506582, "eExp_Ws": 41}]}'