I've got an issue that pertains to how to use jupyter widgets, dropdowns namely, to produce a workflow. Currently my intentions aren't working, and I am aiming to do the following:
Run a function that produces a list
This list is fed into a dropdown, from which I select one (x)
x refers to another function, that has a dictionary, it picks up all values associated with this key, and produces another list
The list is fed into another dropdown, from where I'd pick one value for processing.
Issue that I am coming up with, is that I can get the first list produced and fed into a dropdown. However the subsequent list is not captured, and rather the function is, which of course fails down the road. Let me illustrate with some code:
This bit of code simply goes through a list of dictionaries, and places all the unique league instances into a list:
def league_names():
league_list = []
data_filenames = [data_file for data_file in os.listdir()
if data_file.endswith('.json')]
with open(data_filenames[0]) as json_file:
data = json.load(json_file)
for x in data:
if x['Competition'] is not None and x['Competition'] not in league_list:
league_list.append(x['Competition'])
return league_list[1:]
What the following will then do, is take that list, and search the same set of dictionaries, search for all the teams that are a part of that league, and add them to a list.
def team_names(league_select):
team_list = []
data_filenames = [data_file for data_file in os.listdir()
if data_file.endswith('.json')]
with open(data_filenames[0]) as json_file:
data = json.load(json_file)
for x in data:
if x['Competition'] == league_select and x['Team'] not in team_list:
team_list.append(x['Team'])
return team_list
How I want to interact with this, is that the first league list is passed to a dropdown, from which you pick a league. This passes the league to the second function, to pull all the teams. How this is done is with the following:
def league_interact():
choice = interact(team_names, league_select=league_names())
return type(choice)
league_interact()
Now this works, the list is successfully passed through, however what I simply cannot get to work, is for the interact from here to be transformed into a variable, that I can then pass to a subsequent function for further processing.
Below is an example of the json content:
[{"Team": "Yeovil Town FC", "Gender": "M", "Competition": "National League", "Earliest Season": "2003-2004", "Latest Season": "2020-2021", "Total Seasons": "18", "Championships": "1", "Other Names": "", "Code": "bd5179b9", "Prefix": "Yeovil-Town-Stats"},
{"Team": "Yeovil Town LFC", "Gender": "F", "Competition": "", "Earliest Season": "2017", "Latest Season": "2018-2019", "Total Seasons": "3", "Championships": "", "Other Names": "", "Code": "a506e4a2", "Prefix": "Yeovil-Town-Women-Stats"},
{"Team": "York City FC", "Gender": "M", "Competition": "", "Earliest Season": "2002-2003", "Latest Season": "2019-2020", "Total Seasons": "13", "Championships": "0", "Other Names": "", "Code": "e272e7a8", "Prefix": "York-City-Stats"},
{"Team": "Yorkshire Amateur AFC", "Gender": "M", "Competition": "", "Earliest Season": "2019-2020", "Latest Season": "2020-2021", "Total Seasons": "0", "Championships": "", "Other Names": "", "Code": "66379800", "Prefix": "Yorkshire-Amateur-AFC-Stats"}]
Question: How would I in the above case, use interact to produce the list created by the first choice, rather than a function? I have the type pulled here, where it is a 'function' rather than a list as expected. I tried using .value, and some derivatives, but none of them pushed out a value. Any idea how to approach this, so I can produce a secondary dropdown?
I've tried the following, but getting an error:
def league_interact():
choice = interact(team_names, league_select=league_names())
return choice
def team_interact():
choice2 = interact(team_code, team_select=league_interact())
team_interact()
Error: ValueError: <function team_names at 0x0000021359D20B80> cannot be transformed to a widget
Thanks! I did trawl through the documentation, but how to approach this didn't quite click with me.
Ok so I actually managed to figure this out:
#interact(league = league_box)
def choose_both(league):
team_box.options = team_names(league_box.value)
return
#interact_manual(team = team_box, use_season = season_box)
def choose_team(team, use_season):
return team_choice_cap(team_data(team),use_season)
def team_choice_cap(data_set, use_season):
code = data_set['Code']
prefix = data_set['Prefix']
return parse_seasons(code,prefix,use_season)
The above interact and interact_manual can be used to feed the latter list, that then works to pull up with a manual call the rest of the details.
Related
I have a list of countries and their cities on one website. I take all names of countries and their capitals from this list, and want to put them in JSON file like this:
[
{
"country": "Afghanistan",
"city": "Kabul"
},
{
"country": "Aland Islands",
"city": "Mariehamn"
}
]
there's my code:
cells = soup.table('td')
count = 0
cities_list.write('[\n')
for cell in cells:
if count == len(cells)-2:
break
else:
cities_list.write(json.dumps({"country": "{}".format(cells[count].getText()),
"city": "{}".format(cells[count].next_sibling.getText())},
indent=2))
count += 2
cities.list_write('\n]')
And my problem is that objects are not separated by comma:
[
{
"country": "Afghanistan",
"city": "Kabul"
}{
"country": "Aland Islands",
"city": "Mariehamn"
}
]
How can I make my objects separated by comma, and also is it possible to do without writing '\n]' at the end and beginning?
Python obviously has no way to know that you are writing a list of objects when you are writing them one at a time ... so just don't.
cells = soup.table('td')
cities = []
for cell in cells[:-2]:
cities.append({"country": str(cells[count].getText()),
"city": str(cells[count].next_sibling.getText())})
json.dump(cities, cities_list)
Notice also how "{}".format(value) is just a clumsy way to write str(value) (or just value if value is already a string) and how json.dump lets you pass an open file handle to write to.
i have a dictionary content as below
contentDict = {
"Prefecture": "",
"Total List Number": "",
"Title": "",
"Event Validity": "",
"Available Period": "",
"Available StartDate": "",
"End Date": "",
"Last Updated": "",
"mainText": "",
"Location Tag": "",
"Event Tag": "",
"Main Image URL": "",
"innerWebSiteURL": "",
"ListLink": ""
}
and a list of contents of the same length.
is there a way to assign each content of an list sequentially to the dictionary
i tried using zip() but failed miserably
The dictionary would need to be ordered in the sequence specified because i would like to add it to a mysql database in the order provided..
the end GOAL is to use to_sql() method to put the dictionary into the MySQL database
I initially coded without the MySQL in mind and later was told to use ORM to_SQL method to transfer the data..
any hints or guide would be appreciated
Well, assuming your list is contentList,
contentList = [i for i in range(14)]
one thing would be looping through keys:
for i,k in enumerate(contentDict):
contentDict[k] = contentList[i]
Or 'pythonically':
contentDict = {k:contentList[i] for i,k in enumerate(contentDict)}
I am a beginner to python and scripting so I am unfamiliar with how json innately works, but this is the problem I have. I wrote a script which took values of the "location" variable from the json file I was reading and used googlemaps API to find the country this location was in. However, as some of these locations are repeat, and I did not want to repeatedly check the same location over and over. I stored all the values retrieved from the location variable in a list, then converted the list into a set to get rid of duplicates.
My question is this: once I have retrieved the country data (I have the data stored in a list), how can I add this country data to my original json file?
For instance, these are a few of my tests from the json file I am reading.
{"login": "hi", "name": "hello", "location": "Seoul, South Korea"}
{"login": "hi", "name": "hello", "location": null}
{"login": "hi", "name": "hello", "location": "Berlin, Germany"}
{"login": "hi", "name": "hello", "location": "Pittsburgh, PA"}
{"login": "hi", "name": "hello", "location": "London"}
{"login": "hi", "name": "hello", "location": "Tokyo, Japan"}
input = codecs.open(inputFile, 'r', 'utf8')
for line in input.readlines():
temp = json.loads(line)
if (temp['location'] != None): #some locations are Null
locationList.append(temp['location'])
input.close()
locationList = list(set(locationList))
print(locationList)
#getting location data, storing it in countryList variable
for uniqueLocation in locationList:
geocodeResult = gm.geocode(uniqueLocation)[0] #getting information about each location
geoObject = geocodeResult['address_components'] #gettnig just the address components
for item in geoObject: #iterating in object
if item['types'][0] == 'country': #if first element of this item is country
countryName = item['long_name'] #then retrieve long_name from this item
countryList.append(countryName)
print(countryList)
Check this out:
How to append data to a json file?
I have an input JSON that looks like this:
> {"payment": {"payment_id": "AA340", "payment_amt": "20", "chk_nr": "321749", "clm_list": {"dtl": [{"clm_id": "1A2345", "name": "John", adj:{"adj_id":"W123","adj_cd":"45"}}, {"clm_id": "9999", "name": "Dilton", adj:{"adj_id":"X123","adj_cd":"5"}}]}}}
I need the output to look like this:
{"clm_id": "1A2345",adj:{"adj_id":"W123"},"payment_amt": "20", "chk_nr": "321749"}
{"clm_id": "9999"adj:{"adj_id":"X123"},"payment_amt": "20", "chk_nr": "321749"}
So the code takes in the one JSON doc, parses the claim array section and normalizes it by adding payment info to each section. Even the nested JSON is parsed.
I'm able to parse the data, but unsure on how to normalize only certain section of the data.
The code below will parse the data, but NOT normalize
keep = ["payment","payment_id","payment_amt", "clm_list", "dtl", "clm_id","adj","adj_id"]
old_dict={"payment": {"payment_id": "AA340", "payment_amt": "20", "chk_nr": "321749", "clm_list": {"dtl": [{"clm_id": "1A2345", "name": "John", "adj": {"adj_id": "W123", "adj_cd": "45"}}, {"clm_id": "9999", "name": "Dilton", "adj": {"adj_id": "X123", "adj_cd": "5"}}]}}}
def recursively_prune_dict_keys(obj, keep):
if isinstance(obj, dict):
return dict([(k, recursively_prune_dict_keys(v, keep)) for k, v in obj.items() if k in keep])
elif isinstance(obj, list):
return [recursively_prune_dict_keys(item, keep) for item in obj]
else:
return obj
new_dict = recursively_prune_dict_keys(old_dict, keep)
conv_json=new_dict["payment"]
print json.dumps(conv_json)
It may be the neat way is to simply pick through the data, like;
new_dict = recursively_prune_dict_keys(old_dict, keep)
payment = old_dict['payment']
claims = payment['clm_list']['dtl']
for claim in claims:
claim['payment_amt'] = payment['payment_amt']
claim['chk_nr'] = payment['chk_nr']
print(json.dumps(claims))
This will yield;
[{"chk_nr": "321749", "clm_id": "1A2345", "payment_amt": "20", "adj": {"adj_id": "W123"}}, {"chk_nr": "321749", "clm_id": "9999", "payment_amt": "20", "adj": {"adj_id": "X123"}}]
This contains the output you asked for, but not exactly as you may want to see it.
First, your desired output isn't correct JSON without the square brackets [] that would make it a list. But, we can get rid of that by dumping each claim individually;
new_dict = recursively_prune_dict_keys(old_dict, keep)
payment = old_dict['payment']
claims = payment['clm_list']['dtl']
for claim in claims:
claim['payment_amt'] = payment['payment_amt']
claim['chk_nr'] = payment['chk_nr']
print(json.dumps(claim))
This gives;
{"name": "John", "clm_id": "1A2345", "payment_amt": "20", "adj": {"adj_cd": "45", "adj_id": "W123"}, "chk_nr": "321749"}
{"name": "Dilton", "clm_id": "9999", "payment_amt": "20", "adj": {"adj_cd": "5", "adj_id": "X123"}, "chk_nr": "321749"}
This is close to your desired output, except maybe for the ordering. Python dicts are not inherently ordered. You can sort them, however. So, if the ordering is important, you will want to read through How to Sort Python Dictionaries by Key or Value
This question already has answers here:
How can I extract a single value from a nested data structure (such as from parsing JSON)?
(5 answers)
Closed 4 years ago.
I'm trying to get the zip code for a particular city using zippopotam.us. I have the following code which works, except when I try to access the post code key which returns TypeError: expected string or buffer
r = requests.get('http://api.zippopotam.us/us/ma/belmont')
j = r.json()
data = json.loads(j)
print j['state']
print data['places']['latitude']
Full JSON output:
{
"country abbreviation": "US",
"places": [
{
"place name": "Belmont",
"longitude": "-71.4594",
"post code": "02178",
"latitude": "42.4464"
},
{
"place name": "Belmont",
"longitude": "-71.2044",
"post code": "02478",
"latitude": "42.4128"
}
],
"country": "United States",
"place name": "Belmont",
"state": "Massachusetts",
"state abbreviation": "MA"
}
Places is a list and not a dictionary. This line below should therefore not work:
print(data['places']['latitude'])
You need to select one of the items in places and then you can list the place's properties. So to get the first post code you'd do:
print(data['places'][0]['post code'])
I did not realize that the first nested element is actually an array. The correct way access to the post code key is as follows:
r = requests.get('http://api.zippopotam.us/us/ma/belmont')
j = r.json()
print j['state']
print j['places'][1]['post code']
In your code j is Already json data and j['places'] is list not dict.
r = requests.get('http://api.zippopotam.us/us/ma/belmont')
j = r.json()
print j['state']
for each in j['places']:
print each['latitude']
I'm using this lib to access nested dict keys
https://github.com/mewwts/addict
import requests
from addict import Dict
r = requests.get('http://api.zippopotam.us/us/ma/belmont')
j = Dict(r.json())
print j.state
print j.places[1]['post code'] # only work with keys without '-', space, or starting with number