I can't seems to wrap my head around this silly issue. There are API requests that run simultaneously 23 different section stored in a dictionary :
polygonsDict = {
'Sect1':'100,6.3',
'Sect2':'100,6.0',
'Sect3':'100,5.5' # and more sections
}
urlDict = {
'traffic': 'https://google.com'
}
Here is the code where I iteratively :
section_key = list(polygonsDict.keys()) #convert the dictionary key to list for iteration
for idx, section in enumerate(section_key):
traffics(section ,urlDict['traffic']+polygonsDict[section]).getPolygonTraffics() #This line is constructing the link for request.
Then, the link is send to a class called traffics with getPolygonTraffics function :
class traffics:
def __init__(self, name, traffics):
self.traffics = traffics
self.name = name
def getPolygonTraffics(self):
try :
print("TRF: Running for "+self.name+"...")
raw = req.get(self.traffics).json()
traffics_raw = [raw] #wrap the dict to list
traffics_ls = []
for ls in traffics_raw:
traffics_ls.append(ls)
#Melt the list of dictionary to form a DataFrame
traffics_df = pd.DataFrame(traffics_ls).explode('jams').reset_index(drop=True)
print(traffics_df)
#exception when `jams` is not found in the list of dict.
except KeyError:
print("Skip!")
In getPolygonTraffics, I want to append every traffics_raw (the json requests) to one individual list and eventually, explode them to a dataFrame. How can I achieve this? I'm not very sure how to explain this.
Current output is multiple lists of each dictionary :
[{}]
[{}]
WHat I want is : [{},{},{}]. Then explode to DataFrame.
Related
I'm making a script with Python to search for competitors with a Google API.
Just for you to see how it works:
First I make a request and save data inside a Json:
# make the http GET request to Scale SERP
api_result = requests.get('https://api.scaleserp.com/search', params)
# Save data inside Json
dados = api_result.json()
Then a create some lists to get position, title, domain and things like that, then I create a loop for to append the position from my competitors inside my lists:
# Create the lists
sPositions = []
sDomains = []
sUrls = []
sTitles = []
sDescription = []
sType = []
# Create loop for to look for information about competitors
for sCompetitors in dados['organic_results']:
sPositions.append(sCompetitors['position'])
sDomains.append(sCompetitors['domain'])
sUrls.append(sCompetitors['link'])
sTitles.append(sCompetitors['title'])
sDescription.append(sCompetitors['snippet'])
sType.append(sCompetitors['type'])
The problem is that not every bracket of my Json is going to have the same values. Some of them won't have the "domain" value. So I need something like "when there is no 'domain' value, append 'no domain' to sDomains list.
I'm glad if anyone could help.
Thanks!!
you should use the get method for dicts so you can set a default value incase the key doesn't exist:
for sCompetitors in dados['organic_results']:
sPositions.append(sCompetitors.get('position', 'no position'))
sDomains.append(sCompetitors.get('domain', 'no domain'))
sUrls.append(sCompetitors.get('link', 'no link'))
sTitles.append(sCompetitors.get('title', 'no title'))
sDescription.append(sCompetitors.get('snippet', 'no snippet'))
sType.append(sCompetitors.get('type', 'no type'))
I am trying to parse a "complicated" JSON string that is returned to me by an API.
It looks like this:
{
"data":[
["Distance to last strike","23.0","miles"],
["Time of last strike","1/14/2022 9:23:42 AM",""],
["Number of strikes today","1",""]
]
}
While the end goal will be to extract the distance, date/time, as well as count, for right now I am just trying to successfully get the distance.
My python script is:
import requests
import json
response_API = requests.get('http://localhost:8998/api/extra/lightning.json')
data = response_API.text
parse_json = json.loads(data)
value = parse_json['Distance to last strike']
print(value)
This does not work. If I change the value line to
value = parse_json['data']
then the entire string I listed above is returned.
I am hoping it's just a simple formatting issue. Suggestions?
You have an object with a list of lists. If you fetch
value = parse_json['data']
Then you will have a list containing three lists. So:
print(value[0][1])
will print "23.0".
I'm querying a REST API url & I'm trying to return all the dictionary sets, but only able to return one key pair.
Dictionary Output in the print statement inside for loop is the expected output, when when returned only one set of key pair is appearing.
Expected Dictionary looks like:
{'IncidentID': 'IM10265'}
{'IncidentID': 'IM10266'}
{'IncidentID': 'IM10267'}
{'IncidentID': 'IM10268'}
Code:
import json , requests
sm1 = requests.get('http://Rest Url', auth=('XX','YY'))
z = json.loads(sm1.text)
def get_im_list():
incidentlist_access = z['content']
for im_data in incidentlist_access:
Access_imslist = im_data['Incident']
print(Access_imslist)
#print(type(Access_imslist))
#return Access_imslist
data = get_im_list()
#print(data)
So when when I'm un-commentating
return Access_imslist & print(data)
I'm only receiving the output as:
{'IncidentID': 'IM10265'}
not the complete dictionary.
Every time you loop through the data, Access_imslist gets overwritten, so when you (presumably) return Access_Imlist it's only returning the last value.
You need to create a data structure outside of the for loop, add each bit of data to it, then return that instead. Something like:
def get_im_list():
incident_data = []
incidentlist_access = z['content']
for im_data in incidentlist_access:
incident_data.append(im_data['Incident'])
return incident_data
hope that helps!
you need to define a variable list , and then append the Access_imslist values to that variable.
like this :
data = []
def get_im_list():
incidentlist_access = z['content']
for im_data in incidentlist_access:
Access_imslist = im_data['Incident']
data.append(Access_imslist)
print(data)
I apologize if this is a very basic question, but I built multiple dictionaries using nested loops and I cannot return all the dictionaries, it only returns the first one. I know a part of this is because return is in the wrong indentation, but Python will only allow return data to be indented there. Is there any way to change this function so it returns all the dictionaries?
import urllib.request
import json
def get_url(url):
text = urllib.request.urlopen(url).read().decode()
return text
def display_json(text):
dictionary = json.loads(text)
for item in dictionary['items']:
for article in item['articles']:
line = (article['article'], article['views'])
data = dict([line])
return data
def function(data):
print(data)
def main():
url = "https://wikimedia.org/api/rest_v1/metrics/pageviews/top/en.wikiversity/all-access/2018/01/all-days"
text = get_url(url)
data = display_json(text)
function(data)
main()
Use dictionary comprehension
def display_json(text):
return {
item['article']:item['views'] for item in
json.loads(text)['items'][0]['articles']
}
The JSON is structured as follows:
There is no need to loop through items, b'coz it has only one element. We can just get the first element using json.loads(text)['items'][0]. Even if there is more than one item inside items, you can still manage it inside the comprehension.
use a generator
def display_json(text):
dictionary = json.loads(text)
for item in dictionary['items']:
for article in item['articles']:
line = (article['article'], article['views'])
data = dict([line])
yield data
data = list(display_json(text))
I have just made a program to parse some data from an api. The api gives data back with a JSON format. When I try to parse it it gives me a key error
url = json.loads(r.text)["url"]
KeyError: 'url'
This is the part of the code
url = json.loads(r.text)["url"]
I am trying to get the data in the plain field. Here is the output from the API:
{"updates":[{"id":"a6aa-8bd","description":"Bug fixes and enhancemets","version":"8.1.30","type":"firmware","url":"https://con-man.company.com/api/v1/file-732e844b","updated":"2017-07-25"}]}
You cannot access url since it is inside update (list), therefore you need to Pass index and then key :
One liner:
>>> url = json.loads(r.text)['updates'][0]['url']
'https://con-man.company.com/api/v1/file-732e844b'
Explicit
>>> jobj = json.loads(r.text)
>>> url = jobj['updates'][0]['url']
'https://con-man.company.com/api/v1/file-732e844b'
try this,
url = json.loads(r.text)["updates"][0]["url"]
{
"updates": [
{
"id":"a6aa-8bd",
"description":"Bug fixes and enhancemets",
"version":"8.1.30",
"type":"firmware",
"url":"https://con-man.company.com/api/v1/file-732e844b",
"updated":"2017-07-25"
}
]
}
Try to visualize of your dict, it has only one key "update" in that key value it has another list and into that list, you has another dict
so if in your case
_dict = json.loads(r.text) # read file and load dict
_list = _dict['updates'] # read list inside dict
_dict_1 = _list[0] # read list first value and load dict
url = _dict_1['url'] # read 'url' key from dict
I used this and works now for me.
json_object = json.loads(response.content.decode("utf-8"))['list'][0]['localPercentDynamicObjectsUsed']