I was using the requests module to get some data in JSON form and I want to assign some of the output results into variables in the app; for example the results were like:
{'text': 'example',
'type': 'text'}
I wanted to create variables that automatically store text as example and type as text.
I tried to create a function and put the first code in it but it didn't work.
The code for it was:
import requests
import json
import pprint
def new_func():
url = '***'
r = requests.get(url)
data = r.json()
pprint.pprint(data)
print(data)
text = new_func.text()
print(text)
However, it gives me an error as text is not a member of new_func.
text was part of the output as I mentioned before.
You basically have what is called a dictionary in python.
A dictionary looks like this: dictionary = {key: value}
You can get the value of a key using dictionary.get(key)
For example, consider the code below:
def getValue(key):
data = {'text': 'some text here',
'type': 'some text here 2'}
return data.get(key)
your_value = getValue('type')
This function will return some text here 2 when we get the type from data
You don't even necessarily need a function for this. You can just have this:
data = {'text': 'some text here',
'type': 'some text here 2'}
your_value = data.get('type')
You should be able to apply this to your case.
Hope that helps.
You should take a look at the JSON module in Python. Below are some links that should help:
https://docs.python.org/3/library/json.html
https://www.w3schools.com/python/python_json.asp
You can try something like this, you can wrap any part into a function.
import requests
# get response
response = requests.get('https://api.github.com')
# parse response:
response_code = response.status_code
response_json = response.json()
# pack response:
packed_response = {
'text' : response_json,
'type' : 'text',
'code' : response_code,
}
More on the requests library here: https://realpython.com/python-requests/
Related
Why doesn't the script below return a photo url link? I try to modify the code but it has no effect.
import requests
import json
def get_wiki_main_image(title):
url = 'https://pl.wikipedia.org/wiki/Zamek_Kr%C3%B3lewski_na_Wawelu'
data = {
'action' :'query',
'format' : 'json',
'formatversion' : 2,
'prop' : 'pageimages|pageterms',
'piprop' : 'original',
'titles' : title
}
response = requests.get(url, data)
json_data = json.loads(response.text)
return json_data['query']['pages'][0]['original']['source'] if len(json_data['query']['pages']) >0 else 'Not found'
urllink = get_wiki_main_image('zamek królewski na wawelu')
print (urllink)
Thanks for help.
By observation, we notice that all the pictures in Wikipedia are in the folder https://upload.wikimedia.org/wikipedia/commons/thumb. If without using additional libraries:
import requests
r = requests.get('https://pl.wikipedia.org/wiki/Zamek_Kr%C3%B3lewski_na_Wawelu')
gen = r.iter_lines() # create a byte string generator
for s in gen:
# Is there such a substring, with the folder we need, in this line
if s.find(b'https://upload.wikimedia.org/wikipedia/commons/thumb') == -1:
continue
else:
ss = s.split(b'"') # split the byte string to separate the url
print(ss[3].decode('utf-8')) # take the url and convert it to a string
Console output:
https://upload.wikimedia.org/wikipedia/commons/thumb/e/ea/Royal_Castle%2C_Wawel_Hill%2C_4_Wawel%2C_Old_Town%2C_Krak%C3%B3w%2C_Poland.jpg/1200px-Royal_Castle%2C_Wawel_Hill%2C_4_Wawel%2C_Old_Town%2C_Krak%C3%B3w%2C_Poland.jpg
https://upload.wikimedia.org/wikipedia/commons/thumb/e/ea/Royal_Castle%2C_Wawel_Hill%2C_4_Wawel%2C_Old_Town%2C_Krak%C3%B3w%2C_Poland.jpg/800px-Royal_Castle%2C_Wawel_Hill%2C_4_Wawel%2C_Old_Town%2C_Krak%C3%B3w%2C_Poland.jpg
https://upload.wikimedia.org/wikipedia/commons/thumb/e/ea/Royal_Castle%2C_Wawel_Hill%2C_4_Wawel%2C_Old_Town%2C_Krak%C3%B3w%2C_Poland.jpg/640px-Royal_Castle%2C_Wawel_Hill%2C_4_Wawel%2C_Old_Town%2C_Krak%C3%B3w%2C_Poland.jpg
There are three static pictures on the site with different sizes.
I am currently trying to read out the locations of a company. The information about the locations is inside a script tag (json). So I read out the contet inside the corresponding script tag.
This is my code:
sauce = requests.get('https://www.ep.de/store-finder', verify=False, headers = {'User-Agent':'Mozilla/5.0'})
soup1 = BeautifulSoup(sauce.text, features="html.parser")
all_scripts = soup1.find_all('script')[6]
all_scripts.contents
The output is:
['\n\t\twindow.storeFinderComponent = {"center":{"lat":51.165691,"long":10.451526},"bounds":[[55.655085,5.160441],[46.439648,15.666775]],"stores":[{"code":"1238240","lat":51.411572,"long":10.425264,"name":"EP:Schulze","url":"/schulze-breitenworbis","showAsClosed":false,"isBusinessCard":false,"logoUrl":"https://cdn.prod.team-ec.com/logo/retailer/retailerlogo_epde_1238240.png","address":{"street":"Weststraße 6","zip":"37339","town":"Breitenworbis","phone":"+49 (36074) 31193"},"email":"info#ep-schulze-breitenworbis.de","openingHours":[{"day":"Mo.","openingTime":"09:00","closingTime":"18:00","startPauseTime":"13:00","endPauseTime":"14:30"},{"day":"Di.","openingTime":"09:00","closingTime":"18:00","startPauseTime":"13:00","endPauseTime":"14:30"},{"day":"Mi.","openingTime":"09:00","closingTime":"18:00","startPauseTime":"13:00","endPauseTime":"14:30"},...]
I have problems converting the content to a dictionary and reading all lat and long data.
When I try:
data = json.loads(all_scripts.get_text())
all_scripts.get_text() returns an empty list
So i tryed:
data = json.loads(all_scripts.contents)
But then i get an TypeError: the JSON object must be str, bytes or bytearray, not list
I dont know ho to convert the .content method to json:
data = json.loads(str(all_scripts.contents))
JSONDecodeError: Expecting value: line 1 column 2 (char 1)
Can anyone help me?
You could use regex to pull out the json and read that in.
import requests
import re
import json
html = requests.get('https://www.ep.de/store-finder', verify=False, headers = {'User-Agent':'Mozilla/5.0'}).text
pattern = re.compile('window\.storeFinderComponent = ({.*})')
result = pattern.search(html).groups(1)[0]
jsonData = json.loads(result)
You can removed first part of data and then last character of data and then load data to json
import json
data=all_scripts.contents[0]
removed_data=data.replace("\n\t\twindow.storeFinderComponent = ","")
clean_data=data[:-3]
json_data=json.loads(clean_data)
Output:
{'center': {'lat': 51.165691, 'long': 10.451526},
'bounds': [[55.655085, 5.160441], [46.439648, 15.666775]],
'stores': [{'code': '1238240',
'lat': 51.411572,
....
I was successful in extracting the data about matches I required in the first half of my code, but I can't seem the do the other part. I am reading JSON data and doing it in the same way really but I'm getting strings, not dictionaries with data. I'm sure it's a logic problem or something, please help me. I have the working part on my github : https://github.com/LEvinson2504/Football-Prediction-and-analysis
import urllib.request
import json
#Match odds
#!/usr/bin/python
# -*- coding: utf-8 -*-
import urllib.request
def SportDemo():
# Set url parameter
url = "http://api.isportsapi.com/sport/free/football/odds/main?api_key=" + api_key
# Call iSport Api to get data in json format
f = urllib.request.urlopen(url)
content = f.read()
#data = json.loads((content.decode('utf-8')))
data = content.decode('utf-8')
'''store match ids
matches = []
#English teams match id
for team in data['data']:
if (team == 'English Premier League'):
#store match ids
matches.append(team['matchId'])
'''
#here is the problem, tried several ways to access data
for i in data[data]:
print(i['asia'])
'''
for match in data[data]['asia']:
for coun in match:
print(coun)
'''
'''
if(match == 'asian'):
print(type(match))
'''
#if (match['leagueName'] == 'ENG U23 D1'):
#for odds in data['data']:
#for i in matches:
#print()
SportDemo()
Expected Output, and I want to read inside the dictionaries to get the data iside keys "europe", "asia"
Json data : https://www.isportsapi.com/docs?isportsDocIndex=1-4-24 like her, I'm sorry I couldn't format.
But i get nothing
Firstly, when asking a question please take the time to tidy it up so that it represents what you actually ran and remove any commented-out code.
In your case, the problem can be reduced to:
f = urllib.request.urlopen(url)
content = f.read()
data = json.loads((content.decode('utf-8')))
#here is the problem, tried several ways to access data
for i in data[data]:
print(i['asia'])
and we can actually see what the issue is. data is a dict; within that dict is a key 'data', which is itself a dict. Iterating through a dict gives you the keys. If you just want to access the 'asia' data, then do so, no need to loop at all:
print(data['data']['asia'])
If you did want to iterate through every item, then use items():
for region, matches in data['data'].items():
print(region)
print(matches)
The download data is too big. 6.2M
Change the jupyter notebook configuration file. (jupyter_notebook_config.py)
Edit ~/.jupyter/jupyter_notebook_config.py
If you cannot find the file,
$ jupyter notebook --generate-config
Open the file and edit.
c.NotebookApp.iopub_data_rate_limit = 10000000
and restart $ jupyter notebook.
url = "http://api.isportsapi.com/sport/free/football/odds/main?api_key=" + api_key
# Call iSport Api to get data in json format
f = urllib.request.urlopen(url)
content = f.read()
#print(content.decode('utf-8'))
data = json.loads((content.decode('utf-8')))
print( data['data']['asian'])
# there is no 'asia' field in that content.
Output is
[{'matchId': '4196461', 'companyId': '1', 'initialHandicap': '-0.25', 'initialHome': '0.78', 'initialAway': '1.02', 'instantHandicap': '-0.25', 'instantHome': '0.78', 'instantAway': '1.02', 'modifyTime': 1567434821, 'close': False, 'inPlay': False}, {'matchId': '4196461', 'companyId': '3', 'initialHandicap': '-0.25', 'initialHome': '0.91', 'initialAway': '0.91', 'instantHandicap': '-0.25', 'instantHome': '0.81', 'instantAway': '1.09', 'modifyTime': 1567709243, 'close': False, 'inPlay': True}, {'matchId': '4196461', 'companyId': '8', 'initialHandicap': '-0.25', 'initialHome': '0.85', 'initialAway': '1.00', 'instantHandicap': '-0.25', 'instantHome': '0.80',
...
I'm pretty lost. Not going to lie. I'm trying to figure out how to parse JSON data from the college scorecard API into an HTML file. I used Python to store the JSON data in a dictionary, but other than that, I'm pretty dang lost. How would you write an example sending this data to an HTML file?
def main():
url = 'https://api.data.gov/ed/collegescorecard/v1/schools.json'
payload = {
'api_key': "api_key_string",
'_fields': ','.join([
'school.name',
'school.school_url',
'school.city',
'school.state',
'school.zip',
'2015.student.size',
]),
'school.operating': '1',
'2015.academics.program_available.assoc_or_bachelors': 'true',
'2015.student.size__range': '1..',
'school.degrees_awarded.predominant__range': '1..3',
'school.degrees_awarded.highest__range': '2..4',
'id': '240444',
}
data = requests.get(url, params=payload).json()
for result in data['results']:
print result
main()
Output:
{u'school.city': u'Madison', u'school.school_url': u'www.wisc.edu', u
'school.zip': u'53706-1380', u'2015.student.size': 29579, u'school.st
ate': u'WI', u'school.name': u'University of Wisconsin-Madison'}
Edit: For clarification, I need to insert the return data to an HTML file that formats and removes data styling and places it onto a table.
Edit II: Json2html edit
data = requests.get(url, params=payload).json()
for result in data['results']:
print result
data_processed = json.loads(data)
formatted_table = json2html.convert(json = data_processed)
index= open("index.html","w")
index.write(formatted_table)
index.close()
Edit: Json2html output:
Output image here
Try using the json2html module! This will convert the JSON that was returned into a 'human readable HTML Table representation'.
This code will take your JSON output and create the HTML:
data_processed = json.loads(data)
formatted_table = json2html.convert(json = data_processed)
Then to save it as HTML you can do this:
your_file= open("filename","w")
your_file.write(formatted_table)
your_file.close()
I have a json string and I am trying to print each section (id, name, link, etc) by using labels on tkinter GUI window.
Data:
{"id":"123456789","name":"John Smith","first_name":"John","last_name":"Smith","link":"http:\/\/www.facebook.com\/john.smith","username":"john.smith","gender":"male","locale":"en_GB"}
Code:
URL = https://graph.facebook.com/ + user
info = urlopen(info).read()
json_format = infor.decode("utf-8")
My problem is how do I assign each section of the json data to a variable do it can be printed out on a tkinter label?
thanks in advance
EDIT
Tried this code:
jsonData = json.loads(json_format)
u_name = jsoninfo['username']
and got the following error message
TypeError: string indices must be integers
You want to use the json standard module:
>>> import json
>>> data = '{"id":"123456789","name":"John Smith","first_name":"John","last_name":"Smith","link":"http:\/\/www.facebook.com\/john.smith","username":"john.smith","gender":"male","locale":"en_GB"}'
>>> d = json.loads(data)
This gives you your data as a regular dictionary to use:
>>> d
{u'username': u'john.smith', u'first_name': u'John', u'last_name': u'Smith', u'name': u'John Smith', u'locale': u'en_GB', u'gender': u'male', u'link': u'http://www.facebook.com/john.smith', u'id': u'123456789'}
>>> d['username']
u'john.smith'
try:
import simplejson as json
except ImportError:
import json
json_data = json.dumps(info)
# info here is json string or your variable json_format
You need to import the json library - it is included in the standard library, and load the json. This will convert the json string to a Python dictionary which you can use.
import json
py_dict= json.loads(json_string)
# work away