This question already has answers here:
How can I parse a dictionary string?
(2 answers)
Closed 2 years ago.
I am writing code to obtain geolocation information using the IP Geolocation API. I used the following code in my jupyter notebook...
try:
from urllib.request import urlopen
except ImportError:
from urllib2 import urlopen
ip = '8.8.8.8'
api_key = 'your_api_key'
api_url = 'https://geo.ipify.org/api/v1?'
url = api_url + 'apiKey=' + api_key + '&ipAddress=' + ip
result = urlopen(url).read().decode('utf8')
print(result)
I got the following result, but it returns the following string...
'{"ip":"8.8.8.8","location":{"country":"US","region":"California","city":"Mountain View","lat":37.4223,"lng":-122.085,"postalCode":"94043","timezone":"-07:00","geonameId":5375480},"domains":["0--9.ru","000.lyxhwy.xyz","000180.top","00049ok.com","001998.com.he2.aqb.so"],"as":{"asn":15169,"name":"Google LLC","route":"8.8.8.0\\/24","domain":"https:\\/\\/about.google\\/intl\\/en\\/","type":"Content"},"isp":"Google LLC"}'
I am trying to remove the strings at the beginning and end. I tried changing this string to a list by calling the list function on the result variable but that didn't work. I would like to obtain the following output...
{"ip":"8.8.8.8","location":{"country":"US","region":"California","city":"Mountain View","lat":37.4223,"lng":-122.085,"postalCode":"94043","timezone":"-07:00","geonameId":5375480},"domains":["0--9.ru","000.lyxhwy.xyz","000180.top","00049ok.com","001998.com.he2.aqb.so"],"as":{"asn":15169,"name":"Google LLC","route":"8.8.8.0\\/24","domain":"https:\\/\\/about.google\\/intl\\/en\\/","type":"Content"},"isp":"Google LLC"}
By doing this, I will get a dictionary which I can work with using the various keys. Any help will be greatly appreciated.
You can use json library.
import json
str = '{"ip": "8.8.8.8"}'
res = json.loads(str)
print(res)
Result will be a dict.
If you want a dictionary, you shouldn't try to remove the quotes, that is there because it denotes a string variable. You should instead use JSON as this is a valid JSON string:
import json
try:
from urllib.request import urlopen
except ImportError:
from urllib2 import urlopen
ip = '8.8.8.8'
api_key = 'your_api_key'
api_url = 'https://geo.ipify.org/api/v1?'
url = api_url + 'apiKey=' + api_key + '&ipAddress=' + ip
result = json.loads(urlopen(url).read().decode('utf8'))
print(result)
This will give you the dictionary you are looking for.
Related
I need to take only one value from an XML/JSON file I read into a value.
https://whoisapi.whoisxmlapi.com/whois/google.com?format=XML
here is my code example :
import json
try:
from urllib.request import urlopen
except ImportError:
from urllib2 import urlopen
domainName = 'google.com';
apiKey = 'YourApiKey'
url = 'https://www.whoisxmlapi.com/whoisserver/WhoisService?'\
+ 'domainName=' + domainName + '&apiKey=' + apiKey + "&outputFormat=JSON"
testing=urlopen(url).read().decode('utf8')
test = json.loads(whoisdb)
registrar = test['WhoisRecord']['registrarName']
domain['whois-created'] = test['WhoisRecord']['createdDate']
And if I am using the value "registrar" later, I get a "Keyerror".
Unfortunaly neither of the other Questions helped me...
Edit : https://i.stack.imgur.com/FMBjW.png ,well he gets the value in the variable, but I still get this error.(I printed the error while running)
This question already has answers here:
How to use urllib with username/password authentication in python 3?
(3 answers)
Closed 4 years ago.
I'm trying to modify a code which was written in Python 2 Language with the urllib2 module.I did modified my code with the module urllib in Python 3 but I'm getting error :
req = urllib.request(url)
TypeError: 'module' object is not callable
What I am doing wrong here?
import urllib.request
import json
import datetime
import csv
import time
app_id = "172"
app_secret = "ce3"
def testFacebookPageData(page_id, access_token):
# construct the URL string
base = "https://graph.facebook.com/v2.4"
node = "/" + page_id
parameters = "/?access_token=%s" % access_token
url = base + node + parameters
# retrieve data
req = urllib.request(url)
response = urllib.urlopen(req)
data = json.loads(response.read())
print (json.dumps(data, indent=4, sort_keys=True))
Change the lines
req = urllib.request(url)
response = urllib.urlopen(req)
to:
req = urllib.request.Request(url)
response = urllib.request.urlopen(req)
You can find more information on this module **https://docs.python.org/3/library/urllib.request.html#urllib.request.Request
**https://docs.python.org/3/library/urllib.request.html#urllib.request.urlopen
#kvmahesh's answer is absolutely correct. I'll just provide an alternate solution which supports both the versions. Use Python's requests library for making the call.
import requests
import json
import datetime
import csv
import time
app_id = "172"
app_secret = "ce3"
def testFacebookPageData(page_id, access_token):
# construct the URL string
base = "https://graph.facebook.com/v2.4"
node = "/" + page_id
parameters = "/?access_token=%s" % access_token
url = base + node + parameters
# retrieve data
response = requests.get(url)
data = json.loads(response.text())
print (json.dumps(data, indent=4, sort_keys=True))
For detailed usage of requests: Python Requests Docs
urllib.request is a module. You are calling the module on line 22...
req = urllib.request(url)
To fix do following:
1) Import at the top:
from urllib.request import urlopen
2) then pass the url to urlopen(url)
# remove this line req = urllib.request(url)
response = urlopen(url)
data = json.loads(response.read())
3) See similar error here
TypeError: 'module' object is not callable
I am trying to GET a URL using Python and the response is JSON. However, when I run
import urllib2
response = urllib2.urlopen('https://api.instagram.com/v1/tags/pizza/media/XXXXXX')
html=response.read()
print html
The html is of type str and I am expecting a JSON. Is there any way I can capture the response as JSON or a python dictionary instead of a str.
If the URL is returning valid JSON-encoded data, use the json library to decode that:
import urllib2
import json
response = urllib2.urlopen('https://api.instagram.com/v1/tags/pizza/media/XXXXXX')
data = json.load(response)
print data
import json
import urllib
url = 'http://example.com/file.json'
r = urllib.request.urlopen(url)
data = json.loads(r.read().decode(r.info().get_param('charset') or 'utf-8'))
print(data)
urllib, for Python 3.4
HTTPMessage, returned by r.info()
"""
Return JSON to webpage
Adding to wonderful answer by #Sanal
For Django 3.4
Adding a working url that returns a json (Source: http://www.jsontest.com/#echo)
"""
import json
import urllib
url = 'http://echo.jsontest.com/insert-key-here/insert-value-here/key/value'
respons = urllib.request.urlopen(url)
data = json.loads(respons.read().decode(respons.info().get_param('charset') or 'utf-8'))
return HttpResponse(json.dumps(data), content_type="application/json")
Be careful about the validation and etc, but the straight solution is this:
import json
the_dict = json.load(response)
resource_url = 'http://localhost:8080/service/'
response = json.loads(urllib2.urlopen(resource_url).read())
Python 3 standard library one-liner:
load(urlopen(url))
# imports (place these above the code before running it)
from json import load
from urllib.request import urlopen
url = 'https://jsonplaceholder.typicode.com/todos/1'
you can also get json by using requests as below:
import requests
r = requests.get('http://yoursite.com/your-json-pfile.json')
json_response = r.json()
Though I guess it has already answered I would like to add my little bit in this
import json
import urllib2
class Website(object):
def __init__(self,name):
self.name = name
def dump(self):
self.data= urllib2.urlopen(self.name)
return self.data
def convJSON(self):
data= json.load(self.dump())
print data
domain = Website("https://example.com")
domain.convJSON()
Note : object passed to json.load() should support .read() , therefore urllib2.urlopen(self.name).read() would not work .
Doamin passed should be provided with protocol in this case http
This is another simpler solution to your question
pd.read_json(data)
where data is the str output from the following code
response = urlopen("https://data.nasa.gov/resource/y77d-th95.json")
json_data = response.read().decode('utf-8', 'replace')
None of the provided examples on here worked for me. They were either for Python 2 (uurllib2) or those for Python 3 return the error "ImportError: No module named request". I google the error message and it apparently requires me to install a the module - which is obviously unacceptable for such a simple task.
This code worked for me:
import json,urllib
data = urllib.urlopen("https://api.github.com/users?since=0").read()
d = json.loads(data)
print (d)
I am trying to GET a URL using Python and the response is JSON. However, when I run
import urllib2
response = urllib2.urlopen('https://api.instagram.com/v1/tags/pizza/media/XXXXXX')
html=response.read()
print html
The html is of type str and I am expecting a JSON. Is there any way I can capture the response as JSON or a python dictionary instead of a str.
If the URL is returning valid JSON-encoded data, use the json library to decode that:
import urllib2
import json
response = urllib2.urlopen('https://api.instagram.com/v1/tags/pizza/media/XXXXXX')
data = json.load(response)
print data
import json
import urllib
url = 'http://example.com/file.json'
r = urllib.request.urlopen(url)
data = json.loads(r.read().decode(r.info().get_param('charset') or 'utf-8'))
print(data)
urllib, for Python 3.4
HTTPMessage, returned by r.info()
"""
Return JSON to webpage
Adding to wonderful answer by #Sanal
For Django 3.4
Adding a working url that returns a json (Source: http://www.jsontest.com/#echo)
"""
import json
import urllib
url = 'http://echo.jsontest.com/insert-key-here/insert-value-here/key/value'
respons = urllib.request.urlopen(url)
data = json.loads(respons.read().decode(respons.info().get_param('charset') or 'utf-8'))
return HttpResponse(json.dumps(data), content_type="application/json")
Be careful about the validation and etc, but the straight solution is this:
import json
the_dict = json.load(response)
resource_url = 'http://localhost:8080/service/'
response = json.loads(urllib2.urlopen(resource_url).read())
Python 3 standard library one-liner:
load(urlopen(url))
# imports (place these above the code before running it)
from json import load
from urllib.request import urlopen
url = 'https://jsonplaceholder.typicode.com/todos/1'
you can also get json by using requests as below:
import requests
r = requests.get('http://yoursite.com/your-json-pfile.json')
json_response = r.json()
Though I guess it has already answered I would like to add my little bit in this
import json
import urllib2
class Website(object):
def __init__(self,name):
self.name = name
def dump(self):
self.data= urllib2.urlopen(self.name)
return self.data
def convJSON(self):
data= json.load(self.dump())
print data
domain = Website("https://example.com")
domain.convJSON()
Note : object passed to json.load() should support .read() , therefore urllib2.urlopen(self.name).read() would not work .
Doamin passed should be provided with protocol in this case http
This is another simpler solution to your question
pd.read_json(data)
where data is the str output from the following code
response = urlopen("https://data.nasa.gov/resource/y77d-th95.json")
json_data = response.read().decode('utf-8', 'replace')
None of the provided examples on here worked for me. They were either for Python 2 (uurllib2) or those for Python 3 return the error "ImportError: No module named request". I google the error message and it apparently requires me to install a the module - which is obviously unacceptable for such a simple task.
This code worked for me:
import json,urllib
data = urllib.urlopen("https://api.github.com/users?since=0").read()
d = json.loads(data)
print (d)
This question already has answers here:
How can I parse (read) and use JSON?
(5 answers)
What are the differences between the urllib, urllib2, urllib3 and requests module?
(11 answers)
Closed last month.
I want to dynamically query Google Maps through the Google Directions API. As an example, this request calculates the route from Chicago, IL to Los Angeles, CA via two waypoints in Joplin, MO and Oklahoma City, OK:
http://maps.googleapis.com/maps/api/directions/json?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false
It returns a result in the JSON format.
How can I do this in Python? I want to send such a request, receive the result and parse it.
I recommend using the awesome requests library:
import requests
url = 'http://maps.googleapis.com/maps/api/directions/json'
params = dict(
origin='Chicago,IL',
destination='Los+Angeles,CA',
waypoints='Joplin,MO|Oklahoma+City,OK',
sensor='false'
)
resp = requests.get(url=url, params=params)
data = resp.json() # Check the JSON Response Content documentation below
JSON Response Content: https://requests.readthedocs.io/en/master/user/quickstart/#json-response-content
The requests Python module takes care of both retrieving JSON data and decoding it, due to its builtin JSON decoder. Here is an example taken from the module's documentation:
>>> import requests
>>> r = requests.get('https://github.com/timeline.json')
>>> r.json()
[{u'repository': {u'open_issues': 0, u'url': 'https://github.com/...
So there is no use of having to use some separate module for decoding JSON.
requests has built-in .json() method
import requests
requests.get(url).json()
import urllib
import json
url = 'http://maps.googleapis.com/maps/api/directions/json?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false'
result = json.load(urllib.urlopen(url))
Use the requests library, pretty print the results so you can better locate the keys/values you want to extract, and then use nested for loops to parse the data. In the example I extract step by step driving directions.
import json, requests, pprint
url = 'http://maps.googleapis.com/maps/api/directions/json?'
params = dict(
origin='Chicago,IL',
destination='Los+Angeles,CA',
waypoints='Joplin,MO|Oklahoma+City,OK',
sensor='false'
)
data = requests.get(url=url, params=params)
binary = data.content
output = json.loads(binary)
# test to see if the request was valid
#print output['status']
# output all of the results
#pprint.pprint(output)
# step-by-step directions
for route in output['routes']:
for leg in route['legs']:
for step in leg['steps']:
print step['html_instructions']
just import requests and use from json() method :
source = requests.get("url").json()
print(source)
OR you can use this :
import json,urllib.request
data = urllib.request.urlopen("url").read()
output = json.loads(data)
print (output)
Try this:
import requests
import json
# Goole Maps API.
link = 'http://maps.googleapis.com/maps/api/directions/json?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false'
# Request data from link as 'str'
data = requests.get(link).text
# convert 'str' to Json
data = json.loads(data)
# Now you can access Json
for i in data['routes'][0]['legs'][0]['steps']:
lattitude = i['start_location']['lat']
longitude = i['start_location']['lng']
print('{}, {}'.format(lattitude, longitude))
Also for pretty Json on console:
json.dumps(response.json(), indent=2)
possible to use dumps with indent. (Please import json)