Develop Reference

JSONDecodeError: Expecting value: line 1 column 1 (char 0) / While json parameter include

I'm trying to retrieve data from https://clinicaltrials.gov/ and althought I've specified the format as Json in the request parameter:
fmt=json
the returned value is txt by default.
As a consequence i'm not able to retrieve the response in json()
Good:
import requests
response = requests.get('https://clinicaltrials.gov/api/query/study_fields?expr=heart+attack&fields=NCTId%2CBriefTitle%2CCondition&min_rnk=1&max_rnk=&fmt=json')
response.text
Not Good:
import requests
response = requests.get('https://clinicaltrials.gov/api/query/study_fields?expr=heart+attack&fields=NCTId%2CBriefTitle%2CCondition&min_rnk=1&max_rnk=&fmt=json')
response.json()
Any idea how to turn this txt to json ?
I've tried with response.text which is working but I want to retrieve data in Json()

You can use following code snippet:
import requests, json
response = requests.get('https://clinicaltrials.gov/api/query/study_fields?expr=heart+attack&fields=NCTId%2CBriefTitle%2CCondition&min_rnk=1&max_rnk=&fmt=json')
jsonResponse = json.loads(response.content)

You should use the JSON package (that is built-in python, so you don't need to install anything), that will convert the text into a python object (dictionary) using the json.loads() function. Here you can find some examples.

Finding specific string and printing from collected data of a site using python

Here, i am trying to do that, i have a data from ngrok tunnel (http://127.0.0.1//api/tunnels) and i want to print only 'public_url' : 'https://.....ngrok.io' which i have collected from that site, that data looks like this
{'tunnels': [{'name': 'command_line', 'uri': '/api/tunnels/command_line', 'public_url': 'https://a28e4c77.ngrok.io', 'proto': 'https', 'config': {'addr': 'http://localhost:80', 'inspect': True}....Something more
This is the part of that data.
I have use this code to collect that data.
import requests
url = "http://127.0.0.1:4040/api/tunnels"
r = requests.get(url)
data = r.json()
I have also save this into a ngrok.txt but i have absolutely no idea to find...To write this data i use this code : -
import requests
url = "http://127.0.0.1:4040/api/tunnels"
r = requests.get(url)
data = r.json()
f = open('ngrok.txt', 'w')
f.write(data)
f.close()

You need to convert your json string to a json object. You can do it with function loads() from json library.
Here the code for your example:
import json
json.loads(data)["tunnels"][0]["public_url"]
json.loads(data) converts a string to a json object
["tunnels"] gets the object associated with the name "tunnels"
The resulting object is a list, indeed you need to get the first element with [0]
Finally you get "public_url"

How to read .dat file directly from URL and access columns in it?

I am trying to access this file from URL:
https://data.princeton.edu/wws509/datasets/copen.dat
However, I am unable to access it and split it for training and testing purpose.
Does someone have a solution for this?
Thanks
I have run the following code which converted the data into html. Now how can I access the data eg. if a want to access certain rows and columns, how would I do that?
import urllib.request
weburl=urllib.request.urlopen('https://data.princeton.edu/wws509/datasets/cuse.dat')
print('result code:'+ str(weburl.getcode()))
data=weburl.read()
print(data)

To do this you need to install requests module in python.requests module
As #nekomatic suggests you can convert data to proper format by going through this link Getting list of lists into pandas DataFrame
import requests
response = requests.get('https://data.princeton.edu/wws509/datasets/copen.dat')
data = response.text // you can use response.json() method in this line
print("data is ")
print(data)
// the url we mentioned given data in text/plain format so response.json() doesn't work
data_by_line = data.split('\n')
for i in range(0,len(data_by_line)):
data_by_line[i] = ' '.join(data_by_line[i].split())
data_by_line[i] = data_by_line[i].split(' ')
print(data_by_line[2][2]) // output will be "low". We have converted data to multidimensional list(data_by_line)

Can't extract JSON from an http request

I'm having problems getting data from an HTTP response. The format unfortunately comes back with '\n' attached to all the key/value pairs. JSON says it must be a str and not "bytes".
I have tried a number of fixes so my list of includes might look weird/redundant. Any suggestions would be appreciated.
#!/usr/bin/env python3
import urllib.request
from urllib.request import urlopen
import json
import requests
url = "http://finance.google.com/finance/info?client=ig&q=NASDAQ,AAPL"
response = urlopen(url)
content = response.read()
print(content)
data = json.loads(content)
info = data[0]
print(info)
#got this far - planning to extract "id:" "22144"

When it comes to making requests in Python, I personally like to use the requests library. I find it easier to use.
import json
import requests
r = requests.get('http://finance.google.com/finance/info?client=ig&q=NASDAQ,AAPL')
json_obj = json.loads(r.text[4:])
print(json_obj[0].get('id'))
The above solution prints: 22144
The response data had a couple unnecessary characters at the head, which is why I am only loading the relevant (json) portion of the response: r.text[4:]. This is the reason why you couldn't load it as json initially.

Bytes object has method decode() which converts bytes to string. Checking the response in the browser, seems there are some extra characters at the beginning of the string that needs to be removed (a line feed character, followed by two slashes: '\n//'). To skip the first three characters from the string returned by the decode() method we add [3:] after the method call.
data = json.loads(content.decode()[3:])
print(data[0]['id'])
The output is exactly what you expect:
22144

JSON says it must be a str and not "bytes".
Your content is "bytes", and you can do this as below.
data = json.loads(content.decode())

HTTP requests and JSON parsing in Python [duplicate]

This question already has answers here:
How can I parse (read) and use JSON?
(5 answers)
What are the differences between the urllib, urllib2, urllib3 and requests module?
(11 answers)
Closed last month.
I want to dynamically query Google Maps through the Google Directions API. As an example, this request calculates the route from Chicago, IL to Los Angeles, CA via two waypoints in Joplin, MO and Oklahoma City, OK:
http://maps.googleapis.com/maps/api/directions/json?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false
It returns a result in the JSON format.
How can I do this in Python? I want to send such a request, receive the result and parse it.

I recommend using the awesome requests library:
import requests
url = 'http://maps.googleapis.com/maps/api/directions/json'
params = dict(
origin='Chicago,IL',
destination='Los+Angeles,CA',
waypoints='Joplin,MO|Oklahoma+City,OK',
sensor='false'
)
resp = requests.get(url=url, params=params)
data = resp.json() # Check the JSON Response Content documentation below
JSON Response Content: https://requests.readthedocs.io/en/master/user/quickstart/#json-response-content

The requests Python module takes care of both retrieving JSON data and decoding it, due to its builtin JSON decoder. Here is an example taken from the module's documentation:
>>> import requests
>>> r = requests.get('https://github.com/timeline.json')
>>> r.json()
[{u'repository': {u'open_issues': 0, u'url': 'https://github.com/...
So there is no use of having to use some separate module for decoding JSON.

requests has built-in .json() method
import requests
requests.get(url).json()

import urllib
import json
url = 'http://maps.googleapis.com/maps/api/directions/json?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false'
result = json.load(urllib.urlopen(url))

Use the requests library, pretty print the results so you can better locate the keys/values you want to extract, and then use nested for loops to parse the data. In the example I extract step by step driving directions.
import json, requests, pprint
url = 'http://maps.googleapis.com/maps/api/directions/json?'
params = dict(
origin='Chicago,IL',
destination='Los+Angeles,CA',
waypoints='Joplin,MO|Oklahoma+City,OK',
sensor='false'
)
data = requests.get(url=url, params=params)
binary = data.content
output = json.loads(binary)
# test to see if the request was valid
#print output['status']
# output all of the results
#pprint.pprint(output)
# step-by-step directions
for route in output['routes']:
for leg in route['legs']:
for step in leg['steps']:
print step['html_instructions']

just import requests and use from json() method :
source = requests.get("url").json()
print(source)
OR you can use this :
import json,urllib.request
data = urllib.request.urlopen("url").read()
output = json.loads(data)
print (output)

Try this:
import requests
import json
# Goole Maps API.
link = 'http://maps.googleapis.com/maps/api/directions/json?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false'
# Request data from link as 'str'
data = requests.get(link).text
# convert 'str' to Json
data = json.loads(data)
# Now you can access Json
for i in data['routes'][0]['legs'][0]['steps']:
lattitude = i['start_location']['lat']
longitude = i['start_location']['lng']
print('{}, {}'.format(lattitude, longitude))

Also for pretty Json on console:
json.dumps(response.json(), indent=2)
possible to use dumps with indent. (Please import json)