I've never done any object oriented programming, only basic script writing.
I'm playing around with grequests
rs = (grequests.get('https://api.github.com/repositories?since='+str(page), auth=(login, password)) for page in pages)
blah = grequests.map(rs)
print type(blah[0])
The response is:
<class 'requests.models.Response'>
Normally I convert the response to text and then load it into json so I can parse it, but I can't do that with this response.
I understand the concept of classes but haven't used them or know really what to do with that response.
Is there a way I can convert it to json?
blah[0] in your case is a requests.models.Response class which, according to the source code and the documentation, has json() method that deserializes the JSON response into a Python object using json.loads():
print blah[0].json()
Response object can be converted to JSON in two ways.
Use the method .json()
blah[0].json()
OR
Convert to text and load as json.
json.loads(blah[0].text)
Related
I am putting a JSON response into a variable via requests.json() like this:
response = requests.get(some_url, params=some_params).json()
This however converts JSON's original " to Python's ', true to True, null to None.
This poses a problem when trying to save the response as text and the convert it back to JSON - sure, I can use .replace() for all conversions mentioned above, but even once I do that, I get other funny json decoder errors.
Is there any way in Python to get JSON response and keep original JavaScript format?
json() is the JSON decoder method. You are looking at a Python object, that is why it looks like Python.
Other formats are listed on the same page, starting from Response Content
.text: text - it has no separate link/paragraph, it is right under "Response Content"
.content: binary, as bytes
.json(): decoded JSON, as Python object
.raw: streamed bytes (so you can get parts of content as it comes)
You need .text for getting text, including JSON data.
You can get the raw text of your response with requests.get(some_url, params=some_params).text
It is the json method which converts to a Python friendly format.
I am writing a program (python Python 3.5.2) that uses a HTTPSConnection to get a JSON object as a response. I have it working using some example code, but am not sure where a method comes from.
My question is this: In the code below, the decode('utf-9') method doesn't exist in the documentation at https://docs.python.org/3.4/library/http.client.html#http.client.HTTPResponse under "21.12.2. HTTPResponse Objects". How would I know that the return value from the method "response.read()" has the method "decode('utf-8')" available?
Do Python objects inherit from a base class like C# objects do or am I missing something?
http = HTTPSConnection(get_hostname(token))
http.request('GET', uri_path, headers=get_authorization_header(token))
response = http.getresponse()
print(response.status, response.reason)
feed = json.loads(response.read().decode('utf-8'))
Thank you for your help.
The read method of the response object always returns a byte string (in Python 3, which I presume you are using as you use the print function). The byte string does indeed have a decode method, so there should be no problem with this code. Of course it makes the assumption that the response is encoded in UTF-8, which may or may not be correct.
[Technical note: email is a very difficult medium to handle: messages can be made up of different parts, each of which is differently encoded. At least with web traffic you stand a chance of reading the Content-Type header's charset attribute to find the correct encoding].
header = {'Content-type': 'application/json','Authorization': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' }
url = 'https://sandbox-authservice.priaid.ch/login'
response = requests.post(url, headers = header, verify=False).json()
token = json.dumps(response)
print token['ValidThrough']
I want to print the ValidThrough Attribute in my webhook, which is received as JSON data via a POST call. I know this has been asked a number of times here, but print token['ValidThrough']isnt working for me.I receive the error "TypeError: string indices must be integers, not str"
Since the response already seems to be in json, there is no need to use json.dumps.
json.dumps on a dictionary will return a string which cannot be indexed obviously and hence that error.
a requests response .json() method already loads the content of the string to json.
You should use that, but your code later serializes it back to a string, and hence the error (token is a string representation of the dict you are expecting, not the dict). You should just omit the json.dumps(response) line, and use response['ValidThrough']
There's another error here, even if you assume that the .json() returns a string that should be unserialized again you should've used json.loads(response) in order to load it into a dict (not dumps to serialize it again)
I'm starting to learn Python and I've written the following Python code (some of it omitted) and it works fine, but I'd like to understand it better. So I do the following:
html_doc = requests.get('[url here]')
Followed by:
if html_doc.status_code == 200:
soup = BeautifulSoup(html_doc.text, 'html.parser')
line = soup.find('a', class_="some_class")
value = re.search('[regex]', str(line))
print (value.group(0))
My questions are:
What does html_doc.text really do? I understand that it makes "text" (a string?) out of html_doc, but why isn't it text already? What is it? Bytes? Maybe a stupid question but why doesn't requests.get create a really long string containing the HTML code?
The only way that I could get the result of re.search was by value.group(0) but I have literally no idea what this does. Why can't I just look at value directly? I'm passing it a string, there's only one match, why is the resulting value not a string?
requests.get() return value, as stated in docs, is Response object.
re.search() return value, as stated in docs, is MatchObject object.
Both objects are introduced, because they contain much more information than simply response bytes (e.g. HTTP status code, response headers etc.) or simple found string value (e.g. it includes positions of first and last matched characters).
For more information you'll have to study docs.
FYI, to check type of returned value you may use built-in type function:
response = requests.get('[url here]')
print type(response) # <class 'requests.models.Response'>
Seems to me you are lacking some basic knowledge about Classes, Object and methods...etc, you need to read more about it here (for Python 2.7) and about requests module here.
Concerning what you asked, when you type html_doc = requests.get('url'), you are creating an instance of class requests.models.Response, you can check it by:
>>> type(html_doc)
<class 'requests.models.Response'>
Now, html_doc has methods, thus html_doc.text will return to you the server's response
Same goes for re module, each of its methods generates response object that are not simply int or string
i'm playing a little with google places api and requests
I got :
r = requests.get(self.url, params={'key': KEY, 'location': self.location, 'radius': self.radius, 'types': "airport"}, proxies=proxies)
r returns a 200 code, fine, but I'm confused by what r.json() returns compared to r.content
extract of r.json() :
{u'html_attributions': [],
u'next_page_token': u'CoQC-QAAABT4REkkX9NCxPWp0JcGK70kT4C-zM70b11btItnXiKLJKpr7l2GeiZeyL5y6NTDQA6ASDonIe5OcCrCsUXbK6W0Y09FqhP57ihFdQ7Bw1pGocLs_nAJodaS4U7goekbnKDlV3TaL8JMr4XpQBvlMN2dPvhFayU6RcF5kwvIm1YtucNOAUk-o4kOOziaJfeLqr3bk_Bq6DoCBwRmSEdZj34RmStdrX5RAirQiB2q_fHd6HPuHQzZ8EfdggqRLxpkFM1iRSnfls9WlgEJDxGB91ILpBsQE3oRFUoGoCfpYA-iW7E3uUD_ufby-JRqxgjD2isEIn8tntmFDjzQmjOraFQSEC6RFpAztLuk7l2ayfXsvw4aFO9gIhcXtG0LPucJkEa2nj3PxUDl',
u'results': [{u'geometry': {u'location': {u'lat': -33.939923,
u'lng': 151.175276}},
while extract of r.content :
'{\n "html_attributions" : [],\n "next_page_token" : "CoQC-QAAABT4REkkX9NCxPWp0JcGK70kT4C-zM70b11btItnXiKLJKpr7l2GeiZeyL5y6NTDQA6ASDonIe5OcCrCsUXbK6W0Y09FqhP57ihFdQ7Bw1pGocLs_nAJodaS4U7goekbnKDlV3TaL8JMr4Xp
so r.content has the double quotes like a "correct" json object while r.json() seems to have changed all double-quotes in single-quotes.
Should I care about it or not ? I can still access r.json() contents fine, just wondered if this was normal for requests to return an object with single quotes.
The json() method doesn't actually return JSON. It returns a python object (read: dictionary) that contains the same information as the json data. When you print it out, the quotes are added for the sake of readability, they are not actually in your data.
Should I care about it or not?
Not.
What you can however is to add
jsonresponse=json.dump(requests.get(xxx).json())
in order to get valid json in jsonresponse.
Python uses single or double quotes for strings. By default, it'll display single quote for strings.
However, JSON specification only consider double quotes to mark strings.
Note that requests' response.json() will return native Python types which are slightly different from their JSON representation you can see with response.content.
You are seeing the single quotes because you are looking at Python, not JSON.
Calling Response.json attempts to parse the content of the Response as JSON. If it is successful, it will return a combination of dicts, lists and native Python types as #Two-Bit Alchemist alluded to in his comment.
Behind the scenes, The json method is just calling complexjson.loads on the response text (see here). If you dig further to look at the requests.compat module to figure out what complexjson is, it is the simplejson package if it is importable on the system (i.e. installed) and the standard library json package otherwise (see here). So, modulo considerations about the encoding, you can read a call to Response.json as equivalent to:
import requests
import json
response = requests.get(...)
json.loads(response.text)
TL;DR: nothing exciting is happening and no, what is returned from Response.json is not intended to be valid JSON but rather valid JSON transformed into Python data structures and types.