Develop Reference

How to separate data in a Restful API?

I am working on a program that reads the content of a Restful API from ImportIO. The connection works, and data is returned, but it's a jumbled mess. I'm trying to clean it to only return Asins.
I have tried using the split keyword and delimiter to no success.
stuff = requests.get('https://data.import.io/extractor***')
stuff.content
I get the content, but I want to extract only Asins.
results

While .content gives you access to the raw bytes of the response payload, you will often want to convert them into a string using a character encoding such as UTF-8. the response will do that for you when you access .text.
response.txt
Because the decoding of bytes to str requires an encoding scheme, requests will try to guess the encoding based on the response’s headers if you do not specify one. You can provide an explicit encoding by setting .encoding before accessing .text:
If you take a look at the response, you’ll see that it is actually serialized JSON content. To get a dictionary, you could take the str you retrieved from .text and deserialize it using json.loads(). However, a simpler way to accomplish this task is to use .json():
response.json()
The type of the return value of .json() is a dictionary, so you can access values in the object by key.
You can do a lot with status codes and message bodies. But, if you need more information, like metadata about the response itself, you’ll need to look at the response’s headers.
For More Info: https://realpython.com/python-requests/

What format is the return information in? Typically Restful API's will return the data as json, you will likely have luck parsing the it as a json object.
https://realpython.com/python-requests/#content
stuff_dictionary = stuff.json()
With that, you can load the content is returned as a dictionary and you will have a much easier time.
EDIT:
Since I don't have the full URL to test, I can't give an exact answer. Given the content type is CSV, using a pandas DataFrame is pretty easy. With a quick StackOverflow search, I found the following answer: https://stackoverflow.com/a/43312861/11530367
So I tried the following in the terminal and got a dataframe from it
from io import StringIO
import pandas as pd
pd.read_csv(StringIO("HI\r\ntest\r\n"))
So you should be able to perform the following
from io import StringIO
import pandas as pd
df = pd.read_csv(StringIO(stuff.content))
If that doesn't work, consider dropping the first three bytes you have in your response: b'\xef\xbb\xf'. Check the answer from Mark Tolonen to get parse this.
After that, selecting the ASIN (your second column) from your dataframe should be easy.
asins = df.loc[:, 'ASIN']
asins_arr = asins.array

The response is the byte string of CSV content encoded in UTF-8. The first three escaped byte codes are a UTF-8-encoded BOM signature. So stuff.content.decode('utf-8-sig') should decode it. stuff.text may also work if the encoding was returned correctly in the response headers.

Get JSON response in Python, but in original JavaScript format

I am putting a JSON response into a variable via requests.json() like this:
response = requests.get(some_url, params=some_params).json()
This however converts JSON's original " to Python's ', true to True, null to None.
This poses a problem when trying to save the response as text and the convert it back to JSON - sure, I can use .replace() for all conversions mentioned above, but even once I do that, I get other funny json decoder errors.
Is there any way in Python to get JSON response and keep original JavaScript format?

json() is the JSON decoder method. You are looking at a Python object, that is why it looks like Python.
Other formats are listed on the same page, starting from Response Content
.text: text - it has no separate link/paragraph, it is right under "Response Content"
.content: binary, as bytes
.json(): decoded JSON, as Python object
.raw: streamed bytes (so you can get parts of content as it comes)
You need .text for getting text, including JSON data.

You can get the raw text of your response with requests.get(some_url, params=some_params).text
It is the json method which converts to a Python friendly format.

Scrapy javascript json object loads

I am scraping one web site which has javascript json object. How can i convert that javascript json object to pure json object. I need JSON.stringfy method like javascript. How can i do that on python.
{
title: 'Erkek Gri Sandalet',
description: '<ul><li>Asıl Dış Materyal: Suni Deri</li><li>İç Materyal: Suni Deri</li><li>Taban Materyali: Kauçuk Taban</li></ul>',
url: '/erkek-gri-sandalet-3500250',
code: 'C-362686',
id: '3500250'
}
i am getting an error when passing above string into the json.loads() The error is
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
How can i convert to this javascript json object to native json. I used https://www.freeformatter.com/json-formatter.html website to validate above json and that website validate and converted native JSON easily.

You could do something funky with regular expressions, but actually you should take advantage of the fact that even though this is not valid JSON, it is valid YAML. Install PyYAML and you can just do yaml.load(data).

Push a raw value to Firebase via REST API

I am trying to use the requests library in Python to push data (a raw value) to a firebase location.
Say, I have urladd (the url of the location with authentication token). At the location, I want to push a string, say International. Based on the answer here, I tried
data = {'.value': 'International'}
p = requests.post(urladd, data = sjson.dumps(data))
I get <Response [400]>. p.text gives me:
u'{\n "error" : "Invalid data; couldn\'t parse JSON object, array, or value. Perhaps you\'re using invalid characters in your key names."\n}\n'
It appears that they key .value is invalid. But that is what the answer linked above suggests. Any idea why this may not be working, or how I can do this through Python? There are no problems with connection or authentication because the following works. However, that pushes an object instead of a raw value.
data = {'name': 'International'}
p = requests.post(urladd, data = sjson.dumps(data))
Thanks for your help.

The answer you've linked is a special case for when you want to assign a priority to a value. In general, '.value' is an invalid name and will throw an error.
If you want to write just "International", you should write the stringified-JSON version of that data. I don't have a python example in front of me, but the curl command would be:
curl -X POST -d "\"International\"" https://...

Andrew's answer above works. In case someone else wants to know how to do this using the requests library in Python, I thought this would be helpful.
import simplejson as sjson
data = sjson.dumps("International")
p = requests.post(urladd, data = data)
For some reason I had thought that the data had to be in a dictionary format before it is converted to stringified JSON version. That is not the case, and a simple string can be used as an input to sjson.dumps().

urllib's urlencode returning weird encoded results

I'm trying to use Facebook's REST api, and am encoding a JSON string/dictionary using urllib.urlencode. The result I get however, is different from the correct encoded result (as displayed by pasting the dictionary in the attachment field here http://developers.facebook.com/docs/reference/rest/stream.publish/). I was wondering if anyone could offer any help.
Thanks.
EDIT:
I'm trying to encode the following dictionary:
{"media": [{"type":"flash", "swfsrc":"http://shopperspoll.webfactional.com/media/flashFile.swf", "height": '100', "width": '100', "expanded_width":"160", "expanded_height":"120", "imgsrc":"http://shopperspoll.webfactional.com/media/laptop1.jpg"}]}
This is the encoded string using urllib.urlencode:
"media=%5B%7B%27swfsrc%27%3A+%27http%3A%2F%2Fshopperspoll.webfactional.com%2Fmedia%2FflashFile.swf%27%2C+%27height%27%3A+%27100%27%2C+%27width%27%3A+%27100%27%2C+%27expanded_width%27%3A+%27160%27%2C+%27imgsrc%27%3A+%27http%3A%2F%2Fshopperspoll.webfactional.com%2Fmedia%2Flaptop1.jpg%27%2C+%27expanded_height%27%3A+%27120%27%2C+%27type%27%3A+%27flash%27%7D%5D"
It's not letting me copy the result being thrown out from the facebook rest documentation link, but on copying the above dictionary in the attachment field, the result is different.

urllib.encode isn't meant for urlencoding a single value (as functions of the same name are in many languages), but for encoding a dict of separate values. For example, if I had the dict {"a": 1, "b": 2} it would produce the string "a=1&b=2".
First, you want to encode your dict as JSON.
data = {"media": [{"type":"flash", "swfsrc":"http://shopperspoll.webfactional.com/media/flashFile.swf", "height": '100', "width": '100', "expanded_width":"160", "expanded_height":"120", "imgsrc":"http://shopperspoll.webfactional.com/media/laptop1.jpg"}]}
import json
json_encoded = json.dumps(data)
You can then either use urllib.encode to create a complete query string
import urllib
urllib.encode({"access_token": example, "attachment": json_encoded})
# produces a long string in the form "access_token=...&attachment=..."
or use urllib.quote to just encode your attachment parameter
urllib.quote(json_encoded)
# produces just the part following "&attachment="