Convert list of byte objects to dict - python

I'm consuming an API that returns a list a of objects in a JSON. But when I get its content with Requests lib, the content is a byte array of objects like this :
b'[{"id":44,"id_string":"a2BPQDsGLfLiwo4r5U4JCY","title":"ED_1803_ITAIPAVA_RJ","description":"ED_1803_ITAIPAVA_RJ","url":"https://kobocat.docker.kobo.techo.org/api/v1/data/44"},{"id":57,"id_string":"a3pb3ALiGuQAHD6XzdHAip","title":"ED_v2018_1801_Taba\xc3\xa7u-SP","description":"ED_v2018_1801_Taba\xc3\xa7u-SP","url":"https://kobocat.docker.kobo.techo.org/api/v1/data/57"},{"id":68,"id_string":"a4Gz2dSwRuyQCsjBwNhf3D","title":"ECS_1804_SONHO REAL-BA","description":"ECS_1804_SONHO REAL-BA","url":"https://kobocat.docker.kobo.techo.org/api/v1/data/68"},{"id":2,"id_string":"a4KjYoy8ieCRNykiYb7nGP","title":"ECS_1708_Vila Esperan\xc3\xa7a-SP","description":"ECS_1708_Vila Esperan\xc3\xa7a-SP","url":"https://kobocat.docker.kobo.techo.org/api/v1/data/2"},{"id":38,"id_string":"a7GQQ7xEu4K6HXWYu9SaSC","title":"ECo_1711_Terra Nossa-UF","description":"ECo_1711_Terra Nossa-UF","url":"https://kobocat.docker.kobo.techo.org/api/v1/data/38"},{"id":78,"id_string":"a7NnnbdhBUSsGoxVWBiGFb","title":"ECoSP_1805_Vila Nova Esperan\xc3\xa7a-SP","description":"ECoSP_1805_Vila Nova Esperan\xc3\xa7a-SP","url":"https://kobocat.docker.kobo.techo.org/api/v1/data/78"}]
How can I make it a normal list of dictionares? I tried iterating through the byte array with for in range() , but all I can return are numbers.

You can do this:
import json
dic = json.loads(Your_input)

I can't comment to answers so I'm writing here.
import json
dic = json.loads(Your_input)
This code works perfectly,
but as #jonrsharpe mentioned in the comment section, you don't need to import anything.
You can use:
page = reg.get("your_url")
page.json()

Related

Convert Json format String to Link{"link":"https://i.imgur.com/zfxsqlk.png"}

I try to convert this String to only the link: {"link":"https://i.imgur.com/zfxsqlk.png"}
I'm trying to create a discord bot, which sends random pictures from the API https://some-random-api.ml/img/red_panda.
With imageURL = json.loads(requests.get(redpandaurl).content) I get the json String, but what do I have to do that I only get the Link like this https://i.imgur.com/zfxsqlk.png
Sorry if my question is confusingly written, I'm new to programming and don't really know how to describe this problem.
You can simply do this:
image_url = requests.get(your_api_url).json()["link"]
Directly use requests.json(), no need to load the string with json.loads and other manual stuff.
What you get from json.loads() is a Python dict. You can access values in the dict by specifying their keys.
In your case, there is only one key-value pair in the dict: "link" is the key and "https://i.imgur.com/zfxsqlk.png" is the value. You can get the link and store it in the value by appending ["link"] to your line of code:
imageURL = json.loads(requests.get(redpandaurl).content)["link"]

How to separate data in a Restful API?

I am working on a program that reads the content of a Restful API from ImportIO. The connection works, and data is returned, but it's a jumbled mess. I'm trying to clean it to only return Asins.
I have tried using the split keyword and delimiter to no success.
stuff = requests.get('https://data.import.io/extractor***')
stuff.content
I get the content, but I want to extract only Asins.
results
While .content gives you access to the raw bytes of the response payload, you will often want to convert them into a string using a character encoding such as UTF-8. the response will do that for you when you access .text.
response.txt
Because the decoding of bytes to str requires an encoding scheme, requests will try to guess the encoding based on the response’s headers if you do not specify one. You can provide an explicit encoding by setting .encoding before accessing .text:
If you take a look at the response, you’ll see that it is actually serialized JSON content. To get a dictionary, you could take the str you retrieved from .text and deserialize it using json.loads(). However, a simpler way to accomplish this task is to use .json():
response.json()
The type of the return value of .json() is a dictionary, so you can access values in the object by key.
You can do a lot with status codes and message bodies. But, if you need more information, like metadata about the response itself, you’ll need to look at the response’s headers.
For More Info: https://realpython.com/python-requests/
What format is the return information in? Typically Restful API's will return the data as json, you will likely have luck parsing the it as a json object.
https://realpython.com/python-requests/#content
stuff_dictionary = stuff.json()
With that, you can load the content is returned as a dictionary and you will have a much easier time.
EDIT:
Since I don't have the full URL to test, I can't give an exact answer. Given the content type is CSV, using a pandas DataFrame is pretty easy. With a quick StackOverflow search, I found the following answer: https://stackoverflow.com/a/43312861/11530367
So I tried the following in the terminal and got a dataframe from it
from io import StringIO
import pandas as pd
pd.read_csv(StringIO("HI\r\ntest\r\n"))
So you should be able to perform the following
from io import StringIO
import pandas as pd
df = pd.read_csv(StringIO(stuff.content))
If that doesn't work, consider dropping the first three bytes you have in your response: b'\xef\xbb\xf'. Check the answer from Mark Tolonen to get parse this.
After that, selecting the ASIN (your second column) from your dataframe should be easy.
asins = df.loc[:, 'ASIN']
asins_arr = asins.array
The response is the byte string of CSV content encoded in UTF-8. The first three escaped byte codes are a UTF-8-encoded BOM signature. So stuff.content.decode('utf-8-sig') should decode it. stuff.text may also work if the encoding was returned correctly in the response headers.

Added escaped quotes to JSON in Flask app with mongoDB

I am trying to create API for my Flask project. I have data stored in mongoDB and for building API I am using flask_restful. The problem is that in JSON are added escaped quotes and I cannot figure why and I rather have my JSON without them.
This is how my get function looks like:
from flask_restful import Resource
import json
from bson import json_util
class Harvests(Resource):
def get(self):
json_docs = []
for doc in db.collection.find():
json_doc = json.dumps(doc, default=json_util.default)
json_docs.append(json_doc)
return json_docs
In app.py it is just like that
api = Api(app)
api.add_resource(Harvests, '/api/harvests')
And I get JSON with escaped quotes (in browser or with curl)
[
"{\"_id\": {\"$oid\": \"5c05429cc4247917d66163a7\"},...
]
If I try this outside Flask (print JSON from mongo) and it works just fine. I tried use .replace(), but I think is not most elegant solution, but it did not work anyway. Any idea how I should get rid off these backslashes?
What you see is absolutely what you should expect to see according to your code, so I think there is a misunderstanding at some point. Let me explain what you are doing.
You convert each doc (a data structure) into a jsonified version (a string) of this data. Then you gather these strings in a list. Later you see this list, and of course you see a list of strings. Each of these strings contains a jsonified version of a data structure (a dictionary with opening braces, keys and values inside, and each key is a string itself with quotes, so these quotes are escaped within the jsonified string).
I recommend to collect your documents into a list and then convert that list to json instead:
def get(self):
docs = []
for doc in db.collection.find():
docs.append(doc)
return json.dumps(docs, default=json_util.default)
This way you get one json string representing the list of docs.
Maybe your framework is already applying a jsonifying automatically, in this case just don't do this step yourself:
return docs
Just use this instead.

How to read and assign variables from an API return that's formatted as Dictionary-List-Dictionary?

So I'm trying to learn Python here, and would appreciate any help you guys could give me. I've written a bit of code that asks one of my favorite websites for some information, and the api call returns an answer in a dictionary. In this dictionary is a list. In that list is a dictionary. This seems crazy to me, but hell, I'm a newbie.
I'm trying to assign the answers to variables, but always get various error messages depending on how I write my {},[], or (). Regardless, I can't get it to work. How do I read this return? Thanks in advance.
{
"answer":
[{"widgets":16,
"widgets_available":16,
"widgets_missing":7,
"widget_flatprice":"156",
"widget_averages":15,
"widget_cost":125,
"widget_profit":"31",
"widget":"90.59"}],
"result":true
}
Edited because I put in the wrong sample code.
You need to show your code, but the de-facto way of doing this is by using the requests module, like this:
import requests
url = 'http://www.example.com/api/v1/something'
r = requests.get(url)
data = r.json() # converts the returned json into a Python dictionary
for item in data['answer']:
print(item['widgets'])
Assuming that you are not using the requests library (see Burhan's answer), you would use the json module like so:
data = '{"answer":
[{"widgets":16,
"widgets_available":16,
"widgets_missing":7,
"widget_flatprice":"156",
"widget_averages":15,
"widget_cost":125,
"widget_profit":"31",
"widget":"90.59"}],
"result":true}'
import json
data = json.loads(data)
# Now you can use it as you wish
data['answer'] # and so on...
First I will mention that to access a dictionary value you need to use ["key"] and not {}. see here an Python dictionary syntax.
Here is a step by step walkthrough on how to build and access a similar data structure:
First create the main dictionary:
t1 = {"a":0, "b":1}
you can access each element by:
t1["a"] # it'll return a 0
Now lets add the internal list:
t1["a"] = ["x",7,3.14]
and access it using:
t1["a"][2] # it'll return 3.14
Now creating the internal dictionary:
t1["a"][2] = {'w1':7,'w2':8,'w3':9}
And access:
t1["a"][2]['w3'] # it'll return 9
Hope it helped you.

JSON Python get field as array

This is my response to a get request for some json goodness.
I'm getting this in Python, everything works up to here.
I've been searching for json documentation and reading quite a bit but can't seam to find my answer.
How would I get all the email addresses?
{u'-InFSLzYdyg-OcTosYYs': {u'email': u'hello#gmail.com', u'time': 1360707022892}, u'- InFYJya4K6tZa8YSzme': {u'email': u'me#gmail.com', u'time': 1360708587511}}
What I'd want is a list like so:
email = ['hello#gmail.com', 'me#gmail.com']
Thanks in advance.
Like wRAR said, once you have it as a python dict, it should be as simple as:
[x['email'] for x in l.itervalues()]
Assuming you're converted you JSON string to a python dict (see loads()):
>>> from json import loads
>>> myJSON = loads(somejsonstring)
>>> emails = [a[x]['email'] for x in a]
>>> emails
['hello#gmail.com', 'me#gmail.com']
Or even better, use itervalues() as Luke mentioned.
Just do json.loads and process the resulting dict as usual. There is nothing JSON-specific here.

Categories

Resources