I'm trying to use Facebook's REST api, and am encoding a JSON string/dictionary using urllib.urlencode. The result I get however, is different from the correct encoded result (as displayed by pasting the dictionary in the attachment field here http://developers.facebook.com/docs/reference/rest/stream.publish/). I was wondering if anyone could offer any help.
Thanks.
EDIT:
I'm trying to encode the following dictionary:
{"media": [{"type":"flash", "swfsrc":"http://shopperspoll.webfactional.com/media/flashFile.swf", "height": '100', "width": '100', "expanded_width":"160", "expanded_height":"120", "imgsrc":"http://shopperspoll.webfactional.com/media/laptop1.jpg"}]}
This is the encoded string using urllib.urlencode:
"media=%5B%7B%27swfsrc%27%3A+%27http%3A%2F%2Fshopperspoll.webfactional.com%2Fmedia%2FflashFile.swf%27%2C+%27height%27%3A+%27100%27%2C+%27width%27%3A+%27100%27%2C+%27expanded_width%27%3A+%27160%27%2C+%27imgsrc%27%3A+%27http%3A%2F%2Fshopperspoll.webfactional.com%2Fmedia%2Flaptop1.jpg%27%2C+%27expanded_height%27%3A+%27120%27%2C+%27type%27%3A+%27flash%27%7D%5D"
It's not letting me copy the result being thrown out from the facebook rest documentation link, but on copying the above dictionary in the attachment field, the result is different.
urllib.encode isn't meant for urlencoding a single value (as functions of the same name are in many languages), but for encoding a dict of separate values. For example, if I had the dict {"a": 1, "b": 2} it would produce the string "a=1&b=2".
First, you want to encode your dict as JSON.
data = {"media": [{"type":"flash", "swfsrc":"http://shopperspoll.webfactional.com/media/flashFile.swf", "height": '100', "width": '100', "expanded_width":"160", "expanded_height":"120", "imgsrc":"http://shopperspoll.webfactional.com/media/laptop1.jpg"}]}
import json
json_encoded = json.dumps(data)
You can then either use urllib.encode to create a complete query string
import urllib
urllib.encode({"access_token": example, "attachment": json_encoded})
# produces a long string in the form "access_token=...&attachment=..."
or use urllib.quote to just encode your attachment parameter
urllib.quote(json_encoded)
# produces just the part following "&attachment="
Related
I am working on a program that reads the content of a Restful API from ImportIO. The connection works, and data is returned, but it's a jumbled mess. I'm trying to clean it to only return Asins.
I have tried using the split keyword and delimiter to no success.
stuff = requests.get('https://data.import.io/extractor***')
stuff.content
I get the content, but I want to extract only Asins.
results
While .content gives you access to the raw bytes of the response payload, you will often want to convert them into a string using a character encoding such as UTF-8. the response will do that for you when you access .text.
response.txt
Because the decoding of bytes to str requires an encoding scheme, requests will try to guess the encoding based on the response’s headers if you do not specify one. You can provide an explicit encoding by setting .encoding before accessing .text:
If you take a look at the response, you’ll see that it is actually serialized JSON content. To get a dictionary, you could take the str you retrieved from .text and deserialize it using json.loads(). However, a simpler way to accomplish this task is to use .json():
response.json()
The type of the return value of .json() is a dictionary, so you can access values in the object by key.
You can do a lot with status codes and message bodies. But, if you need more information, like metadata about the response itself, you’ll need to look at the response’s headers.
For More Info: https://realpython.com/python-requests/
What format is the return information in? Typically Restful API's will return the data as json, you will likely have luck parsing the it as a json object.
https://realpython.com/python-requests/#content
stuff_dictionary = stuff.json()
With that, you can load the content is returned as a dictionary and you will have a much easier time.
EDIT:
Since I don't have the full URL to test, I can't give an exact answer. Given the content type is CSV, using a pandas DataFrame is pretty easy. With a quick StackOverflow search, I found the following answer: https://stackoverflow.com/a/43312861/11530367
So I tried the following in the terminal and got a dataframe from it
from io import StringIO
import pandas as pd
pd.read_csv(StringIO("HI\r\ntest\r\n"))
So you should be able to perform the following
from io import StringIO
import pandas as pd
df = pd.read_csv(StringIO(stuff.content))
If that doesn't work, consider dropping the first three bytes you have in your response: b'\xef\xbb\xf'. Check the answer from Mark Tolonen to get parse this.
After that, selecting the ASIN (your second column) from your dataframe should be easy.
asins = df.loc[:, 'ASIN']
asins_arr = asins.array
The response is the byte string of CSV content encoded in UTF-8. The first three escaped byte codes are a UTF-8-encoded BOM signature. So stuff.content.decode('utf-8-sig') should decode it. stuff.text may also work if the encoding was returned correctly in the response headers.
I am trying to get a key code from a json output.
But i cannot seem to get it, I get errors left and right.
Here is my code.
import requests
import time
import threading
import json
def ThreadRequest():
scrape_url = "https://pastebin.com/api_scraping.php?limit=1"
json_data = requests.get(scrape_url)
python_obj = json.loads(json_data.text)
print python_obj["key"]
ThreadRequest()
I either get
TypeError: list indices must be integers, not str
ValueError: No JSON object could be decoded
TypeError: expected string or buffer
I have tried many ways, different ways, even parsing by using .split() function.
I cannot seem to get the understanding of how to parse in json.
Here is the API output
[
{
"scrape_url": "https://pastebin.com/api_scrape_item.php?i=rkFbtGSj",
"full_url": "https://pastebin.com/rkFbtGSj",
"date": "1516914453",
"key": "rkFbtGSj",
"size": "3031",
"expire": "0",
"title": "",
"syntax": "text",
"user": ""
}
]
The first thing is that the requests module has a built-in JSON parsing method so just use that rather than trying to use the raw text response. Change:
python_obj = json.loads(json_data.text)
To:
python_obj = json_data.json()
Second, the data that you're interested in is in a dictionary. However, that dictionary is contained within a list. Take the 0th index of that list to get access to the dictionary, then access that by key (in this case, also called "key").
my_value = python_obj[0]['key']
I'm newbie in python and trying to parse data in my application using these lines of codes
json_str = request.body.decode('utf-8')
py_str = json.loads(json_str)
But I'm getting this error on json.loads
Expecting value: line 1 column 1 (char 0)
this is json formatted data that I send from angular app (Updated)
Object { ClientTypeId: 6, ClientName: "asdasd", ClientId: 0, PhoneNo: "123", FaxNo: "123", NTN: "1238", GSTNumber: "1982", OfficialAddress: "sads", MailingAddress: "asdasd", RegStartDate: "17-Aug-2016", 15 more… }
these are the values that I get in json_str
ClientTypeId=5&ClientName=asdasd&ClientId=0&PhoneNo=123&FaxNo=123&NTN=123&GSTNumber=12&OfficialAddress=adkjh&MailingAddress=adjh&RegStartDate=09-Aug-2016&RegEndDate=16-Aug-2016&Status=1&CanCreateUser=true&UserQuotaFor=11&UserQuotaType=9&MaxUsers=132123&ApplyUserCharges=true&ApplyReportCharges=true&EmailInvoice=true&BillingType=1&UserCharges=132&ReportCharges=123&MonthlyCharges=123&BillingDate=16-Aug-2016&UserSessionId=324
I don't know what's wrong in it.. can anyone mention what's the mistake is??
Your data is not JSON-formatted, not even the one you included in your updated answer. Your data is a JavaScript-object, not an encoded string. Please note the "N" in JSON: Notation -- it is a format inspired from how data is written in JavaScript code, but runtime JavaScript data is not represented in JSON. The "JSON" you pasted is how your browser represents the object to you, it is not proper JSON (that would be {"ClientTypeId": 6, ...} -- note the quotes around the property name).
When sending this data to the server, you have to encode it. You think you are sending it JSON-encoded, but you aren't. You are sending it "web form encoded" (data of type application/x-www-form-urlencoded).
Now either you have to learn how to send the data in JSON format from Angular, or use the correct parsing routine in Python: urllib.parse.parse_qs. Depending on the library you are using, there might be a convenience method to access the data as well, as this is a common use case.
I am currently trying to create a dictionary from a json formatted server response:
{"id": null,{"version": "1.1","result": "9QtirjtH9b","error": null}}
Therefore I am using json.loads(). But I always get the following error:
ValueError: Expecting property name: line 1 column 12 (char 12)
I know that this means that there is an error in the json syntax and I found some threads (like this one) here at stackoverflow, but they did not include an answer that solved my problem.
However, I was not sure if the null value within the json response causes the error, so I had a closer look at the json.org Reference Manual and it seems to be a valid syntax. Any ideas?
It's not valid. The outer object needs a property name for the second element; raw values are not valid in an object.
{"id": null, "somename":{"version": "1.1","result": "9QtirjtH9b","error": null}}
The problem here is the lack of a key for the nested object, not the null. You'd need to find a way to fix that syntax or parse it yourself.
If we make a few assumptions about the syntax, you should be able to use a regular expression to fix the JSON data before decoding:
import re
from itertools import count
def _gen_id(match, count=count()):
return '{1}"generated_id_{0}":{2}'.format(next(count), *match.groups())
_no_key = re.compile(r'(,)({)')
def fix_json(json_data):
return _no_key.sub(_gen_id, json_data)
This assumes that any ,{ combo indicates the location of a missing key, and generates one to insert there. That is a reasonable assumption to make, but may break things if you have string data with exactly that sequence.
Demo:
>>> json_data = '{"id": null,{"version": "1.1","result": "9QtirjtH9b","error": null}}'
>>> fix_json(json_data)
'{"id": null,"generated_id_0":{"version": "1.1","result": "9QtirjtH9b","error": null}}'
>>> json.loads(fix_json(json_data))
{u'id': None, u'generated_id_1': {u'version': u'1.1', u'result': u'9QtirjtH9b', u'error': None}}
I have troubleshooting some code that uses HTTP POST to send data and should return a JSON result whose contents are a dictionary. I am using an XML-RPC wrapper to expose this service. When the wrapper receives the dict information from the http response variable, the dict contents are in a string in this form:
{'created': datetime.datetime(2010, 12, 31, 19, 13, 8, 379909), 'worker': u'GoogleWorker', 'ready': False, 'request_id': '8f1381853a444a42a37ae5152a3af947', 'owner': u'admin', 'shortname': u'test19'}
I'm trying to convert the string below into a JSON result using the following statement:
result = json.loads(response[1])
However, when I try to use json.loads to convert the data to JSON, I get the following error: Fault: <Fault 1: "<type 'exceptions.ValueError'>:Expecting property name: line 1 column 1 (char 1)">
I manually tried to convert the above string to JSON, but I get the same error. Is the dict malformed in some way? Is it due to unicode? I also tried setting the locale to UTF-8, but that was unsuccessful.
Any help would be greatly appreciated.
You are trying to use the wrong method. json.loads is for loading JSON to Python. If you want to convert Python to JSON, you need json.dumps.
result = json.dumps(response[1])
That dict is in Python dict literal format, not JSON. You can do:
import ast
result = ast.literal_eval(response[1])
to read in the response in that format. Are you sure that Django hasn't already JSON-decoded the response?
i have use json on django , i use this :
import simplejson as json
#to encode
final= {'first':first_data,'second':second_data}
json.dumps(final)
#to decode this is the example from python's api
json.loads('["foo", {"bar":["baz", null, 1.0, 2]}]')