Develop Reference

Passing a list of identifiers into an API parameter in Python [duplicate]

I have been trying to figure out how to use python-requests to send a request that the url looks like:
http://example.com/api/add.json?name='hello'&data[]='hello'&data[]='world'
Normally I can build a dictionary and do:
data = {'name': 'hello', 'data': 'world'}
response = requests.get('http://example.com/api/add.json', params=data)
That works fine for most everything that I do. However, I have hit the url structure from above, and I am not sure how to do that in python without manually building strings. I can do that, but would rather not.
Is there something in the requests library I am missing or some python feature I am unaware of?
Also what do you even call that type of parameter so I can better google it?

All you need to do is putting it on a list and making the key as list like string:
data = {'name': 'hello', 'data[]': ['hello', 'world']}
response = requests.get('http://example.com/api/add.json', params=data)

What u are doing is correct only. The resultant url is same what u are expecting.
>>> payload = {'name': 'hello', 'data': 'hello'}
>>> r = requests.get("http://example.com/api/params", params=payload)
u can see the resultant url:
>>> print(r.url)
http://example.com/api/params?name=hello&data=hello
According to url format:
In particular, encoding the query string uses the following rules:
Letters (A–Z and a–z), numbers (0–9) and the characters .,-,~ and _ are left as-is
SPACE is encoded as + or %20
All other characters are encoded as %HH hex representation with any non-ASCII characters first encoded as UTF-8 (or other specified encoding)
So array[] will not be as expected and will be automatically replaced according to the rules:
If you build a url like :
`Build URL: http://example.com/api/add.json?name='hello'&data[]='hello'&data[]='world'`
OutPut will be:
>>> payload = {'name': 'hello', "data[]": 'hello','data[]':'world'}
>>> r = requests.get("http://example.com/api/params", params=payload)
>>> r.url
u'http://example.com/api/params?data%5B%5D=world&name=hello'
This is because Duplication will be replaced by the last value of the key in url and data[] will be replaced by data%5B%5D.
If data%5B%5D is not the problem(If server is able to parse it correctly),then u can go ahead with it.
Source Link

One solution if using the requests module is not compulsory, is using the urllib/urllib2 combination:
payload = [('name', 'hello'), ('data[]', ('hello', 'world'))]
params = urllib.urlencode(payload, doseq=True)
sampleRequest = urllib2.Request('http://example.com/api/add.json?' + params)
response = urllib2.urlopen(sampleRequest)
Its a little more verbose and uses the doseq(uence) trick to encode the url parameters but I had used it when I did not know about the requests module.
For the requests module the answer provided by #Tomer should work.

Some api-servers expect json-array as value in the url query string. The requests params doesn't create json array as value for parameters.
The way I fixed this on a similar problem was to use urllib.parse.urlencode to encode the query string, add it to the url and pass it to requests
e.g.
from urllib.parse import urlencode
query_str = urlencode(params)
url = "?" + query_str
response = requests.get(url, params={}, headers=headers)

The solution is simply using the famous function: urlencode
>>> import urllib.parse
>>> params = {'q': 'Python URL encoding', 'as_sitesearch': 'www.urlencoder.io'}
>>> urllib.parse.urlencode(params)
'q=Python+URL+encoding&as_sitesearch=www.urlencoder.io'

Loading JSON with json.loads in Python gives JSONDecodeError because of "

I am trying to turn a Javascript from a website into a JSON structure with Python's json.loads() but it gives a JSONDecodeError. It's because there are objects in the Javascript which are quoted, and when json.loads() runs, it turns " into a " (double quote), which produces bad JSON.
This is a very small example of the Javascript:
{"key":{"hascookie":"yes"}, "cookiestatus":234, "widget":null, "player":"{"source":true,"country"}"}
There is a lot of JSON, and it's minified.
I am loading it like this:
j = 'JSON text'
result = json.loads(j)
Is the solution prevent the loads() function to unquote the JSON and leave the " as is?

Here is a possible solution (if I understand the problem correctly...):
import json
s = '{"source":true,"country":"GB"}'
class myEncoder(json.JSONDecoder):
def decode(self, s):
s = s.replace('"', '\"')
return json.JSONDecoder.decode(self, s)
decoded = json.loads(s, cls=myEncoder)
print(decoded)
Output is correct JSON: {'source': True, 'country': 'GB'}

python unable to parse JSON Data

I am unable to parse the JSON data using python.
A webpage url is returning JSON Data
import requests
import json
BASE_URL = "https://www.codechef.com/api/ratings/all"
data = {'page': page, 'sortBy':'global_rank', 'order':'asc', 'itemsPerPage':'40' }
r = requests.get(BASE_URL, data = data)
receivedData = (r.text)
print ((receivedData))
when I printed this, I got large text and when I validated using https://jsonlint.com/ it showed VALID JSON
Later I used
import requests
import json
BASE_URL = "https://www.codechef.com/api/ratings/all"
data = {'page': page, 'sortBy':'global_rank', 'order':'asc', 'itemsPerPage':'40' }
r = requests.get(BASE_URL, data = data)
receivedData = (r.text)
print (json.loads(receivedData))
When I validated the large printed text using https://jsonlint.com/ it showed INVALID JSON
Even if I don't print and directly use the data. It is working properly. So I am sure even internally it is not loading correctly.
is python unable to parse the text to JSON properly?

in short, json.loads converts from a Json (thing, objcet, array, whatever) into a Python object - in this case, a Json Dictionary. When you print that, it will print as a itterative and therefore print with single quotes..
Effectively your code can be expanded:
some_dictionary = json.loads(a_string_which_is_a_json_object)
print(some_dictionary)
to make sure that you're printing json-safe, you would need to re-encode with json.dumps

When you use python's json.loads(text) it returns a python dictionary. When you print that dictionary out it is not in json format.
If you want a json output you should use json.dumps(json_object).

Can't extract JSON from an http request

I'm having problems getting data from an HTTP response. The format unfortunately comes back with '\n' attached to all the key/value pairs. JSON says it must be a str and not "bytes".
I have tried a number of fixes so my list of includes might look weird/redundant. Any suggestions would be appreciated.
#!/usr/bin/env python3
import urllib.request
from urllib.request import urlopen
import json
import requests
url = "http://finance.google.com/finance/info?client=ig&q=NASDAQ,AAPL"
response = urlopen(url)
content = response.read()
print(content)
data = json.loads(content)
info = data[0]
print(info)
#got this far - planning to extract "id:" "22144"

When it comes to making requests in Python, I personally like to use the requests library. I find it easier to use.
import json
import requests
r = requests.get('http://finance.google.com/finance/info?client=ig&q=NASDAQ,AAPL')
json_obj = json.loads(r.text[4:])
print(json_obj[0].get('id'))
The above solution prints: 22144
The response data had a couple unnecessary characters at the head, which is why I am only loading the relevant (json) portion of the response: r.text[4:]. This is the reason why you couldn't load it as json initially.

Bytes object has method decode() which converts bytes to string. Checking the response in the browser, seems there are some extra characters at the beginning of the string that needs to be removed (a line feed character, followed by two slashes: '\n//'). To skip the first three characters from the string returned by the decode() method we add [3:] after the method call.
data = json.loads(content.decode()[3:])
print(data[0]['id'])
The output is exactly what you expect:
22144

JSON says it must be a str and not "bytes".
Your content is "bytes", and you can do this as below.
data = json.loads(content.decode())

Python: How to prevent python dictionary from putting quotes around my json?

I am using requests to create a post request on a contractor's API. I have a JSON variable inputJSON that undergoes formatting like so:
def dolayoutCalc(inputJSON):
inputJSON = ast.literal_eval(inputJSON)
inputJSON = json.dumps(inputJSON)
url='http://xxyy.com/API'
payload = {'Project': inputJSON, 'x':y, 'z':f}
headers = {'content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.post(url, data=json.dumps(payload), headers=headers)
My issue arises when I define payload={'Project':inputJSON, 'x':y, 'z':f}
What ends up happening is Python places a pair of quotes around the inputJSON structure. The API I am hitting is not able to handle this. It needs Project value to be the exact same inputJSON value just without the quotes.
What can I do to prevent python from placing quotes around my inputJSON object? Or is there a way to use requests library to handle such POST request situation?

inputJSON gets quotes around it because it's a string. When you call json.dumps() on something a string will come out, and then when it's converted to JSON it will get quotes around it. e.g.:
>>> import json
>>> json.dumps('this is a string')
>>> '"this is a string"'
I'm with AKS in that should be able to remove this line:
inputJSON = json.dumps(inputJSON)
From your description inputJSON sounds like a Python literal (e.g. {'blah': True} instead of {"blah": true}. So you've used the ast module to convert it into a Python value, and then in the final json.dumps() it should be converted to JSON along with everything else.
Example:
>>> import ast
>>> import json
>>> input = "{'a_var': True}" # A string that looks like a Python literal
>>> input = ast.literal_eval(input) # Convert to a Python dict
>>> print input
>>> {'a_var': True}
>>> payload = {'Project': input} # Add to payload as a dict
>>> print json.dumps(payload)
>>> {"Project": {"a_var": true}} # In the payload as JSON without quotes