I'm trying to use a REST API apparently constructed with LoopBack and it requires that my filter parameters be encoded in a specific way.
What is this sort of transformation called?
# this is a JSON type encoding
...?filter={"where": {"StartDate": {"gte": "2021-01-01T00:00:00.000Z"}}, "offset": 0, "limit": 100, "order": "id DESC" }
which needs to be encoded as some sort of HTTP query string
...?filter=%7B%22where%22%3A%20%7B%22StartDate%22%3A%20%7B%22gte%22%3A%20%222021-06-01T00%3A00%3A00.000Z%22%7D%7D%2C%20%22offset%22%3A%200%2C%20%22limit%22%3A%20100%2C%20%22order%22%3A%20%22id%20DESC%22%20%7D
Is there a python function to do this?
This is URL encoding.
URLs cannot contain a lot of different special characters, such as spaces (a space would be %20 in URL encoding).
Note that URL encoding is quite easy to recognize once you know it exists, due to the %xx pattern.
The urllib has functions to deal with encoding/decoding this.
To create an URL encoded string use urllib.parse.quote(string). Relevant docs here...
Example
from urllib.parse import quote
jsonstring = '{"where": {"StartDate": {"gte": "2021-01-01T00:00:00.000Z"}}, "offset": 0, "limit": 100, "order": "id DESC" }'
urlencoded = quote(jsonstring)
print(urlencoded)
# output
# %7B%22where%22%3A%20%7B%22StartDate%22%3A%20%7B%22gte%22%3A%20%222021-01-01T00%3A00%3A00.000Z%22%7D%7D%2C%20%22offset%22%3A%200%2C%20%22limit%22%3A%20100%2C%20%22order%22%3A%20%22id%20DESC%22%20%7D
You are looking for URL encoding. If you already have the JSON encoded in a variable as such, then just encode it to the filter parameter:
from urllib.parse import urlencode, quote
base_url = "..."
filter_string = """{"where": {"StartDate": {"gte": "2021-01-01T00:00:00.000Z"}}, "offset": 0, "limit": 100, "order": "id DESC" }"""
query = urlencode({"filter": filter_string}, quote_via=quote)
url = f"{base_url}?{query}"
Now, I expect that the JSON is probably coming from a Python data structure. You can use the dumps function from json to handle that encoding:
from urllib.parse import urlencode, quote
import json
base_url = "..."
data = {
"where": {
"StartDate": {
"gte": "2021-01-01T00:00:00.000Z"
}
},
"offset": 0,
"limit": 100,
"order": "id DESC"
}
filter_string = json.dumps(data)
query = urlencode({"filter": filter_string}, quote_via=quote)
url = f"{base_url}?{query}"
And you have the URL to call to in the variable url.
Related
How to transfer payload as a string in requests python?
my code:
def send():
url = "url"
cookies = {
"csrf_token":"",
"refresh_token":"",
"access_token":""
}
data = "id%5B%5D=52626995&id%5B%5D=52627067&result_element%5BNAME%5D=%D0%90%D1%80%D0%BA%D0%B0%D0%B4%D0%B8%D0%B9&result_element%5BMAIN_USER_ID%5D=8272126&result_element%5BTAGS%5D%5B%5D=559091&result_element%5BTAGS%5D%5B%5D=559091&result_element%5Bcfv%5D%5B664393%5D%5B%5D=%7B%22DESCRIPTION%22%3A%22WORK%22%2C%22VALUE%22%3A%2271111111111%22%7D&result_element%5Bcfv%5D%5B664393%5D%5B%5D=%7B%22DESCRIPTION%22%3A%22WORK%22%2C%22VALUE%22%3A%2271111111111%22%7D&result_element%5Bcfv%5D%5B1262415%5D=12&result_element%5Bcfv%5D%5B1256527%5D=3&result_element%5Bcfv%5D%5B1272573%5D=817683&result_element%5BLEADS%5D%5B%5D=36375665&result_element%5BID%5D=52627067"
resp = requests.post(url=url, cookies=cookies, data=data)
return resp
But i got error cause data must be dict
They are right. It has to be a dictionary. The data you have here is also an encoded dictionary. Using (https://www.url-encode-decode.com/) you can understand your data better.
id[]=52626995
id[]=52627067
result_element[NAME]=Аркадий
result_element[MAIN_USER_ID]=8272126
result_element[TAGS][]=559091
result_element[TAGS][]=559091
result_element[cfv][664393][]={"DESCRIPTION":"WORK","VALUE":"71111111111"}
result_element[cfv][664393][]={"DESCRIPTION":"WORK","VALUE":"71111111111"}
result_element[cfv][1262415]=12
result_element[cfv][1256527]=3
result_element[cfv][1272573]=817683
result_element[LEADS][]=36375665
result_element[ID]=52627067
As a normal python dictionary, this is the following
{
"id": [52626995, 52627067],
"result_element": {
"NAME": "Аркадий",
"MAIN_USER_ID": 8272126,
"TAGS": [559091, 559091],
"cfv": {
664393: [
{"DESCRIPTION":"WORK","VALUE":"71111111111"},
{"DESCRIPTION":"WORK","VALUE":"71111111111"}],
1262415: 12,
1256527: 3,
1272573: 817683,
},
"LEADS": [36375665],
"ID": 52627067
}
}
So if you have the following code, it should work:
url = "url"
cookies = {
"csrf_token":"",
"refresh_token":"",
"access_token":""
}
data = {"id": [52626995, 52627067],
"result_element": {
"NAME": "Аркадий",
"MAIN_USER_ID": 8272126,
"TAGS": [559091, 559091],
"cfv": {
664393: [
{"DESCRIPTION":"WORK","VALUE":"71111111111"},
{"DESCRIPTION":"WORK","VALUE":"71111111111"}],
1262415: 12,
1256527: 3,
1272573: 817683,
},
"LEADS": [36375665],
"ID": 52627067
}
}
resp = requests.post(url=url, cookies=cookies, data=data)
return resp
I did it manually. First, check whether I have correctly parsed your string.
Check out the requests documentation
data – (optional) Dictionary, list of tuples, bytes, or file-like object to send in the body of the Request.
So only thing you need is to encode your string to bytes:
resp = requests.post(url=url, cookies=cookies, data=data.encode())
The more detailed explanation is that strings in Python 3 are abstract representation of the characters you see. You can use Czech letter "Ř" in Python regardless what encoding is used to represent the characters as bytes in the computer memory.
However, to send it over internet, you have to send it encoded as bytes.
If you would like to get those bytes, you have to specify the encoding that converts characters to bytes representing them. For Czech, the most appropriate is UTF-8 (as for almost anything) or maybe Windows-1250 aka CP-1250:
>>> x = "Ř"
>>> x
'Ř'
>>> x.encode("utf-8")
b'\xc5\x98'
>>> x.encode("cp1250")
b'\xd8'
Plain str.encode without encoding specified uses UTF-8, probably the best choice:
>>> x.encode()
b'\xc5\x98'
With Python I'm pulling a nested json, and I'm seeking to parse it via a loop and write the data to a csv. The structure of the json is below. The values I'm after are in the "view" list, labeled "user_id" and `"message"'
{
"view": [
{
"id": 109205,
"user_id": 6354,
"parent_id": null,
"created_at": "2020-11-03T23:32:49Z",
"updated_at": "2020-11-03T23:32:49Z",
"rating_count": null,
"rating_sum": null,
**"message": "message text",**
"replies": [
# json continues
],
}
After some study and assistance from this helpful tutorial I was able to structure requests like this:
import requests
import json
import pandas as pd
url = "URL"
headers = {'Authorization' : 'Bearer KEY'}
r = requests.get(url, headers=headers)
data = r.json()
print(data['view'][0]['user_id'])
print(data['view'][0]['message'])
Which successfully prints the outputs 6354 and "message test".
Now....how would I approach capturing all the user id's and messages from the json to a csv with Pandas?
I have the following JSON data:
x = df.to_json(orient='records')
print(x)
[{"val":"3760","id":"204","quantity":2},{"val":"8221","id":"220","quantity":8}]
I want to add the data to my REST call, but it results in the following string in the payload (note: the single quotes around the square bracket:
'updateVals': '[{"val":"3760","id":"204","quantity":2},{"val":"8221","id":"220","quantity":8}]'}}
The fact that the JASON values are listed as one big string, the REST call results in an HTTP 400 error.
The code is:
url = 'my_url'
payload = {
'client_id': 'my_id',
'api_key': 'my_key',
"data": {
"uuid": "myUUID",
"timeStamp": "2018-09-12T06:17:48+00:00",
"updateVals": x
}
}
How do I plug the JSON into the REST call? I assume I have to split the string, or maybe there is a more straightforward answer?
Still not sure what you need, JSON is really one big string:
x = [{"val":"3760","id":"204","quantity":2},
{"val":"8221","id":"220","quantity":8}]
>>> json.dumps(x)
>>> '[{"val": "3760", "id": "204", "quantity": 2}, {"val": "8221", "id": "220", "quantity": 8}]'
Here's my function which connects to an API:
def order_summary():
"""Get order summary for a specific order"""
# Oauth2 params
headerKey = api_login()
headers = {'Authorization': headerKey}
# Payload params
payloadOrderSum = {
"domainId": 15,
"domainName": "SGL",
"orderId": 3018361
}
# API response
orderSumResp = requests.post(url + "order/summary", data=payloadOrderSum, headers=headers)
print(orderSumResp.content)
The API expects a JSON array as Payload Params which essentially looks like that:
[
{
"domainId": 0,
"domainName": "string",
"orderId": 0
}
]
The other endpoints I coded for on this API didn't need for the params to be an array so I could just use them as is and send them as a dictionary and it worked.
I've tried a couple things using the JSON library but I can't seem to get it to work. I saw that the JSonEncoder converts lists and tuples to JSON arrays but I couldn't figure it out.
Not sure what other info I could provide but just ask if there are any.
Thanks!
Wrap payloadOrderSum into a list:
payloadOrderSum = {
"domainId": 15,
"domainName": "SGL",
"orderId": 3018361
}
orderSumResp = requests.post(url + "order/summary", json=[payloadOrderSum], headers=headers)
Note that I used json kwarg instead of data (added in version 2.4.2).
dump your dict with json.dumps requests-doc
r = requests.post(url, data=json.dumps(payload))
It could help if you specify what you tried with the JSON library.
However, you might wanna try this if you haven't already done so:
import json
payloadOrderSum = json.dumps(
{
"domainId": 15,
"domainName": "SGL",
"orderId": 3018361
}
)
I've been digging a hole deeper and deeper trying to work out my api script in python. I'm not sure if i'm just doing something wrong, or i'm misunderstanding the way the projection section is to be written.
Here is the information I'll provide.. So we have two ways to get information from our API through JSON or via socketJS. The socket JS code i will provide down the bottom which essentially does the same thing..
What's going wrong is that it doesn't seem to be processing the parameters correctly, all i get is the same values as i would if i have added no filters or projections.. Does anyone have any ideas on what i'm doing wrong? I suspect i'm not using the requests library correctly, however i have looked and i don't seem to find anything for my particular case in the documentation.
Working SocketJS:
{
"address": "service",
"body": {
"action": "getControlers",
"params": {
"filter": {
"deviceClass": {
"$like" : "*mainControllers*"
}
},
"projection": {
"tagValues": {
"IdMap": 1,
"StateDevice": 1
}
},
"limit":1000
}
}
}
equivalent line via API-Rest outside of Python:
https://URLURL/rest/service/controlers?projection={"tagValues":{"StateDevice": 1}}&filter= {"DeviceClass": {"$like" : "*Controlers*"}}
My script is as follows:
import requests
import json
import os
header = {"Authorization": 'access_token *Iputakeyheretomakethingswork*'}
parameters = {"Filter": {"deviceClass": {"$like" : "*Controller*"}},
"Projection": {"tagValues":{"IdStateMap": 1, "stateDevice": 1}}}
response = requests.get("https://urlgoeshere", headers=header, params=parameters)
print(response.status_code)
data = response.json()
with open('data.txt', 'w') as outfile:
json.dump(data, outfile, sort_keys = True, indent = 4,
ensure_ascii = False)
params does not take a nested dictionary structure. Your API is essentially asking for JSON formatted values in the query string, but you are not providing those.
Moreover, your example URL uses lowercased parameter names, your dictionary contains uppercased parameters.
Instead, requests will convert the any container in params to a string by taking each element, before encoding it with URL encoding. For a dictionary, that means only the keys are used; you are essentially producing the following URL:
>>> import requests
>>> parameters = {"Filter": {"deviceClass": {"$like" : "*Controller*"}},
... "Projection": {"tagValues":{"IdStateMap": 1, "stateDevice": 1}}}
>>> prepped = requests.Request('GET', 'http://example.com/', params=parameters).prepare()
>>> prepped.url
'http://example.com/?Filter=deviceClass&Projection=tagValues'
The following would produce the equivalent of your sample URL:
parameters = {
"filter": '{"deviceClass": {"$like": "*Controller*"}}',
"projection": '{"tagValues": {"IdStateMap": 1, "stateDevice": 1}}'
}
Note that I lowercased the keys and the values are just strings. You could use the json.dumps() function to produce those strings from Python dictionaries if need be:
import json
filter = {"deviceClass": {"$like": "*Controller*"}}
projection = {"tagValues": {"IdStateMap": 1, "stateDevice": 1}}
parameters = {
"filter": json.dumps(filter),
"projection": json.dumps(projection),
}
Demo:
>>> parameters = {
... "filter": '{"deviceClass": {"$like" : "*Controller*"}}',
... "projection": '{"tagValues":{"IdStateMap": 1, "stateDevice": 1}}'
... }
>>> prepped = requests.Request('GET', 'http://example.com/', params=parameters).prepare()
>>> prepped.url
'http://example.com/?filter=%7B%22deviceClass%22%3A+%7B%22%24like%22+%3A+%22%2AController%2A%22%7D%7D&projection=%7B%22tagValues%22%3A%7B%22IdStateMap%22%3A+1%2C+%22stateDevice%22%3A+1%7D%7D'
>>> from urllib.parse import urlparse, parse_qsl
>>> parse_qsl(urlparse(prepped.url).query)
[('filter', '{"deviceClass": {"$like" : "*Controller*"}}'), ('projection', '{"tagValues":{"IdStateMap": 1, "stateDevice": 1}}')]