I am writing a python program that calls AWS API Gateway to reach dynamoDB.
And it all works fine when I use python "requests":
r = requests.post("https://xxxxxxx.execute-api.us-east-1.amazonaws.com/Prod/log-entries",
data={'logbookTimestamp': timestamp, 'Name': "Fred"})
Now in order to run this as a lambda function, I want to use "urllib3" instead of "requests"
because urllib3 is included by default in lambda's python. So now I am trying to do the same with urllib3 but can't get it to work. I've read the urllib3 user guide here (https://urllib3.readthedocs.io/en/latest/user-guide.html#json) and it says I need to encode the JSON data before sending it so I've done this:
http = urllib3.PoolManager()
fields = {'logbookTimestamp': timestamp, 'Name': "Fred"}
encoded_fields = json.dumps(fields).encode('utf-8')
link = "https://xxxxxxx.execute-api.us-east-1.amazonaws.com/Prod/log-entries"
r = http.request('POST',
link,
body=encoded_fields,
headers={'Content-Type': 'application/json'}
)
When I look at the output of both in CloudWatch I see that the data are formatted differently.
With requests:
(c084a37e-43d8-464a-9dcf-e40c28922ece) Method request body before transformations: logbookTimestamp=2020%3A12%3A15%3A20%3A11%3A02&Name=Fred
With urllib3:
(9b8d84e9-2403-462c-b25f-945a927d1e66) Method request body before transformations:
{
"logbookTimestamp": "2020:12:15:21:31:21",
"Name": "Fred"
}
and returns the following:
(9b8d84e9-2403-462c-b25f-945a927d1e66) Endpoint response body before transformations:
{
"statusCode": 500,
"headers": {
"Content-Type": "text/html; charset=utf-8",
"Content-Length": "290"
},
"body": "\n500 Internal Server Error\nInternal Server Error\nThe server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.\n"
}
(9b8d84e9-2403-462c-b25f-945a927d1e66) Endpoint response body before transformations: {"statusCode": 500, "headers": {"Content-Type": "text/html; charset=utf-8", "Content-Length": "290"}
I can't figure out how to get the data in the format in "urllib3" that will be accepted like it is with "requests". Any thoughts? Thanks!
If you post your json as data in requests it will be submited like a html form:
"Content-Type":["application/x-www-form-urlencoded"]
If you want it to go as json, you should use
requests.post("https://xxxxxxx.execute-api.us-east-1.amazonaws.com/Prod/log-entries", json={'logbookTimestamp': timestamp, 'Name': "Fred"})
which submits data as:
"Content-Type":["application/json"]
Related
I am trying to post/put to Sheety via Python:
SHEETY_URL = "https://api.sheety.co/gfhgfhjfdghjgfjf/flightDeals/prices/5"
header = {
"Content-Type" : "application/json"
}
params_sheety = {
"price": {
"iataCode": "PLN"
}
}
response_sheety = requests.put(url=SHEETY_URL, params=params_sheety, headers=header)
print(response_sheety)
print(response_sheety.json())
=======================================================
Getting Bad request Error:
<Response [400]> {'errors': [{'detail': "Bad Request. The JSON payload
should be inside a root property called 'price'. Check
https://sheety.co/docs for more details."}]}
The same request works fine with Postman.
When you use params=params_sheety, the parameters are not sent as json.
Use json=params_sheety to send them as json.
This also sets the content-type header to json automatically, so you don't need headers=header.
I'm sure this is a broader question that applies to more than just Reddit, but currently I am attempting to exchange a code for a user access token however I am not understanding how to implement the following steps:
https://github.com/reddit-archive/reddit/wiki/OAuth2#retrieving-the-access-token
If you didn't get an error and the state value checks out,
you may then make a POST request with code to the following URL to retrieve your access token:
https://www.reddit.com/api/v1/access_token
Include the following information in your POST data (NOT as part of the URL)
grant_type=authorization_code&code=CODE&redirect_uri=URI
Okay, so what I did was this:
headers = {
CLIENT_ID: CLIENT_SECRET,
}
r = requests.post(
url="https://www.reddit.com/api/v1/access_token",
data={
"grant_type": "authorization_code",
"code": code,
"redirect_uri": "http://127.0.0.1:5000/callback"
},
headers=headers
)
I think I am failing with the headers, I receive a 429 error, and I don't think I've understood how to arrange the headers correctly as it doesn't clearly explain in the above link.
The "user" is the client_id. The "password" for confidential clients is the client_secret. The "password" for non-confidential clients (installed apps) is an empty string.
CLIENT_ID and CLIENT_SECRET are obviously variables, and they are my Reddit App dev credentials.
EDIT:
I came up with this, it's gross but it seems to work
headers = {
"User-Agent": "MyApp v1.0",
"Authorization": "Basic " + str(base64.b64encode(str.encode(f"{CLIENT_ID}:{CLIENT_SECRET}")))[2:-1],
}
Is there a cleaner way to write that?
Final answer, using an inbuilt method in Python's request:
client_auth = requests.auth.HTTPBasicAuth(CLIENT_ID, CLIENT_SECRET)
r = requests.post(
url="https://www.example.com/api/v1/access_token",
auth=client_auth,
data={
"grant_type": "authorization_code",
"code": code,
"redirect_uri": "http://127.0.0.1:5000/callback"
},
headers={
"User-Agent": "MyApp v1.0",
}
)
I'm trying to trigger a GitLab build using Python Requests. Normally one can kick off a build with a curl command.
Example curl command:
curl -X POST \
-F token=TOKEN \
-F ref=master \
-F "variables[UPLOAD_TO_S3]=true" \
https://gitlab.example.com/api/v4/projects/9/trigger/pipeline
I can get this working using the sh module but I'd prefer using requests. I've tried variations with the following:
data = {
'token': token,
'ref': master,
'variables[UPLOAD_TO_S3]': str(uploadS3),
}
headers = {'Content-Type': 'application/json'}
result = requests.post(_trigger_url, headers=headers, json=data)
I tried with and without the headers param. I've also tried passing the data dict using params, json, files, and data. All keep coming back with 400 or 404 errors. Any suggestions?
The above answer is incomplete as it does not deal with the part that makes this complex that is passing trough variables.
To pass variables around with requests using json data one needs to pass the data in the following structure
data = {
'token': token,
'ref': master,
'variables': [{"key": "UPLOAD_TO_S3", "value": True}, {"key": "S3_SERVER", "value": "https://mys3.example.com"}],
}
result = requests.post(_trigger_url, json=data)
You shouldn't send the Content-Type: "application/json" header as part of your request, and you should only need form encoding, so just pass your data dict as the data argument.
Here's a request to my Gitlab instance that succeeds and triggers a build of my project's master branch.
rsp = requests.post('https://gitlab.instance/api/v4/projects/PROJECT_ID/trigger/pipeline',
data={'token': '125fdsfdf1ab3631d2423434347643', 'ref': 'master'})
Here's the output if I inspect my rsp object in `ipython:
In [3]: rsp
Out[3]: <Response [201]>
You should also be able to trigger the pipeline by sending a POST and including the token and ref in the URL itself.
rsp = requests.post('https://gitlab.instance/api/v4/projects/PROJECT_ID/trigger/pipeline?token=TOKEN&ref=master')
If I pass the Content-Type: "application/json" header, Gitlab responds with an HTTP/400 error.
The first answer doesn't address the "variables" part. The second answer didn't work for me. Here's what I ended up with working:
variables = {
"UPLOAD_TO_S3": True,
"S3_SERVER": "https://mys3.example.com"
}
data = {
"token": token,
"ref": ref_name,
"variables": variables
}
res = requests.post(pipeline_trigger, json=data)
I am using the following filters in Postman to make a POST request in a Web API but I am unable to make a simple POST request in Python with the requests library.
First, I am sending a POST request to this URL (http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets) with the following filters in Postman applied to the Body, with the raw and JSON(application/json) options selected.
Filters in Postman
{
"filter": {
"filters": [
{
"field": "RCA_Assigned_Date",
"operator": "gte",
"value": "2017-05-31 00:00:00"
},
{
"field": "RCA_Assigned_Date",
"operator": "lte",
"value": "2017-06-04 00:00:00"
},
{
"field": "T_Subcategory",
"operator": "neq",
"value": "Temporary Degradation"
},
{
"field": "Issue_Status",
"operator": "neq",
"value": "Queued"
}],
"logic": "and"
}
}
The database where the data is stored is Cassandra and according to the following links Cassandra not equal operator, Cassandra OR operator,
Cassandra Between order by operators, Cassandra does not support the NOT EQUAL TO, OR, BETWEEN operators, so there is no way I can filter the URL with these operators except with AND.
Second, I am using the following code to apply a simple filter with the requests library.
import requests
payload = {'field':'T_Subcategory','operator':'neq','value':'Temporary Degradation'}
url = requests.post("http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets",data=payload)
But what I've got is the complete data of tickets instead of only those that are not temporary degradation.
Third, the system is actually working but we are experiencing a delay of 2-3 mins to see the data. The logic goes as follows: We have 8 users and we want to see all the tickets per user that are not temporary degradation, then we do:
def get_json():
if user_name == "user 001":
with urllib.request.urlopen(
"http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets?user_name=user&001",timeout=15) as url:
complete_data = json.loads(url.read().decode())
elif user_name == "user 002":
with urllib.request.urlopen(
"http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets?user_name=user&002",timeout=15) as url:
complete_data = json.loads(url.read().decode())
return complete_data
def get_tickets_not_temp_degradation(start_date,end_date,complete_):
return Counter([k['user_name'] for k in complete_data if start_date < dateutil.parser.parse(k.get('DateTime')) < end_date and k['T_subcategory'] != 'Temporary Degradation'])
Basically, we get the whole set of tickets from the current and last year, then we let Python to filter the complete set by user and so far there are only 10 users which means that this process is repeated 10 times and makes me no surprise to discover why we get the delay...
My questions is how can I fix this problem of the requests library? I am using the following link Requests library documentation as a tutorial to make it working but it just seems that my payload is not being read.
Your Postman request is a JSON body. Just reproduce that same body in Python. Your Python code is not sending JSON, nor is it sending the same data as your Postman sample.
For starters, sending a dictionary via the data arguments encodes that dictionary to application/x-www-form-urlencoded form, not JSON. Secondly, you appear to be sending a single filter.
The following code replicates your Postman post exactly:
import requests
filters = {"filter": {
"filters": [{
"field": "RCA_Assigned_Date",
"operator": "gte",
"value": "2017-05-31 00:00:00"
}, {
"field": "RCA_Assigned_Date",
"operator": "lte",
"value": "2017-06-04 00:00:00"
}, {
"field": "T_Subcategory",
"operator": "neq",
"value": "Temporary Degradation"
}, {
"field": "Issue_Status",
"operator": "neq",
"value": "Queued"
}],
"logic": "and"
}}
url = "http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets"
response = requests.post(url, json=filters)
Note that filters is a Python data structure here, and that it is passed to the json keyword argument. Using the latter does two things:
Encode the Python data structure to JSON (producing the exact same JSON value as your raw Postman body value).
Set the Content-Type header to application/json (as you did in your Postman configuration by picking the JSON option in the dropdown menu after picking raw for the body).
requests is otherwise just an HTTP API, it can't make Cassandra do any more than any other HTTP library. The urllib.request.urlopen code sends GET requests, and are trivially translated to requests with:
def get_json():
url = "http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets"
response = requests.get(url, params={'user_name': user}, timeout=15)
return response.json()
I removed the if branching and replaced that with using the params argument, which translates a dictionary of key-value pairs to a correctly encoded URL query (passing in the user name as the user_name key).
Note the json() call on the response; this takes care of decoding JSON data coming back from the server. This still takes long, you are not filtering the Cassandra data much here.
I would recommend using the json attribute instead of data. It handles the dumping for you.
import requests
data = {'user_name':'user&001'}
headers = {'Content-Type': 'application/json', 'Accept': 'application/json'}
url = "http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets/"
r = requests.post(url, headers=headers, json=data)
Update, answer for question 3. Is there a reason you are using urllib? I’d use python requests as well for this request.
import requests
def get_json():
r = requests.get("http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets”, params={"user_name": user_name.replace(" ", "&")})
return r.json
# not sure what you’re doing here, more context/code example would help
def get_tickets_not_temp_degradation(start_date, end_date, complete_):
return Counter([k['user_name'] for k in complete_data if start_date < dateutil.parser.parse(k.get('DateTime')) < end_date and k['T_subcategory'] != 'Temporary Degradation'])
Also, is the username really supposed to be user+001 and not user&001 or user 001?
I think, you can use requests library as follows:
import requests
import json
payload = {'field':'T_Subcategory','operator':'neq','value':'Temporary Degradation'}
url = requests.post("http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets",data=json.dumps(payload))
You are sending user in url, use it through post, but its depend upon how end points are implemented. You can try the below code :
import requests
from json import dumps
data = {'user_name':'user&001'}
headers = {'Content-Type': 'application/json', 'Accept': 'application/json'}
url = "http://10.61.202.98:8081/T/a/api/rows/cat/ect/tickets/"
r = requests.post(url, headers=headers, data=dumps(data))
I am trying to send a request to Linkedin's rest share api. I have been receiving this error message:
{
"errorCode": 0,
"message": "Can not parse JSON share document.\nRequest body:\n\nError:\nnull",
"requestId": "ETX9XFEI7N",
"status": 400,
"timestamp": 1437910620120
}
The request is send through the following python code:
import requests,json
auth_token = "some auth token"
url = "https://api.linkedin.com/v1/people/~/shares?format=json&oauth2_access_token="+auth_token
headers = {'content-type': 'application/x-www-form-urlencoded','x-li-format':'json'}
data = {
"comment":"Check out developer.linkedin.com!",
"content":{
"title": "LinkedIn Developers Resources",
"description": "Leverage LinkedIn's APIs to maximize engagement",
"submitted-url": "https://developer.linkedin.com",
"submitted-image-url": "https://example.com/logo.png"
},
"visibility":{
"code": "anyone"
}
}
response = requests.post( url , json= data , headers=headers )
return HttpResponse( response )
I made sure that I followed all the instructions in their documentation and can't find the mistake I am making.
Note: i have tried json=data and data=data both are not working
Remove content-type from the headers dictionary.
requests sets the correct Content-Type when using the json keyword argument.
You have three basic problems:
Please read the documentation on oauth2; because you are not passing in the token correctly.
The share URL does not take a oauth2_token argument.
You have the wrong content-type header.