Develop Reference

How to pass a file from a URL via Python to an API?

I'm trying to use an API where you give it a CSV file and it validates it. For this API you pass it a file.
What I'm trying to do is this:
import requests
import urllib3
api_url="https://validation.openbridge.io/dryrun"
file_url = 'http://winterolympicsmedals.com/medals.csv'
http = urllib3.PoolManager()
response = http.request('GET', file_url)
data = response.data.decode('utf-8')
headers = {
# requests won't add a boundary if this header is set when you pass files=
'Content-Type': 'multipart/form-data',
}
response = requests.post(api_url, headers=headers, files=data)
I'm getting an error I don't quite understand:
'cannot encode objects that are not 2-tuples'
I'm fairly new to python, I'm mostly experienced in other languages but what I think the issue is that I'm probably passing the API the data in the file instead of giving it the file.
Any suggestions on what I"m doing wrong and how I can pass that file to the API?

For anyone who stumbles across this off google in the future. I figured out how to do it.
import requests
import urllib3
api_url="https://validation.openbridge.io/dryrun"
file_url="http://winterolympicsmedals.com/medals.csv"
remote_file = urllib3.PoolManager().request('GET', file_url)
response = requests.post(url=api_url,
json={'data': {'attributes': {'is_async': True }}},
files={ "file": remote_file},
allow_redirects=False)
if response.ok:
print("Upload completed successfully!")
print()
print("Status Code: ")
print(response.status_code)
print()
print(response.headers['Location'])
print()
else:
print("Something went wrong!")
print()
print("Status Code: ")
print(response.status_code)
print()
print(response.text)
print()

TypeError: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str while trying to pass PUT request in python

I am trying to pass a list of dictionaries(strings) to a for a put request. I am getting this error:
TypeError: POST data should be bytes, an iterable of bytes.
Is this the right way to make a put request with list of dictionaries(strings) in python.
list looks like the following:
list1 = ['{"id" : "","email" : "John#fullcontact.com","fullName": "John Lorang"}', '{"id" : "","email" : "Lola#fullcontact.com","fullName": "Lola Dsilva"}']
myData = json.dumps(list1)
myRestRequestObj = urllib.request.Request(url,myData)
myRestRequestObj.add_header('Content-Type','application/json')
myRestRequestObj.add_header('Authorization','Basic %s')
myRestRequestObj.get_method = lambda : 'PUT'
try:
myRestRequestResponse = urllib.request.urlopen(myRestRequestObj)
except urllib.error.URLError as e:
print(e.reason)

As you said in a comment, you cannot use requests (that's pretty sad to hear!), so I did another snippet using urllib (the short answer: you must .encode('utf-8') json.dumps and decode('utf-8') response.read()):
import urllib.request
import urllib.error
import json
url = 'http://httpbin.org/put'
token = 'jwtToken'
list1 = ['{"id" : "","email" : "John#fullcontact.com","fullName": "John Lorang"}', '{"id" : "","email" : "Lola#fullcontact.com","fullName": "Lola Dsilva"}']
# Request needs bytes, so we have to encode it
params = json.dumps(list1).encode('utf-8')
headers = {
'Content-Type': 'application/json',
'Authorization': 'Basic {token}'.format(token=token)
}
# Let's try to create our request with data, headers and method
try:
request = urllib.request.Request(url, data=params, headers=headers, method='PUT')
except urllib.error.URLError as e:
# Unable to create our request, here the reason
print("Unable to create youro request: {error}".format(error=str(e)))
else:
# We did create our request, let's try to use it
try:
response = urllib.request.urlopen(request)
except urllib.error.HTTPError as e:
# An HTTP error occured, here the reason
print("HTTP Error: {error}".format(error=str(e)))
except Exception as e:
# We got another reason, here the reason
print("An error occured while trying to put {url}: {error}".format(
url=url,
error=str(e)
))
else:
# We are printing the result
# We must decode it because response.read() returns a bytes string
print(response.read().decode('utf-8'))
I did try to add some comments. I hope this solution help you!
To help you learn a better way to learn python, you should read the Style Guide for Python Code

I will suppose you can use the requests module (pip install requests).
requests is a simple yet powerful HTTP libraby for Python.
import json
import requests
my_data = json.dumps(list1)
headers = {
'Authorization': 'Basic {token}'.format(token=your_token)
}
response = requests.put(url, headers=headers, json=my_data)
print("Status code: {status}".format(status=response.status_code))
print("raw response: {raw_response}".format(
raw_response=response.text
)
print("json response: {json_response}".format(
json_response=response.json()
)

How can I post a PDF to AWS lambda

I have AWS Lambda set up.
def lambda_handler(event, context):
return {
'statusCode': 200,
'body': json.dumps(event)
}
I would like to POST in a PDF file so that I can operate on it in my lambda function.
Here is my POST code
import requests
headers = {
'X-API-KEY':'1234',
'Content-type': 'multipart/form-data'}
files = {
'document': open('my.pdf', 'rb')
}
r = requests.post(url, files=files, headers=headers)
display(r)
display(r.text)
I am getting the error:
<Response [400]>
'{"message": "Could not parse request body into json: Unexpected character (\\\'-\\\' (code 45)) in numeric value: expected digit (0-9) to follow minus sign, for valid numeric value
How can I POST over my PDF and be able to properly send over my PDF and access it in Lambda?
Note:
I am successful if I do this:
payload = '{"key1": "val1","key2": 22,"key3": 15,"key4": "val4"}'
r = requests.post(url = URL, data=payload, headers=HEADERS)
It is just the PDF part which I can't get

I figured it out. Took me a ton of time but I think I got it. Essentially it's all about encoding and decoding as bytes. Didn't have to touch the API Gateway at all.
Request:
HEADERS = {'X-API-KEY': '12345'}
data = '{"body" : "%s"}' % base64.b64encode(open(path, 'rb').read())
r = requests.post(url, data=data, headers=HEADERS)
In lambda
from io import BytesIO
def lambda_handler(event, context):
pdf64 = event["body"]
# Need this line as it does 'b'b'pdfdatacontent'.
pdf64 = pdf64[2:].encode('utf-8')
buffer = BytesIO()
content = base64.b64decode(pdf64)
buffer.write(content)

I found this worked quite well for me:
Request
import requests
file_loc = 'path/to/test.pdf'
data = open(file_loc,'rb').read() #this is a bytes object
r = requests.post(url, data=data)
r.ok #returns True (also a good idea to check r.text
#one-liner
requests.post(url, data=open(file_loc,'rb').read())
Lambda - Python3.8
import io, base64
body = event["body"]
attachment = base64.b64decode(body.encode()) #this is a bytes object
buff = io.BytesIO(attachment) #this is now useable - read/write etc.
#one-liner
buff = io.BytesIO(base64.b64decode(event["body"].encode()))
Not quite sure why, but for me base64 encoding (even with urlsafe) in the original request corrupted the file and it was no longer recognised as a PDF in Lambda, so the OP's answer didn't work for me.

Python - Microsoft Cognitive Verify API (params)

I'm trying to use the Microsoft Cognitive Verify API with python 2.7: https://dev.projectoxford.ai/docs/services/563879b61984550e40cbbe8d/operations/563879b61984550f3039523a
The code is:
import httplib, urllib, base64
headers = {
# Request headers
'Content-Type': 'application/json',
'Ocp-Apim-Subscription-Key': 'my key',
}
params = '{\'faceId1\': \'URL.jpg\',\'faceId2\': \'URL.jpg.jpg\'}'
try:
conn = httplib.HTTPSConnection('api.projectoxford.ai')
conn.request("POST", "/face/v1.0/verify?%s" % params, "{body}", headers)
response = conn.getresponse()
data = response.read()
print(data)
conn.close()
except Exception as e:
print("[Errno {0}] {1}".format(e.errno, e.strerror))
I tried letting the conn.request line like this:
conn.request("POST", "/face/v1.0/verify?%s" % params, "", headers)
The error is:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Bad Request</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Bad Request</h2>
<hr><p>HTTP Error 400. The request is badly formed.</p>
</BODY></HTML>
I alrealy tried to follow and make work the following codes:
https://github.com/Microsoft/Cognitive-Emotion-Python/blob/master/Jupyter%20Notebook/Emotion%20Analysis%20Example.ipynb
Using Project Oxford's Emotion API
However I just can't make this one work. I guess there is something wrong with the params or body argument.
Any help is very appreciated.

You can refer to this question.
Obviously you did not understand the code. "{body}" means you should replace it with your body which contains your request url, just like the site says:
So you can use this api this way:
body = {
"url": "http://example.com/1.jpg"
}
…………
conn = httplib.HTTPSConnection('api.projectoxford.ai')
conn.request("POST", "/face/v1.0/detect?%s" % params, str(body), headers)

Dawid's comment looks like it should fix it (double quoting), try this for python 2.7:
import requests
url = "https://api.projectoxford.ai/face/v1.0/verify"
payload = "{\n \"faceId1\":\"A Face ID\",\n \"faceId2\":\"A Face ID\"\n}"
headers = {
'ocp-apim-subscription-key': "KEY_HERE",
'content-type': "application/json"
}
response = requests.request("POST", url, data=payload, headers=headers)
print(response.text)
for python 3:
import http.client
conn = http.client.HTTPSConnection("api.projectoxford.ai")
payload = "{\n\"faceId1\": \"A Face ID\",\n\"faceId2\": \"Another Face ID\"\n}"
headers = {
'ocp-apim-subscription-key': "keyHere",
'content-type': "application/json"
}
conn.request("POST", "/face/v1.0/verify", payload, headers)
res = conn.getresponse()
data = res.read()

There are a couple issues with your script:
You must pass face Ids and not URLs or file objects to the REST API.
You must correctly formulate the HTTP request.
However, you may find it easier to use the Python API and not the REST API. For example, once you have the face ids, you can just run result = CF.face.verify(faceid1, another_face_id=faceid2) instead of worrying about setting up the correct POST request.
You will probably need to install cognitive_face with pip. I use this API to get the face Ids for some bonus instruction.
To make this simpler, let's assume you have img1.jpg and img2.jpg on disk.
Here is an example using the REST API:
import cognitive_face as CF
from io import BytesIO
import json
import http.client
# Setup
KEY = "your subscription key"
# Get Face Ids
def get_face_id(img_file):
f = open(img_file, 'rb')
data = f.read()
f.close()
faces = CF.face.detect(BytesIO(data))
if len(faces) != 1:
raise RuntimeError('Too many faces!')
face_id = faces[0]['faceId']
return face_id
# Initialize API
CF.Key.set(KEY)
faceId1 = get_face_id('img1.jpg')
faceId2 = get_face_id('img2.jpg')
# Now that we have face ids, we can setup our request
headers = {
# Request headers
'Content-Type': 'application/json',
'Ocp-Apim-Subscription-Key': KEY
}
params = {
'faceId1': faceId1,
'faceId2': faceId2
}
# The Content-Type in the header specifies that the body will
# be json so convert params to json
json_body = json.dumps(params)
try:
conn = httplib.HTTPSConnection('https://eastus.api.cognitive.microsoft.com')
conn.request("POST", "/face/v1.0/verify", body=json_body, headers=headers)
response = conn.getresponse()
data = response.read()
data = json.loads(data)
print(data)
conn.close()
except Exception as e:
print("[Errno {0}] {1}".format(e.errno, e.strerror))

Get variable outside exception Python

I am calling an API with the urllib. When something is not as expected, the API throws an error at the user (E.G. HTTP Error 415: Unsupported Media Type). But next to that, the API returns a JSON with more information. I would like to pass that json to the exception and parse it there, so I can give information to the user about the error.
Is that possible? And if, how is it done?
Extra info:
Error: HTTPError
--EDIT--
On request, here is some code (I want to read resp in the exception):
def _sendpost(url, data=None, filetype=None):
try:
global _auth
req = urllib.request.Request(url, data)
req.add_header('User-Agent', _useragent)
req.add_header('Authorization', 'Bearer ' + _auth['access_token'])
if filetype is not None:
req.add_header('Content-Type', filetype)
resp = urllib.request.urlopen(req, data)
data = json.loads(resp.read().decode('utf-8'), object_pairs_hook=OrderedDict)
except urllib.error.HTTPError as e:
print(e)
return data
--EDIT 2--
I do not want to use extra library's/modules. As I do not control the target machines.

Code
import urllib.request
import urllib.error
try:
request = urllib.request.urlopen('https://api.gutefrage.net')
response = urllib.urlopen(request)
except urllib.error.HTTPError as e:
error_message = e.read()
print(error_message)
Output
b'{"error":{"message":"X-Api-Key header is missing or invalid","type":"API_REQUEST_FORBIDDEN"}}'
Not asked but with module json you could convert it to dict via
import json
json.loads(error_message.decode("utf-8"))
Which gives you the dict out of the byte string.

If you're stuck with using urllib, then you can use the error to read the text of the response, and load that into JSON.
from urllib import request, error
import json
try:
req = urllib.request.Request(url, data)
req.add_header('User-Agent', _useragent)
req.add_header('Authorization', 'Bearer ' + _auth['access_token'])
if filetype is not None:
req.add_header('Content-Type', filetype)
resp = urllib.request.urlopen(req, data)
data = json.loads(resp.read().decode('utf-8'), object_pairs_hook=OrderedDict)
except error.HTTPError as e:
json_response = json.loads(e.read().decode('utf-8'))
If you're not stuck to urllib, I would highly recommend you use the requests module instead of urllib. With that, you can have something like this instead:
response = requests.get("http://www.example.com/api/action")
if response.status_code == 415:
response_json = response.json()
requests doesn't throw an exception when it encounters a non-2xx series response code; instead it returns the response anyway with the status code added.
You can also add headers and parameters to these requests:
headers = {
'User-Agent': _useragent,
'Authorization': 'Bearer ' + _auth['access_token']
}
response = requests.get("http://www.example.com/api/action", headers=headers)