Develop Reference

Python using Requests to cURL a file upload

I am trying to translate a specific curl method into Python's requests module to upload a file to to an api. My standard method that works for non-file requests looks like this:
import requests
requestObject = requests.Session()
standard_headers = {header1:headerValue1,header2:headerValue2}
payload = {key1:value1,key2:value2}
url = 'https://myUrl.com/apiCall'
requestObject.post(url,headers=standard_headers, json=payload)
This works for non-file requests that I need to make to the API. However for file uploads, the API documentation shows a method using curl:
curl -XPOST -H 'header1' -H 'header2 'https://myUrl.com/apiCall' \
-F 'attachment=#/path/to/my/file' \
-F '_json=<-;type=application/json' << _EOF_
{
"key1":"keyValue1",
"key2":"keyValue2"
}
_EOF_
I tested the curl command and it works successfully.
My question is how do I translate that curl method using the << _EOF_ method in Python requests. One idea I had was simply to use the 'files' option in the requests module:
requestObject = requests.Session()
standard_headers = {header1:headerValue1,header2:headerValue2}
payload = {key1:keyValue1,key2:keyValue2}
url = 'https://myUrl.com/apiCall'
file_to_upload = {'filename': open('/path/to/my/file', 'rb')}
requestObject.post(url,headers=standard_headers, files=file_to_upload, json=payload)
But that does not seem to work as the necessary json parameters (the values in payload) do not appear to get passed to the file upload
I also tried specifying the json parameters directly into the file_to_upload variable:
requestObject = requests.Session()
standard_headers = {header1:headerValue1,header2:headerValue2}
url = 'https://myUrl.com/apiCall'
file_to_upload = {'attachment': open('/path/to/my/file', 'rb'),'{"key1":"keyValue1","key2":"keyValue2"}'}
requestObject.post(url,headers=standard_headers, files=file_to_upload)
Similar result, it seems as though I am not passing the necessary json values correctly. I tried a few other ways but I am overlooking something. Any insight into how I should structure my request is appreciated.

Ok I managed to get it to work and posting for anyone who might need help in the future.
The trick was to include the _json key in the data field. My code ended up looking like so:
import requests
requestObject = requests.Session()
standard_headers = {header1:headerValue1,header2:headerValue2}
json_fields = json.dumps({
"key1": "key1Value",
"key2": "key2Value"
})
payload = {"_json":json_fields)
file = {"attachment": /path/to/my/file}
url = 'https://myUrl.com/apiCall'
requestObject.post(url,headers=standard_headers, files=file, data=payload)
Hope that helps some future person.

Http Basic Auth with RFC2045-MIME variant of Base64 in python

I am currently working with and API that requires "RFC2045-MIME variant of Base64, except not limited to 76 char/line" this seems to be different from the normal basic auth used in the requests library. Curious if anyone else has come across this and been able to solve it? I imagine I will have to write a function to do this encoding and build the header manually.

Yes, you can create a function as shown below where the headers for the request are created manually.
For this implementation, you will need these variables:
API_PUBLIC_KEY,
API_SECRET_KEY,
host_url,
endpoint (where the API is supposed to hit).
import base64
import requests
def create_request():
auth_header = base64.b64encode(bytes(f'{API_PUBLIC_KEY}:{API_SECRET_KEY}'.encode('ascii'))).decode('utf-8')
headers = {
'Host': host_url,
'Authorization': f'Basic {auth_header}',
'Content-Type': 'application/json'
}
api_endpoint = f'https://{host_url}/{endpoint}'
data = {}
response = requests.request("POST", api_endpoint, headers=headers, data=data)
print(response.text.encode('utf8'))

Can someone give a python requests example of uploading a release asset in github?

url = 'https://github.abc.defcom/api/v3/repos/abc/def/releases/401/assets?name=foo.sh'
r = requests.post(url, headers={'Content-Type':'application/binary'}, data=open('sometext.txt','r'), auth=('user','password'))
This is giving me
>>> r.text
u'{"message":"Not Found","documentation_url":"https://developer.github.com/enterprise/2.4/v3"}'
Where am I going wrong?

So I'll preface this with the advice that if you use a library it's as easy as:
from github3 import GitHubEnterprise
gh = GitHubEnterprise(token=my_token)
repository = gh.repository('abc', 'def')
release = repository.release(id=401)
asset = release.upload_asset(content_type='application/binary', name='foo.sh', asset=open('sometext.txt', 'rb'))
With that in mind, I'll also preface this with "application/binary" is not a real media type (see: https://www.iana.org/assignments/media-types/media-types.xhtml)
Next, if you read the documentation, you'll notice that GitHub requires clients that have real SNI (Server Name Indication), so depending on your version of Python, you may also have to install pyOpenSSL, pyasn1, and ndg-httpsclient from PyPI.
I'm not sure what the URL looks like for enterprise instances, but for public GitHub, it looks like:
https://uploads.github.com/repos/octocat/Hello-World/releases/1/assets?name=foo.sh
So you're going to have that as url, plus you're going to want your auth credentials (in your case you seem to want to use basic auth). Then you're going to want a valid media-type in the headers, e.g.,
headers = {'Content-Type': 'text/plain'}
And your call would look pretty much exactly correct:
requests.post(url, headers=headers, data=open('file.txt', 'rb'), auth=(username, password))
To get the correct url, you should do:
release = requests.get(release_url, auth=(username, password))
upload_url = release.json().get('upload_url')
Note this is a URITemplate. You'll need to remove the templating or use a library like uritemplate.py to parse it and use it to build your URL for you.
One last reminder, github3.py (the library in the original example) takes care of all of this for you.

APIv3 upload example without any external dependencies
Usage:
GITHUB_TOKEN=<token> ./create-release username/reponame <tag-name> <path-to-upload>
Script:
#!/usr/bin/env python3
import json
import os
import sys
from urllib.parse import urlencode
from urllib.request import Request, urlopen
repo = sys.argv[1]
tag = sys.argv[2]
upload_file = sys.argv[3]
token = os.environ['GITHUB_TOKEN']
url_template = 'https://{}.github.com/repos/' + repo + '/releases'
# Create.
_json = json.loads(urlopen(Request(
url_template.format('api'),
json.dumps({
'tag_name': tag,
'name': tag,
'prerelease': True,
}).encode(),
headers={
'Accept': 'application/vnd.github.v3+json',
'Authorization': 'token ' + token,
},
)).read().decode())
release_id = _json['id']
# Upload.
with open(upload_file, 'br') as myfile:
content = myfile.read()
_json = json.loads(urlopen(Request(
url_template.format('uploads') + '/' + str(release_id) + '/assets?' \
+ urlencode({'name': os.path.split(upload_file)[1]}),
content,
headers={
'Accept': 'application/vnd.github.v3+json',
'Authorization': 'token ' + token,
'Content-Type': 'application/zip',
},
)).read().decode())
Superset question with any language: How to release a build artifact asset on GitHub with a script?

Python requests library how to pass Authorization header with single token

I have a request URI and a token. If I use:
curl -s "<MY_URI>" -H "Authorization: TOK:<MY_TOKEN>"
etc., I get a 200 and view the corresponding JSON data.
So, I installed requests and when I attempt to access this resource I get a 403 probably because I do not know the correct syntax to pass that token. Can anyone help me figure it out?
This is what I have:
import sys,socket
import requests
r = requests.get('<MY_URI>','<MY_TOKEN>')
r. status_code
I already tried:
r = requests.get('<MY_URI>',auth=('<MY_TOKEN>'))
r = requests.get('<MY_URI>',auth=('TOK','<MY_TOKEN>'))
r = requests.get('<MY_URI>',headers=('Authorization: TOK:<MY_TOKEN>'))
But none of these work.

In python:
('<MY_TOKEN>')
is equivalent to
'<MY_TOKEN>'
And requests interprets
('TOK', '<MY_TOKEN>')
As you wanting requests to use Basic Authentication and craft an authorization header like so:
'VE9LOjxNWV9UT0tFTj4K'
Which is the base64 representation of 'TOK:<MY_TOKEN>'
To pass your own header you pass in a dictionary like so:
r = requests.get('<MY_URI>', headers={'Authorization': 'TOK:<MY_TOKEN>'})

I was looking for something similar and came across this. It looks like in the first option you mentioned
r = requests.get('<MY_URI>', auth=('<MY_TOKEN>'))
"auth" takes two parameters: username and password, so the actual statement should be
r=requests.get('<MY_URI>', auth=('<YOUR_USERNAME>', '<YOUR_PASSWORD>'))
In my case, there was no password, so I left the second parameter in auth field empty as shown below:
r=requests.get('<MY_URI', auth=('MY_USERNAME', ''))
Hope this helps somebody :)

This worked for me:
access_token = #yourAccessTokenHere#
result = requests.post(url,
headers={'Content-Type':'application/json',
'Authorization': 'Bearer {}'.format(access_token)})

You can also set headers for the entire session:
TOKEN = 'abcd0123'
HEADERS = {'Authorization': 'token {}'.format(TOKEN)}
with requests.Session() as s:
s.headers.update(HEADERS)
resp = s.get('http://example.com/')

I found it here, it's working for me with Linkedin:
https://auth0.com/docs/flows/guides/auth-code/call-api-auth-code
The code I used with Linkedin login is:
ref = 'https://api.linkedin.com/v2/me'
headers = {"content-type": "application/json; charset=UTF-8",'Authorization':'Bearer {}'.format(access_token)}
Linkedin_user_info = requests.get(ref1, headers=headers).json()

Requests natively supports basic auth only with user-pass params, not with tokens.
You could, if you wanted, add the following class to have requests support token based basic authentication:
import requests
from base64 import b64encode
class BasicAuthToken(requests.auth.AuthBase):
def __init__(self, token):
self.token = token
def __call__(self, r):
authstr = 'Basic ' + b64encode(('token:' + self.token).encode('utf-8')).decode('utf-8')
r.headers['Authorization'] = authstr
return r
Then, to use it run the following request :
r = requests.get(url, auth=BasicAuthToken(api_token))
An alternative would be to formulate a custom header instead, just as was suggested by other users here.

You can try something like this
r = requests.get(ENDPOINT, params=params, headers={'Authorization': 'Basic %s' % API_KEY})

This worked for me:
r = requests.get('http://127.0.0.1:8000/api/ray/musics/', headers={'Authorization': 'Token 22ec0cc4207ebead1f51dea06ff149342082b190'})
My code uses user generated token.

You have a request needing an authorization maybe you have a result 401.
Suppose your request is like this :
REQ ='https://api.asite.com/something/else/else'
You have your token :
TOKEN = 'fliuzabuvdgfnsuczkncsq12454632'
build your header like this :
HEADER = {'Authorization': f'{TOKEN}'}
and use it like this :
req.get(REQ, headers=HEADER)
display your result like this :
req.get(COACH, headers=HEADER).json()

Is there any way to do HTTP PUT in python

I need to upload some data to a server using HTTP PUT in python. From my brief reading of the urllib2 docs, it only does HTTP POST. Is there any way to do an HTTP PUT in python?

I've used a variety of python HTTP libs in the past, and I've settled on requests as my favourite. Existing libs had pretty useable interfaces, but code can end up being a few lines too long for simple operations. A basic PUT in requests looks like:
payload = {'username': 'bob', 'email': 'bob#bob.com'}
>>> r = requests.put("http://somedomain.org/endpoint", data=payload)
You can then check the response status code with:
r.status_code
or the response with:
r.content
Requests has a lot synactic sugar and shortcuts that'll make your life easier.

import urllib2
opener = urllib2.build_opener(urllib2.HTTPHandler)
request = urllib2.Request('http://example.org', data='your_put_data')
request.add_header('Content-Type', 'your/contenttype')
request.get_method = lambda: 'PUT'
url = opener.open(request)

Httplib seems like a cleaner choice.
import httplib
connection = httplib.HTTPConnection('1.2.3.4:1234')
body_content = 'BODY CONTENT GOES HERE'
connection.request('PUT', '/url/path/to/put/to', body_content)
result = connection.getresponse()
# Now result.status and result.reason contains interesting stuff

You can use the requests library, it simplifies things a lot in comparison to taking the urllib2 approach. First install it from pip:
pip install requests
More on installing requests.
Then setup the put request:
import requests
import json
url = 'https://api.github.com/some/endpoint'
payload = {'some': 'data'}
# Create your header as required
headers = {"content-type": "application/json", "Authorization": "<auth-key>" }
r = requests.put(url, data=json.dumps(payload), headers=headers)
See the quickstart for requests library. I think this is a lot simpler than urllib2 but does require this additional package to be installed and imported.

This was made better in python3 and documented in the stdlib documentation
The urllib.request.Request class gained a method=... parameter in python3.
Some sample usage:
req = urllib.request.Request('https://example.com/', data=b'DATA!', method='PUT')
urllib.request.urlopen(req)

You should have a look at the httplib module. It should let you make whatever sort of HTTP request you want.

I needed to solve this problem too a while back so that I could act as a client for a RESTful API. I settled on httplib2 because it allowed me to send PUT and DELETE in addition to GET and POST. Httplib2 is not part of the standard library but you can easily get it from the cheese shop.

I also recommend httplib2 by Joe Gregario. I use this regularly instead of httplib in the standard lib.

Have you taken a look at put.py? I've used it in the past. You can also just hack up your own request with urllib.

You can of course roll your own with the existing standard libraries at any level from sockets up to tweaking urllib.
http://pycurl.sourceforge.net/
"PyCurl is a Python interface to libcurl."
"libcurl is a free and easy-to-use client-side URL transfer library, ... supports ... HTTP PUT"
"The main drawback with PycURL is that it is a relative thin layer over libcurl without any of those nice Pythonic class hierarchies. This means it has a somewhat steep learning curve unless you are already familiar with libcurl's C API. "

If you want to stay within the standard library, you can subclass urllib2.Request:
import urllib2
class RequestWithMethod(urllib2.Request):
def __init__(self, *args, **kwargs):
self._method = kwargs.pop('method', None)
urllib2.Request.__init__(self, *args, **kwargs)
def get_method(self):
return self._method if self._method else super(RequestWithMethod, self).get_method()
def put_request(url, data):
opener = urllib2.build_opener(urllib2.HTTPHandler)
request = RequestWithMethod(url, method='PUT', data=data)
return opener.open(request)

You can use requests.request
import requests
url = "https://www.example/com/some/url/"
payload="{\"param1\": 1, \"param1\": 2}"
headers = {
'Authorization': '....',
'Content-Type': 'application/json'
}
response = requests.request("PUT", url, headers=headers, data=payload)
print(response.text)

A more proper way of doing this with requests would be:
import requests
payload = {'username': 'bob', 'email': 'bob#bob.com'}
try:
response = requests.put(url="http://somedomain.org/endpoint", data=payload)
response.raise_for_status()
except requests.exceptions.RequestException as e:
print(e)
raise
This raises an exception if there is an error in the HTTP PUT request.

Using urllib3
To do that, you will need to manually encode query parameters in the URL.
>>> import urllib3
>>> http = urllib3.PoolManager()
>>> from urllib.parse import urlencode
>>> encoded_args = urlencode({"name":"Zion","salary":"1123","age":"23"})
>>> url = 'http://dummy.restapiexample.com/api/v1/update/15410' + encoded_args
>>> r = http.request('PUT', url)
>>> import json
>>> json.loads(r.data.decode('utf-8'))
{'status': 'success', 'data': [], 'message': 'Successfully! Record has been updated.'}
Using requests
>>> import requests
>>> r = requests.put('https://httpbin.org/put', data = {'key':'value'})
>>> r.status_code
200