Develop Reference

Response content shorter than Content-Length

I'm doing a FastAPI app, with a function that authenticates to a CouchDB instance. In order to request Couchdb, I use the (yet unmaintained) library python-couchdb.
Here is the relevant portion of code that illustrates my issue:
from fastapi import FastAPI
from couchdb import http
resource = http.Resource(url, http.Session())
from couchdb import Unauthorized
from pydantic import BaseModel
app = FastAPI()
class RegisteredUser(BaseModel):
email: str
password: str
#app.post("/login")
async def log_user(user: RegisteredUser):
# some email format verifications here
# ...
try:
status, headers, _ = resource.post_json('_session', {
'name': user.email,
'password': user.password,
})
except Exception as e:
if isinstance(e, Unauthorized):
return 403
else:
return 500
# tests
print(headers)
The headers look like:
Cache-Control: must-revalidate
Content-Length: 54
Content-Type: application/json
Date: Sat, 08 Aug 2020 19:19:49 GMT
Server: CouchDB/3.1.0 (Erlang OTP/22)
Set-Cookie: AuthSession=am9zZWJvdmVAam9zZWJvdmUuY29tOjVGMkVGQUQ2Op-UUD22VvdxYzbMNp92e30Er_z0; Version=1; Expires=Sat, 08-Aug-2020 20:59:50 GMT; Max-Age=6000; Path=/; HttpOnly
At this point (if no error raised), I'd like to send back to the (browser) client the cookie that CouchDb provides. Something like:
...
# tests
print(headers)
if status == 200 and 'Set-Cookie' in headers:
return JSONResponse(content=True, headers=headers)
else:
return status
I'm not used to sessions-cookie and I'm not sure if I should send back the full headers or just the headers['Set-Cookie'] part
Whatever I do, I end up with the same error message
RuntimeError: Response content shorter than Content-Length
Would you mine to explain to me what the error is saying, and how I can solve my case? Ty!
Found this SO thread but no clue neither FastAPI middleware peeking into responses
Here is one solution:
if status == 200 and 'Set-Cookie' in headers:
return JSONResponse(content=True, headers={
'Set-Cookie' : headers['Set-Cookie']
})
else:
return status
Ty #HernánAlarcón

Discrepency between curl results and Python requests results

I am querying the New Relic API, and trying to pull CPU utilization fron the metrics that they provide. When I run the following curl command (after exporting the correct proxy settings), I see the following information (which contains the percentage value that I want) -
curl -X GET "https://api.newrelic.com/v2/applications/140456413/hosts/21044947/metrics/data.json" -H "X-Api-Key:myapikey" -i -d 'names[]=CPU/User+Time&values[]=percent&summarize=true&from=2018-07-15 16:22:30&to=2018-07-16 16:22:30'
HTTP/1.0 200 Connection Established
Proxy-agent: Apache
HTTP/1.1 200 OK
Server: openresty
Date: Tue, 17 Jul 2018 11:53:22 GMT
Content-Type: application/json
Content-Length: 290
Connection: keep-alive
Status: 200 OK
X-UA-Compatible: IE=Edge,chrome=1
ETag: "a35b451e27a49d0d5f4e16715429a17d"
Cache-Control: max-age=0, private, must-revalidate
X-Request-Id: f4f8675f095aba80d5e089fbcbf1b172
X-Runtime: 0.168283
X-Rack-Cache: miss
{"metric_data":{"from":"2018-07-15T16:22:30+00:00","to":"2018-07-16T16:22:30+00:00","metrics_not_found":[],"metrics_found":["CPU/User Time"],"metrics":[{"name":"CPU/User Time","timeslices":[{"from":"2018-07-15T16:22:00+00:00","to":"2018-07-16T16:22:00+00:00","values":{"percent":9.52}}]}]}}
However, when I try to implement it inside of the Python requests module, the "Percent" value that I am interested in seeing is returning 0. This is my code to call it -
options = {"names[]": "CPU/User+Time", "values[]": "percent", "summarize": "true", "from": str(end_date), "to": str(start_date - datetime.timedelta(hours=6))}
path = "applications/140456413/hosts/" + server_id + "/metrics/data.json"
api_key = "myapikey"
headers = {'X-Api-Key': api_key}
url = "https://api.newrelic.com/v2/" + path
r = requests.get(url, headers=headers, data=options, proxies=myproxy.proxies)
This is what I get instead (notice the percent value is now 0) -
{u'metric_data': {u'metrics': [{u'timeslices': [{u'values': {u'percent': 0}, u'to': u'2018-07-17T01:35:00+00:00', u'from': u'2018-07-16T07:35:00+00:00'}], u'name': u'CPU/User+Time'}], u'to': u'2018-07-17T01:35:59+00:00', u'metrics_found': [u'CPU/User+Time'], u'from': u'2018-07-16T07:35:59+00:00', u'metrics_not_found': []}}
How can I adjust the python request to match the same output as the curl command? We were originally passing in options inside of the "options" variable using ='s instead of key/value pairs, but the requests module would not process them in this format.
These were the docs pages I referencesd -
https://docs.newrelic.com/docs/apis/rest-api-v2/requirements/specifying-time-range-v2
https://docs.newrelic.com/docs/apis/rest-api-v2/application-examples-v2/get-average-cpu-usage-host-app
https://docs.newrelic.com/docs/apis/rest-api-v2/getting-started/introduction-new-relic-rest-api-v2#examples
Thanks.

Change the options a bit:
"values[]": "percent"
to:
"values": ["percent"]

Post request failing with python httplib2, works with curl

I'm trying to request access tokens from the fitbit API, but it keeps returning 401 Unauthorized status, even though I configure the request identical to a corresponding curl query - which succeeds. The error message returned says: "errorType":"invalid_client","message":"Invalid authorization header format. Is there some nuance of how httplib2 builds its requests that is throwing me off here?...
(Working) curl query:
curl -X POST -i
-H 'Authorization: Basic <LONG_CODE>'
-H 'Content-Type: application/x-www-form-urlencoded'
-d "clientId=<CLIENT_ID>"
-d "grant_type=authorization_code"
-d "redirect_uri=http%3A%2F%2F127.0.0.1%3A5000%2Ffitbit-callback"
-d "code=<AUTHORIZATION_GRANT_CODE>"
https://api.fitbit.com/oauth2/token
Non-working python request (edited):
TOKEN_URL = 'https://api.fitbit.com'/oauth2/token'
CALLBACK_URI = 'http://127.0.0.1:5000/fitbit-callback'
auth_header = base64.b64encode(bytes(<CLIENT_ID> + ':' + <CLIENT_SECRET>, 'utf-8'))
headers = {
'Authorization': 'Basic %s' % auth_header,
'Content-Type' : 'application/x-www-form-urlencoded'
}
params = {
'client_id': <CLIENT_ID>,
'grant_type': 'authorization_code',
'redirect_uri': CALLBACK_URI,
'code': <AUTHORIZATION_GRANT_CODE>
}
urlparams = urlencode(params)
resp, content = h.request(TOKEN_URL,
'POST',
urlparams,
headers)
Not evident from code:
the auth_header-variable in python matches <LONG_CODE>
Terminal response after python3 fitbit.py:
send: b"POST /oauth2/token HTTP/1.1\r\nHost: api.fitbit.com\r\nContent-Length: 153\r\nauthorization: Basic b'<LONG_CODE>'\r\ncontent-type: application/x-www-form-urlencoded\r\nuser-agent: Python-httplib2/0.10.3 (gzip)\r\naccept-encoding: gzip, deflate\r\n\r\n"
send: b'client_id=<CLIENT_ID>&grant_type=authorization_code&redirect_uri=http%3A%2F%2F127.0.0.1%3A5000%2Ffitbit-callback&code=<AUTHORIZATION_GRANT_CODE>'
reply: 'HTTP/1.1 401 Unauthorized\r\n'
header: Date header: Content-Type header: Transfer-Encoding header: Connection header: Cache-control header: WWW-Authenticate header: Content-Language header: Content-Encoding header: Vary header: X-Frame-Options header: Server header: CF-RAY
Running print(content):
b'{"errors":[{"errorType":"invalid_client","message":"Invalid authorization header format. Visit https://dev.fitbit.com/docs/oauth2 for more information on the Fitbit Web API authorization process."}],"success":false}'

Well this is awkward. I didn't notice it before I directed the requests towards a request capture service that let me analyze them (like runscope), but I seem to simply have missed the b'<LONG_CODE>' formatting in the python example. This fixed it:
auth_header.decode("utf-8")
Maybe this question should be deleted, if it is unlikely to help anyone else...

Spotify API: add tracks to playlist: error parsing JSON

I'm trying to bulk add songs to a playlist. I cannot for the life of me find the difference between the API documentation and what I am sending, but it errors anyway. Here is the function I'm using and how I'm calling it:
def addToPlaylist(songs, playlistUrl, positions = None):
data = { 'uris': songs }
if position != None:
data.update({ 'position': position })
headers = authHeader.copy()
headers.update({'Content-Type': 'application/json'})
print(headers)
print(json.dumps(data))
req = requests.post(playlistUrl, headers = headers, data = json.dumps(data))
if req.status_code != 201:
print('Error: Request returned status code {}. Returned: {}'.format(req.status_code, req.text))
songs = ["spotify:track:1i1fxkWeaMmKEB4T7zqbzK", "spotify:track:2VKqMKKCFhTPczQb10TMKB", "spotify:track:7Gl9cKtVjRN6KHNMfV1gD3"]
url = "https://api.spotify.com/v1/users/username/playlists/2...Q/tracks"
addToPlaylist(songs, url, 0)
The two debug print()s tell me the following:
{'Accept': 'application/json', 'Content-Type': 'application/json', 'Authorization': 'Bearer BlahblahblA'}
{"position": 0, "uris": ["spotify:track:1i1fxkWeaMmKEB4T7zqbzK", "spotify:track:2VKqMKKCFhTPczQb10TMKB", "spotify:track:7Gl9cKtVjRN6KHNMfV1gD3"]}
The error output tells me this:
Error: Request returned status code 400. Returned: {
"error" : {
"status" : 400,
"message" : "Error parsing JSON."
}
}
When changing the URL to http://localhost:3001 and listening with netcat nc -l -p 3001, I can see the following request:
POST / HTTP/1.1
Host: 0:3001
Accept: application/json
Connection: keep-alive
User-Agent: python-requests/2.11.1
Accept-Encoding: gzip, deflate
Content-Type: application/json
Authorization: Bearer BlahblahblA
Content-Length: 145
{"position": 0, "uris": ["spotify:track:1i1fxkWeaMmKEB4T7zqbzK", "spotify:track:2VKqMKKCFhTPczQb10TMKB", "spotify:track:7Gl9cKtVjRN6KHNMfV1gD3"]}
I might think the body should be URL-encoded, even though that's not what the Content-Type header mentions, but another requests (the DELETE to delete songs from a playlist) works and there I don't use url encoding.
Finally the documentation mentions the following:
POST https://api.spotify.com/v1/users/{user_id}/playlists/{playlist_id}/tracks
[...]
Content-Type: Required if URIs are passed in the request body, otherwise ignored. The content type of the request body: application/json
[...]
uris: array of Spotify URI strings | Optional. A JSON array of the Spotify track URIs to add. For example:
{"uris": ["spotify:track:4iV5W9uYEdYUVa79Axb7Rh",
"spotify:track:1301WleyT98MSxVHPZCA6M"]}
Am I doing something wrong?

While writing this post, it caught my attention that the position parameter was not supposed to be in the request body. It has to be in the URI parameters. Testing this, it works.
In my defense, Spotify's error suggested the JSON data was not syntactically correct, which was not the case. This took entirely too much time.
This works:
if position is not None:
playlistUrl += '?position=' + str(position)
Instead of data.update({'position': position}) (<= this is wrong).

Python POST binary data

I am writing some code to interface with redmine and I need to upload some files as part of the process, but I am not sure how to do a POST request from python containing a binary file.
I am trying to mimic the commands here:
curl --data-binary "#image.png" -H "Content-Type: application/octet-stream" -X POST -u login:password http://redmine/uploads.xml
In python (below), but it does not seem to work. I am not sure if the problem is somehow related to encoding the file or if something is wrong with the headers.
import urllib2, os
FilePath = "C:\somefolder\somefile.7z"
FileData = open(FilePath, "rb")
length = os.path.getsize(FilePath)
password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, 'http://redmine/', 'admin', 'admin')
auth_handler = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
request = urllib2.Request( r'http://redmine/uploads.xml', FileData)
request.add_header('Content-Length', '%d' % length)
request.add_header('Content-Type', 'application/octet-stream')
try:
response = urllib2.urlopen( request)
print response.read()
except urllib2.HTTPError as e:
error_message = e.read()
print error_message
I have access to the server and it looks like a encoding error:
...
invalid byte sequence in UTF-8
Line: 1
Position: 624
Last 80 unconsumed characters:
7z¼¯'ÅÐз2^Ôøë4g¸R<süðí6kĤª¶!»=}jcdjSPúá-º#»ÄAtD»H7Ê!æ½]j):
(further down)
Started POST "/uploads.xml" for 192.168.0.117 at 2013-01-16 09:57:49 -0800
Processing by AttachmentsController#upload as XML
WARNING: Can't verify CSRF token authenticity
Current user: anonymous
Filter chain halted as :authorize_global rendered or redirected
Completed 401 Unauthorized in 13ms (ActiveRecord: 3.1ms)

Basically what you do is correct. Looking at redmine docs you linked to, it seems that suffix after the dot in the url denotes type of posted data (.json for JSON, .xml for XML), which agrees with the response you get - Processing by AttachmentsController#upload as XML. I guess maybe there's a bug in docs and to post binary data you should try using http://redmine/uploads url instead of http://redmine/uploads.xml.
Btw, I highly recommend very good and very popular Requests library for http in Python. It's much better than what's in the standard lib (urllib2). It supports authentication as well but I skipped it for brevity here.
import requests
with open('./x.png', 'rb') as f:
data = f.read()
res = requests.post(url='http://httpbin.org/post',
data=data,
headers={'Content-Type': 'application/octet-stream'})
# let's check if what we sent is what we intended to send...
import json
import base64
assert base64.b64decode(res.json()['data'][len('data:application/octet-stream;base64,'):]) == data
UPDATE
To find out why this works with Requests but not with urllib2 we have to examine the difference in what's being sent. To see this I'm sending traffic to http proxy (Fiddler) running on port 8888:
Using Requests
import requests
data = 'test data'
res = requests.post(url='http://localhost:8888',
data=data,
headers={'Content-Type': 'application/octet-stream'})
we see
POST http://localhost:8888/ HTTP/1.1
Host: localhost:8888
Content-Length: 9
Content-Type: application/octet-stream
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.0.4 CPython/2.7.3 Windows/Vista
test data
and using urllib2
import urllib2
data = 'test data'
req = urllib2.Request('http://localhost:8888', data)
req.add_header('Content-Length', '%d' % len(data))
req.add_header('Content-Type', 'application/octet-stream')
res = urllib2.urlopen(req)
we get
POST http://localhost:8888/ HTTP/1.1
Accept-Encoding: identity
Content-Length: 9
Host: localhost:8888
Content-Type: application/octet-stream
Connection: close
User-Agent: Python-urllib/2.7
test data
I don't see any differences which would warrant different behavior you observe. Having said that it's not uncommon for http servers to inspect User-Agent header and vary behavior based on its value. Try to change headers sent by Requests one by one making them the same as those being sent by urllib2 and see when it stops working.

This has nothing to do with a malformed upload. The HTTP error clearly specifies 401 unauthorized, and tells you the CSRF token is invalid. Try sending a valid CSRF token with the upload.
More about csrf tokens here:
What is a CSRF token ? What is its importance and how does it work?

you need to add Content-Disposition header, smth like this (although I used mod-python here, but principle should be the same):
request.headers_out['Content-Disposition'] = 'attachment; filename=%s' % myfname

You can use unirest, It provides easy method to post request.
`
import unirest
def callback(response):
print "code:"+ str(response.code)
print "******************"
print "headers:"+ str(response.headers)
print "******************"
print "body:"+ str(response.body)
print "******************"
print "raw_body:"+ str(response.raw_body)
# consume async post request
def consumePOSTRequestASync():
params = {'test1':'param1','test2':'param2'}
# we need to pass a dummy variable which is open method
# actually unirest does not provide variable to shift between
# application-x-www-form-urlencoded and
# multipart/form-data
params['dummy'] = open('dummy.txt', 'r')
url = 'http://httpbin.org/post'
headers = {"Accept": "application/json"}
# call get service with headers and params
unirest.post(url, headers = headers,params = params, callback = callback)
# post async request multipart/form-data
consumePOSTRequestASync()