Develop Reference

Upload binary image file using linkedin create share api gives null response using python code but curl command works

Curl command below works perfectly but converted curl command into python code doesn't. However python code return status code 201 but response.text prints nothing.
FYI: I am following this documentation.
Here is Working Curl Command:
curl -i --upload-file "/home/abhay/download.jpeg" --header "Authorization: Bearer xxxxxxxxx" 'https://api.linkedin.com/mediaUpload/C5122AQE4GSagETyxQA/feedshare-uploadedImage/0?ca=vector_feedshare&cn=uploads&m=AQK_YGFfk_crEwAAAWoGPkMNETNZnhoeCKKmD_CKYeasF1NHooQ7pdeZWA&app=5969925&sync=0&v=beta&ut=1DL2-qd0JrroI1'
Converted python code
upload_url = 'https://api.linkedin.com/mediaUpload/C5122AQE4GSagETyxQA/feedshare-uploadedImage/0?ca=vector_feedshare&cn=uploads&m=AQK_YGFfk_crEwAAAWoGPkMNETNZnhoeCKKmD_CKYeasF1NHooQ7pdeZWA&app=5969925&sync=0&v=beta&ut=1DL2-qd0JrroI1'
headers = {
'accept' : 'application/json',
'X-Restli-Protocol-Version': '2.0.0',
'Authorization': 'Bearer xxxxxxxxx',
# 'Content-Type': 'multipart/form-data'
'Content-Type': 'image/jpeg',
}
image = open("/home/abhay/download.jpeg" ,"rb")
files = {'images_file': image }
response = requests.post(upload_url, headers=headers, files=files)
print response.status_code
print response.text
Curl gives correct response:
HTTP/2 201
date: Wed Apr 10 07:59:03 UTC 2019
server: Play
set-cookie: lang=v=2&lang=en-us; Path=/; Domain=api.linkedin.com
x-ambry-creation-time: Wed Apr 10 07:59:03 UTC 2019
access-control-allow-origin: https://www.linkedin.com
content-length: 0
x-li-fabric: prod-lsg1
x-li-pop: prod-tmu1
x-li-proto: http/2
x-li-uuid: FPDYGfsNlBXg+OFSZisAAA==
set-cookie: lidc="b=SB83:g=115:u=5:i=1554883143:t=1554953528:s=AQEVXPybFdnyKyPqSWbf_ax997ap-22S"
x-li-route-key: "b=SB83:g=115:u=5:i=1554883143:t=1554953528:s=AQEVXPybFdnyKyPqSWbf_ax997ap-22S"
Python code gives the following response
201 (status code)
None (response.text)
Can anybody help me out what I am missing out in python code? Thanks!

How to see headers of urllib OpenerDirector before using open method

I'm getting "urllib.error.HTTPError: HTTP Error 401: Unauthorized" after trying this code:
_HTTPHandler = urllib.request.HTTPBasicAuthHandler()
_HTTPHandler.add_password(None,'http://192.168.1.205','admin','password')
opener = urllib.request.build_opener(_HTTPHandler)
opener.open('http://192.168.1.205/api/swis/resource')
I'm sure that the user/password is correct. I've tested it with Google's postman app setting a Basic Auth header and i receive the correct response.
My question is how can a see the headers that are being used by the "opener" so i can check if they are being generated correctly or not.

For manual debugging, you can set the debuglevel of your HTTPHandler:
handler = HTTPHandler(debuglevel=1)
This will produce rich output in stdout, where you can see the full request/response dump:
send: GET / HTTP/1.1
Accept-Encoding: identity
Host: www.example.com
Connection: close
reply: HTTP/1.1 200 OK
header: Date: Mon May 22 2017 12:21:31 GMT
header: Cache-Control: private
header: Content-Type: text/html; charset=ISO-8859-1
Relevant docs: https://docs.python.org/3/library/urllib.request.html

How to send/receive URL parameters in a HTTP POST correctly?

I am using cakephp 2.4.5. I want to send a HTTP POST with URL parameters. I am using python 2.7 request module to send the HTTP POST. Please assume the payload is structured correctly as I have tested that part.
URL_post = http://127.0.0.1/webroot/TestFunc?identity_number=S111A/post
r = requests.post(URL_post, payload)
On the cakephp side, the controller looks something like this;
public function TestFunc($id=null)
{
$identity_number = $this->request->query['identity_number'];
$this->request->data['Model']['associated_id']=$identity_number;
$this->Model->saveAll($this->request->data, array('deep' => true));
}
I have tested that the query is not received correctly. However, if I am not using HTTP POST and just throwing in a normal URL, the query can be received correctly.
What have I done wrong?

The query part of the url is sent correctly:
import requests
requests.post('http://localhost/webroot/TestFunc?identity_number=S111A/post',
{'Model': 'data'})
The Request
POST /webroot/TestFunc?identity_number=S111A/post HTTP/1.1
Host: localhost
User-Agent: python-requests/2.2.1 CPython/3.4 Linux/3.2
Accept: */*
Accept-Encoding: gzip, deflate, compress
Content-Type: application/x-www-form-urlencoded
Content-Length: 10
Model=data
You could also make the requests using params:
requests.post('http://localhost/webroot/TestFunc',
data={'Model': 'data'},
params={'identity_number': 'S111A/post'})
The only difference is that S111A/post is sent as S111A%2Fpost (the same url in the end).

Look at http://docs.python-requests.org/en/latest/user/quickstart/#passing-parameters-in-urls.
payload = {"identity_number": "S111A/post"}
URL_post = "http://127.0.0.1/webroot/TestFunc"
req = requests.post(URL_post, params=payload)
print(req.status_code)

Simulating HTTP POST request produces unexpected result

---SOLVED---
It turned out the request body had literal r"\n" (repr: "\\n") characters in it, and since I simply copy pasted the body as a Python string, Python thought I was giving it newline characters rather than escaped newline characters.
The reason this causes a Bad Request is as follows: So the body was JSON, and in JSON you have to escape all your newline characters by definition. So when the server loads the JSON object from the raw text, an error is thrown causing Bad Request
I realised this because the Content-Length header was different in both cases (\n is one char while \\\n is two chars, although perhaps the Content-Length doesn't actually matter.
Also it is noteworthy that when a lower Content-Length is sent, Bad Request is also returned. I believe this is because the JSON body gets truncated, and the server doesn't accept the important char (e.g. closing brace or something)
--- Problem:---
Summary:
I am trying to use Python to simulate a POST request to bitbucket.org performed within my Firefox web browser. Here is what I did:
Tracked the POST request using Firebug
Copied the POST request headers
Copied the POST request body (in application/json format)
Code:
Here is the code I use to POST my request, but it's a bit long and not very relevant. My Content-Type is application/json, and my POST body is a JSON-encoded string.
dataString = '{"branch":"master","files":[{"path":"readme.txt","content":"ntestxx\n \n"}],"message":"readme.txt edited online with Bitbucket","parents":["465305dc4da32f91da057b65297cda9b72c"],"repository":{"full_name":"minesite/ica-i18n"},"timestamp":"2014-03-20T23:49:29.759Z","transient":false}'
headers = {'X-CSRFToken': '6TqWjCl698U99Iu6ZYGBAloCxZ', 'Content-Length': '2190', 'Accept-Language': 'en,en-us;q=0.7,zh;q=0.3', 'X-NewRelic-ID': 'VwMGVVZSGwIIUFBQDwU=, VwMGVVZSGwIIUFBQDwU=', 'Cookie': 'csrftoken=6TqWjCl698U99Iu6ZYGBAloCxZ; __utma=254090395.1171276563.1394767875.1394776803.1395358874.3; __utmc=254090395; __utmz=254090395.1394776803.2.2.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); bb_session=gpqergylgoa7icpwosqsbpxig0; __utmv=254090395.|1=isBBUser=true=1; recently-viewed-repos_1701252=3802872%2C108928; __utmb=254090395.21.9.1395359363952', 'Connection': 'keep-alive', 'Accept': 'application/json, text/javascript, */*; q=0.01', 'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0', 'Host': 'bitbucket.org', 'X-Requested-With': 'XMLHttpRequest', 'Pragma': 'no-cache', 'Cache-Control': 'no-cache', 'Referer': 'https://bitbucket.org/xxxxxx/xxxxxxx/src/465305dc4da32f91da057b6529a8e4/readme.txt?at=master', 'Content-Type': 'application/json; charset=UTF-8', 'Accept-Encoding': 'gzip, deflate'}
edit = requests.post("https://bitbucket.org/!api/internal/repositories/xxxxxxx/xxxxxxxx/oecommits/", data=dataString, headers=headers)
Results vs. expected results:
When I perform the POST request using my Firefox web browser (using Firebug's "resend request" function), I get a 409 CONFLICT response (Which is the desired response! I am simulating a request to an online editor, so that should be the correct response to a re-sent edit).
However, when I try to simulate the request by copying the request header and the request body, I get a 400 BAD REQUEST response, and the response contains no other information, so I don't even know what my problem is.
Regardless of how many times I send the POST in the web-browser (despite an incorrect timestamp), it achieves the intended outcome, but the server refuses to accept any requests I make using the python requests library.
Response using browser request:
Headers
HTTP/1.1 409 CONFLICT
Server: nginx/1.5.10
Date: Fri, 21 Mar 2014 00:20:55 GMT
Content-Type: text/plain
Content-Length: 45
Connection: keep-alive
x-served-by: app16
X-Render-Time: 0.558492183685
Content-Language: en
X-Static-Version: 48695e7c3140
Vary: Authorization, Accept-Language, Cookie
X-Version: e6778a5040f7
Etag: "92f0b780984e984140de0f8ed0a3992c"
X-Frame-Options: SAMEORIGIN
X-Request-Count: 483
X-NewRelic-App-Data: PxQEVFdXCAITVVlWBgMPUkYdFGQHBDcQUQxLA1tMXV1dSn8UXwJHCwtYGAMPF1pGUw8EFhlQRxYXH1dDC0gKDEQHSgxZVBpaUgtdDVQTQFgrWFsICAZ9V1kQIg1aXF4SLFBYVw4DEUxTEF0DTF0WHgNJCU8EVApUUgUHVFFQCgQCU1FXGwMGX1QdFAEBUVVbA1AJVQEBB1FSA11DHQdSDhdTag==
Body
Specified change not on head of branch master
Response using python request:
Headers
content-length: 11
x-served-by: app10
x-render-time: 0.012787103653
content-language: en
content-type: text/plain
vary: Authorization, Accept-Language, Cookie
connection: keep-alive
server: nginx/1.5.10
x-version: e6778a5040f7
etag: "825644f747baab2c00e420dbbc39e4b3"
x-request-count: 321
x-newrelic-app-data: PxQEVFdXCAITVVlWBgMPUkYdFGQHBDcQUQxLA1tMXV1dSn8UXwJHCwtYGAMPF1pGUw8EFhlQRxYXH1dDC0gRB0MNTRBbXQ5gVhZWFEMCVkBIBhtRSFMJAARQUlsDBw9VXAIBC1tWVU4CUwtUFBpVAwFcWgdTVQIAXQBRWQQAGh9WBQ0RUmw=
date: Fri, 21 Mar 2014 00:51:01 GMT
x-frame-options: SAMEORIGIN
x-static-version: 48695e7c3140
Body
Bad Request
Some of my ideas:
I am thinking that perhaps there is another component to a HTTP POST request that I need to simulate? Perhaps when Firefox sends a POST request, there is some header or wrapper added that makes the request valid?
Or is there something more to a POST request than just a method, headers, and body?
Maybe it's something to do with the fact that it's HTTPS instead of HTTP?
Update:
I have tried sending the "sent cookies" in with the request, to little success.

Or is there something more to a POST request than just a method,
headers, and body?
No. The important part are the request headers. They should be exactly the same in both cases.
Because Firebug can just track the network requests inside Firefox, you'll need an external network analyzer like Wireshark to track the requests coming from your Python script.
Of course you need to run it on the server where the script lies.
Another solution would be to run your request against a local web server and log the request information there.
Then you'll be able to compare the request made in the browser with the one from your script.

how to debug requests library?

I'm trying to convert my python script from issuing a curl command via os.system() to using requests. I thought I'd use pycurl, but this question convinced me otherwise. The problem is I'm getting an error returned from the server that I can see when using r.text (from this answer) but I need more information. Is there a better way to debug what's happening?
for what it's worth I think the issue revoles around converting my --data flag from curl/pycurl to requests. I've created a dictionary of the params i was passing to --data before. My guess is that one of those isn't valid but how can I get more info to know for sure?
example:
headers2 = {"Accept":"*/*", \
"Content-Type":"application/x-www-form-urlencoded", \
"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36", \
"Origin":"https://somedomain.com", \
"X-Requested-With":"XMLHttpRequest", \
"Connection":"keep-alive", \
"Accept-Language":"en-US,en;q=0.8", \
"Referer":"https://somedomain.com/release_cr_new.html?releaseid=%s&v=2&m=a&prev_release_id=%s" % (current_release_id, previous_release_id), \
"Host":"somedomain.com", \
"Accept-Encoding":"gzip,deflate,sdch", \
"Cookie":'cookie_val'}
for bug_id in ids:
print bug_id
data = {'dump_json':'1','releaseid':current_release_id, 'v':'2','m':'a','prev_release_id': previous_release_id,'bug_ids': bug_id, 'set_cols':'sqa_status&sqa_updates%5B0%5D%5Bbugid%5D=' + bug_id + '&sqa_updates%5B0%5D%5Bsqa_status%5D=6'}
print 'current_release_id' , data['releaseid']
print 'previous_release_id', data['prev_release_id']
r = requests.post(post_url, data=json.dumps(data), headers=headers2)
print r.text
The output I'm getting is a pretty generic html message that I've seen before when I've queried the server in the wrong way. So I know I'm reaching the right server at least.
I'm not really expecting any output. This should just post to the server and update a field in the DB.

Anatomy of an http response
Example (loading this page)
HTTP/1.1 200 OK
Cache-Control: public, max-age=60
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Expires: Fri, 27 Sep 2013 19:22:41 GMT
Last-Modified: Fri, 27 Sep 2013 19:21:41 GMT
Vary: *
X-Frame-Options: SAMEORIGIN
Date: Fri, 27 Sep 2013 19:21:41 GMT
Content-Length: 12706
<!DOCTYPE html>
<html>
... truncated rest of body ...
The first line is the status line and consists of the status code and status text.
Headers are key/value pairs. Headers are ended with an empty new line. The empty line denotes there are no more headers and the start of the payload / body follows.
body consumes the rest of the message.
The following explains how to extract the 3 parts:
Status Line
Use the following to get the status line sent back from the server
>>> bad_r = requests.get('http://httpbin.org/status/404')
>>> bad_r.status_code
404
>>> bad_r.raise_for_status()
Traceback (most recent call last):
File "requests/models.py", line 832, in raise_for_status
raise http_error
requests.exceptions.HTTPError: 404 Client Error
(source)
Headers:
r = requests.get('http://en.wikipedia.org/wiki/Monty_Python')
# response headers:
r.headers
# request headers:
r.request.headers
Body
Use r.text.
Post Request Encoding
The 'content-type' you send to the server in the request should match the content-type you're actually sending. In your case, you are sending json but telling the server you're sending form data (which is the default if you do not specify).
From the headers you show above:
"Content-Type":"application/x-www-form-urlencoded",
But your request.post call sets data=json.dumps(data) which is JSON. The headers should say:
"Content-type": "application/json",

The value returned from the request object contains the request information under .request.
Example:
r = requests.request("POST", url, ...)
print("Request headers:", r.request.headers)
print("Request body:", r.request.body)
print("Response status code:", r.status_code)
print("Response text:", r.text.encode('utf8'))