Develop Reference

Python PUT call fails while curl call doesn't

Best wishes (first things first!)
I want to enable/disable a PoE port on my UniFi switch. For this I aim using Python 3.9.1 (first time) with the following code:
import requests
import json
import sys
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
gateway = {"ip": "MYSERVER.COM", "port": "8443"}
headers = {"Accept": "application/json", "Content-Type": "application/json"}
login_url = f"https://{gateway['ip']}:{gateway['port']}/api/login"
login_data = {
"username": "MYUSERNAME",
"password": "MYPASSWORD"
}
session = requests.Session()
login_response = session.post(login_url, headers=headers, data=json.dumps(login_data), verify=False)
if (login_response.status_code == 200):
api_url_portoverrides = 'api/s/default/rest/device/MYDEVICEID'
poe_url = f"https://{gateway['ip']}:{gateway['port']}/{api_url_portoverrides}"
# build json for port overrides
json_poe_state_on = '{"port_overrides": [{"port_idx": 6, "portconf_id": "MYPROFILE1"}]}'
json_poe_state_off = '{"port_overrides": [{"port_idx": 6, "portconf_id": "MYPROFILE2"}]}'
post_response = session.put(poe_url, headers=headers, data=json.dumps(json_poe_state_off))
print('Response HTTP Request {request}'.format(request=post_response.request ))
else:
print("Login failed")
The login works (I get the 2 security cookies and tried them in Paw (a macOS REST API client) to see if these were ok)) but the second call, the. PUT, returns OK but noting happens.
Before I've done this in Python, I tried all my calls in Paw first and there it works. I tried everything in bash with curl and there it works too. So I am a bit at a loss here.
Anyone has an idea?
Thank you for your time!
Best regards!
Peter

Solved it! By looking into what was posted with Wireshark I saw that the payload was different. The culprit was the json.dumps function which encoded the string by putting a backslash in front of each double quote.
It works now!

Http Basic Auth with RFC2045-MIME variant of Base64 in python

I am currently working with and API that requires "RFC2045-MIME variant of Base64, except not limited to 76 char/line" this seems to be different from the normal basic auth used in the requests library. Curious if anyone else has come across this and been able to solve it? I imagine I will have to write a function to do this encoding and build the header manually.

Yes, you can create a function as shown below where the headers for the request are created manually.
For this implementation, you will need these variables:
API_PUBLIC_KEY,
API_SECRET_KEY,
host_url,
endpoint (where the API is supposed to hit).
import base64
import requests
def create_request():
auth_header = base64.b64encode(bytes(f'{API_PUBLIC_KEY}:{API_SECRET_KEY}'.encode('ascii'))).decode('utf-8')
headers = {
'Host': host_url,
'Authorization': f'Basic {auth_header}',
'Content-Type': 'application/json'
}
api_endpoint = f'https://{host_url}/{endpoint}'
data = {}
response = requests.request("POST", api_endpoint, headers=headers, data=data)
print(response.text.encode('utf8'))

Allcoin signed POST request fails

The API i am using requires an MD5 encryption to work on POST requests. I am trying the make an 'userBalance' request to the API using the documentation https://www.allcoin.ca/api_market/market But every time it gives me back {'code': 1, 'msg': '签名校验失败'} (which means signature check failed). The API only asks for 2 parameteres, the api_key and sign. I carefully follow the guidance of the documentation, but the API still rejects the POST. Any suggestions why it fails the signature check? Am i missing something?
import hashlib
import keys
import requests
KEY = keys.allcoin["key"]
API_SECRET = keys.allcoin["secret"]
msg = "api_key='{}&secret_key={}".format(KEY, API_SECRET)
signature = hashlib.md5(msg.encode("utf-8")).hexdigest()
parameters = {
"api_key": KEY,
"sign": signature.upper(),
}
params = "&".join("{}={}".format(a, b) for a, b in parameters.items())
url = "http://www.allcoin.ca/Api_User/userBalance"
r = requests.post(
headers={
"Content-Type": "application/x-www-form-urlencoded",
'user-agent': 'my-app/0.0.1'
},
url=url,
params=parameters
)
print(r.json())

I think you want to change params=parameters to params=params in your requests.post().

How to request the "load more" function with Python?

Hi~I'm now doing my research on Zhihu, a Chinese Q&A website like Quora, using social network analysis. And I'm writing a crawler with Python these days, but met a problem:
I want to scratch the user info that follows a specific user, like Kaifu-Lee. The Kaifu-Lee's followers page is http://www.zhihu.com/people/kaifulee/followers
And the load-more button is at the bottom of the followers list, I need to get full list.
Here's the way I do with python requests:
import requests
import re
s = requests.session()
login_data = {'email': '***', 'password': '***', }
# post the login data.
s.post('http://www.zhihu.com/login', login_data)
# verify if I've login successfully. Surely this step have succeed.
r = s.get('http://www.zhihu.com')
Then, I jumped to the target page:
r = s.get('http://www.zhihu.com/people/kaifulee/followers')
and get 200 return:
In [7]: r
Out[7]: <Response [200]>
So the next step is to analyze the request of load-more under "network" tag using chrome's developer tool, here's the information:
Request URL: http://www.zhihu.com/node/ProfileFollowersListV2
Request Method: POST
Request Headers
Connection:keep-alive
Host:www.zhihu.com
Origin:http://www.zhihu.com
Referer:http://www.zhihu.com/people/kaifulee/followers
Form data
method:next
params:{"hash_id":"12135f10b08a64c54e8bfd537dd7bee7","order-by":"created","offset":20}
_xsrf:ea63beee3a3444bfb853f36b7d968ad1
So I try to POST:
global header_info
header_info = {
'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1581.2 Safari/537.36',
'Host':'www.zhihu.com',
'Origin':'http://www.zhihu.com',
'Connection':'keep-alive',
'Referer':'http://www.zhihu.com/people/zihaolucky/followers',
'Content-Type':'application/x-www-form-urlencoded',
}
# form data.
data = r.text
raw_hash_id = re.findall('hash_id(.*)',data)
hash_id = raw_hash_id[0][14:46]
payload={"method":next,"hash_id":str(hash_id),"order_by":"created","offset":20}
# post with parameters.
url = 'http://www.zhihu.com/node/ProfileFollowersListV2'
r = requests.post(url,data=payload,headers=header_info)
BUT, it returns Response<404>>
If I made any mistake?
Someone said I made a mistake in dealing with the params. The Form Data has 3 parameters:method,params,_xsrfand I lost _xsrf and then I put them into a dictionary.
So I modified the code:
# form data.
data = r.text
raw_hash_id = re.findall('hash_id(.*)',data)
hash_id = raw_hash_id[0][14:46]
raw_xsrf = re.findall('xsrf(.*)',r.text)
_xsrf = raw_xsrf[0][9:-3]
payload = {"method":"next","params":{"hash_id":hash_id,"order_by":"created","offset":20,},"_xsrf":_xsrf,}
# reuse the session object, but still error.
>>> r = s.post(url,data=payload,headers=header_info)
>>> <Response [500]>

You can't pass nested dictionaries to the data parameter. Requests just doesn't know what to do with them.
It's not clear, but it looks like the value of the params key is probably JSON. This means your payload code should look like this:
params = json.dumps({"hash_id":hash_id,"order_by":"created","offset":20,})
payload = {"method":"next", "params": params, "_xsrf":_xsrf,}
Give that a try.

python set-up boundary for POST using multipart/form-data with requests

I want to send a file using requests but the server works with a fixed boundary set at *****. I'm only able to send a file but the requests module creates a random boundary. How do I overwrite it?
import requests
url='http://xxx.xxx.com/uploadfile.php'
fichier= {'uploadedfile':open('1103290736_2016_03_23_13_32_55.zip','rb')}
headers2={'Connection':'Keep-Alive','User-Agent':'Dalvik/1.6.0 (Linux; U; Android 4.4.2; S503+ Build/KOT49H)','Accept-Encoding':'gzip'}
session= requests.Session()
session.post(url,headers=headers2,files=fichier)
session.close()

Boy, that's one very broken server. If you can, fix the server instead.
You can't tell requests what boundary to pick. You can instead build your own multipart/form-data payload, using the email.mime package:
from email.mime.multipart import MIMEMultipart
from email.mime.application import MIMEApplication
related = MIMEMultipart('form-data', '*****') # second argument is the boundary.
file_part = MIMEApplication(
open('1103290736_2016_03_23_13_32_55.zip', 'rb').read(),
# optional: set a subtype: 'zip',
)
file_part.add_header('Content-disposition', 'form-data; name="uploadedfile"')
related.attach(file_part)
body = related.as_string().split('\n\n', 1)[1]
headers = dict(related.items())
headers['User-Agent'] = 'Dalvik/1.6.0 (Linux; U; Android 4.4.2; S503+ Build/KOT49H)'
r = session.post(url, data=body, headers=headers)
This sets Content-Type: multipart/form-data; boundary="*****" as the header, and the body uses ***** as the boundary (with appropriate -- pre- and postfixes).

A simple alternative is using requests-toolbelt; below example taken from this GitHub issue thread:
from requests_toolbelt import MultipartEncoder
fields = {
# your multipart form fields
}
m = MultipartEncoder(fields, boundary='my_super_custom_header')
r = requests.post(url, headers={'Content-Type': m.content_type}, data=m.to_string())
However, this introduces an extra dependency and can be slow to upload large files.

you can actually directly use the requests module to do this:
files = {'file': ('filename', open('filename', 'rb'), 'text/plain')}
body, content_type = requests.models.RequestEncodingMixin._encode_files(files, {})
# this way you ensure having the same boundary defined in
# the multipart/form-data contetn-type header
# the form-data
data = body
headers = {
"Content-Type": content_type
}
response = requests.post(
endpoint,
data=data,
headers=headers
)
This worked for me. Otherwise I was getting an error like this:
"errorMessage":"Required request part 'file' is not present"