Develop Reference

Error passing curl -d parameter in requests library

I have a curl POST request that works perfectly in the terminal (macOS), returning a csv as expected. The following format is provided in the RJMetrcis documentation (see "Export figure data"). Here is the curl request in bash:
 curl -d "format=csv&includeColumnHeaders=1" -H "X-RJM-API-Key: myAPIkey" https://api.rjmetrics.com/0.1/figure/0000/export
My objective is to implement the exact same curl request in Python using requests. When I input the same parameters as a POST request, the code does not work returning an error:
import requests
headers = {'X-RJM-API-Key: myAPIkey'}
data= {'format=csv&includeColumnHeaders=1'}
url = "https://api.rjmetrics.com/0.1/figure/0000/export"
response = requests.post(url, data, headers)
This returns the error:
TypeError: memoryview: a bytes-like object is required, not 'str'
On the second try:
response = requests.post(url, data=data, headers=headers)
returns
AttributeError: 'set' object has no attribute 'items'
What is the correct format in python for constructing a POST request so that it matches the data = {'key':'value'} convention, and returns a csv?Any help would be appreciated translating the bash curl POST into a python POST request

Here you are passing a set and you are expected to pass a dict or str object
data= {'format=csv&includeColumnHeaders=1'}
Replacing with
data= {'format':'csv&includeColumnHeaders=1'}
Should fix it.
On the other hand by seeing your curl request..
It all depends how you want to pass the data, following code (passing the data payload as a string) will post the data directly, that will be the equivalent to --data-raw in curl
import requests
url = "https://api.rjmetrics.com/0.1/figure/0000/export"
payload = "'{\"format\":\"csv&includeColumnHeaders=1\"}'"
headers = {
'X-RJM-API-Key': 'myAPIkey'
}
response = requests.request("POST", url, headers=headers, data = payload)
print(response.text.encode('utf8'))

Python using Requests to cURL a file upload

I am trying to translate a specific curl method into Python's requests module to upload a file to to an api. My standard method that works for non-file requests looks like this:
import requests
requestObject = requests.Session()
standard_headers = {header1:headerValue1,header2:headerValue2}
payload = {key1:value1,key2:value2}
url = 'https://myUrl.com/apiCall'
requestObject.post(url,headers=standard_headers, json=payload)
This works for non-file requests that I need to make to the API. However for file uploads, the API documentation shows a method using curl:
curl -XPOST -H 'header1' -H 'header2 'https://myUrl.com/apiCall' \
-F 'attachment=#/path/to/my/file' \
-F '_json=<-;type=application/json' << _EOF_
{
"key1":"keyValue1",
"key2":"keyValue2"
}
_EOF_
I tested the curl command and it works successfully.
My question is how do I translate that curl method using the << _EOF_ method in Python requests. One idea I had was simply to use the 'files' option in the requests module:
requestObject = requests.Session()
standard_headers = {header1:headerValue1,header2:headerValue2}
payload = {key1:keyValue1,key2:keyValue2}
url = 'https://myUrl.com/apiCall'
file_to_upload = {'filename': open('/path/to/my/file', 'rb')}
requestObject.post(url,headers=standard_headers, files=file_to_upload, json=payload)
But that does not seem to work as the necessary json parameters (the values in payload) do not appear to get passed to the file upload
I also tried specifying the json parameters directly into the file_to_upload variable:
requestObject = requests.Session()
standard_headers = {header1:headerValue1,header2:headerValue2}
url = 'https://myUrl.com/apiCall'
file_to_upload = {'attachment': open('/path/to/my/file', 'rb'),'{"key1":"keyValue1","key2":"keyValue2"}'}
requestObject.post(url,headers=standard_headers, files=file_to_upload)
Similar result, it seems as though I am not passing the necessary json values correctly. I tried a few other ways but I am overlooking something. Any insight into how I should structure my request is appreciated.

Ok I managed to get it to work and posting for anyone who might need help in the future.
The trick was to include the _json key in the data field. My code ended up looking like so:
import requests
requestObject = requests.Session()
standard_headers = {header1:headerValue1,header2:headerValue2}
json_fields = json.dumps({
"key1": "key1Value",
"key2": "key2Value"
})
payload = {"_json":json_fields)
file = {"attachment": /path/to/my/file}
url = 'https://myUrl.com/apiCall'
requestObject.post(url,headers=standard_headers, files=file, data=payload)
Hope that helps some future person.

Python post request returns 500

I am using requests to make a POST request to create a user. The request succeeds with 201 created when I use curl, however fails with a 500 response when I use requests. My curl command is
curl --user administrator:password -H "Content-Type: application/json" https://localhost:8080/midpoint/ws/rest/users -d #user.json -v
And my python script is:
import requests
import json
headers = {
'Content-Type': 'application/json',
}
with open('user.json') as j:
data = json.load(j)
response = requests.post('https://localhost:8080/midpoint/ws/rest/users', headers=headers, data=str(data), auth=('Administrator', 'password'))
print(response)
Can anyone see a reason why my python script would be failing? I am at a loss.

str(data) returns the Python representation of data, not its JSON representation. These two forms can differ in things like ' vs. ", True vs. true, and None vs. null. To properly JSONify data, call json.dumps() on it:
response = requests.post(..., data=json.dumps(data))
or let requests do the JSONification:
response = requests.post(..., json=data)
or use the JSON as it appears in user.json directly:
with open('user.json') as j:
data = j.read()
response = requests.post(..., data=data)

How to make a post request with REQUESTS package for Python?

I am trying to use the toggl api.
I use Requests instead of Urllib2 for doing my GETs en POSTS. But i am stuck.
payload = {
"project":{
"name":"Another Project",
"billable":False,
"workspace":{
"Name":"jorrebor's workspace",
"id":213272
},
"automatically_calculate_estimated_workhours":False
}
}
url = "https://www.toggl.com/api/v6/projects.json"
r = requests.post(url, data=json.dumps(payload), auth=HTTPBasicAuth('j_xxxxx#gmail.com', 'mypassword'))
Authentication seems to be fine, but the payload format probably isn't.
a curl command with the same parameters:
curl -v -u heremytoken:api_token -H "Content-type: application/json" -d "{\"project\":{\"billable\":true,\"workspace\":{\"id\":213272},\"name\":\"Another project\",\"automatically_calculate_estimated_workhours\":false}}" -X POST https://www.toggl.com/api/v6/projects.json
does work fine.
What wrong with my payload? The response is get is:
["Name can't be blank","Workspace can't be blank"]
which leads me to conclude that the authentication works but toggl cannot read my json object.

Seems like you should try setting the header to a JSON application, rather than the default format, and send a JSON object instead of the Python dict. Check it out here:
payload = {"project":{"name":"Another Project",
"billable":False,
"workspace":{"Name":"jorrebor's workspace",
"id":213272},
"automatically_calculate_estimated_workhours":False
} }
parameters_json = json.dumps(payload)
headers = {'Content-Type': 'application/json')
r = client.post(url, data=parameters_json, headers=headers)
which should allow the site to read the json object just fine.

Python equivalent of Curl HTTP post

I am posting to Hudson server using curl from the command line using the following--
curl -X POST -d '<run><log encoding="hexBinary">4142430A</log><result>0</result><duration>2000</duration></run>' \
http://user:pass#myhost/hudson/job/_jobName_/postBuildResult
as shown in the hudson documentation..can I emulate the same thing using python..i don't want to use pyCurl or send this line through os.system()..is there ny way out using raw python??

import urllib2
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
result = response.read()
where data is the encoded data you want to POST.
You can encode a dict using urllib like this:
import urllib
values = { 'foo': 'bar' }
data = urllib.urlencode(values)

The modern day solution to this is much simpler with the requests module (tagline: HTTP for humans! :)
import requests
r = requests.post('http://httpbin.org/post', data = {'key':'value'}, auth=('user', 'passwd'))
r.text # response as a string
r.content # response as a byte string
# gzip and deflate transfer-encodings automatically decoded
r.json() # return python object from json! this is what you probably want!