c = pycurl.Curl()
c.setopt(c.URL, 'https://httpbin.org/post')
post_data = {'field': 'value'}
# Form data must be provided already urlencoded.
postfields = urlencode(post_data)
# Sets request method to POST,
# Content-Type header to application/x-www-form-urlencoded
# and data to send in request body.
c.setopt(c.POSTFIELDS, postfields)
c.perform()
c.close()
This is example from official doc.
I want to know what would be the http request looks like.
Is there a way to do that?
You could use the VERBOSE option, works like curl -v, will print request/response headers to stderr:
c.setopt(pycurl.VERBOSE, 1)
see more detail on pycurl doc.
Related
curl correctly posts data to Solr:
$ curl -v 'http://solr.example.no:12699/solr/my_coll/update?commit=true' \
--data '<add><doc><field name="key">KEY__9927.1</field><field name="value">\
{"result":0,"jobId":"9459695","jobNumber":"9927.1"}</field></doc></add>'
The solr query log says:
[20200306T111354,131] [my_coll_shard1_replica_n85] webapp=/solr path=/update params={commit=true} status=0 QTime=96
I'm trying to do the same thing with Python:
>>> import urllib.request
>>> data = '<add><doc><field name="key">KEY__9927.1</field><field name="value">{"result":0,"jobId":"9459695","jobNumber":"9927.1"}</field></doc></add>'
>>> url = 'http://solr.example.no:12699/solr/my_coll/update?commit=true'
>>> req = urllib.request.Request(url=url, data=data.encode('utf-8'), method='POST')
>>> res = urllib.request.urlopen(req)
But now the solr query log shows that the POST data has been added to the query param string:
[20200306T112358,780] [my_coll_shard1_replica_n87] webapp=/solr path=/update params={commit=true&<add><doc><field+name="key">KEY__9927.1</field><field+name%3D"value">{"result":0,"jobId":"9459695","jobNumber":"9927.1"}</field></doc></add>} status=0 QTime=30
What is happening here?
The issue is that you're not sending the correct Content-Type for your request, and this gets mangled within Jetty (or the Solr app) before being forwarded to the log (any POSTed data that's not multipart can be inserted as part of the query string - Solr parses them all the same). The /update endpoint accepts multiple formats, such as both JSON and XML, and the Content-Type should be set appropriately.
req = urllib.request.Request(url=url, data=data.encode('utf-8'), method='POST', headers={'Content-Type': 'text/xml'})
res = urllib.request.urlopen(req)
In fact, it's the User-Agent string that changes the behaviour. This is by design - curl has been special cased to override the default Content-Type handler. If you're not using curl, you have to explicitly provide the Content-Type being submitted. This has probably been done to make it easier to make manual requests using curl on the command line. The implementation is available in SolrRequestParsers.java, line 782
I'm trying to authenticate a bot which posts on stocktwits (see API) with Python/pycurl. As I understand it, first I need to request that the application is authorized to use Stocktwits user data via oauth/authorize, which returns an authorization code. Then I need to confirm permission via oauth/token, which would send back an access token that can be used to post stocktwits.
The problem I'm running into is that after I make a post request to auth/authorize, the response returned is
Host: api.stocktwits.com
Authorization: Basic bmljay5vc2hpbm92QGdtYWlsLmNvbTp6YWRuaWsxMg==
User-Agent: PycURL/7.43.0.2 libcurl/7.60.0 OpenSSL/1.1.0h zlib/1.2.11 c-
ares/1.14.0 WinIDN libssh2/1.8.0 nghttp2/1.32.0
Accept: */*
Content-Type: application/x-www-form-urlencoded
Expect: 100-continue
< HTTP/1.1 100 Continue
and the program just stalls waiting. How can I handle the http 100 response?
Code for api call is below
def get_auth_token:
auth_data = ""
c = pycurl.Curl()
c.setopt(c.CAINFO, self.CA_CERTS)
c.setopt(pycurl.POST, 1)
c.setopt(c.URL, uri_to_ouath_authorize_call)
c.setopt(c.FOLLOWLOCATION, True)
c.setopt(c.WRITEDATA, auth_data)
c.setopt(c.USERPWD, 'bot_account_username:password')
I'm guessing you need to use POSTFIELDS or READFUNCTION to supply the post data, it doesn't look like you have anything for the request body configured. See https://curl.haxx.se/libcurl/c/CURLOPT_POST.html.
Using pycurl.POST is a common mistake, normally POSTFIELDS should be used instead.
I had been using sockets, with Python, for some time ago and I'm trying to understand why this POST which should send some data on fields data1 and data2 do not work.
POST /method.php HTTP/1.1\r\nHost: localhost\r\nContent-Type: multipart/form-data\r\n\r\ndata1=something&data2= otherthing\r\n\r\n
What is the problem with this request?
There are several things wrong with your request:
POST /method.php HTTP/1.1
Host: localhost
Content-Type: multipart/form-data
data1=something&data2= otherthing
First, whenever a body is used within a HTTP request the length of the body must be known. This is typically done by given the length up-front with Content-length in the HTTP header although also chunked encoding might be used if the full length is not known up front. Your request does not do any of these which means the request is an invalid HTTP request.
Additionally you claim a Content-Type of multipart/form-data although your body is not of this type. With multipart/form-data your body would consist of several MIME parts separated by a text boundary and this boundary would need to have been declared in your Content-type header. The correct type for the body you show would be instead application/x-www-form-urlencoded.
Even with application/x-www-form-urlencoded the body is partly wrong. This type of body should be only pairs of key=value concatenated by &, i.e. there should be neither as space after a key as you have after data2= nor there should be new lines added after the end of the data as you have.
When removing all these problems you should probably send the following request:
body = "data1=something&data2=otherthing"
request = ("POST /method.php HTTP/1.1\r\n" + \
"Host: localhost\r\n" + \
"Content-Type: application/x-www-form-urlencoded\r\n" + \
"Content-Length: %d\r\n" + \
"\r\n%s") % (len(body),body)
But once you have send this request the trouble continues since getting the response correctly is complex too. Generally I recommend to not code your own HTTP handling unless you really know what you do but instead use existing libraries. While HTTP might look simple when just looking at a few example requests it is way more complex than it initially looks. And while your code might seem to work against specific servers it might fail with other servers.
It might be easier to use the requests library so your code would look something like this:
import requests
# Data
data = {
'data1':'something',
'data2':'otherthing'
}
# Custom headers
headers = {
'content-type': 'multipart/form-data'
}
# Get response from server
response = requests.post('http://localhost/', data=data, headers=headers)
# If you care about the response
print(response.json())
You can also send files and a whole lot of other stuff
Have you tried using the Requests library instead, example of a post request below
import requests
header = {"Content-Type": "multipart/form-data"}
data1="something"
data2= "otherthing"
session_requests = requests.session()
result = session_requests.post("http://localhost/", data=dict(data1, data2), headers=header)
I am unable to post data to a rest server because the server doesn't know how to handle a multi part post requests,and it throws an error when it encounters a boundary.
Is there a way to make a non multi part post in pycurl?
Why does a post request need to be multi part?
A POST certainly doesn't have to be multipart, see this example from the pycurl docs that I also paste here:
c = pycurl.Curl()
c.setopt(c.URL, 'http://pycurl.io/tests/testpostvars.php')
post_data = {'field': 'value'}
# Form data must be provided already urlencoded.
postfields = urlencode(post_data)
# Sets request method to POST,
# Content-Type header to application/x-www-form-urlencoded
# and data to send in request body.
c.setopt(c.POSTFIELDS, postfields)
c.perform()
c.close()
I want to login into a website sending POST request using `pycurl' on a HTTPS page. I did the following:
import pycurl, urllib
curl = pycurl.Curl()
curl.setopt(pycurl.COOKIEFILE, "")
post = "_username=something&_password=somethingelse&_submit=Login"
curl.setopt(pycurl.URL, "https://www.example.com/login_check")
curl.setopt(pycurl.POST, 1)
curl.setopt(pycurl.POSTFIELDS, post)
curl.perform()
But I wasn't able to login. May be that was because my code made http call rather. Although I am able to perform the login sending 'POST' request with the same parameters in `POSTMAN'
So seems like login is right but I am doing something wrong implementing the same in python. Pl guide.
According to the pycURL documentation, your post data needs to be key value pairs and url encoded:
post_data = {'field': 'value'}
# Form data must be provided already urlencoded.
postfields = urlencode(post_data)
# Sets request method to POST,
# Content-Type header to application/x-www-form-urlencoded
# and data to send in request body.
c.setopt(c.POSTFIELDS, postfields)