I'm trying to authenticate a bot which posts on stocktwits (see API) with Python/pycurl. As I understand it, first I need to request that the application is authorized to use Stocktwits user data via oauth/authorize, which returns an authorization code. Then I need to confirm permission via oauth/token, which would send back an access token that can be used to post stocktwits.
The problem I'm running into is that after I make a post request to auth/authorize, the response returned is
Host: api.stocktwits.com
Authorization: Basic bmljay5vc2hpbm92QGdtYWlsLmNvbTp6YWRuaWsxMg==
User-Agent: PycURL/7.43.0.2 libcurl/7.60.0 OpenSSL/1.1.0h zlib/1.2.11 c-
ares/1.14.0 WinIDN libssh2/1.8.0 nghttp2/1.32.0
Accept: */*
Content-Type: application/x-www-form-urlencoded
Expect: 100-continue
< HTTP/1.1 100 Continue
and the program just stalls waiting. How can I handle the http 100 response?
Code for api call is below
def get_auth_token:
auth_data = ""
c = pycurl.Curl()
c.setopt(c.CAINFO, self.CA_CERTS)
c.setopt(pycurl.POST, 1)
c.setopt(c.URL, uri_to_ouath_authorize_call)
c.setopt(c.FOLLOWLOCATION, True)
c.setopt(c.WRITEDATA, auth_data)
c.setopt(c.USERPWD, 'bot_account_username:password')
I'm guessing you need to use POSTFIELDS or READFUNCTION to supply the post data, it doesn't look like you have anything for the request body configured. See https://curl.haxx.se/libcurl/c/CURLOPT_POST.html.
Using pycurl.POST is a common mistake, normally POSTFIELDS should be used instead.
Related
I am writing a small application that interprets the http response of a request. I am writing the application in python. I have not found anything that allows me to send the body + headers stored in one file. I can send certain parts like the headers but not the entire request.
For example, if the request is:
GET /index.html HTTP/1.1
Host: localhost
Cookie: bob=lemon
I want to send this entire request in one go. How would I do this in python?
Check out the python requests library. https://requests.readthedocs.io/en/master/user/quickstart/#make-a-request
For the request above it would look something like
import requests
url = 'http://localhost:[YOUR PORT HERE]/'
cookies = {bob : lemon}
r = requests.get(url, cookies=cookies)
To check if you had a successful request you should get a 200 code from.
r.status_code
Check out the library for more, it is very extensive.
I want to be able to send a HTTP request with python without slash "/" in the path.
Simply, here is what the request should be like:
GET test HTTP/1.1
Host: example.com
Connection: keep-alive
Cache-Control: max-age=0
What I want to do is GET test HTTP/1.1 rather than GET /test HTTP/1.1
I am able to send the request using request repeating tools, but I am not sure how to do that with Python.
To clarify more: I don't want the request path to start with "/"
I am looking for the equivalent of this in python.
Thanks!
I am currently writing a HTTP client to do a HTTP POST on a URL that returns a HTTP response.
However, for error messages code 400 and 500, it sends back non chunked HTTP response, and for success messages, 201, it sends a chunked response.
In the request, I am setting the content-length, so I am not sure why it is still sending us the chunked transfer encoding. Is there any other header I can set in the request, that will tell the HTTP server not to send chunked encoding?
headerList.append("POST /v2/charges HTTP/1.1")
headerList.append("Content-Type: application/json")
headerList.append("host: xxxxxxxxx")
headerList.append("request-id: ABCD001123")
headerList.append("Content-length: %d" %len(Msg))
hostReqHeader = "\r\n".join(headerList)
reqData = hostReqHeader + '\r\n\r\n' + qbPosMsg
I am using sockets to send these HTTP messages, and not using httplib or requests library.
Chunked is a required feature of HTTP/1.1. If you do not require any other 1.1-specific features, specify HTTP/1.0 in your request:
headerList.append("POST /v2/charges HTTP/1.0")
The Content-Length header you are specifying in your request applies to the request, not the server's response.
Chunked transfer is only used by the HTTP/1.1 server in a response when the client specifies HTTP/1.1 as the protocol. If you want to disable chunked transfer completely, you specify HTTP/1.0 as the protocol in your request.
Alternatively, make use of an HTTP client library that supports chunked transfer - any one that supports HTTP/1.1 will, because in any HTTP/1.1 conversation the server is free to choose whether to use chunked transfer for any request.
HTTP/1.1 (and indeed HTTP/2) servers still support HTTP/1.0, and HTTP 1.0 remains quite useful for the ability to write simple clients with a few lines of code, that can still query modern web servers (albeit, with the need to TLS-wrap for HTTPS support). So it's quite appropriate in this situation. I think that is the beauty of HTTP, that the basic protocol is so simple. HTTP 1.1 and HTTP 2.0 add progressively more complexity in terms of being able to write a client that supports them, but all that complexity is optional - HTTP 1.0 can still be used.
I have been able to view the attributes of the PreparedRequest that botocore sends, but I'm wondering how I can view the exact request string that is sent to AWS. I need the exact request string to be able to compare it to another application I'm testing AWS calls with.
You could also enable debug logging in boto3. That will log all requests and responses as well as lots of other things. Its a bit obscure to enable it:
import boto3
boto3.set_stream_logger(name='botocore')
The reason you have to specify botocore as the name to log is that all of the actual requests and responses happen at the botocore layer.
So what you probably want to do is to send your request through the proxy (mitmproxy, squid). Then check the proxy for what was sent.
Since HTTPS data is encrypted you must first decrypt it, then log the response, then encrypt it back and send to AWS. One of the options is to use mitmproxy. ( It's really easy to install )
Run mitmproxy
Open up another terminal and point proxy to mitmproxys port:
export http_proxy=127.0.0.1:8080
export https_proxy=$http_proxy
Then set verify=False when creating session/client
In [1]: import botocore.session
In [2]: client = botocore.session.Session().create_client('elasticache', verify=False)
Send request and look at the output of mitmproxy
In [3]: client.describe_cache_engine_versions()
The result should be similar to this:
Host: elasticache.us-east-1.amazonaws.com
Accept-Encoding: identity
Content-Length: 53
Content-Type: application/x-www-form-urlencoded
Authorization: AWS4-HMAC-SHA256 Credential=FOOOOOO/20150428/us-east-1/elasticache/aws4_request, SignedHeaders=host;user-agent;x-amz-date, Signature=BAAAAAAR
X-Amz-Date: 20150428T213004Z
User-Agent: Botocore/0.103.0 Python/2.7.6 Linux/3.13.0-49-generic
<?xml version='1.0' encoding='UTF-8'?>
<DescribeCacheEngineVersionsResponse
xmlns="http://elasticache.amazonaws.com/doc/2015-02-02/">
<DescribeCacheEngineVersionsResult>
<CacheEngineVersions>
<CacheEngineVersion>
<CacheParameterGroupFamily>memcached1.4</CacheParameterGroupFamily>
<Engine>memcached</Engine>
<CacheEngineVersionDescription>memcached version 1.4.14</CacheEngineVersionDescription>
<CacheEngineDescription>memcached</CacheEngineDescription>
<EngineVersion>1.4.14</EngineVersion>
I have a working bit of PHP code that uploads a binary to a remote server I don't have shell access to. The PHP code is:
function upload($uri, $filename) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $uri);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, array('file' => '#' . $filename));
curl_exec($ch);
curl_close($ch);
}
This results in a header like:
HTTP/1.1
Host: XXXXXXXXX
Accept: */*
Content-Length: 208045596
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----------------------------360aaccde050
I'm trying to port this over to python using requests and I cannot get the server to accept my POST. I have tried every which way to use requests.post, but the header will not mimic the above.
This will successfully transfer the binary to the server (can tell by watching wireshark) but because the header is not what the server is expecting it gets rejected. The response_code is a 200 though.
files = {'bulk_test2.mov': ('bulk_test2.mov', open('bulk_test2.mov', 'rb'))}
response = requests.post(url, files=files)
The requests code results in a header of:
HTTP/1.1
Host: XXXX
Content-Length: 160
Content-Type: multipart/form-data; boundary=250852d250b24399977f365f35c4e060
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/2.2.1 CPython/2.7.5 Darwin/13.1.0
--250852d250b24399977f365f35c4e060
Content-Disposition: form-data; name="bulk_test2.mov"; filename="bulk_test2.mov"
--250852d250b24399977f365f35c4e060--
Any thoughts on how to make requests match the header that the PHP code generates?
There are two large differences:
The PHP code posts a field named file, your Python code posts a field named bulk_test2.mov.
Your Python code posts an empty file. There Content-Length header is 160 bytes, exactly the amount of space the multipart boundaries and Content-Disposition part header take up. Either the bulk_test2.mov file is indeed empty, or you tried to post the file multiple times without rewinding or reopening the file object.
To fix the first problem, use 'file' as the key in your files dictionary:
files = {'file': open('bulk_test2.mov', 'rb')}
response = requests.post(url, files=files)
I used just the open file object as the value; requests will get the filename directly from the file object in that case.
The second issue is something only you can fix. Make sure you don't reuse files when repeatedly posting. Reopen, or use files['file'].seek(0) to rewind the read position back to the start.
The Expect: 100-continue header is an optional client feature that asks the server to confirm that the body upload can go ahead; it is not a required header and any failure to post your file object is not going to be due to requests using this feature or not. If an HTTP server were to misbehave if you don't use this feature, it is in violation of the HTTP RFCs and you'll have bigger problems on your hands. It certainly won't be something requests can fix for you.
If you do manage to post actual file data, any small variations in Content-Length are due to the (random) boundary being a different length between Python and PHP. This is normal, and not the cause of upload problems, unless your target server is extremely broken. Again, don't try to fix such brokenness with Python.
However, I'd assume you overlooked something much simpler. Perhaps the server blacklists certain User-Agent headers, for example. You could clear some of the default headers requests sets by using a Session object:
files = {'file': open('bulk_test2.mov', 'rb')}
session = requests.Session()
del session.headers['User-Agent']
del session.headers['Accept-Encoding']
response = session.post(url, files=files)
and see if that makes a difference.
If the server fails to handle your request because it fails to handle HTTP persistent connections, you could try to use the session as a context manager to ensure that all session connections are closed:
files = {'file': open('bulk_test2.mov', 'rb')}
with requests.Session() as session:
response = session.post(url, files=files, stream=True)
and you could add:
response.raw.close()
for good measure.