I am trying to replicate the following client requests via python-requests.
Under client connection I see HTTP Version which is 2.0 and TLS version which is 1.3, as up to my knowledge I know that requests utilizes TLS 1.3. My requests are failing as of now.
And I wonder if I need to pass certificates. I would like to understand how this request is different from regular request which would be simply as
r = requests.get('someurl')
How can I use requests to use the exact client connection show in requests? I don't fully understand each pointer, How would I use h2 ALPN/ with that specific cipher name? I am not expecting an solid answer to the question rather an explanation would be much more helpful!
python-requests doesn't support HTTP 2 request. You can use httpx package.
HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2.
Example
import ssl
import httpx
# create an ssl context
ssl_context = ssl.SSLContext(protocol=ssl.PROTOCOL_TLS)
# ssl.PROTOCOL_TLS - Selects the highest protocol version that both the client and server support.
# Despite the name, this option can select both "SSL" and "TLS" protocols.
# set protocol to use
ssl_context.set_alpn_protocols(["h2"])
CIPHERS = 'ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:DH+HIGH:RSA+AESGCM:RSA+AES:RSA+HIGH:!aNULL:!eNULL:!MD5:!3DES'
# set ciphers
ssl_context.set_ciphers(CIPHERS)
# httpx verify param lets you pass a standard library ssl.SSLContext
response = httpx.get('https://example.com', verify=ssl_context)
print(response.http_version)
# outputs HTTP/2
Instead of using ssl.SSLContext, you can also use httpx.create_ssl_context() to set the ssl context.
As far as I know python-requests is a library which currently1 doesn't support HTTP/2.0. This question has been answered here.
However there are python libraries like Python httpx supporting HTTP/2.0!
Kind regards,
1 Feb 16, 2021
Related
I need to edit the python request to add TLS settings (by TLS settings I mean tls finger printing int, ja3).
You can use curl_cffi. Although you can not modify the TLS settings at python level, you can change them when using C extensions, such as curl.
pip install curl_cffi
It has a requests-like interface:
from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101")
print(r.json())
# output: {'ja3_hash': '53ff64ddf993ca882b70e1c82af5da49'
# the fingerprint should be the same as target browser
The JA3 fingerprint is based on ciphers and order and various TLS extensions and order. While ciphers and order can be changed features like the TLS extension order are not accessible from Python. This means there is no way to emulate a specific JA3 fingerprint from Python and thus also not from requests.
I need to edit the python request to add TLS settings (by TLS settings I mean tls finger printing int, ja3).
You can use curl_cffi. Although you can not modify the TLS settings at python level, you can change them when using C extensions, such as curl.
pip install curl_cffi
It has a requests-like interface:
from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101")
print(r.json())
# output: {'ja3_hash': '53ff64ddf993ca882b70e1c82af5da49'
# the fingerprint should be the same as target browser
The JA3 fingerprint is based on ciphers and order and various TLS extensions and order. While ciphers and order can be changed features like the TLS extension order are not accessible from Python. This means there is no way to emulate a specific JA3 fingerprint from Python and thus also not from requests.
I'm hacking together an amazon api and when only using python requests without proxying, it prompts for a captcha. When routing this python requests traffic through fiddler, it seems to pass without a problem. Is it possible that amazon is fingerprinting python requests and fiddler changes the fingerprint since it's a proxy?
I viewed headers sent from fiddler and python requests and they are the same.
There is no exra proxying/fiddler rules/filters set on fiddler to create a change.
To be clear, all mentioned proxying is only done locally, so it will not change the public ip address.
Thank you!
The reason is that websites are fingerprinting your requests with TLS hello package. There exist libraries like JA3 to generate a fingerprint for each request. They will intentionally block http clients like requests or urllib. If you uses a MITM proxy, because the proxy server create a new TLS connection with the server, the server only sees proxy server's fingerprint, so they will not block it.
If the server only blocks certain popular http libraries, you can simply change the TLS version, then you will have different fingerprint than the default one.
If the server only allows popular real-world browsers, and only accepts them as valid requests, you will need libraries that can simulate browser fingerprints, one of which is curl-impersonate and its python binding curl_cffi.
pip install curl_cffi
from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101")
print(r.json())
# output: {'ja3_hash': '53ff64ddf993ca882b70e1c82af5da49'
# the fingerprint should be the same as target browser
Looking at the documentation for urlib2 it says it supports HTTPS connections. However what it doesn't make clear is how you enable it do you for example take HTTPBasicAuth and replace the HTTP with HTTPS or do you just need to pass an HTTPS in url when you actually open the connection?
< Python 2.7.9:_
You can simply pass an HTTPS URL when you open the connection. Heed the warning in the Urllib2 documentation that states:
"Warning HTTPS requests do not do any verification of the server’s certificate."
As such, I recommend using Python Requests library that provides a better interface and many features, including SSL Cert verification and Unicode support.
Update 20150120:
Python 2.7.9 Added HTTPS Hostname verification as standard. See change comment in https://docs.python.org/2/library/httplib.html#httplib.HTTPSConnection
Thanks to #EnnoGröper for the notice.
I want to make a GET request to retrieve the contents of a web-page or a web service.
I want to send specific headers for this request AND
I want to set the IP address FROM WHICH this request will be sent.
(The server on which this code is running has multiple IP addresses available).
How can I achieve this with Python and its libraries?
I checked urllib2 and it won't set the source address (at least not on Python 2.7). The underlying library is httplib, which does have that feature, so you may have some luck using that directly.
From the httplib documentation:
class httplib.HTTPConnection(host[, port[, strict[, timeout[, source_address]]]])
The optional source_address parameter may be a tuple of a (host, port) to use as the source address the HTTP connection is made from.
You may even be able to convince urllib2 to use this feature by creating a custom HTTPHandler class. You will need to duplicate some code from urllib2.py, because AbstractHTTPHandler is using a simpler version of this call:
class AbstractHTTPHandler(BaseHandler):
# ...
def do_open(self, http_class, req):
# ...
h = http_class(host, timeout=req.timeout) # will parse host:port
Where http_class is httplib.HTTPConnection for HTTP connections.
Probably this would work instead, if patching urllib2.py (or duplicating and renaming it) is an acceptable workaround:
h = http_class(host, timeout=req.timeout, source_address=(req.origin_req_host,0))
There are many options available to you for making http requests. I don't even think there is really a commonly agreed upon "best". You could use any of these:
urllib2: http://docs.python.org/library/urllib2.html
requests: http://docs.python-requests.org/en/v0.10.4/index.html
mechanize: http://wwwsearch.sourceforge.net/mechanize/
This list is not exhaustive. Read the docs and take your pick. Some are lower level and some offer rich browser-like features. All of them let you set headers before making request.