I need to edit the python request to add TLS settings (by TLS settings I mean tls finger printing int, ja3).
You can use curl_cffi. Although you can not modify the TLS settings at python level, you can change them when using C extensions, such as curl.
pip install curl_cffi
It has a requests-like interface:
from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101")
print(r.json())
# output: {'ja3_hash': '53ff64ddf993ca882b70e1c82af5da49'
# the fingerprint should be the same as target browser
The JA3 fingerprint is based on ciphers and order and various TLS extensions and order. While ciphers and order can be changed features like the TLS extension order are not accessible from Python. This means there is no way to emulate a specific JA3 fingerprint from Python and thus also not from requests.
Related
I am trying to replicate the following client requests via python-requests.
Under client connection I see HTTP Version which is 2.0 and TLS version which is 1.3, as up to my knowledge I know that requests utilizes TLS 1.3. My requests are failing as of now.
And I wonder if I need to pass certificates. I would like to understand how this request is different from regular request which would be simply as
r = requests.get('someurl')
How can I use requests to use the exact client connection show in requests? I don't fully understand each pointer, How would I use h2 ALPN/ with that specific cipher name? I am not expecting an solid answer to the question rather an explanation would be much more helpful!
python-requests doesn't support HTTP 2 request. You can use httpx package.
HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2.
Example
import ssl
import httpx
# create an ssl context
ssl_context = ssl.SSLContext(protocol=ssl.PROTOCOL_TLS)
# ssl.PROTOCOL_TLS - Selects the highest protocol version that both the client and server support.
# Despite the name, this option can select both "SSL" and "TLS" protocols.
# set protocol to use
ssl_context.set_alpn_protocols(["h2"])
CIPHERS = 'ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:DH+HIGH:RSA+AESGCM:RSA+AES:RSA+HIGH:!aNULL:!eNULL:!MD5:!3DES'
# set ciphers
ssl_context.set_ciphers(CIPHERS)
# httpx verify param lets you pass a standard library ssl.SSLContext
response = httpx.get('https://example.com', verify=ssl_context)
print(response.http_version)
# outputs HTTP/2
Instead of using ssl.SSLContext, you can also use httpx.create_ssl_context() to set the ssl context.
As far as I know python-requests is a library which currently1 doesn't support HTTP/2.0. This question has been answered here.
However there are python libraries like Python httpx supporting HTTP/2.0!
Kind regards,
1 Feb 16, 2021
I need to edit the python request to add TLS settings (by TLS settings I mean tls finger printing int, ja3).
You can use curl_cffi. Although you can not modify the TLS settings at python level, you can change them when using C extensions, such as curl.
pip install curl_cffi
It has a requests-like interface:
from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101")
print(r.json())
# output: {'ja3_hash': '53ff64ddf993ca882b70e1c82af5da49'
# the fingerprint should be the same as target browser
The JA3 fingerprint is based on ciphers and order and various TLS extensions and order. While ciphers and order can be changed features like the TLS extension order are not accessible from Python. This means there is no way to emulate a specific JA3 fingerprint from Python and thus also not from requests.
I'm hacking together an amazon api and when only using python requests without proxying, it prompts for a captcha. When routing this python requests traffic through fiddler, it seems to pass without a problem. Is it possible that amazon is fingerprinting python requests and fiddler changes the fingerprint since it's a proxy?
I viewed headers sent from fiddler and python requests and they are the same.
There is no exra proxying/fiddler rules/filters set on fiddler to create a change.
To be clear, all mentioned proxying is only done locally, so it will not change the public ip address.
Thank you!
The reason is that websites are fingerprinting your requests with TLS hello package. There exist libraries like JA3 to generate a fingerprint for each request. They will intentionally block http clients like requests or urllib. If you uses a MITM proxy, because the proxy server create a new TLS connection with the server, the server only sees proxy server's fingerprint, so they will not block it.
If the server only blocks certain popular http libraries, you can simply change the TLS version, then you will have different fingerprint than the default one.
If the server only allows popular real-world browsers, and only accepts them as valid requests, you will need libraries that can simulate browser fingerprints, one of which is curl-impersonate and its python binding curl_cffi.
pip install curl_cffi
from curl_cffi import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101")
print(r.json())
# output: {'ja3_hash': '53ff64ddf993ca882b70e1c82af5da49'
# the fingerprint should be the same as target browser
I'm using Python 3 and want to use SSL for a client and server I am writing.
I would like to be able to debug what gets sent/received. Obviously, SSL prevents doing that with Wireshark, unless I can make Python use a non-perfect forward secrecy and give the server key to wireshark, or if I can log the symmetric key generated for each session and save it in a file.
I'd like to do that latter, as it should work with any of the ciphers used.
I should be able to handle the logging to a file part. But I'm not sure how to retrieve the symmetric key used in an TLS session with the Python 'ssl' library. How do I go about retrieving those keys (preferably server side, but I could do it in the client if necessary)?
I have the following code to use:
def createCon(host,auth):
con = httplib.HTTPSConnection(host)
return con
def _readJson(con,url):
con.putrequest("GET",url)
...
r = con.getresponse()
It is working on a specific server, but on another I'm getting SSLError. If I open the url from browser, I need to accept the certificate and it is working well. But how can I either accept the certificate or disable it's validation? This is a self-signed certificate stored in a java keystore, so I would prefer to disable the verification...
The code is meant to reuse the connection between the requests, so I would prefer not to modify it deeply.
How could I do this? I tried to modify the examples but haven't beend succeded.
con.putrequest("GET",url, verify=False)
or
con.request.verify=False
I do not know how could I access the session or request objects or modify the default settings.
UPDATE this does not help:
socket.ssl.cert_reqs='CERT_NONE'
well, the actual error message is weird...:
SSLError:'[Errno 1] _ssl.c:492: error:100AE081:elliptic curve routines:EC_GROUP_new_by_curve_name:unknown group'
Regards:
Bence
Your error message points to a bug in the openssl version you use. See https://bugzilla.redhat.com/show_bug.cgi?id=1022468. In short: the client advertises capabilities it does not have and if the server picks such capability you get this error message. Needs to be fixed by upgrading your local openssl installation. A workaround on the server side should be possible too, if you have control over the server.