In my twisted app I want to make an asynchronous request to Akismet to check for spam. Akismet reasonably uses HTTPS, so I've been following the web client guide on SSL in the docs. But there's this part that worries me:
Here’s an example which shows how to use Agent to request an HTTPS URL with no certificate verification.
I very much want certificate verification to prevent Man-In-The-Middle attacks. So how do I add it?
My test code without verification is this:
from twisted.internet import reactor
from twisted.web.client import Agent
from twisted.internet.ssl import ClientContextFactory
class WebClientContextFactory(ClientContextFactory):
def getContext(self, hostname, port):
print( "getting context for {}:{}".format( hostname, port ) )
# FIXME: no attempt to verify certificates!
return ClientContextFactory.getContext(self)
agent = Agent( reactor, WebClientContextFactory() )
def success( response ):
print( "connected!" )
def failure( failure ):
print( "failure: {}".format( failure ) )
def stop( ignored ):
reactor.stop()
agent.request( "GET", "https://www.pcwebshop.co.uk/" )\ # uses self-signed cert
.addCallbacks( success, failure )\
.addBoth( stop )
reactor.run()
I'd like it to fail due to inability to verify the cert.
I'm using Twisted 15.1.0.
Actually, the default init function of Agent will pass in BrowserLikePolicyForHTTPS as contextFactory and have the ablility to verify server certificate.
Simply using this:
agent = Agent( reactor )
will produce the following error:
failure: [Failure instance: Traceback (failure with no frames):
<class 'twisted.web._newclient.ResponseNeverReceived'>:
[<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>]]
Make sure you installed service_identity package using pip.
If you need custom cert verification, you can create a custom policy by passing the pem in, as described here:
customPolicy = BrowserLikePolicyForHTTPS(
Certificate.loadPEM(FilePath("your-trust-root.pem").getContent())
)
agent = Agent(reactor, customPolicy)
Thanks for pointing this out. This seems to be a bug in the documentation. Prior to version 14.0, it was accurate; Twisted would not validate HTTPS, and that was a big problem. However, as you can see in the release notes for that version, Twisted (at least in versions 14.0 and greater) does validate TLS on HTTPS connections made with Agent. (It still does not do so for getPage, the old, bad, HTTP client; do not use getPage.)
I have filed this bug to track fixing the documentation to be accurate.
Related
I am running a Python code where I have to get some data from HTTPSConnectionPool(host='ssd.jpl.nasa.gov', port=443). But every time I try to run the code I get the following error. I am on MAC OS 12.1
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='ssd.jpl.nasa.gov', port=443): Max retries exceeded with url: /api/horizons.api?format=text&EPHEM_TYPE=OBSERVER&QUANTITIES_[...]_ (Caused by SSLError(SSLError(1, '[SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:997)')))
I really don't know how to bypass this issue.. thank you for the help!
SERIOUS DISCLAIMER: This answer is only left for educational and testing purposes. It described how to easily disable SSL verification through /etc/openssl.conf for all applications (and all systems). When done in a regulated entity, this is likely to be considered a security break and audit break that may constitute grounds for your termination and affect cybersecurity insurances.
WARNING: When enabling Legacy Unsafe Renegotiation, SSL connections will be vulnerable to the Man-in-the-Middle prefix attack as described in CVE-2009-3555.
With the help of https://bugs.launchpad.net/bugs/1963834
and https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1856428
Beware that editing your system's openssl.conf is not recommended, because you might lose your changes once openssl is updated.
Create a custom openssl.cnf file in any directory with these contents:
openssl_conf = openssl_init
[openssl_init]
ssl_conf = ssl_sect
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
Options = UnsafeLegacyRenegotiation
Before running your program, make sure your OPENSSL_CONF environment variable is set to your custom openssl.cnf full path when running the scraper like so:
OPENSSL_CONF=/path/to/custom/openssl.cnf python your_scraper.py
or like so:
export OPENSSL_CONF=/path/to/custom/openssl.cnf
python your_scraper.py
or, if you are using pipenv or systemd or docker, place this into your .env file
OPENSSL_CONF=/path/to/custom/openssl.cnf
I hit the same error on Linux (it happens when the server doesn't support "RFC 5746 secure renegotiation" and the client is using OpenSSL 3, which enforces that standard by default).
Here is a solution (you may have to adjust it slightly).
Import ssl and urllib3 in your Python code
Create a custom HttpAdapter which uses a custom ssl Context
class CustomHttpAdapter (requests.adapters.HTTPAdapter):
'''Transport adapter" that allows us to use custom ssl_context.'''
def __init__(self, ssl_context=None, **kwargs):
self.ssl_context = ssl_context
super().__init__(**kwargs)
def init_poolmanager(self, connections, maxsize, block=False):
self.poolmanager = urllib3.poolmanager.PoolManager(
num_pools=connections, maxsize=maxsize,
block=block, ssl_context=self.ssl_context)
Set up an ssl context which enables OP_LEGACY_SERVER_CONNECT, and use it with your custom adapter.
ssl.OP_LEGACY_SERVER_CONNECT is not available in Python yet (https://bugs.python.org/issue44888). However it turns out that in OpenSSL its value is 0x4 in the bitfield. So we can do the following.
ctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
ctx.options |= 0x4
session.mount('https://', CustomHttpAdapter(ctx))
This error comes up when using OpenSSL 3 to connect to a server which does not support it. The solution is to downgrade the cryptography package in python:
run pip install cryptography==36.0.2 in the used enviroment.
source: https://github.com/scrapy/scrapy/issues/5491
EDIT: Refer to Hally Mallon and ahmkara's answer for a fix without downgrading cryptography
Complete code snippets for Harry Mallon's answer:
Define a method for reuse:
import requests
import urllib3
import ssl
class CustomHttpAdapter (requests.adapters.HTTPAdapter):
# "Transport adapter" that allows us to use custom ssl_context.
def __init__(self, ssl_context=None, **kwargs):
self.ssl_context = ssl_context
super().__init__(**kwargs)
def init_poolmanager(self, connections, maxsize, block=False):
self.poolmanager = urllib3.poolmanager.PoolManager(
num_pools=connections, maxsize=maxsize,
block=block, ssl_context=self.ssl_context)
def get_legacy_session():
ctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
ctx.options |= 0x4 # OP_LEGACY_SERVER_CONNECT
session = requests.session()
session.mount('https://', CustomHttpAdapter(ctx))
return session
Then use it in place of the requests call:
get_legacy_session().get("some-url")
This doesn't really answer the issue, but a coworker switched from Node 18 to 16 and stopped getting this error.
For me, it worked when I downgraded python to v3.10.8.
(If you are facing the issue in docker container, read below)
In my docker image, I was using alpine-10 which was using v3.10.9. Since I couldn't get alpine with v3.10.8, I used 3.10.8-slim-bullseye.
I am trying to create a connection to a TLS (TLSv1) secured MQTT Broker(Rabbitmq with MQTT Plugin enabled) with the python implementation of the eclipse paho client. The same works fine with the MQTTFX application which is based on the java implementation of paho. For this i am using self signed certificates.
Java version uses:
CA-File: ca_certificate.crt
Client Certificate client_cert.crt
Client Key File: client_key.key
Python Version should use:
CA-File: ca_certificate.pem
Client Certificate: client_cert.pem
Client key file: client_key.key
I tried to establish a connection like this:
import ssl
import paho.mqtt.client as paho
# Locations of CA Authority, client certificate and client key file
ca_cert = "ca_certificate.pem"
client_cert = "client_certificate.pem"
client_key = "client_key.pem"
# Create ssl context with TLSv1
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
context.load_verify_locations(ca_cert)
context.load_cert_chain(client_cert, client_key)
# Alternative to using ssl context but throws the exact same error
# client.tls_set(ca_certs=ca_cert, certfile=client_cert, keyfile=client_key, tls_version=ssl.PROTOCOL_TLSv1)
client = paho.Client()
client.username_pw_set(username="USER", password="PASSWORD")
client.tls_set_context(context)
client.tls_insecure_set(False)
client.connect_async(host="HOSTNAME", port="PORT")
client.loop_forever()
Which results in the following error:
ssl.SSLError: [SSL: NO_CIPHERS_AVAILABLE] no ciphers available (_ssl.c:997)
Could it be that I need to explicitly pass a cipher that the broker supports or could it be due of an older openssl version? I am a little bit lost right now, maybe someone has a clue on how to solve this.
Edit: I got it to work by myself but still not sure why exactly it works now.
Changed context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
to context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
Changed client.tls_insecure_set(False)
to client.tls_insecure_set(True)
PROTOCOL_TLSv1 forces the client to only use TLS v1.0 which is old and unless you have explicitly forced your broker to only use the same version unlikely to match.
Using PROTOCOL_TLS_CLIENT will allow Python to negotiate across the full range of TLS v1.0 to TLS v1.3 until it finds one that both the client and the broker support.
Why you are having to set client.tls_insecure_set(True) is hard to answer without knowing more about the certificates you are using with the broker. Does it container a CA/SAN entry that matches the HOSTNAME you are using to connect? The documentation says it will explicitly enforce the hostname check.
ssl.PROTOCOL_TLS_CLIENT
Auto-negotiate the highest protocol version that both the client and
server support, and configure the context client-side connections. The
protocol enables CERT_REQUIRED and check_hostname by default.
In the official doc of HTTP PROTOCOL CLIENT
class http.client.HTTPSConnection(host, port=None, key_file=None,
cert_file=None, [timeout, ]source_address=None, *, context=None,
check_hostname=None)
A subclass of HTTPConnection that uses SSL for communication with
secure servers. Default port is 443. If context is specified, it must
be a ssl.SSLContext instance describing the various SSL options.
Is there any option to disable ssl verification like python requests library as verify=fasle
For some reasons I can't use HTTPConnection class which would be a straight forward solution. I have to use HTTPSConnection and work with
HTTPConnection.putrequest()
to send request without the ssl verification.
while creating https connection make sure that you pass context parameter as follows
import http.client
import ssl
conn = http.client.HTTPSConnection(
HOSTNAME,
context = ssl._create_unverified_context()
)
I try to connect to a FTP Server which only supports TLS 1.2
Using Python 3.4.1
My Code:
import ftplib
import ssl
ftps = ftplib.FTP_TLS()
ftps.ssl_version = ssl.PROTOCOL_TLSv1_2
print (ftps.connect('108.61.166.122',31000))
print(ftps.login('test','test123'))
ftps.prot_p()
print (ftps.retrlines('LIST'))
Error on client side:
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:598)
Error on server side:
Failed TLS negotiation on control channel, disconnected. (SSL_accept():
error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol)
The credentials in the example are working for testing.
See the end of this post for the final solution. The rest are the steps needed to debug the problem.
I try to connect to a FTP Server which only supports TLS 1.2 Using Python 3.4.1
How do you know?
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:598)
I would suggest one of the many SSL problems between client and server, like the server not supporting TLS 1.2, no common ciphers etc. These problems are hard to debug because you either get only some SSL alert or the server will simply close the connection without any obvious reason. If you have access to the server look for error messages on the server side.
You may also try to not to enforce an SSL version but use the default instead, so that client and server will agree to the best SSL version both support. If this will still not work try with a client which is known to work with this server and make a packet capture of the good and bad connections and compare. If you need help with that post the packet captures to cloudshark.org.
Edit#1: just tried it with python 3.4.0 and 3.4.2 against a test server:
python 3.4.0 does a TLS 1.0 handshake, i.e. ignores the setting
python 3.4.2 does a successful TLS 1.2 handshake
In both versions ftplib has the minor bug, that it sends AUTH SSL instead of AUTH TLS if ftps.ssl_version is something else then TLS 1.0, i.e. SSLv3 or TLS1.1.+. While I doubt that this is the origin of the problem it might actually be if the FTP server handles AUTH TLS and AUTH SSL differently.
Edit#2 and Solution:
A packet capture shows that setting ftps.ssl_version has no effect and the SSL handshake will still be done with TLS 1.0 only. Looking at the source code of ftplib in 3.4.0 gives:
ssl_version = ssl.PROTOCOL_TLSv1
def __init__(self, host='', user='', passwd='', acct='', keyfile=None,
certfile=None, context=None,
timeout=_GLOBAL_DEFAULT_TIMEOUT, source_address=None):
....
if context is None:
context = ssl._create_stdlib_context(self.ssl_version,
certfile=certfile,
keyfile=keyfile)
self.context = context
Since __init__ is called when ftplib.FTP_TLS() is called the SSL context will be created with the default ssl_version used by ftplib (ssl.PROTOCOL_TLSv1) and not with your own version. To enforce another SSL version you must to provide your own context with the needed SSL version. The following works for me:
import ftplib
import ssl
ctx = ssl._create_stdlib_context(ssl.PROTOCOL_TLSv1_2)
ftps = ftplib.FTP_TLS(context=ctx)
print (ftps.connect('108.61.166.122',31000))
print(ftps.login('test','test123'))
ftps.prot_p()
print (ftps.retrlines('LIST'))
Alternatively you could set the protocol version globally instead of only for this FTP_TLS object:
ftplib.FTP_TLS.ssl_version = ssl.PROTOCOL_TLSv1_2
ftps = ftplib.FTP_TLS()
And just a small but important observation: it looks like that ftplib does not do any kind of certificate validation, since it accepts this self-signed certificate which does not match the name without complaining. This makes a active man-in-the-middle attack possible. Hopefully they will fix this insecure behavior in the future, in which case the code here will fail because of an invalid certificate.
Firstly AFAIK no ftp supports SSL directly, for which ftps is introduced. Also sftp and ftps are two different concepts: http://en.wikipedia.org/wiki/FTPS .Now, your problem is regarding the programming and not related to SSL or FTPs or any such client-server communication
import ftplib
import ssl
ftps = ftplib.FTP_TLS()
#ftps.ssl_version = ssl.PROTOCOL_TLSv1_2
print (ftps.connect('108.61.166.122',31000))
print(ftps.login('test','test123'))
ftps.prot_p()
print (ftps.retrlines('LIST'))
as ftplib has no attribute PROTOCOL_TLSv1_2 besides which it works fine. and well, your host is not responding !
Hopefully it helps !
How do I validate the SSL server certificates in my twisted SSL client?
I am very much beginner to the SSL, I have gone through the twisted SSL tutorials but still I am unclear about some things.
My queries are:
How should I validate the SSL server certificate using twisted.internet.ssl module,
How ssl.ClientContextFactory.getContext method is useful while dealing with SSL,
How can I tell twisted SSL client about the location of the public key file?
Since Twisted 14.0, optionsForClientTLS is the best way to do this:
from twisted.internet.ssl import optionsForClientTLS
from twisted.internet.endpoints import SSL4ClientEndpoint
ctx = optionsForClientTLS(hostname=u"example.com")
endpoint = SSL4ClientEndpoint(reactor, host, port, ctx)
factory = ...
d = endpoint.connect(factory)
...
optionsForClientTLS takes other (optional) arguments as well which may also be useful.
Prior to Twisted 14.0, the process was a bit more involved:
After the connection is established and the SSL handshake has completed successfully (which means the certificate is currently valid based on its notBefore and notAfter values and that it is signed by a certificate authority certificate which you have indicated is trusted) you can get the certificate from the transport:
certificate = self.transport.getPeerCertificate()
The certificate is represented as a pyOpenSSL X509 instance. It has a method you can use to retrieve the certificate's subject name:
subject_name = certificate.get_subject()
The subject name is a distinguished name, represented as a pyOpenSSL X509Name instance. You can inspect its fields:
common_name = subject_name.commonName
This is a string, for example "example.com".
If you need to inspect the subjectAltName instead (which it's likely you do), then you can find this information in the extensions of the certificate:
extensions = certificate.get_extensions()
This is a list of pyOpenSSL X509Extension instances. You'll have to parse each one to find the subjectAltName and its value.