I am trying to create a connection to a TLS (TLSv1) secured MQTT Broker(Rabbitmq with MQTT Plugin enabled) with the python implementation of the eclipse paho client. The same works fine with the MQTTFX application which is based on the java implementation of paho. For this i am using self signed certificates.
Java version uses:
CA-File: ca_certificate.crt
Client Certificate client_cert.crt
Client Key File: client_key.key
Python Version should use:
CA-File: ca_certificate.pem
Client Certificate: client_cert.pem
Client key file: client_key.key
I tried to establish a connection like this:
import ssl
import paho.mqtt.client as paho
# Locations of CA Authority, client certificate and client key file
ca_cert = "ca_certificate.pem"
client_cert = "client_certificate.pem"
client_key = "client_key.pem"
# Create ssl context with TLSv1
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
context.load_verify_locations(ca_cert)
context.load_cert_chain(client_cert, client_key)
# Alternative to using ssl context but throws the exact same error
# client.tls_set(ca_certs=ca_cert, certfile=client_cert, keyfile=client_key, tls_version=ssl.PROTOCOL_TLSv1)
client = paho.Client()
client.username_pw_set(username="USER", password="PASSWORD")
client.tls_set_context(context)
client.tls_insecure_set(False)
client.connect_async(host="HOSTNAME", port="PORT")
client.loop_forever()
Which results in the following error:
ssl.SSLError: [SSL: NO_CIPHERS_AVAILABLE] no ciphers available (_ssl.c:997)
Could it be that I need to explicitly pass a cipher that the broker supports or could it be due of an older openssl version? I am a little bit lost right now, maybe someone has a clue on how to solve this.
Edit: I got it to work by myself but still not sure why exactly it works now.
Changed context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
to context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
Changed client.tls_insecure_set(False)
to client.tls_insecure_set(True)
PROTOCOL_TLSv1 forces the client to only use TLS v1.0 which is old and unless you have explicitly forced your broker to only use the same version unlikely to match.
Using PROTOCOL_TLS_CLIENT will allow Python to negotiate across the full range of TLS v1.0 to TLS v1.3 until it finds one that both the client and the broker support.
Why you are having to set client.tls_insecure_set(True) is hard to answer without knowing more about the certificates you are using with the broker. Does it container a CA/SAN entry that matches the HOSTNAME you are using to connect? The documentation says it will explicitly enforce the hostname check.
ssl.PROTOCOL_TLS_CLIENT
Auto-negotiate the highest protocol version that both the client and
server support, and configure the context client-side connections. The
protocol enables CERT_REQUIRED and check_hostname by default.
I intend to connect to the remote host example.com over TLS but I have to connect through a proxy IP address with DNS name example-proxy.com.
I don't have control over the SSL certificate and I cannot ask the admin at example.com to add example-proxy.com to its certificate's SAN.
Using example-prxoy.com would cause OpenSSL to error out because the host name does not match the name in the certificate. How can I split the host parameter into two: (1) domain name for the network connection and (2) domain name for the certificate verification.
I don't have the resources to modify the OpenSSL library but I can make changes to the Python libraries. According to this doc, I could have modified the match_hostname method to implement this feature but it is no longer available as of Python 3.7+.
Asks
How can I use Python 3.7+ to specify both a host name and a certificate name?
From the security standpoint, How could my implementation go wrong?
Just give a different hostname for TCP connection and TLS handshake, i.e. set server_hostname in wrap_socket. To modify the example from the official documentation for this:
import socket
import ssl
tls_hostname = 'www.example.com'
context = ssl.create_default_context()
with socket.create_connection(('127.0.0.1',8443)) as sock:
with context.wrap_socket(sock, server_hostname=tls_hostname) as ssock:
print(ssock.version())
This will connect to ('127.0.0.1',8443) but do the TLS handshake with www.example.com.
Note that this will use tls_hostname for both SNI extension in the TLS handshake and for validating the certificate. But this seems to be what you need based on your question anyway: connect to IP:port but do TLS handshake and validation with a specific hostname.
I'm setting up SSL client verification in my python app. At the moment my proof-of-concept code is falling over just establishing a secure connection.
It looks like the certificates I've generated either have a certificate usage without the necessary permissions (more likely IMO) or they have permissions that the server cannot understand or accept (a little less likely IMO).
This should be relatively trivial, but I can't find the right documentation.
I've generated the server and client certificate through OpenSSL. I've done this in the past for other apps without any problem. But I'm much less familiar with creating client certificates. OpenSSL reports that the client certificate I'm using has extensions:
X509v3 extensions:
X509v3 Subject Key Identifier:
AF:AB:9D:AA:88:96:F4:0C:F5:56:9A:2C:DB:B6:BA:D9:DD:11:69:45
X509v3 Subject Alternative Name:
email:a#example.com
X509v3 Basic Constraints:
CA:FALSE
Netscape Cert Type:
SSL Client
X509v3 Authority Key Identifier:
keyid:E1:35:7C:39:7F:39:A4:43:D2:F8:00:59:38:91:71:AF:B9:38:AD:3F
X509v3 Key Usage:
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication
The trivial server test code is:
import ssl
import socket
import logging
_log = logging.getLogger(__name__)
def main():
context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
context.load_cert_chain("1B.pem", "key2.pem")
context.verify_mode = ssl.CERT_REQUIRED
context.load_verify_locations("my_ca.crt")
raw_server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
try:
# domain replaced for SO question
raw_server_socket.bind(('neptune.example.com', 8812))
raw_server_socket.listen(5)
server_socket = context.wrap_socket(raw_server_socket, server_side=True)
except Exception:
raw_server_socket.close()
raise
with server_socket:
while True:
try:
connection_to_client, address = server_socket.accept()
with connection_to_client:
connection_to_client.write(b'Hello')
except Exception as ex:
print(ex)
if __name__ == "__main__":
main()
This gives the error:
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unsupported certificate purpose (_ssl.c:1076)
... When the client connected with this:
import socket
import ssl
context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
context.load_cert_chain("1C.pem", "key.pem")
raw_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Domain changed for SO question
conn = context.wrap_socket(raw_socket, server_side=False, server_hostname="neptune.example.com")
conn.connect(("neptune.example.com", 8812))
conn.close()
As hinted at by Steffen Ullrich, it looks like the problem was not with the certificate itself but the CA certificate I used to sign it.
CA certificates are limited in the rights they can authorize. These are represented in the same extensions as the certificates they sign. So for a CA to sign a certificate as a client SSL certificate both the client certificate and the CA certificate must have:
Key usage with "digital signature".
the 1.3.6.1.5.5.7.3.2 extension AKA TLS Web Client Authentication
That is, openssl should report the following for both the CA certificate and the signed client cetificate:
X509v3 Key Usage:
Digital Signature
X509v3 Extended Key Usage:
TLS Web Client Authentication
In my case, the CA didn't have the extended use 1.3.6.1.5.5.7.3.2. It was fine for server certificates but couldn't sign client certificates.
Can anyone explain me how to add SSL to the Python STOMP client I'm using.
I added the stomp+ssl transport connector in the ActiveMQ configuration file and my basic Python STOMP client is below:
import time
import sys
import stomp
class MyListener(stomp.ConnectionListener):
def on_error(self, headers, message):
print('received an error "%s"' % message)
def on_message(self, headers, message):
print('received a message "%s"' % message)
conn = stomp.Connection()
conn.set_listener('', MyListener())
conn.start()
conn.connect('admin', 'password', wait=True)
conn.subscribe(destination='/queue/test', id=1, ack='auto')
conn.send(body=' '.join(sys.argv[1:]), destination='/queue/test')
time.sleep(2)
conn.disconnect()
I created the key store and trust store given in the http://activemq.apache.org/how-do-i-use-ssl.html docs and added them to the SSL_OPTS environment variable in the broker but I'm unable to find how to initialize the Python STOMP client with the key store and trust store. Should I use the SSL paraments given in the stomp.Connection() method, and if yes how to do so?
Can anyone please explain if there is any other way to add SSL over STOMP?
The Python STOMP client (as of version 4.1.20) uses an SSLContext to process its key pair/certificate, so there is no reason to produce a Java KeyStore for the client.
With this in mind, let us go through the entire process of setting up ApacheMQ to support SSL-wrapped STOMP connections. The process below has been tested on ApacheMQ 5.15.4. We explicitly set up two-way trust by manually moving self-signed certificates between the broker and client; using a certificate authority is also possible but how to do so is a different question.
Create a client certificate
As mentioned above, on the Python side of things, a KeyStore will have little use, and since SSLContext expects PEM encoded certificates, we might as well create the key pair and certificate by hand (that is, using openssl). First, on the client machine, let us create a 4096-bit RSA key:
openssl genrsa -out client.key 4096
Using this, turn the public key part into a certificate and sign it with the key itself; since we will be manually moving the certificate to the broker, self-signing the certificate is not an issue:
openssl req -new -out client.csr -key client.key
openssl x509 -req -days 365 -in client.csr -signkey client.key -out client.pem
rm client.csr
The STOMP client will need both the signed certificate, client.pem, and the private key, client.key, while the broker will only need the certificate.
Create a broker certificate
On the broker, we can follow the first part of the Apache guide and use the Java keytool to create a KeyStore with a key for the server:
keytool -genkeypair -alias broker -keyalg RSA -keysize 4096 -keystore broker.ks
When prompted for "first and last name", provide the hostname of the server, which in our example we will take simply to be localhost; if the broker and client are running on different servers, make sure that this is set to whatever the Python client will end up using to identify the broker:
What is your first and last name?
[Unknown]: localhost
All other input values can be left as "Unknown".
At the end of the day, we will only want to allow connections to the broker from clients with certificates that we know, so at this point copy the client.pem generated above to the broker and add it to a trust store through
keytool -import -alias client -keystore broker.ts -file client.pem
If the broker is to allow connections from any client, then this final step can be skipped.
Setting up ApacheMQ
By default, all connections through STOMP (and indeed all connections) are plaintext ones, and in order to enable STOMP connections over SSL, add the following <transportConnector /> to conf/apachemq.xml:
<transportConnectors>
<transportConnector name="stomp+ssl" uri="stomp+nio+ssl://0.0.0.0:61613?transport.enabledProtocols=TLSv1.2&needClientAuth=true" />
</transportConnectors>
Make sure to remove any existing plaintext connectors such as the default STOMP connector as otherwise clients will be able to simply use those and bypass the SSL requirement. Note also that needClientAuth=true is what forces client certificate validation; without this, clients are able to connect without providing a trusted certificate.
To configure ApacheMQ to use the key and trust stores defined above, define the environment variable ACTIVEMQ_SSL_OPTS through (on Unix)
export ACTIVEMQ_SSL_OPTS = -Djavax.net.ssl.keyStore=/path/to/broker.ks -Djavax.net.ssl.trustStore=/path/to/broker.ts -Djavax.net.ssl.keyStorePassword=passwordForBrokerKs -Djavax.net.ssl.trustStorePassword=passwordForBrokerTs
or (on Windows)
set ACTIVEMQ_SSL_OPTS=-Djavax.net.ssl.keyStore=C:\path\to\broker.ks -Djavax.net.ssl.trustStore=C:\path\to\broker.ts -Djavax.net.ssl.keyStorePassword=passwordForBrokerKs -Djavax.net.ssl.trustStorePassword=passwordForBrokerTs
Here, the two passwords are those chosen after running keytool in the previous step. If client certificate validation is not desired, simply leave out the configuration of trustStore and trustStorePassword.
With this, ActiveMQ can be started as usual through bin/activemq start. To make sure that the SSL configuration matches expectation, pay attention to the JVM args part of the output printed when starting the server.
Testing the STOMP client
With the broker properly set up, we can configure the client as well. Here, we provide stomp.Connection.set_ssl with references to the key and certificate created in the first step. Assuming that the ActiveMQ server is running on localhost:61613, your test script simply becomes
import time
import sys
import stomp
class MyListener(stomp.ConnectionListener):
def on_error(self, headers, message):
print('received an error "%s"' % message)
def on_message(self, headers, message):
print('received a message "%s"' % message)
host = 'localhost'
port = 61613
conn = stomp.Connection([(host, port)])
conn.set_ssl(for_hosts=[(host, port)], key_file='/path/to/client.key', cert_file='/path/to/client.pem')
conn.set_listener('', MyListener())
conn.start()
conn.connect('admin', 'password', wait=True)
conn.subscribe(destination='/queue/test', id=1, ack='auto')
conn.send(body='test message', destination='/queue/test')
time.sleep(2)
conn.disconnect()
To make sure that ApacheMQ is indeed validating the client certificate, we could repeat step 1 and create a new pair, client2.key/client2.pem say, and use that instead. Doing so should result in the following non-sensical error message being printed by ApacheMQ:
ERROR | Could not accept connection from null : {}
java.io.IOException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
Validating the broker certificate
Now, the attentive reader will have noticed that we never actually moved the broker certificate to the client, and yet things seem to work regardless. As it turns out, the default behavior of stomp.py is to perform no certificate validation at all, allowing an (active) attacker to MITM the connection.
As we are rolling self-signed certificates, all we need to do to fix this situation is to provide the actual broker certificate to the Python client. On the broker, export the certificate through
keytool -exportcert -rfc -alias broker -keystore broker.ks -file broker.pem
and move broker.pem to the Python client. Now, in the test script above, include the certificate by replacing the SSL configuration with
conn.set_ssl(for_hosts=[(host, port)], key_file='/path/to/client.key', cert_file='/path/to/client.pem', ca_certs='/path/to/broker.pem')
As above, we can test that this is indeed performing the proper validation by repeating the broker certificate generation process to create a broker2.pem, use that in the test script, and note that it will fail with an
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)
Try this.
conn = stomp.Connection([(host, port)])
conn.set_listener('', MyListener())
conn.set_ssl(for_hosts=[(host, port)], ssl_version=ssl.PROTOCOL_TLS)
conn.start()
conn.connect(login, password, wait=True)
conn.send(body=message, destination=queue)
conn.disconnect()
or
conn.set_ssl(for_hosts=[(host, port)], ssl_version=_ssl.PROTOCOL_TLS)
The key point is understanding the key/cert files must be there - otherwise even stomp.py tests in their repository will not work. TLS will not work in stomp.py if the file names are not specified (correctly).
See the code & key generation instructions here: https://gist.github.com/borislitvak/6ccea503abf1b2f9c89e87309d6dab88
UPDATED: fixed/working code at bottom of question. Re-implemented to use wrap_socket() instead of SSL.Context(). Negotiated cipher seems unaffected by ssl.PROTOCOL_
I've got an XML service which listens on port 8388. The service has it's own CA built in and will issue a P12 file for any client configured on it's admin interface (I need to include a CA cert in my connections).
I've configured a client and downloaded the P12 file; also exported PEM files (for easier debugging) from the P12 and exported a PEM file for the server CA (which was a Java keystore).
If I use openssl s_client and feed it the client-side cert/key and CA file, things seem to work correctly; ie: verify return:1 in the truncated output below. The openssl process does not generate any certificate errors on the server (as does my Python script). From what I've read, this is supposed to mean the certs are all OK and valid.
# openssl s_client -connect srv.domain.net:8388 -CAfile server_ca.pem -cert client_cert.pem -key client_key.pem
CONNECTED(00000003)
depth=1 ... emailAddress = ca#SERVERsystems.com
verify return:1
depth=0 ...CN = srv.domain.net
verify return:1
---
Certificate chain
0 s:/emailAddress=help#mydomain.net...CN=srv.domain.net
i:/C=US/...emailAddress=ca#SERVERsystems.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIJRDCCByygAwIBAgIGAVGIopaoMA0GCSqGSIb3DQEBDQUAMIG...rV
-----END CERTIFICATE-----
subject=/emailAddress=help#mydomain.net...CN=srv.domain.net
issuer=/C=US/...emailAddress=ca#SERVERsystems.com
---
Acceptable client certificate CA names
/C=US/...emailAddress=ca#SERVERsystems.com
/.../CN=srv.domain.net
---
SSL handshake has read 3369 bytes and written 5705 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: ...DF6572FA00D62D83CF0B1A82F
Session-ID-ctx:
Master-Key: ...7F685C0B163A7739C271E9722FC0554108175C4
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1461337153
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
I am now attempting to hook up a Python (2.7.9) script to the XML service using the same cert, key and CA file but I can't get it working. Python is complaining about SSLv3 errors and the server says it can't verify the client. So, the connection works, but the handshake, certs or ciphers or something aren't right.
I've searched out numerous examples and implementations and this one seemed to be the simplest one so I started with it for a template. SSL3 is not the protocol I would stick with (POODLE), but it was supposed to be from a working example so I wanted to make as few changes as possible to get things working and then tweak from there. Anyone know what I've got wrong here? ouput/errors/logs are posted at the far bottom.
#!/usr/bin/python
# -*- coding: utf-8 -*-
import socket
import OpenSSL
from OpenSSL import *
import sys
serverName = sys.argv[1]
print "Using server : " + serverName
ctx = SSL.Context(SSL.SSLv3_METHOD)
ctx.use_privatekey_file('client_key.pem')
ctx.use_certificate_file('client_cert.pem')
ctx.load_verify_locations(cafile='server_ca.pem')
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((serverName, 8388))
sslSocket = socket.ssl(s)
print repr(sslSocket.server())
print repr(sslSocket.issuer())
print ("writing socket..")
sslSocket.write('<transaction><data>14</data></transaction>\n')
s.close()
Python output:
Using server : localhost
Traceback (most recent call last):
File "./test3.py", line 29, in <module>
sslSocket = socket.ssl(s)
File "/usr/lib/python2.7/socket.py", line 64, in ssl
return _realssl.sslwrap_simple(sock, keyfile, certfile)
File "/usr/lib/python2.7/ssl.py", line 993, in sslwrap_simple
ssl_sock.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:581)
Server logs after connection above:
Apr 22 10:39:56 srv.domain.net ERROR server.auth [Thread-183,run:233] Couldn't validate the client certificate. Verify the validity and dates of the client cert.
Apr 22 10:39:56 srv.domain.net javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
Apr 22 10:39:56 srv.domain.net at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:431)
Apr 22 10:39:56 srv.domain.net at com.xml.server.auth.run(auth.java:226)
Apr 22 10:39:56 srv.domain.net at java.lang.Thread.run(Thread.java:745)
Working code:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import socket
import ssl
import sys
import os
serverName = sys.argv[1]
print "Using server : " + serverName
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# ssl.PROTOCOL_xxx does not seem to affect negotiated cipher??
wrapper = ssl.wrap_socket(s,
ca_certs = "server_ca.pem",
certfile = "client_cert.pem",
keyfile = "client_key.pem",
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1)
wrapper.connect((serverName, 8388))
# some info on the connnection/cipher in use
print repr(wrapper.getpeername())
print repr(wrapper.cipher())
print repr(wrapper.getpeercert())
# send server command
print ("writing socket..")
wrapper.send ("<transaction><data>14</data></transaction>\n")
# read server reply
print "server reply: " + wrapper.recv(4096)
wrapper.close()
s.close()
ctx.use_privatekey_file('client_key.pem')
ctx.use_certificate_file('client_cert.pem')
...
sslSocket = socket.ssl(s)
While you create an SSL context with the client certificate you don't use this context within your SSL connection. This means no client certificate will be send and the server complains accordingly.
A much simpler way to use client certificates is by using the certfile and keyfile parameter of ssl.wrap_socket, see the documentation.