A python deployment our team has fails on
import socket
import ssl
context = ssl.wrap_socket(
sock,
ca_certs='root_ca_bundle.pem',
keyfile='pk.pem',
certfile='cert.pem',
do_handshake_on_connect=True,
cert_reqs=ssl.CERT_REQUIRED,
)
which is traced to failing on load_cert chain function with
if certfile:
context.load_cert_chain(certfile, keyfile)
The error I see is this
[X509: KEY_VALUES_MISMATCH] key values mismatch (_ssl.c:4067)
The real issue is that I copied a wrap_socket call using the same certfile,keyfile and ca_certs, and I don't see this issue.
It only appears on deployment.
To my understanding, load_cert_chain just verifies that the certfile and keyfile are mathematically aligned, using something like this:
$ openssl x509 -noout -modulus -in cert.pem
$ openssl rsa -noout -modulus -in key-no-pass.pem
Tried this on key and cert and verified they're the same.
What I am asking is :
Is there an explanation other than wrong keyfile or certfile being used in deployment?
Is there a way I can find source code for load_cert_chain function.
Can anyone explain me how to add SSL to the Python STOMP client I'm using.
I added the stomp+ssl transport connector in the ActiveMQ configuration file and my basic Python STOMP client is below:
import time
import sys
import stomp
class MyListener(stomp.ConnectionListener):
def on_error(self, headers, message):
print('received an error "%s"' % message)
def on_message(self, headers, message):
print('received a message "%s"' % message)
conn = stomp.Connection()
conn.set_listener('', MyListener())
conn.start()
conn.connect('admin', 'password', wait=True)
conn.subscribe(destination='/queue/test', id=1, ack='auto')
conn.send(body=' '.join(sys.argv[1:]), destination='/queue/test')
time.sleep(2)
conn.disconnect()
I created the key store and trust store given in the http://activemq.apache.org/how-do-i-use-ssl.html docs and added them to the SSL_OPTS environment variable in the broker but I'm unable to find how to initialize the Python STOMP client with the key store and trust store. Should I use the SSL paraments given in the stomp.Connection() method, and if yes how to do so?
Can anyone please explain if there is any other way to add SSL over STOMP?
The Python STOMP client (as of version 4.1.20) uses an SSLContext to process its key pair/certificate, so there is no reason to produce a Java KeyStore for the client.
With this in mind, let us go through the entire process of setting up ApacheMQ to support SSL-wrapped STOMP connections. The process below has been tested on ApacheMQ 5.15.4. We explicitly set up two-way trust by manually moving self-signed certificates between the broker and client; using a certificate authority is also possible but how to do so is a different question.
Create a client certificate
As mentioned above, on the Python side of things, a KeyStore will have little use, and since SSLContext expects PEM encoded certificates, we might as well create the key pair and certificate by hand (that is, using openssl). First, on the client machine, let us create a 4096-bit RSA key:
openssl genrsa -out client.key 4096
Using this, turn the public key part into a certificate and sign it with the key itself; since we will be manually moving the certificate to the broker, self-signing the certificate is not an issue:
openssl req -new -out client.csr -key client.key
openssl x509 -req -days 365 -in client.csr -signkey client.key -out client.pem
rm client.csr
The STOMP client will need both the signed certificate, client.pem, and the private key, client.key, while the broker will only need the certificate.
Create a broker certificate
On the broker, we can follow the first part of the Apache guide and use the Java keytool to create a KeyStore with a key for the server:
keytool -genkeypair -alias broker -keyalg RSA -keysize 4096 -keystore broker.ks
When prompted for "first and last name", provide the hostname of the server, which in our example we will take simply to be localhost; if the broker and client are running on different servers, make sure that this is set to whatever the Python client will end up using to identify the broker:
What is your first and last name?
[Unknown]: localhost
All other input values can be left as "Unknown".
At the end of the day, we will only want to allow connections to the broker from clients with certificates that we know, so at this point copy the client.pem generated above to the broker and add it to a trust store through
keytool -import -alias client -keystore broker.ts -file client.pem
If the broker is to allow connections from any client, then this final step can be skipped.
Setting up ApacheMQ
By default, all connections through STOMP (and indeed all connections) are plaintext ones, and in order to enable STOMP connections over SSL, add the following <transportConnector /> to conf/apachemq.xml:
<transportConnectors>
<transportConnector name="stomp+ssl" uri="stomp+nio+ssl://0.0.0.0:61613?transport.enabledProtocols=TLSv1.2&needClientAuth=true" />
</transportConnectors>
Make sure to remove any existing plaintext connectors such as the default STOMP connector as otherwise clients will be able to simply use those and bypass the SSL requirement. Note also that needClientAuth=true is what forces client certificate validation; without this, clients are able to connect without providing a trusted certificate.
To configure ApacheMQ to use the key and trust stores defined above, define the environment variable ACTIVEMQ_SSL_OPTS through (on Unix)
export ACTIVEMQ_SSL_OPTS = -Djavax.net.ssl.keyStore=/path/to/broker.ks -Djavax.net.ssl.trustStore=/path/to/broker.ts -Djavax.net.ssl.keyStorePassword=passwordForBrokerKs -Djavax.net.ssl.trustStorePassword=passwordForBrokerTs
or (on Windows)
set ACTIVEMQ_SSL_OPTS=-Djavax.net.ssl.keyStore=C:\path\to\broker.ks -Djavax.net.ssl.trustStore=C:\path\to\broker.ts -Djavax.net.ssl.keyStorePassword=passwordForBrokerKs -Djavax.net.ssl.trustStorePassword=passwordForBrokerTs
Here, the two passwords are those chosen after running keytool in the previous step. If client certificate validation is not desired, simply leave out the configuration of trustStore and trustStorePassword.
With this, ActiveMQ can be started as usual through bin/activemq start. To make sure that the SSL configuration matches expectation, pay attention to the JVM args part of the output printed when starting the server.
Testing the STOMP client
With the broker properly set up, we can configure the client as well. Here, we provide stomp.Connection.set_ssl with references to the key and certificate created in the first step. Assuming that the ActiveMQ server is running on localhost:61613, your test script simply becomes
import time
import sys
import stomp
class MyListener(stomp.ConnectionListener):
def on_error(self, headers, message):
print('received an error "%s"' % message)
def on_message(self, headers, message):
print('received a message "%s"' % message)
host = 'localhost'
port = 61613
conn = stomp.Connection([(host, port)])
conn.set_ssl(for_hosts=[(host, port)], key_file='/path/to/client.key', cert_file='/path/to/client.pem')
conn.set_listener('', MyListener())
conn.start()
conn.connect('admin', 'password', wait=True)
conn.subscribe(destination='/queue/test', id=1, ack='auto')
conn.send(body='test message', destination='/queue/test')
time.sleep(2)
conn.disconnect()
To make sure that ApacheMQ is indeed validating the client certificate, we could repeat step 1 and create a new pair, client2.key/client2.pem say, and use that instead. Doing so should result in the following non-sensical error message being printed by ApacheMQ:
ERROR | Could not accept connection from null : {}
java.io.IOException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
Validating the broker certificate
Now, the attentive reader will have noticed that we never actually moved the broker certificate to the client, and yet things seem to work regardless. As it turns out, the default behavior of stomp.py is to perform no certificate validation at all, allowing an (active) attacker to MITM the connection.
As we are rolling self-signed certificates, all we need to do to fix this situation is to provide the actual broker certificate to the Python client. On the broker, export the certificate through
keytool -exportcert -rfc -alias broker -keystore broker.ks -file broker.pem
and move broker.pem to the Python client. Now, in the test script above, include the certificate by replacing the SSL configuration with
conn.set_ssl(for_hosts=[(host, port)], key_file='/path/to/client.key', cert_file='/path/to/client.pem', ca_certs='/path/to/broker.pem')
As above, we can test that this is indeed performing the proper validation by repeating the broker certificate generation process to create a broker2.pem, use that in the test script, and note that it will fail with an
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)
Try this.
conn = stomp.Connection([(host, port)])
conn.set_listener('', MyListener())
conn.set_ssl(for_hosts=[(host, port)], ssl_version=ssl.PROTOCOL_TLS)
conn.start()
conn.connect(login, password, wait=True)
conn.send(body=message, destination=queue)
conn.disconnect()
or
conn.set_ssl(for_hosts=[(host, port)], ssl_version=_ssl.PROTOCOL_TLS)
The key point is understanding the key/cert files must be there - otherwise even stomp.py tests in their repository will not work. TLS will not work in stomp.py if the file names are not specified (correctly).
See the code & key generation instructions here: https://gist.github.com/borislitvak/6ccea503abf1b2f9c89e87309d6dab88
UPDATED: fixed/working code at bottom of question. Re-implemented to use wrap_socket() instead of SSL.Context(). Negotiated cipher seems unaffected by ssl.PROTOCOL_
I've got an XML service which listens on port 8388. The service has it's own CA built in and will issue a P12 file for any client configured on it's admin interface (I need to include a CA cert in my connections).
I've configured a client and downloaded the P12 file; also exported PEM files (for easier debugging) from the P12 and exported a PEM file for the server CA (which was a Java keystore).
If I use openssl s_client and feed it the client-side cert/key and CA file, things seem to work correctly; ie: verify return:1 in the truncated output below. The openssl process does not generate any certificate errors on the server (as does my Python script). From what I've read, this is supposed to mean the certs are all OK and valid.
# openssl s_client -connect srv.domain.net:8388 -CAfile server_ca.pem -cert client_cert.pem -key client_key.pem
CONNECTED(00000003)
depth=1 ... emailAddress = ca#SERVERsystems.com
verify return:1
depth=0 ...CN = srv.domain.net
verify return:1
---
Certificate chain
0 s:/emailAddress=help#mydomain.net...CN=srv.domain.net
i:/C=US/...emailAddress=ca#SERVERsystems.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIJRDCCByygAwIBAgIGAVGIopaoMA0GCSqGSIb3DQEBDQUAMIG...rV
-----END CERTIFICATE-----
subject=/emailAddress=help#mydomain.net...CN=srv.domain.net
issuer=/C=US/...emailAddress=ca#SERVERsystems.com
---
Acceptable client certificate CA names
/C=US/...emailAddress=ca#SERVERsystems.com
/.../CN=srv.domain.net
---
SSL handshake has read 3369 bytes and written 5705 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: ...DF6572FA00D62D83CF0B1A82F
Session-ID-ctx:
Master-Key: ...7F685C0B163A7739C271E9722FC0554108175C4
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1461337153
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
I am now attempting to hook up a Python (2.7.9) script to the XML service using the same cert, key and CA file but I can't get it working. Python is complaining about SSLv3 errors and the server says it can't verify the client. So, the connection works, but the handshake, certs or ciphers or something aren't right.
I've searched out numerous examples and implementations and this one seemed to be the simplest one so I started with it for a template. SSL3 is not the protocol I would stick with (POODLE), but it was supposed to be from a working example so I wanted to make as few changes as possible to get things working and then tweak from there. Anyone know what I've got wrong here? ouput/errors/logs are posted at the far bottom.
#!/usr/bin/python
# -*- coding: utf-8 -*-
import socket
import OpenSSL
from OpenSSL import *
import sys
serverName = sys.argv[1]
print "Using server : " + serverName
ctx = SSL.Context(SSL.SSLv3_METHOD)
ctx.use_privatekey_file('client_key.pem')
ctx.use_certificate_file('client_cert.pem')
ctx.load_verify_locations(cafile='server_ca.pem')
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((serverName, 8388))
sslSocket = socket.ssl(s)
print repr(sslSocket.server())
print repr(sslSocket.issuer())
print ("writing socket..")
sslSocket.write('<transaction><data>14</data></transaction>\n')
s.close()
Python output:
Using server : localhost
Traceback (most recent call last):
File "./test3.py", line 29, in <module>
sslSocket = socket.ssl(s)
File "/usr/lib/python2.7/socket.py", line 64, in ssl
return _realssl.sslwrap_simple(sock, keyfile, certfile)
File "/usr/lib/python2.7/ssl.py", line 993, in sslwrap_simple
ssl_sock.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:581)
Server logs after connection above:
Apr 22 10:39:56 srv.domain.net ERROR server.auth [Thread-183,run:233] Couldn't validate the client certificate. Verify the validity and dates of the client cert.
Apr 22 10:39:56 srv.domain.net javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
Apr 22 10:39:56 srv.domain.net at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:431)
Apr 22 10:39:56 srv.domain.net at com.xml.server.auth.run(auth.java:226)
Apr 22 10:39:56 srv.domain.net at java.lang.Thread.run(Thread.java:745)
Working code:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import socket
import ssl
import sys
import os
serverName = sys.argv[1]
print "Using server : " + serverName
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# ssl.PROTOCOL_xxx does not seem to affect negotiated cipher??
wrapper = ssl.wrap_socket(s,
ca_certs = "server_ca.pem",
certfile = "client_cert.pem",
keyfile = "client_key.pem",
cert_reqs=ssl.CERT_REQUIRED,
ssl_version=ssl.PROTOCOL_TLSv1)
wrapper.connect((serverName, 8388))
# some info on the connnection/cipher in use
print repr(wrapper.getpeername())
print repr(wrapper.cipher())
print repr(wrapper.getpeercert())
# send server command
print ("writing socket..")
wrapper.send ("<transaction><data>14</data></transaction>\n")
# read server reply
print "server reply: " + wrapper.recv(4096)
wrapper.close()
s.close()
ctx.use_privatekey_file('client_key.pem')
ctx.use_certificate_file('client_cert.pem')
...
sslSocket = socket.ssl(s)
While you create an SSL context with the client certificate you don't use this context within your SSL connection. This means no client certificate will be send and the server complains accordingly.
A much simpler way to use client certificates is by using the certfile and keyfile parameter of ssl.wrap_socket, see the documentation.
I'm trying to start using new messenger platform from FB.
So i have server with name (i.e.) www.mysite.com
I got a valid SSL certificate for that domain and apache is setup correctly - all good.
I have a python code on my server which runs flask and i point it to these certificates I got for the server:
from flask import Flask, request
from pymessenger.bot import Bot
import requests
import ipdb
from OpenSSL import SSL
app = Flask(__name__)
TOKEN = "<access_token>"
bot = Bot(TOKEN)
#app.route("/webhook", methods = ['GET', 'POST'])
def hello():
if request.method == 'GET':
if (request.args.get("hub.verify_token") == "my_awesome_token"):
return request.args.get("hub.challenge")
if request.method == 'POST':
output = request.json
event = output['entry'][0]['messaging']
for x in event:
if (x.get('message') and x['message'].get('text')):
message = x['message']['text']
recipient_id = x['sender']['id']
bot.send_text_message(recipient_id, message)
else:
pass
return "success"
if __name__ == "__main__":
# tried this - no luck
#context = SSL.Context(SSL.SSLv23_METHOD)
#context.use_privatekey_file('/home/mysite.com.key')
#context.use_certificate_file('/home/intermediate.crt')
# tried this - also same result
context = ('/mysite.com.crt', '/mysite.com.key')
app.run(host='www.mysite.com',port=5002, debug=True, ssl_context=context)
It starts without error and if I navigate with my browser - i see it gets connections.
port 5002 is open inf my firewall.
But when I go to FB and try to subscribe to that URL - it keeps failing with this error:
The URL couldn't be validated.
Callback verification failed with the following errors: curl_errno = 60;
curl_error = SSL certificate problem: unable to get local issuer certificate;
HTTP Status Code = 200;
HTTP Message = Connection established
I've read half the internet on the possible reasons and saw many different suggestions, but I can't make any of those work - i end up with the same result 95% of the time. 5% - some other error due to Flask bugging out with some "crazy" settings I try.
What can be wrong? I'm using certificate files which I got from COMODO.
Also tried in the call back url to use /webhook - no luck either. same error.
As mentioned in the answer above, the issue is that you do not have the issuer certificate which means that a secure connection can not be established. These certificates can be downloaded from your certificate provider, in your case Comodo. Once you have them you need to serve both to the user so a SSL connection can be established. There are multiple ways to do depending on how you are hosting but the easiest way to do it is to concat the certificates together. The order they are concated in IS important and will cause a Key Mismatch error if done in the incorrect order. Your certificate should be first, followed by the issuer certificate and the root certificate, like this:
-----BEGIN CERTIFICATE-----
(Your Primary SSL certificate: your_domain_name.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Your Intermediate certificate: Intermediate.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Your Root certificate: TrustedRoot.crt)
-----END CERTIFICATE-----
This can easily be done from the CLI like this:
cat your_domain_name.crt Intermediate.crt TrustedRoot.crt > combined.crt
This command combines the 3 files in the proper order and stores the output in a new file called combined.crt.
See here for more details.
Add the issuer certificate also. Comodo will issue their own certificate. YOu need to include that while starting the server.
I use certbot and Let's Encrypt.
Follow Installing Client software.
Then run command => sudo ./certbot-auto --apache -d YOUR_DOMAIN_NAME.COM (i tried apache, nginx and flask alone, all works, no need to put https at front)
cd /etc/letsencrypt/live/YOUR_DOMAIN_NAME.COM/
context = ('/etc/letsencrypt/live/YOUR_DOMAIN_NAME.COM/fullchain.pem', '/etc/letsencrypt/live/YOUR_DOMAIN_NAME.COM/privkey.pem')
i used cert.pem instead of fullchain.pem at first and got the above error, succeed after changed cert.pem to fullchain.pem
I'm trying to use imaplib.IMAP4_SSL function but I'm stuck with this error:
[Errno 336265225] _ssl.c:356: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
try:
mail = imaplib.IMAP4_SSL(host='imap.gmail.com', port="993", certfile="certIMAP.pem")
except Exception as e:
print e
sys.exit()
The certificate has been downloaded with:
echo | openssl s_client -connect imap.gmail.com:993 2>/dev/null | openssl x509 > certIMAP.pem
Permission on the file are ok.
If I use the keyfile option, they ask me the cert, and then the same error is obtained.
I could not find example for this function with cert specification.
Thank you.
While it is not obvious from the documentation I would suggest that the parameters certfile and keyfile are not used to verify the servers certificate (that's how you use it - and keyfile would not make any sense here) but are the optional certificate for the client and it's private key, which some server might require to identify the client (e.g. instead or additionally to password).
To verify the server you usually have something called CA file or CA path (CA = certificate agency) and you don't need a private key here. There seem to be no obvious way for given the CA certificates here, but Certificate Authority for imaplib and poplib python might help you with this.