Develop Reference

What is the correct certificate purpose for SSL client in python?

I'm setting up SSL client verification in my python app. At the moment my proof-of-concept code is falling over just establishing a secure connection.
It looks like the certificates I've generated either have a certificate usage without the necessary permissions (more likely IMO) or they have permissions that the server cannot understand or accept (a little less likely IMO).
This should be relatively trivial, but I can't find the right documentation.
I've generated the server and client certificate through OpenSSL. I've done this in the past for other apps without any problem. But I'm much less familiar with creating client certificates. OpenSSL reports that the client certificate I'm using has extensions:
X509v3 extensions:
X509v3 Subject Key Identifier:
AF:AB:9D:AA:88:96:F4:0C:F5:56:9A:2C:DB:B6:BA:D9:DD:11:69:45
X509v3 Subject Alternative Name:
email:a#example.com
X509v3 Basic Constraints:
CA:FALSE
Netscape Cert Type:
SSL Client
X509v3 Authority Key Identifier:
keyid:E1:35:7C:39:7F:39:A4:43:D2:F8:00:59:38:91:71:AF:B9:38:AD:3F
X509v3 Key Usage:
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication
The trivial server test code is:
import ssl
import socket
import logging
_log = logging.getLogger(__name__)
def main():
context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
context.load_cert_chain("1B.pem", "key2.pem")
context.verify_mode = ssl.CERT_REQUIRED
context.load_verify_locations("my_ca.crt")
raw_server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
try:
# domain replaced for SO question
raw_server_socket.bind(('neptune.example.com', 8812))
raw_server_socket.listen(5)
server_socket = context.wrap_socket(raw_server_socket, server_side=True)
except Exception:
raw_server_socket.close()
raise
with server_socket:
while True:
try:
connection_to_client, address = server_socket.accept()
with connection_to_client:
connection_to_client.write(b'Hello')
except Exception as ex:
print(ex)
if __name__ == "__main__":
main()
This gives the error:
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unsupported certificate purpose (_ssl.c:1076)
... When the client connected with this:
import socket
import ssl
context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
context.load_cert_chain("1C.pem", "key.pem")
raw_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Domain changed for SO question
conn = context.wrap_socket(raw_socket, server_side=False, server_hostname="neptune.example.com")
conn.connect(("neptune.example.com", 8812))
conn.close()

As hinted at by Steffen Ullrich, it looks like the problem was not with the certificate itself but the CA certificate I used to sign it.
CA certificates are limited in the rights they can authorize. These are represented in the same extensions as the certificates they sign. So for a CA to sign a certificate as a client SSL certificate both the client certificate and the CA certificate must have:
Key usage with "digital signature".
the 1.3.6.1.5.5.7.3.2 extension AKA TLS Web Client Authentication
That is, openssl should report the following for both the CA certificate and the signed client cetificate:
X509v3 Key Usage:
Digital Signature
X509v3 Extended Key Usage:
TLS Web Client Authentication
In my case, the CA didn't have the extended use 1.3.6.1.5.5.7.3.2. It was fine for server certificates but couldn't sign client certificates.

Send data via ssl from docker container with python paho.mqqt

I have a docker python script with a paho.mqqt client.
import json
import ssl
import paho.mqtt.client as mqtt
# connection parameters
broker = "111.11.111.111"
port = 5000
topic = "v1/devices/me/telemetry"
device_token = "111"
# message creation
msg = dict()
msg["greeting"] = "Hello World"
msg_out = json.dumps(msg)
# create mqtt client
client = mqtt.Client()
# access token
client.username_pw_set(device_token)
# one-way-SSL
client.tls_set(ca_certs="../settings/test-server.pub.pem", certfile=None, keyfile=None, cert_reqs=ssl.CERT_REQUIRED,
tls_version=ssl.PROTOCOL_TLSv1, ciphers=None)
# two-way-SSL
# client.tls_set(ca_certs="tb-test-server.pub.pem",certfile="mqttclient.nopass.pem",keyfile=None,cert_reqs=ssl.CERT_REQUIRED,tls_version=ssl.PROTOCOL_TLSv1,ciphers=None)
# connect, send message and disconnect
client.connect(broker, port, 60)
client.publish(topic, msg_out, 1)
client.disconnect()
When I start the script on my host machine everything works fine. But if I start the script inside a docker container I get the error:
SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '111.11.111.111'.
The The client certificate "test-server.pub.pem":
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 737991734 (0x2bfcdc36)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=DE, ST=B, L=B, O=IAV, OU=Digital-Lab, CN=194.31.198.168
Validity
Not Before: Aug 29 09:53:53 2018 GMT
Not After : Jan 13 09:53:53 2046 GMT
Subject: C=DE, ST=B, L=B, O=IAV, OU=Digital-Lab, CN=194.31.198.168
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:e0:ea:37:1e:19:69:b8:f1:7a:e0:28:af:d6:ff:
57:12:6c:ff:ac:2c:11:68:a4:38:3c:f2:89:bc:64:
2f:78:c4:5c:b0:14:55:d2:c4:8e:84:55:c0:58:80:
75:b9:fb:02:42:6f:8a:dd:47:2f:80:5a:b2:35:be:
cc:c7:4b:15:ed:35:f9:10:36:3b:2e:68:28:89:2c:
04:6e:ac:10:6d:b4:5a:80:a0:5b:da:53:14:3b:ff:
04:a8:bc:45:48:9e:11:b7:b6:62:94:ad:67:8e:82:
2e:42:b5:03:6c:30:eb:1d:72:d3:05:83:30:ae:ce:
e0:8b:98:13:04:5c:49:fe:73:76:ee:7e:fa:33:49:
32:d8:51:9b:15:17:cb:46:1c:2c:a8:00:d0:4b:06:
df:4d:16:9f:dc:83:3b:1b:bd:7f:86:35:68:b6:f1:
12:82:d7:50:a1:9d:d9:db:8b:60:c0:ed:68:85:31:
51:57:a5:13:62:ec:bb:22:a1:a0:4f:c2:45:31:de:
8d:ad:e1:3e:81:fa:62:0a:04:e8:94:ac:eb:80:af:
dc:c8:00:67:94:25:c5:a6:81:a4:82:bc:da:cf:f5:
ad:5b:36:6d:62:70:73:d0:30:84:04:60:dd:25:10:
92:65:aa:29:3a:6a:e1:1d:40:6c:45:c3:5f:77:ad:
31:a5
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
F7:F6:DC:83:8F:9E:E1:2F:68:B5:4A:95:5C:E0:9B:03:B2:0B:A6:3C
Signature Algorithm: sha256WithRSAEncryption
dc:e1:1e:84:03:ee:8e:17:63:9f:73:0a:0d:ac:55:26:61:c5:
62:75:32:00:69:ba:96:21:fc:c3:3c:d8:23:31:a4:6c:3e:63:
57:50:38:55:7a:52:ef:3f:7c:97:94:9c:d3:5b:29:41:c4:d4:
5c:2e:49:b4:7c:c3:f1:69:57:87:fd:57:b5:52:13:62:a4:d4:
88:78:da:b6:f8:d1:4d:6f:4f:87:68:75:8e:20:6b:21:db:8e:
21:f2:c7:23:f0:02:d4:bc:65:ea:75:ec:7f:a7:3a:2a:d1:02:
8e:5b:26:aa:fc:7e:3c:3e:79:95:36:72:48:e1:36:27:09:42:
f4:05:ce:e0:56:93:ac:c1:5b:ce:64:23:25:9d:d1:c7:82:08:
cb:a7:99:9e:e6:88:ab:71:f0:3d:54:37:5b:a2:fa:41:d8:9b:
af:37:85:a8:9c:9c:0a:9b:87:f5:b2:49:51:bb:86:9a:af:ce:
e1:52:83:00:25:50:02:d5:c6:4a:e0:20:e7:33:1f:3f:5a:5c:
8d:ba:11:a8:02:94:17:41:0d:e0:98:11:5f:93:52:7c:bb:2c:
d8:0b:61:bf:ea:bd:f7:b0:b1:c0:99:68:cb:47:4b:79:01:81:
36:5c:dc:43:92:78:58:40:c1:e4:56:46:20:26:07:19:b4:b4:
d8:5a:16:00
-----BEGIN CERTIFICATE-----
MIIDYzCCAkugAwIBAgIEK/zcNjANBgkqhkiG9w0BAQsFADBiMQswCQYDVQQGEwJE
RTEKMAgGA1UECBMBQjEKMAgGA1UEBxMBQjEMMAoGA1UEChMDSUFWMRQwEgYDVQQL
EwtEaWdpdGFsLUxhYjEXMBUGA1UEAxMOMTk0LjMxLjE5OC4xNjgwHhcNMTgwODI5
MDk1MzUzWhcNNDYwMTEzMDk1MzUzWjBiMQswCQYDVQQGEwJERTEKMAgGA1UECBMB
QjEKMAgGA1UEBxMBQjEMMAoGA1UEChMDSUFWMRQwEgYDVQQLEwtEaWdpdGFsLUxh
YjEXMBUGA1UEAxMOMTk0LjMxLjE5OC4xNjgwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQDg6jceGWm48XrgKK/W/1cSbP+sLBFopDg88om8ZC94xFywFFXS
xI6EVcBYgHW5+wJCb4rdRy+AWrI1vszHSxXtNfkQNjsuaCiJLARurBBttFqAoFva
UxQ7/wSovEVInhG3tmKUrWeOgi5CtQNsMOsdctMFgzCuzuCLmBMEXEn+c3bufvoz
STLYUZsVF8tGHCyoANBLBt9NFp/cgzsbvX+GNWi28RKC11Chndnbi2DA7WiFMVFX
pRNi7LsioaBPwkUx3o2t4T6B+mIKBOiUrOuAr9zIAGeUJcWmgaSCvNrP9a1bNm1i
cHPQMIQEYN0lEJJlqik6auEdQGxFw193rTGlAgMBAAGjITAfMB0GA1UdDgQWBBT3
9tyDj57hL2i1SpVc4JsDsgumPDANBgkqhkiG9w0BAQsFAAOCAQEA3OEehAPujhdj
n3MKDaxVJmHFYnUyAGm6liH8wzzYIzGkbD5jV1A4VXpS7z98l5Sc01spQcTUXC5J
tHzD8WlXh/1XtVITYqTUiHjatvjRTW9Ph2h1jiBrIduOIfLHI/AC1Lxl6nXsf6c6
KtECjlsmqvx+PD55lTZySOE2JwlC9AXO4FaTrMFbzmQjJZ3Rx4IIy6eZnuaIq3Hw
PVQ3W6L6QdibrzeFqJycCpuH9bJJUbuGmq/O4VKDACVQAtXGSuAg5zMfP1pcjboR
qAKUF0EN4JgRX5NSfLss2Athv+q997CxwJloy0dLeQGBNlzcQ5J4WEDB5FZGICYH
GbS02FoWAA==
-----END CERTIFICATE-----
I use Thingsboard as MQTT Broaker. thingsboard-config. I created the certificates as described there.
As described, the script works on the host machine of the Docker container, but not inside the Docker container.

IP address mismatch
You have a certificate with an IP address in the CN field (which I will not repeat from your question as you badly obfuscated it), and you are connecting to another IP address. The TLS library hence rejects the connection because of the mismatch.
You will probably need to reissue a certificate with the proper IP address.
Since it is a self signed certificate, you should have no problem creating a new one.
However it is not a good idea to use IP addresses like that. You should use hostnames instead, so both in the certificate and in your client code when connecting. Of course you need to make sure in the same way that you have no mismatch.

Login via starttls method from smtplib to old e-mail server

I use: Python 2.7.15, OpenSSL 1.1.0h (27 Mar 2018), MS Exchange 2007.
My MS exchange allows to send login/pass only after STARTTLS.
In python I try to connect to server like:
from stmplib import SMTP
conn = SMTP(server,port)
conn.set_debuglevel(1)
conn.starttls()
conn.login(username, password)
conn.quit()
And finally I get error in starttls:
/python2.7/ssl.py", line 847, in do_handshake
self._sslobj.do_handshake()
The problem is follow python try to establish connection with TLS v1.2 but Exchange only support TLS v.1.0. I tried ports 25 and 587.
When I tried to connect and login to server by console openssl application it is work fine for both ports with TLS v.1.0:
openssl s_client -connect sever:587 -starttls smtp -no_tls1_2 -no_tls1_1 -crlf
Server answer:
SSL handshake has read 1481 bytes and written 530 bytes
--- New, TLSv1/SSLv3, Cipher is RC4-MD5 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session:
Protocol : TLSv1
Cipher : RC4-MD5
Session-ID: xxxx
Session-ID-ctx:
Master-Key: xxxx
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1533874470
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
--- 250 CHUNKING ehlo 250-xxxxxxx Hello [xxxx] 250-SIZE 26214400 250-PIPELINING 250-DSN 250-ENHANCEDSTATUSCODES 250-AUTH GSSAPI NTLM LOGIN 250-8BITMIME 250-BINARYMIME 250 CHUNKING ^C
I try to inherits standard SMTP class to overload starttls method to use context option like:
# show only changes to standard `starttls` method
def starttls(self, keyfile=None, certfile=None, context=None):
...
if context is None:
context = ssl._create_stdlib_context(certfile=certfile, keyfile=keyfile)
self.sock = context.wrap_socket(self.sock, server_hostname=self._hostname)
...
And use this class in my script:
context = ssl.SSLContext(ssl.PROTOCOL_TLS)
context.options |= ssl.OP_NO_TLSv1_2 | ssl.OP_NO_TLSv1_1
conn = mySMTP(server,port)
conn.set_debuglevel(1)
conn.starttls(context = context)
conn.login(username, password)
conn.quit()
But the error still the same.
What I do wrong? May be options of context have to be other or may be I miss something?
How to set smtplib and starttls method to force use only TLS v.1.0 in this situation?

Something that's been successful for me in the past when connecting to old email-serivces using deprecated SSL-versions is setting up a local tunnel with stunnel or similar and connecting on localhost through that, you can also set up a local sendmail instance as a proxy.

ssl.get_server_certificate for sites with SNI (Server Name Indication)

I am trying to get the server certificate of badssl.com subdomains (ex. https://expired.badssl.com).
import ssl
ssl.get_server_certificate(('expired.badssl.com', 443))
But when examining the above generated certificate I see that the certificate has
Identity: badssl-fallback-unknown-subdomain-or-no-sni
which means SNI is failing. How can I get the server certificate of different subdomains of badssl.com? (I am using python 2.7.12)

Found the answer.
import ssl
hostname = "expired.badssl.com"
port = 443
conn = ssl.create_connection((hostname, port))
context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
sock = context.wrap_socket(conn, server_hostname=hostname)
certificate = ssl.DER_cert_to_PEM_cert(sock.getpeercert(True))

Searching for "Python ssl.get_server_certificate SNI" brought me easily to this answer. Although the OP himself answer is correct, I would like to provide a little more insight for future reference.
With some [hostname]s the fallowing call using Python 3.7:
ssl.get_server_certificate(("example.com", 443)
will complain with a traceback that ends with:
ssl.SSLError: [SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:1045)
Doing some further investigation, making use of the openssl s_client utility, allows to discover that those same [hostname]s which made get_server_certificate to fail, also makes the fallowing command:
openssl s_client -showcerts -connect example.com:443
to fail with this error:
SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:s23_clnt.c:802
Note that the error message is similar to the one returned by the python code.
Using the -servername switch did the trick:
openssl s_client -showcerts -connect example.com:443 -servername example.com
leading to the conclusion that the investigated hostname refers to a secure server that makes use of SNI (a good explanation on what that means is given by the SNI Wikipedia article).
So, switching again to Python and looking at the get_server_certificate method, examining the ssl module source (here for convenience), you can discover that the function includes this call:
context.wrap_socket(sock)
without the server_hostname=hostname key argument, which of course should mean that get_server_certificate cannot be used querying a SNI server. A little more effort is required:
hostname = "example.com"
port = 443
context = ssl.create_default_context()
with socket.create_connection((hostname, port)) as sock:
with context.wrap_socket(sock, server_hostname=hostname) as sslsock:
der_cert = sslsock.getpeercert(True)
# from binary DER format to PEM
pem_cert = ssl.DER_cert_to_PEM_cert(der_cert)
print(pem_cert)

Python OpenSSL giving handshake failure

I'm trying to update a new valid SSL cert provided by DigiCert on my python flask. The same code with the old expired cert works just fine. However for the renewed cert, it did not recognise my certificate chain and giving error:
[root#test~]# openssl s_client -connect somedomain.com:9443
CONNECTED(00000003)
140340901406536:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:744:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 249 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---
The weird thing is, when i'm using openssl command to verify my cert chain, everything is working well as follow:
[root#test ~]$ openssl verify -verbose -CAfile CertChain.crt Server.crt
Server.crt: OK
This is my python code
from OpenSSL import SSL
from flask import Flask, send_file, make_response, request, jsonify, Response
from flask.ext import restful
from flask.ext.restful import reqparse, abort, Api, Resource
#------------#
context = SSL.Context(SSL.SSLv23_METHOD)
context.use_certificate_file("Server.crt")
context.use_privatekey_file("Server.key")
context.use_certificate_chain_file("CertChain.crt")
context.set_options(SSL.OP_NO_SSLv3)
context.set_options(SSL.OP_NO_SSLv2)
context.set_cipher_list('ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS')
app = Flask(__name__)
#app.route("/")
def hello():
return "Hello World!"
if __name__ == "__main__":
app.run(host='0.0.0.0', port=9443, debug=True, use_reloader=False, ssl_context=context)
And for your info key and certificate is matching with no problem
[root#test]# openssl x509 -noout -modulus -in Server.crt | openssl md5
(stdin)= adc3de807ec6a02c5ba9da89f3fe5dd5
[root#test]# openssl rsa -noout -modulus -in Server.key | openssl md5
(stdin)= adc3de807ec6a02c5ba9da89f3fe5dd5
Is there anything affecting this? Python version? OpenSSL version? My cipher? Any body can help? Thanks

SSL handshake has read 7 bytes and written 249 bytes
These are not problems of the validation of the certificate.
The clients starts the SSL handshake but the server sends only 7 bytes back, which might be an SSL alert that something is wrong. The server never sends the certificate back so it cannot be a problem of the client side validation.
This means you should better check the server side for problems and not the client side. And since you say that exactly the same code works with the old certificate it can not be a problem of protocol version or ciphers but must relate to the certificate. My guess is thus that the private key you use does not match the certificate you use, which gets not checked when you do the openssl verify.

I got the solution. Previously my certificate chain only contains the Intermediate and Root certificate which is not complete.
The solution is to also include the server certificate into the chain which is weird since context.use_certificate_file is already defined.
Anyways, the handshake is now working fine. Thanks!