I am getting the following error when trying to read a certificate:
OpenSSL.crypto.Error: [('PEM routines', 'PEM_read_bio', 'no start line')]
when running OpenSSL.crypto.load_certificate(FILETYPE_PEM, filename). I have made some research but was not able to find an answer specific to my case.
I tried checking if the file existed with os.path.isfile(filename) which returns True, but loading the certificate ONLY raises the error above.
Also, when executing on the terminal openssl X509 -in file.pem, it works like a charm.
The file.pem looks like this:
-----BEGIN CERTIFICATE-----
<<sensitive data>>
-----END CERTIFICATE-----
It seems to be valid since I able to perform basic openssl operations on the terminal. I am running CentOS 7, if that helps.
Any ideas?
Thanks!
According to http://www.pyopenssl.org/en/stable/api/crypto.html#OpenSSL.crypto.load_certificate,load_certificate() takes a buffer (string will do) containing the certificate, not a filename.
Your need to do:
with open(filename, "r") as my_cert_file:
my_cert_text = my_cert_file.read()
cert = load_certificate(FILETYPE_PEM, my_cert_text)
Related
A python deployment our team has fails on
import socket
import ssl
context = ssl.wrap_socket(
sock,
ca_certs='root_ca_bundle.pem',
keyfile='pk.pem',
certfile='cert.pem',
do_handshake_on_connect=True,
cert_reqs=ssl.CERT_REQUIRED,
)
which is traced to failing on load_cert chain function with
if certfile:
context.load_cert_chain(certfile, keyfile)
The error I see is this
[X509: KEY_VALUES_MISMATCH] key values mismatch (_ssl.c:4067)
The real issue is that I copied a wrap_socket call using the same certfile,keyfile and ca_certs, and I don't see this issue.
It only appears on deployment.
To my understanding, load_cert_chain just verifies that the certfile and keyfile are mathematically aligned, using something like this:
$ openssl x509 -noout -modulus -in cert.pem
$ openssl rsa -noout -modulus -in key-no-pass.pem
Tried this on key and cert and verified they're the same.
What I am asking is :
Is there an explanation other than wrong keyfile or certfile being used in deployment?
Is there a way I can find source code for load_cert_chain function.
This code
import requests
requests.get("https://hcaidcs.phe.org.uk/WebPages/GeneralHomePage.aspx")
is giving me this error
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
I know practically nothing about SSL, but I've tried downloading the site's certificate and pointing to that file using the verify option, but it hasn't worked. Am I missing something?
As already pointed out in a comment: the site has a bad SSL implementation as can be seen from the SSLLabs report. The main part of this report regarding your problem is:
This server's certificate chain is incomplete. Grade capped to B.
This means that the server is not sending the full certificate chain as is needed to verify the certificate. This means you need to add the missing certificates yourself when validating. For this you need to include the PEM for the missing chain certificate C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert SHA2 High Assurance Server CA and also for the root CA C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert High Assurance EV Root CA info a file my_trust_store.pem and then you can call:
requests.get("https://...", verify='my_trust_store.pem')
... but I've tried downloading the site's certificate and pointing to that file using the verify option
This will not work with normal leaf certificates. Since the SSL stack of Python is based on OpenSSL and OpenSSL expects only trusted certificate authorities in the trust store (i.e. given with verify) and a server certificate is not CA certificate it will not help to add it to the trust store.
cat institution-certificate.pem >> venv/lib/python3.9/site-packages/certifi/cacert.pem
This should solve the problem if your network requires a CA
using the certifi doesn't seem to be implied, so i'll show you what made my solution:
import urllib, urllib2, ssl
import certifi
request = urllib2.Request(url=url)
kw = dict()
if url.startswith('https://'):
certifi_context = ssl.create_default_context(cafile=certifi.where())
kw.update(context=certifi_context)
urllib2.urlopen(request, **kw)
i found this solution and more on RealPython, here
If you can avoid the certificate verification (not secure), set PYTHONHTTPSVERIFY environment variable to 0:
export PYTHONHTTPSVERIFY=0
This will skip the certificate verification.
import requests
html = requests.get("https://hcaidcs.phe.org.uk/WebPages/GeneralHomePage.aspx",verify=False).text
You should write it like this, and I've verified it
I have a very simple flask HTTPS server running using a self-signed certificate on a google compute instance. Here is the code for the server:
from flask import Flask
app = Flask(__name__)
with open("test_certificate.pem",'r') as inputFile:
print(inputFile.read())
if __name__=="__main__":
app.run(ssl_context=("test_certificate.pem", "test_key.pem"),
debug=True, host="0.0.0.0", port=443)
When I start the server I receive the following output:
-----BEGIN CERTIFICATE-----
MIID7DCCAtSgAwIBAgIJANdrMgD+3bUnMA0GCSqGSIb3DQEBCwUAMIGKMQswCQYD
VQQGEwJVUzELMAkGA1UECAwCUEExFTATBgNVBAcMDFBoaWxhZGVscGhpYTEKMAgG
A1UECgwBQTERMA8GA1UECwwIYXNkZmF3ZWYxFzAVBgNVBAMMDjM1LjIzMS4xMTAu
MTQ5MR8wHQYJKoZIhvcNAQkBFhBhc2RmYXNAZ21haWwuY29tMB4XDTE5MDMyMjIw
NTcyM1oXDTIwMDMyMTIwNTcyM1owgYoxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJQ
QTEVMBMGA1UEBwwMUGhpbGFkZWxwaGlhMQowCAYDVQQKDAFBMREwDwYDVQQLDAhh
c2RmYXdlZjEXMBUGA1UEAwwOMzUuMjMxLjExMC4xNDkxHzAdBgkqhkiG9w0BCQEW
EGFzZGZhc0BnbWFpbC5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQCl16hzFOqYQozHh4G46oY32HP59LgfbT7hGyAcYPC3jt42jiKa9Yfpa8JaaZrm
xWagPNZG73xqyejMoaLKEPbZc650e+mwNuVoer2ic2CZIKp5iVy0si85ecxAkFYR
3RF+eDCqL8kT5gLbfj8knq3PQtVp2AmNlnN8c3cV6It5XJMUoPyfGdCGzphzVha+
XBpWJ25pWmezB38lTZpBdeZx2KDTDNpbtxsmtxy52IHQ/HXCnlW8bSbvhFvTn0IH
q3vOJnKG7GNOiOtyi+i6bSg0QxbQAhN5xVVB3Qq6i5yaIFTz/1Wedr4FpvbJqShZ
wf5lt4PxSaO6hDXbnWOgiiW3AgMBAAGjUzBRMB0GA1UdDgQWBBSs3T312Qv3r7QN
bs/duHyQrvO9RTAfBgNVHSMEGDAWgBSs3T312Qv3r7QNbs/duHyQrvO9RTAPBgNV
HRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBLcpXBcafyR2SRa6jDDnwt
rDxr4aKnC6C99JPmrpBS6aD5+LrCUn6eY7B0MS5/+EoBEWxMtRoCW6MpBKdaOgPw
1etjfoYICPTvkvnIJRn3re1ziV54wSaXWlATTCoJ9Cjg8daHk7RS5JFr9T13iRwR
dzbAx6/dAFur6mmVcH5yD+qSJWSIBq/EpOjy4xWt1Pqu7oClzGLCStsR4sAIg0+T
gQIwxy9pMukxZOCzER9Ega+Wnac/CpA9pNUs+ayMebjo9ZVnq95vBzy87UMptvz9
W00TV+NN/zbnV24gmnOTWla4x8ylxxhq+7bYImZjnwinMe3FCIcZA350dcuOZI8n
-----END CERTIFICATE-----
I was having issues sending HTTPS requests (I was receiving SSL "bad handshake" errors), so I executed the following code to see what the certificate was that was being returned:
import ssl
serverSSL = ssl.get_server_certificate(('20.42.490.329',443))
print(serverSSL)
with open(VERIFY_PATH, 'w') as outputFile:
outputFile.write(serverSSL)
This code outputs the following:
-----BEGIN CERTIFICATE-----
MIIDmDCCAoCgAwIBAgIJT8O3F1rTZQgnMA0GCSqGSIb3DQEBCwUAMFgxCzAJBgNV
BAYTAlVTMQswCQYDVQQIEwJQQTEVMBMGA1UEBxMMUEhJTEFERUxQSElBMQ0wCwYD
VQQKEwRVUEhTMRYwFAYDVQQDFA1VUEhTX0ZJUkVXQUxMMB4XDTE5MDMyMjIwNTcy
M1oXDTIwMDMyMTIwNTcyM1owgYoxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJQQTEV
MBMGA1UEBwwMUGhpbGFkZWxwaGlhMQowCAYDVQQKDAFBMREwDwYDVQQLDAhhc2Rm
YXdlZjEXMBUGA1UEAwwOMzUuMjMxLjExMC4xNDkxHzAdBgkqhkiG9w0BCQEWEGFz
ZGZhc0BnbWFpbC5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC3
Bd7d0ty9wbw8G+ongIs6Ezy0njO0JLulMNV/SVnBg+V0yHfCy0emA4JXyHTw+LfX
fQcQeiv0znXUFNnl+i2T0vQuPb2U99wbj+c/MEsiP+H9NRDWzgYgQTat/nGZv31l
grH4+IJSYVuQ/iKquSseKrQRJzDQes1BbuA8hNoD2iIuV2hIPzXMUOEbBExXoAF8
U8M0UMWUrvsCYrzf/qJlg+jciUqhXiRc3SomIxrs6P7J/LJHl+xWFsUflWgzoxDn
NCYaf3nCGi55cqWbstJytvtDoDqS3+HoYeCLz3ab/HCGMiTPJOtLi/nLMhkCZy2U
bowedqlFX6C7wIG/rczTAgMBAAGjMjAwMB0GA1UdDgQWBBSs3T312Qv3r7QNbs/d
uHyQrvO9RTAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBtgJZ3
dQMV36kXDKKEBEV04RTFD6XrKhEx0/TRyiCzuCG05GtF0efbA0pz18uDMNYla70i
R4n/KWTRjbJ8aVvQwYvT7vCgTzHtM0lPwIiZYzssvlaauBEiZCBYAEV1hFEK53A2
eqUmDGul7x9w4IO0800pC8vkdxL6UGa6i/cEp4xXZca+A4Ukhoq+I8sENPTiBIy1
FT+wWgz8d7V65wM+r9pe4OvUubLJ4qHwKBGRGCOAh5SeluKIL1L8sSul3MKOr+Cj
t5mHuGsjXTyIjB3nYkW94tBkFiRvEYRKtvoDOo5wm8v4g/GGSZtDlNLtq7523J3n
maGrCR2AQw4PA5i3
-----END CERTIFICATE-----
Why do these certificates not match? I am assuming this is why I am getting the bad handshake error. Any suggestions are much appreciated
Ok, so you are using invalid certificates, and the reason why the file changes every time is that you are corrupting it.
This is what I use to get a TRUSTED certificate that most browsers can use: https://certbot.eff.org/instructions. Comment and tell me if this works for you.
Tho, if you really want to test this, just use
if __name__ == "__main__":
app.run(ssl_context='adhoc')
I have a simple Python script runs pretty well on Python 2.7 on Windows 10.
however it gives me
"SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:726)"
error when I run it from Android.
the code is quite simple as below：
import urllib
import urllib2
import ssl
import traceback
#ssl._https_verify_certificates(False)
#ssl._create_default_https_context = ssl._create_unverified_context
try:
url="https://stackoverflow.com/"
https_context = ssl._create_unverified_context(cert_reqs=ssl.CERT_NONE,check_hostname=False)
https_handler = urllib2.HTTPSHandler(context=https_context)
opener = urllib2.build_opener(https_handler)
rep=opener.open(url)
print(rep.read())
except Exception, e:
urllib.urlopen("http://c.seechentech.com/exception?code=484&error=%s&u=%s" % (traceback.format_exc(), url))
After some investigation, I found this exception is raised during socket handshake, but I have already set certificate verification to False (There's no needs to verify certificate for me), why it checks signature? How to fix this problem?
this issue has been resolved and I'd like to share my findings.
Finding 1:
Regardless set to verify certificate or not, OpenSSL socket, which called by Python Socket, will always do SSL handshake where PEM certificate is required. This finding answers why my code fails during socket handshake.
Finding 2:
Python is able to load system trusted CA from windows 10 default location, however, it fails on Android. It might be caused by two potential reasons:
1) Python could not find default CA location on Andorid which is /system/etc/security/cacerts/
2) Python expects CA to be PEM format, but Android trusted CA has additional lines after ---END CERTIFICAT--- which is a sign of end of CA content.
I did not verify which reason actually caused the failure of loading CA certificate. What I did to give Python proper CA data is combined all CA files I extracted from one Android device into a single file, and set to ssl.SSLContext with load_verify_locations method.
Finding 3:
There're some trusted CAs missing on Android comparing to trusted CAs on Windows, so I have to manually add them into my combined CA file.
Finding 4:
Python document says "If you want maximum compatibility between clients and servers, it is recommended to use PROTOCOL_SSLv23 as the protocol version" which is not true. It causes invalid certification error when I use this protocol version. ssl.PROTOCOL_TLSv1 protocol version works fine in my case.
Finding 5:
Python document says to give a certificate as an "ASCII PEM string" which is not right. We have to give a unicode string.
Here is the code for reference:
#!/usr/bin/python
# -*- coding: UTF-8 -*-
import sys
reload(sys)
sys.setdefaultencoding("utf-8")
import os
from os import path
import urllib
import urllib2
import ssl
import traceback
import shutil
def loadCA():
concatedCA = u'''
-----BEGIN CERTIFICATE-----
MIIEIDCCAwigAwIBAgIQNE7VVyDV7exJ9C/ON9srbTANBgkqhkiG9w0BAQUFADCBqTELMAkGA1UEBhMCVVMxFTATBgNVBAoTDHRoYXd0ZSwgSW5jLjEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjE4MDYGA1UECxMvKGMpIDIwMDYgdGhhd3RlLCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxHzAdBgNVBAMTFnRoYXd0ZSBQcmltYXJ5IFJvb3QgQ0EwHhcNMDYxMTE3MDAwMDAwWhcNMzYwNzE2MjM1OTU5WjCBqTELMAkGA1UEBhMCVVMxFTATBgNVBAoTDHRoYXd0ZSwgSW5jLjEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjE4MDYGA1UECxMvKGMpIDIwMDYgdGhhd3RlLCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxHzAdBgNVBAMTFnRoYXd0ZSBQcmltYXJ5IFJvb3QgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCsoPD7gFnUnMekz52hWXMJEEUMDSxuaPFsW0hoSVk3/AszGcJ3f8wQLZU0HObrTQmnHNK4yZc2AreJ1CRfBsDMRJSUjQJib+ta3RGNKJpchJAQeg29dGYvajig4tVUROsdB58Hum/u6f1OCyn1PoSgAfGcq/gcfomk6KHYcWUNo1F77rzSImANuVud37r8UVsLr5iy6S7pBOhih94ryNdOwUxkHt3Ph1i6Sk/KaAcdHJ1KxtUvkcx8cXIcxcBn6zL9yZJclNqFwJu/U30rCfSMnZEfl2pSy94JNqR32HuHUETVPm4pafs5SSYeCaWAe0At6+gnhcn+Yf1+5nyXHdWdAgMBAAGjQjBAMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBR7W0XPr87Lev0xkhpqtvNG61dIUDANBgkqhkiG9w0BAQUFAAOCAQEAeRHAS7ORtvzw6WfUDW5FvlXok9LOAz/t2iWwHVfLHjp2oEzsUHboZHIMpKnxuIvW1oeEuzLlQRHAd9mzYJ3rG9XRbkREqaYB7FViHXe4XI5ISXycO1cRrK1zN44veFyQaEfZYGDm/Ac9IiAXxPcW6cTYcvnIc3zfFi8VqT79aie2oetaupgf1eNNZAqdE8hhuvU5HIe6uL17In/2/qxAeeWsEG89jxt5dovEN7MhGITlNgDrYyCZuen+MwS7QcjBAvlEYyCegc5C09Y/LHbTY5xZ3Y+m4Q6gLkH3LpVHz7z9M/P2C2F+fpErgUfCJzDupxBdN49cOSvkBPB7jVaMaA==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIGCDCCA/CgAwIBAgIQKy5u6tl1NmwUim7bo3yMBzANBgkqhkiG9w0BAQwFADCBhTELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxKzApBgNVBAMTIkNPTU9ETyBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTQwMjEyMDAwMDAwWhcNMjkwMjExMjM1OTU5WjCBkDELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxNjA0BgNVBAMTLUNPTU9ETyBSU0EgRG9tYWluIFZhbGlkYXRpb24gU2VjdXJlIFNlcnZlciBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAI7CAhnhoFmk6zg1jSz9AdDTScBkxwtiBUUWOqigwAwCfx3M28ShbXcDow+G+eMGnD4LgYqbSRutA776S9uMIO3Vzl5ljj4Nr0zCsLdFXlIvNN5IJGS0Qa4Al/e+Z96e0HqnU4A7fK31llVvl0cKfIWLIpeNs4TgllfQcBhglo/uLQeTnaG6ytHNe+nEKpooIZFNb5JPJaXyejXdJtxGpdCsWTWM/06RQ1A/WZMebFEh7lgUq/51UHg+TLAchhP6a5i84DuUHoVS3AOTJBhuyydRReZw3iVDpA3hSqXttn7IzW3uLh0nc13cRTCAquOyQQuvvUSH2rnlG51/ruWFgqUCAwEAAaOCAWUwggFhMB8GA1UdIwQYMBaAFLuvfgI9+qbxPISOre44mOzZMjLUMB0GA1UdDgQWBBSQr2o6lFoL2JDqElZz30O0Oija5zAOBgNVHQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwGwYDVR0gBBQwEjAGBgRVHSAAMAgGBmeBDAECATBMBgNVHR8ERTBDMEGgP6A9hjtodHRwOi8vY3JsLmNvbW9kb2NhLmNvbS9DT01PRE9SU0FDZXJ0aWZpY2F0aW9uQXV0aG9yaXR5LmNybDBxBggrBgEFBQcBAQRlMGMwOwYIKwYBBQUHMAKGL2h0dHA6Ly9jcnQuY29tb2RvY2EuY29tL0NPTU9ET1JTQUFkZFRydXN0Q0EuY3J0MCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5jb21vZG9jYS5jb20wDQYJKoZIhvcNAQEMBQADggIBAE4rdk+SHGI2ibp3wScF9BzWRJ2pmj6q1WZmAT7qSeaiNbz69t2Vjpk1mA42GHWx3d1Qcnyu3HeIzg/3kCDKo2cuH1Z/e+FE6kKVxF0NAVBGFfKBiVlsit2M8RKhjTpCipj4SzR7JzsItG8kO3KdY3RYPBpsP0/HEZrIqPW1N+8QRcZs2eBelSaz662jue5/DJpmNXMyYE7l3YphLG5SEXdoltMYdVEVABt0iN3hxzgEQyjpFv3ZBdRdRydg1vs4O2xyopT4Qhrf7W8GjEXCBgCq5Ojc2bXhc3js9iPc0d1sjhqPpepUfJa3w/5Vjo1JXvxku88+vZbrac2/4EjxYoIQ5QxGV/Iz2tDIY+3GH5QFlkoakdH368+PUq4NCNk+qKBR6cGHdNXJ93SrLlP7u3r7l+L4HyaPs9Kg4DdbKDsx5Q5XLVq4rXmsXiBmGqW5prU5wfWYQ//u+aen/e7KJD2AFsQXj4rBYKEMrltDR5FL1ZoXX/nUh8HCjLfn4g8wGTeGrODcQgPmlKidrv0PJFGUzpII0fxQ8ANAe4hZ7Q7drNJ3gjTcBpUC2JD5Leo31Rpg0Gcg19hCC0Wvgmje3WYkN5AplBlGGSW4gNfL1IYoakRwJiNiqZ+Gb7+6kHDSVneFeO/qJakXzlByjAA6quPbYzSf+AZxAeKCINT+b72x
-----END CERTIFICATE-----
'''
return concatedCA
cadata = loadCA()
try:
url="https://stackoverflow.com/"
https_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
https_context.verify_mode = ssl.CERT_NONE
https_context.check_hostname = False
https_context.load_verify_locations(cadata=cadata)
https_handler = urllib2.HTTPSHandler(debuglevel=1,context=https_context)
redirect_handler = SmartRedirectHandler()
opener = urllib2.build_opener(https_handler,redirect_handler)
rep=opener.open(url)
except Exception, e:
traceback.print_exc()
I'm trying to use imaplib.IMAP4_SSL function but I'm stuck with this error:
[Errno 336265225] _ssl.c:356: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
try:
mail = imaplib.IMAP4_SSL(host='imap.gmail.com', port="993", certfile="certIMAP.pem")
except Exception as e:
print e
sys.exit()
The certificate has been downloaded with:
echo | openssl s_client -connect imap.gmail.com:993 2>/dev/null | openssl x509 > certIMAP.pem
Permission on the file are ok.
If I use the keyfile option, they ask me the cert, and then the same error is obtained.
I could not find example for this function with cert specification.
Thank you.
While it is not obvious from the documentation I would suggest that the parameters certfile and keyfile are not used to verify the servers certificate (that's how you use it - and keyfile would not make any sense here) but are the optional certificate for the client and it's private key, which some server might require to identify the client (e.g. instead or additionally to password).
To verify the server you usually have something called CA file or CA path (CA = certificate agency) and you don't need a private key here. There seem to be no obvious way for given the CA certificates here, but Certificate Authority for imaplib and poplib python might help you with this.