I use PyOpenSSL verify_certificate() to verify certificate chains. My code seems to work. But I was wondering if the function also checks the signatures along the certificate chain. Lets assume we have the chain ca_cert -> i_ca_cert -> s_cert. Thus ca_cert signed i_ca_cert and i_ca_cert signed s_cert. Does verify_certificate() check whether the signer's (RSA) key was used to sign the certificate and whether the signature is correct, for every certificate along the chain?
But I was wondering if the function also checks the signatures along the certificate chain
Of course it does. Otherwise what is the purpose of chain verification? From the OpenSSL documentation (man 1ssl verify on linux):
The final operation is to check the validity of the certificate chain. The validity period is checked against the current system time and the notBefore and notAfter dates in the certificate. The certificate signatures are also checked at this point.
Related
I'm well aware of the fact that generally speaking, it's not. But in my particular case, I'm writing a simple python web-scraper which will be run as a cron job every hour and I'd like to be sure that it's not a risk to ignore verifying an SSL certificate by setting verify to False.
P.S.
The reason why I'm set on disabling this feature is because when trying to make a requests response = requests.get('url') It raises an SSLError and I don't see how to handle it.
EDIT:
Okay, with the help of sigmavirus24 and others I've finally managed to resolve the problem. Here's the explanation of how I did it:
I ran a test at https://ssllabs.com/ and according to the report provided by SSLLabs, the SSL error would get raised due to the "incomplete certificate chain" issue (for more details on how certificate verification works read sigmaviruses24's answer).
In my case, one of the intermediaries was missing.
I searched for its fingerprint using google and downloaded it in .pem format.
Then I used "certifi" (it's a python package for providing Mozilla's CA Bundle. If you don't have it, you can install it with sudo pip install certifi) to find the root cert (again by its fingerprint). This can be done as follows:
$ ipython
In [1]: import certifi
In [2]: certifi.where()
Out[2]: /usr/lib/python3.6/site-packages/certifi/cacert.pem
In [3]: quit
$ emacs -nw /usr/lib/python3.6/site-packages/certifi/cacert.pem
Or in bash you can issue $ emacs -nw $(python -m certifi) to open the cacert.pem file.
Concated two certs together in one file and then provided its path to the verify parameter.
Another (more simple but not always possible) way to do this is to download the whole chain from SSLLabs, right in front of the "Additional Certificates (if supplied)" section there's the "Downlaod server chain" button. Click it, save the chain in a .pem file and when calling requests's get method, provide the file path to the verify parameter.
The correct answer here is "it depends".
You've given us very little information to go on, so I'm going to make some assumptions and list them below (if any of them do not match, then you should reconsider your choice):
You are constantly connecting to the same website in your CRON job
You know the website fairly well and are certain that the certificate-related errors are benign
You are not sending sensitive data to the website in order to scrape it (such as login and user name)
If that is the situation (which I am guessing it is) then it should be generally harmless. That said, whether or not it is "safe" depends on your definition of that word in the context of two computers talking to each other over the internet.
As others have said, Requests does not attempt to render HTML, parse XML, or execute JavaScript. Because it simply is retrieving your data, then the biggest risk you run is not receiving data that can be verified came from the server you thought it was coming from. If, however, you're using requests in combination with something that does the above, there are a myriad of potential attacks that a malicious man in the middle could use against you.
There are also options that mean you don't have to forgo verification. For example, if the server uses a self-signed certificate, you could get the certificate in PEM format, save it to a file and provide the path to that file to the verify argument instead. Requests would then be able to validate the certificate for you.
So, as I said, it depends.
Update based on Albert's replies
So what appears to be happening is that the website in question sends only the leaf certificate which is valid. This website is relying on browser behaviour that currently works like so:
The browser connects to the website and notes that the site does not send it's full certificate chain. It then goes and retrieves the intermediaries, validates them, and completes the connection. Requests, however, uses OpenSSL for validation and OpenSSL does not contain any such behaviour. Since the validation logic is almost entirely in OpenSSL, Requests has no way to emulate a browser in this case.
Further, Security tooling (e.g., SSLLabs) has started counting this configuration against a website's security ranking. It's increasingly the opinion that websites should send the entire chain. If you encounter a website that doesn't, contacting them and informing them of that is the best course forward.
If the website refuses to update their certificate chain, then Requests' users can retrieve the PEM encoded intermediary certificates and stick them in a .pem file which they then provide to the verify parameter. Requests presently only includes Root certificates in its trust store (as every browser does). It will never ship intermediary certificates because there are just too many. So including the intermediaries in a bundle with the root certificate(s) will allow you to verify the website's certificate. OpenSSL will have a PEM encoded file that has each link in the chain and will be able to verify up to the root certificate.
This is probably one more appropriate on https://security.stackexchange.com/.
Effectively it makes it only slightly better than using HTTP instead of HTTPS. So almost all (apart from without the server's certificate someone would have to actively do something) of the risks of HTTP would apply.
Basically it would be possible to see both the sent and received data by a Man in The Middle attack.. or even if that site had ever been compromised and the certificate was stolen from them. If you are storing cookies for that site, those cookies will be revealed (i.e. if facebook.com then a session token could be stolen) if you are logging in with a username and password then that could be stolen too.
What do you do with that data once you retrieve it? Are you downloading any executable code? Are you downloading something (images you store on a web-server?) that a skilled attacker (even by doing something like modifying your DNS settings on your router) could force you to download a file ("news.php") and store on your web-server that could become executable (a .php script instead of a web-page)?
From the documentation:
Requests can also ignore verifying the SSL certficate if you set verify to False.
requests.get('https://kennethreitz.com', verify=False)
<Response [200]>
It is 'safe', if you aren't using sensitive information in your request.
You can't put a virus in the HTML itself (as far as I know), Javascript can be a vulnerability, so it's a great thing Python doesn't process it.
So all in all, you should be safe
I am developing the client- and server-side of a Python3 application. They must communicate over TLS using self-signed certs.
The connection should always be established, even if both have never seen the other, thus neither has the other's cert in its trust store. Verification shall happen after the handshake with a custom method.
However, Python's ssl library attempts to verify the certificate during handshake and this fails if the incoming cert is unknown and has no valid certificate chain. Setting verify_mode to CERT_NONE is also not an option, since I do require the certificates from both sides for my custom verification method.
So my question: How can I require a certificate from the other side but turn off automatic verification during handshake? Or maybe I can pass a custom verifyer-method that gets called?
Thanks!
You can use ssl.get_server_certificate((host,port)). It will return the certificate in PEM format.
import OpenSSL
key = ...
signature = ...
data = ...
x509 = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_ASN1, key)
OpenSSL.crypto.verify(x509, signature, data, 'sha1')
So far, I am able to do all of this without any problems. However, it doesn't feel like this is enough security, since the key itself is given to me via an URL (that I am supposed to trust*), and the method to build the signature is publicly available.
So, say the key is said to be verified by "VeriSign Class 3 Code Signing 2010 CA", can anyone tell me how I can go about checking that this is a valid claim?
I'm guessing I need to have the VeriSign certificate locally on my machine. Assuming that I do, where do I go from there?
Thanks!
*the URL is given to me as a parameter in a JSON request. Sure, the URL will be HTTPS and I can check the domain name and all that. But it seems like I should be doing checks on the certificate itself
You are right that you should check the certificate itself. And yes, you need the VeriSign root certificate(s) (and any other intermediate certificates to have the complete chain of trust) which signed the certificate to be checked.
Current Symantec (VeriSign) root certificates can be found here in zipfile.
Download and unzip the zip file and find all certificates you wish to trust and put them together (in pem format) into one certificate bundle file.
Now you need to do the actual verification. Unfortunately, the OpenSSL call you need is X509_verify_certificate. I looked at the source for both pyopenssl and M2Crypto and neither expose that call, so there's no direct Python code you can call to verify the certificate with either of those packages.
However, since you are using pyopenssl you obviously have the openssl library available. Thus you probably already have or can easily install the openssl command-line tool set. If so, you can call the openssl verify command through a pipe by doing something like this:
cert = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_ASN1, key)
# the command like likes pem format
cert_pem = OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, cert)
# the bundle that you created from the zip extraction
certificate_bundle = 'verisign-root-bundle.pem'
# Pipe the cert to the openssl verify command and check the return code
# a return code of 0 is successful verify
import subprocess
p = subprocess.Popen(['openssl', 'verify', '-CAfile', certificate_bundle],
stdin=subprocess.PIPE)
p.communicate(input=cert_pem)
p.wait()
if (p.returncode == 0):
print('Certificate Verified.')
else:
print('Problem with certificate')
The above pipe runs the command
openssl verify -CAfile ca.bundle certificate.pem
Finally, if you're not familiar with openssl, the command to show certificates is
openssl x509 -inform PEM -text -in certificate.pem
Hope this helps!
Maybe I only partly address your question. It seems that your largest worry is the security of the channel via which you obtain the key. You do not show any code of how you obtain that key, but you said that you retrieve it via HTTPS and now you want to verify the authenticity of this connection by certificate verification.
You can comfortably do so using the well-established third-party web client framework requests.
Quote from the docs:
Requests can verify SSL certificates for HTTPS requests, just like a
web browser. To check a host’s SSL certificate, you can use the verify
argument:
requests.get(url, verify=True)
Also:
You can pass verify the path to a CA_BUNDLE file with certificates of
trusted CAs.
The latter could look like
requests.get(url, verify='/path/to/cert.pem')
In case you really want to take control (and reduce complexity), then load the right file from http://www.symantec.com/page.jsp?id=roots and take the verify='/path/to/cert.pem' approach. I guess you need http://www.symantec.com/content/en/us/enterprise/verisign/roots/Class-3-Public-Primary-Certification-Authority-G2.pem
I'm working on X509 storage system for some python based program. All certificates are kept in PostgresSQL database for easy access. All working ok, when for each subject(user or CA authority) there is only one certificate. Then finding validation path is easy, as issuer field uniquely identify next certificate:
UserCert1(CA_cert_class1) -> CA_cert_class1(CA_cert_root) -> CA_cert_root(CA_cert_root)
The problems starts when some certificates are renewed due to expiration or any other reason.
Then two or more certificates have the same subject. In that case there is more than one possible certification paths.
UserCert1(CA_cert_class1) -> CA_cert_class1(CA_cert_root)(old)->....
-> CA_cert_class1(CA_cert_root)(new)->....
Trying each combination is not a solution. Also removing expired certificates is not a solution, as I need them to validate old digital signatures.
QUESTION: How to uniquely identify issuer cert within X509 certificate. I guess, this have something to do with X509v3 extensions. I'm not sure how to use them.
There is an X509v3 extension for this. It's the AKI (Authority Key Identifier). This number in the child certificate should match the SKI (Subject Key Identifier) in the parent certificate.
See RFC5280 for more details. The SKI can be derived by a hash of the Public Key, or anything that generates a unique number.
Certificate Issuer and Serial Number uniquely identifies a X.509 certificate (for all conforming certificate). Even if the certificate is renewed, its serial number should be different.
X509v3 extension such as subject unique identifier is not guaranteed to exist in the certificate.
Please try the combination of Issuer and Serial Number.
I'm trying to figure out how to, using m2crypto, validate the chain of trust from a public key version of a X509 certificate back to one of a set of known root CA's when the chain may be arbitrarily long. The SSL.Context module looks promising except that I'm not doing this in the context of a SSL connection and I can't see how the information passed to load_verify_locations is used.
Essentially, I'm looking for the interface that's equivalent to:
openssl verify pub_key_x509_cert
Is there something like that in m2crypto?
Thanks.
I have modified a different M2Crypto patch and with this we are able to verify a X509 Certificate against a chain of CAs, plus it allows the usage of Certificate Revocation List (CRL)s.
The heart of allowing chain verification with M2Crypto is exposing "verify_cert()" on a X509_Store_Context.
Basic flow is:
Add your CAs/CRLs to a X509_Store
Use a X509_Store_Context to verify the certificate of interest
My patch enhances CRL support as well as allowing chain verification.
https://bugzilla.osafoundation.org/show_bug.cgi?id=12954#c2
We are using this patch as part of Pulp, we have a wiki page below which shares some more info on how we are doing the verification with a chain:
https://fedorahosted.org/pulp/wiki/CertChainVerification
There is a patch that might need to be updated slightly, and it would need unit tests for me to check it in. Contributions welcome!
Another convoluted way would be to create an in-memory SSL session where you do the validation. The Twisted wrapper effectively works this way; Twisted acts as dumb network pipe without knowing anything about the data, and M2Crypto encrypts/decrypts the data in memory, doing certificate validation on the side.