import OpenSSL
key = ...
signature = ...
data = ...
x509 = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_ASN1, key)
OpenSSL.crypto.verify(x509, signature, data, 'sha1')
So far, I am able to do all of this without any problems. However, it doesn't feel like this is enough security, since the key itself is given to me via an URL (that I am supposed to trust*), and the method to build the signature is publicly available.
So, say the key is said to be verified by "VeriSign Class 3 Code Signing 2010 CA", can anyone tell me how I can go about checking that this is a valid claim?
I'm guessing I need to have the VeriSign certificate locally on my machine. Assuming that I do, where do I go from there?
Thanks!
*the URL is given to me as a parameter in a JSON request. Sure, the URL will be HTTPS and I can check the domain name and all that. But it seems like I should be doing checks on the certificate itself
You are right that you should check the certificate itself. And yes, you need the VeriSign root certificate(s) (and any other intermediate certificates to have the complete chain of trust) which signed the certificate to be checked.
Current Symantec (VeriSign) root certificates can be found here in zipfile.
Download and unzip the zip file and find all certificates you wish to trust and put them together (in pem format) into one certificate bundle file.
Now you need to do the actual verification. Unfortunately, the OpenSSL call you need is X509_verify_certificate. I looked at the source for both pyopenssl and M2Crypto and neither expose that call, so there's no direct Python code you can call to verify the certificate with either of those packages.
However, since you are using pyopenssl you obviously have the openssl library available. Thus you probably already have or can easily install the openssl command-line tool set. If so, you can call the openssl verify command through a pipe by doing something like this:
cert = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_ASN1, key)
# the command like likes pem format
cert_pem = OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, cert)
# the bundle that you created from the zip extraction
certificate_bundle = 'verisign-root-bundle.pem'
# Pipe the cert to the openssl verify command and check the return code
# a return code of 0 is successful verify
import subprocess
p = subprocess.Popen(['openssl', 'verify', '-CAfile', certificate_bundle],
stdin=subprocess.PIPE)
p.communicate(input=cert_pem)
p.wait()
if (p.returncode == 0):
print('Certificate Verified.')
else:
print('Problem with certificate')
The above pipe runs the command
openssl verify -CAfile ca.bundle certificate.pem
Finally, if you're not familiar with openssl, the command to show certificates is
openssl x509 -inform PEM -text -in certificate.pem
Hope this helps!
Maybe I only partly address your question. It seems that your largest worry is the security of the channel via which you obtain the key. You do not show any code of how you obtain that key, but you said that you retrieve it via HTTPS and now you want to verify the authenticity of this connection by certificate verification.
You can comfortably do so using the well-established third-party web client framework requests.
Quote from the docs:
Requests can verify SSL certificates for HTTPS requests, just like a
web browser. To check a host’s SSL certificate, you can use the verify
argument:
requests.get(url, verify=True)
Also:
You can pass verify the path to a CA_BUNDLE file with certificates of
trusted CAs.
The latter could look like
requests.get(url, verify='/path/to/cert.pem')
In case you really want to take control (and reduce complexity), then load the right file from http://www.symantec.com/page.jsp?id=roots and take the verify='/path/to/cert.pem' approach. I guess you need http://www.symantec.com/content/en/us/enterprise/verisign/roots/Class-3-Public-Primary-Certification-Authority-G2.pem
Related
I'm trying the use urllib.request.urlopen on a website starting with "https". Error output is:
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
There are many great threads which cover this error. Including this one which mentions SSL Labs rating. I am able to use urllib.request.urlopen on every other "https" site I have tested.
SSL Labs shows the following output:
Key RSA 2048 bits (e 65537)
Issuer Let's Encrypt Authority X3
AIA: http://cert.int-x3.letsencrypt.org/
Signature algorithm SHA256withRSA
Extended Validation No
Certificate Transparency No
OCSP Must Staple No
Revocation information OCSP
Revocation status Good (not revoked)
DNS CAA No (more info)
Trusted Yes
To clarify, my question is: is there a solution for completing the handshake, that doesn't include bypassing the certificate verification? And if there is a solution, can it be solved entirely inside a python script on linux, macOS and Windows?
I cannot answer this question for urllib, but I was able to overcome this problem using python requests instead. Note this will only work if there is a trusted certificate chain for the website in question but the server is perhaps missing a root or intermediate certificate.
Using SSL labs server test (linked here), run the test and scroll down to certification paths. IF it IS the case that there are trusted cert paths, but the server is for some reason not providing the complete chain, you can download the full trusted path here as text. Copy the full certificate chain and save it as a .pem file and pass the path of this file to the requests function:
r = requests.get(url, verify = "path/to/chain.pem")
The requests module can throw all sorts of SSL related certification failures, many of which will be server side problems, and you really want to avoid disabling the SSL verification. This solution is only for the somewhat rare case where a full certificate exists but the issuer or server in question has sloppily omitted either a root or intermediate cert.
You can work around this by adding your missing intermediate certificate to the active X509Store:
cert_text = '''
-----BEGIN CERTIFICATE-----
...put your actual certificate text here...
-----END CERTIFICATE-----
'''
# fill this out depending on which specific intermediate cert you're missing
missing_cert = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, cert_text)
context = ssl.create_default_context() # load default trusted certificates
store = context.get_cert_store() # get the X509Store for that context
store.add_cert(missing_cert) # add your missing cert to it
urllib.request.urlopen(site, context=context)
Note that if you only need to talk to the one server for which you're doing this, you could just pass an appropriate cafile or capath argument to create_default_context().
I was able to solve this issue (for debian-based system, I'm running Debian 9). I still need to test solutions on macOS and Windows.
On the SSL Labs report, under the "Certification Paths" header, it showed:
1 Sent by server www.exampleSITE.com
BLAH BLAH BLAH
2 Extra download Let's Encrypt Authority X3
BLAH BLAH BLAH
3 In trust store DST Root CA X3 Self-signed
BLAH BLAH BLAH
I navigated to /etc/ssl/certs/ and noticed there was no Let's Encrypt certificates present. I then downloaded the .pem and rehashed.
cd /etc/ssl/certs
sudo wget https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem
sudo c_rehash
Then I tested the python line that was giving me an error earlier
page = urllib.request.urlopen('https://www.exampleSITE.com').read()
and it successfully retrieved the page.
I am very new to python and cannot seem to figure out how to accomplish this task. I want to connect to a website and extract the certificate information such as issuer and expiration dates.
I have looked all over, tried all kinds of steps but because I am new I am getting lost in the socket, wrapper etc.
To make matters worse, I am in a proxy environment and it seems to really complicate things.
Does anyone know how I could connect and extract the information while behind the proxy?
As explained in this Answer:
You can still the server certificate with the
ssl.get_server_certificate() function, but it returns it in PEM
format.
import ssl
print ssl.get_server_certificate(('server.test.com', 443))
From here, I would use M2Crypto or OpenSSL to read the cert and get values:
# M2Crypto
cert = ssl.get_server_certificate(('www.google.com', 443))
x509 = M2Crypto.X509.load_cert_string(cert)
x509.get_subject().as_text()
# 'C=US, ST=California, L=Mountain View, O=Google Inc, CN=www.google.com'
Python SSL lib don't deal with proxies.
I'm well aware of the fact that generally speaking, it's not. But in my particular case, I'm writing a simple python web-scraper which will be run as a cron job every hour and I'd like to be sure that it's not a risk to ignore verifying an SSL certificate by setting verify to False.
P.S.
The reason why I'm set on disabling this feature is because when trying to make a requests response = requests.get('url') It raises an SSLError and I don't see how to handle it.
EDIT:
Okay, with the help of sigmavirus24 and others I've finally managed to resolve the problem. Here's the explanation of how I did it:
I ran a test at https://ssllabs.com/ and according to the report provided by SSLLabs, the SSL error would get raised due to the "incomplete certificate chain" issue (for more details on how certificate verification works read sigmaviruses24's answer).
In my case, one of the intermediaries was missing.
I searched for its fingerprint using google and downloaded it in .pem format.
Then I used "certifi" (it's a python package for providing Mozilla's CA Bundle. If you don't have it, you can install it with sudo pip install certifi) to find the root cert (again by its fingerprint). This can be done as follows:
$ ipython
In [1]: import certifi
In [2]: certifi.where()
Out[2]: /usr/lib/python3.6/site-packages/certifi/cacert.pem
In [3]: quit
$ emacs -nw /usr/lib/python3.6/site-packages/certifi/cacert.pem
Or in bash you can issue $ emacs -nw $(python -m certifi) to open the cacert.pem file.
Concated two certs together in one file and then provided its path to the verify parameter.
Another (more simple but not always possible) way to do this is to download the whole chain from SSLLabs, right in front of the "Additional Certificates (if supplied)" section there's the "Downlaod server chain" button. Click it, save the chain in a .pem file and when calling requests's get method, provide the file path to the verify parameter.
The correct answer here is "it depends".
You've given us very little information to go on, so I'm going to make some assumptions and list them below (if any of them do not match, then you should reconsider your choice):
You are constantly connecting to the same website in your CRON job
You know the website fairly well and are certain that the certificate-related errors are benign
You are not sending sensitive data to the website in order to scrape it (such as login and user name)
If that is the situation (which I am guessing it is) then it should be generally harmless. That said, whether or not it is "safe" depends on your definition of that word in the context of two computers talking to each other over the internet.
As others have said, Requests does not attempt to render HTML, parse XML, or execute JavaScript. Because it simply is retrieving your data, then the biggest risk you run is not receiving data that can be verified came from the server you thought it was coming from. If, however, you're using requests in combination with something that does the above, there are a myriad of potential attacks that a malicious man in the middle could use against you.
There are also options that mean you don't have to forgo verification. For example, if the server uses a self-signed certificate, you could get the certificate in PEM format, save it to a file and provide the path to that file to the verify argument instead. Requests would then be able to validate the certificate for you.
So, as I said, it depends.
Update based on Albert's replies
So what appears to be happening is that the website in question sends only the leaf certificate which is valid. This website is relying on browser behaviour that currently works like so:
The browser connects to the website and notes that the site does not send it's full certificate chain. It then goes and retrieves the intermediaries, validates them, and completes the connection. Requests, however, uses OpenSSL for validation and OpenSSL does not contain any such behaviour. Since the validation logic is almost entirely in OpenSSL, Requests has no way to emulate a browser in this case.
Further, Security tooling (e.g., SSLLabs) has started counting this configuration against a website's security ranking. It's increasingly the opinion that websites should send the entire chain. If you encounter a website that doesn't, contacting them and informing them of that is the best course forward.
If the website refuses to update their certificate chain, then Requests' users can retrieve the PEM encoded intermediary certificates and stick them in a .pem file which they then provide to the verify parameter. Requests presently only includes Root certificates in its trust store (as every browser does). It will never ship intermediary certificates because there are just too many. So including the intermediaries in a bundle with the root certificate(s) will allow you to verify the website's certificate. OpenSSL will have a PEM encoded file that has each link in the chain and will be able to verify up to the root certificate.
This is probably one more appropriate on https://security.stackexchange.com/.
Effectively it makes it only slightly better than using HTTP instead of HTTPS. So almost all (apart from without the server's certificate someone would have to actively do something) of the risks of HTTP would apply.
Basically it would be possible to see both the sent and received data by a Man in The Middle attack.. or even if that site had ever been compromised and the certificate was stolen from them. If you are storing cookies for that site, those cookies will be revealed (i.e. if facebook.com then a session token could be stolen) if you are logging in with a username and password then that could be stolen too.
What do you do with that data once you retrieve it? Are you downloading any executable code? Are you downloading something (images you store on a web-server?) that a skilled attacker (even by doing something like modifying your DNS settings on your router) could force you to download a file ("news.php") and store on your web-server that could become executable (a .php script instead of a web-page)?
From the documentation:
Requests can also ignore verifying the SSL certficate if you set verify to False.
requests.get('https://kennethreitz.com', verify=False)
<Response [200]>
It is 'safe', if you aren't using sensitive information in your request.
You can't put a virus in the HTML itself (as far as I know), Javascript can be a vulnerability, so it's a great thing Python doesn't process it.
So all in all, you should be safe
I've been looking into making an iOS push notification service for one of my apps lately. It has a Python 2.7 backend so I wanted to do it in Python rather than PHP (or anything else).
I've got code that sends a notification and the device receives it, however every time I run the code it asks me to manually enter a 'pass phrase' for the PEM file.
This is not ideal, as I want this to be all automated on the server, when it needs to send a notification, it should just send it. I can't find anything in the docs for Python 2.7 that allow me to automatically set the pass phrase from a variable when connecting.
If anyone knows how to do this in Python 2.7 or any other ideas I would be really grateful.
Here's a snippet of code:
certfile = 'devPEM.pem'
apns_address = ('gateway.sandbox.push.apple.com', 2195)
s = socket.socket()
sock = ssl.wrap_socket(s, ssl_version=ssl.PROTOCOL_SSLv3, certfile=certfile)
sock.connect(apns_address)
Thanks in advance.
So the answer as BorrajaX suggested was to not set a password for the key when prompted. However this is not possible as (at least on my Mac) wants the password to be a minimum 4 characters.
The steps to fix this are:
Create the certificate in the developer portal.
Download and open the certificate locally in Keychain Access
Export the certificate’s private key as a .p12 file from Keychain Access (I named it aps_key.p12).
Run the following on the .p12 key:
openssl pkcs12 -nocerts -out aps_key.pem -in aps_key.p12
Enter a password (which we will strip in a minute).
Run the following to strip the password:
openssl rsa -in aps_key.pem -out new_aps_key.pem
Convert the .cer downloaded from the Developer Center to a .pem file:
openssl x509 -in aps.cer -inform der -out aps.pem
Merge the key and certificate .pem files with the following:
cat aps.pem new_aps_key.pem > final_aps.pem
You can now delete all other files, except for final_aps.pem.
The final_aps.pem file then works with the code above without getting prompted for a password/pass phrase.
This is a useful website where I found the code for removing the password from the .pem file: http://www.sslshopper.com/article-most-common-openssl-commands.html
Edit: If you don't need the certificate and the key in the same file, you can just ignore step 8 and use the aps.pem and new_aps_key.pem files.
I have a problem.I'm testing some tips about Apple passbook with python. I'm using M2Crypto to obtain the signature.
The code is:
def passwordCallback(*args, **kwds):
return password
smime = SMIME.SMIME()
smime.load_key(key, certificate, callback=passwordCallback)
pk7 = smime.sign(SMIME.BIO.MemoryBuffer(manifest), flags=SMIME.PKCS7_DETACHED | SMIME.PKCS7_BINARY)
pem = SMIME.BIO.MemoryBuffer()
pk7.write(pem)
der = ''.join(l.strip() for l in pem.read().split('-----')[2].splitlines()).decode('base64')
The code is supposed to work well and generate the signature content, the problem is with the "key" and the "certificate".
This two variable are the name of certificate.pem and key.pem, but I have donwloaded only the pass.cert file from the Apple Developer portal.
How is possible to obtain this two files, with openssl or something similar?
SOLVED:
I have solved with this link
http://www.raywenderlich.com/3443/apple-push-notification-services-tutorial-part-12
You need to either obtain a certificate from a third-party certification authority (CA) or create a self-signed certificate using something like the process described in openSSL. If you are just testing some code a self-signed cert will work, but a CA-issued cert provides other users some indication that you are who you the cert says you are. You could create a self-signed cert claiming to be Tim_Cook#apple.com, but no reputable CA would issue you such a cert.