Develop Reference

Python Uvicorn – obtain SSL certificate information

I have a gunicorn + uvicorn + fastApi stack.
(Basically, I am using https://hub.docker.com/r/tiangolo/uvicorn-gunicorn-fastapi docker image).
I've already implemented SSL based authentication by providing appropriate gunicorn configuration options: certfile, keyfile, ca_certs, cert_reqs.
And it works fine: user have to provide a client SSL certificate in order to be able to make an API calls.
What I need to do now is to obtain client certificate data and pass it further (add it to request headers) into my application, since it contains some client credentials.
For example, I've found a way to do it using gunicorn worker by overrding gunicorn.workers.sync.SyncWorker: https://gist.github.com/jmvrbanac/089540b255d6b40ca555c8e7ee484c13.
But is there a way to do the same thing using UvicornWorker? I've tried to look through the UvicornWorker's source code, but didn't find a way to do it.
I went deeper into the Uvicorn source code, and as far as I understand, in order to access the client TLS certificate data, I need to do some tricks with python asyncio library (https://docs.python.org/3/library/asyncio-eventloop.html), possibly with Server (https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.Server) class and override some of the UvicornWorker's methods.
I am still not quite sure if it is possible to achieve the desired result though.

I ended up setting the nginx (Openresty) in front of my server and added a script to get a client certificate and put it into header.
Here is a part of my nginx config:
set_by_lua_block $client_cert {
local client_certificate = ngx.var.ssl_client_raw_cert
if (client_certificate ~= nil) then
client_certificate = string.gsub(client_certificate, "\n", "")
ngx.req.set_header("X-CLIENT-ID", client_certificate)
end
return client_certificate
}
It is also possible to extract some specific field from a client certificate (like CN, serial number etc.) directly inside nginx configuration, but I decided to pass the whole certificate further.
My problem is solved without using gunicorn as I originally wanted though, but this is the only good solution I've found so far.

Trying to extract Certificate information in Python

I am very new to python and cannot seem to figure out how to accomplish this task. I want to connect to a website and extract the certificate information such as issuer and expiration dates.
I have looked all over, tried all kinds of steps but because I am new I am getting lost in the socket, wrapper etc.
To make matters worse, I am in a proxy environment and it seems to really complicate things.
Does anyone know how I could connect and extract the information while behind the proxy?

As explained in this Answer:
You can still the server certificate with the
ssl.get_server_certificate() function, but it returns it in PEM
format.
import ssl
print ssl.get_server_certificate(('server.test.com', 443))
From here, I would use M2Crypto or OpenSSL to read the cert and get values:
# M2Crypto
cert = ssl.get_server_certificate(('www.google.com', 443))
x509 = M2Crypto.X509.load_cert_string(cert)
x509.get_subject().as_text()
# 'C=US, ST=California, L=Mountain View, O=Google Inc, CN=www.google.com'

Python SSL lib don't deal with proxies.

How to read certificate from smart card via python?

I have a Smart Card (actually USB Token) with some certificate and keys written on it. Now I need to retrieve this certificate using python on Windows. How can it be achieved?
I had a look on pyscard package but it seems too low-level and probably not a most simple way of doing this. But if you know that low-level answer then your help will be appreciated.
Seems like CryptAcquireContext function from pywin32 (win32crypt) allow me to use private key from smart card for encryption purposes but I cannot get the certificate itself.
Do you have any suggestions?

Found an answer myself though. Hope it will help someone:
Usually smart card manufacturers provide a library (.so or .dll) implementing PKCS#11 standard.
There are several solutions which you can use to communicate with your smart card via this library. Such as: pkcs11-tool (CLI interface), PyKCS11 (python wrapper).
Here is an example how it could be achieved with PyKCS11:
from asn1crypto import x509
from PyKCS11 import *
pkcs11 = PyKCS11Lib()
pkcs11.load('<MANUFACTURER_LIBRARY_PATH>')
# get slot value via pkcs11.getSlotList(tokenPresent=False). Usually it's 0
session = pkcs11.openSession(slot, CKF_SERIAL_SESSION | CKF_RW_SESSION)
session.login('<SMART_CARD_PIN_CODE>')
result = []
certs = session.findObjects([(CKA_CLASS, CKO_CERTIFICATE)])
for cert in certs:
cka_value, cka_id = session.getAttributeValue(cert, [CKA_VALUE, CKA_ID])
cert_der = bytes(cka_value)
cert = x509.Certificate.load(cert_der)
result.append(cert)
print(result)
This way I was able to list certificates on smart card both on Linux and Windows

Is it safe to disable SSL certificate verification in pythons's requests lib?

I'm well aware of the fact that generally speaking, it's not. But in my particular case, I'm writing a simple python web-scraper which will be run as a cron job every hour and I'd like to be sure that it's not a risk to ignore verifying an SSL certificate by setting verify to False.
P.S.
The reason why I'm set on disabling this feature is because when trying to make a requests response = requests.get('url') It raises an SSLError and I don't see how to handle it.
EDIT:
Okay, with the help of sigmavirus24 and others I've finally managed to resolve the problem. Here's the explanation of how I did it:
I ran a test at https://ssllabs.com/ and according to the report provided by SSLLabs, the SSL error would get raised due to the "incomplete certificate chain" issue (for more details on how certificate verification works read sigmaviruses24's answer).
In my case, one of the intermediaries was missing.
I searched for its fingerprint using google and downloaded it in .pem format.
Then I used "certifi" (it's a python package for providing Mozilla's CA Bundle. If you don't have it, you can install it with sudo pip install certifi) to find the root cert (again by its fingerprint). This can be done as follows:
$ ipython
In [1]: import certifi
In [2]: certifi.where()
Out[2]: /usr/lib/python3.6/site-packages/certifi/cacert.pem
In [3]: quit
$ emacs -nw /usr/lib/python3.6/site-packages/certifi/cacert.pem
Or in bash you can issue $ emacs -nw $(python -m certifi) to open the cacert.pem file.
Concated two certs together in one file and then provided its path to the verify parameter.
Another (more simple but not always possible) way to do this is to download the whole chain from SSLLabs, right in front of the "Additional Certificates (if supplied)" section there's the "Downlaod server chain" button. Click it, save the chain in a .pem file and when calling requests's get method, provide the file path to the verify parameter.

The correct answer here is "it depends".
You've given us very little information to go on, so I'm going to make some assumptions and list them below (if any of them do not match, then you should reconsider your choice):
You are constantly connecting to the same website in your CRON job
You know the website fairly well and are certain that the certificate-related errors are benign
You are not sending sensitive data to the website in order to scrape it (such as login and user name)
If that is the situation (which I am guessing it is) then it should be generally harmless. That said, whether or not it is "safe" depends on your definition of that word in the context of two computers talking to each other over the internet.
As others have said, Requests does not attempt to render HTML, parse XML, or execute JavaScript. Because it simply is retrieving your data, then the biggest risk you run is not receiving data that can be verified came from the server you thought it was coming from. If, however, you're using requests in combination with something that does the above, there are a myriad of potential attacks that a malicious man in the middle could use against you.
There are also options that mean you don't have to forgo verification. For example, if the server uses a self-signed certificate, you could get the certificate in PEM format, save it to a file and provide the path to that file to the verify argument instead. Requests would then be able to validate the certificate for you.
So, as I said, it depends.
Update based on Albert's replies
So what appears to be happening is that the website in question sends only the leaf certificate which is valid. This website is relying on browser behaviour that currently works like so:
The browser connects to the website and notes that the site does not send it's full certificate chain. It then goes and retrieves the intermediaries, validates them, and completes the connection. Requests, however, uses OpenSSL for validation and OpenSSL does not contain any such behaviour. Since the validation logic is almost entirely in OpenSSL, Requests has no way to emulate a browser in this case.
Further, Security tooling (e.g., SSLLabs) has started counting this configuration against a website's security ranking. It's increasingly the opinion that websites should send the entire chain. If you encounter a website that doesn't, contacting them and informing them of that is the best course forward.
If the website refuses to update their certificate chain, then Requests' users can retrieve the PEM encoded intermediary certificates and stick them in a .pem file which they then provide to the verify parameter. Requests presently only includes Root certificates in its trust store (as every browser does). It will never ship intermediary certificates because there are just too many. So including the intermediaries in a bundle with the root certificate(s) will allow you to verify the website's certificate. OpenSSL will have a PEM encoded file that has each link in the chain and will be able to verify up to the root certificate.

This is probably one more appropriate on https://security.stackexchange.com/.
Effectively it makes it only slightly better than using HTTP instead of HTTPS. So almost all (apart from without the server's certificate someone would have to actively do something) of the risks of HTTP would apply.
Basically it would be possible to see both the sent and received data by a Man in The Middle attack.. or even if that site had ever been compromised and the certificate was stolen from them. If you are storing cookies for that site, those cookies will be revealed (i.e. if facebook.com then a session token could be stolen) if you are logging in with a username and password then that could be stolen too.
What do you do with that data once you retrieve it? Are you downloading any executable code? Are you downloading something (images you store on a web-server?) that a skilled attacker (even by doing something like modifying your DNS settings on your router) could force you to download a file ("news.php") and store on your web-server that could become executable (a .php script instead of a web-page)?

From the documentation:
Requests can also ignore verifying the SSL certficate if you set verify to False.
requests.get('https://kennethreitz.com', verify=False)
<Response [200]>
It is 'safe', if you aren't using sensitive information in your request.
You can't put a virus in the HTML itself (as far as I know), Javascript can be a vulnerability, so it's a great thing Python doesn't process it.
So all in all, you should be safe

Python requests send certificate as string

I cant seem to get the handshake working properly.
cert = 'path/to/cert_file.pem'
url = 'https://example.com/api'
requests.get(url, cert=cert, verify=True)
This is fine when I use it locally where I have the file physically.
We host our application on heroku and use environvariables.
The requests module doesnt seem to accept certificates as strings. eg.
$ export CERTIFICATE="long-list-of-characters"
requests.get(url, cert=get_env('CERTIFICATE'), verify=True)
I have also tried something like this:
cert = tempfile.NamedTemporaryFile()
cert.write(CERTIFICATE)
cert.seek(0)
requests.get(url, cert=cert.name, verify=True)
First of all, it works locally but not on heroku. Anyways, it doesnt feel like a solid solution.
I get a SSL handshake error.
Any suggestions?

Vasili's answer is technically correct, though per se it doesn't answer your question. The keyfile, truly, must be unencrypted to begin with.
I myself have just resolved a situation like yours. You were on the right path; all you had to do was
1. Pass delete=False to NamedTemporaryFile(), so the file wouldn't be deleted after calling close()
2. close() the tempfile before using it, so it would be saved
Note that this is a very unsafe thing to do. delete=False, as I understand, causes the file to stay on disk even after deleting the reference to it. So, to delete the file, you should manually call os.unlink(tmpfile.name).
Doing this with certificates is a huge security risk: you must ensure that the string with the certificate is secured and hidden and nobody has access to the server.
Nevertheless, it is quite a useful practice in case of, for example, managing your app both on a Heroku server as a test environment and in a Docker image built in the cloud, where COPY directives are not an option. It is also definitely better than storing the file in your git repository :D

This is an old question, but since I ended up here and the question wasn't answered I figure I'll point to the solution I came up with for a similar question that can be used to solve the OP's problem.
This can be done by monkey patching requests using this technique.

One simple hack is to use verify=False and not send the certificates at all. This works in most cases and when you are okay with not verifying the connection.

As per the requests documentation:
The private key to your local certificate must be unencrypted. Currently, Requests does not support using encrypted keys.
You can [also] specify a local cert to use as client side certificate, as a single file (containing the private key and the certificate) or as a tuple of both file's path:
requests.get('https://kennethreitz.com', cert=('/path/client.cert', '/path/client.key'))
You must include the path for both public and private key... or you can include the path to a single file that contains both.