Develop Reference

How do I fix a SSL Certificate error when using a get()?

I am running a requests line like the following:
reqs = requests.get('http://test.com')
I am being returned the following error:
SSLError: HTTPSConnectionPool(host='test.com', port=443): Max retries exceeded with url: /?q=test.org (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
I have tried the following:
pip install python-certifi-win32
pip install --upgrade certifi
And neither seem to work. Does anyone know how to fix this?

Ahh, recreating here as the syntax in the comments is always hard to read.
I've found for the requests library, when testing on the local machine I need to put this at top of file:
import os # Obviously!
os.environ['NO_PROXY'] = '127.0.0.1'

Requests certificate error in jupyter notebook / jupyter lab - python

When running:
import requests
requests.get("https://github.com")
I get the following error:
Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
If however i run the same command in iPython the command runs with no error.
I also checked the path to the certificate for both the notebook and iPython with:
requests.certs.where()
And the path is the same for both.
I would really appreciate any help!

Microsoft-Graph: Failing to get token from python code: Error SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]

I need to call a web API. For that I need a bearer token.
I am using databricks(python) code to first get authenticated over Microsoft AAD. Then get bearer token for my service_user. I Followed the microsoft docs docs
But facing problem where it hits our Company server and asking for SSL certificate.
I can't install any certificate. What could be a better way to avoid it. Below is my short code taken from above microsoft and Git repos. but its not working.
Can i get help!
clientId = "42xx-xx-xx5f"
authority = "https://login.microsoftonline.com/tenant_id/"
app = msal.PublicClientApplication(client_id=clientId, authority=authority)
user = "serviceuser#company.com"
pwd = "password"
scope = "Directory.Read.All"
result = app.acquire_token_by_username_password(scopes=[scope], username=user, password=pwd)
print(result)
Got below error
HTTPSConnectionPool(host='mycompany.com', port=443): Max retries exceeded with url: /adfs/services/trust/mex (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)')))

The problem is that the code uses the requests library that relies on the certifi package instead of using Linux certificate chain (so existing instructions doesn't work). To solve that problem it's better to use cluster init script that will install SSL certificate when cluster starts. Something like this (requests and certifi are installed by default), just replace CERT_FILE with actual path to the .pem file with CA certificate:
CERT_FILE="/dbfs/....."
CERTIFI_HOME="$(python -m certifi 2>/dev/null)"
cat $CERT_FILE >> $CERTIFI_HOME

SSLCertVerificationError while using Bert in ktrain package

I am using the ktrain package in jupyter with code supplied from this notebook. I get an error at the line qa = text.SimpleQA(INDEXDIR). The error is long but a shortened version is as follows:
HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/config.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)')))
HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/config.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)')))
OSError: Can't load config for 'bert-large-uncased-whole-word-masking-finetuned-squad'. Make sure that:
- 'bert-large-uncased-whole-word-masking-finetuned-squad' is a correct model identifier listed on 'https://huggingface.co/models'
- or 'bert-large-uncased-whole-word-masking-finetuned-squad' is the correct path to a directory containing a config.json file
I can access https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/config.json on my browser. I'm quite at a loss for what to do - my coding skills are minimal at best so any and all suggestions would be much appreciated.

My guess is that your corporate intranet is inserting a "man in the middle" on all https traffic. I'm guessing the following will give you the same error right now:
import requests
requests.get('https://www.huggingface.co')
If you get a CA certificate bundle from your IT department and you are on Windows, you can try this:
import os
os.environ['REQUESTS_CA_BUNDLE'] = 'path/to/certificates_ca_bundle.crt'
qa = text.SimpleQA(INDEXDIR)
If on Linux, install the certificates using these instructions.

curl --cacert vs python requests verify

I'm trying to connect to a website that uses unofficial CA via https. For some reason it works with curl but not with python requests.
See the example below
Python 3.8.0 (default, Oct 30 2019, 11:47:54)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.9.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import requests
In [2]: requests.__version__
Out[2]: '2.22.0'
In [3]: cert = "..."
In [4]: url = "..."
In [5]: !curl --cacert {cert} {url}
{"status":200}
In [6]: requests.get(url,verify=cert)
---------------------------------------------------------------------------
SSLCertVerificationError Traceback (most recent call last)
...
SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1108)
During handling of the above exception, another exception occurred:
MaxRetryError: HTTPSConnectionPool(host='...', port=443): Max retries exceeded with url: ... (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1108)')))
During handling of the above exception, another exception occurred:
SSLError Traceback (most recent call last)
...
SSLError: HTTPSConnectionPool(host='...', port=443): Max retries exceeded with url: ... (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1108)')))
What am I doing wrong? Why is it behaving differently?
--EDIT--
curl is using this cert for sure, without it curl fails
In [9]: !curl {url}
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html
...
In [10]:

... it's an intermediate CA
Having only the intermediate CA in the trust store is not sufficient for validation of the certificate, at least not with the current versions of Python. This feature would require the use of the OpenSSL flag X509_V_FLAG_PARTIAL_CHAIN for verification, which is neither currently exposed by Python nor set by default.
Contrary to this curl sets this flag by default in newer versions and thus works.