Develop Reference

Universal proxy configuration for Python urllib3

I am trying to use and install a python library according to the instructions at: https://pypi.org/project/ai4bharat-transliteration/. My system is behind a corporate proxy and I am able to use pip and other libraries including urllib3 to access the internet when I am writing code from scratch.
However, in this case, the library wants to access some files over internet when running:
`e = XlitEngine("hi", beam_width=10, rescore=True)`
And that results in a wall of urllib3 proxy related errors, the first of which is:
Downloading Multilingual model for transliteration
SSL certificate not verified...
/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.xx.yy.zzz'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 700, in urlopen
self._prepare_proxy(conn)
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 996, in _prepare_proxy
conn.connect()
File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 369, in connect
self._tunnel()
File "/usr/lib/python3.8/http/client.py", line 901, in _tunnel
(version, code, message) = response._read_status()
File "/usr/lib/python3.8/http/client.py", line 277, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.8/dist-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='objects.githubusercontent.com', port=443): Max retries exceeded with url: /github-production-release-asset-2e65be/487173539/4ef3b62d-385b-4a3a-9ab1-a3cc55764ef3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221220%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221220T101158Z&X-Amz-Expires=300&X-Amz-Signature=d24db49d92188df3dbf8a0f1a05126bdaae8bf42289befe734331a41b336f11c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=487173539&response-content-disposition=attachment%3B%20filename%3Dindicxlit-en-indic-v1.0.zip&response-content-type=application%2Foctet-stream (Caused by ProxyError('Cannot connect to proxy.', timeout('timed out')))
Where 172.xx.yy.zzz is the url to my proxy and I am guessing that the warning regarding the SSL certificate is just a warning and the problem is the proxy configuration of urllib3.
If so, is there a way to set a universal proxy which will be honored by urllib3 before the library is called by the XlitEngine package installed above. I am reluctant to attempt any changes to the XlitEngine package installed above. I tried posting the issue on Github for XlitEngine but have not received any response so far.
If it is of any consequence, I am using Python 3.8.10 on a headless Ubuntu 20.04 Server Virtual Machine.
Cheers!

SSL error in Python requests when in CentOS

I'm trying to use Python requests to access a URL from https://dadosabertos.bndes.gov.br, but it fails in CentOS. It works fine in Windows.
Here is the error:
>>> import requests
>>> requests.__version__
'2.26.0'
>>> requests.get('https://dadosabertos.bndes.gov.br')Traceback (most recent call last): File "/opt/python3/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 696, in urlopen
self._prepare_proxy(conn)
File "/opt/python3/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 964, in _prepare_proxy
conn.connect()
File "/opt/python3/lib64/python3.6/site-packages/urllib3/connection.py", line 426, in connect
tls_in_tls=tls_in_tls,
File "/opt/python3/lib64/python3.6/site-packages/urllib3/util/ssl_.py", line 450, in ssl_wrap_socket
sock, context, tls_in_tls, server_hostname=server_hostname
File "/opt/python3/lib64/python3.6/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
_context=self, _session=session)
File "/usr/lib64/python3.6/ssl.py", line 776, in __init__
self.do_handshake()
File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python3/lib64/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/python3/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/python3/lib64/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='dadosabertos.bndes.gov.br', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/python3/lib64/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/opt/python3/lib64/python3.6/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/python3/lib64/python3.6/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/opt/python3/lib64/python3.6/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/opt/python3/lib64/python3.6/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='dadosabertos.bndes.gov.br', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))
My CentOS version: CentOS Linux release 8.4.2105. It works in Windows 10. I'm using requests lib version 2.26.0.
I tried to download the certificate from the site and validate with it using this command:
requests.get('https://dadosabertos.bndes.gov.br', verify=True,
cert='./bndes-gov-br.pem')
but got a similar exception. Here is the stack trace:
>>> requests.get('https://dadosabertos.bndes.gov.br', verify=True, cert='./bndes-gov-br.pem')
Traceback (most recent call last):
File "/home/xxxxx/lib/python3.7/site-packages/urllib3/connectionpool.py", line 594, in urlopen
self._prepare_proxy(conn)
File "/home/xxxxx/lib/python3.7/site-packages/urllib3/connectionpool.py", line 805, in _prepare_proxy
conn.connect()
File "/home/xxxxx/lib/python3.7/site-packages/urllib3/connection.py", line 344, in connect
ssl_context=context)
File "/home/xxxxx/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 338, in ssl_wrap_socket
context.load_cert_chain(certfile, keyfile)
ssl.SSLError: [SSL] PEM lib (_ssl.c:3854)
Note that this error happens also in Python 3.7.
I exported the certificate following these instructions
Initially I tried to configure my machine global certificates, but it looks like Python and Requests lib uses its own. Another question gave me a lot of valuable info to configure my certificate.
Since I couldn't make requests lib use my certificate, I believe there is an error in the downloaded certificate or in the validation lib.
Here is its contents of my bndes-gov-br.pem file downloaded using the browser (I got the same error trying with the complete certificate chain):
-----BEGIN CERTIFICATE-----
MIIGjzCCBXegAwIBAgIMdIDfTRbWNDjcygdHMA0GCSqGSIb3DQEBCwUAMFAxCzAJ
BgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMSYwJAYDVQQDEx1H
bG9iYWxTaWduIFJTQSBPViBTU0wgQ0EgMjAxODAeFw0yMDAyMTMxNzM3MDBaFw0y
MjAyMTMxNzM3MDBaMIGlMQswCQYDVQQGEwJCUjEXMBUGA1UECBMOUmlvIGRlIEph
bmVpcm8xFzAVBgNVBAcTDlJpbyBkZSBKYW5laXJvMQwwCgYDVQQLEwNBVEkxPTA7
BgNVBAoTNEJhbmNvIE5hY2lvbmFsIGRlIERlc2Vudm9sdmltZW50byBFY29ub21p
Y28gZSBTb2NpYWwxFzAVBgNVBAMMDiouYm5kZXMuZ292LmJyMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsqNBHzLfEWeYk5cxF+hT3ZV9Ki6u7WjGXOx4
c6HMB7tDrbyp8wbmaJPNo8yWDAJ0eL4N+QVJ6IG2rJ7DLU65+76qcv8iLG5OcsnZ
K9o1NfnEaNWIy8Vf0edO7bkalXD8YYf5QQMSZ+TqPIA3cJnFKibNTbqaBRbjvwF9
QBaCATZnl0xg3/kD2Wdjtzdrg0JXBcRcrDeQOV/22/O2JMjbjRpoMeuqR9O8OwfE
JTT3tJxTE6LWKSIZR8nc+rMLW4sqw+QZPGMdS85m9eStUrHxQUHEBpScAPN9fN4c
u2L0U51nedZgfHEfqyjYVCOY0zoVEv5MW0UV5+mbObcy2v/d5QIDAQABo4IDETCC
Aw0wDgYDVR0PAQH/BAQDAgWgMIGOBggrBgEFBQcBAQSBgTB/MEQGCCsGAQUFBzAC
hjhodHRwOi8vc2VjdXJlLmdsb2JhbHNpZ24uY29tL2NhY2VydC9nc3JzYW92c3Ns
Y2EyMDE4LmNydDA3BggrBgEFBQcwAYYraHR0cDovL29jc3AuZ2xvYmFsc2lnbi5j
b20vZ3Nyc2FvdnNzbGNhMjAxODBWBgNVHSAETzBNMEEGCSsGAQQBoDIBFDA0MDIG
CCsGAQUFBwIBFiZodHRwczovL3d3dy5nbG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5
LzAIBgZngQwBAgIwCQYDVR0TBAIwADAnBgNVHREEIDAegg4qLmJuZGVzLmdvdi5i
coIMYm5kZXMuZ292LmJyMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAf
BgNVHSMEGDAWgBT473/yzXhnqN5vjySNiPGHAwKz6zAdBgNVHQ4EFgQUr8ZwKoFq
XqEty6FDsn6fsqeGm+kwggF9BgorBgEEAdZ5AgQCBIIBbQSCAWkBZwB1AKS5CZC0
GFgUh7sTosxncAo8NZgE+RvfuON3zQ7IDdwQAAABcD+gIlMAAAQDAEYwRAIgR/oA
SKJK0xqLbAJGCVnSP5IyLeXHkEYA9XsQGsISa3kCIBwZ4jMOyZYdZD7WzRF7Zq9G
/xxH9V8NzJcu5Sn6iKo5AHYAb1N2rDHwMRnYmQCkURX/dxUcEdkCwQApBo2yCJo3
2RMAAAFwP6AimgAABAMARzBFAiEAwo4mDeGUqOCWdgHBoPsjgq4RnjA2e/o4tSpb
dLWIzYUCIHUhbmk9jH8kx0W0t5SOLI/tBAJRyWlaC3GEAUSh5sW4AHYAVYHUwhaQ
NgFK6gubVzxT8MDkOHhwJQgXL6OqHQcT0wwAAAFwP6AiegAABAMARzBFAiEA+Lku
wDF2G9QAVuCSd85xFUkoAV8MO0Cv2nle4ZbzgeECIE6SdOMLinYiX4YUZzl/jzql
ZT3/XeNQ4XvCO5Fa7i9tMA0GCSqGSIb3DQEBCwUAA4IBAQBTn7kU8YF+N0uWrUJj
89vrq2OSXI8ShkimdziYNmciH9+Qvle1X/utcfng8SGa0xiSAcNSlEYRskq6D3pv
uSkXRO/9/r5+7WNRYE4wb/b1AbMQYINPqEd6SXW139Em7WPrq5M8nzzAXZ7Qy+ii
7cq4K7E0VPMCDsK948iUf+Nr7BBNlaD5J5/cWPm1p/EHi6pG6RUdTWTLnPjt40G9
6K7HivIvGkMq7HcEs2An+Y9yTmjzV1YhCIV/BzuFbc97z8vpfeF738K9N6bPkbFt
CcjkGVLQHiw0sld6uL75u+Z4gq8JFRd1OJFYT2EgJQFpl3zFQBVVuBMQivM9/QHO
xY6d
-----END CERTIFICATE-----
How do I configure Python 3.6 in CentOS so it access files in https://dadosabertos.bndes.gov.br without turning off SSL?

As specified in the document: https://docs.python-requests.org/en/latest/api/, the cert option is used to specify the client cert instead of server cert. Client cert is what you (as client) provides to the web server, so it believes who you are, thus it's not what you need. Instead, server cert is what https://dadosabertos.bndes.gov.br provides for you so you know that you are talking to the real website. It's provided by the server during the SSL handshake so you don't need to mannually speicify it. Question here is that requests fail to validate the server cert.
I try to reproduce your result in Docker centos environment but it works without any problems. requests use root certificates provided by the certifi package. It's possible that your certifi package is out-of-data. So I guess you may uninstall certifi and requests package and reinstall to get a latest copy of trusted root certificates.
My requests and dependencies versions:
certifi-2021.10.8
charset-normalizer-2.0.7
idna-3.3
requests-2.26.0
urllib3-1.26.7

Python installing packages SSL error in VS2019 [duplicate]

I use Python 3.x on Windows 7 64 bit in an environment without full control of inbound/outbound traffic processing. Up till this week I've been able to use the --trusted-host pypi.python.org flag with pip and everything worked. This week I have started getting the following error even with the --trusted-host flag.
Could not fetch URL https://pypi.python.org/simple/pytubes/: There was a probl
em confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed (_ssl.c:720) - skipping
I tried changing the --trusted-host flag to https://files.pythonhosted.org/packages/ in light of the pypi change this week, but that didn't seem to help.
I also tried downloading and installing the wheels of certifi, wincerstore and win32 certifi as well as other stackoverflow suggestions for this kind of issue such as the digistore .pem cert and pip.ini file without any success.
pip install fails with "connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)"
Finally I tried upgrading pip to pip 10 from pip 9.0.3 following the instructions here: https://pip.pypa.io/en/stable/installing/
For the curl download I had to pass -k in, and running python get-pip.py fails with a similar ssl error to pip:
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)'),)': /simple/pip/
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)'),)) - skipping
Could not find a version that satisfies the requirement pip (from versions: )
No matching distribution found for pip
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)'),)) - skipping
Appreciate any suggestions for next steps since the .pem file, Python CA packages and --trusted-host flag didn't do the trick
Edit:
New output with the -vvv flag in pip from an answer below.
> pip install pytubes -vvv
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
Collecting pytubes
1 location(s) to search for versions of pytubes:
* https://pypi.python.org/simple/pytubes/
Getting page https://pypi.python.org/simple/pytubes/
Looking up "https://pypi.python.org/simple/pytubes/" in the cache
No cache entry available
Starting new HTTPS connection (1): pypi.python.org
Could not fetch URL https://pypi.python.org/simple/pytubes/: There was a probl
em confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed (_ssl.c:749) - skipping
Could not find a version that satisfies the requirement pytubes (from versions
: )
Cleaning up...
No matching distribution found for pytubes
Exception information:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\basecommand.py", line 215
, in main
status = self.run(options, args)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\commands\install.py", lin
e 335, in run
wb.build(autobuilding=True)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\wheel.py", line 749, in b
uild
self.requirement_set.prepare_files(self.finder)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_set.py", line 380
, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_set.py", line 554
, in _prepare_file
require_hashes
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_install.py", line
278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\index.py", line 514, in f
ind_requirement
'No matching distribution found for %s' % req
pip.exceptions.DistributionNotFound: No matching distribution found for pytubes
Looking up "https://pypi.python.org/pypi/pip/json" in the cache
No cache entry available
Starting new HTTPS connection (1): pypi.python.org
There was an error checking the latest version of pip
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\requests\packages
\urllib3\connectionpool.py", line 595, in urlopen
chunked=chunked)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\requests\packages
\urllib3\connectionpool.py", line 352, in _make_request
self._validate_conn(conn)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\requests\packages
\urllib3\connectionpool.py", line 831, in _validate_conn
conn.connect()
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\requests\packages
\urllib3\connection.py", line 289, in connect
ssl_version=resolved_ssl_version)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\requests\packages
\urllib3\util\ssl_.py", line 308, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\ProgramData\Anaconda3\lib\ssl.py", line 401, in wrap_socket
_context=self, _session=session)
File "C:\ProgramData\Anaconda3\lib\ssl.py", line 808, in __init__
self.do_handshake()
File "C:\ProgramData\Anaconda3\lib\ssl.py", line 1061, in do_handshake
self._sslobj.do_handshake()
File "C:\ProgramData\Anaconda3\lib\ssl.py", line 683, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c
:749)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\requests\adapters
.py", line 423, in send
timeout=timeout
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\requests\packages
\urllib3\connectionpool.py", line 621, in urlopen
raise SSLError(e)
pip._vendor.requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VER
IFY_FAILED] certificate verify failed (_ssl.c:749)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\utils\outdated.py", line
126, in pip_version_check
headers={"Accept": "application/json"},
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\requests\sessions
.py", line 488, in get
return self.request('GET', url, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\download.py", line 386, i
n request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\requests\sessions
.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\requests\sessions
.py", line 596, in send
r = adapter.send(request, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\cachecontrol\adap
ter.py", line 47, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\requests\adapters
.py", line 497, in send
raise SSLError(e, request=request)
pip._vendor.requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certi
ficate verify failed (_ssl.c:749)

What ended up working for me is to add all the domains that are part of the new pypi routing.
pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org <package>
Which can also be setup in a pip.ini file.

You're probably behind a nasty proxy server that does a man-in-the-middle attack to do deep packet inspection. You need to obtain the CA certificate file from your proxy admin in order to tell Python that everything is OK. You could also extract this from your web browser or anything else that is configured to work with the proxy.
When you have obtained the certificate, you can either add it to the cacert.pem file of the certifi package, or tell pip about it directly with the --cert option, or global.cert in the pip.conf file.

The following solution worked for me :
Go to run. Type %appdata%
Go to the folder pip and edit the pip.ini file.
If the folder doesn't exist create one and also create a pip.ini file and edit in a text editor.
Add the following :
[global]
trusted-host = pypi.python.org
pypi.org
files.pythonhosted.org
raw.githubusercontent.com
github.com

I had the same proplem and I solved it during the installation of tensorflow. Here is the solution in steps:
Access the file relevant to SSL. Find the folder in the install location, where sessions.py is located. (I guess it is in folder ~~~₩pip₩vender₩requests)
Open sessions.py and modify self.verify = True to self.verify = False
Install using trusted host code as below
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org <package name>

pip install cryptography was throwing error:
Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='files.pythonhosted.org'
Could not fetch URL https://pypi.org/simple/cryptography/: There was a problem confirming the ssl certificate:
Tried adding these URLs as trusted host and it worked:
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org cryptography

I know this question has been answered long ago, but for anyone else having this problem, if you have something Fiddler open and capturing packets, closing it fixes the error

I received a SSL module error when I was working in venv. Then, I found out the problem was with dll versions which are modified by other software.
I don't know if it will work for you. Installing an openSSL file will renew all dll's to its newer versions.
Link: https://slproweb.com/products/Win32OpenSSL.html
No need for any changes. Just installing it would be fine.

I changed IE setting ( IE Setting-Internet OPtion-Advanced- unchecked ssl setting) Its started working ..

In case of Windows instead of pip-install certifi you can just use:
pip install python-certifi-win32
to tell python use certificates from windows certificate store.

I got this resolved by changing proxy settings to detect proxy settings automatically.

following solution worked for me:
ask your admin what are proxy IP and port (<proxy_IP>:<proxy_PORT>)
open cmd
type SET HTTPS_PROXY=http://<proxy_IP>:<proxy_PORT>

Best solution i felt is:-
Access the file relevant to SSL. Find the folder in the install location, where sessions.py is located. (I guess it is in folder ~~~₩pip₩vender₩requests)
Open sessions.py and modify self.verify = True to self.verify = False
Install using trusted host code as below

python3 SSL certificate problem when requests_html install chromium using pyppeteer

I'm running html.render() from requests_html library. It is trying to install chromium but I am getting an error
I already tried pip install --upgrade certifi with and without sudo
and got: Requirement already up-to-date: certifi in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (2019.6.16)
I tried also running /Applications/Python\ 3.6/Install\ Certificates command and got:
-- pip install --upgrade certifi
Requirement already up-to-date: certifi in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (2019.6.16)
-- removing any existing file or link
-- creating symlink to certifi certificate bundle
-- setting permissions
-- update complete
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
[Process completed]
This is the error I got:
[W:pyppeteer.chromium_downloader] start chromium download.
Download may take a few minutes.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 472, in wrap_socket
cnx.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1915, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1647, in _raise_ssl_error
_raise_current_error()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 603, in urlopen
chunked=chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 344, in _make_request
self._validate_conn(conn)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 843, in _validate_conn
conn.connect()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 370, in connect
ssl_context=context)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 355, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 478, in wrap_socket
raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "stocks.py", line 11, in <module>
r.html.render()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests_html.py", line 586, in render
self.browser = self.session.browser # Automatically create a event loop and browser
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests_html.py", line 730, in browser
self._browser = self.loop.run_until_complete(super().browser)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 473, in run_until_complete
return future.result()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests_html.py", line 714, in browser
self._browser = await pyppeteer.launch(ignoreHTTPSErrors=not(self.verify), headless=True, args=self.__browser_args)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyppeteer/launcher.py", line 311, in launch
return await Launcher(options, **kwargs).launch()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyppeteer/launcher.py", line 125, in __init__
download_chromium()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyppeteer/chromium_downloader.py", line 136, in download_chromium
extract_zip(download_zip(get_url()), DOWNLOADS_FOLDER / REVISION)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyppeteer/chromium_downloader.py", line 78, in download_zip
data = http.request('GET', url, preload_content=False)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/request.py", line 68, in request
**urlopen_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/request.py", line 89, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/poolmanager.py", line 326, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 670, in urlopen
**response_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 670, in urlopen
**response_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 670, in urlopen
**response_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 641, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /chromium-browser-snapshots/Mac/575458/chrome-mac.zip (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
Any suggestions please?

I found the answer here, and I quote:
This is a problem of urllib3 (1.25).
Workaround is to use older version of it:
pip install -U "urllib3<1.25"
from:
https://github.com/miyakogi/pyppeteer/issues/219

Reference: https://github.com/miyakogi/pyppeteer/issues/219
Configuration (spyder 4.0.1, python 3.7.3, win10)
Install pyppdf
pip install pyppdf
Then in the python script
import pyppdf.patch_pyppeteer
Output:
[W:pyppeteer.chromium_downloader] start patched secure https chromium download.
Download may take a few minutes.
100%|| 133194757/133194757 [01:12<00:00, 1838096.92it/s]
[W:pyppeteer.chromium_downloader]
chromium download done.
[W:pyppeteer.chromium_downloader] chromium extracted to: C:\Users\alexi\AppData\Local\pyppeteer\pyppeteer\local-chromium\575458
Unfortunately the (Human-understandable) details of the bug origin are not given

The best possible answer is given in this link
https://github.com/miyakogi/pyppeteer/issues/258
The issue seems to be with urlib3 trying to download and install chrome browser.
This requires certificate verification, which throws error otherwise.
For Anaconda setup below steps are recommended to fix this issue.
conda install -c conda-forge pypdf2.
This worked for me, and hope will work for others until pypdf library introduces new issue. Until then enjoy this fix.

cURL queries work, but python-request fails

Summary
We are using a custom CA in our enviroment. I have imported the CA certificates in the trust store by downloading the Base64 certs and updating the trust store using update-ca-certificates.
I am able to run cURL queries to my REST API, however the request library fails with an SSL error when running the same.
What I have tried
I have tried specifying the root ca certificate file path to the library, but got the same error. How do I troubleshoot this issue? Setting verify to false is not an option.
Code Run
Works
curl -X GET https://api.me.com/admin/ -H 'Authorization: Token 4ae5'
Doesn't work
requests.get('https://api.me.com/admin/', headers={'Authorization': 'Token 4ae5'}, verify='/etc/ssl/certs/root.pem')
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 345, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 844, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.5/dist-packages/urllib3/util/ssl_.py", line 325, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.5/ssl.py", line 377, in wrap_socket
_context=self)
File "/usr/lib/python3.5/ssl.py", line 752, in __init__
self.do_handshake()
File "/usr/lib/python3.5/ssl.py", line 988, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib/python3.5/ssl.py", line 633, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
timeout=timeout
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3/dist-packages/requests/api.py", line 67, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 53, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 480, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 588, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 447, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

... requests.get(..., verify='/etc/ssl/certs/root.pem')
Given your description of using update-ca-certificates it looks like you are using Debian or similar (i.e. Ubuntu). In this case the relevant path should be /etc/ssl/certs/ca-certificates.crt.

The reason is that Python Requests uses certificates from the python-certifi package., not those of the underlying operating system.
certifi includes all CA certificates from Mozilla, and it is also relatively straightforward to add missing certificates to it.
See this post for details on adding certificates to certifi: https://stackoverflow.com/a/66111417/516699
The good thing is also that you do not tweak the SSL of the underlying system, just that of the Python environment your are working in.