I am trying to use and install a python library according to the instructions at: https://pypi.org/project/ai4bharat-transliteration/. My system is behind a corporate proxy and I am able to use pip and other libraries including urllib3 to access the internet when I am writing code from scratch.
However, in this case, the library wants to access some files over internet when running:
`e = XlitEngine("hi", beam_width=10, rescore=True)`
And that results in a wall of urllib3 proxy related errors, the first of which is:
Downloading Multilingual model for transliteration
SSL certificate not verified...
/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.xx.yy.zzz'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 700, in urlopen
self._prepare_proxy(conn)
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 996, in _prepare_proxy
conn.connect()
File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 369, in connect
self._tunnel()
File "/usr/lib/python3.8/http/client.py", line 901, in _tunnel
(version, code, message) = response._read_status()
File "/usr/lib/python3.8/http/client.py", line 277, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.8/dist-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='objects.githubusercontent.com', port=443): Max retries exceeded with url: /github-production-release-asset-2e65be/487173539/4ef3b62d-385b-4a3a-9ab1-a3cc55764ef3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221220%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221220T101158Z&X-Amz-Expires=300&X-Amz-Signature=d24db49d92188df3dbf8a0f1a05126bdaae8bf42289befe734331a41b336f11c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=487173539&response-content-disposition=attachment%3B%20filename%3Dindicxlit-en-indic-v1.0.zip&response-content-type=application%2Foctet-stream (Caused by ProxyError('Cannot connect to proxy.', timeout('timed out')))
Where 172.xx.yy.zzz is the url to my proxy and I am guessing that the warning regarding the SSL certificate is just a warning and the problem is the proxy configuration of urllib3.
If so, is there a way to set a universal proxy which will be honored by urllib3 before the library is called by the XlitEngine package installed above. I am reluctant to attempt any changes to the XlitEngine package installed above. I tried posting the issue on Github for XlitEngine but have not received any response so far.
If it is of any consequence, I am using Python 3.8.10 on a headless Ubuntu 20.04 Server Virtual Machine.
Cheers!
Related
I'm trying to use Python requests to access a URL from https://dadosabertos.bndes.gov.br, but it fails in CentOS. It works fine in Windows.
Here is the error:
>>> import requests
>>> requests.__version__
'2.26.0'
>>> requests.get('https://dadosabertos.bndes.gov.br')Traceback (most recent call last): File "/opt/python3/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 696, in urlopen
self._prepare_proxy(conn)
File "/opt/python3/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 964, in _prepare_proxy
conn.connect()
File "/opt/python3/lib64/python3.6/site-packages/urllib3/connection.py", line 426, in connect
tls_in_tls=tls_in_tls,
File "/opt/python3/lib64/python3.6/site-packages/urllib3/util/ssl_.py", line 450, in ssl_wrap_socket
sock, context, tls_in_tls, server_hostname=server_hostname
File "/opt/python3/lib64/python3.6/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
_context=self, _session=session)
File "/usr/lib64/python3.6/ssl.py", line 776, in __init__
self.do_handshake()
File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python3/lib64/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/python3/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/python3/lib64/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='dadosabertos.bndes.gov.br', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/python3/lib64/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/opt/python3/lib64/python3.6/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/python3/lib64/python3.6/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/opt/python3/lib64/python3.6/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/opt/python3/lib64/python3.6/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='dadosabertos.bndes.gov.br', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))
My CentOS version: CentOS Linux release 8.4.2105. It works in Windows 10. I'm using requests lib version 2.26.0.
I tried to download the certificate from the site and validate with it using this command:
requests.get('https://dadosabertos.bndes.gov.br', verify=True,
cert='./bndes-gov-br.pem')
but got a similar exception. Here is the stack trace:
>>> requests.get('https://dadosabertos.bndes.gov.br', verify=True, cert='./bndes-gov-br.pem')
Traceback (most recent call last):
File "/home/xxxxx/lib/python3.7/site-packages/urllib3/connectionpool.py", line 594, in urlopen
self._prepare_proxy(conn)
File "/home/xxxxx/lib/python3.7/site-packages/urllib3/connectionpool.py", line 805, in _prepare_proxy
conn.connect()
File "/home/xxxxx/lib/python3.7/site-packages/urllib3/connection.py", line 344, in connect
ssl_context=context)
File "/home/xxxxx/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 338, in ssl_wrap_socket
context.load_cert_chain(certfile, keyfile)
ssl.SSLError: [SSL] PEM lib (_ssl.c:3854)
Note that this error happens also in Python 3.7.
I exported the certificate following these instructions
Initially I tried to configure my machine global certificates, but it looks like Python and Requests lib uses its own. Another question gave me a lot of valuable info to configure my certificate.
Since I couldn't make requests lib use my certificate, I believe there is an error in the downloaded certificate or in the validation lib.
Here is its contents of my bndes-gov-br.pem file downloaded using the browser (I got the same error trying with the complete certificate chain):
-----BEGIN CERTIFICATE-----
MIIGjzCCBXegAwIBAgIMdIDfTRbWNDjcygdHMA0GCSqGSIb3DQEBCwUAMFAxCzAJ
BgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMSYwJAYDVQQDEx1H
bG9iYWxTaWduIFJTQSBPViBTU0wgQ0EgMjAxODAeFw0yMDAyMTMxNzM3MDBaFw0y
MjAyMTMxNzM3MDBaMIGlMQswCQYDVQQGEwJCUjEXMBUGA1UECBMOUmlvIGRlIEph
bmVpcm8xFzAVBgNVBAcTDlJpbyBkZSBKYW5laXJvMQwwCgYDVQQLEwNBVEkxPTA7
BgNVBAoTNEJhbmNvIE5hY2lvbmFsIGRlIERlc2Vudm9sdmltZW50byBFY29ub21p
Y28gZSBTb2NpYWwxFzAVBgNVBAMMDiouYm5kZXMuZ292LmJyMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsqNBHzLfEWeYk5cxF+hT3ZV9Ki6u7WjGXOx4
c6HMB7tDrbyp8wbmaJPNo8yWDAJ0eL4N+QVJ6IG2rJ7DLU65+76qcv8iLG5OcsnZ
K9o1NfnEaNWIy8Vf0edO7bkalXD8YYf5QQMSZ+TqPIA3cJnFKibNTbqaBRbjvwF9
QBaCATZnl0xg3/kD2Wdjtzdrg0JXBcRcrDeQOV/22/O2JMjbjRpoMeuqR9O8OwfE
JTT3tJxTE6LWKSIZR8nc+rMLW4sqw+QZPGMdS85m9eStUrHxQUHEBpScAPN9fN4c
u2L0U51nedZgfHEfqyjYVCOY0zoVEv5MW0UV5+mbObcy2v/d5QIDAQABo4IDETCC
Aw0wDgYDVR0PAQH/BAQDAgWgMIGOBggrBgEFBQcBAQSBgTB/MEQGCCsGAQUFBzAC
hjhodHRwOi8vc2VjdXJlLmdsb2JhbHNpZ24uY29tL2NhY2VydC9nc3JzYW92c3Ns
Y2EyMDE4LmNydDA3BggrBgEFBQcwAYYraHR0cDovL29jc3AuZ2xvYmFsc2lnbi5j
b20vZ3Nyc2FvdnNzbGNhMjAxODBWBgNVHSAETzBNMEEGCSsGAQQBoDIBFDA0MDIG
CCsGAQUFBwIBFiZodHRwczovL3d3dy5nbG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5
LzAIBgZngQwBAgIwCQYDVR0TBAIwADAnBgNVHREEIDAegg4qLmJuZGVzLmdvdi5i
coIMYm5kZXMuZ292LmJyMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAf
BgNVHSMEGDAWgBT473/yzXhnqN5vjySNiPGHAwKz6zAdBgNVHQ4EFgQUr8ZwKoFq
XqEty6FDsn6fsqeGm+kwggF9BgorBgEEAdZ5AgQCBIIBbQSCAWkBZwB1AKS5CZC0
GFgUh7sTosxncAo8NZgE+RvfuON3zQ7IDdwQAAABcD+gIlMAAAQDAEYwRAIgR/oA
SKJK0xqLbAJGCVnSP5IyLeXHkEYA9XsQGsISa3kCIBwZ4jMOyZYdZD7WzRF7Zq9G
/xxH9V8NzJcu5Sn6iKo5AHYAb1N2rDHwMRnYmQCkURX/dxUcEdkCwQApBo2yCJo3
2RMAAAFwP6AimgAABAMARzBFAiEAwo4mDeGUqOCWdgHBoPsjgq4RnjA2e/o4tSpb
dLWIzYUCIHUhbmk9jH8kx0W0t5SOLI/tBAJRyWlaC3GEAUSh5sW4AHYAVYHUwhaQ
NgFK6gubVzxT8MDkOHhwJQgXL6OqHQcT0wwAAAFwP6AiegAABAMARzBFAiEA+Lku
wDF2G9QAVuCSd85xFUkoAV8MO0Cv2nle4ZbzgeECIE6SdOMLinYiX4YUZzl/jzql
ZT3/XeNQ4XvCO5Fa7i9tMA0GCSqGSIb3DQEBCwUAA4IBAQBTn7kU8YF+N0uWrUJj
89vrq2OSXI8ShkimdziYNmciH9+Qvle1X/utcfng8SGa0xiSAcNSlEYRskq6D3pv
uSkXRO/9/r5+7WNRYE4wb/b1AbMQYINPqEd6SXW139Em7WPrq5M8nzzAXZ7Qy+ii
7cq4K7E0VPMCDsK948iUf+Nr7BBNlaD5J5/cWPm1p/EHi6pG6RUdTWTLnPjt40G9
6K7HivIvGkMq7HcEs2An+Y9yTmjzV1YhCIV/BzuFbc97z8vpfeF738K9N6bPkbFt
CcjkGVLQHiw0sld6uL75u+Z4gq8JFRd1OJFYT2EgJQFpl3zFQBVVuBMQivM9/QHO
xY6d
-----END CERTIFICATE-----
How do I configure Python 3.6 in CentOS so it access files in https://dadosabertos.bndes.gov.br without turning off SSL?
As specified in the document: https://docs.python-requests.org/en/latest/api/, the cert option is used to specify the client cert instead of server cert. Client cert is what you (as client) provides to the web server, so it believes who you are, thus it's not what you need. Instead, server cert is what https://dadosabertos.bndes.gov.br provides for you so you know that you are talking to the real website. It's provided by the server during the SSL handshake so you don't need to mannually speicify it. Question here is that requests fail to validate the server cert.
I try to reproduce your result in Docker centos environment but it works without any problems. requests use root certificates provided by the certifi package. It's possible that your certifi package is out-of-data. So I guess you may uninstall certifi and requests package and reinstall to get a latest copy of trusted root certificates.
My requests and dependencies versions:
certifi-2021.10.8
charset-normalizer-2.0.7
idna-3.3
requests-2.26.0
urllib3-1.26.7
Sitting behind a very strict firewall with SSL decryption, I usually install python packages (on macOS 10.15.) with these options pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org <packagename>. But pipenv install --trusted-host pypi.org --trusted-host files.pythonhosted.org <packagename> doesn't work:
pipenv.vendor.requirementslib.exceptions.RequirementError: Failed parsing requirement from '--trusted-host'
Since ignoring SSL didn't work, I tried to place the certificate of the firewall into a folder and set REQUESTS_CA_BUNDLE=/path/to/company/certificates.pem but without success (maybe I did it wrong). User #Shanti made a promising comment in this question, but I don't know how he accomplished feeding the certificate to pipenv.
So on the bottom line I am looking for a way to make pipenv use my firewall's certificate.
EDIT: here's the output when running pipenv install:
Creating a virtualenv for this project…
Pipfile: /Users/admin/Code/test/Pipfile
Using /Users/admin/.pyenv/versions/3.8.6/bin/python3.8 (3.8.6) to create virtualenv…
⠧ Creating virtual environment...created virtual environment CPython3.8.6.final.0-64 in 404ms
creator CPython3Posix(dest=/Users/admin/.local/share/virtualenvs/test-NSydZlln, clear=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/admin/Library/Application Support/virtualenv)
added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
✔ Successfully created virtual environment!
Virtualenv location: /Users/admin/.local/share/virtualenvs/test-NSydZlln
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✘ Locking Failed!
Traceback (most recent call last):
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/contrib/pyopenssl.py", line 488, in wrap_socket
cnx.do_handshake()
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1934, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1671, in _raise_ssl_error
_raise_current_error()
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 976, in _validate_conn
conn.connect()
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/connection.py", line 361, in connect
self.sock = ssl_wrap_socket(
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/util/ssl_.py", line 377, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/contrib/pyopenssl.py", line 494, in wrap_socket
raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 724, in urlopen
retries = retries.increment(
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/urllib3/util/retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/wheel/json (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in <module>
main()
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
_main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
results, resolver = resolve(
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
return resolve_deps(
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/utils.py", line 1096, in actually_resolve_deps
results = resolver.clean_results()
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/utils.py", line 1002, in clean_results
collected_hashes = self.collect_hashes(ireq)
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/utils.py", line 885, in collect_hashes
r = session.get(pkg_url, timeout=10)
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/requests/sessions.py", line 543, in get
return self.request('GET', url, **kwargs)
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/Users/admin/.pyenv/versions/3.8.6/lib/python3.8/site-packages/pipenv/vendor/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/wheel/json (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
As already stated in the comments, setting the environment variable would solve the problem. It should look like this:
export REQUESTS_CA_BUNDLE=/path/to/certificates.pem
Complete Chain
In certificates.pem you must have a complete chain that includes the root certificate. Therefore certificates.pem should look like this:
-----BEGIN CERTIFICATE-----
MII...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MII...
-----END CERTIFICATE-----
...
You can split the file into single files with suffix .pem including the begin and end marker like so:
-----BEGIN CERTIFICATE-----
MII...
-----END CERTIFICATE-----
In Finder you can now select the individual .pem files, enter <alt> + <tab> so that you can see the contents of each certificate. The chain must be complete, e.g. you should find the corresponding signing certificate for each certificate when you look in the 'Issuer' section under 'Common Name'.
If one or more are missing, use the Keychain Access application (/Applications/Utilities/) to search for the certificate with the missing 'Common Name', export the cert in .PEM format and simply append the resulting file to the end of your certificates.pem file.
Test
Locally tested like this:
setting a HTTPS proxy (in this case Charles)
save the Charles certificate in a .pem file
try to call pipenv install requests (or any other package), it fails with a SSLCertVerificationError
set REQUESTS_CA_BUNDLE environment variable
call pipenv install requests again -> works
Screenshot
I'm trying to deploy code to a server to which I don't have root access. So the solution I was thinking was to deploy using pyinstaller and staticx.
My code runs in python 3.7, in a nutshell, does something like:
import requests
response = requests.get('http://example.com/api/action')
# Do something with the response
When I run it in my environment, even after "building" with pyinstaller and staticx, it works flawlessly.
However, when I try to deploy it (the server is running Red Hat Enterprise Linux Server release 7.4, which does not have python 3), I get:
Traceback (most recent call last):
File "site-packages/urllib3/connection.py", line 160, in _new_conn
File "site-packages/urllib3/util/connection.py", line 57, in create_connection
File "socket.py", line 748, in getaddrinfo
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "site-packages/urllib3/connectionpool.py", line 603, in urlopen
File "site-packages/urllib3/connectionpool.py", line 355, in _make_request
File "http/client.py", line 1229, in request
File "http/client.py", line 1275, in _send_request
File "http/client.py", line 1224, in endheaders
File "http/client.py", line 1016, in _send_output
File "http/client.py", line 956, in send
File "site-packages/urllib3/connection.py", line 183, in connect
File "site-packages/urllib3/connection.py", line 169, in _new_conn
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f912ca96f28>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "site-packages/requests/adapters.py", line 449, in send
File "site-packages/urllib3/connectionpool.py", line 641, in urlopen
File "site-packages/urllib3/util/retry.py", line 399, in increment
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='someserver.com', port=80): Max retries exceeded with url: /api/action (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f912ca96f28>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Everything works if I use IP addresses instead of domains. Unfortunately, it's not a possibility because I have to hit an proxy that uses hosts. Also IP addresses may change in the future.
The server resolves names perfectly fine if I use nslookup or ping.
Any ideas why this could be happening? Any alternatives on how I could deploy my code in a way that it works in the remote server may also be a valid answer. Note however:
I have no root access and requesting the installation of python3, any libs or any other package such as docker, etc would most likely be rejected by my customer
The server can resolve external names, such as google.com but does not have access to the outer internet (the code is ment to work on internal servers, with internal DNS such as someserver.mycustomer.example)
I'm running html.render() from requests_html library. It is trying to install chromium but I am getting an error
I already tried pip install --upgrade certifi with and without sudo
and got: Requirement already up-to-date: certifi in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (2019.6.16)
I tried also running /Applications/Python\ 3.6/Install\ Certificates command and got:
-- pip install --upgrade certifi
Requirement already up-to-date: certifi in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (2019.6.16)
-- removing any existing file or link
-- creating symlink to certifi certificate bundle
-- setting permissions
-- update complete
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
[Process completed]
This is the error I got:
[W:pyppeteer.chromium_downloader] start chromium download.
Download may take a few minutes.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 472, in wrap_socket
cnx.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1915, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1647, in _raise_ssl_error
_raise_current_error()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 603, in urlopen
chunked=chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 344, in _make_request
self._validate_conn(conn)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 843, in _validate_conn
conn.connect()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 370, in connect
ssl_context=context)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 355, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 478, in wrap_socket
raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "stocks.py", line 11, in <module>
r.html.render()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests_html.py", line 586, in render
self.browser = self.session.browser # Automatically create a event loop and browser
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests_html.py", line 730, in browser
self._browser = self.loop.run_until_complete(super().browser)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 473, in run_until_complete
return future.result()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests_html.py", line 714, in browser
self._browser = await pyppeteer.launch(ignoreHTTPSErrors=not(self.verify), headless=True, args=self.__browser_args)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyppeteer/launcher.py", line 311, in launch
return await Launcher(options, **kwargs).launch()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyppeteer/launcher.py", line 125, in __init__
download_chromium()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyppeteer/chromium_downloader.py", line 136, in download_chromium
extract_zip(download_zip(get_url()), DOWNLOADS_FOLDER / REVISION)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyppeteer/chromium_downloader.py", line 78, in download_zip
data = http.request('GET', url, preload_content=False)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/request.py", line 68, in request
**urlopen_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/request.py", line 89, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/poolmanager.py", line 326, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 670, in urlopen
**response_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 670, in urlopen
**response_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 670, in urlopen
**response_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 641, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /chromium-browser-snapshots/Mac/575458/chrome-mac.zip (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
Any suggestions please?
I found the answer here, and I quote:
This is a problem of urllib3 (1.25).
Workaround is to use older version of it:
pip install -U "urllib3<1.25"
from:
https://github.com/miyakogi/pyppeteer/issues/219
Reference: https://github.com/miyakogi/pyppeteer/issues/219
Configuration (spyder 4.0.1, python 3.7.3, win10)
Install pyppdf
pip install pyppdf
Then in the python script
import pyppdf.patch_pyppeteer
Output:
[W:pyppeteer.chromium_downloader] start patched secure https chromium download.
Download may take a few minutes.
100%|| 133194757/133194757 [01:12<00:00, 1838096.92it/s]
[W:pyppeteer.chromium_downloader]
chromium download done.
[W:pyppeteer.chromium_downloader] chromium extracted to: C:\Users\alexi\AppData\Local\pyppeteer\pyppeteer\local-chromium\575458
Unfortunately the (Human-understandable) details of the bug origin are not given
The best possible answer is given in this link
https://github.com/miyakogi/pyppeteer/issues/258
The issue seems to be with urlib3 trying to download and install chrome browser.
This requires certificate verification, which throws error otherwise.
For Anaconda setup below steps are recommended to fix this issue.
conda install -c conda-forge pypdf2.
This worked for me, and hope will work for others until pypdf library introduces new issue. Until then enjoy this fix.
I have written Robot Framework test scripts in .tsv format to test web-services/APIs. Everything was working fine until today (probably because of the new updates of Robot Framework) when I started to get the following error:
SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)
This error keeps popping up for the following code in a test script:
${headers}= Create Dictionary Content-Type application/json Accept application/json
RequestsKeywords.Get Request httpbin ${url} headers=${headers} //ERROR SHOWS FOR THIS STATEMENT
I did get a detailed traceback for this error which is as follows:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/RequestsLibrary/RequestsKeywords.py", line 298, in get_request
session, uri, params, headers, redir, timeout)
File "/Library/Python/2.7/site-packages/RequestsLibrary/RequestsKeywords.py", line 801, in _get_request
cookies=self.cookies)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 480, in get
return self.request('GET', url, **kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Python/2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/Library/Python/2.7/site-packages/requests/adapters.py", line 447, in send
raise SSLError(e, request=request)
My system configuration:
Mac OS X (10.11.3)
Python (2.7.10)
openssl (1.0.2f)
requests (2.9.1)
robotframework (3.0)
robotframework-httplibrary (0.4.2)
robotframework-requests (0.4.4)
robotframework-ride (1.5.2.1)
robotframework-sshlibrary (2.1.2)
pyOpenSSL (0.15.1)
How do I resolve this issue?
You're all up to date, so there are two possibilities:
The handshake isn't going smoothly because of a break in the trust chain. Start from the bottom and work up. Are you testing this on a local area network? Is the certificate up to date? Can you access the site from your location and others without an error? SSL labs showing anything up? What about firewall rules?
There's a bug in the recent update. OpenSSL reports in my experience are caused by connection issues rather than obscure software problems - this is the less likely of the two.
Try installing requests[security] instead of requests. It uses PyOpenSSL, which is better than OpenSSL: pip install requests[security] vs pip install requests: Difference