Summary
We are using a custom CA in our enviroment. I have imported the CA certificates in the trust store by downloading the Base64 certs and updating the trust store using update-ca-certificates.
I am able to run cURL queries to my REST API, however the request library fails with an SSL error when running the same.
What I have tried
I have tried specifying the root ca certificate file path to the library, but got the same error. How do I troubleshoot this issue? Setting verify to false is not an option.
Code Run
Works
curl -X GET https://api.me.com/admin/ -H 'Authorization: Token 4ae5'
Doesn't work
requests.get('https://api.me.com/admin/', headers={'Authorization': 'Token 4ae5'}, verify='/etc/ssl/certs/root.pem')
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 345, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 844, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.5/dist-packages/urllib3/util/ssl_.py", line 325, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.5/ssl.py", line 377, in wrap_socket
_context=self)
File "/usr/lib/python3.5/ssl.py", line 752, in __init__
self.do_handshake()
File "/usr/lib/python3.5/ssl.py", line 988, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib/python3.5/ssl.py", line 633, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
timeout=timeout
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3/dist-packages/requests/api.py", line 67, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 53, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 480, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 588, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 447, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
... requests.get(..., verify='/etc/ssl/certs/root.pem')
Given your description of using update-ca-certificates it looks like you are using Debian or similar (i.e. Ubuntu). In this case the relevant path should be /etc/ssl/certs/ca-certificates.crt.
The reason is that Python Requests uses certificates from the python-certifi package., not those of the underlying operating system.
certifi includes all CA certificates from Mozilla, and it is also relatively straightforward to add missing certificates to it.
See this post for details on adding certificates to certifi: https://stackoverflow.com/a/66111417/516699
The good thing is also that you do not tweak the SSL of the underlying system, just that of the Python environment your are working in.
Related
I'm trying to use Python requests to access a URL from https://dadosabertos.bndes.gov.br, but it fails in CentOS. It works fine in Windows.
Here is the error:
>>> import requests
>>> requests.__version__
'2.26.0'
>>> requests.get('https://dadosabertos.bndes.gov.br')Traceback (most recent call last): File "/opt/python3/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 696, in urlopen
self._prepare_proxy(conn)
File "/opt/python3/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 964, in _prepare_proxy
conn.connect()
File "/opt/python3/lib64/python3.6/site-packages/urllib3/connection.py", line 426, in connect
tls_in_tls=tls_in_tls,
File "/opt/python3/lib64/python3.6/site-packages/urllib3/util/ssl_.py", line 450, in ssl_wrap_socket
sock, context, tls_in_tls, server_hostname=server_hostname
File "/opt/python3/lib64/python3.6/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
_context=self, _session=session)
File "/usr/lib64/python3.6/ssl.py", line 776, in __init__
self.do_handshake()
File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/python3/lib64/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/python3/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/opt/python3/lib64/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='dadosabertos.bndes.gov.br', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/python3/lib64/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/opt/python3/lib64/python3.6/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/python3/lib64/python3.6/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/opt/python3/lib64/python3.6/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/opt/python3/lib64/python3.6/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='dadosabertos.bndes.gov.br', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))
My CentOS version: CentOS Linux release 8.4.2105. It works in Windows 10. I'm using requests lib version 2.26.0.
I tried to download the certificate from the site and validate with it using this command:
requests.get('https://dadosabertos.bndes.gov.br', verify=True,
cert='./bndes-gov-br.pem')
but got a similar exception. Here is the stack trace:
>>> requests.get('https://dadosabertos.bndes.gov.br', verify=True, cert='./bndes-gov-br.pem')
Traceback (most recent call last):
File "/home/xxxxx/lib/python3.7/site-packages/urllib3/connectionpool.py", line 594, in urlopen
self._prepare_proxy(conn)
File "/home/xxxxx/lib/python3.7/site-packages/urllib3/connectionpool.py", line 805, in _prepare_proxy
conn.connect()
File "/home/xxxxx/lib/python3.7/site-packages/urllib3/connection.py", line 344, in connect
ssl_context=context)
File "/home/xxxxx/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 338, in ssl_wrap_socket
context.load_cert_chain(certfile, keyfile)
ssl.SSLError: [SSL] PEM lib (_ssl.c:3854)
Note that this error happens also in Python 3.7.
I exported the certificate following these instructions
Initially I tried to configure my machine global certificates, but it looks like Python and Requests lib uses its own. Another question gave me a lot of valuable info to configure my certificate.
Since I couldn't make requests lib use my certificate, I believe there is an error in the downloaded certificate or in the validation lib.
Here is its contents of my bndes-gov-br.pem file downloaded using the browser (I got the same error trying with the complete certificate chain):
-----BEGIN CERTIFICATE-----
MIIGjzCCBXegAwIBAgIMdIDfTRbWNDjcygdHMA0GCSqGSIb3DQEBCwUAMFAxCzAJ
BgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMSYwJAYDVQQDEx1H
bG9iYWxTaWduIFJTQSBPViBTU0wgQ0EgMjAxODAeFw0yMDAyMTMxNzM3MDBaFw0y
MjAyMTMxNzM3MDBaMIGlMQswCQYDVQQGEwJCUjEXMBUGA1UECBMOUmlvIGRlIEph
bmVpcm8xFzAVBgNVBAcTDlJpbyBkZSBKYW5laXJvMQwwCgYDVQQLEwNBVEkxPTA7
BgNVBAoTNEJhbmNvIE5hY2lvbmFsIGRlIERlc2Vudm9sdmltZW50byBFY29ub21p
Y28gZSBTb2NpYWwxFzAVBgNVBAMMDiouYm5kZXMuZ292LmJyMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsqNBHzLfEWeYk5cxF+hT3ZV9Ki6u7WjGXOx4
c6HMB7tDrbyp8wbmaJPNo8yWDAJ0eL4N+QVJ6IG2rJ7DLU65+76qcv8iLG5OcsnZ
K9o1NfnEaNWIy8Vf0edO7bkalXD8YYf5QQMSZ+TqPIA3cJnFKibNTbqaBRbjvwF9
QBaCATZnl0xg3/kD2Wdjtzdrg0JXBcRcrDeQOV/22/O2JMjbjRpoMeuqR9O8OwfE
JTT3tJxTE6LWKSIZR8nc+rMLW4sqw+QZPGMdS85m9eStUrHxQUHEBpScAPN9fN4c
u2L0U51nedZgfHEfqyjYVCOY0zoVEv5MW0UV5+mbObcy2v/d5QIDAQABo4IDETCC
Aw0wDgYDVR0PAQH/BAQDAgWgMIGOBggrBgEFBQcBAQSBgTB/MEQGCCsGAQUFBzAC
hjhodHRwOi8vc2VjdXJlLmdsb2JhbHNpZ24uY29tL2NhY2VydC9nc3JzYW92c3Ns
Y2EyMDE4LmNydDA3BggrBgEFBQcwAYYraHR0cDovL29jc3AuZ2xvYmFsc2lnbi5j
b20vZ3Nyc2FvdnNzbGNhMjAxODBWBgNVHSAETzBNMEEGCSsGAQQBoDIBFDA0MDIG
CCsGAQUFBwIBFiZodHRwczovL3d3dy5nbG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5
LzAIBgZngQwBAgIwCQYDVR0TBAIwADAnBgNVHREEIDAegg4qLmJuZGVzLmdvdi5i
coIMYm5kZXMuZ292LmJyMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAf
BgNVHSMEGDAWgBT473/yzXhnqN5vjySNiPGHAwKz6zAdBgNVHQ4EFgQUr8ZwKoFq
XqEty6FDsn6fsqeGm+kwggF9BgorBgEEAdZ5AgQCBIIBbQSCAWkBZwB1AKS5CZC0
GFgUh7sTosxncAo8NZgE+RvfuON3zQ7IDdwQAAABcD+gIlMAAAQDAEYwRAIgR/oA
SKJK0xqLbAJGCVnSP5IyLeXHkEYA9XsQGsISa3kCIBwZ4jMOyZYdZD7WzRF7Zq9G
/xxH9V8NzJcu5Sn6iKo5AHYAb1N2rDHwMRnYmQCkURX/dxUcEdkCwQApBo2yCJo3
2RMAAAFwP6AimgAABAMARzBFAiEAwo4mDeGUqOCWdgHBoPsjgq4RnjA2e/o4tSpb
dLWIzYUCIHUhbmk9jH8kx0W0t5SOLI/tBAJRyWlaC3GEAUSh5sW4AHYAVYHUwhaQ
NgFK6gubVzxT8MDkOHhwJQgXL6OqHQcT0wwAAAFwP6AiegAABAMARzBFAiEA+Lku
wDF2G9QAVuCSd85xFUkoAV8MO0Cv2nle4ZbzgeECIE6SdOMLinYiX4YUZzl/jzql
ZT3/XeNQ4XvCO5Fa7i9tMA0GCSqGSIb3DQEBCwUAA4IBAQBTn7kU8YF+N0uWrUJj
89vrq2OSXI8ShkimdziYNmciH9+Qvle1X/utcfng8SGa0xiSAcNSlEYRskq6D3pv
uSkXRO/9/r5+7WNRYE4wb/b1AbMQYINPqEd6SXW139Em7WPrq5M8nzzAXZ7Qy+ii
7cq4K7E0VPMCDsK948iUf+Nr7BBNlaD5J5/cWPm1p/EHi6pG6RUdTWTLnPjt40G9
6K7HivIvGkMq7HcEs2An+Y9yTmjzV1YhCIV/BzuFbc97z8vpfeF738K9N6bPkbFt
CcjkGVLQHiw0sld6uL75u+Z4gq8JFRd1OJFYT2EgJQFpl3zFQBVVuBMQivM9/QHO
xY6d
-----END CERTIFICATE-----
How do I configure Python 3.6 in CentOS so it access files in https://dadosabertos.bndes.gov.br without turning off SSL?
As specified in the document: https://docs.python-requests.org/en/latest/api/, the cert option is used to specify the client cert instead of server cert. Client cert is what you (as client) provides to the web server, so it believes who you are, thus it's not what you need. Instead, server cert is what https://dadosabertos.bndes.gov.br provides for you so you know that you are talking to the real website. It's provided by the server during the SSL handshake so you don't need to mannually speicify it. Question here is that requests fail to validate the server cert.
I try to reproduce your result in Docker centos environment but it works without any problems. requests use root certificates provided by the certifi package. It's possible that your certifi package is out-of-data. So I guess you may uninstall certifi and requests package and reinstall to get a latest copy of trusted root certificates.
My requests and dependencies versions:
certifi-2021.10.8
charset-normalizer-2.0.7
idna-3.3
requests-2.26.0
urllib3-1.26.7
Im trying to setup elasticsearch-curator (version 5.6.0) to delete indices in elasticsearch (version 7.3.1).
Theirs versions should be compatible (https://www.elastic.co/guide/en/elasticsearch/client/curator/current/version-compatibility.html).
Elasticseach is protected by SSL using self-signed certificate, so I need to turn off certificate verification.
This is my curator.yml conf:
client:
hosts:
- 127.0.0.1
port: 9201
url_prefix:
use_ssl: True
certificate: /opt/elastic-stack/curator/security/ca.crt
client_cert:
client_key:
ssl_no_validate: True
http_auth: curator:************
timeout: 30
master_only: False
logging:
loglevel: INFO
logfile: /var/log/elastic-stack/curator/curator.log
logformat: default
blacklist: ['elasticsearch', 'urllib3']
When I run
curator --config /opt/elastic-stack/curator/curator.yml /opt/elastic-stack/curator/actions.yml
Even though ssl_no_validate is set to True, I am getting:
/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/utils.py:53: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
return yaml.load(read_file(path))
/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py:175: UserWarning: Connecting to 127.0.0.1 using SSL with verify_certs=False is insecure.
% host
Traceback (most recent call last):
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
conn.connect()
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connection.py", line 394, in connect
ssl_context=context,
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 383, in ssl_wrap_socket
return context.wrap_socket(sock)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 814, in __init__
self.do_handshake()
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 1068, in do_handshake
self._sslobj.do_handshake()
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 217, in perform_request
method, url, body, retries=Retry(False), headers=request_headers, **kw
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 376, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
conn.connect()
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/connection.py", line 394, in connect
ssl_context=context,
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 383, in ssl_wrap_socket
return context.wrap_socket(sock)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 814, in __init__
self.do_handshake()
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 1068, in do_handshake
self._sslobj.do_handshake()
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/utils.py", line 899, in get_client
check_version(client)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/utils.py", line 685, in check_version
version_number = get_version(client)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/utils.py", line 658, in get_version
version = client.info()['version']['number']
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 245, in info
return self.transport.perform_request("GET", "/", params=params)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/transport.py", line 353, in perform_request
timeout=timeout,
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 226, in perform_request
raise SSLError("N/A", str(e), e)
elasticsearch.exceptions.SSLError: ConnectionError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)) caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/python36/python/opt/rh/rh-python36/root/usr/bin//curator", line 11, in <module>
sys.exit(cli())
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/cli.py", line 213, in cli
run(config, action_file, dry_run)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/cli.py", line 160, in run
client = get_client(**client_args)
File "/app/python36/python/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/curator/utils.py", line 906, in get_client
'Error: {0}'.format(e)
elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch. Error: ConnectionError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)) caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777))
I got successful response, when I run
curl -k -u curator:******** https://127.0.0.1:9201
Also kibana and logstash communicates with elasticsearch properly.
Does anyone knows, what might be the problem or how to get more info?
Edit 1:
unfortunately I dont have permissions to use yum, when I installed elasticsearch-curator-5.8.1-1.x86_64.rpm to my home dir using
cd {{ python_installation_dest }} && rpm2cpio ../elasticsearch-curator-5.8.1-1.x86_64.rpm| cpio -idmB
and then run curator, I got:
Fatal Python error: initfsencoding: Unable to get the locale encoding Traceback (most recent call last): File "/opt/python/3.7.4/lib/python3.7/encodings/__init__.py", line 31, in <module> zipimport.ZipImportError: can't decompress data; zlib not available
I dont use pythin 3.7.4, how can I change path to python?
The error appears to be tied to Python issues within RedHat and associated variants. As the official RPM version of Curator 5.8.x now bundles both its own version of Python 3.7.4 and its own up-to-date OpenSSL shared library (1.1.1c in Curator 5.8.1), you will get better results using the official RPM build.
I also faced a similar issue when I was using Curator 5.5.4 for my Elastic Search 5.6. The issue seems to be tied to some python dependency associated with Curator with versions below 5.8.
To fix this, I first installed Curator 5.8 which internally upgraded all the other Python dependencies of Curator. I then uninstalled Curator 5.8 which just removed Curator 5.8 and not its upgraded dependencies.
In the end, I installed Curator 5.5.4. It worked like a charm after that.
Just FYI, it has been working well on production.
pip install --no-cache-dir elasticsearch_curator==5.8 --user
pip uninstall elasticsearch_curator==5.8
pip install --no-cache-dir elasticsearch_curator==5.5.4 --user
I'm running html.render() from requests_html library. It is trying to install chromium but I am getting an error
I already tried pip install --upgrade certifi with and without sudo
and got: Requirement already up-to-date: certifi in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (2019.6.16)
I tried also running /Applications/Python\ 3.6/Install\ Certificates command and got:
-- pip install --upgrade certifi
Requirement already up-to-date: certifi in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (2019.6.16)
-- removing any existing file or link
-- creating symlink to certifi certificate bundle
-- setting permissions
-- update complete
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
[Process completed]
This is the error I got:
[W:pyppeteer.chromium_downloader] start chromium download.
Download may take a few minutes.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 472, in wrap_socket
cnx.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1915, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1647, in _raise_ssl_error
_raise_current_error()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 603, in urlopen
chunked=chunked)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 344, in _make_request
self._validate_conn(conn)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 843, in _validate_conn
conn.connect()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connection.py", line 370, in connect
ssl_context=context)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 355, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 478, in wrap_socket
raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "stocks.py", line 11, in <module>
r.html.render()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests_html.py", line 586, in render
self.browser = self.session.browser # Automatically create a event loop and browser
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests_html.py", line 730, in browser
self._browser = self.loop.run_until_complete(super().browser)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 473, in run_until_complete
return future.result()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests_html.py", line 714, in browser
self._browser = await pyppeteer.launch(ignoreHTTPSErrors=not(self.verify), headless=True, args=self.__browser_args)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyppeteer/launcher.py", line 311, in launch
return await Launcher(options, **kwargs).launch()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyppeteer/launcher.py", line 125, in __init__
download_chromium()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyppeteer/chromium_downloader.py", line 136, in download_chromium
extract_zip(download_zip(get_url()), DOWNLOADS_FOLDER / REVISION)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyppeteer/chromium_downloader.py", line 78, in download_zip
data = http.request('GET', url, preload_content=False)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/request.py", line 68, in request
**urlopen_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/request.py", line 89, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/poolmanager.py", line 326, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 670, in urlopen
**response_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 670, in urlopen
**response_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 670, in urlopen
**response_kw)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/connectionpool.py", line 641, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /chromium-browser-snapshots/Mac/575458/chrome-mac.zip (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
Any suggestions please?
I found the answer here, and I quote:
This is a problem of urllib3 (1.25).
Workaround is to use older version of it:
pip install -U "urllib3<1.25"
from:
https://github.com/miyakogi/pyppeteer/issues/219
Reference: https://github.com/miyakogi/pyppeteer/issues/219
Configuration (spyder 4.0.1, python 3.7.3, win10)
Install pyppdf
pip install pyppdf
Then in the python script
import pyppdf.patch_pyppeteer
Output:
[W:pyppeteer.chromium_downloader] start patched secure https chromium download.
Download may take a few minutes.
100%|| 133194757/133194757 [01:12<00:00, 1838096.92it/s]
[W:pyppeteer.chromium_downloader]
chromium download done.
[W:pyppeteer.chromium_downloader] chromium extracted to: C:\Users\alexi\AppData\Local\pyppeteer\pyppeteer\local-chromium\575458
Unfortunately the (Human-understandable) details of the bug origin are not given
The best possible answer is given in this link
https://github.com/miyakogi/pyppeteer/issues/258
The issue seems to be with urlib3 trying to download and install chrome browser.
This requires certificate verification, which throws error otherwise.
For Anaconda setup below steps are recommended to fix this issue.
conda install -c conda-forge pypdf2.
This worked for me, and hope will work for others until pypdf library introduces new issue. Until then enjoy this fix.
I have this plugin which I want to install, but somehow the installation always breaks due to the error:
urllib.error.URLError: <urlopen error [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:720)>
So I think the problem is maybe that the SSL version is too old, so I printed the version information out:
OpenSSL 0.9.8zh 14 Jan 2016
And after seeing some answers,enter link description here, enter link description here, enter link description here now I have the newest SSL on my mac, but it seems like this plugin is using some external python which is not installed on my disk, and this python always tries to use its own SSL. I found this python which blender uses, Show Package Contents -> Burrow down to Contents -> Resources -> 2.79 -> python, seems like it's python 3.5, and I have python 3.7 installed on the disk.
Here's the installation code of the plugin:
import bpy
import os
import addon_utils
from subprocess import call
from urllib.request import urlretrieve
from zipfile import ZipFile
from tempfile import TemporaryDirectory
from shutil import copytree,rmtree
from os.path import join
python_exec = bpy.app.binary_path_python
path_to_addons = bpy.utils.user_resource('SCRIPTS', "addons")
print('Install Pip')
try:
import pip
except:
rc = call([python_exec,"-m","ensurepip","--default-pip", "--upgrade"])
import pip
print('Download RD')
import ssl
print(ssl.OPENSSL_VERSION)
URL = "https://github.com/HBPNeurorobotics/BlenderRobotDesigner/archive/master.zip"
addon_dir = 'robot_designer_plugin'
zip_dir = "BlenderRobotDesigner-master"
print('Unzip RD')
with TemporaryDirectory() as tmp:
zip_file = join(tmp,"master.zip")
print(zip_file)
urlretrieve(URL,zip_file)
print('Downloaded!')
rc = call([python_exec,"-m","zipfile","-e",zip_file,tmp])
with ZipFile(zip_file, "r") as z:
z.extractall(tmp)
print('Unzip finished')
addon_dir_src = join(tmp,zip_dir,addon_dir)
addon_dir_dst = join(path_to_addons,addon_dir)
print('remove previous addon')
rmtree(addon_dir_dst,True)
print('add latest addon')
copytree(addon_dir_src,addon_dir_dst)
print('enable addon')
addon_utils.enable("robot_designer_plugin", persistent=True)
bpy.ops.wm.save_userpref()
with open(join(addon_dir_src,"requirements.txt")) as f:
for line in f:
rc = call([python_exec,"-m","pip","install",line])
#pip.main(['install', line])
print('RD Installation Done!')
Und this is the error raised in the terminal:
Install Pip
Download RD
OpenSSL 0.9.8zh 14 Jan 2016
Unzip RD
/var/folders/nm/9nfcg98x4hxf1kh08dj8p92h0000gn/T/tmpvd4k0lfi/master.zip
Traceback (most recent call last):
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/urllib/request.py", line 1254, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/http/client.py", line 1107, in request
self._send_request(method, url, body, headers)
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/http/client.py", line 1152, in _send_request
self.endheaders(body)
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/http/client.py", line 1103, in endheaders
self._send_output(message_body)
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/http/client.py", line 934, in _send_output
self.send(msg)
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/http/client.py", line 877, in send
self.connect()
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/http/client.py", line 1261, in connect
server_hostname=server_hostname)
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/ssl.py", line 385, in wrap_socket
_context=self)
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/ssl.py", line 760, in __init__
self.do_handshake()
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/ssl.py", line 996, in do_handshake
self._sslobj.do_handshake()
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/ssl.py", line 641, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:720)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/gaoyingqiang/Downloads/installer.blend/Text", line 40, in <module>
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/urllib/request.py", line 188, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/urllib/request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/urllib/request.py", line 466, in open
response = self._open(req, data)
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/urllib/request.py", line 484, in _open
'_open', req)
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/urllib/request.py", line 1297, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Applications/Blender/blender.app/Contents/Resources/2.79/python/lib/python3.5/urllib/request.py", line 1256, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:720)>
Error: Python script fail, look in the console for now...
I really don't know where goes wrong since the plugin provides no further imformation on this issue. How can I fix this?
New edition:
I now want to solve the issue of how to upgrade the openssl version in the python which is in the Blender. Blender brings python3.5 with it and how can I brew install openssl to this python but not the python on my disk?
Curling
$ curl https://10.0.0.2
$ curl: (60) Peer certificate cannot be authenticated with known CA certificates
Tried curling with a certificate
$ curl -E /opt/software/lib/python2.7/site-packages/certifi-2017.11.5-py2.7.egg/certifi/cacert.pem https://10.0.0.2
And this works. so it looks to have a valid .pem file. Now trying in a python script with the same .pem file
test_bb.py
#!/opt/software/bin/python2.7
import ssl
import urllib2
import certifi
BB_URL = "https://10.0.0.2"
mycafile = certifi.where()
print "pem: %s" % mycafile
urllib2.urlopen(BB_URL, cafile=mycafile)
OUTPUT
$ ./test_bb.py
pem: /opt/software/lib/python2.7/site-packages/certifi-2017.11.5-py2.7.egg/certifi/cacert.pem
Traceback (most recent call last):
File "./test_bb.py", line 14, in <module>
urllib2.urlopen(BB_URL, cafile=mycafile)
File "/opt/software/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/opt/software/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/opt/software/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/opt/software/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/opt/software/lib/python2.7/urllib2.py", line 1241, in https_open
context=self._context)
File "/opt/software/lib/python2.7/urllib2.py", line 1198, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)>
Why is it throwing SSL: CERTIFICATE_VERIFY_FAILED for python when curl worked with the same .pem file?
How can I fix the python script so it doesnt throw the error?