I am using confluent kafka python 'https://github.com/confluentinc/confluent-kafka-python' for writing application. Both kafka and schema registry is secured and uses https endpoints.
While running the application, i am getting following error
Result: Failure Exception: SSLError: HTTPSConnectionPool(host='hostname', port=443):
Max retries exceeded with url: //subjects/schema-value/versions (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
Question 1:
For connecting to schema registry, where to specify the ceritificate value ?
Question 2:
For testing, i want to disable SSL verification in python, What is the option to do that ?
Thanks in advance.
This is the config that I used for my avro producer:
avro_producer_conf = {
"bootstrap.servers": "SSL://127.0.0.1:9094",
"security.protocol": "ssl",
# Certificates used by simple Producer
"ssl.ca.location": "/ssl/root/intermediate/ca-chain.cert.pem",
"ssl.certificate.location": "/ssl/root/intermediate/producer/producer.cert.pem",
"ssl.key.location": "/ssl/root/intermediate/producer/producer.key.pem",
'schema.registry.url': "https://schemaregistry:8081",
# Certificates used by Schema Registry
"schema.registry.ssl.ca.location": "/ssl/root/intermediate/ca-chain.cert.pem",
"schema.registry.ssl.certificate.location": "/ssl/root/intermediate/producer/producer.cert.pem",
"schema.registry.ssl.key.location": "/ssl/root/intermediate/producer/producer.key.pem"
}
The AvroProducer __init__() method is doing the separation of parameters. Everything you want to pass to SchemaRegistry needs to start with schema.registry.<parameter>. To use SSL with Schema registry make sure you use a non-encrypted key(private key without password). Make sure you don't have REQUESTS_CA_BUNDLE environment variable set, it will confuse the library.
Related
I am trying to connect to the server doing:
nexus_config = nexuscli.nexus_config.NexusConfig(username=NEXUS_USER,
password=NEXUS_PASSWORD,
url=NEXUS_URL,
x509_verify=True)
nexus_client = nexuscli.nexus_client.NexusClient(config=nexus_config)
print(nexus_client.repositories.list)
But, I get the error:
nexuscli.exception.NexusClientConnectionError: HTTPSConnectionPool(host='my_site.net', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)')))
https://nexus3-cli.readthedocs.io/en/latest/nexuscli.html#module-nexuscli.nexus_config
checking the URL using the API endpoing with the requests library and setting validate=False I get a warning and does work.
So I change my code to: x509_verify=True hoping that the same from requests will happen that i will be getting warnings but instead i got.
Is there something I am missing? or maybe there is a bug in the library?
thanks guys.
nexuscli.exception.NexusClientAPIError: <exception str() failed>
You are getting the exception: NexusClientAPIError. According to the documents:
exception nexuscli.exception.NexusClientAPIError Bases: Exception
Unexpected response from Nexus service.
I think your code is fine and the issue with x.509 certificate is resolved and the problem might be on the server's side.
I use python-requests to talk to HTTPS web services, some of which present incomplete certificate X509 chains. I'm having trouble figuring out how to access the invalid/incomplete certificates in order to explain the error to the user.
Here's an example illustrated by https://ssllabs.com/ssltest, where the server sends only the leaf certificate, and not the intermediate certificate which is necessary for validation, but missing from certifi's root CA store:
When I try to connect with python-requests, I get an exception that isn't very useful:
request.get('https://web.service.com/path')
SSLError: HTTPSConnectionPool(host='web.service.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
Obviously, I can use separate tools to figure out what's wrong in any particular case (e.g. gnutls-cli, openssl s_client, SSLLabs, etc.).
However, what I really want to be able to do is to be able to catch and diagnose the problem with the certificate chain in my Python code, so that I can present a more specific error message to the user. This answer suggests a monkey-patch to the response object; it's not particularly elegant, but it works—though only when the response object is returned successfully, and not in the case of an exception.
What are the cleanest ways to instrument requests to save the peer's certificate chain in the exception object returned when requests fails to validate the certificate chain itself?
Take requests.get("https://151.101.1.69") # stackoverflow's ip as an example:
try:
requests.get("https://151.101.1.69")
except requests.exceptions.SSLError as e:
cert = e.args[0].reason.args[0]._peer_cert
Then cert is a dict contains the peer's certificate. As I'm not very familiar with SSL, I don't know if it is enough for your case.
BTW, in this case the error is "hostname '151.101.1.69' doesn't match either of '*.stackexchange.com', ...omitted. I'm not sure about the structure of exception in your real case, so you may need to find it on your own. I think it should have the same name _peer_cert.
update
The above method doesn't work when handshake fails... But it still can be done:
try:
requests.get("https://fpslinux1.finalphasesystems.com/")
except requests.exceptions.SSLError:
import ssl
import OpenSSL
cert = ssl.get_server_certificate(('fpslinux1.finalphasesystems.com', 443))
cert = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, cert)
print(cert.get_issuer())
print(cert.get_subject().get_components())
Yes it is a little dirty but I don't have a better method as a ssl socket doesn't
even return invalid certs from C level :/
To use OpenSSL, you need to install pyopenssl.
I'm trying to access a table on factfinder.census.gov via their API.
I've tried the solutions listed here and tried using http, all sorts of variants of this code, etc. In fact, I can't even seem to do a simple get (I'm using python requests) on http://factfinder.census.gov/ at all.
E.g.,
https://factfinder.census.gov/service/data/v1/en/programs/DEC/datasets/10_SF1/tables/GCTPH1/data/0100000US.04000
I had to tack on a user access key (from https://factfinder.census.gov/service/UserAccessKey.html) to get access, and then I could just paste https://factfinder.census.gov/service/data/v1/en/programs/DEC/datasets/10_SF1/tables/GCTPH1/data/0100000US.04000?key=MYKEY and it worked fine in private/incognito mode by just pasting it into my browser url bar)
However, I'm getting
Error
requests.exceptions.SSLError: HTTPSConnectionPool(host='factfinder.census.gov', port=443): Max retries exceeded with url: /service/data/v1/en/programs/DEC/datasets/10_SF1/tables/GCTPH1/data/0100000US.04000?key=MYKEY (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:598)'),))
I'd hazard a guess that you're querying the link you posted (http://factfinder.census.gov/) using http not https (notice if you click that link it redirects to https), the site uses SSL encryption and is rejecting the unencrypted connection attempt.
I keep getting the following error intermittently, i'm suspecting a proxy on the network is causing this, since i can run my python script when using a different connection.
I'm using python 2.7 and also using Fiddler to help with the proxy authentication.
SSLError: HTTPSConnectionPool(host='api.bogus.com', port=443): Max
retries exceeded with url: /api/v1/oauth2/token (Caused by
SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected
EOF')",),))
At the moment I am having some limited success with the following setup, it does fail quite often but it manages to work a few times. I am using a session with the following parameters:
def get_session():
session = requests.Session()
# limit connection pool and connection numbers
session.mount('https://', HTTPAdapter(max_retries=5, pool_connections=100, pool_maxsize=100))
# make sure all connections are closed
session.headers.update({"Connection": "close"})
return session
I've looked at and tried several stack solutions regarding the https requests using a proxy with python, and I've seen the discussions on GitHub.
My impression was that the requests library in python 3 now has support for https requests using a proxy and so I don't understand why mine doesn't work:
import requests
proxydict = {
'http':'http://xx.xx.xx.xxx:5555/',
'https':'https://xx.xx.xxx.xx:5555/'
}
requests.get('https://www.google.co.uk',proxies = proxydict).
When I run this code I get:
ConnectionError: HTTPSConnectionPool(host='www.python.org', port=443):
Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake:
Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),))
I was used a postman proxy to do this.
My http requests with a proxy work fine. Can someone help?
Edit:
I wanted to do this with a corporate proxy so I can't simply use a different proxy. Also, the https addresses work fine in the browser, it's simply when I do a python request with the proxy as described in the documentation that this error occurs.
Thanks.
Hi because your proxy is not working with ssl or your proxy is used try other
from here
https://www.sslproxies.org/
scroll down and you see https proxies