I need to connect to Kafka broker in python using SASL/SCRAM-SHA-256 authentication.
I use the kafka-python (2.0.1) python library which allows this kind of connection/authentication.
This is an example of code, but i get the error ... not Brokers available
prod = KafkaProducer(bootstrap_servers='server:9093',
security_protocol='SASL_SSL',
ssl_certfile='path to ca in pem format',
sasl_mechanism='SCRAM-SHA-256',
sasl_plain_username='xxx',
sasl_plain_password='xxx')
Getting the following error
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
20-04-06 09:58:52 - root - 1 - 140658864072256 - MainThread - ERROR - Error in connection to kafka
Traceback (most recent call last):
File "/*****/*****/dao/kafka_dao.py", line 44, in __init__
**self.consumer_additional_params)
File "/opt/app-root/lib/python3.6/site-packages/kafka/consumer/group.py", line 355, in __init__
self._client = KafkaClient(metrics=self._metrics, **self.config)
File "/opt/app-root/lib/python3.6/site-packages/kafka/client_async.py", line 242, in __init__
self.config['api_version'] = self.check_version(timeout=check_timeout)
File "/opt/app-root/lib/python3.6/site-packages/kafka/client_async.py", line 907, in check_version
version = conn.check_version(timeout=remaining, strict=strict, topics=list(self.config['bootstrap_topics_filter']))
File "/opt/app-root/lib/python3.6/site-packages/kafka/conn.py", line 1228, in check_version
if not self.connect_blocking(timeout_at - time.time()):
File "/opt/app-root/lib/python3.6/site-packages/kafka/conn.py", line 337, in connect_blocking
self.connect()
File "/opt/app-root/lib/python3.6/site-packages/kafka/conn.py", line 426, in connect
if self._try_handshake():
File "/opt/app-root/lib/python3.6/site-packages/kafka/conn.py", line 505, in _try_handshake
self._sock.do_handshake()
File "/usr/lib64/python3.6/ssl.py", line 1033, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib64/python3.6/ssl.py", line 645, in do_handshake
self._sslobj.do_handshake()
Where am I doing wrong??
Related
#!/usr/local/bin/python3.7
#Producer.py
from kafka import KafkaProducer
kafkaBrokers='kafka1.xyz.com:443,kafka2.xyz.com:443,kafka3.xyz.com:443'
caRootLocation='CARoot.pem'
certLocation='certificate.pem'
keyLocation='key.pem'
topic='demo_topic'
password='something'
producer = KafkaProducer(bootstrap_servers=kafkaBrokers,
security_protocol='SSL',
ssl_check_hostname=True,
ssl_cafile=caRootLocation,
ssl_certfile=certLocation,
ssl_keyfile=keyLocation,
ssl_password=password)
# Send to a particular partition
producer.send(topic, bytes('Hello Kafka!','utf-8'),partition=0)
producer.flush()
Followed this approach below:
https://dev.to/adityakanekar/connecting-to-kafka-cluster-using-ssl-with-python-k2e
enter code here
-------------------- Error --------------------------------- Traceback (most recent call last): File "kafka_with_pem.py", line 20, in
ssl_password=password) File "/home/vdina01/.local/lib/python3.7/site-packages/kafka/producer/kafka.py",
line 383, in init
**self.config) File "/home/vdina01/.local/lib/python3.7/site-packages/kafka/client_async.py",
line 244, in init
self.config['api_version'] = self.check_version(timeout=check_timeout) File
"/home/vdina01/.local/lib/python3.7/site-packages/kafka/client_async.py",
line 909, in check_version
version = conn.check_version(timeout=remaining, strict=strict, topics=list(self.config['bootstrap_topics_filter'])) File
"/home/vdina01/.local/lib/python3.7/site-packages/kafka/conn.py", line
1238, in check_version
if not self.connect_blocking(timeout_at - time.time()): File "/home/vdina01/.local/lib/python3.7/site-packages/kafka/conn.py", line
340, in connect_blocking
self.connect() File "/home/vdina01/.local/lib/python3.7/site-packages/kafka/conn.py", line
401, in connect
self._wrap_ssl() File "/home/vdina01/.local/lib/python3.7/site-packages/kafka/conn.py", line
484, in _wrap_ssl
password=self.config['ssl_password']) ssl.SSLError: [SSL] PEM lib (_ssl.c:3845)
I would like to deploy my blog(based in python3.7 and django2.0) into server(centos6.) and I made a virtual environment(based in python3.7). after installing pypi-sever,I run pypi-server in terminal for testing whether it works. Unfortunately, this command does't work.I found many methods including update openssl,setuptools and soon,but they doesn't work.
Can anyone advise on what else can I try??
(pypi-env3) [root#edward pypi-env3]# pypi-server
[I 181029 20:18:20 __init__:29] Opening sqlite database: /root/virtual_path/pypi-env3/packages/metadata.db
[I 181029 20:18:20 __init__:35] Database initialized as '/root/virtual_path/pypi-env3/packages/metadata.db'. Checking migrations...
[I 181029 20:18:20 server:165] Init thread pool with 6 threads
[I 181029 20:18:20 server:194] Starting server http://127.0.0.1:8080/
[W 181029 20:18:25 iostream:1332] SSL Error on 9 ('151.101.108.223', 443): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)
[E 181029 20:18:25 ioloop:638] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7fc60131ad08>, <tornado.concurrent.Future object at 0x7fc6013928d0>)
Traceback (most recent call last):
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/ioloop.py", line 605, in _run_callback
ret = callback()
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/ioloop.py", line 626, in _discard_future_result
future.result()
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/gen.py", line 1063, in run
yielded = self.gen.throw(*exc_info)
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/pypi_server/cache.py", line 194, in wrap_gen
f = gen.throw(e)
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/pypi_server/handlers/pypi/proxy/client.py", line 49, in packages
(yield cls.XMLRPC.list_packages())
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/pypi_server/cache.py", line 189, in wrap_gen
res = yield f
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/gen.py", line 1063, in run
yielded = self.gen.throw(*exc_info)
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado_xmlrpc/client.py", line 111, in _remote_call
response = yield self.client.fetch(req)
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/stack_context.py", line 316, in wrapped
ret = fn(*args, **kwargs)
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/gen.py", line 289, in <lambda>
future, lambda future: callback(future.result()))
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/gen.py", line 1063, in run
yielded = self.gen.throw(*exc_info)
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/tcpclient.py", line 189, in connect
server_hostname=host)
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/root/virtual_path/pypi-env3/lib/python3.7/site-packages/tornado/iostream.py", line 1315, in _do_ssl_handshake
self.socket.do_handshake()
File "/usr/local/lib/python3.7/ssl.py", line 1108, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)
I have a file (it's called conversation_template.py) which I use to seed my DynamoDB database, and it worked until very recently.
import boto3
dynamodb = boto3.resource('dynamodb', region_name='us-east-1')
table = dynamodb.Table('tablename')
data = {'<data_goes_here>'}
def add_to_table():
table.put_item(Item=data)
add_to_table()
Now, when I run it in my command line, I get this:
Traceback (most recent call last):
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
body=body, headers=headers)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
self._validate_conn(conn)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 761, in _validate_conn
conn.connect()
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connection.py", line 238, in connect
ssl_version=resolved_ssl_version)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/util/ssl_.py", line 279, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/ssl.py", line 401, in wrap_socket
_context=self, _session=session)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/ssl.py", line 808, in __init__
self.do_handshake()
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/ssl.py", line 1061, in do_handshake
self._sslobj.do_handshake()
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/ssl.py", line 683, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:748)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/vendored/requests/adapters.py", line 370, in send
timeout=timeout
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py", line 574, in urlopen
raise SSLError(e)
botocore.vendored.requests.packages.urllib3.exceptions.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:748)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "conversation_template.py", line 74, in <module>
add_to_table()
File "conversation_template.py", line 71, in add_to_table
table.put_item(Item=episode)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/client.py", line 312, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/client.py", line 588, in _make_api_call
operation_model, request_dict)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/endpoint.py", line 141, in make_request
return self._send_request(request_dict, operation_model)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/endpoint.py", line 170, in _send_request
success_response, exception):
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/endpoint.py", line 249, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/endpoint.py", line 204, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/Users/user/anaconda/envs/django_pocketcoach/lib/python3.6/site-packages/botocore/vendored/requests/adapters.py", line 431, in send
raise SSLError(e, request=request)
botocore.vendored.requests.exceptions.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:748)
I tried to figure out where the error comes from but I couldn’t do it. Even the error messages didn’t get me anywhere.
I get an SSLError; the protocol is unknown. I found that this type of error could be related to me trying to connect to proxy servers but I am not using one and haven’t done so in the past either.
Any ideas on where this error could come from?
What's worked in the end is to exit my current virtual environment and run:
/Applications/Python\ 3.6/Install\ Certificates.command
And then I entered my virtual environment again and ran:
pip install certifi
That fixed it.
I got help from a friend who suspected it could have to do with the certification verification (related to this: urllib and "SSL: CERTIFICATE_VERIFY_FAILED" Error ) but it remains unclear why I didn't get a "Certificate Verification Failed" error.
Question I want to be able to connect to and download a file from my remote FTP filezilla. I am able to connect however as soon as it executes "ftp.retrlines('LIST')" I get an error as described below. Any help would be greatly appreciated!!!
Here is my script
from ftplib import FTP_TLS
ftp = FTP_TLS('111.111.111.111) # Hidden IP for my FTP SERVER
ftp.login('ftpuser', 'ftppassword12345')
ftp.prot_p() # securing data connection explicitly
ftp.cwd('/') # change working directory to the root
ftp.retrlines('LIST') # This is where it fails
Here is the error
Traceback (most recent call last):
File "ftps_.py", line 7, in <module>
ftp.retrlines('LIST')
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ftplib.py", line 735, in retrlines
conn = self.transfercmd(cmd)
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ftplib.py", line 376, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ftplib.py", line 713, in ntransfercmd
server_hostname=self.host)
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 352, in wrap_socket
_context=self)
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 579, in __init__
self.do_handshake()
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 808, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:590)
I am working on a simple script that calls Instagram API.
Error -
File "C:\Python32\lib\ssl.py", line 168, in wrap_socket
_context=self)
File "C:\Python32\lib\ssl.py", line 255, in __init__
raise x
File "C:\Python32\lib\ssl.py", line 251, in __init__
self.do_handshake()
File "C:\Python32\lib\ssl.py", line 430, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [Errno 8] _ssl.c:392: EOF occurred in violation of protocol