I'm trying to publish messages via python to a kafka topic and am receiving an error. I can connect and publish via the CLI. Hoping for some guidance. I've googled and read the docs. Thanks!!
Successful CLI command:
kafka-console-producer --broker-list
123.45.67.891:1234,123.45.67.892:1234,123.45.67.893:1234 --
producer.config C:\Users\fake_user\Kafka\client-ssl.properties --topic FakeTopic
Contents of client-ssl.properties:
security.protocol = SSL
ssl.truststore.location = C:/Users/fake_user/Kafka/kafka-truststore
ssl.truststore.password = fakepass
Code:
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers=['123.45.67.891:1234', '123.45.67.892:1234', '123.45.67.893:1234'],
security_protocol='SSL',
ssl_certfile=r'C:\Users\fake_user\Kafka\kafka-truststore',
ssl_password='fakepass')
producer.send('FakeTopic', value='python_test', key='test')
Resultant Error:
Traceback (most recent call last):
File "kafka_post_test.py", line 6, in <module>
ssl_password='fakepass')
File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kafka\producer\kafka.py", line 381, in __init__
**self.config)
File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kafka\client_async.py", line 239, in __init__
self.config['api_version'] = self.check_version(timeout=check_timeout)
File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kafka\client_async.py", line 874, in check_version
version = conn.check_version(timeout=remaining, strict=strict, topics=list(self.config['bootstrap_topics_filter']))
File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kafka\conn.py", line 1078, in check_version
if not self.connect_blocking(timeout_at - time.time()):
File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kafka\conn.py", line 331, in connect_blocking
self.connect()
File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kafka\conn.py", line 420, in connect
if self._try_handshake():
File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\site-packages\kafka\conn.py", line 496, in _try_handshake
self._sock.do_handshake()
File "C:\Users\fake_user\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line 1117, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1051)
Check out this link.
You have to add the SSL Cert to the JVM keystore for any pretty much any program that is run by Java.
I've found that by default, the python-kafka library, sets the ssl_cafile property to None. Setting it to the default os (/etc/pki/tls/cert.pem on linux) allowed me to connect to the kafka brokers.
https://kafka-python.readthedocs.io/en/master/_modules/kafka/producer/kafka.html#KafkaProducer.send
Related
I'm working on an implicit TLS connection program with Python ftplib. I tried the solution provided in question python-ftp-implicit-tls-connection-issue(including Rg Glpj's and Juan Moreno's answers) to make the connection. But when I call retrline or retrbinary after logging into the ftp server like this(FTP_ITLS is the subclass of FTP_TLS):
58 server = FTP_ITLS()
59 server.connect(host="x.x.x.x", port=990)
60 server.login(user="user", passwd="******")
61 server.prot_p()
62
63 server.cwd("doc")
64 print(server.retrlines('LIST'))
65 # server.retrbinary('RETR contents.7z', open('contents.7z', 'wb').write)
66 server.quit()
I got an EOF error:
Traceback (most recent call last):
File "D:/Coding/test/itls.py", line 64, in <module>
print(server.retrlines('LIST'))
File "D:\Python\Python27\lib\ftplib.py", line 735, in retrlines
conn = self.transfercmd(cmd)
File "D:\Python\Python27\lib\ftplib.py", line 376, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "D:\Python\Python27\lib\ftplib.py", line 713, in ntransfercmd
server_hostname=self.host)
File "D:\Python\Python27\lib\ssl.py", line 352, in wrap_socket
_context=self)
File "D:\Python\Python27\lib\ssl.py", line 579, in __init__
self.do_handshake()
File "D:\Python\Python27\lib\ssl.py", line 808, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:590)
As it seems ftplib uses PROTOCOL_SSLv23 as the default protocol in Python 2.7, I tried
PROTOCOL_TLSv1, PROTOCOL_TLSv1_1 and PROTOCOL_TLSv1_2, but none of them worked. And I also tried overriding ntransfercmd and auth, or setting ctx = ssl._create_stdlib_context(ssl.PROTOCOL_TLSv1) as Steffen Ullrich said in question connect-to-ftp-tls-1-2-server-with-ftplib, but the error never disappeared. What can I do then? Thanks.
I ran into this trying to connect to a FileZilla FTP server. FileZilla has a setting in the "FTP over TLS settings" called "Require TLS session resumption on data connection when using PROT P". Disabling this option fixed this problem.
If you don't have control over the server, check out FTPS with Python ftplib - Session reuse required which goes over how to enable session reuse. This seems to require Python 3.6+, however.
I want to connect to an FTPS server containing some not trusted certificate. When I use simple:
lftp -u user hostname
then after dir command there's an error:
ls: Fatal error: Certificate verification: Not trusted
The problem can be solved in lftp by executing the following command:
lftp -e "set ssl:verify-certificate false" -u user hostname
I'm trying to make the same connection in Python, using for example ftplib module:
import ftplib
ftp = ftplib.FTP_TLS()
ftp.connect(hostname, port)
ftp.login(username, password)
ftp.prot_p()
ftp.dir()
But it raises OSError exception:
Traceback (most recent call last):
File "/usr/lib/python3.8/code.py", line 90, in runcode
exec(code, self.locals)
File "<console>", line 1, in <module>
File "/usr/lib/python3.8/ftplib.py", line 558, in dir
self.retrlines(cmd, func)
File "/usr/lib/python3.8/ftplib.py", line 451, in retrlines
with self.transfercmd(cmd) as conn, \
File "/usr/lib/python3.8/ftplib.py", line 382, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "/usr/lib/python3.8/ftplib.py", line 783, in ntransfercmd
conn = self.context.wrap_socket(conn,
File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
OSError: [Errno 0] Error
The problem seems to be similar to te OSError during authenticating to an ftps server with ftplib.FTP_TLS so I also tried to use some other context, like:
import ssl
ctx = ssl._create_stdlib_context(ssl.PROTOCOL_TLSv1_2)
ftp = FTP_TLS(context=ctx)
or
ctx = ssl.ssl._create_unverified_context(ssl.PROTOCOL_TLSv1_2)
ftp = FTP_TLS(context=ctx)
But the error is still the same. Any ideas how to disable certificate verification?
It cannot be a certificate problem, as you are getting error only at dir. The connect succeeds.
You get a TLS error when opening FTP data connection. It quite possible that the root cause is that the server require TLS session resumption.
See FTPS with Python ftplib - Session reuse required.
I am using a private key authentication to connect to Snowflake using python,
**This is working successfully when connecting directly using Java Client
import snowflake.connector
import os
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives.asymmetric import dsa
from cryptography.hazmat.primitives import serialization
with open("rsa_key.p8", "rb") as key:
p_key= serialization.load_pem_private_key(
key.read(),
password='XXXXX'.encode(),
backend=default_backend()
)
pkb = p_key.private_bytes(
encoding=serialization.Encoding.DER,
format=serialization.PrivateFormat.PKCS8,
encryption_algorithm=serialization.NoEncryption())
conn = snowflake.connector.connect(
user=XXXXX,
password=XXXXXXX,
account=XXXXXXXXX,
private_key=pkb,
warehouse=XXX,
database=XXXXXX,
schema=XXXX
)
Have masked real values where needed, but these are correct as same as work direct with Java client.
Error:
/usr/lib/python3/dist-packages/jwt/algorithms.py:179: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
self.hash_alg()
Traceback (most recent call last):
File "tryconnection.py", line 37, in <module>
schema='PUBLIC'
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/__init__.py", line 53, in Connect
return SnowflakeConnection(**kwargs)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/connection.py", line 189, in __init__
self.connect(**kwargs)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/connection.py", line 493, in connect
self.__open_connection()
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/connection.py", line 710, in __open_connection
self.__authenticate(auth_instance)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/connection.py", line 963, in __authenticate
session_parameters=self._session_parameters,
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/auth.py", line 217, in authenticate
socket_timeout=self._rest._connection.login_timeout)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/network.py", line 530, in _post_request
_include_retry_params=_include_retry_params)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/network.py", line 609, in fetch
**kwargs)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/network.py", line 711, in _request_exec_wrapper
raise e
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/network.py", line 653, in _request_exec_wrapper
method, full_url, headers, data, conn)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/network.py", line 758, in _handle_unknown_error
u'errno': ER_FAILED_TO_REQUEST,
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/errors.py", line 100, in errorhandler_wrapper
connection.errorhandler(connection, cursor, errorclass, errorvalue)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/errors.py", line 73, in default_errorhandler
done_format_msg=errorvalue.get(u'done_format_msg'))
snowflake.connector.errors.OperationalError: 250003: None: Failed to get the response. Hanging? method: post, url:
Thank you in advance for your help.
I can not see the rest of the error message so I couldn't be sure. Are you sure enter your account name (and region and cloud if needed) as the account parameter, instead of full Snowflake URL?
https://docs.snowflake.com/en/user-guide/python-connector-example.html#connecting-to-snowflake
When connecting Jira, usually people uses JDBC connection string which includes full snowflake URL:
https://docs.snowflake.com/en/user-guide/python-connector-example.html#connecting-to-snowflake
I'm working on an implicit TLS connection program with Python ftplib. I tried the solution provided in question python-ftp-implicit-tls-connection-issue(including Rg Glpj's and Juan Moreno's answers) to make the connection. But when I call retrline or retrbinary after logging into the ftp server like this(FTP_ITLS is the subclass of FTP_TLS):
58 server = FTP_ITLS()
59 server.connect(host="x.x.x.x", port=990)
60 server.login(user="user", passwd="******")
61 server.prot_p()
62
63 server.cwd("doc")
64 print(server.retrlines('LIST'))
65 # server.retrbinary('RETR contents.7z', open('contents.7z', 'wb').write)
66 server.quit()
I got an EOF error:
Traceback (most recent call last):
File "D:/Coding/test/itls.py", line 64, in <module>
print(server.retrlines('LIST'))
File "D:\Python\Python27\lib\ftplib.py", line 735, in retrlines
conn = self.transfercmd(cmd)
File "D:\Python\Python27\lib\ftplib.py", line 376, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "D:\Python\Python27\lib\ftplib.py", line 713, in ntransfercmd
server_hostname=self.host)
File "D:\Python\Python27\lib\ssl.py", line 352, in wrap_socket
_context=self)
File "D:\Python\Python27\lib\ssl.py", line 579, in __init__
self.do_handshake()
File "D:\Python\Python27\lib\ssl.py", line 808, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:590)
As it seems ftplib uses PROTOCOL_SSLv23 as the default protocol in Python 2.7, I tried
PROTOCOL_TLSv1, PROTOCOL_TLSv1_1 and PROTOCOL_TLSv1_2, but none of them worked. And I also tried overriding ntransfercmd and auth, or setting ctx = ssl._create_stdlib_context(ssl.PROTOCOL_TLSv1) as Steffen Ullrich said in question connect-to-ftp-tls-1-2-server-with-ftplib, but the error never disappeared. What can I do then? Thanks.
I ran into this trying to connect to a FileZilla FTP server. FileZilla has a setting in the "FTP over TLS settings" called "Require TLS session resumption on data connection when using PROT P". Disabling this option fixed this problem.
If you don't have control over the server, check out FTPS with Python ftplib - Session reuse required which goes over how to enable session reuse. This seems to require Python 3.6+, however.
I have a Python SSL server that uses a self-signed certificate. I start my server like this:
httpd = BaseHTTPServer.HTTPServer(('', 443), MyHTTPHandler)
httpd.socket = ssl.wrap_socket (httpd.socket, certfile='server.pem', server_side=True, cert_reqs=0)
httpd.serve_forever()
I get the following error when I connect using Firefox:
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 51194)
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 655, in __init__
self.handle()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py", line 340, in handle
self.handle_one_request()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py", line 310, in handle_one_request
self.raw_requestline = self.rfile.readline(65537)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 734, in recv
return self.read(buflen)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 621, in read
v = self._sslobj.read(len or 1024)
SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1751)
----------------------------------------
I do not see this behavior with Chrome or another client. It only happens on the first connection (complaints about certificate) until I accept the certificate. The exception actually does not cause the program to quit.
Why am I getting an error on the server? How can I avoid the exception?
The TLv1 unknown CA alert is sent by some clients if they cannot verify the certificate of the server because it is signed by an unknown issuer CA. You can avoid this kind of exception if you use a certificate which is already trusted by the client or which can be validated against a root CA of the client (don't forget to include the chain certificates too).
If you cannot avoid this error (for instance because you are using a self-signed certificate) then you have to catch the exception and deal with it by closing the connection. You might need to do this by using handle_request to handle each request by itself and catch exceptions instead of using serve_forever.
I had the same error as you, even though I had a signed certificate from Sectigo. Turns out, you need a certificate chain rather than only your domain's certificate itself.
Source
While referring to this site, and the following video: https://www.youtube.com/watch?v=_YjX7rtiAsk
, I found that I need to create a new file called certificate-chain.pem, and manually/with the help of scripts concatenate (join) three certificates - domain certificate, CA certificate and the USERTrust certificate, one after the other.
Then, in the file you need, point to this certificate bundle/chain. This is a solution which I wept on for 7 hours.