I have an Ubuntu VM on cloud, where I downloaded Kafka version 2.8.1 from the Official Kafka site and followed the instructions in Kafka's official quickstart guide.
I am using a python client to consume one of the topics that I created as part of the quickstart guide. When I run it on the VM, everything runs fine, however, when I run the same program on my local system, I get the below error
Unable connect to node with id 0: [Errno 8] nodename nor servname provided, or not known
Traceback (most recent call last):
...
...
File "/Path/python3.9/site-packages/aiokafka/client.py", line 547, in check_version
raise KafkaConnectionError(
kafka.errors.KafkaConnectionError: KafkaConnectionError: No connection to node with id 0
The python program I am using:
import asyncio
import aiokafka
async def consume(self):
consumer = aiokafka.AIOKafkaConsumer(
"quickstart-events", bootstrap_servers="IP:9092"
)
try:
await consumer.start()
async for msg in self.consumer:
print(
"consumed: ",
msg.topic,
msg.partition,
msg.offset,
msg.key,
msg.value,
msg.timestamp,
)
finally:
await consumer.stop()
asyncio.run(consume())
I have ensured that the necessary ports (9022) on Ubuntu is open -
I checked that I could telnet into port 9022 from my local system.
I am not sure what could be the reason that I am unable to access Kafka over internet. Am I missing something obvious?
Change the following attribute in config/server.properties to bootstrap server address you are using in your code.
advertised.listeners = PLAINTEXT://IP or FQDN:9092
Related
I am trying to create a simple client with pykafka. For this I need SSL certificates. The client runs under RHEL 7 and Python 3.6.x
It looks like the connection works but I don't get any feedback or data, only a black screen.
How can I check the connection or get error messages.
#!/usr/bin/scl enable rh-python36 -- python3
from pykafka import KafkaClient, SslConfig
config = SslConfig(cafile='key/root_ca.crt',
certfile='key/cert.crt',
keyfile='key/key.key',
password='xxxx')
client = KafkaClient(hosts="xxxxxxx:9093",
ssl_config=config)
print("topics", client.topics)
topic = client.topics['xxxxxx']
consumer = topic.get_simple_consumer(
consumer_group="yyyyy",
auto_offset_reset=OffsetType.EARLIEST,
reset_offset_on_start=False
)
for message in consumer:
if message is not None:
print (message.offset, message.value)
I am trying to connect to my Neo4j graph database server from a new machine. I can successfully connect from an older machine but do not wish to use the older one anymore.
I have reduced the problem to a simple script that returns an exception:
from neo4j.v1 import GraphDatabase, basic_auth
auth = basic_auth("username","password")
session = GraphDatabase.driver("bolt://remote.server:7687",auth=auth).session()
statement = """MATCH (a:Protein)
WHERE a.name={name}
RETURN a.Accession"""
tx = session.begin_transaction()
record = tx.run(statement,{'name':"ARCH_HUMAN"}).single()
print record['a.Accession']
session.close()
And the error message is:
File "Test.py", line 10, in <module>
tx = session.begin_transaction()
File "/home/username/anaconda2/lib/python2.7/site-packages/neo4j/v1/api.py", line 432, in begin_transaction
self._connect()
File "/home/username/anaconda2/lib/python2.7/site-packages/neo4j/v1/api.py", line 269, in _connect
self._connection = self._acquirer(access_mode)
File "/home/username/anaconda2/lib/python2.7/site-packages/neo4j/v1/direct.py", line 52, in acquire
raise ServiceUnavailable("Cannot acquire connection to {!r}".format(self.address))
neo4j.exceptions.ServiceUnavailable: Cannot acquire connection to Address(host='remote.server', port=7687)
Port 7687 is open (confirmed via netstat -tulpn and iptables -L), and neo4j is configured to listen to 0.0.0.0:7687. In addition, .neo4j/known_hosts contains an entry for host 0.0.0.0
What's strange is that I get a different error message (neo4j.exceptions.AuthError) if I break the authentication by using an incorrect password. So the connection is being made to check the password, but still I cannot connect with the correct auth.
What's going on?
I too had the same issue and turns out the driver was the issue.
I did some experiments and found out that the last driver that it works for is neo4j-driver==v1.1.0 but the next version neo4j-driver==v1.2.0 it stops working for some reason.
Try uncomment dbms.connectors.default_listen_address=0.0.0.0 And check this
# Bolt connector
dbms.connector.bolt.enabled=true
dbms.connector.bolt.tls_level=OPTIONAL
dbms.connector.bolt.listen_address=:7687
# HTTP Connector. There must be exactly one HTTP connector.
dbms.connector.http.enabled=true
dbms.connector.http.listen_address=:7474
# HTTPS Connector. There can be zero or one HTTPS connectors.
dbms.connector.https.enabled=true
dbms.connector.https.listen_address=:7473
local-host --->Aterm server (security server ) -----> target-machine(
I am trying to write a code in Python using Paramiko to first SSH from local-host to the target-machine. From the target-machine, I want to capture some outputs and store them locally either as a variable or as a file (havent got to that point yet). I found an example from stackoverflow where they talk about using nested SSH with paramiko, and I follow it but I get stuck here:
i need just reaching the target-machine
My code:
import paramiko
import sys
import subprocess
hostname = '10.10.10.1'
port = 22
username = 'mohamed.hosseny'
password ='Pass#1'
client = paramiko.Transport((hostname, port))
client.connect(username=username, password=password)
client.close()
but i found the below error message :
Traceback (most recent call last):
File "C:/Users/mohamed.hosseny/Desktop/Paramiko.py", line 13, in <module>
client = paramiko.Transport((hostname, port))
File "C:\Python27\lib\site-packages\paramiko\transport.py", line 332, in
__init__
'Unable to connect to {}: {}'.format(hostname, reason))
SSHException: Unable to connect to 10.10.10.1: [Errno 10060] A connection
attempt failed because the connected party did not properly respond after
a period of time, or established connection failed because connected host
has failed to respond
paramiko.Transport is a lower level API. Don't use it unless you have a good reason. Instead, you can use paramiko.SSHClient.
I have successfully used python-ldap to connect to a windows 2012 R2 server over ldaps in the past. The procedure I used for this was as follows:
python code:
import ldap
ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER)
ldap.set_option(ldap.OPT_DEBUG_LEVEL, 255)
ip = '<redacted>'
url = "%s://%s:%d" % ('ldaps', ip, 636)
ld = ldap.initialize(url)
ld.protocol_version = 3
ld.set_option(ldap.OPT_REFERRALS, ldap.OPT_OFF)
user = '<redacted>'
passwd = '<redacted>'
ld.simple_bind_s('<redacted>\%s' % user, passwd)
And on the windows server, I used the 'server manager' to add a 'AD CS' role, and created a root certificate. I do not care about verifying the certificate, just using some encryption. After creating the root certificate, LDAPS was enabled on the server, and this code runs without error.
Now, I have followed the exact same procedure on windows server 2016, and the results are not so nice. I have managed to get a few errors from the same script. Usually either 'A TLS packet with unexpected length was received.' or 'Error in the push function.'. I have searched for a few hours but I have not been able to find a solution. Does anyone know if extra steps are needed for configuration on the windows server, or if something about my script is incorrect?
The client I am testing with is using python 2.7 and ubuntu 14.04. pip2.7 has updated the python-ldap library to the latest version. Lere is an example of the failed script run:
ldap_create
ldap_url_parse_ext(ldaps://<redacted>:636)
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP <redacted>:636
ldap_new_socket: 3
ldap_prepare_socket: 3
ldap_connect_to_host: Trying <redacted>:636
ldap_pvt_connect: fd: 3 tm: -1 async: 0
TLS: can't connect: Error in the push function..
ldap_err2string
Traceback (most recent call last):
File "test_ldap.py", line 13, in <module>
ld.simple_bind_s('<redacted>\%s' % user, passwd)
File "/usr/local/lib/python2.7/dist-packages/ldap/ldapobject.py", line 228, in simple_bind_s
msgid = self.simple_bind(who,cred,serverctrls,clientctrls)
File "/usr/local/lib/python2.7/dist-packages/ldap/ldapobject.py", line 222, in simple_bind
return self._ldap_call(self._l.simple_bind,who,cred,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls))
File "/usr/local/lib/python2.7/dist-packages/ldap/ldapobject.py", line 108, in _ldap_call
result = func(*args,**kwargs)
ldap.SERVER_DOWN: {'info': 'Error in the push function.', 'errno': 104, 'desc': "Can't contact LDAP server"}
So it seems that this is one of those windows things which I just do not understand. After coming into work on the next day, the same code above just started working. It seems that windows server may require many hours in order to allow LDAPS to become available to connect to.
I have my RabbitMQ Server running on AWS EC2
I have run the producer and consumer code locally. It is working.
I am able to access the rabbitMQ management web app as well.
When I am trying to push data from my laptop to EC2
I am getting this error on this line:
connection = pika.BlockingConnection(pika.ConnectionParameters('xx.xx.xx.xx',5672,'/',credentials))
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 339, in init
self._process_io_for_connection_setup()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 374, in _process_io_for_connection_setup
self._open_error_result.is_ready)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 395, in _flush_output
raise exceptions.ConnectionClosed()
pika.exceptions.ConnectionClosed
where xx.xx.xx.xx: public IP address of my instance
Pls tell me, if I am using the correct parameters. What should be the IP address , virtual hostname.
I have checked the credentials , the user that I am using exists and it has the rights to access '/' virtual host
I have made the needed changes in the Security Groups.
This is a screenshot of it:
When I am running the same producer code from within the instance it's working properly. No Exceptions and the consumer is able to receive it as well.
This is my complete code for reference:
import pika
print("Start")
credentials=pika.PlainCredentials('manish','manish')#RabbitMQ user created on EC2
connection=pika.BlockingConnection(pika.ConnectionParameters('xx.xx.xx.xx',5672,'/',credentials))
channel = connection.channel()
channel.queue_declare(queue='hello')
channel.basic_publish(exchange='',
routing_key='hello',
body='Hello World!')
print(" [x] Sent 'Hello World!'")
connection.close()
I tried socket_timeout and it worked for me, you could try something like :
credentials = pika.PlainCredentials('username,'password')
connection = pika.BlockingConnection(pika.ConnectionParameters('hostname',port,'virtual host',credentials,**socket_timeout=10000**))