Develop Reference

Python Kafka client cannot connect to remote Kafka server

I have an Ubuntu VM on cloud, where I downloaded Kafka version 2.8.1 from the Official Kafka site and followed the instructions in Kafka's official quickstart guide.
I am using a python client to consume one of the topics that I created as part of the quickstart guide. When I run it on the VM, everything runs fine, however, when I run the same program on my local system, I get the below error
Unable connect to node with id 0: [Errno 8] nodename nor servname provided, or not known
Traceback (most recent call last):
...
...
File "/Path/python3.9/site-packages/aiokafka/client.py", line 547, in check_version
raise KafkaConnectionError(
kafka.errors.KafkaConnectionError: KafkaConnectionError: No connection to node with id 0
The python program I am using:
import asyncio
import aiokafka
async def consume(self):
consumer = aiokafka.AIOKafkaConsumer(
"quickstart-events", bootstrap_servers="IP:9092"
)
try:
await consumer.start()
async for msg in self.consumer:
print(
"consumed: ",
msg.topic,
msg.partition,
msg.offset,
msg.key,
msg.value,
msg.timestamp,
)
finally:
await consumer.stop()
asyncio.run(consume())
I have ensured that the necessary ports (9022) on Ubuntu is open -
I checked that I could telnet into port 9022 from my local system.
I am not sure what could be the reason that I am unable to access Kafka over internet. Am I missing something obvious?

Change the following attribute in config/server.properties to bootstrap server address you are using in your code.
advertised.listeners = PLAINTEXT://IP or FQDN:9092

Google Cloud Speech to Text Audio Timeout Error when used with Twilio "Stream" verb and Websocket

I'm currently trying to make a system that can transcribe a phone call in real time and then display the conversation in my command line. To do this, im using a Twilio phone number which sends out a http request when called. Then using Flask, Ngrok and Websockets to compile my server code, make my local port public and to transfer the data, the TwiML verb "Stream" is used to stream the audio data to the Google Cloud Speech-Text API. I have so far used Twilio's python demo on GitHub (https://github.com/twilio/media-streams/tree/master/python/realtime-transcriptions).
My server code:
from flask import Flask, render_template
from flask_sockets import Sockets
from SpeechClientBridge import SpeechClientBridge
from google.cloud.speech_v1 import enums
from google.cloud.speech_v1 import types
import json
import base64
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "./<KEY>.json"
HTTP_SERVER_PORT = 8080
config = types.RecognitionConfig(
encoding=enums.RecognitionConfig.AudioEncoding.MULAW,
sample_rate_hertz=8000,
language_code='en-US')
streaming_config = types.StreamingRecognitionConfig(
config=config,
interim_results=True)
app = Flask(__name__)
sockets = Sockets(app)
#app.route('/home')
def home():
return render_template("index.html")
#app.route('/twiml', methods=['POST'])
def return_twiml():
print("POST TwiML")
return render_template('streams.xml')
def on_transcription_response(response):
if not response.results:
return
result = response.results[0]
if not result.alternatives:
return
transcription = result.alternatives[0].transcript
print("Transcription: " + transcription)
#sockets.route('/')
def transcript(ws):
print("WS connection opened")
bridge = SpeechClientBridge(
streaming_config,
on_transcription_response
)
while not ws.closed:
message = ws.receive()
if message is None:
bridge.terminate()
break
data = json.loads(message)
if data["event"] in ("connected", "start"):
print(f"Media WS: Received event '{data['event']}': {message}")
continue
if data["event"] == "media":
media = data["media"]
chunk = base64.b64decode(media["payload"])
bridge.add_request(chunk)
if data["event"] == "stop":
print(f"Media WS: Received event 'stop': {message}")
print("Stopping...")
break
bridge.terminate()
print("WS connection closed")
if __name__ == '__main__':
from gevent import pywsgi
from geventwebsocket.handler import WebSocketHandler
server = pywsgi.WSGIServer(('', HTTP_SERVER_PORT), app, handler_class=WebSocketHandler)
print("Server listening on: http://localhost:" + str(HTTP_SERVER_PORT))
server.serve_forever()
streams.xml:
<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Say> Thanks for calling!</Say>
<Start>
<Stream url="wss://<ngrok-URL/.ngrok.io/"/>
</Start>
<Pause length="40"/>
</Response>
Twilio WebHook:
http://<ngrok-URL>.ngrok.io/twiml
Im am getting the following error when I run the server code and then call the Twilio number:
C:\Users\Max\Python\Twilio>python server.py
Server listening on: http://localhost:8080
POST TwiML
WS connection opened
Media WS: Received event 'connected': {"event":"connected","protocol":"Call","version":"0.2.0"}
Media WS: Received event 'start': {"event":"start","sequenceNumber":"1","start":{"accountSid":"AC8abc5aa74496a227d3eb489","streamSid":"MZe6245f23e2385aa2ea7b397","callSid":"CA5864313b4992607d3fe46","tracks":["inbound"],"mediaFormat":{"encoding":"audio/x-mulaw","sampleRate":8000,"channels":1}},"streamSid":"MZe6245f2397c1285aa2ea7b397"}
Exception in thread Thread-4:
Traceback (most recent call last):
File "C:\Users\Max\AppData\Local\Programs\Python\Python37\lib\site-packages\google\api_core\grpc_helpers.py", line 96, in next
return six.next(self._wrapped)
File "C:\Users\Max\AppData\Local\Programs\Python\Python37\lib\site-packages\grpc\_channel.py", line 416, in __next__
return self._next()
File "C:\Users\Max\AppData\Local\Programs\Python\Python37\lib\site-packages\grpc\_channel.py", line 689, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.OUT_OF_RANGE
details = "Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time."
debug_error_string = "{"created":"#1591738676.565000000","description":"Error received from peer ipv6:[2a00:1450:4009:807::200a]:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time.","grpc_status":11}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\Max\AppData\Local\Programs\Python\Python37\lib\threading.py", line 917, in _bootstrap_inner
self.run()
File "C:\Users\Max\AppData\Local\Programs\Python\Python37\lib\threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\Max\Python\Twilio\SpeechClientBridge.py", line 37, in process_responses_loop
for response in responses:
File "C:\Users\Max\AppData\Local\Programs\Python\Python37\lib\site-packages\google\api_core\grpc_helpers.py", line 99, in next
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.OutOfRange: 400 Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time.
Media WS: Received event 'stop': {"event":"stop","sequenceNumber":"752","streamSid":"MZe6245f2397c125aa2ea7b397","stop":{"accountSid":"AC8abc5aa74496a60227d3eb489","callSid":"CA5842bc6431314d502607d3fe46"}}
Stopping...
WS connection closed
I cant work out why im getting the audio timeout error? Is it a firewall issue with Twilio and Google? An encoding issue?
Any help would be greatly appreciated.
System:
Windows 10
Python 3.7.1
ngrok 2.3.35
Flask 1.1.2

As your streams.xml returned socket url "wss://<ngrok-URL/.ngrok.io/", please make sure it matches with your routing (e.g. #sockets.route('/'))
If your socket starting with '/', then your should rewrite the streams.xml, see below as an example.
<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Say> Thanks for calling!</Say>
<Start>
<Stream url="wss://YOUR_NGROK_ID.ngrok.io/"/>
</Start>
<Pause length="40"/>
</Response>

I ran some tests on this to try to establish what was happening. I put a timer over the
bridge = SpeechClientBridge(
streaming_config,
on_transcription_response)
section of code and found that it was taking ~10.9s to initialize. I believe the google API has a timeout of 10s. I tried running this on my google cloud instance which has more oomph than my laptop and it works perfectly well. Either this, or there are some different versions of libraries/code etc installed on the GCP instance, which I need to check.

This is related to gevent (used by flask_sockets) and grpc (used by google cloud speech) conflict described in this issue https://github.com/grpc/grpc/issues/4629
the solution is to add the following code
import grpc.experimental.gevent as grpc_gevent
grpc_gevent.init_gevent()

How to run a Python Discord bot on a cPanel (o2switch)

I started to host my Discord bot on a cPanel (o2switch), but I don't know how to run the bot. I have to define an entry point for my app, but I don't know what it should be.
I tried to set it as a function that just returns "Launched!", but that doesn't work.
# imports
def application():
return "Launched!"
# bot code
Does anyone know what code should I add for my bot runs?
EDIT: Added "runner" thing. The bot still doesn't launch, but I have this log:
App 16078 output: /opt/passenger-5.3.7-5.el7.cloudlinux/src/helper-scripts/wsgi-loader.py:26: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
App 16078 output: import sys, os, re, imp, threading, signal, traceback, socket, select, struct, logging, errno
App 16078 output: [ pid=16078, time=2020-01-15 16:18:24,002 ]: PyNaCl is not installed, voice will NOT be supported
App 16078 output: [ pid=16078, time=2020-01-15 16:18:24,033 ]: WSGI application raised an exception!
App 16078 output: Traceback (most recent call last):
App 16078 output: File "/opt/passenger-5.3.7-5.el7.cloudlinux/src/helper-scripts/wsgi-loader.py", line 199, in main_loop
App 16078 output: socket_hijacked = self.process_request(env, input_stream, client)
App 16078 output: File "/opt/passenger-5.3.7-5.el7.cloudlinux/src/helper-scripts/wsgi-loader.py", line 333, in process_request
App 16078 output: result = self.app(env, start_response)
App 16078 output: File "/home/bist1484/virtualenv/bot/3.7/lib/python3.7/site-packages/discord/client.py", line 598, in run
App 16078 output: return future.result()
App 16078 output: File "/home/bist1484/virtualenv/bot/3.7/lib/python3.7/site-packages/discord/client.py", line 579, in runner
App 16078 output: await self.start(*args, **kwargs)
App 16078 output: File "/home/bist1484/virtualenv/bot/3.7/lib/python3.7/site-packages/discord/client.py", line 542, in start
App 16078 output: await self.login(*args, bot=bot)
App 16078 output: TypeError: login() takes 2 positional arguments but 4 positional arguments (and 1 keyword-only argument) were given

I myself host my bot on cPanel. I'll help you with hosting your bot. Make sure your bot is in the home directory, with permissions set to 755.
You'll need a start script and a stop script. Make a new file in the cgi-bin of your public_html and you'll be able to start the bot at yourmaindomain.com/cgi-bin/startbot.py considering you name the start script to startbot.py. Place the following code in the start script :
#!/usr/bin/python3.6
import os, subprocess, signal
print("Content-Type: text/html\n\n")
counter = 0
p = subprocess.Popen(['ps', '-u', 'username'], stdout=subprocess.PIPE)
# must match your username --------^^^^^^^^
out, err = p.communicate()
for line in out.splitlines():
if 'heliobot.py'.encode('utf-8') in line:
# ^^^^^^^^^^^----- this has to match the filename of your bot script
counter += 1
print("Bot already running.")
if counter == 0:
subprocess.Popen("/home/username/heliobot.py")
# ^^^^^^^^-- be sure to update it to your username
print("Bot started!")
For the stop script, you can create a stopbot.py file in the same cgi-bin where you'll be able to stop the bot at yourmaindomain.com/cgi-bin/stopbot.py, place the following code in the script:
!/usr/bin/python3.6
import os, subprocess, signal
print("Content-Type: text/html\n\n")
counter = 0
p = subprocess.Popen(['ps', '-u', 'username'], stdout=subprocess.PIPE)
# must match your username --------^^^^^^^^
out, err = p.communicate()
for line in out.splitlines():
if 'heliobot.py'.encode('utf-8') in line:
# ^^^^^^^--- this has to match the filename of your loop
counter += 1
pid = int(line.split(None, 1)[0])
print("Stopping bot.")
os.kill(pid, signal.SIGTERM)
if counter == 0:
print("Already stopped.")
Replace the first line, that's the shebang, with the Python path of your hosting provider. Ensure that the modules used are installed, else request the host to install it for you. Also, ensure that the permissions of all these files are 755, otherwise you'll get Internal Server Errors.
Do remember to replace those parameters I've highlighted in the scripts. That's exactly how I've been hosting my bot on cPanel free hosting since I started developing. I never had money to get a VPS so this was the best and seemingly the only option for me. (I don't prefer Heroku and other application hosts for a variety of reasons). Hope that helps and solves your problem! If you need help with anything else, just comment it down and I'll try to help you out. :)
Regards,
Sayan Bhattacharyya.

cPanel is designed for web hosting, not for applications like Discord bots.
The application entry point is for web application frameworks that support WSGI. It does not apply to Discord bots.

You need to call Client.run. Specifically, it looks like you need to prepare a partial function that you can pass to this other application:
from functools import partial
from discord import Client
client = Client()
#client.event
async def on_message(message):
print(message.content)
runner = partial(client.run, "your token") # runner() then starts the bot

Python websockets: "Malformed HTTP message" when trying to connect to server in Docker container

I'm currently writing a server application which communicates to clients using websockets. I want to deploy the server in a Docker container. To test the connection, I wrote a client in Python. When running the server without Docker on my local system, the client can connect, send and receive messages without a problem. As soon as I run the server in a Docker container though, the client fails to connect with the following error message:
Traceback (most recent call last): File "./client-ws.py", line 40, in <module>
asyncio.get_event_loop().run_until_complete(loop(msg, sockaddr)) File "/usr/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
return future.result() File "./client-ws.py", line 11, in loop
async with websockets.connect(sockaddr) as sock: File "/usr/lib/python3.7/site-packages/websockets/py35/client.py", line 2, in __aenter__
return await self File "/usr/lib/python3.7/site-packages/websockets/py35/client.py", line 20, in __await_impl__
extra_headers=protocol.extra_headers, File "/usr/lib/python3.7/site-packages/websockets/client.py", line 283, in handshake
status_code, response_headers = yield from self.read_http_response() File "/usr/lib/python3.7/site-packages/websockets/client.py", line 92, in read_http_response
raise InvalidMessage("Malformed HTTP message") from exc websockets.exceptions.InvalidMessage: Malformed HTTP message
Here's my client's code:
async def loop(msg, sockaddr):
async with websockets.connect(sockaddr) as sock:
while True:
print("Sending position {}".format(i))
message = '{\"type\" : \"PositionUpdate\",\"position\": {\"latitude\": ' + msg[i][0] +',\"longitude\": ' + msg[i][1] + ',\"elevation\": ' + msg[i][2] + ',\"speed\": ' + msg[i][3] + '}}'
await sock.send(message)
print(await sock.recv())
i += 1
if i >= len(msg):
i = 0
time.sleep(1)
parser = argparse.ArgumentParser()
parser.add_argument('--host', help='The host to which the socket should bind', required=True)
parser.add_argument('--port', help='The port to which the socket should bind', required=True)
parser.add_argument('--file', help='Path to the coordinate resource file. The file should have the following structure: latitude,longitude,elevation,speed;latitude,longitude,...', required=True)
args = parser.parse_args()
file = open(args.file, 'r')
locationsStr = file.read()
locations = locationsStr.split(';')
msg = []
for l in locations:
tmp = l.split(',')
msg.append(tmp)
sockaddr = "ws://" + args.host + ":" + args.port
i = 0
asyncio.get_event_loop().run_until_complete(loop(msg, sockaddr))
On my server, which is written in C++, I use the library websocketpp to open a websocket. I can't post all of the server's code, but here is how I open the websocket:
broadcastServer::broadcastServer(SpatMessageHandler spatHandler, MapMessageHandler mapHandler,
ClearSpatCache spatCclear, ClearMapCache mapCclear) {
this->spatHandler = spatHandler;
this->mapHandler = mapHandler;
this->spatCclear = spatCclear;
this->mapCclear = mapCclear;
m_server.init_asio();
m_server.clear_access_channels(websocketpp::log::alevel::all);
m_server.set_open_handler(bind(&broadcastServer::onOpen, this, ::_1));
m_server.set_close_handler(bind(&broadcastServer::onClose, this, ::_1));
m_server.set_message_handler(bind(&broadcastServer::onMessage, this, ::_1, ::_2));
}
...
void broadcastServer::run(uint16_t port) {
m_server.listen(port);
m_server.start_accept();
m_server.run();
}
As far as I can judge, websocketpp binds to 0.0.0.0 by default. I run the docker container with the following command:
sudo docker run -p 9001:9001 -it $dockerhash
EDIT: I've just looked if something's wrong with the websocket in the container. Executing netstat -an inside the container shows, that the websocket didn't even bind to a port.

The issue was that I was running an older version of boost in my Docker container as on my local machine. I solved the issue by installing the boost headers from source.

Connection Error while sending message to RabbitMQ server on EC2

I have my RabbitMQ Server running on AWS EC2
I have run the producer and consumer code locally. It is working.
I am able to access the rabbitMQ management web app as well.
When I am trying to push data from my laptop to EC2
I am getting this error on this line:
connection = pika.BlockingConnection(pika.ConnectionParameters('xx.xx.xx.xx',5672,'/',credentials))
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 339, in init
self._process_io_for_connection_setup()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 374, in _process_io_for_connection_setup
self._open_error_result.is_ready)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 395, in _flush_output
raise exceptions.ConnectionClosed()
pika.exceptions.ConnectionClosed
where xx.xx.xx.xx: public IP address of my instance
Pls tell me, if I am using the correct parameters. What should be the IP address , virtual hostname.
I have checked the credentials , the user that I am using exists and it has the rights to access '/' virtual host
I have made the needed changes in the Security Groups.
This is a screenshot of it:
When I am running the same producer code from within the instance it's working properly. No Exceptions and the consumer is able to receive it as well.
This is my complete code for reference:
import pika
print("Start")
credentials=pika.PlainCredentials('manish','manish')#RabbitMQ user created on EC2
connection=pika.BlockingConnection(pika.ConnectionParameters('xx.xx.xx.xx',5672,'/',credentials))
channel = connection.channel()
channel.queue_declare(queue='hello')
channel.basic_publish(exchange='',
routing_key='hello',
body='Hello World!')
print(" [x] Sent 'Hello World!'")
connection.close()

I tried socket_timeout and it worked for me, you could try something like :
credentials = pika.PlainCredentials('username,'password')
connection = pika.BlockingConnection(pika.ConnectionParameters('hostname',port,'virtual host',credentials,**socket_timeout=10000**))