Develop Reference

How to handle failed connections with Flask-MQTT

I have a Flask-Mqtt client running, and everything is fine if broker is accessible. The problem starts when the MQTT broker is not accessible. Usually, I would be able to handle the failed connections with exceptions on the on_connect function, however I don't quite understand how to implement it with Flask-Mqtt
I would want to handle failed broker connections while allowing the web server to run the web pages.
Documentations and example of Flask-MQTT: https://flask-mqtt.readthedocs.io/en/latest/usage.html
The code for handling broker connection failure (based on Steve's Internet Guide)
def on_connect(client, userdata, flags, rc):
if rc==0:
client.connected_flag=True #set flag
print("connected OK")
else:
print("Bad connection Returned code=",rc)
client.bad_connection_flag=True
The error on Flask app when broker not connected:
C:\Users\USER\Documents\College\FYP1\flask_testing\venv\Scripts\python.exe C:/Users/USER/Documents/College/FYP1/flask_testing/main.py
Traceback (most recent call last):
File "C:/Users/USER/Documents/College/FYP1/flask_testing/main.py", line 37, in <module>
mqtt = Mqtt(app)
File "C:\Users\USER\Documents\College\FYP1\flask_testing\venv\lib\site-packages\flask_mqtt\__init__.py", line 104, in __init__
self.init_app(app)
File "C:\Users\USER\Documents\College\FYP1\flask_testing\venv\lib\site-packages\flask_mqtt\__init__.py", line 183, in init_app
self._connect()
File "C:\Users\USER\Documents\College\FYP1\flask_testing\venv\lib\site-packages\flask_mqtt\__init__.py", line 209, in _connect
res = self.client.connect(
File "C:\Users\USER\Documents\College\FYP1\flask_testing\venv\lib\site-packages\paho\mqtt\client.py", line 941, in connect
return self.reconnect()
File "C:\Users\USER\Documents\College\FYP1\flask_testing\venv\lib\site-packages\paho\mqtt\client.py", line 1075, in reconnect
sock = self._create_socket_connection()
File "C:\Users\USER\Documents\College\FYP1\flask_testing\venv\lib\site-packages\paho\mqtt\client.py", line 3546, in _create_socket_connection
return socket.create_connection(addr, source_address=source, timeout=self._keepalive)
File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\socket.py", line 807, in create_connection
raise err
File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\socket.py", line 796, in create_connection
sock.connect(sa)
socket.timeout: timed out

I faced a similar error. Everything was working fine when the mqtt server was running but if I started the app with the server switched off, the app failed with the same message than yours.
I finally solved it by using connect_async parameter when creating the MQTT object instance.
mqtt_client = Mqtt(app, connect_async=True)
With that parameter, the web pages will be displayed and, when the mqtt server is available, it will automatically connect.

Server Flask app with cheroot server results to error in HTTPServer.tick after each request

I'm trying to serve a Flask (v1.1.2) wsgi application using cheroot server of CherryPy (v18.6.0) and after each request executed via Postman or browser I'm getting the following exception in my console. I'm running python v3.8.5
Error in HTTPServer.tick
Traceback (most recent call last):
File "C:\myproject\venv\lib\site-packages\cheroot\server.py", line 1795, in serve
self.tick()
File "C:\myproject\venv\lib\site-packages\cheroot\server.py", line 2030, in tick
self.connections.expire()
File "C:\myproject\venv\lib\site-packages\cheroot\connections.py", line 107, in expire
for sock_fd, conn in timed_out_connections:
File "C:\myproject\venv\lib\site-packages\cheroot\connections.py", line 103, in <genexpr>
(sock_fd, conn)
File "C:\python\lib\_collections_abc.py", line 743, in __iter__
for key in self._mapping:
RuntimeError: dictionary changed size during iteration
Code as follows:
from cheroot.wsgi import Server
from flask import Flask
app = Flask(__name__)
#app.route("/", methods=["GET"])
def index():
return "Hello"
if __name__ == "__main__":
server = Server(bind_addr=("0.0.0.0", 3000), wsgi_app=app)
try:
server.start()
finally:
server.stop()
Any idea causing that exception and how we can resolve it?

This is a recent and acknowledged issue with cheroot, take a look to the cheroot GitHub Issue 312.

How to solve 'request that this server could not understand' error?

I have built an API with Flask and keep getting an error when testing it with postman.
I only started getting this error when I added threading so that I could continue running my scraper after returning data to postman, now I'm not sure how to get past this issue
My code looks something like this:
from threading import Thread
from flask import Flask
application = Flask(__name__)
class Compute(Thread):
def __init__(self, request):
print("init")
Thread.__init__(self)
self.request = request
def run(self):
print("RUN")
command = './webscraper.py -us "{user}" -p "{password}" -url "{url}"'.format(**self.request.json)
output = subprocess.call(['bash','-c', command])
print("done")
#application.route('/scraper/run', methods=['POST'])
def init_scrape():
thread_a = Compute(request.__copy__())
thread_a.start()
return jsonify({'Scraping this site: ': request.json["url"]}), 201
if __name__ == '__main__':
application.run(host="0.0.0.0", port="8080")
My POST data is just a site url and details to login to it, looks something like this
data = {
{
"user":"username",
"password":"password",
"url":"www.mysite.com/"
}
If I make a POST request to localhost:8080/scraper/run with postman I get this error:
init
RUN
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "api_app.py", line 19, in run
command = './portal_scrape.py -us "{user}" -p "{password}" -start {start} -end {end} -fav "{favourite}" -url "{url}"'.format(**self.request.json)
File "/home/connor/Desktop/portal_dl/venv36/lib/python3.6/site-packages/flask/wrappers.py", line 47, in json
return self.get_json()
File "/home/connor/Desktop/portal_dl/venv36/lib/python3.6/site-packages/flask/wrappers.py", line 71, in get_json
data = self._get_data_for_json(cache=cache)
File "/home/connor/Desktop/portal_dl/venv36/lib/python3.6/site-packages/flask/wrappers.py", line 50, in _get_data_for_json
return self.get_data(cache=cache)
File "/home/connor/Desktop/portal_dl/venv36/lib/python3.6/site-packages/werkzeug/wrappers.py", line 514, in get_data
rv = self.stream.read()
File "/home/connor/Desktop/portal_dl/venv36/lib/python3.6/site-packages/werkzeug/wsgi.py", line 1307, in read
return self.on_disconnect()
File "/home/connor/Desktop/portal_dl/venv36/lib/python3.6/site-packages/werkzeug/wsgi.py", line 1275, in on_disconnect
raise ClientDisconnected()
werkzeug.exceptions.ClientDisconnected: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand.
I am sending the same POST request I had used when it was working without threading

Python P2P sockets error "This operation would block forever"

Hello I'm trying to create a P2P program to distribute messages throgh the clients.
My server.py:
import requests
from flask import Flask, jsonify, request
import logging
from daemon import Daemon
import time
import argparse
import sys
from transaction import Transaction
import settings
import socket
import threading
bind_ip = '139.59.26.12'
bind_port = 9998
my_node_addr = str(bind_ip) + ":" + str(bind_port)
clients = [("0.0.0.0", 9999), ("0.0.0.0", 9998)]
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((bind_ip, bind_port))
server.listen(5) # max backlog of connections
print('Listening on {}:{}'.format(bind_ip, bind_port))
numbers = []
def broadcast_info(info):
for n in clients:
node_str_addr = str(n[0]) + ":" + str(n[1])
print("Selected: " + node_str_addr)
if node_str_addr != my_node_addr:
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect((n[0], n[1]))
client.send(info)
response = client.recv(4096)
def handle_client_connection(client_socket):
request = client_socket.recv(1024)
request_str = request.decode()
if request_str == 'print':
client_socket.send(str(numbers).encode('utf-8'))
print(numbers)
else:
client_socket.send('Added!'.encode('utf-8'))
client_socket.close()
num_rec = int(request.decode())
if num_rec not in numbers:
numbers.append(num_rec)
#broadcast_info(request)
while True:
client_sock, address = server.accept()
print('Accepted connection from {}:{}'.format(address[0], address[1]))
client_handler = threading.Thread(
target=handle_client_connection,
args=(client_sock,)
# without comma you'd get a... TypeError: handle_client_connection() argument after * must be a sequence, not _socketobject
)
client_handler.start()
And my client:
import socket
hostname, sld, tld, port = 'www', 'integralist', 'co.uk', 80
target = '{}.{}.{}'.format(hostname, sld, tld)
# create an ipv4 (AF_INET) socket object using the tcp protocol (SOCK_STREAM)
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# connect the client
# client.connect((target, port))
client.connect(('139.59.76.32', 9998))
# send some data (in this case a HTTP GET request)
client.send("2".encode('utf-8')) # Number or "print"
# receive the response data (4096 is recommended buffer size)
response = client.recv(4096)
print (response)
client.close()
Sometimes it works well. But then after a few tries this error appears:
Exception in thread Thread-11:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "server.py", line 43, in handle_client_connection
request = client_socket.recv(1024)
File "/usr/local/lib/python3.5/dist-packages/gevent/_socket3.py", line 336, in recv
self._wait(self._read_event)
File "/usr/local/lib/python3.5/dist-packages/gevent/_socket3.py", line 156, in _wait
self.hub.wait(watcher)
File "/usr/local/lib/python3.5/dist-packages/gevent/hub.py", line 651, in wait
result = waiter.get()
File "/usr/local/lib/python3.5/dist-packages/gevent/hub.py", line 898, in get
return self.hub.switch()
File "/usr/local/lib/python3.5/dist-packages/gevent/hub.py", line 630, in switch
return RawGreenlet.switch(self)
gevent.hub.LoopExit: ('This operation would block forever', <Hub at 0x7fdf21aa2af8 epoll pending=0 ref=0 fileno=9>)
I tried everything. In my local machine works fine. But when I try to do it on my Digitalocean VPS, then this happens. I've search about this error over the Internet but I can't find anything that helps me.
Do you know what can be the cause of this error?
Thanks

Python + Flask web app reports "[Errno 9] Bad file descriptor" with pexpect module

I am developing a web app using Python + Flask. In the simplest sense, a request from the client to a specific URL will trigger the app to login into a remote machine on the server side, perform a series of shell commands, and parse and send the output (formatted in JSON) to the client as part of the response. These commands are fairly simple. To log in to the remote machine, the only method available to me is rlogin, so I used the pexpect module since I couldn't find any standard Python modules for rlogin.
Now the problem is that while I am able to get the correct output from pexpect/rlogin, sending that output (a string) as a response results in an error:
----------------------------------------
Exception happened during processing of request from ('171.71.55.54', 62736)
Traceback (most recent call last):
File "/isan/python/python2.7/SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "/isan/python/python2.7/SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "/isan/python/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/isan/python/python2.7/SocketServer.py", line 651, in __init__
self.finish()
File "/isan/python/python2.7/SocketServer.py", line 710, in finish
self.wfile.close()
File "/isan/python/python2.7/socket.py", line 279, in close
self.flush()
File "/isan/python/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 9] Bad file descriptor
----------------------------------------
I've stripped the code down to the bare minimum required to reproduce the error. An HTTP request to this app results in the "Bad file descriptor" error. Sorry for the varying indentation, I was using two different editors to modify the code!
import os
import subprocess
import pexpect
from flask import Flask
app = Flask(__name__)
class rlogin:
def __init__(self, host, prompt):
self.child = pexpect.spawn("rlogin " + host)
self.prompt = prompt
self.child.expect_exact(self.prompt)
def command(self, command):
self.child.sendline(command)
self.child.expect_exact(self.prompt)
response = self.child.before
return response
def close(self):
self.child.close()
#app.route('/')
def index():
rl = rlogin("myserver", "root#myserver:~# ")
output = rl.command("pwd")
rl.close()
# The output of the next line is just as I expect:
print output
# This is probably where it fails:
return output
if __name__ == '__main__':
app.run(host='0.0.0.0', debug=True)
Removing the calls to rlogin(), rl.command() and rl.close(), and returning a simple string like "A" resolves the error. I've been stuck on this issue for a while now, and would hugely appreciate any help. Thanks!

You will have to make sure that the return type is str, unicode, response class or WSGI function. It looks like your output doesn't belong to any of the accepted Flask route return types.
#app.route('/')
def index():
rl = rlogin("myserver", "...")
output = rl.command("pwd")
rl.close()
print type(output)
# Convert output into something that flask can understand
value = str(output)
resp = make_response(value)
resp.headers['Content-Type'] = 'text/plain'
return resp
You can read more about this at http://flask.pocoo.org/docs/0.10/api/#flask.Flask.make_response