I'm trying to write a script that saves mqtt data and sends it to influxDB. The issue I'm having is that the callback function of the mqtt-paho module keeps giving the error:
AttributeError: 'Client' object has no attribute 'write_api'. I think this is because of the self in the internal 'Client' class of the mqtt-paho. My full script can be found below:
# Imported modules
# standard time module
from datetime import datetime
import time
# InfluxDB specific modules
from influxdb_client import InfluxDBClient, Point, WritePrecision
from influxdb_client.client.write_api import SYNCHRONOUS
#MQTT paho specific modules
import paho.mqtt.client as mqtt
class data_handler(): # Default namespaces are just for all the ESPs.
def __init__(self, namespace_list=["ESP01","ESP02","ESP03","ESP04","ESP05","ESP06","ESP07","ESP08"]):
# initialize influxdb client and define access token and data bucket
token = "XXXXXXXXXX" # robotlab's token
self.org = "Home"
self.bucket = "HomeSensors"
self.flux_client = InfluxDBClient(url="http://localhost:8086", token=token)
self.write_api = self.flux_client.write_api(write_options=SYNCHRONOUS)
# Initialize and establish connection to MQTT broker
broker_address="XXX.XXX.XXX.XXX"
self.mqtt_client = mqtt.Client("influx_client") #create new instance
self.mqtt_client.on_message=data_handler.mqtt_message #attach function to callback
self.mqtt_client.connect(broker_address) #connect to broker
# Define list of namespaces
self.namespace_list = namespace_list
print(self.namespace_list)
def mqtt_message(self, client, message):
print("message received " ,str(message.payload.decode("utf-8")))
print("message topic=",message.topic)
print("message qos=",message.qos)
print("message retain flag=",message.retain)
sequence = [message.topic, message.payload.decode("utf-8")]
self.write_api.write(self.bucket, self.org, sequence)
def mqtt_listener(self):
for namespace in self.namespace_list:
self.mqtt_client.loop_start() #start the loop
print("Subscribing to topics!")
message = namespace+"/#"
self.mqtt_client.subscribe(message, 0)
time.sleep(4) # wait
self.mqtt_client.loop_stop() #stop the loop
def main():
influxHandler = data_handler(["ESP07"])
influxHandler.mqtt_listener()
if __name__ == '__main__':
main()
The code works fine until I add self.someVariable in the callback function. What would be a good way to solve this problem? I don't really want to be making global variables hence why I chose to use a class.
Thanks in advance!
Dealing with self when there are multiple classes involved can get confusing. The paho library calls on_message as follows:
on_message(self, self._userdata, message)
So the first argument passed is the instance of Client so what you are seeing is expected (in the absence of any classes).
If the callback is a method object (which appears to be your aim) "the instance object is passed as the first argument of the function". This means your function would take four arguments and the definition be:
mqtt_message(self, client, userdata, msg)
Based upon this you might expect your application to fail earlier than it is but lets look at how you are setting the callback:
self.mqtt_client.on_message=data_handler.mqtt_message
datahandler is the class itself, not an instance of the class. This means you are effectively setting the callback to a static function (with no binding to any instance of the class - this answer might help). You need to change this to:
self.mqtt_client.on_message=self.mqtt_message
However this will not work as the method currently only takes three arguments; update the definition to:
def mqtt_message(self, client, userdata, msg)
with those changes I believe this will work (or at least you will find another issue :-) ).
An example might be a better way to explain this:
class mqtt_sim():
def __init__(self):
self._on_message = None
#property
def on_message(self):
return self._on_message
#on_message.setter
def on_message(self, func):
self._on_message = func
# This is what you are doing
class data_handler1(): # Default namespaces are just for all the ESPs.
def __init__(self):
self.mqtt = mqtt_sim()
self.mqtt.on_message = data_handler1.mqtt_message # xxxxx
def mqtt_message(self, client, message):
print("mqtt_message1", self, client, message)
# This is what you should be doing
class data_handler2(): # Default namespaces are just for all the ESPs.
def __init__(self):
self.mqtt = mqtt_sim()
self.mqtt.on_message = self.mqtt_message #attach function to callback
def mqtt_message(self, mqttself, client, message):
print("mqtt_message2", self, mqttself, client, message)
# Lets try using both of the above
d = data_handler1()
d.mqtt._on_message("self", "userdata", "message")
d = data_handler2()
d.mqtt._on_message("self", "userdata", "message")
Related
I want to use grpc-python in the following scenario, but I don' t know how to realize it.
The scenario is that, in the python server, it uses class to calculate and update the instance' s state, then sends such state to corresponding client; in the client side, more than one clients need to communicate with the server to get its one result and not interfered by others.
Specifically, suppose there is a class with initial value self.i =0, then each time the client calls the class' s update function, it does self.i=self.i+1 and returns self.i. Actually there are two clients call such update function simultaneously, like when client1 calls update at third time, client2 calls update at first time.
I think this may can be solved by creating thread for each client to avoid conflict. If the new client calls, new thead will be created; if existing client calls, existing thread will be used. But I don' t know how to realize it?
Hope you can help me. Thanks in advance.
I think I solved this problem by myself. If you have any other better solutions, you can post here.
I edited helloworld example in grpc-python introduction to explain my aim.
For helloworld.proto
syntax = "proto3";
option java_multiple_files = true;
option java_package = "io.grpc.examples.helloworld";
option java_outer_classname = "HelloWorldProto";
option objc_class_prefix = "HLW";
package helloworld;
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
rpc Unsubscribe (HelloRequest) returns (HelloReply) {}
}
// The request message containing the user's name.
message HelloRequest {
string name = 1;
}
// The response message containing the greetings
message HelloReply {
string message = 1;
}
I add Unsubsribe function to allow one specific client to diconnect from server.
In hello_server.py
import grpc
import helloworld_pb2
import helloworld_pb2_grpc
import threading
from threading import RLock
import time
from concurrent import futures
import logging
class Calcuate:
def __init__(self):
self.i = 0
def add(self):
self.i+=1
return self.i
class PeerSet(object):
def __init__(self):
self._peers_lock = RLock()
self._peers = {}
self.instances = {}
def connect(self, peer):
with self._peers_lock:
if peer not in self._peers:
print("Peer {} connecting".format(peer))
self._peers[peer] = 1
a = Calcuate()
self.instances[peer] = a
output = a.add()
return output
else:
self._peers[peer] += 1
a = self.instances[peer]
output = a.add()
return output
def disconnect(self, peer):
print("Peer {} disconnecting".format(peer))
with self._peers_lock:
if peer not in self._peers:
raise RuntimeError("Tried to disconnect peer '{}' but it was never connected.".format(peer))
del self._peers[peer]
del self.instances[peer]
def peers(self):
with self._peers_lock:
return self._peers.keys()
class Greeter(helloworld_pb2_grpc.GreeterServicer):
def __init__(self):
self._peer_set = PeerSet()
def _record_peer(self, context):
return self._peer_set.connect(context.peer())
def SayHello(self, request, context):
output = self._record_peer(context)
print("[thread {}] Peers: {}, output: {}".format(threading.currentThread().ident, self._peer_set.peers(), output))
time.sleep(1)
return helloworld_pb2.HelloReply(message='Hello, {}, {}!'.format(request.name, output))
def Unsubscribe(self, request, context):
self._peer_set.disconnect(context.peer())
return helloworld_pb2.HelloReply(message='{} disconnected!'.format(context.peer()))
def serve():
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
helloworld_pb2_grpc.add_GreeterServicer_to_server(Greeter(), server)
server.add_insecure_port('[::]:50051')
server.start()
server.wait_for_termination()
if __name__ == '__main__':
logging.basicConfig()
serve()
The use of context.peer() is adapted from Richard Belleville' s answer in this post. You can change add() function to any other functions that can be used to update instance' s state.
In hello_client.py
from __future__ import print_function
import logging
import grpc
import helloworld_pb2
import helloworld_pb2_grpc
def run():
# NOTE(gRPC Python Team): .close() is possible on a channel and should be
# used in circumstances in which the with statement does not fit the needs
# of the code.
with grpc.insecure_channel('localhost:50051') as channel:
stub = helloworld_pb2_grpc.GreeterStub(channel)
response = stub.SayHello(helloworld_pb2.HelloRequest(name='you'))
print("Greeter client received: " + response.message)
response = stub.SayHello(helloworld_pb2.HelloRequest(name='Tom'))
print("Greeter client received: " + response.message)
response = stub.SayHello(helloworld_pb2.HelloRequest(name='Jerry'))
print("Greeter client received: " + response.message)
stub.Unsubscribe(helloworld_pb2.HelloRequest(name="end"))
if __name__ == '__main__':
logging.basicConfig()
run()
If we run serveral hello_client.py simultaneously, the server can distinguish the different clients and send correct corresponding info to them.
I am trying to store the content of a callback function, in order to access and manipulate the data within the script.
As far as I am concerned, the given function .subscribe() in my code below does not return anything (None). My function is only passed as a reference to the function as an argument. Is there a way to return the data from the function that calls my function?
My code is a simple example with roslibpy (a library for Python that interacts with the open-source robotics framework ROS through Websockets). It is mentioned, that the data is published as a stream via a Websocket each time a message is published into the topic /turtle1/pose. My goal here is to return the data that is being published into the topic. The print command provides a nice visualization of the data, which just works fine.
import roslibpy
client = roslibpy.Ros(host='localhost', port=9090)
client.run()
def myfunc(msg):
print(msg)
listener = roslibpy.Topic(client, '/turtle1/pose', 'turtlesim/Pose')
#my function is passed as an argument
listener.subscribe(myfunc)
try:
while True:
pass
except KeyboardInterrupt:
client.terminate()
The subscribe() method in the roslibpy library is defined as follows:
def subscribe(self, callback):
"""Register a subscription to the topic.
Every time a message is published for the given topic,
the callback will be called with the message object.
Args:
callback: Function to be called when messages of this topic are published.
"""
# Avoid duplicate subscription
if self._subscribe_id:
return
self._subscribe_id = 'subscribe:%s:%d' % (
self.name, self.ros.id_counter)
self.ros.on(self.name, callback)
self._connect_topic(Message({
'op': 'subscribe',
'id': self._subscribe_id,
'type': self.message_type,
'topic': self.name,
'compression': self.compression,
'throttle_rate': self.throttle_rate,
'queue_length': self.queue_length
}))
Is there common way to deal with such problems? Does it make more sense to store the output as an external source (e.g. .txt) and then access the source trough the script?
You can define a Python class that acts like a function, that can modify its own state when called, by defining the magic __call__ method. When obj(whatever) is done on a non-function obj, Python will run obj.__call__(whatever). subscribe only needs its input to be callable; whether it is an actual function or an object with a __call__ method does not matter to subscribe.
Here's an example of what you could do:
class MessageRecorder():
def __init__(self):
self.messages = []
# Magic python 'dunder' method
# Whenever a MessageRecorder is called as a function
# This function defined here will be called on it
# In this case, adds the message to a list of received messages
def __call__(self, msg):
self.messages.append(msg)
recorder = MessageRecorder()
# recorder can be called like a function
recorder("Hello")
listener.subscribe(recorder)
try:
while True:
pass
except KeyboardInterrupt:
client.terminate()
"""Now you can do whatever you'd like with recorder.messages,
which contains all messages received before termination,
in order of reception. If you wanted to print all of them, do:"""
for m in recorder.messages:
print(m)
I just jump into websocket programing with basic knowledge of "Asynchronous" and "Threads", i have something like this
import tornado.httpserver
import tornado.websocket
import tornado.ioloop
import tornado.web
import socket
import uuid
import json
import datetime
class WSHandler(tornado.websocket.WebSocketHandler):
clients = []
def open(self):
self.id = str(uuid.uuid4())
self.user_info = self.request.remote_ip +' - '+ self.id
print (f'[{self.user_info}] Conectado')
client = {"sess": self, "id" : self.id}
self.clients.append(client.copy())
def on_message(self, message):
print (f'[{self.user_info}] Mensaje Recivido: {message}')
print (f'[{self.user_info}] Respuesta al Cliente: {message[::-1]}')
self.write_message(message[::-1])
self.comm(message)
def on_close(self):
print (f'[{self.user_info}] Desconectado')
for x in self.clients:
if x["id"] == self.id :
self.clients.remove(x)
def check_origin(self, origin):
return True
application = tornado.web.Application([
(r'/', WSHandler),
])
if __name__ == "__main__":
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(80)
myIP = socket.gethostbyname(socket.gethostname())
print ('*** Websocket Server Started at %s***' % myIP)
tornado.ioloop.IOLoop.instance().start()
my question is where do I add code ?, should I add everything inside the WShandler class, or outside, or in another file ? and when to use #classmethod?. for now there is no problem with the code when i add code inside the handler but i have just few test clients.
maybe not the full solution but just a few thoughts..
You can maybe look at the tornado websocket chat example,
here.
First good change is, that their clients (waiters) is a set()
which makes sure that every client is only contained once by default. And it is defined and accessed as a class variable. So you don't use self.waiters but cls.waiters or ClassName.waiters (in this case ChatSocketHandler.waiters) to access it.
class ChatSocketHandler(tornado.websocket.WebSocketHandler):
waiters = set()
Second change is that they update every client (you could choose here
to send the update not to all but only some) as a #classmethod, since
they dont want to receive the instance (self) but the class (cls) and
refer to the class variables (in their case waiters, cache and cach_size)
We can forget about the cache and cache size here.
So like this:
#classmethod
def send_updates(cls, chat):
logging.info("sending message to %d waiters", len(cls.waiters))
for waiter in cls.waiters:
try:
waiter.write_message(chat)
except:
logging.error("Error sending message", exc_info=True)
On every API call a new instance of your handler will be created, refered to as self. And every parameter in self is really unique to the instance and related to the actual client, calling your methods. This is good to identify a client on each call.
So a instance based client list like (self.clients) would always be empty on each call. And adding a client would only add it to this instance's view of the world.
But sometimes you want to have some variables like the list of clients the same for all instances created from your class.
This is where class variables (the ones you define directly under the class definition) and the #classmethod decorator come into play.
#classmethod makes the method call independant from the a instance. This means that you can only access class variables in those methods. But in the case of a
message broker this is pretty much what we want:
add clients to the class variable which is the same for all instances of your handler. And since it is defined as a set, each client is unique.
when receiving messages, send them out to all (or a subset of clients)
so on_message is a "normal" instance method but it calls something like: send_updates() which is a #classmethod in the end.
send_updates() iterates over all (or a subset) of clients (waiters) and uses this to send the actual updates in the end.
From the example:
#classmethod
def send_updates(cls, chat):
logging.info("sending message to %d waiters", len(cls.waiters))
for waiter in cls.waiters:
try:
waiter.write_message(chat)
except:
logging.error("Error sending message", exc_info=True)
Remember that you added waiters with waiters.append(self) so every waiter is really an instance and you are "simply" calling the instances (the instance is representing a caller) write_message() method. So this is not broadcasted but send to every caller one by one. This would be the place where you can separate by some criteria like topics or groups ...
So in short: use #classmethod for methods that are independant from a specific instance (like caller or client in your case) and you want to make actions for "all" or a subset of "all" of your clients. But you can only access class variables in those methods. Which should be fine since it's their purpose ;)
I am a newbie in Python. I am now having a project that using a raspberry pi to connect to local web, a web server & a mobile apps via websocket. Both of them are interactive. I can now communicate with each of them separately with 3 different program. But I face some problems when I want to integrate them into 1.
I read a number of post in here & find that all answers suggest to use a list to store each websocket and all of them sending the same message. Is it possible to send different message? Maybe something this
import tornado.ioloop
import tornado.web
import tornado.websocket
class WebSocketHandler_web(tornado.websocket.WebSocketHandler):
def __init__(self, application, **kwargs):
pass
def open(self):
# do something
def on_message(self, msg):
# do something
def on_close(self):
# do something
class WebSocketHandler_mobile(tornado.websocket.WebSocketHandler):
def __init__(self, application, **kwargs):
pass
def open(self):
# do something
def on_message(self, msg):
# do something
def on_close(self):
# do something
class WebSocketHandler_server(tornado.websocket.WebSocketHandler):
def __init__(self, application, **kwargs):
pass
def open(self):
# do something
def on_message(self, msg):
# do something
def on_close(self):
# do something
app_web = tornado.web.Application([(r'/ws/', WebSocketHandler_web),])
app_mobile = tornado.web.Application([(r'/ws/', WebSocketHandler_mobile),])
app_server = tornado.web.Application([(r'/ws/', WebSocketHandler_server),])
def main_task():
# do something
if(mode == 1):
webSocket_web.write_message("Mode 1")
elif(mode == 2):
webSocket_mobile.write_message("Mode 2")
elif(mode == 3):
webSocket_server.write_message("Mode 3")
# do something
if __name__ == "__main__":
app_web.listen(7777)
app_mobile.listen(8888)
app_server.listen(9999)
webSocket_web = WebSocketHandler_web(app_web)
webSocket_mobile = WebSocketHandler_mobile(app_mobile)
webSocket_server = WebSocketHandler_server(app_server)
tornado.ioloop.PeriodicCallback(main_task,1000).start()
tornado.ioloop.IOLoop.instance.start()
But seems that websocket.init needs a parameter request. What is that?
Is it possible to send different message Just iterate over a list of stored websockets and send different message, simple as that.
init needs a parameter request Tornado on each incoming request creates an instance of a request handler, and request is passed to init to bind a handler to request. When you override an init method in your handler you should follow parent's method interface. Don't worry about value of request argument, tornado will pass it for you.
I'm new to Python, and I think I'm trying to do something simple. However, I am confused with the results I am getting. I am declaring a class that has 2 class methods, add and remove, which in my simple example add or remove a client from a list class variable. Here's my code:
Service.py
from Client import Client
class Service:
clients = []
#classmethod
def add(cls, client):
cls.clients.append(client)
#classmethod
def remove(cls, client):
if client in cls.clients:
cls.clients.remove(client)
if __name == '__main__'
a = Client()
b = Client()
c = Client()
Service.add(a)
Service.add(b)
Service.add(c)
print(Service.clients)
c.kill()
print(Service.clients)
Service.remove(c)
print(Service.clients)
Client.py
class Client:
def kill(self):
from Service import Service
Service.remove(self)
I would expect calling c.kill() would remove the instance from the clients list.
However, when I evaluate the clients list, it is showing 0 items. when I call Service.remove(c), it shows the correct list, and removes it as expected. I am not sure what I am missing here.
If it matters, I am currently using PyCharm with my code running in a Virtualenv with Python 3.6.5.
Your current code is using circular imports, as both files utilize each other. Also, instead of relying on the client to destroy the connections, use a contextmanager to facilitate the updating of clients, and at the end of the procedure, empty clients:
import contextlib
class Client:
pass
class Service:
clients = []
#classmethod
def add(cls, client):
cls.clients.append(client)
#classmethod
#contextlib.contextmanager
def thread(cls):
yield cls
cls.clients = []
with Service.thread() as t:
t.add(Client())
t.add(Client())