Develop Reference

Python async events and serial async?

I have code (python class, I'll call it Arduino) that writes control packets to a serial port (using serial_asyncio), and the serial port replies with confirmation packets. However the remote device also sends event packets to the python side randomly. I want the class to provide the decoded packets to the class that instantiates my class (I'll call it Controller). I am confused on how to handle this.
My first thought is to provide callback to the Arduino class:
class Controller:
def __init__(self):
self.arduino = Arduino("/dex/ttyS0", self._serialPacketCallback)
def _serialPacketCallback(self, obj: dict):
pass # handle spontaneous event packet from remote
But this does not seem very async-y. What is the asyncio way to do this? I think this would look like:
class Controller:
def __init__(self):
self.arduino = Arduino("/dex/ttyS0")
async readEventPacket(self):
pass
#somewhere, not sure where, or how to start it:
async def _handleEvents(self)
while True:
event = await self._readEventPacket()
async def start(self):
await self.arduino.start()
await asyncio.wait([self._handleEvents()])
if __name__ == '__main__':
controller = Controller()
loop = asyncio.get_event_loop()
loop.create_task(controller.start())
loop.run_forever()
I've looked around and I've seen suggestions of using callbacks, multi-processing pipes, additional event loops, and I am sure they work, but I'm not sure what the proper approach is. For me, I don't want to start any additional event loops or threads, leading me to think the callback is best, but that is not very async, and I would like to know how to do this as async, without additional event loops or callbacks.
An additional concern that I want to articulate is that I'd like this as loosely coupled as the Arduino class will be used in other controllers.
Side note: I am not sure when in Python a new event loop is required to be created?
Another question: how does the Arduino class generate an event and have Controller pick it up in await self._readEventPacket()?

The nice thing about asyncio is that you can always convert a callback-based interface to a coroutine-based one, and vice versa.
Let's assume your Arduino class implements a callback-based interface, like this (untested):
class Arduino:
def __init__(self, device, callback):
self._device = device
self._callback = callback
async def start(self):
reader, writer = await serial_asyncio.connect(url=self._device, baudrate=115200)
while True:
data = await reader.read(1024)
self._callback(data)
You can convert that interface into a coroutine-based one by using a queue:
def callback_to_coro():
# Return two functions, one that can be passed as callback to
# code that expects it, and the other a coroutine that can be
# awaited to get the values the callback was invoked with.
queue = asyncio.Queue()
return queue.put_nowait, queue.get
With that code you can implement Controller.read_event_packet like this:
class Controller:
def __init__(self):
callback, wait = callback_to_coro()
self.arduino = Arduino("/dex/ttyS0", callback)
self.read_event_packet = wait

Mulitprocess management in Python with aiomultiprocess

I have a problem with the multiprocessing in Python. I need to create async processes, which run a undefined time and the number of processes is also undefined. As soon as a new request arrives, a new process must be created with the specifications from the request. We use ZeroMQ for messaging. There is also a Process which is started at the beginning and only ends if the whole script terminates.
Now I am searching for a solution how I can await all processes, while being able to add additional processes.
asyncio.gather()
Was my first idea, but it needs the list of processes before it's been called.
class Object:
def __init__(self, var):
self.var = var
async def run(self):
*do async things*
class object_controller:
def __init__(self):
self.ctx = zmq.Context()
self.socket = self.ctx.socket(zmq.PULL)
self.socket.connect("tcp://127.0.0.1:5558")
self.static_process = AStaticProcess()
self.sp = aiomultiprocess.Process(target=self.static_process.run)
self.sp.start()
#here I need a good way to await this process
def process(self, var):
object = Object(var)
process = aiomultiprocess.Process(target=object.run)
process.start()
def listener(self)
while True:
msg = self.socket.recv_pyobj()
# here I need to find a way how I can start and await this process while beeing able to
# receive additional request, which result in additional processes which need to be awaited
This is some code which hopefully explains my problem. I need a kind of Collector which awaits the Processes.
After initialization, there is no interaction between the object and the controller, only over zeroMQ (between the static process and the variable processes). There is also no return.

If you need to start up proceses while concurrently waiting for new ones, instead of explicitly calling await to know when the Processes finish, let them execute in the background using asyncio.create_task(). This will return a Task object, which has an add_done_callback method, which you can use to do some work when the process completes:
class Object:
def __init__(self, var):
self.var = var
async def run(self):
*do async things*
class object_controller:
def __init__(self):
self.ctx = zmq.Context()
self.socket = self.ctx.socket(zmq.PULL)
self.socket.connect("tcp://127.0.0.1:5558")
self.static_process = AStaticProcess()
self.sp = aiomultiprocess.Process(target=self.static_process.run)
self.sp.start()
asyncio.create_task(self.sp.join() self.handle_proc_finished)
def process(self, var):
object = Object(var)
process = aiomultiprocess.Process(target=object.run)
process.start()
def listener(self)
while True:
msg = self.socket.recv_pyobj()
process = aiomultiprocess.Process(...)
process.start()
t = asyncio.create_task(process.join())
t.add_done_callback(self.handle_other_proc_finished)
def handle_proc_finished(self, task):
# do something
def handle_other_proc_finished(self, task):
# do something else
If you want to avoid using callbacks, you can also pass create_task a coroutine you define yourself, which waits for the process to finish and does whatever needs to be done afterward.
self.sp.start()
asyncio.create_task(wait_for_proc(self.sp))
async def wait_for_proc(proc):
await proc.join()
# do other stuff

You need to create a list of tasks or a future object for the processes. Also you cannot add process to the event loop while awaiting other tasks

Threads persisting with irc.bot.SingleServerIRCBot (using with twitch)

What is the correct way to send a disconnect signal to a thread containing a SingleServerIRCBot?
I am instantiating bots that connect to twitch with
import threading
import irc.bot
class MyBot(irc.bot.SingleServerIRCBot):
...
bot = MyBot(...)
threads = []
t = threading.Thread(target=bot.start()
threads.append(t)
t.start()
When the stream no longer exists, no matter what I've tried, I haven't been able to get the thread to successfully end. How should I go about sending a signal to the thread that tells it to exit the channel kill the bot and then itself?
The code for the .start method can be found here https://github.com/jaraco/irc/blob/master/irc/bot.py#L331
My first thought is to override that method with a while loop that has an exit condition. I haven't had any luck with that so far though.
Furthermore, there is a .die method here https://github.com/jaraco/irc/blob/master/irc/bot.py#L269 but how can I call that method when the thread is executing an infinite loop?
Trying to kill the threads directly ends up with them persisting, and eventually throwing errors about the total number of threads that my process is running.
Edit for the bounty: I would also accept an answer that describes a better way to handle multiple IRC bots at once.

I don't think you could (or should) kill a thread directly, but you could stop the task running on that thread. Then the thread would be inactive and you could remove it from the threads list, if you like. I'm not familiar with SingleServerIRCBot, but I'll use the class below as an example.
import time
class MyTask:
def __init__(self):
self._active = True
def start(self):
while self._active:
print('running')
time.sleep(1)
def die(self):
self._active = False
In Python3, threads have a _target attribute, from which we can access the target function/method. We could use this attribute to access the target's object and call the die method (eg: thread._target.__self__.die()). However I think it would be best to subclass Thread and store the the target object in a variable, as _target is a private attribute, and also for compatibility reasons.
import threading
class MyThread(threading.Thread):
def __init__(self, target, args=()):
super(MyThread, self).__init__()
self.target = target
self.args = args
def run(self):
self.target.start(*self.args)
def stop_task(self):
self.target.die()
Using this class we would pass a MyTask object as a target, and the start method would be called from MyThread.run. Now we can use MyThread.stop_task to stop the task running on this thread.
o = MyTask()
t = MyThread(target=o)
t.start()
t.stop_task()
time.sleep(1.1)
print(t.is_alive())
Note that I'm waiting 1.1 sec to test if the thread is alive. That's because the target (MyTask.start) will take up to one second to stop. This method doesn't kill the thread, but calls MyTask.die and waits for the task to finish. If you want to end the task immediately (and loose any resources used by the task) you could use a Process and end it with .terminate. You should also choose multiprocessing over multithreading if your task is performing more CPU operations than IO operations, because processes are not limited by the GIL.
Afrer stydying the source code, I noticed that .die() calls sys.exit, so we can't use it to terminate the task because it would stop the program. It seems the reason for this is that .start() calls the parent object's .start(), which then calls the .process_forever() method of a Reactor object. This method starts running Reactor.process_once() in an infinite loop with no break condition.
A possible solution is to subclass SingleServerIRCBot and use a boolean variabe to break the loop. This class should override .start() and .die(), in order to stop the bot running on a thread. The .die() method would set the flag to false, and .start() would call Reactor.process_once() in a loop.
import irc.bot
class MyBot(irc.bot.SingleServerIRCBot):
def __init__(self, channel, nickname, server, port=6667):
super(MyBot, self).__init__([(server, port)], nickname, nickname)
self.channel = channel
self._active = True
def start(self):
self._connect()
while self._active:
self.reactor.process_once(timeout=0.2)
def die(self, msg="Bye, cruel world!"):
self.connection.disconnect(msg)
self._active = False
Now we can stop the bot either by calling .stop_task() on the thread running the bot, or by calling the .die() method of the bot directly.
host, port = 'irc.freenode.net', 6667
nick = 'My-Bot'
channel = '#python'
bot = MyBot(channel, nick, host, port)
t = MyThread(bot)
t.start()
t.stop_task()
#bot.die()

how to create python parallel sockets in asyncio and transport_base class?

I used asyncio for my non-stop server in python and implemented
connection_made , connection_lost , data_received
funtions in my ServerClientProtocol
I used this class first beacause of using multiple times repeatedly sending data to socket class socket
got closed and program exited
and second becuase I thought its async and have parallel answering multiple coming sockets in same time,
but it's not.
how should I use that in one async thread and parallel answering socket?
this is my code:
class ServerClientProtocol(asyncio.Protocol):
def connection_made(self,transport):
self.transport = transport
def connection_lost(self,exc):
pass
def data_received(self, data):
server.server(self,data)
def main(*args):
loop = get_event_loop()
coro = loop.create_server(ServerClientProtocol, '127.0.0.1', 50008)
srv = loop.run_until_complete(coro)
loop.run_forever()
if __name__ == '__main__':
main()

server.server() might be blocking the other connections. If this is a long-running call, try using asyncio.start_server (example here) instead, and call server.server() using await loop.run_in_executor(None, server.server, data)

Is it possible to run the asyncio.Server instance while the event loop is already running

I'm trying to understand, is it possible to run the asyncio.Server instance while the event loop is already running by run_forever method (from a separate thread, of course).
As I understand, the server can be started either by loop.run_until_complete(asyncio.start_server(...)) or by
await asyncio.start_server(...), if the loop is already running.
The first way is not acceptable for me, since the loop is already running by run_forever method. But I also can't use the await expression, since I'm going to start it from outside the "loop area" (e.g. from the main method, which can't be marked as async, right?)
def loop_thread(loop):
asyncio.set_event_loop(loop)
try:
loop.run_forever()
finally:
loop.close()
print("loop clesed")
class SchedulerTestManager:
def __init__(self):
...
self.loop = asyncio.get_event_loop()
self.servers_loop_thread = threading.Thread(
target=loop_thread, args=(self.loop, ))
...
def start_test(self):
self.servers_loop_thread.start()
return self.servers_loop_thread
def add_router(self, router):
r = self.endpoint.add_router(router)
host = router.ConnectionParameters.Host
port = router.ConnectionParameters.Port
srv = TcpServer(host, port)
server_coro = asyncio.start_server(
self.handle_connection, self.host, self.port)
# does not work since add_router is not async
# self.server = await server_coro
# does not work, since the loop is already running
# self.server = self.loop.run_until_complete(server_coro)
return r
def maind():
st_manager = SchedulerTestManager()
thread = st_manager.start_test()
router = st_manager.add_router(router)
Of cource, the simplest solution is to add all routers (servers) before starting the test (running the loop). But I want try to implement it, so it would be possible to add a router when a test is already running. I thought the loop.call_soon (call_soon_threadsafe) methods can help me, but it seems the can't shedule a coroutine, but just a simple function.
Hope that my explanation is not very confusing. Thanks in advance!

For communicating between event loop executed in one thread and conventional old good threaded code executed in other thread you might use janus library.
It's a queue with two interfaces: async and thread-safe sync one.
This is usage example:
import asyncio
import janus
loop = asyncio.get_event_loop()
queue = janus.Queue(loop=loop)
def threaded(sync_q):
for i in range(100):
sync_q.put(i)
sync_q.join()
#asyncio.coroutine
def async_coro(async_q):
for i in range(100):
val = yield from async_q.get()
assert val == i
async_q.task_done()
fut = loop.run_in_executor(None, threaded, queue.sync_q)
loop.run_until_complete(async_coro(queue.async_q))
loop.run_until_complete(fut)
You may create a task waiting new messages from the queue in a loop and starting new servers on request. Other thread may push new message into the queue asking for a new server.