Develop Reference

Is there a way to 'pause' pyserial's ReaderThread to allow a direct read of a serial port

I've got a gui that I'm playing with that uses pyserial. In it I'm using pyserial's ReaderThread to monitor the serial output of my serial device and print it out on a console window.
I also am using pyserial's Serial() implementation for sending commands to the serial device.
Usually I don't need to grab the response to a ser.write() and just let the ReaderThread handle it.
However there are now occasions where I'd like in pause the ReaderThread do a ser.read() to a variable, act on the variable, and unpause the ReaderThread to let it continue it's thing.
Tried ReaderThread.stop(), but it seems to be dropping the connection.
Also tried creating my own readerThread.run() function that has mutex locking and replacing the run method with it, but that's turning out to be a bit squirrelly.
Am I missing an easy way to do this?

Figured a way by monkey patching the ReaderThread Class:
def localinit(self, serial_instance, protocol_factory):
"""\
Initialize thread.
Note that the serial_instance' timeout is set to one second!
Other settings are not changed.
"""
super(ReaderThread, self).__init__()
self.daemon = True
self.serial = serial_instance
self.protocol_factory = protocol_factory
self.alive = True
self._lock = threading.Lock()
self._connection_made = threading.Event()
self.protocol = None
self._stop_event = threading.Event()
print("****************************************************")
print(" localinit ")
print("****************************************************")
def localrun(self):
"""Reader loop"""
print("****************************************************")
print(" localrun ")
print("****************************************************")
if not hasattr(self.serial, 'cancel_read'):
self.serial.timeout = 1
self.protocol = self.protocol_factory()
try:
self.protocol.connection_made(self)
except Exception as e:
self.alive = False
self.protocol.connection_lost(e)
self._connection_made.set()
return
error = None
self._connection_made.set()
while self.alive and self.serial.is_open:
while self._stop_event.is_set():
#print("local run while")
time.sleep(1)
try:
data = self.serial.read(self.serial.in_waiting or 1)
except serial.SerialException as e:
# probably some I/O problem such as disconnected USB serial
# adapters -> exit
error = e
break
else:
if data:
# make a separated try-except for called user code
try:
self.protocol.data_received(data)
except Exception as e:
error = e
break
self.alive = False
self.protocol.connection_lost(error)
self.protocol = None
def localpause(self):
self._stop_event.set()
def localresume(self):
self._stop_event.clear()
Then in my main code:
ReaderThread.run = localrun
ReaderThread.__init__ = localinit
ReaderThread.pause = localpause
ReaderThread.resume = localresume
self.reader = ReaderThread(serialPort, SerialReaderProtocolLine)
self.reader.start()
def write_read_cmd(self, cmd_str):
if(serialPort.isOpen() == False):
print("Serial port not yet open")
return
app.serialcom.reader.pause()
serialPort.reset_input_buffer() # flush the buffer
serialPort.reset_input_buffer() # flush the buffer
serialPort.reset_input_buffer() # flush the buffer
serialPort.write(bytes(cmd_str, encoding='utf-8'))
line = serialPort.readline()
app.serialcom.reader.resume()
line = line.decode("utf-8")
return line

Threaded Buffer crashes

I tried to implement a 5 element buffer through threading and a list.
import threading
class Buffer1 :
def __init__(self,size) :
self.empty = True
self.content = None
self.lock = threading.Condition()
self.list = []
def take(self) :
with self.lock :
while not self.list :
self.lock.wait()
help = self.list[len(self.list)-1]
del self.list[len(self.list)-1]
self.lock.notify_all()
return help
def put(self,v) :
with self.lock :
while len(self.list) >4:
self.lock.wait()
#self.content = v
self.list.append(v)
self.empty = False
self.lock.notify_all()
def show_list(self):
return self.list
a = Buffer1(5)
a.put(7)
Theoretically it works, but when you exceed the limitations of the buffer, either by buffering 6 values or by trying to „take()“ when there is no value buffered, the IDE becomes unresponsive. How could I go about fixing this?

You are using only one thread to add elements in the buffer, so the list contains 5 items, and your main thread is waiting indefinitely on self.lock.wait(). You can use another thread that would take some elements in parallel and then it will notify the producer thread.
For example, creation of a consumer thread that takes 5 items:
def consume(buffer):
import time
for i in range(5):
print(threading.current_thread(), "consume", buffer.take())
time.sleep(2)
print(threading.current_thread(), "Bye")
buffer = Buffer1(5)
t = threading.Thread(target=consume, args=(buffer,), name="consumer")
t.start()
buffer.put(1)
buffer.put(2)
buffer.put(3)
buffer.put(4)
buffer.put(5)
buffer.put(6)
print(buffer.show_list())

... the IDE becomes unresponsive. How could I go about fixing this?
You only showed adding to the buffer from the main thread and nothing ever takes anything out.
If the buffer gets filled up or becomes empty the next put/take will cause its Condition (lock) to wait until something notifies it to continue. I didn't see any of that signaling in your example.
The buffer is a shared resource. The buffer and the threads that use it need to have good control so that everyone can stay out of everyone else's way and enough logic to keep from getting stuck somewhere.
Presumably you need a thread that puts stuff into the buffer and a thread that takes stuff out of the buffer - both having ample signaling to notify everyone when they are done messing with the buffer.
Set up logging so that the program execution could be traced with log messages.
Buffer1 changes:
Changed the list to a collections.deque to simplify things a bit.
Added properties for empty and full
Added an Event attribute to stop putting/taking when the process gets shut down.
Added a timeout while waiting to put/take to forestall any timing issues when the threads are shut down
Added notifications for empty and full conditions.
Made two threads: one to add to the buffer and one to take from the buffer.
Each will add/take while its Event is not set.
In each iteration a random number of items are taken or added.
When taking/putting the buffer's Condition is acquired and all Waiter's are notified when complete.
In the main thread:
An Event is created - for signaling the thread to quit
A Timer is created to limit thread execution time - when it times out its callback function sets the Event and uses the buffer's Condition (lock) to notify anyone that is waiting and free them up.
The threads are created, started, and joined.
import threading
import collections
import random
import string
import time
import logging
import sys
# logging setup
root = logging.getLogger()
root.setLevel(logging.INFO)
formatter = logging.Formatter(fmt='%(asctime)s.%(msecs)03d %(message)s',
datefmt='%S')
class WarningFilter(logging.Filter):
def filter(self, record):
return record.levelno == logging.WARNING
class InfoFilter(logging.Filter):
def filter(self, record):
return record.levelno == logging.INFO
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.INFO)
handler.setFormatter(formatter)
handler.addFilter(InfoFilter())
root.addHandler(handler)
handler = logging.StreamHandler(sys.stderr)
handler.setLevel(logging.WARNING)
handler.setFormatter(formatter)
handler.addFilter(WarningFilter())
root.addHandler(handler)
# logging setup end
class Buffer1:
'''FILO buffer.
'''
def __init__(self,size,evt) :
self.content = None
self.size = size
self.evt = evt
self.lock = threading.Condition()
self.list = collections.deque()
#property
def full(self):
return len(self.list) >= self.size
#property
def empty(self):
return bool(self.list)
def take(self) :
with self.lock :
while not self.empty:
root.warning('buffer empty waiting to take')
self.lock.wait(timeout=5)
if self.evt.is_set():
help = None
break
else:
help = self.list.pop()
self.lock.notify_all()
return help
def put(self,v):
success = False
with self.lock :
while self.full:
root.warning('buffer full waiting to put')
self.lock.wait(timeout=5)
if self.evt.is_set():
break
else:
self.list.append(v)
success = True
self.lock.notify_all()
return success
def show_list(self):
return self.list
class Prod(threading.Thread):
'''Puts stuff onto buffer, quits on Event.
Contrived toy - periodically puts random n items in buffer.
'''
def __init__(self,buffer,evt):
super().__init__(name='producer')
self.buffer = buffer
self.evt = evt
def run(self):
n = 0
while not self.evt.is_set():
howmany = random.randint(1,9)
payload = random.sample(string.ascii_letters,howmany)
payload = collections.deque(payload)
root.info(f'{self.name} putting {howmany}')
with self.buffer.lock:
while payload and (not self.evt.is_set()):
c = payload.popleft()
root.info(f'{self.name} -----> {c}')
if not self.buffer.put(c):
root.warning(f'{self.name} last put failed')
self.buffer.lock.notify_all()
time.sleep(.04)
n += 1
root.info(f'{self.name} dying n={n}')
with self.buffer.lock:
self.buffer.lock.notify_all()
root.info(f'{self.name} is done')
class Cons(threading.Thread):
'''Takes stuff off of buffer, quits on Event set.
Contrived toy - periodically takes random n items from buffer.
'''
def __init__(self,buffer,evt):
super().__init__(name='consumer')
self.buffer = buffer
self.evt = evt
def run(self):
n = 0
while not self.evt.is_set():
howmany = random.randint(1,9)
root.info(f'{self.name} taking {howmany}')
with self.buffer.lock:
while (howmany > 0) and (not self.evt.is_set()):
c = self.buffer.take()
root.info(f'{self.name} <----- {c}')
howmany -= 1
self.buffer.lock.notify_all()
time.sleep(.04)
n += 1
root.info(f'{self.name} dying n={n}')
with self.buffer.lock:
self.buffer.lock.notify_all()
root.info(f'{self.name} is done')
if __name__ == '__main__':
# use an Event to shut down the whole process
evt = threading.Event()
buffer = Buffer1(5,evt)
def kill(evt=evt,buffer=buffer):
root.warning('killing everything')
evt.set()
with buffer.lock:
buffer.lock.notify_all()
# don't let this toy example run forever
t = threading.Timer(5,kill)
t.start()
p1 = Prod(buffer,evt)
c1 = Cons(buffer,evt)
c1.start()
p1.start()
p1.join()
c1.join()
print('exit')

Here is another take using asyncio instead of threads to exercise your buffer.
import asyncio
import collections
import random
import string
import time
import logging
import sys
# logging setup
root = logging.getLogger()
root.setLevel(logging.INFO)
formatter = logging.Formatter(fmt='%(asctime)s.%(msecs)03d %(message)s',
datefmt='%S')
class WarningFilter(logging.Filter):
def filter(self, record):
return record.levelno == logging.WARNING
class InfoFilter(logging.Filter):
def filter(self, record):
return record.levelno == logging.INFO
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.INFO)
handler.setFormatter(formatter)
handler.addFilter(InfoFilter())
root.addHandler(handler)
handler = logging.StreamHandler(sys.stderr)
handler.setLevel(logging.WARNING)
handler.setFormatter(formatter)
handler.addFilter(WarningFilter())
root.addHandler(handler)
class Buffer:
'''FILO buffer.
'''
def __init__(self,size,evt) :
self.content = None
self.size = size
self.stop_evt = evt
self.lock = asyncio.Condition()
self.list = collections.deque()
def full(self):
return len(self.list) >= self.size
def not_full(self):
return len(self.list) < self.size
def empty(self):
return not bool(self.list)
def not_empty(self):
return bool(self.list)
async def take(self) :
async with self.lock:
#root.info(f'take:lock acquired - wait for not empty')
while self.empty():
waiters = [thing for thing in self.lock._waiters]
#root.warning(f'take:{waiters} waiting')
await self.lock.wait()
if self.stop_evt.is_set(): # shutting down
val = None
else:
#root.info('take: not empty')
val = self.list.pop()
self.lock.notify_all()
return val
async def put(self,v):
success = False
async with self.lock:
#root.info(f'put:lock acquired - wait for not full')
while self.full():
waiters = [thing for thing in self.lock._waiters]
#root.warning(f'put:{waiters} waiting')
await self.lock.wait()
if self.stop_evt.is_set(): # shutting down
break
else:
#root.info('put: not full')
self.list.append(v)
success = True
self.lock.notify_all()
return success
def show_list(self):
return self.list
async def random_stuff():
howmany = random.randint(1,9)
payload = random.sample(string.ascii_letters,howmany)
return collections.deque(payload)
async def produce(buffer,stop_evt,name):
puts = []
try:
while True:
payload = await random_stuff()
root.warning(f'producer{name} putting {len(payload)}')
while payload:
c = payload.popleft()
root.info(f'producer{name} -----> {c}')
success = await buffer.put(c)
if not success:
root.warning(f'producer{name} failed to put {c}')
else:
puts.append(c)
await asyncio.sleep(.03)
except asyncio.CancelledError as e:
root.warning('producer canceled')
root.info(f'producer{name} dying n={len(puts)}')
root.info(f'producer{name} is done')
return puts
async def consume(buffer, stop_evt, name):
'''Takes stuff off of buffer, quits on Event set.
Contrived toy - periodically takes random n items from buffer.
'''
takes = []
try:
while True:
howmany = random.randint(1,9)
msg = f'consumer{name} taking {howmany}'
root.warning(f'{msg:>38}')
while howmany > 0:
c = await buffer.take()
takes.append(c)
msg = f'consumer{name} <----- {c}'
root.info(f'{msg:>38}')
howmany -= 1
await asyncio.sleep(.02)
except asyncio.CancelledError as e:
root.warning('consumer canceled')
root.info(f'consumer{name} dying n={len(takes)}')
root.info(f'consumer{name} is done')
return takes
async def timer(n,buffer,evt, tasks):
root.warning('timer started')
await asyncio.sleep(n)
evt.set()
root.warning('timed out - event set')
root.warning('canceling tasks')
for task in tasks:
task.cancel()
async def main():
loop = asyncio.get_running_loop()
loop.set_debug(True)
# use an Event to shut down the whole process
evt = asyncio.Event()
buffer = Buffer(5,evt)
put_task = asyncio.create_task(produce(buffer,evt,1))
take_task = asyncio.create_task(consume(buffer,evt,1))
timer_task = asyncio.create_task(timer(5,buffer,evt,[put_task,take_task]))
root.info('tasks created')
await timer_task
puts = await put_task
takes = await take_task
print('exit')
return puts,takes,buffer.list
if __name__ == '__main__':
puts,takes,remains = asyncio.run(main())
puts = collections.Counter(puts)
takes = collections.Counter(takes)
remains = collections.Counter(remains)
#print(remains == (puts-takes))

How to close sockets of a class from a non class

my script works like this:
# other part of code
class request(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
while True:
try:
socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket.connect((host), (port))
socket.send(str.encode("test"))
except:
socket.close()
def loop():
for x in range(5):
request(x).start()
# other
# part
# of code
def startall():
# some other code
choice = input("command: ")
if choice == "request":
loop()
elif choice == "stop":
# ?
# some other code
startall()
Is there a way to stop sending request if the input is "stop"? Note that this is just a sample, my script doesn't work like this. I put this code just to let you understand what is my problem

If you want to stop all requests at once you can modify your class as folows:
class request(threading.Thread):
REQUESTS_ALLOWED = True
def run(self):
while request.REQUESTS_ALLOWED:
socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
socket.connect((host), (port))
socket.send(str.encode("test"))
except:
pass # Do what you need
finally:
socket.close()
Notice the alternation of closing socket. In your code the socket was closed when garbage collector destroyed your variable socket. With my alternation it's guaranteed that socket is closed with every iteration.
The start and stop events now can change the state of all request objects.
if choice == "request":
request.REQUESTS_ALLOWED = True
loop()
elif choice == "stop":
request.REQUESTS_ALLOWED = False
After setting REQUESTS_ALLOWED to False you should join() all running threads. It's just recommendation (and you don't have to do it) because normally when function returns it indicates that something is done. So after return from function startall() with choice = "stop" I would expect that all started threads are stopped.
Full code example:
import threading
import time
class Request(threading.Thread):
REQUESTS_ALLOWED = True
active_threads = set()
def __init__(self):
threading.Thread.__init__(self)
def start(self):
Request.active_threads.add(self) # Add thread to set for later use
super().start()
def run(self):
while Request.REQUESTS_ALLOWED:
print("Thread {} is alive.".format(self.name))
time.sleep(1)
print("Thread {} is done.".format(self.name))
def loop():
for x in range(5):
Request().start()
def startall(choice):
if choice == "request":
Request.REQUESTS_ALLOWED = True
loop()
elif choice == "stop":
Request.REQUESTS_ALLOWED = False
# Iterate through active threads and wait for them
for thread in Request.active_threads:
thread.join()
Request.active_threads.clear()
startall("request")
time.sleep(3)
startall("stop")
The code was tested in Python 3.6.1

Python: How to terminate multithreaded python program?

I want to create one program in which two lists of hosts are available. I want to read data from each host. It will take around 5-10 seconds so I want to read each host data with different thread.
I created below code and it is working as per my expectations but only problem is when I'm pressing Ctrl+c, program didn't terminate.
My code:
import threading
import time,os,sys
import signal
is_running = True
def signal_handler(signal, frame):
print "cleaning up...please wait..."
v1.stop()
v2.stop()
global is_running
is_running = False
class Thread2(threading.Thread):
def __init__(self, function,args):
self.running = False
self.function = function
self.args = args
super(Thread2, self).__init__()
def start(self):
self.running = True
super(Thread2, self).start()
def run(self):
while is_running:
self.function(self.args)
time.sleep(time_interval)
def stop(self):
self.running = False
def b_iterate(hostnames):
for host_name in hostnames:
v = Thread2(function = read_cet_data,args = host_name)
v.start()
def read_b_data(host):
#
#reading some data from current host (5-10 seconds processing)
#
#here, this thread is not neccessary, want to stop or kill or terminate it
if threading.current_thread().isAlive():
threading.current_thread().stop()
def a_iterate(entp_hostnames):
for host_name in entp_hostnames:
v = Thread2(function = read_entp_data,args = host_name)
v.start()
def read_a_data(host):
#
#reading some data from current host (5-10 seconds processing)
#
#here, this thread is not neccessary, want to stop or kill or terminate it
if threading.current_thread().isAlive():
threading.current_thread().stop()
if __name__ == "__main__":
signal.signal(signal.SIGINT, signal_handler)
#a_hostnmaes & b_hostnmaes are the lists of hostnames
v1 = Thread2(function = a_iterate,args = a_hostnames)
v2 = Thread2(function = b_iterate,args = b_hostnames)
v1.start()
v2.start()
while is_running:
pass
How I can make this program terminate after pressing Ctrl+c. Am I missing something?

If you just want control C to finish everything, there is no need to use a stop function in threads. You can just daemonise them:
v1 = Thread2(function = a_iterate,args = a_hostnames)
v2 = Thread2(function = b_iterate,args = b_hostnames)
v1.daemon = True
v2.daemon = True
v1.start()
v2.start()
As soon as your main program dies, these threads die as well. You need to add .daemon = True to all other locations in the code where a thread is created.
Hannu

You can either
catch KeyboardInterrupt in main thread
set a flag so another threads can detect it and exit
or
catch KeyboardInterrupt
call os._exit()

Run function in background and continue with program

I am trying to run a function in the background, whilst continuing with said code in python.
The function I want to run in the background is from socket. Looking for specific data to cut the program off.
Here is the function:
def receive():
host = ""
port = 13000
buf = 1024
addr = (host,port)
Sock = socket(AF_INET, SOCK_DGRAM)
Sock.bind(addr)
(data, addr) = Sock.recvfrom(buf)
return data
Here is the code I want to run:
while True:
r = receive()
if r == "stop":
break
#Cannot get past here, because of the function running.
#Should loop over and over, until stop data is received
print "Running program"
I have tried threading, with r = threading.Thread(target=receive()) with no joy.

Rookie error:
r = threading.Thread(target=receive())
I did not take the brackets off the receive():
r = threading.Thread(target=receive)

You can't return to the invoking thread from an invoked thread's target function. Instead, you need some inter-thread communication system. Below, is an example using Python's Queue to pass received datagrams between the two threads. I've used a threading.Event to signal when the receiver thread should stop.
#!/usr/bin/env python
import socket
import threading
from queue import Empty, Queue
class DatagramReceiver(threading.Thread):
def __init__(self, stop, queue):
super().__init__()
self._stop = stop
self._queue = queue
def run(self):
with socket.socket(AF_INET, SOCK_DGRAM) as sock:
sock.bind(('', 13000))
while not self._stop.is_set():
data = sock.recvfrom(1024)[0]
if data == 'stop':
self._stop.set()
break
self._queue.put(data)
def main():
stop = threading.Event()
queue = Queue()
reader = DatagramReceiver(stop, queue)
reader.deamon = True
reader.start()
while not stop.is_set():
user_input = input('Press RETURN to print datagrams, or q quit')
if user_input == 'q':
break
while True:
try:
datagram = queue.get_nowait()
except Empty:
break
print(datagram)
stop.set()
reader.join()