Develop Reference

Is there a way to 'pause' pyserial's ReaderThread to allow a direct read of a serial port

I've got a gui that I'm playing with that uses pyserial. In it I'm using pyserial's ReaderThread to monitor the serial output of my serial device and print it out on a console window.
I also am using pyserial's Serial() implementation for sending commands to the serial device.
Usually I don't need to grab the response to a ser.write() and just let the ReaderThread handle it.
However there are now occasions where I'd like in pause the ReaderThread do a ser.read() to a variable, act on the variable, and unpause the ReaderThread to let it continue it's thing.
Tried ReaderThread.stop(), but it seems to be dropping the connection.
Also tried creating my own readerThread.run() function that has mutex locking and replacing the run method with it, but that's turning out to be a bit squirrelly.
Am I missing an easy way to do this?

Figured a way by monkey patching the ReaderThread Class:
def localinit(self, serial_instance, protocol_factory):
"""\
Initialize thread.
Note that the serial_instance' timeout is set to one second!
Other settings are not changed.
"""
super(ReaderThread, self).__init__()
self.daemon = True
self.serial = serial_instance
self.protocol_factory = protocol_factory
self.alive = True
self._lock = threading.Lock()
self._connection_made = threading.Event()
self.protocol = None
self._stop_event = threading.Event()
print("****************************************************")
print(" localinit ")
print("****************************************************")
def localrun(self):
"""Reader loop"""
print("****************************************************")
print(" localrun ")
print("****************************************************")
if not hasattr(self.serial, 'cancel_read'):
self.serial.timeout = 1
self.protocol = self.protocol_factory()
try:
self.protocol.connection_made(self)
except Exception as e:
self.alive = False
self.protocol.connection_lost(e)
self._connection_made.set()
return
error = None
self._connection_made.set()
while self.alive and self.serial.is_open:
while self._stop_event.is_set():
#print("local run while")
time.sleep(1)
try:
data = self.serial.read(self.serial.in_waiting or 1)
except serial.SerialException as e:
# probably some I/O problem such as disconnected USB serial
# adapters -> exit
error = e
break
else:
if data:
# make a separated try-except for called user code
try:
self.protocol.data_received(data)
except Exception as e:
error = e
break
self.alive = False
self.protocol.connection_lost(error)
self.protocol = None
def localpause(self):
self._stop_event.set()
def localresume(self):
self._stop_event.clear()
Then in my main code:
ReaderThread.run = localrun
ReaderThread.__init__ = localinit
ReaderThread.pause = localpause
ReaderThread.resume = localresume
self.reader = ReaderThread(serialPort, SerialReaderProtocolLine)
self.reader.start()
def write_read_cmd(self, cmd_str):
if(serialPort.isOpen() == False):
print("Serial port not yet open")
return
app.serialcom.reader.pause()
serialPort.reset_input_buffer() # flush the buffer
serialPort.reset_input_buffer() # flush the buffer
serialPort.reset_input_buffer() # flush the buffer
serialPort.write(bytes(cmd_str, encoding='utf-8'))
line = serialPort.readline()
app.serialcom.reader.resume()
line = line.decode("utf-8")
return line

How to create mutliple tasks asyncio and get returns values

I'm beggining with asyncio and I'm struggling awaiting multiple return values from differents functions.
I want to do something like thread but using asyncio since I'm using async librairies from Azure.
I need to launch in my Thread class some function and I want to launch them in conccurency and await for their results and see if the results are good (I'm in a django test).
Here my current code:
class DeviceThread(threading.Thread):
def __init__(self, test_case, device_id, module_id, cert_prim, cert_sec):
threading.Thread.__init__(self)
self.test_case = test_case
self.device_id = device_id
self.module_id = module_id
self.cert_prim = cert_prim
self.cert_sec = cert_sec
async def get_twin(self, device):
try:
# 0 - Get the twin for the device
twin = await device.get_twin()
info(twin)
return twin['desired']['test'] == "test1device"
except Exception as e:
info(format_exc())
return False
async def receive_message_cloud(self, device):
try:
# 1- Waiting for the first message from the server (Blocking call)
message = await device.receive_message_cloud()
return message.data.decode() == "testmessage1"
except Exception as e:
info(format_exc())
return False
async def send_message_cloud(self, device):
try:
# 2 -Sending a message to the backend
await device.send_message_cloud("testmessage1")
return True
except Exception as e:
info(format_exc())
return False
async def receive_direct_call_and_respond(self, device):
try:
# 3 - Receiving a direct call method from the backend
method_request = await device.receive_method_request()
# 4 - Sending result of direct call
payload = {"test": "testok"}
status = 200
await device.send_method_response(method_request, status, payload)
return (method_request.name == "testmethod1") and (method_request.payload == "testpayload1")
except Exception as e:
info(format_exc())
return False
async def update_twin_properties(self, device):
try:
# 5 - Updating twin properties
new_properties = {'test', 'test2device'}
device.patch_twin_reported_properties(new_properties)
return True
except Exception as e:
info(format_exc())
return False
async def perform(self, device):
# Creating the tasks that will execute in parrallel
twin_get_res = asyncio.create_task(self.get_twin(device))
rec_mess_cloud = asyncio.create_task(self.receive_message_cloud(device))
send_mess_cloud = asyncio.create_task(self.send_message_cloud(device))
rec_dir_call = asyncio.create_task(self.receive_direct_call_and_respond(device))
up_twin_prop = asyncio.create_task(self.update_twin_properties(device))
# Verify the execution of the routine when done
self.test_case.assertTrue(await twin_get_res)
self.test_case.assertTrue(await rec_mess_cloud)
self.test_case.assertTrue(await send_mess_cloud)
self.test_case.assertTrue(await rec_dir_call)
self.test_case.assertTrue(await up_twin_prop)
def run(self):
try:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
# Getting writing the cert content to disk
open("cert.pem", "w").write(self.cert_prim)
# Creating a device
device = IOTHubDevice()
device.authenticate_device(self.device_id, "cert.pem")
# Removing cert previously created
remove("cert.pem")
asyncio.run(self.perform(device))
except Exception as e:
info(format_exc())
self.test_case.assertFalse(True)
As you've seen I'm in a thread and I want to verify some function of Azure IOTHub.
But I get this error :
RuntimeError: Task got Future attached to a different loop
So my question is : how can I get all these tasks running in the loop and get their individual results ?
Thanks for your answer !

Threaded Python TCP Client Class continuously calls Receiver method and blocks Send method

I hope the title is appropriate. If not please suggest an alternative. I am working with the following Python Client Class.
import Queue
import socket
import struct
import threading
import time
class ClientCommand(object):
CONNECT, SEND, RECEIVE, CLOSE = range(4)
def __init__(self, type, data=None):
self.type = type
self.data = data
class ClientReply(object):
ERROR, SUCCESS = range(2)
def __init__(self, type, data = None):
self.type = type
self.data = data
class SocketClientThread(threading.Thread):
def __init__(self, cmd_q = Queue.Queue(), reply_q = Queue.Queue()):
super(SocketClientThread, self).__init__()
self.cmd_q = cmd_q
self.reply_q = reply_q
self.alive = threading.Event()
self.alive.set()
self.socket = None
#self.stopped = False
self.handlers = {
ClientCommand.CONNECT: self._handle_CONNECT,
ClientCommand.CLOSE: self._handle_CLOSE,
ClientCommand.SEND: self._handle_SEND,
ClientCommand.RECEIVE: self._handle_RECEIVE
}
def run(self):
while self.alive.isSet():
#while not self.stopped:
try:
cmd = self.cmd_q.get(True, 0.1)
self.handlers[cmd.type](cmd)
except Queue.Empty as e:
continue
def stop(self):
self.alive.clear()
def join(self, timeout=None):
self.alive.clear()
threading.Thread.join(self, timeout)
def _handle_CONNECT(self, cmd):
try:
self.socket = socket.socket(
socket.AF_INET, socket.SOCK_STREAM)
self.socket.connect((cmd.data[0], cmd.data[1]))
self.reply_q.put(self._success_reply())
except IOError as e:
self.reply_q.put(self._error_reply(str(e)))
def _handle_CLOSE(self, cmd):
self.socket.close()
reply = ClientReply(ClientReply.SUCCESS)
self.reply_q.put(reply)
def _handle_SEND(self, cmd):
try:
print "about to send: ", cmd.data
self.socket.sendall(cmd.data)
print "sending data"
self.reply_q.put(self._success_reply())
except IOError as e:
print "Error in sending"
self.reply_q.put(self._error_reply(str(e)))
def _handle_RECEIVE(self, cmd):
try:
#TODO Add check for len(data)
flag = True
while flag:
print "Receiving Data"
data = self._recv_n_bytes()
if len(data) != '':
self.reply_q.put(self._success_reply(data))
if data == "Stop":
print "Stop command"
flag = False
except IOError as e:
self.reply_q.put(self._error_reply(str(e)))
def _recv_n_bytes(self):
data = self.socket.recv(1024)
return data
def _error_reply(Self, errstr):
return ClientReply(ClientReply.ERROR, errstr)
def _success_reply(self, data = None):
return ClientReply(ClientReply.SUCCESS, data)
My main script code -
import socket
import time
import Queue
import sys
import os
from client import *
sct = SocketClientThread()
sct.start()
host = '127.0.0.1'
port = 1234
sct.cmd_q.put(ClientCommand(ClientCommand.CONNECT, (host, port)))
try:
while True:
sct.cmd_q.put(ClientCommand(ClientCommand.RECEIVE))
reply = sct.reply_q
tmp = reply.get(True)
data = tmp.data
if data != None:
if data != "step1":
//call function to print something
else:
// call_function that prints incoming data till server stops sending data
print "Sending OK msg"
sct.cmd_q.put(ClientCommand(ClientCommand.SEND, "Hello\n"))
print "Done"
else:
print "No Data"
except:
#TODO Add better error handling than a print
print "Server down"
So here is the issue. Once the thread starts, and the Receive handler is called, I get some data, if that data is not "Step1", I just call a function (another script) to print it.
However, if the data is "step1", I call a function which will then continue printing whatever data the server sends next, till the server sends a "Stop" message. At this point, I break out of the "Receive Handler", and try to send an "Ok" message to the Server.
However, as soon as I break out of the "Receive Handler", it automatically calls upon that function again. So while I am trying to send back a message, the client is again waiting for data from the server. So due to the "Receiver function" being called again, the "Send function" blocks.
I can't seem to understand how to switch between receiving and sending. What is wrong with my approach here and how should I fix this? Do I need to re-write the code to have two separate threads for sending and receiving?
If you require any more details please let me know before you decide to flag my question for no reason.

However, as soon as I break out of the "Receive Handler", it
automatically calls upon that function again.
This is because you call sct.cmd_q.put(ClientCommand(ClientCommand.RECEIVE)) within the while True loop that's run through for each single chunk of data received, i. e. for each data before "step1" one more command to call the "Receive Handler" (which itself loops until "Stop") is put into the ClientCommand queue, and those commands are of course then executed before the SEND command. If you place the RECEIVE call before this while True loop, your approach can work.

The error is
if msgid != "step1":
NameError: name 'msgid' is not defined
Instead of
#TODO Add better error handling than a print
print "Server down"
you had better written
raise
and spotted it immediately.

Python, re-entering the `with` block

I am currently working on an import script that import listings from a database that regularly shuts down every 15 mins for re-snap.
I have created a with block as below to look after the retry mechanism when creating connections:
class DBRetryController(object):
conn_obj = None
connection = None
cursor = None
retry_count_down = None
sleep_time = None
def __init__(self, conn_obj, retry_count_down=5, sleep_time=10):
self.conn_obj = conn_obj
self.retry_count_down = retry_count_down
self.sleep_time = sleep_time
def __enter__(self):
ex = None
while self.retry_count_down > 0:
try:
if hasattr(self.conn_obj, '__call__'):
self.connection = self.conn_obj()
else:
self.connection = self.conn_obj
self.cursor = self.connection.cursor()
self.retry_count_down = False
except OperationalError as ex:
log.warning('Caught db error, possibly due to sql server gone away, retrying in a few moment')
self.retry_count_down -= 1
time.sleep(self.sleep_time)
if ex:
raise ex
return self.connection, self.cursor
def __exit__(self, type, value, traceback):
try:
self.cursor.close()
self.connection.close()
except:
pass
if value:
raise value
And use as below:
with DBRetryController(self.connection) as (_, cursor):
cursor.execute(self.LISTING_QUERY)
But the problem is the server can shutdown during execution of the query, is it possible to modifying the DBRetryController to make the nested block of code to re-enter?

If I understand your question correctly, I think you can use such a scheme:
notCompleted = 1
class TestClass():
def run(self):
global notCompleted
notCompleted = 1
#do_something here
notCompleted = 0
test = TestClass()
test.run()
while(notCompleted):
test.run()
Let assume that I want to be sure, even if any error occure during execution of run() method, my program will retry to finish to run it complete again. The notCompleted is 1 by default. when I call the run method at the beginning I assign 1 to it and at the end of my run method I assigned 0 to it. Anywhere inside the run() if I have any problem, in the while loop the function will called again.
I think you need to add a Try...Catch too.

Multithread proxy change but once

Let say I have a code like this:
def func1(a,b,c):
try:
p = pycurl.Curl()
p.setopt(pycurl.PROXY, "127.0.0.1")
p.setopt(pycurl.PROXYPORT, 9050)
p.setopt(pycurl.PROXYTYPE, pycurl.PROXYTYPE_SOCKS5)
p.perform()
p.close()
except pycurl.error as error:
if error[0] == 28: # timeout - change proxy
print "Tor timeout, need to change"
queue.put((a,b,c))
new_tor()
return
def new_tor():
# send_signal_for_new_ident_is_here
I start this code in 7 threads.
And when a thread receives error 28 it change the identify.
But it happens that ALL 7 THREADS sending signal to change identify.
How to do this:
If thread received error 28, then it calls new_tor() and other 6 threads don't but waiting for result and only then they proceed to work. How to sync this?

Just put an error "id" into the queue and if you encounter it, put the value back into the queue, and then handle as needed.
You don't wish to end the thread, which is what I did.
So, you can have some unique identifier for each thread, such that if once a thread encounters an error, it also adds the data (it's identifier) that says it encountered this error before, so that if all threads have encountered this error, the error is removed from the queue.
Code:
import threading
import Queue
y = 0
def f1():
global y
y += 1
if y > 100:
raise ValueError('trial')
def f2():
return
class Test(threading.Thread):
def __init__(self, func, name):
threading.Thread.__init__(self)
self.func = func
self.name = name
def run(self):
while True:
x = ''
if not queue.empty():
x = queue.get()
if x == 'error':
queue.put(x)
print 'Stopping %s' % (self.name,)
return
try:
self.func()
except Exception as e:
queue.put('error')
queue = Queue.Queue()
thread1 = Test(f1, '1')
thread2 = Test(f2, '2')
thread1.start()
thread2.start()