Develop Reference

Why can't I access a specific variable inside of a threaded class

The bounty expires in 5 days. Answers to this question are eligible for a +50 reputation bounty.
Haley Mueller wants to draw more attention to this question.
I'm new to Python so this could be a simple fix.
I am using Flask and sockets for this Python project. I am starting the socket on another thread so I can actively listen for new messages. I have an array variable called 'SocketConnections' that is within my UdpComms class. The variable gets a new 'Connection' appended to it when a new socket connection is made. That works correctly. My issue is that when I try to read 'SocketConnections' from outside of the thread looking in, it is an empty array.
server.py
from flask import Flask, jsonify
import UdpComms as U
app = Flask(__name__)
#app.route('/api/talk', methods=['POST'])
def talk():
global global_server_socket
apples = global_server_socket.SocketConnections
return jsonify(message=apples)
global_server_socket = None
def start_server():
global global_server_socket
sock = U.UdpComms(udpIP="127.0.0.1", portTX=8000, portRX=8001, enableRX=True, suppressWarnings=True)
i = 0
global_server_socket = sock
while True:
i += 1
data = sock.ReadReceivedData() # read data
if data != None: # if NEW data has been received since last ReadReceivedData function call
print(data) # print new received data
time.sleep(1)
if __name__ == '__main__':
server_thread = threading.Thread(target=start_server)
server_thread.start()
app.run(debug=True,host='192.168.0.25')
UdpComms.py
import json
import uuid
class UdpComms():
def __init__(self,udpIP,portTX,portRX,enableRX=False,suppressWarnings=True):
self.SocketConnections = []
import socket
self.udpIP = udpIP
self.udpSendPort = portTX
self.udpRcvPort = portRX
self.enableRX = enableRX
self.suppressWarnings = suppressWarnings # when true warnings are suppressed
self.isDataReceived = False
self.dataRX = None
# Connect via UDP
self.udpSock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # internet protocol, udp (DGRAM) socket
self.udpSock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) # allows the address/port to be reused immediately instead of it being stuck in the TIME_WAIT state waiting for late packets to arrive.
self.udpSock.bind((udpIP, portRX))
# Create Receiving thread if required
if enableRX:
import threading
self.rxThread = threading.Thread(target=self.ReadUdpThreadFunc, daemon=True)
self.rxThread.start()
def __del__(self):
self.CloseSocket()
def CloseSocket(self):
# Function to close socket
self.udpSock.close()
def SendData(self, strToSend):
# Use this function to send string to C#
self.udpSock.sendto(bytes(strToSend,'utf-8'), (self.udpIP, self.udpSendPort))
def SendDataAddress(self, strToSend, guid):
# Use this function to send string to C#
print('finding connection: ' + guid)
if self.SocketConnections:
connection = self.GetConnectionByGUID(guid)
print('found connection: ' + guid)
if connection is not None:
self.udpSock.sendto(bytes(strToSend,'utf-8'), connection.Address)
def ReceiveData(self):
if not self.enableRX: # if RX is not enabled, raise error
raise ValueError("Attempting to receive data without enabling this setting. Ensure this is enabled from the constructor")
data = None
try:
data, _ = self.udpSock.recvfrom(1024)
print('Socket data recieved from: ', _)
if self.IsNewConnection(_) == True:
print('New socket')
self.SendDataAddress("INIT:" + self.SocketConnections[-1].GUID, self.SocketConnections[-1].GUID)
data = data.decode('utf-8')
except WindowsError as e:
if e.winerror == 10054: # An error occurs if you try to receive before connecting to other application
if not self.suppressWarnings:
print("Are You connected to the other application? Connect to it!")
else:
pass
else:
raise ValueError("Unexpected Error. Are you sure that the received data can be converted to a string")
return data
def ReadUdpThreadFunc(self): # Should be called from thread
self.isDataReceived = False # Initially nothing received
while True:
data = self.ReceiveData() # Blocks (in thread) until data is returned (OR MAYBE UNTIL SOME TIMEOUT AS WELL)
self.dataRX = data # Populate AFTER new data is received
self.isDataReceived = True
# When it reaches here, data received is available
def ReadReceivedData(self):
data = None
if self.isDataReceived: # if data has been received
self.isDataReceived = False
data = self.dataRX
self.dataRX = None # Empty receive buffer
if data != None and data.startswith('DIALOG:'): #send it info
split = data.split(':')[1]
return data
class Connection:
def __init__(self, gUID, address) -> None:
self.GUID = gUID
self.Address = address
def IsNewConnection(self, address):
for connection in self.SocketConnections:
if connection.Address == address:
return False
print('Appending new connection...')
connection = self.Connection(str(uuid.uuid4()),address)
self.SocketConnections.append(connection)
return True
def GetConnectionByGUID(self, guid):
for connection in self.SocketConnections:
if connection.GUID == guid:
return connection
return None
As mentioned above. When IsNewConnection() is called in UdpComms it does append a new object to SocketConnections. It's just trying to view the SocketConnections in the app.route that is empty. My plans are to be able to send socket messages from the app.routes

For interprocess communication you may try to use something like shared memory documented here
Instead of declaring your self.SocketConnections as a list = []
you'd use self.SocketConnections = Array('i', range(10)) (you are then limited to remembering only 10 connections though).

Opening 2 serial ports simultaneously in python (one tx one for rx)

I am making a throughput test for a bluetooth link, and I need to send data through a serial port to one bluetooth device which will then transport that data wirelessly to another bluetooth device. The other device will then complete the circuit by sending the data back to the host PC via a different serial port.
The problem seems to be when I attempt to open up 2 different instances of PySerial, the program simply hangs. I have isolated it down to running vs. hanging when I comment out one of the two serial port instantiations. Anyone see a problem with how I'm doing this? If so, what is the proper way to do this? See code below:
#/usr/bin/python
import serial
import time
import sys
DEFAULT_BAUD = 115200
SEND_SIZE = 100
def addPath(file):
pth, fl = os.path.split(__file__)
return os.path.join(pth, file)
def is_number(s):
try:
int(s, 16)
return True
except:
return False
class SerialReader():
def __init__(self, portRx, portTx):
self.portTx = portTx
self.portRx = portRx
self.start_time__sec = time.time()
self.interval__sec = 0
self.buffer = []
self.sendtext = ''.join([str(i) for i in range(SEND_SIZE)])
# send first batch of data
self.portTx.write(self.sendtext)
def didDataArrive(self):
# Read port
self.buffer.extend(list(self.portRx.read(1024)))
# Step through the buffer byte and byte and see if the tick text
# is at the front.
while len(self.buffer) >= len(self.sendtext):
if self.buffer[:len(self.sendtext)] == self.sendtext:
# Discard the tick text
self.buffer = self.buffer[len(self.sendtext):]
# Record time
snapshot__sec = time.time()
self.interval__sec = snapshot__sec - self.start_time__sec
self.start_time__sec = snapshot__sec
# send more data
self.portTx.write(self.sendtext)
return True
else:
self.buffer.pop(0)
return False
def main(port1, port2, baudrate1 = DEFAULT_BAUD, baudrate2 = DEFAULT_BAUD):
try:
import serial
except:
traceback.print_exc()
print "="*60
print "You need to install PySerial"
print "Windows: easy_install pyserial"
print "Mac/Linux: sudo easy_install pyserial"
try:
s1 = serial.Serial(port1, baudrate1, timeout = 0.1)
s2 = serial.Serial(port2, baudrate2, timeout = 0.1)
print "Loading serial ports"
except:
print "Serial port error"
exit()
plot_stop = False
dataread = SerialReader(s2, s1)
try:
while plot_stop == False:
if dataread.didDataArrive():
print dataread.interval__sec
except KeyboardInterrupt:
print "Keyboard Interrupt"
plot_stop = True
finally:
print "Closing"
s1.close()
s2.close()
if __name__ == '__main__':
if (len(sys.argv) < 3):
print "Usage: python extract_data.py phonelink_serialport phonelinkclient_serialport [baudrate1] [baudrate2]"
else:
main(*sys.argv[1:])
If I remove one of the following lines (doesn't matter which one), the python script runs (although it eventually crashes because in the code it eventually tries to reference both ports). If I leave these lines in, the program seems to just hang (it just seems to sit there and run indefinitely):
s1 = serial.Serial(port1, baudrate1, timeout = 0.1)
s2 = serial.Serial(port2, baudrate2, timeout = 0.1)

Python - write() adding content I didn't expect

I am playing with the file I/O functions, and I am having issues writing to a file.
To get a feel for it, I have either run a FOR loop on a range, adding each to a new line, or done the same for a list. Either way, I get the following appended to the file after the loop:
98
99
is dropped.
"""
global quitting
try:
raise
except SystemExit:
raise
except EOFError:
global exit_now
exit_now = True
thread.interrupt_main()
except:
erf = sys.__stderr__
print>>erf, '\n' + '-'*40
print>>erf, 'Unhandled server exception!'
print>>erf, 'Thread: %s' % threading.currentThread().getName()
print>>erf, 'Client Address: ', client_address
print>>erf, 'Request: ', repr(request)
traceback.print_exc(file=erf)
print>>erf, '\n*** Unrecoverable, server exiting!'
print>>erf, '-'*40
quitting = True
thread.interrupt_main()
class MyHandler(rpc.RPCHandler):
def handle(self):
"""Override base method"""
executive = Executive(self)
self.register("exec", executive)
self.console = self.get_remote_proxy("console")
sys.stdin = PyShell.PseudoInputFile(self.console, "stdin",
IOBinding.encoding)
sys.stdout = PyShell.PseudoOutputFile(self.console, "stdout",
IOBinding.encoding)
sys.stderr = PyShell.PseudoOutputFile(self.console, "stderr",
IOBinding.encoding)
# Keep a reference to stdin so that it won't try to exit IDLE if
# sys.stdin gets changed from within IDLE's shell. See issue17838.
self._keep_stdin = sys.stdin
self.interp = self.get_remote_proxy("interp")
rpc.RPCHandler.getresponse(self, myseq=None, wait=0.05)
def exithook(self):
"override SocketIO method - wait for MainThread to shut us down"
time.sleep(10)
<ad nauseum>
The code for creating this is:
f = open('test.txt', 'w+')
for x in range(100):
f.write((str(x) + '\n'))
f.read()
But even if I close it and open the file itself, this stuff is appended.
How can I just write the data to the file without this extra stuff?

How do I recover from a serialException using pySerial

I have an application that reads and transmits data to a device connected via USB. I'm using pySerial to facilitate this communication. Everything works fine until the USB cable is unplugged from the PC and an exception is thrown. Once the cable is plugged back in, I can't seem to recover and reconnect to my device. The only way for me to recover is to close down the application and unplug and plug the cable in again. Any help in understanding what's going on would be much appreciated.
This is basic test code that I'm useing to help me understand the process.
# Class used to communicate with USB Dongle
import serial
import time
import sys
class LPort:
def __init__(self, port=0):
"initialize the LPort class"
self.error = ""
self.traffic = ""
self.dest = None
if port == None:
self.simulation = True
else:
self.simulation = False
self.port = port # serial port we should use
self.reset()
self.time = time.time()
def reInit(self):
self.close()
def reset(self):
"flush port, reset the LPort, initialize LPort"
if self.simulation:
r = "LPort simulator"
else:
self.port.flushInput()
self.port.flushOutput()
self.fail = False
self.command("/H1")
self.dest = None
r = "reset"
self.error = ""
self.traffic = ""
return r
def status(self):
"return accumulated status info, reset collection"
s = self.error
self.error = ""
return s
def data(self):
"return accumulated traffic data, reset collection"
s = self.traffic
self.traffic = ""
return s
def set_dest(self, addr):
"set the destination address (if necessary)"
if addr != self.dest:
self.dest = addr
self.command("/O")
r = self.command("/D%02X" % addr)
if r != "*":
self.dest = None
self.error += r
else:
r = True
return r
def checksum(self, bytes):
"calculate the CRC-8 checksum for the given packet"
crc_table = [
# this table is taken from the CP rectifier code
0x00,0x07,0x0E,0x09,0x1C,0x1B,0x12,0x15,0x38,0x3F,
0x36,0x31,0x24,0x23,0x2A,0x2D,0x70,0x77,0x7E,0x79,
0x6C,0x6B,0x62,0x65,0x48,0x4F,0x46,0x41,0x54,0x53,
0x5A,0x5D,0xE0,0xE7,0xEE,0xE9,0xFC,0xFB,0xF2,0xF5,
0xD8,0xDF,0xD6,0xD1,0xC4,0xC3,0xCA,0xCD,0x90,0x97,
0x9E,0x99,0x8C,0x8B,0x82,0x85,0xA8,0xAF,0xA6,0xA1,
0xB4,0xB3,0xBA,0xBD,0xC7,0xC0,0xC9,0xCE,0xDB,0xDC,
0xD5,0xD2,0xFF,0xF8,0xF1,0xF6,0xE3,0xE4,0xED,0xEA,
0xB7,0xB0,0xB9,0xBE,0xAB,0xAC,0xA5,0xA2,0x8F,0x88,
0x81,0x86,0x93,0x94,0x9D,0x9A,0x27,0x20,0x29,0x2E,
0x3B,0x3C,0x35,0x32,0x1F,0x18,0x11,0x16,0x03,0x04,
0x0D,0x0A,0x57,0x50,0x59,0x5E,0x4B,0x4C,0x45,0x42,
0x6F,0x68,0x61,0x66,0x73,0x74,0x7D,0x7A,0x89,0x8E,
0x87,0x80,0x95,0x92,0x9B,0x9C,0xB1,0xB6,0xBF,0xB8,
0xAD,0xAA,0xA3,0xA4,0xF9,0xFE,0xF7,0xF0,0xE5,0xE2,
0xEB,0xEC,0xC1,0xC6,0xCF,0xC8,0xDD,0xDA,0xD3,0xD4,
0x69,0x6E,0x67,0x60,0x75,0x72,0x7B,0x7C,0x51,0x56,
0x5F,0x58,0x4D,0x4A,0x43,0x44,0x19,0x1E,0x17,0x10,
0x05,0x02,0x0B,0x0C,0x21,0x26,0x2F,0x28,0x3D,0x3A,
0x33,0x34,0x4E,0x49,0x40,0x47,0x52,0x55,0x5C,0x5B,
0x76,0x71,0x78,0x7F,0x6A,0x6D,0x64,0x63,0x3E,0x39,
0x30,0x37,0x22,0x25,0x2C,0x2B,0x06,0x01,0x08,0x0F,
0x1A,0x1D,0x14,0x13,0xAE,0xA9,0xA0,0xA7,0xB2,0xB5,
0xBC,0xBB,0x96,0x91,0x98,0x9F,0x8A,0x8D,0x84,0x83,
0xDE,0xD9,0xD0,0xD7,0xC2,0xC5,0xCC,0xCB,0xE6,0xE1,
0xE8,0xEF,0xFA,0xFD,0xF4,0xF3]
for i in range(len(bytes)):
b = int(bytes[i])
if i == 0: chksum = crc_table[b]
else: chksum = crc_table[chksum ^ b]
return chksum
def command(self, cmd):
"transmit distinct commands to unit, and accept response"
if self.simulation:
r = "*"
else:
try:
self.port.write(cmd + chr(13))
except serial.serialutil.SerialTimeoutException:
r = "/TO"
return r
except:
print "Unexpected error:", sys.exc_info()[0]
r = "/Unknown"
return r
r = ""
eol = False
while True:
c = self.port.read(1)
if not c:
r = "/FAIL " + r + " " + cmd
self.error = r
break
else:
r += c
ordc = ord(c)
if ordc == 13 or ordc == 42:
break
return r
def checkRawDataForErrors(self, raw, errors = []):
errorCodes = {'/SNA':'Slave Not Acknowledging',
'/I81':'Busy, Command Ignored',
'/I88':'Connection Not Open',
'/I89':'Invalid Command Argument',
'/I8A':'Transmit Not Active',
'/I8F':'Invalid Command',
'/I90':'Buffer Overflow',
'/DAT':'Data Error',
'/BADPEC':'Bad PEC Value',
'/NO_MRC':'No Master Read Complete Signal',
'/FAIL':'General Failure',
'/LEN':'Data Length Error'}
for ekey, eval in errorCodes.items():
if ekey in raw:
errors.append(eval)
return errors
# self-testing module
if __name__ == "__main__":
com = serial.Serial(port=4, baudrate=115200, timeout=1, xonxoff=0)
if com:
port = LPort(com)
print port
time.sleep(5)
port = LPort(com)
print "/V =", port.command("/V")
print "/V", port.data(), port.status()
print "/O =", port.command("/O")
print "/O", port.data(), port.status()
print "/A =", port.command("/A")
print "/A", port.data(), port.status()
print "/L =", port.command("/L")
print "/L", port.data(), port.status()
com.close()
else:
print "cannot open com port"
UPDATE:
The following is the code around the creatfile() in serialwin32.py which returns the following message:
serial.serialutil.SerialException: could not open port COM5: [Error 2] The system cannot find the file specified.
self.hComPort = win32.CreateFile(port,
win32.GENERIC_READ | win32.GENERIC_WRITE,
0, # exclusive access
None, # no security
win32.OPEN_EXISTING,
win32.FILE_ATTRIBUTE_NORMAL | win32.FILE_FLAG_OVERLAPPED,
0)
if self.hComPort == win32.INVALID_HANDLE_VALUE:
self.hComPort = None # 'cause __del__ is called anyway
raise SerialException("could not open port %s: %s" % (self.portstr, ctypes.WinError()))

Assuming your device is well-behaved, all you must do is this:
close your serial port (serial.Serial instance)
find the COMX name of your port again
open the serial port
The 2nd part is problematic because Windows tries to be clever. In your case the following happens:
USB device is connected and is assigned name COM2
Your program opens the device
USB disconnects
USB reconnects quickly before your program noticed that device died
Windows sees that COM2 is busy and assigns a different name to this USB device
(optional) your program closes the device
your program tries to open COM2 again, but there's no hardware at that name
The are way to get around Windows being clever -- you can specifically assign fixed COMX name to this device in Device Manager, COM ports, your port, advanced options.
Another option is to detect device dying very fast and closing the file handle. If you are lucky then by the time device reconnects original COM2 is free again.
Yet another option is to use a USB-serial converter from another manufacturer that uses another driver. Somehow COMX letter assignment is driver-specific. Better drivers may give you a stable name.

I've come across this problem as well. Sometimes my program has locked up when the device is plugged in again.
NB. I have fixed the COMx name of the port as mentioned by #qarma
I've rearranged my program so that as soon as an exception is thrown from the read() or write() methods of Serial I stop calling those methods.
I then have a function which periodically retries opening the port to try to detect when the device has been plugged in again.
This function creates a new instance of Serial with the same parameters as the original and tries to open it:
def try_to_open_new_port(self):
ret = False
test = serial.Serial(baudrate=9600, timeout=0, writeTimeout=0)
test.port = self.current_port_name
try:
test.open()
if test.isOpen():
test.close()
ret = True
except serial.serialutil.SerialException:
pass
return ret
A return of True indicates that the port is present once again.

Python pyserial and runaway CPU usage

MacOS 10.7.3, python 2.5
I am using pyserial to open a connection to an external server. The connection is opened as:
HOST = '10.0.0.1'
PORT = '16010'
theURL = 'socket://' + HOST + ':' + PORT
conn = serial.serial_for_url(theURL, baudrate=115200)
conn.timeout = 2
and then the read looks like this:
try:
while len(rawData) == 0 and self.shutdown == False:
rawData = conn.readline()
except:
some error handling code...
The problem is, if I kill the server at 10.0.0.1:16010 the code keeps running, but the cpu usage goes to 100%. No error is thrown, so the except is never entered.
This looks like a problem in pyserial, but maybe someone here has run into this before and knows how to detect the lost connection so the situation can be handled gracefully.
Thanks.

If you're not depending on .readline() you could do it like this:
self.plugin.conn = Serial(..., timeout = 1)
...
if not self.shutdown:
rawData = self.plugin.conn.read(1)
if rawData:
rawData += self.plugin.conn.read(self.plugin.conn.inWaiting())
else:
raise Exception("timeout")
I'm not sure if I got your intent right so you might have to adjust...

The fact that your CPU usage is pegged probably indicates the the readline call is not blocking to timeout but returns instantly. So if your normal timeout
is 2, you could use:
from time import time
try:
while len(rawData) == 0 and self.shutdown == False:
before = time()
rawData = conn.readline()
if (len(rawData)==0) and ((time()-before)<2):
raise Exception("Early readline return.")
except:
some error handling code...

A very good solution to this can be found here:
class ReadLine:
def __init__(self, s):
self.buf = bytearray()
self.s = s
def readline(self):
i = self.buf.find(b"\n")
if i >= 0:
r = self.buf[:i+1]
self.buf = self.buf[i+1:]
return r
while True:
i = max(1, min(2048, self.s.in_waiting))
data = self.s.read(i)
i = data.find(b"\n")
if i >= 0:
r = self.buf + data[:i+1]
self.buf[0:] = data[i+1:]
return r
else:
self.buf.extend(data)
ser = serial.Serial('COM7', 9600)
rl = ReadLine(ser)
while True:
print(rl.readline())