I want to trigger a read event whenever there is data to be read from a serial port using pySerial. The pySerial module has some threading functionality for that, but it isn't well documented (only example is the testing example present).
Currently I am reading using a blocking mechanism and the code looks like this:
def read_serial_port(ser, output_value = b''):
time.sleep(1)
while True:
new_read = b''
if ser.in_waiting > 0:
new_read = ser.read(ser.in_waiting)
output_value += new_read
else:
break
#blocking to let other threads run and
#also allow buffer to collect data while waiting
time.sleep(1.5)
if output_value:
output_value = output_value.decode()
output_value = strip_ansi(output_value)
else:
print("Sorry, the read from serial port was empty")
return output_value
Running this helps me read most of the data but in cases where there is a larger interval in data being outputted from the serial port I miss the read? Is there a "recommended" way to do this as by using the functionality of pySerial itself?
Related
I am a Python newbie and my first task is to create a small server program that will forward events from a network unit to a rest api.
The overall structure of my code seems to work, but I have one problem. After I receive the first package, nothing happens. Is something wrong with my loop such that new packages (from the same client) aren't accepted?
Packages look something like this: EVNTTAG 20190219164001132%0C%3D%E2%80h%90%00%00%00%01%CBU%FB%DF ... not that it matters, but I'm sharing just for clarity.
My code (I skipped the irrelevant init of rest etc. but the main loop is the complete code):
# Configure TAGP listener
ipaddress = ([l for l in ([ip for ip in socket.gethostbyname_ex(socket.gethostname())[2] if not ip.startswith("127.")][:1], [[(s.connect(('8.8.8.8', 53)), s.getsockname()[0], s.close()) for s in [socket.socket(socket.AF_INET, socket.SOCK_DGRAM)]][0][1]]) if l][0][0])
server_name = ipaddress
server_address = (server_name, TAGPListenerPort)
print ('starting TAGP listener on %s port %s' % server_address)
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(server_address)
sock.listen(1)
sensor_data = {'tag': 0}
# Listen for TAGP data and forward events to ThingsBoard
try:
while True:
data = ""
connection, client_address = sock.accept()
data = str(connection.recv(1024))
if data.find("EVNTTAG") != -1:
timestamp = ((data.split())[1])[:17]
tag = ((data.split())[1])[17:]
sensor_data['tag'] = tag
client.publish('v1/devices/me/telemetry', json.dumps(sensor_data), 1)
print (data)
except KeyboardInterrupt:
# Close socket server (TAGP)
connection.shutdown(1)
connection.close()
# Close client to ThingsBoard
client.loop_stop()
client.disconnect()
There are multiple issues with your code:
First of all you need a loop over what client sends. So you first connection, client_address = sock.accept() and you now have a client. But in the next iteration of the loop you do .accept() again overwriting your old connection with a new client. If there is no new client this simply waits forever. And that's what you observe.
So this can be fixed like this:
while True:
conn, addr = sock.accept()
while True:
data = conn.recv(1024)
but this code has another issue: no new client can connect until the old one disconnects (well, at the moment it just loops indefinitly regardless of whether the client is alive or not, we'll deal with it later). To overcome it you can use threads (or async programming) and process each client independently. For example:
from threading import Thread
def client_handler(conn):
while True:
data = conn.recv(1024)
while True:
conn, addr = sock.accept()
t = Thread(target=client_handler, args=(conn,))
t.start()
Async programming is harder and I'm not gonna address it here. Just be aware that there are multiple advantages of async over threads (you can google those).
Now each client has its own thread and the main thread only worries about accepting connections. Things happen concurrently. So far so good.
Let's focus on the client_handler function. What you misunderstand is how sockets work. This:
data = conn.recv(1024)
does not read 1024 bytes from the buffer. It actually reads up to 1024 bytes with 0 being possible as well. Even if you send 1024 bytes it can still read say 3. And when you receive a buffer of length 0 then this is an indication that the client disconnected. So first of all you need this:
def client_handler(conn):
while True:
data = conn.recv(1024)
if not data:
break
Now the real fun begins. Even if data is nonempty it can be of arbitrary length between 1 and 1024. Your data can be chunked and may require multiple .recv calls. And no, there is nothing you can do about it. Chunking can happen due to some other proxy servers or routers or network lag or cosmic radiation or whatever. You have to be prepared for it.
So in order to work with that correctly you need a proper framing protocol. For example you have to somehow know how big is the incoming packet (so that you can answer the question "did I read everything I need?"). One way to do that is by prefixing each frame with (say) 2 bytes that combine into total length of the frame. The code may look like this:
def client_handler(conn):
while True:
chunk = conn.recv(1) # read first byte
if not chunk:
break
size = ord(chunk)
chunk = conn.recv(1) # read second byte
if not chunk:
break
size += (ord(chunk) << 8)
Now you know that the incoming buffer will be of length size. With that you can loop to read everything:
def handle_frame(conn, frame):
if frame.find("EVNTTAG") != -1:
pass # do your stuff here now
def client_handler(conn):
while True:
chunk = conn.recv(1)
if not chunk:
break
size = ord(chunk)
chunk = conn.recv(1)
if not chunk:
break
size += (ord(chunk) << 8)
# recv until everything is read
frame = b''
while size > 0:
chunk = conn.recv(size)
if not chunk:
return
frame += chunk
size -= len(chunk)
handle_frame(conn, frame)
IMPORTANT: this is just an example of handling a protocol that prefixes each frame with its length. Note that the client has to be adjusted as well. You either have to define such protocol or if you have a given one you have to read the spec and try to understand how framing works. For example this is done very differently with HTTP. In HTTP you read until you meet \r\n\r\n which signals the end of headers. And then you check Content-Length or Transfer-Encoding headers (not to mention hardcore things like protocol switch) to determine next action. This gets quite complicated though. I just want you to be aware that there are other options. Nevertheless framing is necessary.
Also network programming is hard. I'm not gonna dive into things like security (e.g. against DDOS) and performance. The code above should be treated as extreme simplification, not production ready. I advice using some existing soft.
I've a target board which sends me CPU load for every 1sec which communicates through serial port. I want to write a python code which reads data from serial port for every 1sec second and writes to a file and also it should check for user input i.e. if user enters command it should pass that command to target board.
You can create two threads, one listening for user input, the other polling the serial port every second. Check out threading for information on multithreading in python and pyserial for information on reading from serial ports. The threading package provides your desired repetition functionality: https://docs.python.org/2/library/threading.html#timer-objects
[EDIT]
Sample code, obviously replace /dev/ttyS1 with the desired port and do something useful with the user input:
import serial
import threading
def read_from_serial():
# port = "/dev/ttyS1"
# ser = serial.Serial(port, 19200, timeout = 1)
# print ser.readline()
# ser.close()
print "read"
threading.Timer(1, read_from_serial).start()
if __name__ == "__main__":
serial_thread = threading.Thread(target = read_from_serial())
serial_thread.start()
while True:
print raw_input("User input: ")
I have trouble reading an arbitrary port with pySerial. I can only read the port when the port hasn't been used by any program after it's turned on. Otherwise, I can open the port but ser.inWaiting() returns 0, telling me there is no data in the port buffer. Actually, I can see data in a port monitor, so why does pySerial gets no data? I am using os x 10.9.5., python 2.7.8 and pySerial 2.7
Here is the code:
def usb():
ser = serial.Serial('/dev/tty.usbmodem422651', 115200)
try:
while True:
print ser.read(1000)
except KeyboardInterrupt:
ser.close()
exit
if __name__ == "__main__":
testUSB()
Thank you for your help!
You need to understand how serial works.
If you already opened some terminal and read the data, then it's the input buffer of the serial port is now empty.
while True:
print ser.read(1000)
It's not a good practice, you can try the following using inWating:
if ser.inWaiting:
ser.read(len(ser.inWaiting()))
You cannot open the port from 2 different apps or software.
A good code will look like:
import serial
def my_usb():
s_handle = serial.Serial('/dev/tty.usbmodem422651', 115200)
# or any other end condition
while True:
try:
if s_handle.inWaiting():
print s_handle.read(s_handle.inWaiting())
except KeyboardInterrupt:
print "exiting...."
finally:
s_handle.close()
if __name__ == "__main__":
my_usb()
In Python how do I fill a buffer with lines of data (strings) and consume it with a second process? There are ample of examples here adding and reading lines from a string, but I need to remove the consumed line from the string for the string to work as a buffer.
Example: read sporadic data from a serial port and send it via TCP/IP to a server. Line-by-line within one loop and no buffering = no problem, but in case the destination is unreachable the data should be stored in the buffer and then sent once connection is available.
#!/usr/bin/python
import serial
import socket
from multiprocessing import Process
ip = "someURL"
port = 12345
ser = serial.Serial("/dev/ttyUSB0", 57600, timeout=0)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def serial_reader():
while True:
for line in ser.read():
try:
response = ser.readlines(None)
response = str(response)
message = response[7:]
except:
print datetime.datetime.now(), " No data from serial connection."
##
def data_sender():
s.connect((ip, port))
while True:
for line in queue():
try:
s.send(message)
except:
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((ip, port))
continue
except:
s.close()
##
if __name__ == '__main__':
Process(target=serial_reader).start()
Process(target=data_sender).start()
I think the best way to achieve what you want is to use a queue:
from multiprocessing import Queue
specifically use queue.put() to put a string on the queue, queue.get() to retrieve it, and queue.task_done() to indicate that the task is complete.
https://docs.python.org/2/library/queue.html#Queue.Queue
if you need a bigger gun take a look at RabbitMQ and python libraries that implement the AMPQ protocol such as rabbitpy. This is the defacto standard for inter process/inter service communication and has a lot of usefyl stuff already baked in, such as persisting messages in case the processes shut down, load balancing tasks across multiple processes, etc.
I have Python 2.7.4 and pyserial-2.5 win32 installed on my PC. Here I am using a microcontroller device as a master (primary) and my pc as a slave (secondary). Here every time microcontroller will transmit data, and my PC has to receive the data through serial port. I want a code in Python to receive continuous data. Here the transmitted data size will vary all the time. Here I wrote a code to transmit data, and the code is
import serial
ser= serial.serial("COM10", 9600)
ser.write("Hello world\n")
x = ser.readline()
print(x)
With this code I can transmit data to the other PC, I crosschecked by opening HyperTerminal on the other PC and I can see the transmitted data (hello world).
I also wrote the code to receive data:
import serial
ser=serial.serial("COM10", 9600)
while 1:
if ser.inwaiting():
val = ser.readline(ser.inwaiting())
print(val)
if I send the data (how are you) from HyperTerminal, I can receive the data in my PC, with the above code.
Until this every thing is fine.
My question now is, when the microcontroller is transmitting variable data at variable time periods, I need to receive that data in my PC with Python. Do I need to use a buffer to store the received data? If yes, how will the code be? Why and how to use a buffer in Python? According to my search in internet, buffer is used to slice the string.
Typically what you do for communicating with a micro is to use single characters for something lightweight or create a communication protocol. Basically you have a start flag, end flag, and some sort of checksum to make sure the data gets across correctly. There are many ways to do this.
The below code is for Python 3. You may have to make changes for bytes data.
# On micro
data = b"[Hello,1234]"
serial.write(data)
On the computer you would run
def read_data(ser, buf=b'', callback=None):
if callback is None:
callback = print
# Read enough data for a message
buf += ser.read(ser.inwaiting()) # If you are using threading +10 or something so the thread has to wait for more data, this makes the thread sleep and allows the main thread to run.
while b"[" not in buf or b"]" not in buf:
buf += ser.read(ser.inwaiting())
# There may be multiple messages received
while b"[" in buf and b']' in buf:
# Find the message
start = buf.find(b'[')
buf = buf[start+1:]
end = buf.find(b']')
msg_parts = buf[:end].split(",") # buf now has b"Hello, 1234"
buf = buf[end+1:]
# Check the checksum to make sure the data is valid
if msg_parts[-1] == b"1234": # There are many different ways to make a good checksum
callback(msg_parts[:-1])
return buf
running = True
ser = serial.serial("COM10", 9600)
buf = b''
while running:
buf = read_data(ser, buf)
Threading is useful if you are using a GUI. Then you can have your thread read data in the background while your GUI displays the data.
import time
import threading
running = threading.Event()
running.set()
def thread_read(ser, callback=None):
buf = b''
while running.is_set():
buf = read_data(ser, buf, callback)
def msg_parsed(msg_parts):
# Do something with the parsed data
print(msg_parsed)
ser = serial.serial("COM10", 9600)
th = threading.Thread(target=thread_read, args=(ser, msg_parsed))
th.start()
# Do other stuff while the thread is running in the background
start = time.clock()
duration = 5 # Run for 5 seconds
while running.is_set():
time.sleep(1) # Do other processing instead of sleep
if time.clock() - start > duration
running.clear()
th.join() # Wait for the thread to finish up and exit
ser.close() # Close the serial port
Note that in the threading example I use a callback which is a function that gets passed as a variable and gets called later. The other way this is done is by putting the data in a Queue and then processing the data in the Queue in a different part of the code.