I am new to Python and programming in general. I am trying to write a device driver using pyserial. I opened a thread that would read data from the device and send it to std out. On my main loop I used a function that would read instructions from std in as strings and write them to the device using a dictionary.
My program is reading my instructions but is not displaying any data that should be coming out of the device - I know its writing to the device because it crashes when I use an instruction not in my dictionary. Here is how my code is structured:
import serial
import threading
#ser is my serial object
def writeInstruction(ser):
#reads an instruction string from stdin and writes the corresponding instruction to the device
instruction = raw_input('cmd> ')
if instr == 'disable_all': defaultMode(ser)
else: ser.write(dictionaryOfInstructions[instruction])
time.sleep(.5)
def readData(ser):
# - Reads one package from the device, calculates the checksum and outputs through stdout
# - the package content (excludes the Package head, length, and checksum) as a string
while True:
packetHead = binascii.hexlify(ser.read(2))
packetLength = binascii.hexlify(ser.read(1))
packetContent = binascii.hexlify(ser.read(int(packetLength, 16) - 1))
if checkSum(packetHead + packetLength + packetContent):
print packetContent
readThread = threading.Thread (target = readData, args = ser)
readThread.start()
while True:
writeInstr(ser)
What is the proper way to handle serial objects in multi-threaded programming?
Related
i wrote a script in python for serial communication between my M5Stack Stick C (like raduino) and the raspberry pi.
all work fine. i can send "X","Y" or "Z" from raspberry py to the stick and he will reply the value (G-Force) back to the raspi! so far so good
Codes:
Python on raspy:
import serial
import time
import threading
ser = serial.Serial('/dev/rfcomm5') #init serial port
input_line = []#init input char array
def process_data(_data):
#called every time a sream is terminated by \n
#and the command string is ready to use
command = convert(_data)
print(command)
def convert(s): #convert the char list in a string
new = "" #init string to append all chars from char array
for x in s: # traverse in the string
new += str(x)
return new # return string
def processIncomingByte(inByte):#adding incoming chars to input_line
global input_line# globalize the input_line
if(inByte == '\n'):#if \n is incoming, end the chararray and release process data method
process_data(input_line)
input_line = []#reset input_line for next incoming string
elif(inByte == '\r'):
pass
else:#put all incoming chars in input_line
input_line.append(inByte)
while True:
while(ser.in_waiting > 0):#while some data is waiting to read....
processIncomingByte(ser.read())#.... process bytes whit method
ser.write(b'X\n')
time.sleep(0.5)
before the script work, i have to manually bind the m5Stak Stick-C over Blueman
to /dev/Rfcomm5. it work just fine over GUI or Console....
but now i would like to connect the stick via python to rfcomm5 (just by know the MAC adress, will be found in a config file later on...)
i startet to investigate a bit, but the more i research the more confused i am!!
i read some stuff over sockets and server-client aproaches. over a seperated script and so on....
i tested this code:
from bluetooth import *
target_name = "M5-Stick-C"
target_address = None
nearby_devices = discover_devices()
for address in nearby_devices:
if (target_name == lookup_name( address )):
target_address = address
break
if (target_address is not None):
print ("found target bluetooth device with address ", target_address)
else:
print ("could not find target bluetooth device nearby")
and indeed it found the device (just testing)!
but do i realy need to make a second script/process to connect to from my script?
is the the M5stack Stick-C the server? (i think so)
im so confused about all that stuff. i coded a lot, but never whit sockets, server-client stuff.
basically the communication (server/client?) works.
i just need to connect the device i found in the second script via macadress to rfcomm5 (or whatever rfcomm).
do i need a bluetooth socket? like in this example
https://gist.github.com/kevindoran/5428612
isnt the rfcomm the socket or am i wrong?
There are a number of layers that are used in the communication process and depending where you tap into that stack will depend what coding you need to do. The other complication is that BlueZ (the Bluetooth stack on linux) changed how it works over recent times leaving a lot of out of date information on the internet and easy for people to get confused.
With two Bluetooth devices, they need to establish a pairng. This is typically a one off provisioning step. This can be done with tools like Blueman or on the command line with bluetoothctl. Once you have a pairing established between your RPi and the M5Stack Stick, you shouldn't need to discover nearby devices again. Your script should just be able to connect if you tell it which device to connect to.
The M5Stack stick is advertising as having a Serial Port Profile (SPP). This is a layer on top of rfcomm.
There is a blog post about how this type of connection can be done with the standard Python3 installation: http://blog.kevindoran.co/bluetooth-programming-with-python-3/
My expectation is that you will only have to do the client.py on your RPi as the M5Stack Stick is the server. You will need to know its address and which port to connect on. Might be some trial and error on the port number (1 and 3 seem to be common).
Another library that I find helpful for SPP, is bluedot as it abstracts away some of the boilerplate code: https://bluedot.readthedocs.io/en/latest/btcommapi.html#bluetoothclient
So in summary, my recommendation is to use the standard Python Socket library or Bluedot. This will allow you to specify the address of the device you wish to connect to in your code and the underlying libraries will take care of making the connection and setting up the serial port (as long as you have already paired the two devices).
Example of what the above might look like with Bluedot
from bluedot.btcomm import BluetoothClient
from signal import pause
from time import sleep
# Callback to handle data
def data_received(data):
print(data)
sleep(0.5)
c.send("X\n")
# Make connection and establish serial connection
c = BluetoothClient("M5-Stick-C", data_received)
# Send initial requests
c.send("X\n")
# Cause the process to sleep until data received
pause()
Example using the Python socket library:
import socket
from time import sleep
# Device specific information
m5stick_addr = 'xx:xx:xx:xx:xx:xx'
port = 5 # This needs to match M5Stick setting
# Establish connection and setup serial communication
s = socket.socket(socket.AF_BLUETOOTH, socket.SOCK_STREAM, socket.BTPROTO_RFCOMM)
s.connect((m5stick_addr, port))
# Send and receive data
while True:
s.sendall(b'X\n')
data = s.recv(1024)
print(data)
sleep(0.5)
s.close()
I'm having difficulties getting pyserial to play nicely with a virtual port. I know this is an area which a few others have written about, but I couldn't find anything which solved my problem in those answers. Forgive me if I'm just being dense, and the solution exists ready-made elsewhere.
This is what I'm trying to achieve: I want to set up a virtual port, to which I can write data in one .py file, and from which I can then read data in another .py file. This is for the purposes of development and testing; I don't always have access to the device around which my current project is built.
This is my code so far:
dummy_serial.py
import os, pty, serial, time
master, slave = pty.openpty()
m_name = os.ttyname(master)
s_name = os.ttyname(slave)
# This tells us which ports "openpty" has happened to choose.
print("master: "+m_name)
print("slave: "+s_name)
ser = serial.Serial(s_name, 9600)
message = "Hello, world!"
encoded = message.encode("ascii")
while True:
ser.write(encoded)
time.sleep(1)
reader.py
import serial, time
# The port will change, depending on what port "openpty" (in the other file)
# happens to choose.
ser = serial.Serial("/dev/pts/1", 9600)
while True:
time.sleep(1)
incoming_bytes = ser.inWaiting()
# This print statement gives us an idea of what's going on.
print(incoming_bytes)
if incoming_bytes != 0:
data = ser.read(incoming_bytes)
print(data)
At present, dummy_serial.py seems to run okay. However, reader.py just keeps saying that there are no bytes waiting to be read, and hence reads no data.
What I would like:
An explanation of why ser.inWaiting() keeps returning 0, and a solution which makes ser.read(x) actually spit out "Hello, world!"
Or an explanation of why what I'm trying to do is fundamentally silly, and a better means of creating a writeable/readable virtual port.
I am trying to convert RS252 Ascii string data from a sensor to Modbus TCP Input/Holding registers using pymodbus Callback Sever, the server is the master reporting data when requested to a client logger, and I am not sure what I need to do to get this to work. I am currently able to read the data and log it to a csv file using this
#!/usr/bin/env python
# Log data from serial port
import argparse
import serial
import datetime
import time
import os
parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument("-d", "--device", help="device to read from", default="/dev/ttyUSB0")
parser.add_argument("-s", "--speed", help="speed in bps", default=9600, type=int)
args = parser.parse_args()
outputFilePath = os.path.join(os.path.dirname(__file__),
datetime.datetime.now().strftime("%Y-%m-%d") + ".csv")
with serial.Serial(args.device, args.speed) as ser, open(outputFilePath,'w') as outputFile:
print("Logging started. Ctrl-C to stop.")
try:
while True:
time.sleep(0.2)
x = (ser.read(ser.inWaiting()))
data = x.decode('UTF-8')
if data !="":
outputFile.write(time.strftime("%Y/%m/%d %H:%M ") + " " + data )
outputFile.flush()
except KeyboardInterrupt:
print("Logging stopped")
The string from the sensor comes out of the device as:
0.00 0.0 0.0 346.70 25.14
I need to have each piece as its own Modbus register and I am trying to use pymodbus on a Raspberry Pi Zero. The sensor updates 4 times a second and I am able to break the data into parts, I just haven;t gotten to that yet because I am not sure what I need to do in the Callback script, I am not that versed in Python yet I am I am still learning. I do have an understanding of Modbus TCP and have used it before on Arduino systems. Any help would be appreciated.
What you need is updating server, which you could use to populate the registers . You will have to focus on function def updating_writer and do the serial reads, process them and write to registers of your choice. The example is hard to read and understand in first go. I have modified the example to meet your needs. But here are some key concepts which will be handy to understand the code.
ModbusSlaveContext
BinaryPayloadBuilder
Also note, the example uses asynchronous server based on twisted, If you are new to twisted or have some constraints which will not allow you to use twisted on your target, you can achieve the same with simple threads as well. The design would be roughly like this.
Start your updating function in a separate thread
Start your TCP server at the end (blocking)
# Complete gist here --> https://gist.github.com/dhoomakethu/540b15781c62de6d1f7c318c3fc8ae22
def updating_writer(context, device, baudrate):
""" A worker process that runs every so often and
updates live values of the context. It should be noted
that there is a race condition for the update.
:param arguments: The input arguments to the call
"""
log.debug("updating the context")
log.debug("device - {}, baudrate-{}".format(device, baudrate))
data = serial_reader(device, baudrate) # Implement your serial reads, return list of floats.
if data:
# We can not directly write float values to registers, Use BinaryPayloadBuilder to convert float to IEEE-754 hex integer
for d in data:
builder.add_32bit_float(d)
registers = builder.to_registers()
context = context
register = 3 # Modbus function code (3) read holding registers. Just to uniquely identify what we are reading from /writing in to.
slave_id = 0x01 # Device Unit address , refer ModbusSlaveContext below
address = 0x00 # starting offset of register to write (0 --> 40001)
log.debug("new values: " + str(registers))
context[slave_id].setValues(register, address, registers)
Once the server is running and the values are being updated, you can use a client to read values and parse it back to float.
from pymodbus.client.sync import ModbusTcpClient as Client
from pymodbus.payload import BinaryPayloadDecoder, Endian
client = Client(<IP-ADDRESS>, port=5020)
# Each 32 bit float is stored in 2 words, so we will read 10 registers
raw_values = client.read_holding_registers(0, 10, unit=1)
if not registers.isError():
registers = raw_values.registers
decoder = BinaryPayloadDecoder.fromRegisters(registers,
wordorder=Endian.Big, byteorder=Endian.Big)
for _ in range(5):
print(decoder.decode_32bit_float())
I am trying to read frames sent to a ZigBee module plugged in the USB. Every frame gets discarded by the Python xBee package because the delimiter is 0x00 when it should be 0x7E. Actually it seems that every byte is also zero.
XCTU receives the frames perfectly.
I work with OS X, PyCharm, Python 3.4 and borrowed this code from Internet:
# Open serial port
ser = serial.Serial(PORT, BAUD_RATE)
# Create API object
xbee = ZigBee(ser,escaped=True)
# Continuously read and print packets
while True:
try:
response = xbee.wait_read_frame()
sa = hex(response['source_addr_long'][4:])
rf = hex(response['rf_data'])
datalength=len(rf)
# if datalength is compatible with two floats
# then unpack the 4 byte chunks into floats
if datalength==16:
h=struct.unpack('f',response['rf_data'][0:4])[0]
t=struct.unpack('f',response['rf_data'][4:])[0]
print (sa,' ',rf,' t=',t,'h=',h)
# if it is not two floats show me what I received
else:
print (sa,' ',rf)
except KeyboardInterrupt:
ser.close()
break
ser.close()
The program executes the xbee.wait_read_frame() call and waits there forever because no frame arrives.
I have tracked the call to "base.py" from the xBee package:
while True:
if self._callback and not self._thread_continue:
raise ThreadQuitException
if self.serial.inWaiting() == 0:
time.sleep(.01)
continue
byte = self.serial.read()
if byte != APIFrame.START_BYTE:
continue
The call to serial.read() always returns a zero.
I can't see anything wrong in code fragments you have provided. If you are reading just zeroes (are you?) from that serial port - there most likely something is wrong with serial port settings (e.g. you are reading at 115200 while data is being transmitted at 9600). What is the BAUD_RATE that you are using?
It's also could be a worth to test if you can access device with just dumb terminal.
I am reading serial data like this:
connected = False
port = 'COM4'
baud = 9600
ser = serial.Serial(port, baud, timeout=0)
while not connected:
#serin = ser.read()
connected = True
while True:
print("test")
reading = ser.readline().decode()
The problem is that it prevents anything else from executing including bottle py web framework. Adding sleep() won't help.
Changing "while True"" to "while ser.readline():" doesn't print "test", which is strange since it worked in Python 2.7. Any ideas what could be wrong?
Ideally I should be able to read serial data only when it's available. Data is being sent every 1,000 ms.
Using a separate thread is totally unnecessary. Just follow the example below for your infinite while loop instead.
I use this technique in my eRCaGuy_PyTerm serial terminal program here (search the code for inWaiting() or in_waiting).
Notes:
To check your python3 version, run this:
python3 --version
My output when I first wrote and tested this answer was Python 3.2.3.
To check your pyserial library (serial module) version, run this--I first learned this here:
python3 -c 'import serial; \
print("serial.__version__ = {}".format(serial.__version__))'
This simply imports the serial module and prints its serial.__version__ attribute.
My output as of Oct. 2022 is: serial.__version__ = 3.5.
If your pyserial version is 3.0 or later, use property in_waiting in the code below. If your pyserial version is < 3.0, use function inWaiting() in the code below. See the official pyserial documentation here: https://pyserial.readthedocs.io/en/latest/pyserial_api.html#serial.Serial.in_waiting.
Non-blocking, single-threaded serial read example
import serial
import time # Optional (required if using time.sleep() below)
ser = serial.Serial(port='COM4', baudrate=9600)
while (True):
# Check if incoming bytes are waiting to be read from the serial input
# buffer.
# NB: for PySerial v3.0 or later, use property `in_waiting` instead of
# function `inWaiting()` below!
if (ser.inWaiting() > 0):
# read the bytes and convert from binary array to ASCII
data_str = ser.read(ser.inWaiting()).decode('ascii')
# print the incoming string without putting a new-line
# ('\n') automatically after every print()
print(data_str, end='')
# Put the rest of your code you want here
# Optional, but recommended: sleep 10 ms (0.01 sec) once per loop to let
# other threads on your PC run during this time.
time.sleep(0.01)
This way you only read and print if something is there. You said, "Ideally I should be able to read serial data only when it's available." This is exactly what the code above does. If nothing is available to read, it skips on to the rest of your code in the while loop. Totally non-blocking.
(This answer originally posted & debugged here: Python 3 non-blocking read with pySerial (Cannot get pySerial's "in_waiting" property to work))
pySerial documentation: http://pyserial.readthedocs.io/en/latest/pyserial_api.html
UPDATE:
27 Dec. 2018: added comment about in_waiting vs inWaiting(). Thanks to #FurkanTürkal for pointing that out in the comments below. See documentation here: https://pyserial.readthedocs.io/en/latest/pyserial_api.html#serial.Serial.in_waiting.
27 Oct. 2018: Add sleep to let other threads run.
Documentation: https://docs.python.org/3/library/time.html#time.sleep
Thanks to #RufusV2 for bringing this point up in the comments.
Note on multi-threading:
Even though reading serial data, as shown above, does not require using multiple threads, reading keyboard input in a non-blocking manner does. Therefore, to accomplish non-blocking keyboard input reading, I've written this answer: How to read keyboard input?.
References:
Official pySerial serial.Serial() class API - https://pyserial.readthedocs.io/en/latest/pyserial_api.html
Put it in a separate thread, for example:
import threading
import serial
connected = False
port = 'COM4'
baud = 9600
serial_port = serial.Serial(port, baud, timeout=0)
def handle_data(data):
print(data)
def read_from_port(ser):
while not connected:
#serin = ser.read()
connected = True
while True:
print("test")
reading = ser.readline().decode()
handle_data(reading)
thread = threading.Thread(target=read_from_port, args=(serial_port,))
thread.start()
http://docs.python.org/3/library/threading
I would warn against using blocking IO in a thread. Remember Python has a GIL and at one time only one thread can execute. Now please note that pyserial module is a wrapper over an OS implementation of accessing the serial port. That means it calls code external to the Python. If that code blocks, then the interpreter also get blocked and nothing will execute in the Python program, even the main thread.
This can even happen when using non-blocking IO or timeout based polling if the underlying device driver does not implement timeout well.
A more robust approach is to use multiprocessing module with a queue. Run serial read code in a separate process. This will make sure main and other threads don't block and the program can exit in clean way.
Use a timer driven event to test and read the serial port.
Untested example:
import threading
class serialreading():
def __init__(self):
self.active = True
self.test()
def test(self):
n_in =comport.in_waiting()
if n_in> 0:
self.data = self.data + comport.read(size=n_in)
if len(self.data) > 0:
print(self.data)
self.data=""
if self.active:
threading.Timer(1, test).start() # start new timer of 1 second
def stop(self):
self.active = False