Develop Reference

Xbee image transfer python

Hi there i'm quite new with xBee and struggle with data transferring. My objective is to take picture with Raspberry pi wide and send it back to computer via xBee by turn that image to hexlify code. after recieving the code with python on computer i use binascii library to turn those code back to image by this code
ASCII to IMG:
import binascii
with open("file.txt", "r") as f:
data=f.read()
data = data.strip()
data = data.replace('\n', '')
data = binascii.a2b_hex(data)
with open('image.png', 'wb') as image_file:
image_file.write(data)
but after running that code the image is corrupted. So i start taking a look at the receiving code but I'm not sure if the code is correct, because the text file that i got has a lot of "0" in it
Receiving code:
from digi.xbee.devices import XBeeDevice
PORT = 'COM11'
BAUD = 19200
ser = XBeeDevice(PORT, BAUD)
try :
ser.open()
def data_receive_callback(xbee_message):
data = xbee_message.data.decode("utf-8")
with open("file.txt","a") as f:
f.write(data)
ser.add_data_received_callback(data_receive_callback)
print("Waiting for data...\n")
input()
finally:
if ser is not None and ser.is_open():
ser.close()
camera code in RPi:
from picamera import PiCamera
import serial
import binascii
ser =serial.Serial(
port='/dev/ttyS0',
baudrate=19200,
parity= serial.PARITY_NONE,
stopbits=serial.STOPBITS_ONE,
bytesize=serial.EIGHTBITS,
timeout=1
)
camera=PiCamera()
camera.resolution(1920,1080)
camera.capture("img.png")
with open("image.png",'rb') as f:
content=f.read()
a=binascii.hexlify(content)
ser.write(a)
ser.close
What should I do or try to fix the code. I think its the receiving code that is a main problem.
Ps. i already try to convert image file to hexlify in both computer and Raspberry Pi and reverse it back and it still work fine.

Questions:
Why are you using an XBee instead of Wi-Fi and a standard TCP protocol like HTTP or FTP?
Work on debugging a piece of your system at a time. Instead of capturing an image on the Pi, use a small text file and see if it comes through correctly. It will be easier to see if you're dropping bytes in the middle, beginning or end of the file.
Whenever using an XBee module, be sure to enable hardware flow control (pins D6 and D7) so you don't lose any serial data. With hardware flow control, each side of the connection has the ability to signal the other side to temporarily stop sending while it processes data.
I also recommend increasing your baud rate to 115200 or even 230400 bps. That, in conjunction with the hardware flow control, will result in quicker transfers without lost bytes.
There's no need to hexlify the data -- the XBee is capable of sending 8-bit bytes when running in transparent serial mode (essentially a serial cable replacement).
That said, you don't have any way to indicate the start or end of the file -- the receiver doesn't know when the image begins or when to finish and close the file. If you continue to hexlify the data, you could send characters other than 0-9a-f to indicate that it's the start of an image or the image is complete.

Measuring time interval between serial Tx and Rx in Python

I've got an issue where I am trying to measure the time interval between a serial write and serial read in python. I'm using python 3.8 and a USB->RS485 adaptor on windows. Essentially, the code follows as so:
def write_packet(self, packet, flush=True):
if flush:
self.clear_buffers()
self.ser.write(packet)
self.tx_time = time.time_ns()
and this is immediately followed by:
def read_packet(self):
first_byte = True
while True:
byte = self.ser.read(1)
# Check if the read timed out:
if byte == b'':
**(notify of timeout)**
if byte == b'\x00':
**(end of packet, decode and break)**
else:
if first_byte:
self.time_rx = time.time_ns()
first_byte = False
As you can probably see, I'm trying to capture the time between just after transmitting, and receiving the first byte. After that I do something like this to get the time in ms:
time_diff_ms = (self.rx_time - self.tx_time)/1000000
My issue is the timings time_diff_ms seem to be way off. The scope image below of the RS485 signals shows it should read times of around ~1ms yet the script reads values of 6ms, 11ms, etc., almost random values.
https://i.stack.imgur.com/eJFAJ.jpg
I've also tried running the script on Linux, but not much difference. I'm also working with fairly high baud rates of 921600.

Processing raw data read from serial port with Python serial library?

I am not a Python programmer but am rather electronic circuit designer, however this time I must process some raw data sent by a microcontroller via RS232 port towards Python script (which is called by PHP script).
I've spent quite a few hours trying to determine the best ways of reading raw bytes from serial (RS232) port using Python and I did get the results - but I would like if someone could clarify certain inconsistencies I noticed during researching and here they are:
1:
I can see a lot of people who asked similar question had been asked whether they are using serial or pySerial module and how did they install the serial library. I can only say I don't really know which module I am using as the module worked out-of-the-box. Somewhere I read serial and pySerial is the same thing but I cannot find if that is true. All I know is I am using Python 2.7.9 with Raspbian OS.
2:
I've read there are read() and readline() methods for reading from the serial port but in the pySerial API docs there is no mention of the readline() method. Futhermore, I discovered the 'number of bytes to read' argument can be passed to readline() method as well as to the read() method (and works the same way, limiting the number of bytes to be read) but I cannot find that to be documented.
3:
When searching for how to determine if all of the data from the RS232 buffer has been read I have here found the following code:
read_byte = ser.read()
while read_byte is not None:
read_byte = ser.read()
print '%x' % ord(read_byte)
but that results with the:
Traceback (most recent call last):
File "./testread.py", line 53, in <module>
read_all()
File "./testread.py", line 32, in read_all
print '%x' % ord(read_byte)
TypeError: ord() expected a character, but string of length 0 found
upon reading the last byte from the buffer and I was able to detect the empty buffer only with the following code:
while True:
c = rs232.read()
if len(c) == 0:
break
print int(c.encode("hex"), 16), " ",
so I am not sure if the code that didn't work for me is for some serial library that is other than mine. My code for openinig port is BTW:
rs232 = serial.Serial(
port = '/dev/ttyUSB0',
baudrate = 2400,
parity = serial.PARITY_NONE,
stopbits = serial.STOPBITS_ONE,
bytesize = serial.EIGHTBITS,
timeout = 1
)
4:
The data I am receiving from µC is in the format:
0x16 0x02 0x0b 0xc9 ... 0x0d 0x0a
That is some raw bytes + \r\n. Since 'raw bytes' can contain 0x00, can someone confirm that is not a problem regarding reading the bytes into the Python string variable? As I understand that should work well but am not 100% sure.

PySerial works for me although haven't used it on a Pi.
3: Read() returns a string - this will be zero length if no data is read, so your later version is correct. As a string is not a character, you should use e.g. ord(read_byte[0]) to print the number corresponding to the first character (if the length of the string >0)
Your function:
while True:
c = rs232.read()
if len(c) == 0:
break
print int(c.encode("hex"), 16), " ",
Needs something adding to accumulate the data read, otherwise it is thrown away
rcvd = ""
while True:
c = rs232.read()
if len(c) == 0:
break
rcvd += c
for ch in c:
print ord(ch), " ",
4:
Yes you can receive and put nul (0x00) bytes in a string. For example:
a="\x00"
print len(a)
will print length 1

Pyserial writes data but does not read

I'm relatively new to programming, so bear with me. I'm trying to communicate with the measurement device METEX M-4650CR https://sigrok.org/wiki/Voltcraft_M-4650CR and I'm using a windows 7 64bit OS. I simply want to read out the data the device measures to my python procedure and display it and calculate with it.
I found in the manual http://elektron.pol.lublin.pl/elekp/labor_instr/METEX_M-4650CR_Manual.pdf (page 25ff), that it works with a baudrate of 1200, a bytesize of 7 (with ASCII coding) and 2 stopbits.
Furthermore, it can be requested to send data to the computer by simply giving it the command "M". It then returns 14 bytes to the computer. Without anything to measured connected to it, it should return something like 'DC 00.0000V CR'. CR is the terminator here (I hope that is the right name).
Here is my code:
import pyserial
import time
ser = serial.Serial(port='COM5', baudrate=1200,
bytesize=7, stopbits=2, timeout=1,
rtscts=False, dsrdtr=True)
time.sleep(1)
ser.write("M")
time.sleep(1)
bytestoread = ser.inWaiting()
print bytestoread
output = ''
output += ser.read(1000)
print 'output:' + str(output)
time.sleep(1)
ser.close()
My problem is, that I cannot read out the data properly with pyserial. I send the command "M" to the METEX and in the display it says 'send' for a short moment, so I guess my write command works fine. But after that (it should have send the data), all I get when from ser.inWaitung is '0L' or '1L' and the ser.read command gives nothing at all.
I don't think it is a problem of the hardware, because with another programme, called 'serialwatcher', I'm able read out the data correctly. It gives exactly the characters described in the manual.
I also tried the following while loop, having the problem, that most of the time inWaiting == 0, such that it never initialises the loop.
while ser.inWaiting() > 0:
output += ser.read(1)
if output != '':
output = outpus.rstrip()
print output
So, how can I read out the data correctly, that were send to the serial port? Thanks in advance.

Unfortunately I cannot test your code because I have no serial device with me, but you could try the following:
You could set a flag, e.g. alive when you are expecting data and simply try to read something. This worked for me when I was trying to receive data from a really old spectrometer.
while alive: #loop
text = ser.read(1) #try to read one line
if text: #if there is data
n = ser.inWaiting() #look if there is more to read
if n: #if so
text = text + ser.read(n) #get all of it
A more sophisticated example can be found here wxTerminal - Pyserial example You could also simply try to modify this brilliant code for your purpose and see if you are more successful.

'Drunk' input from readline, OK from other programs (reading smart meters P1 port)

I'm new to Python and want to read my smart meters P1 port using a Raspberry Pi and Python. Problem: the input looks like some component is drunk.
I'm sure it's pretty simple to fix, but after several hours of searching and trying, had to seek help.
When reading the P1 port with CU etc. everything is fine so the hardware etc. is OK. Using a serial to USB converter from dx.com (this one)
Command and (part of) the output: cu -l /dev/ttyUSB0 -s 9600 --parity=none
0-0:96.1.1(205A414246303031363631323463949271)
1-0:1.8.1(03118.000*kWh)
However, when trying to read it from Python, the input becomes gibberish (but at least sort of consistant):
0-0:96.±.±(²05A´±´²´630303±39363±3²3´639·3±3²©
±-0:±.¸.±(03±±¸.000ªë×è©
How to fix this? The code I'm using is:
import serial
ser = serial.Serial()
ser.baudrate = 9600
ser.bytesize=serial.SEVENBITS
ser.parity=serial.PARITY_EVEN
ser.stopbits=serial.STOPBITS_ONE
ser.xonxoff=0
ser.rtscts=0
ser.timeout=20
ser.port="/dev/ttyUSB0"
ser.close()
ser.open()
print ("Waiting for P1 output on " + ser.portstr)
counter=0
#read 20 lines
while counter < 20:
print ser.readline()
counter=counter+1
try:
ser.close()
print ("Closed serial port.")
except:
sys.exit ("Couldn't close serial port.")
Have already tried messing with baudrate etc. but that doesn't make any difference.

I'm not very familiar with the serial module, but I noticed that your cu command assumes there is no parity bit (--parity=none), but your python script assumes there is an even parity bit (ser.parity=serial.PARITY_EVEN). I would try
ser.parity=serial.PARITY_NONE
And if there's no parity bit, you'll also probably want
ser.bytesize=serial.EIGHTBITS

UPDATE: found a workaround by replacing the naughty characters.
This may work for others with the same problem, but I dont know if the bad characters are exactly the same. So the replacement part may need some work to make it work for others.
It's not exactly a solution as the incoming telegram is still messed up, but the following code will work around that. My telegram is completely clean now.
Relevant part of the code I'm using now:
#Define 2 variables
P1_numbers = {'±':'1', '²':'2', '´':'4', '·':'7', '¸':'8'}
P1_rest = {'¯':'/', 'ª':'*', '©':')', 'Æ':'F', 'ë':'k', '×':'W', 'è':'h', 'í':'m'}
# Define function to read the telegram. Calls a function to clean it.
def P1_read(stack):
counter = 0
while counter < TelegramLength:
stack.append(P1_clean(ser.readline()))
counter=counter+1
return stack
# Define function to clean up P1 output
def P1_clean(line):
for i, j in P1_numbers.iteritems():
line = line.replace(i, j)
for i, j in P1_rest.iteritems():
line = line.replace(i, j)
return line

I quess you have a smart meter with P1 protocol: DSMR 3.0?
Then these are the correct serial port settings, which you already had:
serialport = serial.Serial( # Configure Serial communication port
baudrate = 9600,
timeout = 11,
bytesize = serial.SEVENBITS,
parity = serial.PARITY_EVEN,
stopbits = serial.STOPBITS_ONE )
Probably some encoding or interpretation of the data is going wrong at your side. Here is an other method the read the smart meter:
To make the readout of the p1 protocol as easy as possible I'd suggest to use TextIOWrapper, this way you can read the serial port with the readline method. The "!" always ends the P1 telegram, so that can be used to detect the end of the message. when a full telegram has been received, the telegram can be processed. Example:
import io
p1port = io.TextIOWrapper(io.BufferedReader(serialport, buffer_size=1), newline='\n', encoding='ascii')
P1Message = []
while True:
try:
rawline = self.p1port.readline()
except UnicodeDecodeError:
print "Encode error on readline"
if '!' in rawline:
# Process your P1Message here
P1Message = [] # Clear message, wait for new one
else:
P1Message.append(rawline)

The OP is looooong gone, but the problem is of sufficiently general interest, so here's a fresh answer. User #Brionius is right: A look at the bit patterns involved shows that it's definitely a parity problem. Here's how to inspect the bit pattern of the characters "1" and "±":
>>> "{0:b}".format(ord("1"))
'110001'
>>> "{0:b}".format(ord("±"))
'10110001'
Get it? The characters are getting corrupted by having their high (8th) bit turned on. Or you can see this by setting the high bit of ascii "1":
>>> chr(ord("1") | 0b10000000)
'±'
Now, "1", "2" and "4" have three bits set (odd parity), and are corrupted. "0", "3", "5", etc. have even parity (2 or 4 bits set), and are preserved. So the communication channel is using even parity, which is not decoded properly at the receiving end.

I had the same problem, also in the context of the P1 Smart Meter port, and it took me quite a while to find it.
'cu' was displaying the data correctly, but Python wasn't (nor were some other programs). Apparently the parity bit is somehow not handled correctly. The following solves this problem:
p1_raw = ser.readline()
print 'raw:', p1_raw
decoded = ''.join(chr(ord(ch) & 0x7f) for ch in p1_raw)
print 'decoded:', decoded
I still find it strange that this is happening, because this was actually happening when I was trying to read the output of a Smart Meter for the second time. I already had a script successfully monitoring another Smart Meter at a different house for a couple of years and I never ran into this problem.
Perhaps there's a small difference in the USB-Serial adapters that causes this?!?