Develop Reference

Bind Bluetooth device programmatically to rfcomm via python in

i wrote a script in python for serial communication between my M5Stack Stick C (like raduino) and the raspberry pi.
all work fine. i can send "X","Y" or "Z" from raspberry py to the stick and he will reply the value (G-Force) back to the raspi! so far so good
Codes:
Python on raspy:
import serial
import time
import threading
ser = serial.Serial('/dev/rfcomm5') #init serial port
input_line = []#init input char array
def process_data(_data):
#called every time a sream is terminated by \n
#and the command string is ready to use
command = convert(_data)
print(command)
def convert(s): #convert the char list in a string
new = "" #init string to append all chars from char array
for x in s: # traverse in the string
new += str(x)
return new # return string
def processIncomingByte(inByte):#adding incoming chars to input_line
global input_line# globalize the input_line
if(inByte == '\n'):#if \n is incoming, end the chararray and release process data method
process_data(input_line)
input_line = []#reset input_line for next incoming string
elif(inByte == '\r'):
pass
else:#put all incoming chars in input_line
input_line.append(inByte)
while True:
while(ser.in_waiting > 0):#while some data is waiting to read....
processIncomingByte(ser.read())#.... process bytes whit method
ser.write(b'X\n')
time.sleep(0.5)
before the script work, i have to manually bind the m5Stak Stick-C over Blueman
to /dev/Rfcomm5. it work just fine over GUI or Console....
but now i would like to connect the stick via python to rfcomm5 (just by know the MAC adress, will be found in a config file later on...)
i startet to investigate a bit, but the more i research the more confused i am!!
i read some stuff over sockets and server-client aproaches. over a seperated script and so on....
i tested this code:
from bluetooth import *
target_name = "M5-Stick-C"
target_address = None
nearby_devices = discover_devices()
for address in nearby_devices:
if (target_name == lookup_name( address )):
target_address = address
break
if (target_address is not None):
print ("found target bluetooth device with address ", target_address)
else:
print ("could not find target bluetooth device nearby")
and indeed it found the device (just testing)!
but do i realy need to make a second script/process to connect to from my script?
is the the M5stack Stick-C the server? (i think so)
im so confused about all that stuff. i coded a lot, but never whit sockets, server-client stuff.
basically the communication (server/client?) works.
i just need to connect the device i found in the second script via macadress to rfcomm5 (or whatever rfcomm).
do i need a bluetooth socket? like in this example
https://gist.github.com/kevindoran/5428612
isnt the rfcomm the socket or am i wrong?

There are a number of layers that are used in the communication process and depending where you tap into that stack will depend what coding you need to do. The other complication is that BlueZ (the Bluetooth stack on linux) changed how it works over recent times leaving a lot of out of date information on the internet and easy for people to get confused.
With two Bluetooth devices, they need to establish a pairng. This is typically a one off provisioning step. This can be done with tools like Blueman or on the command line with bluetoothctl. Once you have a pairing established between your RPi and the M5Stack Stick, you shouldn't need to discover nearby devices again. Your script should just be able to connect if you tell it which device to connect to.
The M5Stack stick is advertising as having a Serial Port Profile (SPP). This is a layer on top of rfcomm.
There is a blog post about how this type of connection can be done with the standard Python3 installation: http://blog.kevindoran.co/bluetooth-programming-with-python-3/
My expectation is that you will only have to do the client.py on your RPi as the M5Stack Stick is the server. You will need to know its address and which port to connect on. Might be some trial and error on the port number (1 and 3 seem to be common).
Another library that I find helpful for SPP, is bluedot as it abstracts away some of the boilerplate code: https://bluedot.readthedocs.io/en/latest/btcommapi.html#bluetoothclient
So in summary, my recommendation is to use the standard Python Socket library or Bluedot. This will allow you to specify the address of the device you wish to connect to in your code and the underlying libraries will take care of making the connection and setting up the serial port (as long as you have already paired the two devices).
Example of what the above might look like with Bluedot
from bluedot.btcomm import BluetoothClient
from signal import pause
from time import sleep
# Callback to handle data
def data_received(data):
print(data)
sleep(0.5)
c.send("X\n")
# Make connection and establish serial connection
c = BluetoothClient("M5-Stick-C", data_received)
# Send initial requests
c.send("X\n")
# Cause the process to sleep until data received
pause()
Example using the Python socket library:
import socket
from time import sleep
# Device specific information
m5stick_addr = 'xx:xx:xx:xx:xx:xx'
port = 5 # This needs to match M5Stick setting
# Establish connection and setup serial communication
s = socket.socket(socket.AF_BLUETOOTH, socket.SOCK_STREAM, socket.BTPROTO_RFCOMM)
s.connect((m5stick_addr, port))
# Send and receive data
while True:
s.sendall(b'X\n')
data = s.recv(1024)
print(data)
sleep(0.5)
s.close()

How to send an API frame to a remote XBee using Digi XBee Python module?

I am moving code from Python XBee library to Digi's Python XBee library and I am unable to locate syntax in the docs on how to send a API frame to a remote XBee. I am using S1 and S1 Pro devices where the "local" device is attached to a Beaglebone and the "remote" devices are standalone out in the wild.
I have the basic framework down:
from digi.xbee.devices import XBeeDevice, RemoteXBeeDevice, XBee64BitAddress
PORT = '/dev/ttyO1'
BAUD = 9600
API_FRAME = 'long-hex-string-here'
local_xbee = XBeeDevice(PORT, BAUD)
local_xbee.open()
# Instantiate a remote XBee device object.
remote_xbee = RemoteXBeeDevice(local_xbee, XBee64BitAddress.from_hex_string("0013A20040DD7DCD"))
# Transmit frame to remote_xbee - unsure of correct method and syntax
# perhaps XBeePacket.create_packet(hex string here)
# local_xbee.send_data(remote_xbee, "api frame here?") ??
local_xbee.close()
but I am unable to locate syntax for how to transmit my constructed API frame. Based upon the Introduction section in the docs I presume that is the correct approach. I am not interested in broadcasting to all devices on the network but rather unicast communication.

I have some older model of the source when I could get it to work.
…
I have some WiFi Xbee modules that I used w/ some converter boards (base boards). I used it to attach the communication of the Xbee from one computer to another, e.g. random desktop to BeagleBone Black via USB instead of UART. So, I would use the source, listed below, to attach my USB dongle for the Xbee communication from the BBB to the other field module.
Their I/O stuff can be found here: https://github.com/digidotcom/xbee-python/tree/master/examples/io.
Also...changing just some of their lines in their source w/ the USB dongle WiFi adapter boards proved valuable in signaling LEDs and other sensors.
Oh and you will need what they are now calling Carrier Boards. It is the adapter board I just typed out. So, if you have already got a Carrier Board, use lsusb as the command in Linux to find your USB "name."
So, for instance, if lsusb brings up /dev/ttyUSB0, then that is the port identification.
And you can use that section, from lsusb, to then change your xbee modules in the xtcu software from Digi.
…
from digi.xbee.devices import XBeeDevice
from digi.xbee.io import IOLine, IOMode
import time
import threading
# TODO: Replace with the serial port where your local module is connected to.
PORT = "/dev/ttyUSB0"
# TODO: Replace with the baud rate of your local module.
BAUD_RATE = 9600
REMOTE_NODE_ID = "Xbee_B"
IOLINE_IN = IOLine.DIO2_AD2
IOLINE_OUT = IOLine.DIO4_AD4
def main():
print(" +-----------------------------------------------+")
print(" | XBee Python Library Get/Set Remote DIO Sample |")
print(" +-----------------------------------------------+\n")
stop = False
th = None
local_device = XBeeDevice(PORT, BAUD_RATE)
try:
local_device.open()
print("local device: ", local_device.get_node_id())
# Obtain the remote XBee device from the XBee network.
xbee_network = local_device.get_network()
remote_device = xbee_network.discover_device(REMOTE_NODE_ID)
if local_device is None:
print("Could not find the remote device")
exit(2)
def io_detection_callback():
while not stop:
# Read the digital value from the input line.
io_value = remote_device.get_dio_value(IOLINE_IN)
print("%s: %s" % (IOLINE_IN, io_value))
# Set the previous value to the local output line.
local_device.set_dio_value(IOLINE_OUT, io_value)
time.sleep(2)
th = threading.Thread(target=io_detection_callback)
remote_device.set_io_configuration(IOLINE_IN, IOMode.DIGITAL_IN)
local_device.set_io_configuration(IOLINE_OUT, IOMode.DIGITAL_OUT_HIGH)
time.sleep(1)
th.start()
input()
finally:
stop = True
if th is not None and th.is_alive():
th.join()
if local_device is not None and local_device.is_open():
local_device.close()
if __name__ == '__main__':
main()
So, see the PORT = "/dev/ttyUSB0" section of the source?
This is where I attached my Xbee module to the Carrier Board and then attached the Carrier Board to the BBB by way of USB.
Um, this may not answer a question but give more insight as to how to handle Digi Devices/Modules.
I also think that if you want to venture in this direction of UART communication w/ Xbee and the BeagleBone Black, it may be more complicated. I will keep searching my text.
P.S. This book goes over some methods to connect, Experiment 10 and Experiment 16, your "BBB" to a UART, Xbee, and how to communicate. It is a bit too in depth to get all of the communication ideas from this book but this is it:
The Hands-on XBEE Lab Manual, Experiments that Teach you XBEE Wireless Communications – Jonathan A Titus

Python TCP Socket Data Sometimes Missing Parts. Socket Overflow?

Short description:
Client sends server data via TCP socket. Data varies in length and is strings broken up by the delimiter "~~~*~~~"
For the most part it works fine. For a while. After a few minutes data winds up all over the place. So I start tracking the problem and data is ending up in the wrong place because the full thing has not been passed.
Everything comes into the server script and is parsed by a different delimiter -NewData-* then placed into a Queue. This is the code:
Yes I know the buffer size is huge. No I don't send data that kind of size in one go but I was toying around with it.
class service(SocketServer.BaseRequestHandler):
def handle(self):
data = 'dummy'
#print "Client connected with ", self.client_address
while len(data):
data = self.request.recv(163840000)
#print data
BigSocketParse = []
BigSocketParse = data.split('*-New*Data-*')
print "Putting data in queue"
for eachmatch in BigSocketParse:
#print eachmatch
q.put(str(eachmatch))
#print data
#self.request.send(data)
#print "Client exited"
self.request.close()
class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
pass
t = ThreadedTCPServer(('',500), service)
t.serve_forever()
I then have a thread running on while not q.empty(): which parses the data by the other delimiter "~~~*~~~"
So this works for a while. An example of the kind of data I'm sending:
2016-02-23 18:01:24.140000~~~*~~~Snowboarding~~~*~~~Blue Hills~~~*~~~Powder 42
~~~*~~~Board Rental~~~*~~~15.0~~~*~~~1~~~*~~~http://bigshoes.com
~~~*~~~No Wax~~~*~~~50.00~~~*~~~No Ramps~~~*~~~2016-02-23 19:45:00.000000~~~*~~~-15
But things started to break. So I took some control data and sent it in a loop. Would work for a while then results started winding up in the wrong place. And this turned up in my queue:
2016-02-23 18:01:24.140000~~~*~~~Snowboarding~~~*~~~Blue Hills~~~*~~~Powder 42
~~~*~~~Board Rental~~~*~~~15.0~~~*~~~1~~~*~~~http://bigshoes.com
~~~*~~~No Wax~~~*~~~50.00~~~*~~~No Ramps~~~*~~~2016-02-23 19:45:00.000000~~~*~
Cutting out the last "~~-15".
So the exact same data works then later doesn't. That suggests some kind of overflow to me.
The client connects like this:
class Connect(object):
def connect(self):
host = socket.gethostname() # Get local machine name
#host = "127.0.0.1"
port = 500 # Reserve a port for your service.
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#print('connecting to host')
sock.connect((host, port))
return sock
def send(self, command):
sock = self.connect()
#recv_data = ""
#data = True
#print('sending: ' + command)
sock.sendall(command)
sock.close()
return
It doesn't wait for a response because I don't want it hanging around waiting for one. But it closes the socket and (as far as I understand) I don't need to flush the socket buffer or anything it should just be clearing itself when the connection closes.
Would really appreciate any help on this one. It's driving me a little spare at this point.
Updates:
I'm running this on both my local machine and a pretty beefy server and I'd be pushed to believe it's a hardware issue. The server/client both run locally and sockets are used as a way for them to communicate so I don't believe latency would be the cause.
I've been reading into the issues with TCP communication. An area where I feel I'll quickly be out of my depth but I'm starting to wonder if it's not an overflow but just some king of congestion.
If sendall on the client does not ensure everything is sent maybe some kind of timer/check on the server side to make sure nothing more is coming.

The basic issue is that your:
data = self.request.recv(163840000)
line is not guaranteed to receive all the data at once (regardless of how big you make the buffer).
In order to function properly, you have to handle the case where you don't get all the data at once (you need to track where you are, and append to it). See the relevant example in the Python docs on using a socket:
Now we come to the major stumbling block of sockets - send and recv operate on the network buffers. They do not necessarily handle all the bytes you hand them (or expect from them), because their major focus is handling the network buffers. In general, they return when the associated network buffers have been filled (send) or emptied (recv). They then tell you how many bytes they handled. It is your responsibility to call them again until your message has been completely dealt with.

As mentioned, you are not receiving the full message even though you have a large buffer size. You need to keep receiving until you get zero bytes. You can write your own generator that takes the request object and yields the parts. The nice side is that you can start processing messages while some are still coming in
def recvblocks(request):
buf = ''
while 1:
newdata = request.recv(10000)
if not newdata:
if buf:
yield buf
return
buf += newdata
parts = buf.split('*-New*Data-*')
buf = parts.pop()
for part in parts:
yield part
But you need a fix on your client also. You need to shutdown the socket before close to really close the TCP connection
sock.sendall(command)
sock.shutdown(request.SHUT_RDWR)
sock.close()

Can reading sessions from a pcap file with Scapy be made more memory efficient

At the moment I'm trying to write a quick Python program that reads in a .pcap file and writes out data about the various sessions that are stored in there.
The information I write out includes srcip, dstip, srcport and dstport etc.
However, even for a fairly small pcap this takes a lot of memory and ends up running for a very long time. We're talking 8GB+ of memory used for a pcap of a size of 212MB.
As usual I guess there might be a more efficient way of doing this that I'm just unaware of.
Here is a quick skeleton of my code - no vital parts missing.
import socket
from scapy.all import *
edges_file = "edges.csv"
pcap_file = "tcpdump.pcap"
try:
print '[+] Reading and parsing pcap file: %s' % pcap_file
a = rdpcap(pcap_file)
except Exception as e:
print 'Something went wrong while opening/reading the pcap file.' \
'\n\nThe error message is: %s' % e
exit(0)
sessions = a.sessions()
print '[+] Writing to edges.csv'
f1 = open(edges_file, 'w')
f1.write('source,target,protocol,sourceport,destinationport,'
'num_of_packets\n')
for k, v in sessions.iteritems():
tot_packets = len(v)
if "UDP" in k:
proto, source, flurp, target = k.split()
srcip, srcport = source.split(":")
dstip, dstport = target.split(":")
f1.write('%s,%s,%s,%s,%s,%s\n' % (srcip, dstip, proto, srcport,
dstport, tot_packets))
continue
elif "TCP" in k:
proto, source, flurp, target = k.split()
srcip, srcport = source.split(":")
dstip, dstport = target.split(":")
f1.write('%s,%s,%s,%s,%s,%s\n' % (srcip, dstip, proto, srcport,
dstport, tot_packets))
continue
elif "ICMP" in k:
continue # Not bothered about ICMP right now
else:
continue # Or any other 'weird' pacakges for that matter ;)
print '[+] Closing the edges file'
f1.close()
As always - grateful for any assistance.

I know I'm late to the party, but hopefully this will be useful to future visitors.
rdpcap() dissects the entire pcap file and retains an in-memory representation of each and every packet, which explains why it eats up a lot of memory.
As far as I am aware (I am a novice Scapy user myself), the only two ways you can invoke Scapy's session reassembly are:
By calling scapy.plist.PacketList.sessions(). This is what you're currently doing (rdpcap(pcap_file) returns a scapy.plist.PacketList).
By reading the pcap using sniff() in offline mode while also providing the function with a session decoder implementation. For example, for TCP reassembly you'd do sniff(offline='stackoverflow.pcap', session=TCPSession). (This was added in Scapy 2.4.3).
Option 1 is obviously a dead end (as it requires that we keep all packets of all sessions in memory at one time), so let's explore option 2...
Let's launch Scapy in interactive mode to access the documentation for sniff():
$ scapy
>>> help(sniff)
Help on function sniff in module scapy.sendrecv:
sniff(*args, **kwargs)
Sniff packets and return a list of packets.
Args:
count: number of packets to capture. 0 means infinity.
store: whether to store sniffed packets or discard them
prn: function to apply to each packet. If something is returned, it
is displayed.
--Ex: prn = lambda x: x.summary()
session: a session = a flow decoder used to handle stream of packets.
e.g: IPSession (to defragment on-the-flow) or NetflowSession
filter: BPF filter to apply.
lfilter: Python function applied to each packet to determine if
further action may be done.
--Ex: lfilter = lambda x: x.haslayer(Padding)
offline: PCAP file (or list of PCAP files) to read packets from,
instead of sniffing them
timeout: stop sniffing after a given time (default: None).
L2socket: use the provided L2socket (default: use conf.L2listen).
opened_socket: provide an object (or a list of objects) ready to use
.recv() on.
stop_filter: Python function applied to each packet to determine if
we have to stop the capture after this packet.
--Ex: stop_filter = lambda x: x.haslayer(TCP)
iface: interface or list of interfaces (default: None for sniffing
on all interfaces).
monitor: use monitor mode. May not be available on all OS
started_callback: called as soon as the sniffer starts sniffing
(default: None).
The iface, offline and opened_socket parameters can be either an
element, a list of elements, or a dict object mapping an element to a
label (see examples below).
Notice the store parameter. We can set this to False to make sniff() operate in a streamed fashion (read a single packet, process it, then release it from memory):
sniff(offline='stackoverflow.pcap', session=TCPSession, store=False)
I just tested this with a 193 MB pcap. For store=True (default value), this eats up about 1.7 GB of memory on my system (macOS), but only approximately 47 MB when store=False.
Processing the reassembled TCP sessions (open question)
So we managed to reduce our memory footprint - great! But how do we process the (supposedly) reassembled TCP sessions? The usage instructions indicates that we should use the prn parameter of sniff() to specify a callback function that will then be invoked with the reassembled TCP session (emphasis mine):
sniff() also provides Sessions, that allows to dissect a flow of
packets seamlessly. For instance, you may want your sniff(prn=...)
function to automatically defragment IP packets, before executing the
prn.
The example is in the context of IP fragmentation, but I'd expect the TCP analog to be to group all packets of a session and then invoke prn once for each session. Unfortunately, that's not how it works: I tried this on my example pcap, and the callback is invoked once for every packet---exactly as indicated in sniff()'s documentation shown above.
The usage instructions linked above also states the following about using session=TCPSession in sniff():
TCPSession -> defragment certain TCP protocols*. Only HTTP 1.0 currently uses this functionality.
With the output of the experiment above in mind, I now interpret this as that whenever Scapy finds an HTTP (1.0) request/response that spans across multiple TCP segments, it'll create a single packet in which the payload is the merged payload of those TCP segments (which in total is the full HTTP request/response). I'd appreciate it if anyone can help clarify the meaning of the above quote on TCPSession---or even better: clarify if TCP reassembly is indeed possible this way and that I'm just misunderstanding the API.

Reading serial data in realtime in Python

I am using a script in Python to collect data from a PIC microcontroller via serial port at 2Mbps.
The PIC works with perfect timing at 2Mbps, also the FTDI usb-serial port works great at 2Mbps (both verified with oscilloscope)
Im sending messages (size of about 15 chars) about 100-150x times a second and the number there increments (to check if i have messages being lost and so on)
On my laptop I have Xubuntu running as virtual machine, I can read the serial port via Putty and via my script (python 2.7 and pySerial)
The problem:
When opening the serial port via Putty I see all messages (the counter in the message increments 1 by 1). Perfect!
When opening the serial port via pySerial I see all messages but instead of receiving 100-150x per second i receive them at about 5 per second (still the message increments 1 by 1) but they are probably stored in some buffer as when I power off the PIC, i can go to the kitchen and come back and im still receiving messages.
Here is the code (I omitted most part of the code, but the loop is the same):
ser = serial.Serial('/dev/ttyUSB0', 2000000, timeout=2, xonxoff=False, rtscts=False, dsrdtr=False) #Tried with and without the last 3 parameters, and also at 1Mbps, same happens.
ser.flushInput()
ser.flushOutput()
While True:
data_raw = ser.readline()
print(data_raw)
Anyone knows why pySerial takes so much time to read from the serial port till the end of the line?
Any help?
I want to have this in real time.
Thank you

You can use inWaiting() to get the amount of bytes available at the input queue.
Then you can use read() to read the bytes, something like that:
While True:
bytesToRead = ser.inWaiting()
ser.read(bytesToRead)
Why not to use readline() at this case from Docs:
Read a line which is terminated with end-of-line (eol) character (\n by default) or until timeout.
You are waiting for the timeout at each reading since it waits for eol. the serial input Q remains the same it just a lot of time to get to the "end" of the buffer, To understand it better: you are writing to the input Q like a race car, and reading like an old car :)

A very good solution to this can be found here:
Here's a class that serves as a wrapper to a pyserial object. It
allows you to read lines without 100% CPU. It does not contain any
timeout logic. If a timeout occurs, self.s.read(i) returns an empty
string and you might want to throw an exception to indicate the
timeout.
It is also supposed to be fast according to the author:
The code below gives me 790 kB/sec while replacing the code with
pyserial's readline method gives me just 170kB/sec.
class ReadLine:
def __init__(self, s):
self.buf = bytearray()
self.s = s
def readline(self):
i = self.buf.find(b"\n")
if i >= 0:
r = self.buf[:i+1]
self.buf = self.buf[i+1:]
return r
while True:
i = max(1, min(2048, self.s.in_waiting))
data = self.s.read(i)
i = data.find(b"\n")
if i >= 0:
r = self.buf + data[:i+1]
self.buf[0:] = data[i+1:]
return r
else:
self.buf.extend(data)
ser = serial.Serial('COM7', 9600)
rl = ReadLine(ser)
while True:
print(rl.readline())

You need to set the timeout to "None" when you open the serial port:
ser = serial.Serial(**bco_port**, timeout=None, baudrate=115000, xonxoff=False, rtscts=False, dsrdtr=False)
This is a blocking command, so you are waiting until you receive data that has newline (\n or \r\n) at the end:
line = ser.readline()
Once you have the data, it will return ASAP.

From the manual:
Possible values for the parameter timeout:
…
x set timeout to x seconds
and
readlines(sizehint=None, eol='\n') Read a list of lines,
until timeout. sizehint is ignored and only present for API
compatibility with built-in File objects.
Note that this function only returns on a timeout.
So your readlines will return at most every 2 seconds. Use read() as Tim suggested.