Develop Reference

High data-rate parallelism between python and arduino uno

I'm working on a project where a data capture system sends EMG data to a python script through an Arduino Uno's serial port and the python script classifies the user's intent by processing a chunk of that data. The data capture system sends samples at 250Hz and I require my python script to not miss samples the capture system is sending while classifying.
For this I want two parallelly running processes:
A process continuously capturing the data and buffering it. (using PySerial) - running every 0.004s
A process taking data from the buffer, preprocess it and classifying it. - this process approximately takes 0.03 seconds to run per loop excluding data acquisition
I don't know how to proceed with this. I've tried and read through a lot of articles but to no avail.
I tried threading, but apparently threading is not true parallelism, so I gave up trying to make it work.
I tried using multiprocessing but the multiprocessing module doesn't like pickling pyserial.
Here's what I tried: (It's probably full of mistakes, I hacked it up but I expected some output, also this is not indefinitely buffering, but I'm trying to get at least the first buffer printed)
Driver Code:
import GatherData as gd
import serial.tools.list_ports
from multiprocessing import Pool
if __name__ == '__main__':
arr = []
serialInst = serial.Serial()
ports = serial.tools.list_ports.comports()
portList = []
for Port in ports:
portList.append(str(Port))
print(str(Port))
val = input("Select Port: COM")
for x in range(0,len(portList)):
if(portList[x].startswith("COM"+str(val))):
portVar = "COM"+str(val)
print(portList[x])
serialInst.baudrate = 115200
serialInst.port = portVar
serialInst.open()
p = Pool(6) # Number of concurrent processes
arr = p.starmap(gd.collect, (arr, serialInst)) # Start all processes
p.map(gd.printnum(arr))
p.close()
p.join()
GatherData.py:
import serial.tools.list_ports
import numpy as np
import pandas as pd
# import Dependencies as dp
def collect(array,serialInst):
ch = 4
if not array:
return makearr()
else:
nextw = array[1:]
temparr = []
if (serialInst.in_waiting):
packet = serialInst.readline()
# now = time.time()
line_as_list = packet.split(b'\t')
#print(line_as_list)
for i in range(1,ch+1):
try:
rand1=line_as_list[i]
#print("rand1", rand1)
rand1List = rand1.split(b'\n')
rand1f = float(rand1List[0])
temparr.append(rand1f)
#print(temparr)
except IndexError:
return collect(array) #add pca
except ValueError:
return collect(array) #add pca
nextw.append(temparr)
print(nextw)
return nextw
def makearr(serialInst):
final = []
counter = 0
while(counter<50):
temparr = []
if (serialInst.in_waiting):
packet = serialInst.readline()
line_as_list = packet.split(b'\t')
flag = 0
for i in range(1,5):
try:
rand1=line_as_list[i]
#print("rand1", rand1)
rand1List = rand1.split(b'\n')
rand1f = float(rand1List[0])
temparr.append(rand1f)
except IndexError:
flag = 1
continue
except ValueError:
flag = 1
continue
if(flag == 0):
final.append(temparr)
counter=counter+1
return final
def printnum(array):
while True:
if array:
print(len(array), end='\r')
The errors:
COM8 - Arduino Uno (COM8)
Select Port: COM8
COM8 - Arduino Uno (COM8)
Traceback (most recent call last):
File "D:\anaconda\envs\tf\lib\site-packages\spyder_kernels\py3compat.py", line 356, in compat_exec
exec(code, globals, locals)
File "c:\users\indra\documents\igibup.py", line 34, in <module>
arr = p.starmap(gd.collect, (arr, serialInst)) # Start all processes
File "D:\anaconda\envs\tf\lib\multiprocessing\pool.py", line 276, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "D:\anaconda\envs\tf\lib\multiprocessing\pool.py", line 657, in get
raise self._value
File "D:\anaconda\envs\tf\lib\multiprocessing\pool.py", line 431, in _handle_tasks
put(task)
File "D:\anaconda\envs\tf\lib\multiprocessing\connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "D:\anaconda\envs\tf\lib\multiprocessing\reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
ValueError: ctypes objects containing pointers cannot be pickled
I don't understand how to proceed.

Python test http-server: what am I doing wrong?

I'm trying to write a test http-server for a personal IoT project I"ve got going. My project has its own web-server that responds to certain 'page load' requests. Now I'm trying to expand the web-page itself. Now I could develop the actual server itself simultaneously but downloading it to the IoT device every iteration is extraordinarily slow so I've decided to write a test-server on my machine instead.
The test-server is written in python. The code below is the test-server in its entirety:
import http.server
import socketserver
import string
PORT = 8000
class MyHandler(http.server.SimpleHTTPRequestHandler):
def __init__(self):
self.pageFuncs = { "/Lights": self.doLightsPage,
"/Show": self.doShowPage,
"/Static": self.doShowTypePage,
"/Effects": self.doShowTypePage,
"/Playlist": self.doShowTypePage,
"/Select": self.doControlTypePage,
"/Controls": self.doControlTypePage,
"/query": self.doQueryResultsPage }
self.STATIC_SHOW_TYPE = 0
self.EFFECTS_SHOW_TYPE = 1
self.PLAYLIST_SHOW_TYPE = 2
self.SELECT_CTRL_TYPE = 0
self.CONTROL_CTRL_TYPE = 1
self.LightsOn = 0
self.LightShowOn = 0
self.showType = STATIC_SHOW_TYPE
self.ctrlType = SELECT_CTRL_TYPE
def do_GET(self):
try:
page_name = self.path
if (page_name in pageFuncs):
print (page_name)
self.send_response(200)
self.send_header("Content-type", "text/json")
self.end_headers()
self.pageFuncs[page_name](self)
return
else:
super().do_GET()
except IOError:
self.send_error(404,'File Not Found: %s' % self.path)
except :
pass
def doLightsPage(self):
self.bLightsOn ^= 1
return doQueryResultsPage()
def doShowPage(self):
self.bLightShowOn ^= 1
return doQueryResultsPage()
def doShowTypePage(self):
if (self.path == '/Static') :
self.showType = STATIC_SHOW_TYPE
elif (self.path == '/Effects') :
self.showType = EFFECTS_SHOW_TYPE
else :
self.showType = PLAYLIST_SHOW_TYPE
return doQueryResultsPage()
def doControlTypePage(self):
if (self.path == '/Select') :
self.ctrlType = SELECT_CTRL_TYPE
else :
self.ctrlType = CONTROL_CTRL_TYPE
return doQueryResultsPage()
def doQueryResultsPage(self):
json_data = '{ \"power\": {}, \"show\": {},'
json_data += '\"show_type\": {}, \"control_type\": {} }'
json_data = format(json_data, bLightsOn, bLightShowOn, showType, ctrlType);
self.write(json_data)
def main():
try:
server = socketserver.TCPServer(("", PORT), MyHandler)
print ('started httpserver...')
server.serve_forever()
except KeyboardInterrupt:
print ('^C received, shutting down server')
server.socket.close()
if __name__ == '__main__':
main()
The point of the test is to receive a valid request; change a value and return a json string back reflecting the internal state of the server.
However, as soon as I try to load any of the pages, I get the following error message:
Exception happened during processing of request from ('127.0.0.1', 50149)
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.752.0_x64__qbz5n2kfra8p0\lib\socketserver.py", line 316, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.752.0_x64__qbz5n2kfra8p0\lib\socketserver.py", line 347, in process_request
self.finish_request(request, client_address)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.752.0_x64__qbz5n2kfra8p0\lib\socketserver.py", line 360, in finish_request
self.RequestHandlerClass(request, client_address, self)
TypeError: __init__() takes 0 positional arguments but 4 were given
So my question is, what am I doing wrong?
I've taken the code format from the following stackoverflow topic which appears to be trying to do the same things, maybe I'm misunderstanding something?

Thread missing 1 required positional argument

Just to give some context: I'm currently learning Python and to do so I started a little Project.
I wrote two Python Scripts: a Host and a Client script.
I'm already at the point where multiple Clients can connect to one "Host" and the Host can send a random String to all Clients.
Now I wanted to solve the problem, that if a Client disconnects from the Host nobody knows until the CLIENTSOCKET is called again.
So I wrote the checkConnection method to ping all Clients every 5 sec, for now.
class Connection():
def __init__(self):
self.s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
def bound(self,RHOST,PORT):
self.s.bind((RHOST,PORT))
self.s.listen(1000)
def connect(self):
((CLIENTSOCKET,ADRESS)) = self.s.accept()
return (CLIENTSOCKET,ADRESS)
def newConnection(Connection,CLIENTS):
con = Connection
cli = CLIENTS
while True:
c = con.connect()
if c != None:
if len(cli) != 0:
for x in range (len(cli)):
if c[1][0] == cli[x][1][0]:
pass
else:
cli.append(c)
else:
cli.append(c)
def checkConnection(CLIENTS):
cli = CLIENTS
while True:
if(len(cli)!=0):
for x in range(len(cli)):
try:
cli[x][0].sendall(b'')
except:
del cli[x]
time.sleep(5)
I'm creating to Threads with each method as a target and give them the needed parameters. The first thread starts like it should, but the second doesn't.
I really don't know why?
t = threading.Thread(target = newConnection,name = "newConnection",args = (g,CLIENTS))
t2 = threading.Thread(target = checkConnection,name = "checkConnection",args = (CLIENTS))
t.start()
t2.start()
Exception in thread checkConnection:
Traceback (most recent call last):
File "C:\Users\User\AppData\Local\Programs\Python\Python35-32\lib\threading.py", line 914, in _bootstrap_inner
self.run()
File "C:\Users\User\AppData\Local\Programs\Python\Python35-32\lib\threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
TypeError: checkConnection() missing 1 required positional argument: 'CLIENTS'

WebSocket server in Python: 'module' object has no attribute 'AF_INET' [duplicate]

This question already has answers here:
Importing installed package from script with the same name raises "AttributeError: module has no attribute" or an ImportError or NameError
(2 answers)
Closed 7 months ago.
I am trying to run this simple Python WebSocket, with a couple very minor changes. I am running Python 2.4.3 because I cannot use an newer version, but I'm not sure how much that matters.
Here is the error I'm getting:
Traceback (most recent call last):
File "socket.py", line 258, in ?
server = WebSocketServer("localhost", 8000, WebSocket)
File "socket.py", line 205, in __init__
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
AttributeError: 'module' object has no attribute 'AF_INET'
And here is my code:
import time
import struct
import socket
import base64
import sys
from select import select
import re
import logging
from threading import Thread
import signal
# Simple WebSocket server implementation. Handshakes with the client then echos back everything
# that is received. Has no dependencies (doesn't require Twisted etc) and works with the RFC6455
# version of WebSockets. Tested with FireFox 16, though should work with the latest versions of
# IE, Chrome etc.
#
# rich20b#gmail.com
# Adapted from https://gist.github.com/512987 with various functions stolen from other sites, see
# below for full details.
# Constants
MAGICGUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
TEXT = 0x01
BINARY = 0x02
# WebSocket implementation
class WebSocket(object):
handshake = (
"HTTP/1.1 101 Web Socket Protocol Handshake\r\n"
"Upgrade: WebSocket\r\n"
"Connection: Upgrade\r\n"
"Sec-WebSocket-Accept: %(acceptstring)s\r\n"
"Server: TestTest\r\n"
"Access-Control-Allow-Origin: http://localhost\r\n"
"Access-Control-Allow-Credentials: true\r\n"
"\r\n"
)
# Constructor
def __init__(self, client, server):
self.client = client
self.server = server
self.handshaken = False
self.header = ""
self.data = ""
# Serve this client
def feed(self, data):
# If we haven't handshaken yet
if not self.handshaken:
logging.debug("No handshake yet")
self.header += data
if self.header.find('\r\n\r\n') != -1:
parts = self.header.split('\r\n\r\n', 1)
self.header = parts[0]
if self.dohandshake(self.header, parts[1]):
logging.info("Handshake successful")
self.handshaken = True
# We have handshaken
else:
logging.debug("Handshake is complete")
# Decode the data that we received according to section 5 of RFC6455
recv = self.decodeCharArray(data)
# Send our reply
self.sendMessage(''.join(recv).strip());
# Stolen from http://www.cs.rpi.edu/~goldsd/docs/spring2012-csci4220/websocket-py.txt
def sendMessage(self, s):
"""
Encode and send a WebSocket message
"""
# Empty message to start with
message = ""
# always send an entire message as one frame (fin)
b1 = 0x80
# in Python 2, strs are bytes and unicodes are strings
if type(s) == unicode:
b1 |= TEXT
payload = s.encode("UTF8")
elif type(s) == str:
b1 |= TEXT
payload = s
# Append 'FIN' flag to the message
message += chr(b1)
# never mask frames from the server to the client
b2 = 0
# How long is our payload?
length = len(payload)
if length < 126:
b2 |= length
message += chr(b2)
elif length < (2 ** 16) - 1:
b2 |= 126
message += chr(b2)
l = struct.pack(">H", length)
message += l
else:
l = struct.pack(">Q", length)
b2 |= 127
message += chr(b2)
message += l
# Append payload to message
message += payload
# Send to the client
self.client.send(str(message))
# Stolen from http://stackoverflow.com/questions/8125507/how-can-i-send-and-receive-websocket-messages-on-the-server-side
def decodeCharArray(self, stringStreamIn):
# Turn string values into opererable numeric byte values
byteArray = [ord(character) for character in stringStreamIn]
datalength = byteArray[1] & 127
indexFirstMask = 2
if datalength == 126:
indexFirstMask = 4
elif datalength == 127:
indexFirstMask = 10
# Extract masks
masks = [m for m in byteArray[indexFirstMask : indexFirstMask+4]]
indexFirstDataByte = indexFirstMask + 4
# List of decoded characters
decodedChars = []
i = indexFirstDataByte
j = 0
# Loop through each byte that was received
while i < len(byteArray):
# Unmask this byte and add to the decoded buffer
decodedChars.append( chr(byteArray[i] ^ masks[j % 4]) )
i += 1
j += 1
# Return the decoded string
return decodedChars
# Handshake with this client
def dohandshake(self, header, key=None):
logging.debug("Begin handshake: %s" % header)
# Get the handshake template
handshake = self.handshake
# Step through each header
for line in header.split('\r\n')[1:]:
name, value = line.split(': ', 1)
# If this is the key
if name.lower() == "sec-websocket-key":
# Append the standard GUID and get digest
combined = value + MAGICGUID
response = base64.b64encode(combined.digest())
# Replace the placeholder in the handshake response
handshake = handshake % { 'acceptstring' : response }
logging.debug("Sending handshake %s" % handshake)
self.client.send(handshake)
return True
def onmessage(self, data):
#logging.info("Got message: %s" % data)
self.send(data)
def send(self, data):
logging.info("Sent message: %s" % data)
self.client.send("\x00%s\xff" % data)
def close(self):
self.client.close()
# WebSocket server implementation
class WebSocketServer(object):
# Constructor
def __init__(self, bind, port, cls):
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.socket.bind((bind, port))
self.bind = bind
self.port = port
self.cls = cls
self.connections = {}
self.listeners = [self.socket]
# Listen for requests
def listen(self, backlog=5):
self.socket.listen(backlog)
logging.info("Listening on %s" % self.port)
# Keep serving requests
self.running = True
while self.running:
# Find clients that need servicing
rList, wList, xList = select(self.listeners, [], self.listeners, 1)
for ready in rList:
if ready == self.socket:
logging.debug("New client connection")
client, address = self.socket.accept()
fileno = client.fileno()
self.listeners.append(fileno)
self.connections[fileno] = self.cls(client, self)
else:
logging.debug("Client ready for reading %s" % ready)
client = self.connections[ready].client
data = client.recv(4096)
fileno = client.fileno()
if data:
self.connections[fileno].feed(data)
else:
logging.debug("Closing client %s" % ready)
self.connections[fileno].close()
del self.connections[fileno]
self.listeners.remove(ready)
# Step though and delete broken connections
for failed in xList:
if failed == self.socket:
logging.error("Socket broke")
for fileno, conn in self.connections:
conn.close()
self.running = False
# Entry point
if __name__ == "__main__":
logging.basicConfig(level=logging.DEBUG, format="%(asctime)s - %(levelname)s - %(message)s")
server = WebSocketServer("localhost", 8000, WebSocket)
server_thread = Thread(target=server.listen, args=[5])
server_thread.start()
# Add SIGINT handler for killing the threads
def signal_handler(signal, frame):
logging.info("Caught Ctrl+C, shutting down...")
server.running = False
sys.exit()
signal.signal(signal.SIGINT, signal_handler)
while True:
time.sleep(100)

It appears that you've named your own file socket.py, so when you import socket, you're not getting the system library (it's just re-importing the file you're currently in - which has no AF_INET symbol). Try renaming your file something like mysocket.py.

Even after changing the file name, if you are running the python from the terminal.
(you may get the same error)
Kindly
rm -rf socket.pyc
(previously compiled bytecode)

I had the same problem, I was literally stuck here for hours, tried re installing it a million times, but found the solution.
1) Make sure the file name is not socket.py,
2) Change the directory, it will not work in the home directory due to some permission issues.
If you have by anychance saved the file as socket.py, do not copy the same file or rename it to something else, the problem will persist.
What I advice you to do is, open a new folder in a different directory, write a simple socket code which involved AF_INET. Try to run it. It should work.

Issue can be that you have a file or Cache name socket.py or socket.pyc
rm -rf socket.py
rm -rf socket.pyc
Hopefully this will resolve your import issue. Gud Luck

enter the current working directory
and remove the files named 'socket.py' and 'socket.pyc'

Ping and Pong in a threaded WebSocket server (Python)

I've written a threaded websocket server in Python, using the lastest websocket spec and I'm trying to make it send a ping request to every client every x seconds. The only way I came up with to do this is overriding BaseServer.server_forever() like this:
# override BaseServer's serve_forever to send a ping request every now and then
class ModTCPServer(SocketServer.TCPServer):
def __init__(self, server_address, RequestHandlerClass, bind_and_activate=True):
SocketServer.TCPServer.__init__(self, server_address, RequestHandlerClass, bind_and_activate)
self.__is_shut_down = threading.Event()
self.__shutdown_request = False
def serve_forever(self, poll_interval=0.5):
###
global PING_EVERY_SEC
self.lastPing = int(time())
###
self.__is_shut_down.clear()
try:
while not self.__shutdown_request:
r, w, e = select.select([self], [], [], poll_interval)
if self in r:
self._handle_request_noblock()
###
now = int(time())
if (now - self.lastPing) >= PING_EVERY_SEC:
self.socket.send(WebSocketPing(['0x89','0x21','0xa7','0x4b'], now)) # arbitrary key
self.lastPing = now
###
finally:
self.__shutdown_request = False
self.__is_shut_down.set()
class LoginServer(SocketServer.ThreadingMixIn, ModTCPServer):
pass
server = LoginServer(("", PORT), ApplicationHandler)
print "serving at port", PORT
server_thread = threading.Thread(target=server.serve_forever)
server_thread.daemon = True
server_thread.start()
while server_thread.isAlive():
pass
server.shutdown()
Here is the function that constructs the Ping frame, it just puts the timestamp in the contents:
def WebSocketPing(key, timestamp=False):
data = ['0x89','0x8a'] # 0x89 = fin,ping 0x8a = masked,len=10
data.extend(key)
if timestamp:
t = str(timestamp)
else:
t = str(int(time()))
for i in range(10):
masking_byte = int(key[i%4],16)
masked = ord(t[i])
data.append(hex(masked ^ masking_byte))
frame = ''
for i in range(len(data)):
frame += chr(int(data[i],16))
return frame
Bad things happen when I run this
Traceback (most recent call last):
File "LoginServer.py", line 91, in <module>
server = LoginServer(("", PORT), ApplicationHandler)
File "LoginServer.py", line 63, in __init__
SocketServer.TCPServer.__init__(self, server_address, RequestHandlerClass, bind_and_activate)
File "/usr/lib/python2.6/SocketServer.py", line 400, in __init__
self.server_bind()
File "/usr/lib/python2.6/SocketServer.py", line 411, in server_bind
self.socket.bind(self.server_address)
File "<string>", line 1, in bind
socket.error: [Errno 112] Address already in use
I assume this is down to my lack of understanding of how overriding works in Python or to a fundamentally wrong approach to this problem. Is there a better way to do this or a way to make this code work?

The codes does not set the properties __is_shut_down and __shutdown_request anywhere. Therefore, trying to access them fails. Create them in the constructor, like this:
class ModTCPServer(SocketServer.TCPServer):
def __init__(self, *args, **kwargs):
SocketServer.TCPServer.__init__(self, *args, **kwargs)
self.__is_shut_down = threading.Event()
self.__shutdown_request = threading.Event()
In response to the update:
socket.error: [Errno 112] Address already in use
means that another process has already bound to the specified port. On Unix, you can find that process with sudo netstat -ltpn. Alternatively, choose a different port.