Develop Reference

Flask-SocketIO 'Connect' callback

I'm trying to access variables that are being passed from the client (iOS; Swift) to the server on a Flask-SocketIO connection on the connect action. Let me explain. When you want to do a random action you have something like this on the server which includes a callback (see data in the code below):
#socketio.on('custom action', namespace = '/mynamespace')
def handle_custom_action(data):
print data
There are some preset actions (like connect) and apparently connect does not have any callback when it's called so the client cannot send any data on the connect action:
#socketio.on('connect', namespace = '/mynamespace')
def handle_connection(data):
print data # nothing gets printed
I looked into the code a bit deeper and found this. The definition of the on function is:
def on(self, message, namespace=None):
And then within that function (I'm omitting a bit of code to get to the point):
if message == 'connect':
ret = handler()
else:
ret = handler(*args)
I could be wrong but it appears that code explicitly does not return anything back on connect and I'm not sure why? I've found some evidence that this is possible in node.js (I will update this with proper links when I find them) so I'm wondering why this isn't possible in the Flask-SocketIO library or whether I'm just misunderstanding what I'm looking at (and if so, how to get those parameters).
Thanks!
Update:
I did find a way to access connection parameters but it doesn't seem like the 'right' way. I'm using the global request and splitting the GET parameters / query string that come through on the request:
data = dict(item.split("=") for item in request.event["args"][0]["QUERY_STRING"].split("&"))
OR as two lines:
data = request.event["args"][0]["QUERY_STRING"].split("&"))
data = dict(item.split("=") for item in data.split("&"))
Flask-SocketIO adds event which connects a dictionary with keys of message and args and within args is the QUERY_STRING which I then split add turn into a dictionary. This works fine but it doesn't necessarily answer the original question as to why there is no callback?
Here is an example of the iOS connection params being passed:
let connectParams = SocketIOClientOption.connectParams(["user_id" : Int(user.userId)!, "connection_id" : self.socketConnectionId])
self.socket = SocketIOClient(socketURL: URL(string: "http://www.myurl.com")!, config: [.nsp("/namespace"), .forceWebsockets(true), .forceNew(true), connectParams])

How to make multiple python programs communicate in this situation?

I'm a bit new at Python and I am working on a robotics project. The short form of my question is that I am trying to find the best way (for my situation) to run multiple python programs at once.
A little bit of context, my robot is a platform for a service robot that is capable of following markers and paths using image algorithms and also receive commands from a remote computer. I want to have separate programs for the image processing, the driving, and so on, and then manage all of them through a main program. I know I can't use anything basic like functions or classes, because each of these processes must be looping continuously, and I don't want to combine all the code to run in a single while loop, because it runs very slowly and it is significantly harder to manage.
So, in short, how do I make two separate, looping programs "talk"? Like I want the imaging program to send information about what it sees to the driving and steering program, etc.
I did some research and I found some information on multithreading and API's and stuff like that, though I can't really tell which one is actually the thing I'm looking for.
To clarify, I just need to be pointed in the right direction. This doesn't seem like a very high-level thing, and I know there are definitely tutorials out there, I'm just really confused as to where to start as I am teaching myself this as I go.

After some sniffing around, I found that using IPC was a good solution. The process I used wasn't too difficult, I just made some very simple server and client classes and had them communicate over the Localhost IP. There's undoubtedly a better way to do this, but for a beginner like myself, it was a simple way to make two programs talk without modifying code too much. For those who are trying to do a similar thing as I did, here's the classes I made for myself. Fair warning, they're not exactly pristine or even very complex, but they got the job done.
Here's the class I made for the server:
import socket
from random import random
from time import sleep
class ServerObject:
def __init__(self,host_address,port):
self._host_address = host_address
self._s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self._s.bind((self._host_address,port))
def handshake(self):
print "Server Started. Awaiting Connection"
while True:
_data, _addr = self._s.recvfrom(1024)
if str(self._s.recvfrom(1024)[0]) == 'marco':
break
print 'marco recieved. sending polo...'
while True:
self._s.sendto('polo',_addr)
if str(self._s.recvfrom(1024)[0]) == 'confirm':
break
sleep(.5)
print 'connection verified'
self._addr = _addr
return True
def send(self,data):
self._s.sendto(str(data),self._addr)
def recieve(self,mode = 0):
_data, _addr = self._s.recvfrom(1024)
if mode == 0:
return str(_data)
if mode == 1:
return int(_data)
if mode == 2:
return float(_data)
if mode == 3:
return tuple(_data)
def change_port(self,port):
self._s.bind((self._host_address,port))
def close(self):
self._s.close()
print '_socket closed_'
if __name__ == '__main__':
host = '127.0.0.1'
talk = ServerObject(host,6003)
talk.handshake()
And here's the class I made for the client:
import socket
from time import sleep
class ClientObject:
def __init__(self,host_address,server_port,port = 0):
self._server = (host_address,server_port)
self._s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self._s.bind((host_address,port))
def handshake(self):
print ' sending marco'
self._s.sendto('marco',self._server)
sleep(.1)
self._s.sendto('marco',self._server)
while True:
if str(self._s.recvfrom(1024)[0]) == 'polo':
break
#self._s.sendto('marco',self._server)
#self._s.sendto('marco',self._server)
print ' connection verified'
self._s.sendto('confirm',self._server)
self._s.setblocking(0)
return True
def recieve(self,mode = 0):
_data, _addr = self._s.recvfrom(1024)
if mode == 0:
return str(_data)
if mode == 1:
return int(_data)
if mode == 2:
return float(_data)
if mode == 3:
return tuple(_data)
def send(self,data):
self._s.sendto(str(data),self._server)
def close(self):
self._s.close()
print '_socket closed_'
if __name__ == '__main__':
host = '127.0.0.1'
port = 0
talk = ClientObject(host,24603,port)
talk.handshake()
#while True:
#print talk.recieve()
Use the ServerObject class on the program that will primarily send data and the ClientObject class on the program that will primarily recieve data. These can be flipped around in many situations, but I found it's best to do it this way to take advantage of UDP. The client class has an optional port variable that is set to 0 by default. This is because for UDP the client needs another port to establish itself on. 0 means it will pick an available port, but if you specify one, it's possible to re-establish a connection if the client goes offline without needing to restart both programs.
Use the handshake first on both programs being sure to use the same IP and port (not referring to the last variable on the client) and then use the send and receive functions to pass data back and forth.
again, these aren't that good, in fact there's many problems that cab arise with using this method, but for a simple task, they got the job done. I set up the handshake to print verifications of what is happening, but if those get annoying, you can just remove those lines.
Hope this helps!

I think multiproccessing library could be a solution.
You will be able to run several processes in parallel when each process could perform it specific work, while sending data to each other.
You can check this example
This is generic directory walker, which have process that scans directory tree and passes the data to other process, which scans files in already discovered folders. All this done in parallel.

This is probably a little bit outside the scope of your project, but have you considered using ROS? It lets you run a bunch of different nodes (can be Python scripts) at the same time that communicate by publishing and subscribing to topics. They can be on the same system (i.e. one or more nodes on the robot) or different systems (i.e. one node on the robot, multiple nodes on the PC). ROS also has a lot of awesome built in tools and libraries that are specifically made for robotic systems such as visualization, mapping, odometry, etc. Here's a bit of starting info:
https://en.wikipedia.org/wiki/Robot_Operating_System
http://wiki.ros.org/ROS/StartGuide
It's usually used for much larger frameworks than you seem to be describing, and beware that it takes quite a bit of time (in my experience) to implement, but it is very easy to expand once its up and running. Like I said, it all depends on the scope of your project!
Good luck!

Broken Pipe with socket connection

I am using a socket connection to download data through a third party API. It works fine for a while but every now and then my script will crash giving the following error: BrokenPipeError: [Errno 32] Broken pipe
After some research it seems the suggestion (link here) is to do the following:
from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE,SIG_DFL)
However im firstly not sure what this actually does (im still confused after reading the python manual on signal). And I also don't know where to put the code.
If anyone is familiar with this error please could you advise if this is infact the correct solution and where the signal(SIGPIPE,SIG_DFL) would be placed. Should there be a try/except block inside which this is placed, or is it simply placed at the start of the program? Im confused.
Here's some of the relevant code. I basically have a dataframe consisting of several thousand items. I loop through each item passing it to the download method. The download method downloads the data via the api and then writes it to a database. I then move to the next item to download.
def recv_data(sock, recv_buffer=4096, delim='\n'):
buffer = ''
data = True
while data:
data = sock.recv(recv_buffer)
buffer += str(data.decode('latin-1'))
while buffer.find(delim) != -1:
line, buffer = buffer.split('\n', 1)
yield line
def update_existing_symbol_data(engine, sock, exchange, exchange_id, symbol, symbol_id, start_date):
data = ''
message = #request data message
sock.sendall(message.encode())
for line in recv_data(sock):
if "!ENDMSG!" in line:
break
data += line[:-2] + '\n'
df = pd.read_csv(io.StringIO(data))
df.set_index('date', inplace=True)
df.to_sql('daily', engine, if_exists='append')
def main():
df = #dataframe all symbols that need to be downloaded
for index, row in df.iterrows():
update_existing_symbol_data(args)

SIGPIPE is a POSIX thing that gets sent when a socket write operation fails. The default behavior is for the signal (this is an OS/socket thing, not a Python thing) to just kill your process. Python instead gives it to you as an exception so that it's possible to write more robust programs. But if you don't need to handle that event, which it sounds like you don't considering your use case, you can safely ignore it. There's no logic you need to do when you receive the signal, so the solution from that blog post should be fine. No try/except needed.
If your use case changes at a later date and you do need to handle the SIGPIPE, then wrapping that in a try/except and handling it there would be the way to go.

Python 3 (Bot) script stops working

I'm trying to connect to a TeamSpeak server using the QueryServer to make a bot. I've taken advice from this thread, however I still need help.
This is The TeamSpeak API that I'm using.
Before the edits, this was the summary of what actually happened in my script (1 connection):
It connects.
It checks for channel ID (and it's own client ID)
It joins the channel and starts reading everything
If someone says an specific command, it executes the command and then it disconnects.
How can I make it so it doesn't disconnect? How can I make the script stay in a "waiting" state so it can keep reading after the command is executed?
I am using Python 3.4.1.
I tried learning Threading but either I'm dumb or it doesn't work the way I thought it would. There's another "bug", once waiting for events, if I don't trigger anything with a command, it disconnects after 60 seconds.
#Librerias
import ts3
import threading
import datetime
from random import choice, sample
# Data needed #
USER = "thisisafakename"
PASS = "something"
HOST = "111.111.111.111"
PORT = 10011
SID = 1
class BotPrincipal:
def __init__(self, manejador=False):
self.ts3conn = ts3.query.TS3Connection(HOST, PORT)
self.ts3conn.login(client_login_name=USER, client_login_password=PASS)
self.ts3conn.use(sid=SID)
channelToJoin = Bot.GettingChannelID("TestingBot")
try: #Login with a client that is ok
self.ts3conn.clientupdate(client_nickname="The Reader Bot")
self.MyData = self.GettingMyData()
self.MoveUserToChannel(ChannelToJoin, Bot.MyData["client_id"])
self.suscribirEvento("textchannel", ChannelToJoin)
self.ts3conn.on_event = self.manejadorDeEventos
self.ts3conn.recv_in_thread()
except ts3.query.TS3QueryError: #Name already exists, 2nd client connect with this info
self.ts3conn.clientupdate(client_nickname="The Writer Bot")
self.MyData = self.GettingMyData()
self.MoveUserToChannel(ChannelToJoin, Bot.MyData["client_id"])
def __del__(self):
self.ts3conn.close()
def GettingMyData(self):
respuesta = self.ts3conn.whoami()
return respuesta.parsed[0]
def GettingChannelID(self, nombre):
respuesta = self.ts3conn.channelfind(pattern=ts3.escape.TS3Escape.unescape(nombre))
return respuesta.parsed[0]["cid"]
def MoveUserToChannel(self, idCanal, idUsuario, passCanal=None):
self.ts3conn.clientmove(cid=idCanal, clid=idUsuario, cpw=passCanal)
def suscribirEvento(self, tipoEvento, idCanal):
self.ts3conn.servernotifyregister(event=tipoEvento, id_=idCanal)
def SendTextToChannel(self, idCanal, mensajito="Error"):
self.ts3conn.sendtextmessage(targetmode=2, target=idCanal, msg=mensajito) #This works
print("test") #PROBLEM HERE This doesn't work. Why? the line above did work
def manejadorDeEventos(sender, event):
message = event.parsed[0]['msg']
if "test" in message: #This works
Bot.SendTextToChannel(ChannelToJoin, "This is a test") #This works
if __name__ == "__main__":
Bot = BotPrincipal()
threadprincipal = threading.Thread(target=Bot.__init__)
threadprincipal.start()
Prior to using 2 bots, I tested to launch the SendTextToChannel when it connects and it works perfectly, allowing me to do anything that I want after it sends the text to the channel. The bug that made entire python code stop only happens if it's triggered by the manejadorDeEventos
Edit 1 - Experimenting with threading.
I messed it up big time with threading, getting to the result where 2 clients connect at same time. Somehow i think 1 of them is reading the events and the other one is answering. The script doesn't close itself anymore and that's a win, but having a clone connection doesn't looks good.
Edit 2 - Updated code and actual state of the problem.
I managed to make the double connection works more or less "fine", but it disconnects if nothing happens in the room for 60 seconds. Tried using Threading.timer but I'm unable to make it works. The entire question code has been updated for it.
I would like an answer that helps me to do both reading from the channel and answering to it without the need of connect a second bot for it (like it's actually doing...) And I would give extra points if the answer also helps me to understand an easy way to make a query to the server each 50 seconds so it doesn't disconnects.

From looking at the source, recv_in_thread doesn't create a thread that loops around receiving messages until quit time, it creates a thread that receives a single message and then exits:
def recv_in_thread(self):
"""
Calls :meth:`recv` in a thread. This is useful,
if you used ``servernotifyregister`` and you expect to receive events.
"""
thread = threading.Thread(target=self.recv, args=(True,))
thread.start()
return None
That implies that you have to repeatedly call recv_in_thread, not just call it once.
I'm not sure exactly where to do so from reading the docs, but presumably it's at the end of whatever callback gets triggered by a received event; I think that's your manejadorDeEventos method? (Or maybe it's something related to the servernotifyregister method? I'm not sure what servernotifyregister is for and what on_event is for…)
That manejadorDeEventos brings up two side points:
You've declared manejadorDeEventos wrong. Every method has to take self as its first parameter. When you pass a bound method, like self.manejadorDeEventos, that bound self object is going to be passed as the first argument, before any arguments that the caller passes. (There are exceptions to this for classmethods and staticmethods, but those don't apply here.) Also, within that method, you should almost certainly be accessing self, not a global variable Bot that happens to be the same object as self.
If manejadorDeEventos is actually the callback for recv_in_thread, you've got a race condition here: if the first message comes in before your main threads finishes the on_event assignment, the recv_on_thread won't be able to call your event handler. (This is exactly the kind of bug that often shows up one time in a million, making it a huge pain to debug when you discover it months after deploying or publishing your code.) So, reverse those two lines.
One last thing: a brief glimpse at this library's code is a bit worrisome. It doesn't look like it's written by someone who really knows what they're doing. The method I copied above only has 3 lines of code, but it includes a useless return None and a leaked Thread that can never be joined, not to mention that the whole design of making you call this method (and spawn a new thread) after each event received is weird, and even more so given that it's not really explained. If this is the standard client library for a service you have to use, then you really don't have much choice in the matter, but if it's not, I'd consider looking for a different library.

Twisted transport.write

Is there any way to force self.transport.write(response) to write immediately to its connection so that the next call to self.transport.write(response) does not get buffered into the same call.
We have a client with legacy software we cannot amend, that reads for the 1st request and then starts reading again, and the problem I have is twisted joins the two writes together which breaks the client any ideas i have tried looking into deferreds but i don't think it will help in this case
Example:
self.transport.write("|123|") # amount of messages to follow
a loop to generate next message
self.transport.write("|message 1 text here|")
Expected:
|123|
|message 1 text here|
Result:
|123||message 1 text here|

I was having a somewhat related problem using down level Python 2.6. The host I was talking to was expecting a single ACK character, and THEN a separate data buffer, and they all came at once. On top
of this, it was a TLS connection. However, if you reference the socket DIRECTLY, you can invoke a
sendall() as:
self.transport.write(Global.ACK)
to:
self.transport.getHandle().sendall(Global.ACK)
... and that should work. This does not seem to be a problem on Python 2.7 with Twisted on X86, just
Python 2.6 on a SHEEVAPlug ARM processor.

Can you tell which transport you are using. For most implementations,
This is the typical approach :
def write(self, data):
if data:
if self.writeInProgress:
self.outQueue.append(data)
else:
....
Based on the details the behavior of write function can be changed to do as desired.

Maybe You can register your protocol as a pull producer to the transport
self.transport.registerProducer(self, False)
and then create a write method in your protocol that has it's job buffering
the data until the transport call your protocol resumeProducing method to fetch
the data one by one.
def write(self, data):
self._buffers.append(data)
def resumeProducing(self):
data = self._buffers.pop()
self.transport.write(data)