Develop Reference

How do I send input from python cmd to autobahn websocket client running in the same interpreter?

I'm trying to take input from an interactive prompt via python's cmd library and pass the input to an autobahn websocket client to send up to a websocket server. The cmd loop and the autobahn websocket client loop are running in the same interpreter. I'm trying to use crochet to make that work. The websocket client connects to the server successfully but when I enter something at the cmd prompt to invoke sendMessage, I get the exception shown at the bottom of this post. Any guidance on where I may have messed up would be much appreciated. If there's a better approach to accomplish what I'm trying to do, I'm all ears.
These are the relevant imports and setup:
from cmd import Cmd
from crochet import setup, run_in_reactor, wait_for, retrieve_result, TimeoutError
# Setup crochet before importing twisted
setup()
from twisted.internet import reactor, ssl
from twisted.python import log
from autobahn.twisted.websocket import WebSocketClientFactory, \
WebSocketClientProtocol, \
connectWS
This is the websocket client protocol class:
class MyClientProtocol(WebSocketClientProtocol):
def __init__(self, *args, **kwargs):
super(MyClientProtocol, self).__init__(*args, **kwargs)
def onConnect(self, response):
print("Connected")
def onMessage(self, payload, isBinary):
if not isBinary:
print('Message received: {}'.format(payload.decode('utf8')))
def sendTask(self, payload):
payload = json.dumps(payload, ensure_ascii = False).encode('utf8')
self.sendMessage(payload)
This is the websocket client factory class:
class MyClientFactory(WebSocketClientFactory):
def __init__(self, *args, **kwargs):
super(MyClientFactory, self).__init__(*args, **kwargs)
def buildFactory(self, uri, headers):
factory = WebSocketClientFactory(uri, headers=headers)
factory.protocol = MyClientProtocol
return factory
This is the cmd class that sends input to the websocket client:
class mycmd(Cmd):
def do_send(self, inp):
payload = {'task': inp}
m = MyClientProtocol()
reactor.callFromThread(m.sendTask, payload)
This is how I'm calling the websocket client and cmd loop:
if __name__ == '__main__':
#run_in_reactor
def start_connectWS():
headers = {'header1': 'value1'}
f = MyClientFactory()
connectStatement = f.buildFactory(uri, headers)
if connectStatement.isSecure:
contextFactory = ssl.ClientContextFactory()
else:
contextFactory = None
connectWS(connectStatement, contextFactory)
start_connectWS()
mycmd().cmdloop()
This is the exception:
Unhandled Error
Traceback (most recent call last):
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/Users/tomd/project/lib/python3.7/site-packages/crochet/_eventloop.py", line 412, in <lambda>
target=lambda: self._reactor.run(installSignalHandlers=False),
File "/Users/tomd/project/lib/python3.7/site-packages/twisted/internet/base.py", line 1283, in run
self.mainLoop()
File "/Users/tomd/project/lib/python3.7/site-packages/twisted/internet/base.py", line 1292, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File "/Users/tomd/project/lib/python3.7/site-packages/twisted/internet/base.py", line 886, in runUntilCurrent
f(*a, **kw)
File "./client.py", line 62, in sendTask
self.sendMessage(payload)
File "/Users/tomd/project/lib/python3.7/site-packages/autobahn/websocket/protocol.py", line 2215, in sendMessage
if self.state != WebSocketProtocol.STATE_OPEN:
builtins.AttributeError: 'MyClientProtocol' object has no attribute 'state'

Your copmmand class creates a new, unconnected protocol instance and then tries to use it as if it were connected:
class mycmd(Cmd):
def do_send(self, inp):
payload = {'task': inp}
m = MyClientProtocol()
reactor.callFromThread(m.sendTask, payload)
Specifically, this creates a new instance of your protocol class:
m = MyClientProtocol()
And this tries to use it as if it were connected:
reactor.callFromThread(m.sendTask, payload)
Later on you have code that actually connects a protocol to something:
connectWS(connectStatement, contextFactory)
However this code is not connected to your command class in any useful way.
Instead of creating a new MyClientProtocol instance you need to use the connection that results from calling connectWS.
There are many ways you could go about this which different trade-offs. One way which happens to be easy to explain is to use mutable state shared between the websocket code and the command interpreter code.
For example, MyClientProtocol.onConnect can set itself as an attribute on the factory instance and your command line code could accept the factory instance as an argument, then read the connected protocol instance from the attribute.
class MyClientProtocol(...):
def onConnect(self, response):
self.factory.connectedProtocol = self
...
class mycmd(Cmd):
# ... __init__ that accepts factory and sets it on self
def do_send(self, inp):
payload = {'task': inp}
m = self.factory.connectedProtocol
if m is None:
print("No connection")
else:
reactor.callFromThread(m.sendTask, payload)

Why Thrift raises AssertionError when requested by multiple threads?

i'm building a kind of simulator that uses thrift protocol.
But when im executing multiple threads of my virtual equipments sending messages, the program breaks after a short time by receiving them, think the buffer is overloaded or something like that, or not, so i'm here asking for some help if its possible.
Here's the main pieces of my code
A class for threading:
class ThreadManager (threading.Thread):
def __init__(self, name, obj, client, layout):
threading.Thread.__init__(self)
self.name = name
self.obj = obj
self.client = client
self.layout = layout
def run(self):
print ("Starting " + self.name)
while True:
sleep(2)
self.obj.auto_gen_msg(self.client, layout=self.layout)
The method for generating messages:
def auto_gen_msg(self, client, layout='', min_delay=15, max_delay=30):
if not layout:
msg = self.gen_message(self.draw_random_model())
else:
msg = self.gen_message(layout)
wait = randint(min_delay, max_delay)
sleep(wait)
print(self.eqp_type, " delivered a message ...")
getattr(client, msg[0])(*msg[1])
The main:
def start(layout, equipment, number):
try:
host = 'localhost'
transport = TSocket.TSocket(host, port=9090)
transport = TTransport.TBufferedTransport(transport)
protocol = TCompactProtocol.TCompactProtocol(transport)
client = SuiteService.Client(protocol)
transport.open()
equips = [Equipment(equipment) for i in range(number)]
threads = [ThreadManager(i.eqp_type, i, client, layout) for i in equips]
for i in range(len(threads)):
threads[i].start()
sleep(2)
while True:
pass
transport.close()
except Thrift.TException as tx:
print ("%s " % (tx.message))
The error haunting me:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/Users/lem4fia/Documents/sg/loki/loki-thrift/loki_thrift/loki_thrift/lib/thread_manager.py", line 39, in run
self.obj.auto_gen_msg(self.client, layout=self.layout)
File "/Users/lem4fia/Documents/sg/loki/loki-thrift/loki_thrift/loki_thrift/lib/virtual.py", line 281, in auto_gen_msg
getattr(client, msg[0])(*msg[1])
File "/Users/lem4fia/Documents/sg/loki/thrift-server/thrift_server/suite/SuiteService.py", line 4895, in v1
self.send_v1(ir, ts, ch, o1, o2, o3, o4, o5, o6, o7)
File "/Users/lem4fia/Documents/sg/loki/thrift-server/thrift_server/suite/SuiteService.py", line 4899, in send_v1
self._oprot.writeMessageBegin('v1', TMessageType.CALL, self._seqid)
File "/Users/lem4fia/Documents/sg/loki/lokiv/lib/python3.6/site-packages/thrift-0.11.0-py3.6-macosx-10.6-intel.egg/thrift/protocol/TCompactProtocol.py", line 156, in writeMessageBegin
assert self.state == CLEAR
AssertionError
Curiously, it doesnt bug if instancing 2 virtual equipments in thread, but 10 virtual equipments (sometimes less than this) is sufficient to raise this error.
Can someone please gimme a light? :)

The problem there is that it seems that you have to use one diferent Transport object for each thread. This is probably related to Thrift's implementation!
Reference here : http://grokbase.com/t/thrift/user/134s16ks4m/single-connection-and-multiple-threads

As a general rule 1), Thrift is not intended to be used across multiple threads.
This is, at least to my knowledge, true for all currently supported languages.
One instance per thread will do.
1) aside from server-end things like TThreadedServer or TThreadPoolServer

PyZMQ Subscriber doesn't receive messages when working with request socket

I am working with PyZMQ and I have what seems to be a rather peculiar issue. I have two classes that are wrapping sockets for communication, MZLSubscriber and MZLRequester. There is a class that contains both of them, MZLLink. For each of these, I also have tests MZLSubscriberTest, MZLRequesterTest, and MZLinkTest. The tests for the subscriber and requester work as they should, but MZLinkTest does not receive any subscriber messages.
Below is what seems to be the relative code, which are the constructors for the 3 classes as well as run() for MZLSubscriber and the tests for MZLink and MZLSubscriber.
MZLink Constructor:
# Host information
self.host = host
self.requestPort = requestPort
self.subscriberPort = subscriberPort
# Set up zmq context
self.zmq_context = zmq.Context()
# Set up subscriber and replier
self.subscriber = MZLSubscriber(self.zmq_context, self.host, self.subscriberPort)
self.requester = MZLRequester(self.zmq_context, self.host, self.requestPort)
# Start subscriber
self.subscriber.start()
MZLink Test:
# Constants
HOST = "localhost"
REQ_PORT = 5555
SUB_PORT = 5556
# Create Link
link = MIDASZMQLink(HOST, REQ_PORT, SUB_PORT)
link.close()
MZLRequester Constructor:
# Initialize class member variables
self.zmq_context = zmq_context
self.host = host
self.port = port
# Set up reply socket
self.socket = self.zmq_context.socket(zmq.REQ)
# Connect socket
self.socket.connect("tcp://{0}:{1}".format(self.host, self.port))
MZLSubscriber Constructor:
# Initialize parent process
Process.__init__(self)
# Store zmq context and connection host/port
self.zmq_context = zmq_context
self.host = host
self.port = port
# Sockets. Don't set them up here because sockets are not thread safe
self.socket = None
# Queue to store data in
# TODO: Make queue not overflow if events come in too quickly
self.queue = Queue()
MZLSubscriber.run():
# Parent call
Process.run(self)
# Set up subscriber socket in this thread
self.socket = self.zmq_context.socket(zmq.SUB)
self.socket.setsockopt_string(zmq.SUBSCRIBE, unicode())
# Connect socket
self.socket.connect("tcp://{0}:{1}".format(self.host, self.port))
# While the thread is alive, poll for data to put into queue
# Calling MZLSubscriber.stop() will automatically change this
while self.is_alive():
datum = self.socket.recv()
self.queue.put(datum)
# Disconnect and close socket.
#FIXME: Doesn't get here because terminate() immediately stops the process
self.socket.disconnect("tcp://{0}:{1}".format(self.host, self.port))
self.socket.close()
MZLSubscriber Test:
# Host information
HOST = "localhost"
SUBSCRIBER_PORT = "5556"
# Set up zmq context
zmq_context = zmq.Context()
# Set up subscriber
subscriber = MZLSubscriber(zmq_context, HOST, SUBSCRIBER_PORT)
# Start subscriber
subscriber.start()
# Stop and join subscriber
subscriber.close()
subscriber.join()
The subscriber thread seems to block at datum = self.socket.recv(), which makes me think it could be some issue with the socket creation. However, it does seem to work when only working with the subscriber. The requester seems to work in both cases. In addition, everything goes smoothly by just commenting out the two lines dealing with requester.
I apologize for the wall of code, but I can't even narrow what code the issue is coming from at this point. When I do, I'll remove the irrelevant code. The test code that deals with the incoming data has been removed.
As a bit of clarification, I am using Python 2.7 with PyZMQ 14.3.1.
UPDATE: It seems that running MZLSubscriber in the main thread rather than creating another Process results in the expected result, so it seems that this could be some sort of thread safety. To my knowledge, zmq contexts are thread-safe, but sockets are not. I thought this wouldn't cause an issue because I'm explicitly making sure there is a socket for each thread.
UPDATE 2: If the calls setting up the socket in MZLSubscriber are moved from run() to __init__, the socket seems to receive a small portion of the published message, but does have an error:
Process MZLSubscriber-1:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/Users/user/Repos/midas-client/client/midasclient/mzlSubscriber.py", line 45, in run
datum = self.socket.recv()
File "socket.pyx", line 628, in zmq.backend.cython.socket.Socket.recv (zmq/backend/cython/socket.c:5616)
File "socket.pyx", line 662, in zmq.backend.cython.socket.Socket.recv (zmq/backend/cython/socket.c:5436)
File "socket.pyx", line 139, in zmq.backend.cython.socket._recv_copy (zmq/backend/cython/socket.c:1771)
File "checkrc.pxd", line 21, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:6032)
ZMQError: Interrupted system call
I have gotten a workaround to this by creating a new zmq.Context in MZLSubscriber.run(), although I feel that this shouldn't be necessary if zmq contexts are thread-safe.

It seems that my issue was using multiple zmq contexts over different processes. While the PyZMQ documentation states that the zmq context is thread-safe, I can only assume it meant Python threads rather than processes. This is quite confusing as in C, zmq contexts are thread safe despite running in a way similar to the Python multiprocessing.Process.
The issue was solved by creating a zmq context for each Process.

python httpserver on separate thread

I am trying to create an httpserver on a separate thread which should process a single get request and shutdown if a single parameter is passed in the url.
import sys
import threading
import BaseHTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler
class WebServer(SimpleHTTPRequestHandler,threading.Thread):
port = 9000
protocol_version = "HTTP/1.1"
code = ""
httpd = None
isRunning = False
def do_GET(self):
self.send_response(200)
self.send_header("Content-type","text/html")
self.end_headers()
self.wfile.write("webpage")
try:
self.code = split(self.path,"=")[1]
except:
pass
else:
self.isRunning=False
self.stop()
def do_POST(self):
pass
def __init__(self,port=9000):
threading.Thread.__init__(self)
self.port=port
def run(self):
while True:
server_address = ("127.0.0.1",self.port)
try:
self.httpd = BaseHTTPServer.HTTPServer(server_address,WebServer)
except:
self.port +=1
else:
break
self.isRunning=True
self.httpd.serve_forever()
server = WebServer(1)
server.start()
while(server.isRunning==False):
pass
print "Server running on port: %d" %(server.port)
server.join()
print "code: "+server.code
But an error occurs when a request is sent to it:
Server running on port: 1025
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 56462)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
TypeError: __init__() takes at most 2 arguments (4 given)
----------------------------------------
when using curl to connect to the server,it gives this error:
curl: (52) Empty reply from server

you're trying to do too many things with one class.
the way that the SimpleHTTPRequestHandler works is that the BaseHTTPServer creates a new instance to handle each request. it does that by calling the constructor. but you have added a new constructor which you are using to create the server itself.
is that clear?
when the BaseHttpServer receives a request it takes the WebServer class (which you pass in as a subclass of SimpleHTTPRequestHandler where you create self.httpd) and tries to create an instance. it expects the constructor for the SimpleHTTPRequestHandler subclass to have four arguments, but you've added a new constructor which takes only two (and is intended to create a web server, not a handler).
it's weird because you're probably not used to things creating a new instance. but that's how it works.
so separate your server and your request handler into two separate classes (as kichik says).

Python Sharing a network socket with multiprocessing.Manager

I am currently writing a nginx proxy server module with a Request queue in front, so the requests are not dropped when the servers behind the nginx can't handle the requests (nginx is configured as a load balancer).
I am using
from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler
The idea is to put the request in a queue before handling them. I know multiprocessing.Queue supports only simple object and cannot support raw sockets, so I tried using a multiprocess.Manager to make a shared dictionary. The Manager also uses sockets for connection, so this method failed too. Is there a way to share network sockets between processes?
Here is the problematic part of the code:
class ProxyServer(Threader, HTTPServer):
def __init__(self, server_address, bind_and_activate=True):
HTTPServer.__init__(self, server_address, ProxyHandler,
bind_and_activate)
self.manager = multiprocessing.Manager()
self.conn_dict = self.manager.dict()
self.ticket_queue = multiprocessing.Queue(maxsize= 10)
self._processes = []
self.add_worker(5)
def process_request(self, request, client):
stamp = time.time()
print "We are processing"
self.conn_dict[stamp] = (request, client) # the program crashes here
#Exception happened during processing of request from ('172.28.192.34', 49294)
#Traceback (most recent call last):
# File "/usr/lib64/python2.6/SocketServer.py", line 281, in _handle_request_noblock
# self.process_request(request, client_address)
# File "./nxproxy.py", line 157, in process_request
# self.conn_dict[stamp] = (request, client)
# File "<string>", line 2, in __setitem__
# File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in _callmethod
# conn.send((self._id, methodname, args, kwds))
#TypeError: expected string or Unicode object, NoneType found
self.ticket_queue.put(stamp)
def add_worker(self, number_of_workers):
for worker in range(number_of_workers):
print "Starting worker %d" % worker
proc = multiprocessing.Process(target=self._worker, args = (self.conn_dict,))
self._processes.append(proc)
proc.start()
def _worker(self, conn_dict):
while 1:
ticket = self.ticket_queue.get()
print conn_dict
a=0
while a==0:
try:
request, client = conn_dict[ticket]
a=1
except Exception:
pass
print "We are threading!"
self.threader(request, client)

U can use multiprocessing.reduction to transfer the connection and socket objects between processes
Example Code
# Main process
from multiprocessing.reduction import reduce_handle
h = reduce_handle(client_socket.fileno())
pipe_to_worker.send(h)
# Worker process
from multiprocessing.reduction import rebuild_handle
h = pipe.recv()
fd = rebuild_handle(h)
client_socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)
client_socket.send("hello from the worker process\r\n")

Looks like you need to pass file descriptors between processes (assuming Unix here, no clue about Windows). I've never done this in Python, but here is link to python-passfd project that you might want to check.

You can look at this code - https://gist.github.com/sunilmallya/4662837 which is
multiprocessing.reduction socket server with parent processing passing connections to client after accepting connections