Develop Reference

Simulate a Long Network Request for Python Testing

I need to test a device update function. The function opens a socket on a host and sends a block of text.
The update can take up to 120 seconds. It returns a code for success/failure. To allow continued functioning of the program the update is launched in a thread.
I cannot control the response of the device. The simulation needs to be able to hold an open connection for at least 120 seconds.
It does not need to be safe or scalable since it will only be used for an integration test. The simplest solution is preferred. Pure python is best, but a docker is also acceptable.

I wrote this up based on rdas's pointer.
import json
import logging
import socket
import socketserver
import threading
import time
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
class LongRequestHandler(socketserver.BaseRequestHandler):
def handle(self):
# Echo the back to the client
data = json.loads(self.request.recv(1024).decode())
t = 0
while t < data['delay']:
time.sleep(1)
print(".", end='')
t += 1
if t % 80 == 0:
print("\n")
print("\n")
self.request.send(b"ok")
class Server():
def __init__(self, host='localhost', port=0):
self.host = host
self.port = port
self.ip = None
self.server = None
def run(self):
address = (self.host, self.port) # let the kernel assign port if port=0
self.server = socketserver.TCPServer(address, LongRequestHandler)
self.ip, self.port = self.server.server_address # what port was assigned?
t = threading.Thread(target=self.server.serve_forever)
t.setDaemon(True) # don't hang on exit
t.start()
return True
def send_request(self, data: dict ):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((self.ip, self.port))
message = json.dumps(data).encode()
s.send(message)
response = s.recv(1024)
s.close()
return response
def __exit__(self):
self.server.shutdown()
self.server.socket.close()
if __name__ == '__main__':
# For simple testing and config example...
server = Server()
server.run()
# Send the data
d = dict(delay=5) # set delay here to desired
out = server.send_request(d)
print('Received: {!r}'.format(out))

Python sockets; sending from client receiving on server

I am trying to send messages on TCP/IP all on host machine. This is working, although for some reason the socket needs to be re-instantiated for every new message on the client side only. For example here is a basic client that sends three separate messages:
import socket
host = '127.0.0.1'
class Client:
def __init__(self):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def connect(self):
self.sock.connect((host,12347))
def send(self,message):
self.sock.sendall(message)
def close(self):
self.sock.close()
if __name__ == "__main__":
message1 = "I am message 1"
message2 = "I am message 2"
message3 = "I am message 3"
#exp = Client()
#exp.connect()
for i in range(0,3):
try:
exp = Client()
exp.connect()
if i == 0:
txt = message1
elif i == 1:
txt = message2
elif i == 2:
txt = message3
exp.send(txt)
exp.close()
print i
exp.send(txt)
except:
pass
and the server that receives:
#!/usr/bin/env python
import socket
class communication:
def __init__(self):
try:
host = '127.0.0.1'
self.Server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.Server.bind((host,12347))
self.Server.listen(1)
finally:
print "setup finished"
def recieve(self):
(connection, client_address) = self.Server.accept()
data = connection.recv(128)
return data
def close(self):
self.server.close()
if __name__ == "__main__":
exp = communication()
while True:
try:
(connection,client_address) = exp.Server.accept()
message = connection.recv(128)
finally:
print message
if message == "I am message 3":
exp.close()
You see how I re-call the Client class in each iteration of the for loop. This seems to be necessary for sending messages 2 and 3. If the socket is instantiated only once at the start of the main code along with the connect() function, then the server hangs on the recv() after the first message has been sent.
I can't understand why this is happening and the socket only needs to be setup once on the server side. I am doing something wrong, or is this normal?
Thanks!

It's even worse than you think. Take a look at your server code. exp.Server.accept() accepts a connection from the client, but connection.receive() ignores that connection completely and does a second self.Server.accept(). You ignore half of your connections!
Next, your server only does a single receive.... Even if you tried to send more messages on the connection, the server would ignore them.
But you can't just add a recv loop. Your client and server need some way to mark message boundaries so the server knows how to pull them out. Some text based systems use a new line. Others send a message size or fixed size header that the server can read. HTTP for example uses a combination of new lines and data count.
If you want to learn sockets from the ground up just know that they are complicated and you'll need to study. There are lots of ways to build a server and you'll need to understand the trade-offs. Otherwise, there are many frameworks from XMLRPC to zeromq that do some of the heavy lifting for you.

UPnP event subscription using blocking Socket on a Thread

I am tyring to subsribe to an event on a UPnP device (the WeMo motion sensor). I first send an HTTP subscribe request to the device, and the device should start sending me event notification on the designated address. That part is working fine (except that I am getting too many notifications; even when the status is not changing, but it is a different problem for a different thread)
If I run the keepListening Function on a separate python process, everything is working fine . However, when I run the function as a thread, it doesn't work;
import socket
import requests
from threading import Thread
def keepListening(): #running this function on a separate process works
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.settimeout(600)
sock.bind(('192.168.10.231',1234))
sock.listen(5)
while 1:
notification = ''
try:
conn, addr = sock.accept()
conn.setblocking(1)
notification= conn.recv(1024)
conn.sendall(r'''HTTP/1.1 200 OK
Content-Type: text/plain
''')
except Exception as er:
print er
print notification
x = Thread(target=keepListening)
x.start()
message = {
'CALLBACK': '<http://192.168.10.231:1234>',
'NT': 'upnp:event',
'TIMEOUT': 'Second-600',
'HOST': '192.168.10.159:49153'}
k = requests.request('SUBSCRIBE','http://192.168.10.159:49153/upnp/event/basicevent1',headers=message)
print k
# keep doing other important works
Each event notification must be replied with a 200 OK reply, otherwise the device won't send further notification; a fact I learned the hard way. A doubt I have, which might be silly, is that, when running in a thread, as opposed to a separate process, the reply message doesn't get sent in timely manner, so the device doesn't send any more notifications.
It is worth mentioning that, even when I run the function in a Thread, I do get the initial notification after the subscription (Devices must mandatorily send an initial notification right after a subscription according to UPnP protocol), but I get no further notification (indicating that my 200 OK reply didn't get through properly; I do see it in wireshark though)
Any idea on what might be the difference in running the function in a thread (as opposed to a separate process) that makes it fail?
Thank you.

I would assume, what is happening is that you end up sending your subscribe request before thread becomes active and starts listening on the interface. So the device can not connect to the socket.

A few day ago I got a wemo motion sensor, switch and RaspberryPi, so I started tinkering.
The script subscribes to the „binaryState“-event of the wemo-device.
Every time the event occurs it prints out an „Alert“ (you can do other things there).
After 250 seconds it renews the subscription.
To adapt the script to your needs, you have to change the IPs:
localIp : your Computer
remoteIp: the ip of the wemo-sensor or switch
I’m new to python (started 3 days ago), so the script might need some revision, but it works.
import socket
import threading
import requests
host = ''
port = 1234
localIp = '<http://192.168.1.32:1234>' # local IP of your computer
remoteIp = '192.168.1.47:49153' # the ip of the wemo device
global uid # stores the uuid of the event
uid = ''
class client(threading.Thread):
def __init__(self, conn):
super(client, self).__init__()
self.conn = conn
self.data = ""
def run(self):
global uid
while True:
self.data = self.data + self.conn.recv(1024)
if self.data.endswith(u"\r\n"):
print self.data # data from the wemo device
uidPos = self.data.find("uuid")
if uidPos != -1: # data contains the uuid of the event
uid = self.data[uidPos+5:uidPos+41]
if "<BinaryState>1</BinaryState>" in self.data:
print "ALERT ------------------------------------------Alert"
# NOTIFICATION !
if "/e:propertyset" in self.data:
self.conn.sendall('HTTP/1.1 200 OK\r\nContent-Type:text/html\r\n\r\n')
return
self.data = ""
def send_msg(self,msg):
self.conn.send(msg)
def close(self):
self.conn.close()
class connectionThread(threading.Thread):
def __init__(self, host, port):
super(connectionThread, self).__init__()
try:
self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.s.bind((host,port))
self.s.listen(5)
except socket.error:
print 'Failed to create socket'
sys.exit()
self.clients = []
def run(self):
while True:
print uid
conn, address = self.s.accept()
c = client(conn)
c.start()
print '[+] Client connected: {0}'.format(address[0])
def main():
get_conns = connectionThread(host, port)
get_conns.start()
print get_conns.clients
while True:
try:
response = raw_input()
except KeyboardInterrupt:
sys.exit()
def setCalback():
global uid
threading.Timer(250.0, setCalback).start()
if uid == "": # no uuid set so we subscribe to the event
eventSubscribe()
else: # uuid is set, so we renew the subsciption
eventRefresh()
def eventSubscribe(): # subscribe to the wemo-event
message = {
'CALLBACK': localIp,
'NT': 'upnp:event',
'TIMEOUT': 'Second-300',
'HOST': remoteIp}
k = requests.request('SUBSCRIBE', "http://"+remoteIp+'/upnp/event/basicevent1',headers=message)
print k
def eventRefresh() # refresh the subscription with the known uuid
myuid = "uuid:"+uid
message = {
'SID': myuid,
'TIMEOUT': 'Second-300',
'HOST': remoteIp }
k = requests.request('SUBSCRIBE',"http://"+remoteIp+'/upnp/event/basicevent1',headers=message)
print k
if __name__ == '__main__':
threading.Timer(2.0, setCalback).start() # wait 2 sec. then subscribe to the service
main()

When combine TCPServer with ThreadingMixIn, it block

the server-side code (tcp_server.py):
from SocketServer import TCPServer, ThreadingMixIn, StreamRequestHandler
class Server(ThreadingMixIn, TCPServer):
pass
class Handler(StreamRequestHandler):
def handle(self):
print 'got a connection from: ', self.request.getpeername()
print self.rfile.read(1024)
msg = 'hello'
self.wfile.write(msg)
server = Server(('127.0.0.1', 8888), Handler)
server.serve_forever()
the client-side code (tcp_client.py):
from socket import *
import threading
def do_connection():
s = socket(AF_INET, SOCK_STREAM)
s.connect(('127.0.0.1', 8888))
s.sendall('this is client')
print s.recv(1024)
ts = []
for x in xrange(100):
print x
t = threading.Thread(target=do_connection())
t.daemon = True
ts.append(t)
for t in ts:
t.start()
I runned tcp_server.py, and then tcp_client.py. tcp_client.py should have been over soon. However, tcp_client.py seemed just run only one thread and blocked, and tcp_server.py got only one connection. When I interrupted tcp_client.py，tcp_server.py got one message this is client。
Is there any mistake in my code ?

This line:
t = threading.Thread(target=do_connection())
Should be
t = threading.Thread(target=do_connection)
When you use do_connection(), you end up executing do_connection in the main thread, and then pass the return value of that call to the Thread object. What you want to do is pass the do_connection function object to Thread, so that the Thread object can execute do_connection in a new thread when you call t.start.
Also, note that starting 100 threads concurrently to connect to your server may not perform very well. You may want to consider starting with fewer threads, and working your way up to a higher number once you know things are working properly.

because the server is blocked by the first request, I try to change the read(1024) to
readline in the server.py and add a '\n' to the content sended from the client, it
works.
it seems the rfile.read(1024) will block the how process, so the goodway is use readline
or use the self.request.recv(1024)
server.py
from SocketServer import TCPServer, ThreadingMixIn, StreamRequestHandler
class Server(ThreadingMixIn, TCPServer):
pass
class Handler(StreamRequestHandler):
def handle(self):
print 'got a connection from: ', self.request.getpeername()
print self.rfile.readline()
#print self.request.recv(1024).strip()
msg = 'hello'
self.wfile.write(msg)
# Create the server, binding to localhost on port 9999
server = Server(("127.0.0.1", 8888), Handler)
server.serve_forever()
client.py
from socket import *
import threading
def do_connection():
print "start"
s = socket(AF_INET, SOCK_STREAM)
s.connect(('127.0.0.1', 8888))
s.sendall('this is client\n')
print s.recv(1024)
ts = []
for x in xrange(100):
print x
t = threading.Thread(target=do_connection)
ts.append(t)
for t in ts:
print "start t"
t.start()

SocketServer Python

So I found the following sample code which allows for a basic python HTTP server to be established at a given url and port. I am quite inexperienced with web servers and am trying to create handlers for certain GET requests to this server. However, I cannot figure out how to actually create handlers for a GET request made by another computer when accessing this URL remotely. Any suggestions?
import SocketServer
class MyTCPHandler(SocketServer.BaseRequestHandler):
"""
The RequestHandler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def handle(self):
# self.request is the TCP socket connected to the client
self.data = self.request.recv(1024).strip()
print "{} wrote:".format(self.client_address[0])
print self.data
# just send back the same data, but upper-cased
self.request.sendall(self.data.upper())
if __name__ == "__main__":
HOST, PORT = "url" , PORT
# Create the server, binding to localhost on port 9999
server = SocketServer.TCPServer((HOST, PORT), MyTCPHandler)
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
server.serve_forever()

Here is very simple example of how it could work. You would start this and call it with this, for example:
curl -i 'http://localhost:5001/foo/bar?foo=bar' -X POST -d '{"Foo":"Bar"}'
HTTP/1.1 200 OK
Some response%
It is missing tons of things, but this should at least give you some sort of idea.
import SocketServer
class MyTCPHandler(SocketServer.BaseRequestHandler):
def handle(self):
self.data = self.request.recv(1024).strip()
print self.data
self.parse_request(self.data)
func, args = self.path.split("/", 1)
args = args.split("/")
resp = getattr(self, func)(*args)
self.request.sendall("HTTP/1.1 200 OK\n")
self.request.sendall("\n")
self.request.sendall(resp)
def parse_request(self, req):
headers = {}
lines = req.splitlines()
inbody = False
body = ''
for line in lines[1:]:
if line.strip() == "":
inbody = True
if inbody:
body += line
else:
k, v = line.split(":", 1)
headers[k.strip()] = v.strip()
method, path, _ = lines[0].split()
self.path = path.lstrip("/")
self.method = method
self.headers = headers
self.body = body
self.path, self.query_string = self.path.split("?")
def foo(self, *args):
print self.path
print self.query_string
print self.body
print self.headers
print self.method
return "Some response"
if __name__ == "__main__":
server = SocketServer.TCPServer(("localhost", 5001), MyTCPHandler)
server.serve_forever()