Help to build python ForkingTCPServer. This server returns an error when client connects to the server. When I use ThreadingTCPServer instead of ForkingTCPServer it works fine. socketFileIO module contains functions for pickle to/from socket:
http://code.activestate.com/recipes/577667-pickle-tofrom-socket/
server.py:
import SocketServer, time
from socketFileIO import write, read
class MyClientHandler(SocketServer.BaseRequestHandler):
def handle(self):
print self.client_address
data=read(self.request)
print 'Client:', data
time.sleep(10) # for testing
write(self.request,data)
self.request.close()
HOST, PORT = '', 50007
server = SocketServer.ForkingTCPServer((HOST, PORT), MyClientHandler)
server.serve_forever()
client.py:
import socket
from socketFileIO import write, read
HOST = '127.0.0.1'
PORT = 50007
data=["A","B","C","End"]
for x in data:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
write(s,x)
y=read(s)
print 'Server:', y
s.close()
Error:
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 1469)
Traceback (most recent call last):
File "c:\Python26\lib\SocketServer.py", line 283, in _handle_request_noblock
self.process_request(request, client_address)
File "c:\Python26\lib\SocketServer.py", line 525, in process_request
pid = os.fork()
AttributeError: 'module' object has no attribute 'fork'
----------------------------------------
I have a several other questions:
How to shutdown the server by sending data "End" from client ?
How to restrict the number of connections (for example no more then 5) ?
Your fork question was already answered in comments.
For "how to shut down the server": in a similar application (but much older Python version and using xmlrpc), I used a loop invoking one request at a time until a "stop" variable was set. This appears to be supported directly in Python 2.7 now: in your threaded server, call shutdown() on the server instance, from one of the request-handler threads. (Not sure if this is available in 2.6, I de-installed my old 2.6 version a couple of months ago.) I.e.:
class MyClientHandler(SocketServer.BaseRequestHandler):
def handle(self):
...
# if we're supposed to shut down the server, do that
if somecond:
self.server.shutdown()
For limiting the number of connections: there's nothing built-in that I can see, but if you redefined the process_request handler you could count the number of active threads and wait for one to finish if there are "too many" outstanding (or work it however else you like). See the ThreadingMixIn class in SocketServer.py and compare it to the max_children code in ForkingMixIn.
Related
I'm trying to start a UDP server in python 3.
I copied the code from this example.
This is my exact code.
import socketserver
class MyUDPHandler(socketserver.BaseRequestHandler):
"""
This class works similar to the TCP handler class, except that
self.request consists of a pair of data and client socket, and since
there is no connection the client address must be given explicitly
when sending data back via sendto().
"""
def handle(self):
data = self.request[0].strip()
socket = self.request[1]
print("{} wrote:".format(self.client_address[0]))
print(data)
if __name__ == "__main__":
HOST, PORT = "localhost", 19446
with socketserver.UDPServer((HOST, PORT), MyUDPHandler) as server:
server.serve_forever()
I have only removed the reply from the handle method and changed the port number.
when i try to run it i get this
$ sudo python3 test.py
File "test.py", line 19, in <module>
with socketserver.UDPServer((HOST, PORT), MyUDPHandler) as server:
AttributeError: __exit__
I am attempting to run this in Python 3.4.2 installed on a Raspberry Pi 3, it was working this morning.
I googled AttributeError: __exit__ and found that with uses built in methods such as __exit__ to gracefully close after it has finished running instructions indented after it.
The exact same code runs fine on my windows machine (Python 3.6.2) and the code used to run on my raspberry pi and the only thing i have done with it all day was install x11vnc server and plugged in a lot of USB devices. (A capture card and arduinos, no usb drives from untrusted sources).
So my question is, what can cause an Attribute Error: __exit__ in the socketserver library.
context manager was added to socketserver in 3.6: https://bugs.python.org/issue26404, with commit.
below 3.6, in 3.4.2, you have to call server_close() manually:
try:
server = socketserver.UDPServer((HOST, PORT), MyUDPHandler)
server.serve_forever()
finally:
server.server_close()
I'm working on a camera endpoint for an IoT application and I want to start and stop a flask app on command to show a webpage with streaming camera data. I have all the individual pieces working but it's failing when I put them together.
If I start a flask process after starting another thread it complains that address is already in use. Here's a stripped down example:
#!/usr/bin/env python3
import os
import threading
from flask import Flask
import socket
import time
from multiprocessing import Process
app = Flask(__name__)
def main():
start_thread(udp_listener)
start_process(runflask)
while True:
time.sleep(1)
def start_thread(function, arguments=()):
th = threading.Thread(target=function, args=arguments)
th.daemon = True
th.start()
def start_process(function):
server = Process(target=function)
server.start()
while True:
time.sleep(60)
server.terminate()
server.join()
def udp_listener():
while True:
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s:
s.bind(('', 9000))
print('udp listening on port 9000')
while True:
data, server = s.recvfrom(4069)
print('received:')
def runflask():
app.run(host='0.0.0.0', port=8001, debug=True)
if __name__ == "__main__":
main()
Comment out start_thread() and just run start_process() and it's ok.
Comment out start_process() and just run start_thread() and it's ok.
Run both and get an address already in use error even though they're listening on different ports.
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "./webcam_test.py", line 35, in udp_listener
s.bind(('', 9000))
OSError: [Errno 98] Address already in use
I want to run the flask app in a way that will let me start and stop it on command. Should I be doing it a different way?
I was able to resolve this by using a ServerThread as specified here: How to stop flask application without using ctrl-c
It's important to note that calling this ServerThread automatically blocks so if you want an interruptible thread, you need to instantiate this as the child of another thread. My design already spawned threads for each camera mode so now I have a thread calling a thread calling a thread :).
I still don't know why flask, when run as a process, would complain about unrelated listen sockets but now I have a workaround.
Don't how relevant this still is, but:
From a quick glance I don't see any place where the socket you are binding (s.bind(('', 9000))) is being closed. Since it's on a loop it tries to bind a socket that's in use again, hence the error? It should be stopped before binded again:
s.shutdown(1)
s.close()
If this is the case, the first solution I suggest is to close the socket properly, and not have it started in the loop (why do you bind the socket in loop in the first place?)
If this alone does not help, allow the socket to reuse the address:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
Python: Binding Socket: "Address already in use"
I am new to python and I am using socketserver to try to create a server that broadcasts all the received message from one client to all the connected clients, but facing a problem. I get the following error in the end:
Exception happened during processing of request from ('127.0.0.1', 14872)
Traceback (most recent call last):
File "C:\Users\umair\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 628, in process_request_thread
self.finish_request(request, client_address)
File "C:\Users\umair\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 357, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Users\umair\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 684, in init
self.handle()
File "C:\Users\umair\Desktop\socketserverthread_server.py", line 52, in handle
clients.send(data)
OSError: [WinError 10038] An operation was attempted on something that is not a socket
My server code is as follows:
import socketserver
import threading
host = '127.0.0.1'
port = 6666
all_clients = []
class ThreadingHandler (socketserver.BaseRequestHandler):
def handle(self):
if (self.request) not in all_clients:
all_clients.append(self.request)
data = self.request.recv(1024)
print('%s writes: ' % str(self.client_address), end = " ")
print(data.decode())
for clients in all_clients:
clients.send(data)
class ThreadingServer (socketserver.ThreadingMixIn, socketserver.TCPServer):
pass
myserv = ThreadingServer((host, port), ThreadingHandler)
t = threading.Thread(target = myserv.serve_forever)
t.setDaemon(True)
t.start()
print('The server is online')
The client code is:
import socket
host = '127.0.0.1'
port = 6666
while True:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((host, port))
data = input('You: ')
sock.send(data.encode())
received = sock.recv(1024)
print('Received: ', received.decode())
I am running two of these client codes to test it. The first message from a client doesn't get broadcast, and is just received by the same client. On sending the second message, I receive the above mentioned error. Is it the problem due to the fact that I am creating a new socket in every cycle? I have to create the socket in the loop because if I don't, then I cannot send and receive continuously. I think the socket object is destroyed after a request and a response. I have no idea what's going on. So any help is appreciated. Thank you.
On sending the second message, I receive the above mentioned error. Is
it the problem due to the fact that I am creating a new socket in
every cycle? … I think the socket
object is destroyed after a request and a response. I have no idea
what's going on.
You do have an idea what's going on; indeed the client's socket object sock from a previous loop cycle is destroyed at the time when the new socket is assigned to the variable, which causes the old socket to be closed. But independently thereof the server already closes its request socket due to the use of socketserver, just after one request has been handled, i. e. after your handle(self) returns. The above mentioned error arises from the omission to take this closed socket object in all_clients into account, which could be done e. g. so:
for clients in all_clients[:]:
if clients._closed: all_clients.remove(clients)
else: clients.send(data)
The first message from a client doesn't get broadcast, and is just
received by the same client.
The message does get broadcast (sent to each client), it's just that the other client doesn't bother about receiving it, because it is waiting in input('You: ').
I have to create the socket in the loop because if I don't, then I
cannot send and receive continuously.
That's only true in a sense because you use socketserver, which closes the connection after each request - not quite helpful in case of a chat server.
See the question "Handle multiple requests with select" for an example server without the hassle brought by socketserver et al.
I want to send data from a client to the server in a TLS TCP socket from multiple client subprocesses so I share the same ssl socket with all subprocesses. Communication works with one subprocess, but if I use more than one subprocesses, the TLS server crashes with an ssl.SSLError (SSL3_GET_RECORD:decryption failed or bad record mac).
More specific: It does not depend which process first calls the SSLSocket.write() method, but this process is the only one from this time on which can call it. If another process calls write(), the server will result in the exception described above.
I used this basic code:
tlsserver.py
import socket, ssl
def deal_with_client(connstream):
data = connstream.read()
while data:
print data
data = connstream.read()
connstream.close()
bindsocket = socket.socket()
bindsocket.bind(('127.0.0.1', 9998))
bindsocket.listen(5)
while True:
newsocket, fromaddr = bindsocket.accept()
connstream = ssl.wrap_socket(newsocket,
server_side=True,
certfile="srv.crt",
keyfile="srv.key",
ssl_version=ssl.PROTOCOL_TLSv1)
deal_with_client(connstream)
tlsclient.py
import socket, ssl
import multiprocessing
class SubProc:
def __init__(self, sock):
self.sock = sock
def do(self):
self.sock.write("Test")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ssl_sock = ssl.wrap_socket(s)
ssl_sock.connect(('127.0.0.1', 9998))
print "Connected to", repr(ssl_sock.getpeername())
for x in (1,2):
subproc = SubProc(ssl_sock)
proc = multiprocessing.Process(target=subproc.do)
And this is the backtrace:
Traceback (most recent call last):
File "tlsserver.py", line 21, in <module>
deal_with_client(connstream)
File "tlsserver.py", line 7, in deal_with_client
data = connstream.read()
File "/usr/lib64/python2.6/ssl.py", line 136, in read
return self._sslobj.read(len)
ssl.SSLError: [Errno 1] _ssl.c:1325: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
The problem is that you're re-using the same connection for both processes. The way SSL encrypts data makes this fail -- the two processes would have to communicate with each other about the state of the shared SSL connection. Even if you do make it work, or if you didn't use SSL, the data would arrive at the server all jumbled up; you would have no real way of distinguishing which bytes came from which process.
What you need to do is give each process its own SSL connection, by making the connection in subproc.do. Alternatively, don't have the subprocesses communicate with the server at all, but rather communicate with the main process, and have the main process relay it over the SSL connection.
I'm teaching myself Python networking, and I recalled that back when I was teaching myself threading, I came across this page, so I copied the scripts, updated them for Python 3.1.1 and ran them. They worked perfectly.
Then I made a few modifications. My goal is to do something simple:
The client pickles an integer and sends it to the server.
The server receives the pickled integer, unpickles it, doubles it, then pickles it and sends it back to the client.
The client receives the pickled (and doubled) integer, unpickles it, and outputs it.
Here's the server:
import pickle
import socket
import threading
class ClientThread(threading.Thread):
def __init__(self, channel, details):
self.channel = channel
self.details = details
threading.Thread.__init__ ( self )
def run(self):
print('Received connection:', self.details[0])
request = self.channel.recv(1024)
response = pickle.dumps(pickle.loads(request) * 2)
self.channel.send(response)
self.channel.close()
print('Closed connection:', self.details [ 0 ])
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('', 2727))
server.listen(5)
while True:
channel, details = server.accept()
ClientThread(channel, details).start()
And here is the client:
import pickle
import socket
import threading
class ConnectionThread(threading.Thread):
def run(self):
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('localhost', 2727))
for x in range(10):
client.send(pickle.dumps(x))
print('Sent:',str(x))
print('Received:',repr(pickle.loads(client.recv(1024))))
client.close()
for x in range(5):
ConnectionThread().start()
The server runs fine, and when I run the client it successfully connects and starts sending integers and receiving them back doubled as expected. However, very quickly it exceptions out:
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Python30\lib\threading.py", line 507, in _bootstrap_inner
self.run()
File "C:\Users\Imagist\Desktop\server\client.py", line 13, in run
print('Received:',repr(pickle.loads(client.recv(1024))))
socket.error: [Errno 10053] An established connection was aborted by the softwar
e in your host machine
The server continues to run and receives connections just fine; only the client crashes. What's causing this?
EDIT: I got the client working with the following code:
import pickle
import socket
import threading
class ConnectionThread(threading.Thread):
def run(self):
for x in range(10):
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('localhost', 2727))
client.send(pickle.dumps(x))
print('Sent:',str(x))
print('Received:',repr(pickle.loads(client.recv(1024))))
client.close()
for x in range(5):
ConnectionThread().start()
However, I still don't understand what's going on. Isn't this just opening and closing the socket a bunch of times? Shouldn't there be time limitations to that (you shouldn't be able to open a socket so soon after closing it)?
Your client is now correct - you want to open the socket send the data, receive the reply and then close the socket.
The error original error was caused by the server closing the socket after it sent the first response which caused the client to receive a connection closed message when it tried to send the second message on the same connection.
However, I still don't understand
what's going on. Isn't this just
opening and closing the socket a bunch
of times?
Yes. This is acceptable, if not the highest performance way of doing things.
Shouldn't there be time
limitations to that (you shouldn't be
able to open a socket so soon after
closing it)?
You can open a client socket as quickly as you like as every time you open a socket you will get a new local port number, meaning that the connections won't interfere. In the server code above, it will start a new thread for each incoming connection.
There are 4 parts to every IP connection (source_address, source_port, destination_address, destination_port) and this quad (as it is known) must change for ever connection. Everything except source_port is fixed for a client socket so that is what the OS changes for you.
Opening server sockets is more troublesome - if you want to open a new server socket quickly, your
server.bind(('', 2727))
Above then you need to read up on SO_REUSEADDR.