I want to send data from a client to the server in a TLS TCP socket from multiple client subprocesses so I share the same ssl socket with all subprocesses. Communication works with one subprocess, but if I use more than one subprocesses, the TLS server crashes with an ssl.SSLError (SSL3_GET_RECORD:decryption failed or bad record mac).
More specific: It does not depend which process first calls the SSLSocket.write() method, but this process is the only one from this time on which can call it. If another process calls write(), the server will result in the exception described above.
I used this basic code:
tlsserver.py
import socket, ssl
def deal_with_client(connstream):
data = connstream.read()
while data:
print data
data = connstream.read()
connstream.close()
bindsocket = socket.socket()
bindsocket.bind(('127.0.0.1', 9998))
bindsocket.listen(5)
while True:
newsocket, fromaddr = bindsocket.accept()
connstream = ssl.wrap_socket(newsocket,
server_side=True,
certfile="srv.crt",
keyfile="srv.key",
ssl_version=ssl.PROTOCOL_TLSv1)
deal_with_client(connstream)
tlsclient.py
import socket, ssl
import multiprocessing
class SubProc:
def __init__(self, sock):
self.sock = sock
def do(self):
self.sock.write("Test")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ssl_sock = ssl.wrap_socket(s)
ssl_sock.connect(('127.0.0.1', 9998))
print "Connected to", repr(ssl_sock.getpeername())
for x in (1,2):
subproc = SubProc(ssl_sock)
proc = multiprocessing.Process(target=subproc.do)
And this is the backtrace:
Traceback (most recent call last):
File "tlsserver.py", line 21, in <module>
deal_with_client(connstream)
File "tlsserver.py", line 7, in deal_with_client
data = connstream.read()
File "/usr/lib64/python2.6/ssl.py", line 136, in read
return self._sslobj.read(len)
ssl.SSLError: [Errno 1] _ssl.c:1325: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
The problem is that you're re-using the same connection for both processes. The way SSL encrypts data makes this fail -- the two processes would have to communicate with each other about the state of the shared SSL connection. Even if you do make it work, or if you didn't use SSL, the data would arrive at the server all jumbled up; you would have no real way of distinguishing which bytes came from which process.
What you need to do is give each process its own SSL connection, by making the connection in subproc.do. Alternatively, don't have the subprocesses communicate with the server at all, but rather communicate with the main process, and have the main process relay it over the SSL connection.
Related
I have a client-server model where the client will constantly checking a log file and as soon as a new line comes in the log file it sends that line to the server.
Somehow I managed to work this thing using the following code.
server.py
import SocketServer
class MyTCPSocketHandler(SocketServer.BaseRequestHandler):
def handle(self):
# self.request is the TCP socket connected to the client
data = self.request.recv(1024).strip()
print data
# process the data..
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
server = SocketServer.TCPServer((HOST, PORT), MyTCPSocketHandler)
server.serve_forever()
client.py
import time
import socket
def follow(thefile):
thefile.seek(0, 2)
while True:
line = thefile.readline()
if not line:
time.sleep(0.1)
continue
yield line
def connect_socket():
HOST, PORT = "localhost", 9999
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, PORT))
return sock
if __name__ == '__main__':
logfile = open("my_log.log")
loglines = follow(logfile)
for line in loglines:
sock = connect_socket()
# send data
sock.sendall(bytes(line))
the problem is every time I need to call the connect_socket() method to send a new line.
I'm quite new to this topic so somebody please let me know is there any workaround for this to work in a such a way that once the connection is established between client and server I need to send data continuously to the server without making a new connection again and again.
If I'm connecting only one time and using the same socket object to send data it was throwing
socket.error: [Errno 32] Broken pipe
Some StackOverflow links which I have followed are given below,
1, 2, 3
One thing I found is
Broken Pipe occurs when one end of the connection tries sending data while the other end has already closed the connection.
How can I keep the connection open on both ends?
For this use case should I go for an asynchronous method and if so which framework will be the best match tornado or twisted?
After a line is transmitted, you close the connection on the client, but don't close it on the server.
From the docs:
RequestHandler.finish()
Called after the handle() method to perform any clean-up actions required. The default implementation does nothing.
So you should implement finish as well and close the socket (self.request) there.
Another option is not to close the connection on the client:
sock = connect_socket()
for line in loglines:
# send data
sock.sendall(bytes(line))
sock.sendall(b'bye') # EOF
sock.close()
However in this case you should modify the server code and make sure it can serve multiple clients and it understands when to close the socket. With all these difficulties it is nevertheless the preferable way of doing things. Ideally, a client has a single TCP connection per session for they're costly to establish.
I am new to python and I am using socketserver to try to create a server that broadcasts all the received message from one client to all the connected clients, but facing a problem. I get the following error in the end:
Exception happened during processing of request from ('127.0.0.1', 14872)
Traceback (most recent call last):
File "C:\Users\umair\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 628, in process_request_thread
self.finish_request(request, client_address)
File "C:\Users\umair\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 357, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Users\umair\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 684, in init
self.handle()
File "C:\Users\umair\Desktop\socketserverthread_server.py", line 52, in handle
clients.send(data)
OSError: [WinError 10038] An operation was attempted on something that is not a socket
My server code is as follows:
import socketserver
import threading
host = '127.0.0.1'
port = 6666
all_clients = []
class ThreadingHandler (socketserver.BaseRequestHandler):
def handle(self):
if (self.request) not in all_clients:
all_clients.append(self.request)
data = self.request.recv(1024)
print('%s writes: ' % str(self.client_address), end = " ")
print(data.decode())
for clients in all_clients:
clients.send(data)
class ThreadingServer (socketserver.ThreadingMixIn, socketserver.TCPServer):
pass
myserv = ThreadingServer((host, port), ThreadingHandler)
t = threading.Thread(target = myserv.serve_forever)
t.setDaemon(True)
t.start()
print('The server is online')
The client code is:
import socket
host = '127.0.0.1'
port = 6666
while True:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((host, port))
data = input('You: ')
sock.send(data.encode())
received = sock.recv(1024)
print('Received: ', received.decode())
I am running two of these client codes to test it. The first message from a client doesn't get broadcast, and is just received by the same client. On sending the second message, I receive the above mentioned error. Is it the problem due to the fact that I am creating a new socket in every cycle? I have to create the socket in the loop because if I don't, then I cannot send and receive continuously. I think the socket object is destroyed after a request and a response. I have no idea what's going on. So any help is appreciated. Thank you.
On sending the second message, I receive the above mentioned error. Is
it the problem due to the fact that I am creating a new socket in
every cycle? … I think the socket
object is destroyed after a request and a response. I have no idea
what's going on.
You do have an idea what's going on; indeed the client's socket object sock from a previous loop cycle is destroyed at the time when the new socket is assigned to the variable, which causes the old socket to be closed. But independently thereof the server already closes its request socket due to the use of socketserver, just after one request has been handled, i. e. after your handle(self) returns. The above mentioned error arises from the omission to take this closed socket object in all_clients into account, which could be done e. g. so:
for clients in all_clients[:]:
if clients._closed: all_clients.remove(clients)
else: clients.send(data)
The first message from a client doesn't get broadcast, and is just
received by the same client.
The message does get broadcast (sent to each client), it's just that the other client doesn't bother about receiving it, because it is waiting in input('You: ').
I have to create the socket in the loop because if I don't, then I
cannot send and receive continuously.
That's only true in a sense because you use socketserver, which closes the connection after each request - not quite helpful in case of a chat server.
See the question "Handle multiple requests with select" for an example server without the hassle brought by socketserver et al.
I'd like to implement the example from python official documentation on socket module using AF_UNIX socket instead of AF_INET. So here's how server code looks now:
import socket
import os
filename = '/tmp/mysock'
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, 0)
if os.path.exists(filename):
os.remove(filename)
sock.bind(filename)
while 1:
data = sock.recv(1024)
print 'Received', repr(data)
if not data:
break
sock.sendall(data)
sock.close()
When I run it, I get this:
Traceback (most recent call last):
File "/home/somecode/server.py", line 10, in <module>
data = sock.recv(2048)
socket.error: [Errno 22] Invalid argument
SOCK_STREAM is a stateful socket type, so any connection must first be acknowledged by a listening socket and then new dynamic socket (connection) is used for sending and receiving instead. You can't just receive from listening socket.
Basically you forgot to call sock.listen() on socket after binding and connection, address = sock.accept(), which blocks until a connection is established and then returns tuple with new socket (connection) and client address. Then simply use connection.recv(). This is universal for UNIX sockets and TCP/IP sockets.
For example have a look here.
Help to build python ForkingTCPServer. This server returns an error when client connects to the server. When I use ThreadingTCPServer instead of ForkingTCPServer it works fine. socketFileIO module contains functions for pickle to/from socket:
http://code.activestate.com/recipes/577667-pickle-tofrom-socket/
server.py:
import SocketServer, time
from socketFileIO import write, read
class MyClientHandler(SocketServer.BaseRequestHandler):
def handle(self):
print self.client_address
data=read(self.request)
print 'Client:', data
time.sleep(10) # for testing
write(self.request,data)
self.request.close()
HOST, PORT = '', 50007
server = SocketServer.ForkingTCPServer((HOST, PORT), MyClientHandler)
server.serve_forever()
client.py:
import socket
from socketFileIO import write, read
HOST = '127.0.0.1'
PORT = 50007
data=["A","B","C","End"]
for x in data:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
write(s,x)
y=read(s)
print 'Server:', y
s.close()
Error:
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 1469)
Traceback (most recent call last):
File "c:\Python26\lib\SocketServer.py", line 283, in _handle_request_noblock
self.process_request(request, client_address)
File "c:\Python26\lib\SocketServer.py", line 525, in process_request
pid = os.fork()
AttributeError: 'module' object has no attribute 'fork'
----------------------------------------
I have a several other questions:
How to shutdown the server by sending data "End" from client ?
How to restrict the number of connections (for example no more then 5) ?
Your fork question was already answered in comments.
For "how to shut down the server": in a similar application (but much older Python version and using xmlrpc), I used a loop invoking one request at a time until a "stop" variable was set. This appears to be supported directly in Python 2.7 now: in your threaded server, call shutdown() on the server instance, from one of the request-handler threads. (Not sure if this is available in 2.6, I de-installed my old 2.6 version a couple of months ago.) I.e.:
class MyClientHandler(SocketServer.BaseRequestHandler):
def handle(self):
...
# if we're supposed to shut down the server, do that
if somecond:
self.server.shutdown()
For limiting the number of connections: there's nothing built-in that I can see, but if you redefined the process_request handler you could count the number of active threads and wait for one to finish if there are "too many" outstanding (or work it however else you like). See the ThreadingMixIn class in SocketServer.py and compare it to the max_children code in ForkingMixIn.
I'm teaching myself Python networking, and I recalled that back when I was teaching myself threading, I came across this page, so I copied the scripts, updated them for Python 3.1.1 and ran them. They worked perfectly.
Then I made a few modifications. My goal is to do something simple:
The client pickles an integer and sends it to the server.
The server receives the pickled integer, unpickles it, doubles it, then pickles it and sends it back to the client.
The client receives the pickled (and doubled) integer, unpickles it, and outputs it.
Here's the server:
import pickle
import socket
import threading
class ClientThread(threading.Thread):
def __init__(self, channel, details):
self.channel = channel
self.details = details
threading.Thread.__init__ ( self )
def run(self):
print('Received connection:', self.details[0])
request = self.channel.recv(1024)
response = pickle.dumps(pickle.loads(request) * 2)
self.channel.send(response)
self.channel.close()
print('Closed connection:', self.details [ 0 ])
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('', 2727))
server.listen(5)
while True:
channel, details = server.accept()
ClientThread(channel, details).start()
And here is the client:
import pickle
import socket
import threading
class ConnectionThread(threading.Thread):
def run(self):
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('localhost', 2727))
for x in range(10):
client.send(pickle.dumps(x))
print('Sent:',str(x))
print('Received:',repr(pickle.loads(client.recv(1024))))
client.close()
for x in range(5):
ConnectionThread().start()
The server runs fine, and when I run the client it successfully connects and starts sending integers and receiving them back doubled as expected. However, very quickly it exceptions out:
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Python30\lib\threading.py", line 507, in _bootstrap_inner
self.run()
File "C:\Users\Imagist\Desktop\server\client.py", line 13, in run
print('Received:',repr(pickle.loads(client.recv(1024))))
socket.error: [Errno 10053] An established connection was aborted by the softwar
e in your host machine
The server continues to run and receives connections just fine; only the client crashes. What's causing this?
EDIT: I got the client working with the following code:
import pickle
import socket
import threading
class ConnectionThread(threading.Thread):
def run(self):
for x in range(10):
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('localhost', 2727))
client.send(pickle.dumps(x))
print('Sent:',str(x))
print('Received:',repr(pickle.loads(client.recv(1024))))
client.close()
for x in range(5):
ConnectionThread().start()
However, I still don't understand what's going on. Isn't this just opening and closing the socket a bunch of times? Shouldn't there be time limitations to that (you shouldn't be able to open a socket so soon after closing it)?
Your client is now correct - you want to open the socket send the data, receive the reply and then close the socket.
The error original error was caused by the server closing the socket after it sent the first response which caused the client to receive a connection closed message when it tried to send the second message on the same connection.
However, I still don't understand
what's going on. Isn't this just
opening and closing the socket a bunch
of times?
Yes. This is acceptable, if not the highest performance way of doing things.
Shouldn't there be time
limitations to that (you shouldn't be
able to open a socket so soon after
closing it)?
You can open a client socket as quickly as you like as every time you open a socket you will get a new local port number, meaning that the connections won't interfere. In the server code above, it will start a new thread for each incoming connection.
There are 4 parts to every IP connection (source_address, source_port, destination_address, destination_port) and this quad (as it is known) must change for ever connection. Everything except source_port is fixed for a client socket so that is what the OS changes for you.
Opening server sockets is more troublesome - if you want to open a new server socket quickly, your
server.bind(('', 2727))
Above then you need to read up on SO_REUSEADDR.