I'm trying to implement a simple XML-RPC server on Python 3, and I want it to run over HTTPS using the standard ssl library (included in Python 2.6 and Python 3.x).
I've seen some code that does it with OpenSSL or M2Crypto modules, but I want to avoid any dependency.
I implemented the next code which should wrap the SSL protocol over the socket:
"""Monkey patching standard xmlrpc.server.SimpleXMLRPCServer
to run over TLS (SSL)
Changes inspired on http://www.cs.technion.ac.il/~danken/SecureXMLRPCServer.py
"""
import socket
import socketserver
import ssl
from xmlrpc.server import SimpleXMLRPCServer, SimpleXMLRPCDispatcher, SimpleXMLRPCRequestHandler
try:
import fcntl
except ImportError:
fcntl = None
class SimpleXMLRPCServerTLS(SimpleXMLRPCServer):
def __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler,
logRequests=True, allow_none=False, encoding=None, bind_and_activate=True):
"""Overriding __init__ method of the SimpleXMLRPCServer
The method is an exact copy, except the TCPServer __init__
call, which is rewritten using TLS
"""
self.logRequests = logRequests
SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding)
"""This is the modified part. Original code was:
socketserver.TCPServer.__init__(self, addr, requestHandler, bind_and_activate)
which executed:
def __init__(self, server_address, RequestHandlerClass, bind_and_activate=True):
BaseServer.__init__(self, server_address, RequestHandlerClass)
self.socket = socket.socket(self.address_family,
self.socket_type)
if bind_and_activate:
self.server_bind()
self.server_activate()
"""
socketserver.BaseServer.__init__(self, addr, requestHandler)
self.socket = ssl.wrap_socket(
socket.socket(self.address_family, self.socket_type),
server_side=True,
cert_reqs=ssl.CERT_NONE,
ssl_version=ssl.PROTOCOL_TLSv1,
)
if bind_and_activate:
self.server_bind()
self.server_activate()
"""End of modified part"""
# [Bug #1222790] If possible, set close-on-exec flag; if a
# method spawns a subprocess, the subprocess shouldn't have
# the listening socket open.
if fcntl is not None and hasattr(fcntl, 'FD_CLOEXEC'):
flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD)
flags |= fcntl.FD_CLOEXEC
fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags)
But for some reason I couldn't determine, it's raising this error:
[Errno 8] _ssl.c:502: EOF occurred in violation of protocol
before calling the remote method.
Does anyone knows what could be happening, or has any idea on how to gather more information to get a clue on what could be the problem?
Thank you so much in advance!
UPDATE:
Fixed. There were two errors on the code. First, it's necessary to specify the certificate file (which can include the key too).
Second is that xmlrpc.client.ServerProxy (included in Python) is using SSLv2, so TLSv1 does not work.
Working code is:
"""Monkey patching standard xmlrpc.server.SimpleXMLRPCServer
to run over TLS (SSL)
Changes inspired on http://www.cs.technion.ac.il/~danken/SecureXMLRPCServer.py
"""
import socket
import socketserver
import ssl
from xmlrpc.server import SimpleXMLRPCServer, SimpleXMLRPCDispatcher, SimpleXMLRPCRequestHandler
try:
import fcntl
except ImportError:
fcntl = None
class SimpleXMLRPCServerTLS(SimpleXMLRPCServer):
def __init__(self, addr, requestHandler=SimpleXMLRPCRequestHandler,
logRequests=True, allow_none=False, encoding=None, bind_and_activate=True):
"""Overriding __init__ method of the SimpleXMLRPCServer
The method is an exact copy, except the TCPServer __init__
call, which is rewritten using TLS
"""
self.logRequests = logRequests
SimpleXMLRPCDispatcher.__init__(self, allow_none, encoding)
"""This is the modified part. Original code was:
socketserver.TCPServer.__init__(self, addr, requestHandler, bind_and_activate)
which executed:
def __init__(self, server_address, RequestHandlerClass, bind_and_activate=True):
BaseServer.__init__(self, server_address, RequestHandlerClass)
self.socket = socket.socket(self.address_family,
self.socket_type)
if bind_and_activate:
self.server_bind()
self.server_activate()
"""
socketserver.BaseServer.__init__(self, addr, requestHandler)
self.socket = ssl.wrap_socket(
socket.socket(self.address_family, self.socket_type),
server_side=True,
certfile='cert.pem',
cert_reqs=ssl.CERT_NONE,
ssl_version=ssl.PROTOCOL_SSLv23,
)
if bind_and_activate:
self.server_bind()
self.server_activate()
"""End of modified part"""
# [Bug #1222790] If possible, set close-on-exec flag; if a
# method spawns a subprocess, the subprocess shouldn't have
# the listening socket open.
if fcntl is not None and hasattr(fcntl, 'FD_CLOEXEC'):
flags = fcntl.fcntl(self.fileno(), fcntl.F_GETFD)
flags |= fcntl.FD_CLOEXEC
fcntl.fcntl(self.fileno(), fcntl.F_SETFD, flags)
Why don't you write just like this:
server = SimpleXMLRPCServer(...)
...
server.socket = ssl.wrap_socket(srv.socket, ...)
server.serve_forever()
Move your Python server behind a reverse proxy like Apache, NGinx or whatever and let the reverse proxy to the SSL job...this works much better and smoother than on the Python level.
Related
I have been trying to get a simple Python SSL example working for a day now with no luck. I want to create an SSL server and SSL client. The server should authenticate the client. The Python docs are pretty light on examples for the SSL module, and in general I can't find many working examples. The code I am working with is as follows;
Client:
import socket
import ssl
class SSLClient:
def __init__(self, server_host, server_port, client_cert, client_key):
self.server_host = server_host
self.server_port = server_port
self._context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
self._context.load_cert_chain(client_cert, client_key)
self._sock = None
self._ssock = None
def __del__(self):
self.close()
def connect(self):
self._sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self._ssock = self._context.wrap_socket(
self._sock, server_hostname=self.server_host
)
self._ssock.connect((self.server_host, self.server_port))
def send(self, msg):
self._ssock.send(msg.encode())
def close(self):
self._ssock.close()
Server:
import socket
import ssl
from threading import Thread
class SSLServer:
def __init__(self, host, port, cafile, chunk_size=1024):
self.host = host
self.port = port
self.chunk_size = chunk_size
self._context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
self._context.load_verify_locations(cafile)
self._ssock = None
def __del__(self):
self.close()
def connect(self):
with socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) as sock:
sock.bind((self.host, self.port))
sock.listen(5)
with self._context.wrap_socket(sock, server_side=True) as self._ssock:
conn, _ = self._ssock.accept()
while True:
data = conn.recv(self.chunk_size).decode()
print(data)
if data is None:
break
def close(self):
self._ssock.close()
class SSLServerThread(Thread):
def __init__(self, server):
super().__init__()
self._server = server
self.daemon = True
def run(self):
self._server.connect()
def stop(self):
self._server.close()
Test script:
import client, server
from os import path
from time import sleep
server_host = "localhost"
server_port = 11234
client_cert = path.join(path.dirname(__file__), "client.crt")
client_key = path.join(path.dirname(__file__), "client.key")
s = server.SSLServer(server_host, server_port, client_cert)
s_thread = server.SSLServerThread(s)
s_thread.start()
sleep(2)
c = client.SSLClient(server_host, server_port, client_cert, client_key)
c.connect()
c.send("This is a test message!")
c.close()
s.close()
I generated my client certificate and key using the following command:
openssl req -newkey rsa:2048 \
-x509 \
-sha256 \
-days 3650 \
-nodes \
-out client.crt \
-keyout client.key \
-subj "/C=UK/ST=Scotland/L=Glasgow/O=Company A/OU=Testing/CN=MyName"
The test script seems to start the server and allow the client to connect, but I am getting a BrokenPipeError when I try to send the test message.
Annoyingly I have been getting various different error messages as I go, so it's likely a combination of things. This is a simple example I created to try and get something working. On my more complex example I get "NO_SHARED_CIPHERS" when the client attempts to connect to the server. Annoyingly I can't see why this simple example seems to get further than the more complex one (i.e. the connection seems to be established successfully) even though they are set up almost identically.
I have uploaded a repo at git#github.com:stevengillies87/python-ssl-client-auth-example.git if anyone would like to test it.
I realised the first bug came from copy pasting and example and not realising how it differed from my code in its setup. It used socket.socket() to create the socket whereas my example used socket.create_connection(), which also connects the socket. This was the reason I was getting a BrokenPipeError. Now both my simple example and the actual code I am writing both have a consistent NO_SHARED_CIPHER error. I added a line to the source code to connect the client after the socket has been wrapped.
So, as expected it was a combination of things.
Before I added the SSL layer to my code it worked with TCP sockets. I was using socket.create_connection() in the client to create and connect a socket in one call. When I added SSL I continued to do this but because I was attempting to connect to an SSL server via a TCP socket I was getting a NO_SHARED_CIPHER error.
The solution to this problem was to only create the TCP socket with sock = socket.socket(), wrap it with ssock = ssl_context.wrap_context(sock) and then call connect on the SSL layer, ssock.connect((host, port)).
However, I was still getting a handshaking error on connection. I found this link, https://www.electricmonk.nl/log/2018/06/02/ssl-tls-client-certificate-verification-with-python-v3-4-sslcontext/, which provided a detailed example of how to create mutually authenticating SSL client/server. Crucially, the author pointed out that hostname used for server authentication must match the "common name" entered when creating the server.crt and server.key files. Previously I had just been using the same host that I was connecting to, "localhost" in this case. They also noted that the ssl_context verify mode should be set to verify_mode = ssl.CERT_REQUIRED for client auth.
Once the example worked I set about removing the client auth of the server. This was done by changing the client SSL context from ssl.create_default_context(ssl.Purpose.CLIENT_AUTH) to ssl.SSLContext(). The client now does not require the server.crt file to connect successfully.
Frustratingly I still need to create server cert/key files and load them into the server using ssl_context.load_cert_chain(), even though I do not need the server to be authenticated. If I try to remove this step from the server I get a NO_SHARED_CIPHER error again. If anyone knows how I can avoid this then please let me know, or explain why it is necessary.
Working code below, and updated at the github link in the question.
Client:
import socket
import ssl
class SSLClient:
def __init__(
self, server_host, server_port, sni_hostname, client_cert, client_key,
):
self.server_host = server_host
self.server_port = server_port
self.sni_hostname = sni_hostname
self._context = ssl.SSLContext()
self._context.load_cert_chain(client_cert, client_key)
self._sock = None
self._ssock = None
def __del__(self):
self.close()
def connect(self):
self._sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self._ssock = self._context.wrap_socket(self._sock,)
self._ssock.connect((self.server_host, self.server_port))
def send(self, msg):
self._ssock.send(msg.encode())
def close(self):
self._ssock.close()
Server:
import socket
import ssl
from threading import Thread
class SSLServer:
def __init__(
self, host, port, server_cert, server_key, client_cert, chunk_size=1024
):
self.host = host
self.port = port
self.chunk_size = chunk_size
self._context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
self._context.verify_mode = ssl.CERT_REQUIRED
self._context.load_cert_chain(server_cert, server_key)
self._context.load_verify_locations(client_cert)
def connect(self):
with socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) as sock:
sock.bind((self.host, self.port))
sock.listen(5)
while True:
conn, _ = sock.accept()
with self._context.wrap_socket(conn, server_side=True) as sconn:
self._recv(sconn)
def _recv(self, sock):
while True:
data = sock.recv(self.chunk_size)
if data:
print(data.decode())
else:
break
class SSLServerThread(Thread):
def __init__(self, server):
super().__init__()
self._server = server
self.daemon = True
def run(self):
self._server.connect()
Test:
import client, server
from os import path
from time import sleep
server_host = "127.0.0.1"
server_port = 35689
server_sni_hostname = "www.company-b.com"
client_cert = path.join(path.dirname(__file__), "client.crt")
client_key = path.join(path.dirname(__file__), "client.key")
server_cert = path.join(path.dirname(__file__), "server.crt")
server_key = path.join(path.dirname(__file__), "server.key")
s = server.SSLServer(server_host, server_port, server_cert, server_key, client_cert)
s_thread = server.SSLServerThread(s)
s_thread.start()
sleep(2)
c = client.SSLClient(
server_host, server_port, server_sni_hostname, client_cert, client_key
)
c.connect()
c.send("This is a test message!")
c.close()
I'm trying to create a subclass of socket.socket class in Python2.7 with overridden send() and read() methods so that they dump the data transferred over the socket into the terminal. The code looks like this:
import socket
class SocketMonkey(socket.socket):
def send(self, buf):
print('BUF: {}'.format(buf))
return super(SocketMonkey, self).send(buf)
def recv(self, size=-1):
buf = super(SocketMonkey, self).recv(size)
print('BUF: {}'.format(buf))
return buf
socket.socket = SocketMonkey
And then this is how I instantiate and use this class:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('www.domain.com', 80))
sock.send('GET / HTTP/1.1\n\n')
I've monkey patched socket module, but while socket functionality works as before, data is not being dumped. Any idea where am I going wrong?
I had the same problem too, it's solved using the following code
import socket
import logging
class TestSocket:
"""
Defines a custom socket class that'll be used for testing.
"""
def __init__(self, af, sock_type):
logging.log(logging.DEBUG,
f"FAKE SOCKET CONSTRUCT [{af}] [{sock_type}]")
def sendto(self, data, address):
logging.log(logging.DEBUG,
f"SEND TO [{data}] [{address}]")
def socket_get(af, sock_type):
return TestSocket(af, sock_type)
def test_download_file(monkeypatch, submission):
with monkeypatch.context() as m:
# This replaces the socket constructor.
m.setattr(socket, "socket", socket_get)
# use socket now.
# ...
Given that I have implemented a UDP Client in Twisted with the DatagramProtocol, and using it to communicate to a UDP Server, which at one point goes offline (due to a restart - so it does not change it's IP address), stopProtocol in my protocol is called, however the transport itself is set to None by Twisted.
How can I solve a simple reconnect in Twisted or re-initiate the transport?
I cannot connect again with udp according to the docs.
Given that in UDP the sender should be able to send packets even after the server is dead, and given that the protocol has it's own connection handling in the packets, I could reconnect the logical part entirely over the Packet layer, if the transport would not disappear.
I suppose running listenUDP again with a new protocol while the core is running won't work.
from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor
class UDPClientProtocol(DatagramProtocol):
def __init__(self, host, port):
self.host = host
self.port = port
def startProtocol(self):
# Called when transport is connected
self.transport.connect(self.host, self.port)
self.transport.write('initiate protocol') # pseudo code.
def stopProtocol(self):
print "I have lost connection and self.transport is gone!"
# wait some time and try to reconnect somehow?
t = reactor.listenUDP(0, UDPClientProtocol('127.0.0.1', 12345))
reactor.run()
I used following technique that I saw at Twisted DNS source file. It survives server disconnection and even network failures.
from twisted.internet import reactor, protocol, task
import time
class EchoClientDatagramProtocol(protocol.DatagramProtocol):
def __init__(self, host, port, reactor):
self.host = host
self.port = port
self._reactor = reactor
def startProtocol(self):
self.transport.connect(self.host, self.port)
def stopProtocol(self):
#on disconnect
self._reactor.listenUDP(0, self)
def sendDatagram(self):
datagram = ntp_packet
try:
self.transport.write(datagram, (self.host, self.port))
print "{:0.6f}".format(time.time())
except:
pass
def datagramReceived(self, datagram, host):
pass
#print 'Datagram received: ', repr(datagram)
#self.sendDatagram()
def main():
protocol = EchoClientDatagramProtocol('127.0.0.1', 8000, reactor)
t = reactor.listenUDP(0, protocol)
l = task.LoopingCall(protocol.sendDatagram)
l.start(1.0) # call every second
reactor.run()
if __name__ == '__main__':
main()
Interesting. That sounds like something that shouldn't happen. Is this due to a network interface restart or something? What conditions reproduce this?
The simple answer to your question is probably "call listenUDP again, with 'self'", but I am curious what could cause this error to happen in the first place.
i want to add ssl-support to an existing TCP-server which is based on the SocketServer.TCPServer class.
So i overrode the default constructor of the TCPServer class and added the ssl.wrap_socket(...)-call:
class MyTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
def __init__(self, server_address, RequestHandlerClass, bind_and_activate=True):
# See SocketServer.TCPServer.__init__
# (added ssl-support):
SocketServer.BaseServer.__init__(self, server_address,
RequestHandlerClass)
self.socket = ssl.wrap_socket(
socket.socket(self.address_family, self.socket_type),
server_side=True,
certfile='cert.pem'
)
if bind_and_activate:
self.server_bind()
self.server_activate()
When starting the server, no error occurrs.
So i modified my simple test-client to support ssl, too:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock = ssl.wrap_socket(s)
sock.connect(('192.168.1.1', 54321))
Again no error occurrs, but the connect-call is blocking. When closing the client using Ctrl+C it shows the following:
Traceback (most recent call last):
File "exampleClient.py", line 10, in <module>
sock.do_handshake()
File "/usr/lib/python2.6/ssl.py", line 293, in do_handshake
self._sslobj.do_handshake()
KeyboardInterrupt
So the do_handshake is blocking when connecting. Does anyone knows how to fix the problem? I simply want to use an encrypted TCP-connection :)
The handshake is blocking because you are wrapping the socket after binding; the socket is listening for new connections, there is no client yet to accept your connections.
Wrap the socket when accepting a connection instead:
class MyTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
def get_request(self):
(socket, addr) = SocketServer.TCPServer.get_request(self)
return (ssl.wrap_socket(socket, server_side=True, certfile="cert.pem"),
addr)
Now the handshake succeeds because there is a client on the other side to shake hands with.
There is no additional work necessary for the stream handler; the python ssl library gives you objects with the same interface as socket.socket().
You can also wrap the socket early, but do postpone the handshake until you accept a connection:
class MyTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
def server_bind(self):
SocketServer.TCPServer.server_bind(self)
self.socket = ssl.wrap_socket(
self.socket, server_side=True, certfile="cert.pem",
do_handshake_on_connect=False)
def get_request(self):
(socket, addr) = SocketServer.TCPServer.get_request(self)
socket.do_handshake()
return (socket, addr)
Ok, i found a solution. Now i use something similar to
this using the OpenSSL-package:
Inside the MyTCPServer-Constructor:
SocketServer.BaseServer.__init__(self, server_address, RequestHandlerClass)
ctx = SSL.Context(SSL.SSLv23_METHOD)
cert = 'cert.pem'
ctx.use_privatekey_file(cert)
ctx.use_certificate_file(cert)
self.socket = SSL.Connection(ctx, socket.socket(self.address_family,
self.socket_type))
if bind_and_activate:
self.server_bind()
self.server_activate()
And in the setup-method of the StreamRequestHandler:
self.connection = self.request
self.rfile = socket._fileobject(self.request, "rb", self.rbufsize)
self.wfile = socket._fileobject(self.request, "wb", self.wbufsize)
This seems to work fine :-)
That's because you have to set the do_handshake_on_connect argument in your call to ssl.wrap_socket:
The parameter do_handshake_on_connect
specifies whether to do the SSL
handshake automatically after doing a
socket.connect(), or whether the
application program will call it
explicitly, by invoking the
SSLSocket.do_handshake() method.
Calling SSLSocket.do_handshake()
explicitly gives the program control
over the blocking behavior of the
socket I/O involved in the handshake.
Source: http://docs.python.org/library/ssl.html
The code below binds an ip address to urllib, urllib2, etc.
import socket
true_socket = socket.socket
def bound_socket(*a, **k):
sock = true_socket(*a, **k)
sock.bind((sourceIP, 0))
return sock
socket.socket = bound_socket
Is it also able to bind an ip address to telnetlib?
telnetlib at least in recent Python releases uses socket.create_connection (see telnetlib's sources here) but that should also be caught by your monkeypatch (sources here -- you'll see it uses a bare identifier socket but that's exactly in the module you're monkeypatching). Of course monkeypatching is always extremely fragile (the tiniest optimization in some future release, hoisting the global lookup of socket in create_connection, and you're toast...;-) so maybe you'll want to monkeypath create_connection directly as a modestly-stronger approach.