I am trying to modify a socket server I wrote with the python socket library to use encryption using python's SSL library.
I am no able to successfully open a connection to the server, wrap it with an SSL context and send data to the server, but data sent back to the client is not what it should be.
My suspicion is that the server responses are not being decrypted on the client side, but I don't know why. I'm pretty new to SSL/TLS, and networking in general so... what am I missing?
The client is also written in python (for now, to facilitate testing)
Code:
Relevant Server stuff:
def sslServerLoop():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host, port))
s.listen(5)
context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
context.load_cert_chain('cert.pem')
while True:
conn, addr = s.accept()
sslConn = context.wrap_socket(conn, server_side=True)
data = sslConn.recv(1024)
sslConn.sendall(response)
sslConn.close()
Relevant Client stuff:
context = ssl.create_default_context(cafile='cert.pem')
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s = context.wrap_socket(s, server_hostname=server_addr)
s.connect((address, port))
s.sendall(msg)
s.shutdown(socket.SHUT_WR)
response = s.recv(1024)
Sending from client to server works fine, but data sent back to the client is wrong. For example if I set response = bytes([1]) on the server side, I receive b'\x17\x03\x03\x00\x19\xac\xb6\x7f#\xc0\xd3\xce%\x13G\x01\xbd\x88y\xf0\xda..\x02\xf9\xe4o\xdd\x1a\xdb' on the client side. Most of that changes every time I try to run it, but the first 5 bytes are always the same (which is partly why I suspect it isn't being decrypted).
cert.pem is a self signed certificate generated using openssl as described in the python 3 SSL module documentation
It is not legal to shutdown a socket that is being used for SSL. It is a protocol violation. You must close via the SSL/TLS API you are using.
I need to simulate a tls renegotiation behaviour (I understand this as a new handshake) by python. Is that possible?
I tried below codes, the first do_handshake works but the second one do nothing.
import socket, ssl, pprint, re, time
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ssl_sock = ssl.wrap_socket(s,ca_certs="cacert.pem",do_handshake_on_connect=False)
ssl_sock.connect(('172.18.7.162', 443))
time.sleep(3)
ssl_sock.do_handshake()
print repr(ssl_sock.getpeername())
print ssl_sock.cipher()
print pprint.pformat(ssl_sock.getpeercert())
send_content="aaaa"
ssl_sock.write(send_content)
time.sleep(2)
print "do_handshake_again"
ssl_sock.do_handshake()
print "do_handshake_again done"
ssl_sock.write(send_content)
Thanks for helping!
To answer my own question:
Finally I implement this behaviour by using python openssl lib.
from OpenSSL import SSL
import sys, os, select, socket
........
# Initialize context
ctx = SSL.Context(SSL.SSLv23_METHOD)
ctx.set_options(SSL.OP_NO_SSLv2)
#ctx.set_verify(SSL.VERIFY_PEER|SSL.VERIFY_FAIL_IF_NO_PEER_CERT, verify_cb) # Demand a certificate
ctx.use_privatekey_file (os.path.join(dir, 'server.pkey'))
ctx.use_certificate_file(os.path.join(dir, 'server.cert'))
ctx.load_verify_locations(os.path.join(dir, 'CA.cert'))
# Set up server
server = SSL.Connection(ctx, socket.socket(socket.AF_INET, socket.SOCK_STREAM))
server.bind(('', int(sys.argv[1])))
server.listen(3)
server.setblocking(0)
........
for cli in w:
try:
ret = cli.send(writers[cli])
cli.renegotiate()
cli.do_handshake()
ret = cli.send(writers[cli])
......
The important thing is the last 4 lines:
1, sending something with the socket
2-3, trigger renegotiate and do handshake
4, sending something again
Because it's non-blocking socket, I can see this code send out two packets: the first packet will only send application data(content type 23), the second packet will have two payloads: one is ssl handshake(type 22) and another one is application data(type 23).
By the way, this is trying to simulate re-negotiate packet has application data in same packet. If for pure tls re-negotiate, we can use openssl to send "R" to trigger a pure tls renegotiation behaviour.
I'm trying to use tor, socksipy and ssl to proxy a ssl connection. My client looks like this:
import socks, ssl
s = socks.socksocket()
s.setproxy(socks.PROXY_TYPE_SOCKS5,"127.0.0.1", 9050)
ssl_sock = ssl.wrap_socket(s, ssl_version=ssl.PROTOCOL_TLSv1)
ssl_sock.connect(('127.0.0.1', 443))
The server just accepts connections and prints getpeername.
The peer name is always 127.0.0.1. It doesn't even matter if I give it a non-valid proxy. The client won't complain, it will connect anyway.
How do I make it connect through the proxy?
I managed to figure it out so I will leave the answer here for future reference.
The first problem was that I tried to connect to 127.0.0.1. As the request was proxied, the proxy would try to connect to 127.0.0.1, so it would try to connect to itself, not to me.
I had to configure my router to forward requests on port 443 to my laptop and then I replaced 127.0.0.1 with my routers IP.
After that was out of the way, I found out that socksipy doesn't play very well with ssl.
I had to call connect on the socket before wrapping it, otherwise I'd get a handshake failure. The code became:
import socks, ssl
s = socks.socksocket()
s.setproxy(socks.PROXY_TYPE_SOCKS5,"127.0.0.1", 9050)
s.connect(('127.0.0.1', 443))
ssl_sock = ssl.wrap_socket(s, ssl_version=ssl.PROTOCOL_TLSv1)
After that, everything was working fine.
I'm trying to write a python web server using the socket library. I've been through several sources and can't figure out why the code I've written doesn't work. Others have run very similar code and claim it works. I'm new to python so I might be missing something simple.
The only way it will work now is I send the data variable back to the client. The browser prints the original GET request. When I try to send an HTTP response, the connection times out.
import socket
##Creates several variables, including the host name, the port to use
##the size of a transmission, and how many requests can be handled at once
host = ''
port = 8080
backlog = 5
size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen(backlog)
while 1:
client, address = s.accept()
data = client.recv(16)
if data:
client.send('HTTP/1.0 200 OK\r\n')
client.send("Content-Type: text/html\r\n\r\n")
client.send('<html><body><h1>Hello World</body></html>')
client.close()
s.close()
You need to consume the input before responding, and you shouldn't close the socket in your while loop:
Replace client.recv(16) with client.recv(size), to consume the request.
Move your last line, s.close() back one indent, so that it is not in your while loop. At the moment you are closing the connection, then trying to accept from it again, so your server will crash after the first request.
Unless you are doing this as an exercise, you should extend SimpleHTTPServer instead of using sockets directly.
Also, adding this line after your create the socket (before bind) fixes any "Address already in use" errors you might be getting.
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
Good luck!
I have a question regarding client socket on TCP/IP network. Let's say I use
try:
comSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
comSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
except socket.error, msg:
sys.stderr.write("[ERROR] %s\n" % msg[1])
sys.exit(1)
try:
comSocket.bind(('', 5555))
comSocket.connect()
except socket.error, msg:
sys.stderr.write("[ERROR] %s\n" % msg[1])
sys.exit(2)
The socket created will be bound to port 5555. The problem is that after ending the connection
comSocket.shutdown(1)
comSocket.close()
Using wireshark, I see the socket closed with FIN,ACK and ACK from both sides, I can't use the port again. I get the following error:
[ERROR] Address already in use
I wonder how can I clear the port right away so that next time I still can use that same port.
comSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
setsockopt doesn't seem to be able to resolve the problem
Thank you!
Try using the SO_REUSEADDR socket option before binding the socket.
comSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
Edit:
I see you're still having trouble with this. There is a case where SO_REUSEADDR won't work. If you try to bind a socket and reconnect to the same destination (with SO_REUSEADDR enabled), then TIME_WAIT will still be in effect. It will however allow you to connect to a different host:port.
A couple of solutions come to mind. You can either continue retrying until you can gain a connection again. Or if the client initiates the closing of the socket (not the server), then it should magically work.
Here is the complete code that I've tested and absolutely does NOT give me a "address already in use" error. You can save this in a file and run the file from within the base directory of the HTML files you want to serve. Additionally, you could programmatically change directories prior to starting the server
import socket
import SimpleHTTPServer
import SocketServer
# import os # uncomment if you want to change directories within the program
PORT = 8000
# Absolutely essential! This ensures that socket resuse is setup BEFORE
# it is bound. Will avoid the TIME_WAIT issue
class MyTCPServer(SocketServer.TCPServer):
def server_bind(self):
self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.socket.bind(self.server_address)
Handler = SimpleHTTPServer.SimpleHTTPRequestHandler
httpd = MyTCPServer(("", PORT), Handler)
# os.chdir("/My/Webpages/Live/here.html")
httpd.serve_forever()
# httpd.shutdown() # If you want to programmatically shut off the server
According to this link
Actually, SO_REUSEADDR flag can lead to much greater consequences:
SO_REUSADDR permits you to use a port that is stuck in TIME_WAIT, but
you still can not use that port to establish a connection to the last
place it connected to. What? Suppose I pick local port 1010, and
connect to foobar.com port 300, and then close locally, leaving that
port in TIME_WAIT. I can reuse local port 1010 right away to connect
to anywhere except for foobar.com port 300.
However you can completely avoid TIME_WAIT state by ensuring that the remote end initiates the closure (close event). So the server can avoid problems by letting the client close first. The application protocol must be designed so that the client knows when to close. The server can safely close in response to an EOF from the client, however it will also need to set a timeout when it is expecting an EOF in case the client has left the network ungracefully. In many cases simply waiting a few seconds before the server closes will be adequate.
I also advice you to learn more about networking and network programming. You should now at least how tcp protocol works. The protocol is quite trivial and small and hence, may save you a lot of time in future.
With netstat command you can easily see which programs ( (program_name,pid) tuple) are binded to which ports and what is the socket current state: TIME_WAIT, CLOSING, FIN_WAIT and so on.
A really good explanation of linux network configurations can be found https://serverfault.com/questions/212093/how-to-reduce-number-of-sockets-in-time-wait.
In case you face the problem using TCPServer or SimpleHTTPServer,
override SocketServer.TCPServer.allow_reuse_address (python 2.7.x)
or socketserver.TCPServer.allow_reuse_address (python 3.x) attribute
class MyServer(SocketServer.TCPServer):
allow_reuse_address = True
server = MyServer((HOST, PORT), MyHandler)
server.serve_forever()
You need to set the allow_reuse_address before binding. Instead of the SimpleHTTPServer run this snippet:
Handler = SimpleHTTPServer.SimpleHTTPRequestHandler
httpd = SocketServer.TCPServer(("", PORT), Handler, bind_and_activate=False)
httpd.allow_reuse_address = True
httpd.server_bind()
httpd.server_activate()
httpd.serve_forever()
This prevents the server from binding before we got a chance to set the flags.
As Felipe Cruze mentioned, you must set the SO_REUSEADDR before binding. I found a solution on another site - solution on other site, reproduced below
The problem is that the SO_REUSEADDR socket option must be set before
the address is bound to the socket. This can be done by subclassing
ThreadingTCPServer and overriding the server_bind method as follows:
import SocketServer, socket
class MyThreadingTCPServer(SocketServer.ThreadingTCPServer):
def server_bind(self):
self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.socket.bind(self.server_address)
I found another reason for this exception.
When running the application from Spyder IDE (in my case it was Spyder3 on Raspbian) and the program terminated by ^C or an exception, the socket was still active:
sudo netstat -ap | grep 31416
tcp 0 0 0.0.0.0:31416 0.0.0.0:* LISTEN 13210/python3
Running the program again found the "Address already in use"; the IDE seems to start the new 'run' as a separate process which finds the socket used by the previous 'run'.
socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
did NOT help.
Killing process 13210 helped.
Starting the python script from command-line like
python3 <app-name>.py
always worked well when SO_REUSEADDR was set to true. The new Thonny IDE or Idle3 IDE did not have this problem.
socket.socket() should run before socket.bind() and use REUSEADDR as said
I know you've already accepted an answer but I believe the problem has to do with calling bind() on a client socket. This might be OK but bind() and shutdown() don't seem to play well together. Also, SO_REUSEADDR is generally used with listen sockets. i.e. on the server side.
You should be passing and ip/port to connect(). Like this:
comSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
comSocket.connect(('', 5555))
Don't call bind(), don't set SO_REUSEADDR.
For me the better solution was the following. Since the initiative of closing the connection was done by the server, the setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) had no effect and the TIME_WAIT was avoiding a new connection on the same port with error:
[Errno 10048]: Address already in use. Only one usage of each socket address (protocol/IP address/port) is normally permitted
I finally used the solution to let the OS choose the port itself, then another port is used if the precedent is still in TIME_WAIT.
I replaced:
self._socket.bind((guest, port))
with:
self._socket.bind((guest, 0))
As it was indicated in the python socket documentation of a tcp address:
If supplied, source_address must be a 2-tuple (host, port) for the socket to bind to as its source address before connecting. If host or port are ‘’ or 0 respectively the OS default behavior will be used.
another solution, in development environment of course, is killing process using it, for example
def serve():
server = HTTPServer(('', PORT_NUMBER), BaseHTTPRequestHandler)
print 'Started httpserver on port ' , PORT_NUMBER
server.serve_forever()
try:
serve()
except Exception, e:
print "probably port is used. killing processes using given port %d, %s"%(PORT_NUMBER,e)
os.system("xterm -e 'sudo fuser -kuv %d/tcp'" % PORT_NUMBER)
serve()
raise e
I think the best way is just to kill the process on that port, by typing in the terminal fuser -k [PORT NUMBER]/tcp, e.g. fuser -k 5001/tcp.
I had the same problem and I couldn't find any other solution (reuse options didn't work) except restarting Raspberry Pi each time. Then I found a workaround;
comSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
comSocket.close()
comSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
comSocket.connect(('', 5555))
This means, define socket first, close it, then define again, so you can use the same port if it is stuck.