Develop Reference

TLS handshake not completing

I have a client and server application with self-signed certificates. The do_handshake method is not working properly. In the client the SSL negotiation is finished successfully, but not on the server. The server says before SSL initialization all the time (using get_state_string()).
See the code.
Client
from OpenSSL import SSL, crypto
import socket
HOST = "localhost"
PORT = 8080
def verify_cb(conn, cert, errnum, depth, ok):
print(f"Got certificate: %s {cert.get_subject()}")
print(f"Issued by: {cert.get_issuer()}")
return ok
# Initialise SSL context:
ctx = SSL.Context(SSL.TLSv1_2_METHOD)
ctx.set_verify(SSL.VERIFY_PEER, verify_cb) # Demand a server certificate
ctx.load_verify_locations("serverpath.pem")
ctx.use_privatekey_file('clientkey.pem')
ctx.use_certificate_file('clientpath.pem')
# Set up client:
sock = SSL.Connection(ctx, socket.socket(socket.AF_INET, socket.SOCK_STREAM))
sock.connect((HOST, PORT))
sock.set_connect_state()
print(sock.get_state_string())
while True:
try:
sock.do_handshake()
break
except SSL.WantReadError:
pass
print(sock.get_state_string())
sock.write("HELLO")
# Read response:
while True:
try:
print(sock.recv(4096))
except SSL.ZeroReturnError:
break
Server
from OpenSSL import SSL, crypto
import socket
HOST = "localhost"
PORT = 8080
def verify_cb(conn, cert, errnum, depth, ok):
print(f"Got certificate: %s {cert.get_subject()}")
print(f"Issued by: {cert.get_issuer()}")
return ok
# Initialise SSL context:
ctx = SSL.Context(SSL.TLSv1_2_METHOD)
ctx.set_verify(SSL.VERIFY_PEER, verify_cb) # Demand a client certificate
ctx.load_verify_locations("clientpath.pem")
ctx.use_privatekey_file('serverkey.pem')
ctx.use_certificate_file('serverpath.pem')
# Set up sever:
sock = SSL.Connection(ctx, socket.socket(socket.AF_INET, socket.SOCK_STREAM))
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind((HOST, PORT))
sock.listen(1)
print("Waiting for connections.")
#Wait for clients to connect:
(conn, address) = sock.accept()
sock.set_accept_state()
print(f"Got connection from {address}")
print(sock.get_state_string())
while True:
try:
print(sock.get_state_string())
print(conn.recv(4096))
print(sock.get_state_string())
except SSL.ZeroReturnError:
break
Please, can anyone tell me what am I doing wrong?

(conn, address) = sock.accept()
sock.set_accept_state()
print(f"Got connection from {address}")
print(sock.get_state_string())
while True:
try:
print(sock.get_state_string())
print(conn.recv(4096))
print(sock.get_state_string())
You need to operate on the accepted socket conn and not on the server socket sock. While you read from the accepted socket you print the state of the server socket instead, which does not reflect the state of the connected socket. Also, you don't need to set the accept state since you've already called accept on the SSL server socket:
(conn, address) = sock.accept()
print(f"Got connection from {address}")
print(conn.get_state_string())
while True:
try:
print(conn.get_state_string())
print(conn.recv(4096))
print(conn.get_state_string())

What details about the host machine does python send to the web server while establishing a connection through sockets?

I am building a passive reconnaissance tool and one of it functionality is to get certificate info about a domain.
What details of my machine are being to sent to the webserver?
The below code is used to get the certificate info.
from OpenSSL import SSL
from cryptography import x509
from cryptography.x509.oid import NameOID
import idna
from socket import socket
from collections import namedtuple
HostInfo = namedtuple(field_names='cert hostname peername', typename='HostInfo')
HOSTS = [
('google.com', 443),
('yahoo.com', 443),
('yahoo.com', 443),
]
def get_certificate(hostname, port):
hostname_idna = idna.encode(hostname)
sock = socket()
sock.connect((hostname, port))
peername = sock.getpeername()
ctx = SSL.Context(SSL.SSLv23_METHOD) # most compatible
ctx.check_hostname = False
ctx.verify_mode = SSL.VERIFY_NONE
sock_ssl = SSL.Connection(ctx, sock)
sock_ssl.set_connect_state()
sock_ssl.set_tlsext_host_name(hostname_idna)
sock_ssl.do_handshake()
cert = sock_ssl.get_peer_certificate()
crypto_cert = cert.to_cryptography()
sock_ssl.close()
sock.close()
return HostInfo(cert=crypto_cert, peername=peername, hostname=hostname)
import concurrent.futures
if __name__ == '__main__':
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e:
for hostinfo in e.map(lambda x: get_certificate(x[0], x[1]), HOSTS):
print_basic_info(hostinfo)
Is my IP address being sent?
Is python sending any user-agent and what other details are being sent while establishing a connection?

While Python doesn't explicitly send your IP, it is always sent with any TCP request. If it isn't sent, you cannot receive the response from the server.
According to http://evanhahn.com/python-requests-library-useragent, the useragent is
python-requests/{package version} {runtime}/{runtime version} {uname}/{uname -r} when using python-requests, but you can override it by setting the request header.
However, you are not using python-requests but raw sockets, so no data is transferred except what you tell it to (and your IP and some metadata for the TCP connection)

How can a Python proxy server (using SSL socket) pretend to be an HTTPS server and specify my own keys to get decrypted data?

I'm trying to handle HTTPS connections from web browsers in decrypted data using Python SSL socket. I thought I can do this like a man-in-the-middle attack via my Python proxy server program, by pretending to be an HTTPS server, and I've already generated my keys using OpenSSL. However, browser treats my program as a proxy server, not as an HTTPS server. How can my program pretend to be an HTTPS server and specify my own keys to communicate with the browser?
I'm using Chrome and a Chrome extension SwitchySharp to forward requests to another local port (HTTP to 127.0.0.1:8080, and HTTPS to 127.0.0.1:8081). It works well for HTTP requests, but doesn't work for HTTPS.
Background Knowledge
When handling HTTPS with a proxy server, the browser first sends a CONNECT request to proxy:
CONNECT xxx.xxx:443 HTTP/1.1
Host: xxx.xxx:443
Proxy-Connection: keep-alive
(some other headers)
and proxy creates a tunnel to target server, and then browser sends encrypted data via this tunnel. Therefore, the proxy server never gets decrypted data.
To get decrypted data, I've tried two ways.
First, I tried ssl.wrap_socket():
import socket
import ssl
import thread
def handle(conn):
request = conn.recv(4096)
print request
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('127.0.0.1', 8081))
sock.listen(10)
sock = ssl.wrap_socket(sock, 'pkey.pem', 'cert.pem', True)
while True:
conn, addr = sock.accept()
thread.start_new_thread(handle, (conn,))
But I got the following error:
Traceback (most recent call last):
File "https.py", line 12, in <module>
conn, addr = sock.accept()
File "C:\Python27\lib\ssl.py", line 898, in accept
server_side=True)
File "C:\Python27\lib\ssl.py", line 369, in wrap_socket
_context=self)
File "C:\Python27\lib\ssl.py", line 617, in __init__
self.do_handshake()
File "C:\Python27\lib\ssl.py", line 846, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: HTTPS_PROXY_REQUEST] https proxy request (_ssl.c:726)
The cause of this error is that the CONNECT request is not encrypted so ssl can't handle it. I have no idea how to get rid of this error.
Then, I tried the following code:
import socket
import thread
def handler(conn, tunn):
while True:
data = conn.recv(40960)
if data:
tunn.sendall(data)
else:
break
def handle(conn):
request = conn.recv(40960)
print request
i = request.find(' ') + 1
j = request.find(' ', i)
host, port = request[i : j].split(':')
port = int(port)
tunn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tunn.connect((host, port))
conn.sendall('HTTP/1.1 200 Connection Established\r\n\r\n')
thread.start_new_thread(handler, (conn, tunn))
thread.start_new_thread(handler, (tunn, conn))
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('127.0.0.1', 8081))
sock.listen(10)
while True:
conn, addr = sock.accept()
thread.start_new_thread(handle, (conn,))
It worked (without ssl module), but I can't specify my own keys and therefore can't get decrypted data.
It seems that my program has to pretend to be an HTTPS server to "directly" communicate with browser, so I can specify my own key and get decrypted data. But I really have no idea how to pretend to be an HTTPS server and specify my own keys.
Can anyone help me? Thanks in advance.
UPD: I finally solved this problem using ssl module.
I was wrong --- I thought that after connection established the browser sends encrypted data. But it doesn't. In fact, those unreadable data are just SSL handshake. So what I need to do is just to use ssl.wrap_socket() after connection established. Besides, I have to additionally call do_handshake() of the ssl-wrapped socket.
import socket
import ssl
import thread
def handle(tid, conn):
request = conn.recv(40960)
i = request.find(' ') + 1
j = request.find(' ', i)
if request[i : j].find(':') != -1:
host, port = request[i : j].split(':')
else:
host, port = request[i : j], 80
port = int(port)
tunn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tunn.connect((host, port))
conn.sendall('HTTP/1.1 200 Connection Established\r\n\r\n')
conn_s = ssl.wrap_socket(conn, keyfile = 'cert/userkey.pem', certfile = 'cert/usercert.pem', server_side = True, do_handshake_on_connect = False)
conn_s.do_handshake()
data = conn_s.recv(40960)
print data
conn_s.sendall('HTTP/1.1 200 OK\r\n\r\n<h1>Hello, world!</h1>')
conn_s.close()
tunn.close()
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('127.0.0.1', 8081))
sock.listen(10)
buf = {}
while True:
conn, addr = sock.accept()
thread.start_new_thread(handle, (conn,))
Anyway, it doesn't pretend to be an HTTPS server -- it still behaves as a proxy server. I just misunderstood those unreadable data.
Thanks to those who helped me in the comments.

Received data from python SSL server is incorrect

I am trying to modify a socket server I wrote with the python socket library to use encryption using python's SSL library.
I am no able to successfully open a connection to the server, wrap it with an SSL context and send data to the server, but data sent back to the client is not what it should be.
My suspicion is that the server responses are not being decrypted on the client side, but I don't know why. I'm pretty new to SSL/TLS, and networking in general so... what am I missing?
The client is also written in python (for now, to facilitate testing)
Code:
Relevant Server stuff:
def sslServerLoop():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host, port))
s.listen(5)
context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
context.load_cert_chain('cert.pem')
while True:
conn, addr = s.accept()
sslConn = context.wrap_socket(conn, server_side=True)
data = sslConn.recv(1024)
sslConn.sendall(response)
sslConn.close()
Relevant Client stuff:
context = ssl.create_default_context(cafile='cert.pem')
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s = context.wrap_socket(s, server_hostname=server_addr)
s.connect((address, port))
s.sendall(msg)
s.shutdown(socket.SHUT_WR)
response = s.recv(1024)
Sending from client to server works fine, but data sent back to the client is wrong. For example if I set response = bytes([1]) on the server side, I receive b'\x17\x03\x03\x00\x19\xac\xb6\x7f#\xc0\xd3\xce%\x13G\x01\xbd\x88y\xf0\xda..\x02\xf9\xe4o\xdd\x1a\xdb' on the client side. Most of that changes every time I try to run it, but the first 5 bytes are always the same (which is partly why I suspect it isn't being decrypted).
cert.pem is a self signed certificate generated using openssl as described in the python 3 SSL module documentation

It is not legal to shutdown a socket that is being used for SSL. It is a protocol violation. You must close via the SSL/TLS API you are using.

Python SSL verification fails

I am trying to setup a Python TCP client-server session, but the client side is throwing this error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
I am using a freshly generated LetsEncrypt certificate. I've stored the fullchain.pem (public certificate) on my computer which I'm running the client from, and yet it can't verify the authenticity. I am sure that fullchain.pem is the public key and privkey.pem is the private key, so I don't see why this isn't working. I've also tried using cert.pem for public too, which does not work either. Is anyone able to provide some insight on as to why the verification fails? Below is the client and server programs.
Client:
import sys
import socket
import os
import ssl
from backports.ssl_match_hostname import match_hostname, CertificateError
def connect(hostname, port, message):
"""
Connects to the hostname on the specified port and sends a message
:param hostname:
:param port:
:param message:
:return:
"""
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # create a TCP socket
try:
s.connect((hostname, port)) # establish a TCP connection to the host
#ca_certs_path = os.path.join(os.path.dirname(sys.argv[0]), 'fullchain.pem')
ctx = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH)
#sslsock = ctx.wrap_socket(s, ssl_version=ssl.PROTOCOL_SSLv23, cert_reqs=ssl.CERT_REQUIRED, ca_certs=ca_certs_path)
sslsock = ctx.wrap_socket(s, server_hostname='encryptio.tk')
# Check if the server really matches the hostname to which we are trying to connect
match_hostname(sslsock.getpeercert(), hostname)
sslsock.send(message.encode('utf-8')) # send the message
except Exception as e:
sys.exit('[-]' + str(e))
def main():
connect('encryptio.tk', 12000, 'Client says hello') # replace '' with the hostname, 12000 as the port, and 'Hello' as the message
if __name__ == '__main__':
main()
Server:
import sys
import socket
import ssl
from backports.ssl_match_hostname import match_hostname, CertificateError
# Servers do not care whether clients connect with certificates
def listen(port):
"""
Listens on the specified port and accepts incoming connections
:param port:
:return:
"""
# server_socket - welcoming socket
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(('', port)) # establish welcoming socket
ca_certs_path = '/etc/letsencrypt/live/encryptio.tk/fullchain.pem'
priv_certs_path = '/etc/letsencrypt/live/encryptio.tk/privkey.pem'
# listen for TCP connection requests.
server_socket.listen(1) # parameter specifies the maximum number of queued connections (at least 1)
ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
ctx.load_cert_chain(certfile=ca_certs_path, keyfile=priv_certs_path)
server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
print('[*] The server is ready to receive')
while 1:
try:
# When a client knocks on this door, the program invokes the accept() method for
# server_socket, which creates a new socket in the server, called connec-
# tion_ocket, dedicated to this particular client. The client and server then complete
# the handshaking, creating a TCP connection between the client’s client_socket
# and the server’s connection_socket
#connection_socket, addr = server_socket.accept()
#sslsock = ssl.wrap_socket(server_socket, ssl_version=ssl.PROTOCOL_SSLv23, cert_reqs=ssl.CERT_REQUIRED, server_side=True, certfile=ca_certs_path, keyfile=priv_certs_path)
connection_socket, addr = server_socket.accept()
sslsock = ctx.wrap_socket(connection_socket, server_side=True)
data = sslsock.read()
#message = connection_socket.recv(1500)
print('[+] From' + str(addr) + ': ' +
data.decode('utf-8'))
except Exception as e:
sys.exit('[-] ' + str(e))
#connection_socket.close()
sslsock.shutdown(socket.SHUT_RDWR)
sslsock.close()
def main():
listen(12000)
if __name__ == '__main__':
main()
EDIT:
I tried setting the CA in the client side program to a root certificate, and now I get this error:
[X509] PEM lib (_ssl.c:3053)

In your client you have this:
ca_certs_path = os.path.join(os.path.dirname(sys.argv[0]), 'fullchain.pem')
sslsock = ssl.wrap_socket(s, ssl_version=ssl.PROTOCOL_SSLv23,
cert_reqs=ssl.CERT_REQUIRED, ca_certs=ca_certs_path)
The default for ca_certs is None in ssl.wrap_socket which means that you have to provide a useful path if you use the non-default ssl.CERT_REQUIRED. Only, the path you provide seems to be the certificate chain you use in your server, i.e. server certificate and chain certificates but not the root certificate. Instead the ca_certs on the client side should only contain the root certificates, since this is the trust anchor which is used to build the trust chain using the certificates sent by the server (i.e. server certificate and chain).
Instead of using your own CA store you might also use the system wide store. With current versions of Python you can simply use the following code which creates a SSL context using the default CA store and with certificate and hostname validation enabled and then upgrades the socket to SSL and checks the certificate properly:
s.connect((hostname, port))
ctx = ssl.create_default_context()
sslsock = ctx.wrap_socket(s, server_hostname='example.com')