Develop Reference

Python Async download certificate chain

I want to download the TLS certificate chain for a given website.
I have a running code using blocking sockets, code provided here,
Getting certificate chain with Python 3.3 SSL module.
from OpenSSL import SSL
import socket
def get_certificates(hostname, port):
context = SSL.Context(method=SSL.TLSv1_METHOD)
conn = SSL.Connection(context, socket=socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM))
conn.settimeout(1)
conn.connect((hostname, port))
conn.setblocking(1)
conn.do_handshake()
conn.set_tlsext_host_name(hostname.encode())
chain = conn.get_peer_cert_chain()
conn.close()
return chain
def main():
hostname = 'www.google.com'
port = 443
chain = get_certificates(hostname, port)
This code is running fine. I want to use async to make multiprocessing with a large list of hostnames more performant. I didn't find a clear way to do it. What's the best way?

Listen to websockets running Raspbian from Windows

I've created a websocket using Python's asyncio and websockets modules. This servers works properly in the same machine. This is the actual code for the server:
import sys
import os
import asyncio
import websockets
#asyncio.coroutine
def receive(websocket, path):
data = yield from websocket.recv()
print('< {}'.format(data))
output = 'Sent data from server: {}'.format(data)
yield from websocket.send(output)
print('> {}'.format(output))
start_server = websockets.serve(receive, '127.0.0.1', 8765)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
It runs properly, and the connection from a client residing on the same machine connects to it without any problem.
But when I try to access it from a client on a LAN network, it generates a ConnectionRefusedError. This is the client code:
import asyncio
import websockets
#asyncio.coroutine
def hello():
websocket = yield from websockets.connect(
'ws://192.168.0.26:8765')
try:
name = input("What's your name? ")
yield from websocket.send(name)
print("> {}".format(name))
greeting = yield from websocket.recv()
print("< {}".format(greeting))
finally:
yield from websocket.close()
asyncio.get_event_loop().run_until_complete(hello())
I've installed ufw on Raspbian to enable the port 8765 with this command:
ufw allow 8765
But it doesn't work. On the Windows machine, the command
nmap -p 8765 192.168.0.26
generates this result:
PORT STATE SERVICE
8765/tcp closed ultraseek-http
And... the command
ufw status
Could someone give some suggestions to solve this communication problem between the client and the server.

Here is one problem:
start_server = websockets.serve(receive, '127.0.0.1', 8765)
You have told websockets to listen only on 127.0.0.1, thus you can only receive connections originating from the local host, and only on legacy IPv4. Both localhost IPv6 connections (the default) and all connections from other computers will receive the connection refused error.
If you want to receive connections from outside the local machine, you should set the Host to None or the empty string. This will accept connections from anywhere, on both IPv6 and IPv4, subject of course to any firewall rules.
start_server = websockets.serve(receive, None, 8765)
The host and port are passed directly to asyncio.create_server() which documents Host as:
If host is a string, the TCP server is bound to a single network interface specified by host.
If host is a sequence of strings, the TCP server is bound to all network interfaces specified by the sequence.
If host is an empty string or None, all interfaces are assumed and a list of multiple sockets will be returned (most likely one for IPv4 and another one for IPv6).

How can I do certificate pinning for TLS sockets in python 2.7?

I have a python TLS server using a self signed certificate. That works. The code looks like this for now:
#!/usr/bin/python
import socket, ssl
context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
context.load_cert_chain(certfile="server.crt", keyfile="server.key")
bindsocket = socket.socket()
bindsocket.bind(('127.0.0.1', 8888))
bindsocket.listen(5)
while True:
newsocket, fromaddr = bindsocket.accept()
connstream = context.wrap_socket(newsocket, server_side=True)
try:
print("Got connection!")
finally:
connstream.shutdown(socket.SHUT_RDWR)
connstream.close()
I am now trying to make a client in python that connects to this server. On the first connection attempt, I'd like to retrieve either the public key, or a hash of the public key, and then verify this on all future connections. How can I do with with python and the ssl package?
This is the code I'm playing with:
#!/usr/bin/python
import ssl, socket, pprint
context = ssl.SSLContext(ssl.PROTOCOL_TLS)
context.verify_mode = ssl.CERT_REQUIRED
context.check_hostname = False
cnx = context.wrap_socket(socket.socket(socket.AF_INET), certfile="server.crt")
cnx.connect(('127.0.0.1', 8888))
pprint.pprint(cnx.getpeercert())
As it stands right now, it fails because there is no certificate chain to verify the cert. I don't care about that, though. All I care about is that the server I'm talking to has the private key that matches the public key. What do I do?

So I've made this work, though it's not exactly what I hoped for. My biggest complaint is that it requires storing the entire server certificate in a file on disk, whereas really all I care about is the public key. However, here's what I have working:
#!/usr/bin/python
import ssl, socket, pprint, os, sys
# If we haven't retrieved the certificate previously...
if not os.path.isfile('server_cert.pem'):
# Grab the cert
cert = ssl.get_server_certificate(('127.0.0.1', 8888))
# If it worked...
if cert:
# Write it to a file
with open('server_cert.pem', 'w') as f:
f.write(cert)
else:
sys.exit()
# Prepare context, including reference to the server's certificate
context = ssl.SSLContext(ssl.PROTOCOL_TLS)
context.verify_mode = ssl.CERT_REQUIRED
context.check_hostname = False
context.load_verify_locations(cafile='server_cert.pem')
cnx = context.wrap_socket(socket.socket(socket.AF_INET))
# Connect and evaluate
cnx.connect(('127.0.0.1', 8888))
I'll wait a bit to mark this answer accepted to see if someone can find a cleaner way of doing this.

got stuck programming https proxy - what to do next?

I'm currently making a proxy which sits between the browser and the web. Everything works except https. I'm having troubles understanding some passages of it and haven't found many resources on the web. And so I'm stuck.
The code I'm using is:
conn, addr = server.accept()
request = conn.recv(9999) #get a CONNECT request
conn.send(b'HTTP/1.1 200 Connection estabilished\n\n')
enc_req = conn.recv(9999) #this gets an encrypted request
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #plaintext client
client.connect((host, 443)) #connect to chosen host
client.send(enc_req)
resp1 = client.recv(9999) #this gets something unreadable (encrypted?)
#could it be the certificate?
#now what?
Is the resp1 I'm getting the certificate? And what do I need to do after that? (Or, which is the same, what does usually happens next with https?)
P.S. I know the question is somewhat generic, but please don't judge me too harshly. I've tried researching on the web but all I keep finding is the encryption method used for ssl. I really don't know how to proceed.

I haven't tested this code (and it's mainly pseudo code), but this should give you an idea of what you need to do.
conn, addr = server.accept()
request = conn.recv(9999) #get a CONNECT request
# Here, parse the CONNECT string and get the host and port (not sure if you were doing that already.
# Then, try to connect *before* you tell the client the connection was established (in case it fails)
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #plaintext client
client.connect((host, 443)) #connect to chosen host
conn.send(b'HTTP/1.1 200 Connection estabilished\n\n')
# Then loop until the connections are closed.
while True:
# Read from the client, send the data to the server.
enc_req = conn.recv(9999) #this gets an encrypted request
client.send(enc_req)
# Read from the server, send the data to the client.
resp1 = client.recv(9999) #this gets something unreadable (encrypted?)
#could it be the certificate?
#now what?
# The first time it's certainly the Client Hello message, not encrypted, but in a binary format indeed.
# Just send everything you've just read to the server.
conn.send(resp1)
This is just a quick overview of the idea of the loop you need to write. In reality, you may be able to process both in parallel. You'd also want to be a bit more careful when closing the connection (allowing it to happen in any order while still relaying the last data sent by either party).

As mentioned in the comments, a proxy handling encrypted end-to-end traffic can only pass it on.
Here is a fully working proxy written using circuits that has been fully tested with passing and proxying SSH traffic so it should work equally as well as a pass-through TCP proxy even if SSL is involved:
#!/usr/bin/env python
from uuid import uuid4 as uuid
from circuits import Component
from circuits.net.events import close, connect, write
from circuits.net.sockets import TCPClient, TCPServer
class Client(Component):
channel = "client"
def init(self, sock, host, port, channel=channel):
self.sock = sock
self.host = host
self.port = port
TCPClient(channel=self.channel).register(self)
def ready(self, *args):
self.fire(connect(self.host, self.port))
def disconnect(self, *args):
self.fire(close(self.sock), self.parent.channel)
def read(self, data):
self.fire(write(self.sock, data), self.parent.channel)
class Proxy(Component):
channel = "server"
def init(self, bind, host, port):
self.bind = bind
self.host = host
self.port = port
self.clients = dict()
TCPServer(self.bind).register(self)
def connect(self, sock, host, port):
channel = uuid()
client = Client(
sock, self.host, self.port, channel=channel
).register(self)
self.clients[sock] = client
def disconnect(self, sock):
client = self.clients.get(sock)
if client is not None:
client.unregister()
del self.clients[sock]
def read(self, sock, data):
client = self.clients[sock]
self.fire(write(data), client.channel)
app = Proxy(("0.0.0.0", 3333), "127.0.0.1", 22)
from circuits import Debugger
Debugger().register(app)
app.run()

How do I use TLS with asyncore?

An asyncore-based XMPP client opens a normal TCP connection to an XMPP server. The server indicates it requires an encrypted connection. The client is now expected to start a TLS handshake so that subsequent requests can be encrypted.
tlslite integrates with asyncore, but the sample code is for a server (?) and I don't understand what it's doing.
I'm on Python 2.5. How can I get the TLS magic working?
Here's what ended up working for me:
from tlslite.api import *
def handshakeTls(self):
"""
Encrypt the socket using the tlslite module
"""
self.logger.info("activating TLS encrpytion")
self.socket = TLSConnection(self.socket)
self.socket.handshakeClientCert()

Definitely check out twisted and wokkel. I've been building tons of xmpp bots and components with it and it's a dream.

I've followed what I believe are all the steps tlslite documents to make an asyncore client work -- I can't actually get it to work since the only asyncore client I have at hand to tweak for the purpose is the example in the Python docs, which is an HTTP 1.0 client, and I believe that because of this I'm trying to set up an HTTPS connection in a very half-baked way. And I have no asyncore XMPP client, nor any XMPP server requesting TLS, to get anywhere close to your situation. Nevertheless I decided to share the fruits of my work anyway because (even though some step may be missing) it does seem to be a bit better than what you previously had -- I think I'm showing all the needed steps in the __init__. BTW, I copied the pem files from the tlslite/test directory.
import asyncore, socket
from tlslite.api import *
s = open("./clientX509Cert.pem").read()
x509 = X509()
x509.parse(s)
certChain = X509CertChain([x509])
s = open("./clientX509Key.pem").read()
privateKey = parsePEMKey(s, private=True)
class http_client(TLSAsyncDispatcherMixIn, asyncore.dispatcher):
ac_in_buffer_size = 16384
def __init__(self, host, path):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.connect( (host, 80) )
TLSAsyncDispatcherMixIn.__init__(self, self.socket)
self.tlsConnection.ignoreAbruptClose = True
handshaker = self.tlsConnection.handshakeClientCert(
certChain=certChain,
privateKey=privateKey,
async=True)
self.setHandshakeOp(handshaker)
self.buffer = 'GET %s HTTP/1.0\r\n\r\n' % path
def handle_connect(self):
pass
def handle_close(self):
self.close()
def handle_read(self):
print self.recv(8192)
def writable(self):
return (len(self.buffer) > 0)
def handle_write(self):
sent = self.send(self.buffer)
self.buffer = self.buffer[sent:]
c = http_client('www.readyhosting.com', '/')
asyncore.loop()
This is a mix of the asyncore example http client in the Python docs, plus what I've gleaned from the tlslite docs and have been able to reverse engineer from their sources. Hope this (even though incomplete/not working) can at least advance you in your quest...
Personally, in your shoes, I'd consider switching from asyncore to twisted -- asyncore is old and rusty, Twisted already integrates a lot of juicy, useful bits (the URL I gave is to a bit in the docs that already does integrate TLS and XMPP for you...).