I want to make a little update script for a software that runs on a Raspberry Pi and works like a local server. That should connect to a master server in the web to get software updates and also to verify the license of the software.
For that I set up two python scripts. I want these to connect via a TLS socket. Then the client checks the server certificate and the server checks if it's one of the authorized clients. I found a solution for this using twisted on this page.
Now there is a problem left. I want to know which client (depending on the certificate) is establishing the connection. Is there a way to do this in Python 3 with twisted?
I'm happy with every answer.
In a word: yes, this is quite possible, and all the necessary stuff is
ported to python 3 - I tested all the following under Python 3.4 on my Mac and it seems to
work fine.
The short answer is
"use twisted.internet.ssl.Certificate.peerFromTransport"
but given that a lot of set-up is required to get to the point where that is
possible, I've constructed a fully working example that you should be able to
try out and build upon.
For posterity, you'll first need to generate a few client certificates all
signed by the same CA. You've probably already done this, but so others can
understand the answer and try it out on their own (and so I could test my
answer myself ;-)), they'll need some code like this:
# newcert.py
from twisted.python.filepath import FilePath
from twisted.internet.ssl import PrivateCertificate, KeyPair, DN
def getCAPrivateCert():
privatePath = FilePath(b"ca-private-cert.pem")
if privatePath.exists():
return PrivateCertificate.loadPEM(privatePath.getContent())
else:
caKey = KeyPair.generate(size=4096)
caCert = caKey.selfSignedCert(1, CN="the-authority")
privatePath.setContent(caCert.dumpPEM())
return caCert
def clientCertFor(name):
signingCert = getCAPrivateCert()
clientKey = KeyPair.generate(size=4096)
csr = clientKey.requestObject(DN(CN=name), "sha1")
clientCert = signingCert.signRequestObject(
csr, serialNumber=1, digestAlgorithm="sha1")
return PrivateCertificate.fromCertificateAndKeyPair(clientCert, clientKey)
if __name__ == '__main__':
import sys
name = sys.argv[1]
pem = clientCertFor(name.encode("utf-8")).dumpPEM()
FilePath(name.encode("utf-8") + b".client.private.pem").setContent(pem)
With this program, you can create a few certificates like so:
$ python newcert.py a
$ python newcert.py b
Now you should have a few files you can use:
$ ls -1 *.pem
a.client.private.pem
b.client.private.pem
ca-private-cert.pem
Then you'll want a client which uses one of these certificates, and sends some
data:
# tlsclient.py
from twisted.python.filepath import FilePath
from twisted.internet.endpoints import SSL4ClientEndpoint
from twisted.internet.ssl import (
PrivateCertificate, Certificate, optionsForClientTLS)
from twisted.internet.defer import Deferred, inlineCallbacks
from twisted.internet.task import react
from twisted.internet.protocol import Protocol, Factory
class SendAnyData(Protocol):
def connectionMade(self):
self.deferred = Deferred()
self.transport.write(b"HELLO\r\n")
def connectionLost(self, reason):
self.deferred.callback(None)
#inlineCallbacks
def main(reactor, name):
pem = FilePath(name.encode("utf-8") + b".client.private.pem").getContent()
caPem = FilePath(b"ca-private-cert.pem").getContent()
clientEndpoint = SSL4ClientEndpoint(
reactor, u"localhost", 4321,
optionsForClientTLS(u"the-authority", Certificate.loadPEM(caPem),
PrivateCertificate.loadPEM(pem)),
)
proto = yield clientEndpoint.connect(Factory.forProtocol(SendAnyData))
yield proto.deferred
import sys
react(main, sys.argv[1:])
And finally, a server which can distinguish between them:
# whichclient.py
from twisted.python.filepath import FilePath
from twisted.internet.endpoints import SSL4ServerEndpoint
from twisted.internet.ssl import PrivateCertificate, Certificate
from twisted.internet.defer import Deferred
from twisted.internet.task import react
from twisted.internet.protocol import Protocol, Factory
class ReportWhichClient(Protocol):
def dataReceived(self, data):
peerCertificate = Certificate.peerFromTransport(self.transport)
print(peerCertificate.getSubject().commonName.decode('utf-8'))
self.transport.loseConnection()
def main(reactor):
pemBytes = FilePath(b"ca-private-cert.pem").getContent()
certificateAuthority = Certificate.loadPEM(pemBytes)
myCertificate = PrivateCertificate.loadPEM(pemBytes)
serverEndpoint = SSL4ServerEndpoint(
reactor, 4321, myCertificate.options(certificateAuthority)
)
serverEndpoint.listen(Factory.forProtocol(ReportWhichClient))
return Deferred()
react(main, [])
For simplicity's sake we'll just re-use the CA's own certificate for the
server, but in a more realistic scenario you'd obviously want a more
appropriate certificate.
You can now run whichclient.py in one window, then python tlsclient.py a;
python tlsclient.py b in another window, and see whichclient.py print out
a and then b respectively, identifying the clients by the commonName
field in their certificate's subject.
The one caveat here is that you might initially want to put that call to
Certificate.peerFromTransport into a connectionMade method; that won't
work.
Twisted does not presently have a callback for "TLS handshake complete";
hopefully it will eventually, but until it does, you have to wait until you've
received some authenticated data from the peer to be sure the handshake has
completed. For almost all applications, this is fine, since by the time you
have received instructions to do anything (download updates, in your case) the
peer must already have sent the certificate.
Related
I implemented a Twisted SSH server to test a component that uses fabric to run commands on a remote machine via SSH. I have found this example but I don't understand how I have to implement the execCommand method to be compatible with fabric. Here is my implementation of the SSH server:
from pathlib import Path
from twisted.conch import avatar, recvline
from twisted.conch.insults import insults
from twisted.conch.interfaces import ISession
from twisted.conch.ssh import factory, keys, session
from twisted.cred import checkers, portal
from twisted.internet import reactor
from zope.interface import implementer
SSH_KEYS_FOLDER = Path(__file__).parent.parent / "resources" / "ssh_keys"
#implementer(ISession)
class SSHDemoAvatar(avatar.ConchUser):
def __init__(self, username: str):
avatar.ConchUser.__init__(self)
self.username = username
self.channelLookup.update({b"session": session.SSHSession})
def openShell(self, protocol):
pass
def getPty(self, terminal, windowSize, attrs):
return None
def execCommand(self, protocol: session.SSHSessionProcessProtocol, cmd: bytes):
protocol.write("Some text to return")
protocol.session.conn.sendEOF(protocol.session)
def eofReceived(self):
pass
def closed(self):
pass
#implementer(portal.IRealm)
class SSHDemoRealm(object):
def requestAvatar(self, avatarId, _, *interfaces):
return interfaces[0], SSHDemoAvatar(avatarId), lambda: None
def getRSAKeys():
with open(SSH_KEYS_FOLDER / "ssh_key") as private_key_file:
private_key = keys.Key.fromString(data=private_key_file.read())
with open(SSH_KEYS_FOLDER / "ssh_key.pub") as public_key_file:
public_key = keys.Key.fromString(data=public_key_file.read())
return public_key, private_key
if __name__ == "__main__":
sshFactory = factory.SSHFactory()
sshFactory.portal = portal.Portal(SSHDemoRealm())
users = {
"admin": b"aaa",
"guest": b"bbb",
}
sshFactory.portal.registerChecker(checkers.InMemoryUsernamePasswordDatabaseDontUse(**users))
pubKey, privKey = getRSAKeys()
sshFactory.publicKeys = {b"ssh-rsa": pubKey}
sshFactory.privateKeys = {b"ssh-rsa": privKey}
reactor.listenTCP(22222, sshFactory)
reactor.run()
Trying to execute a command via fabric yields the following output:
[paramiko.transport ][INFO ] Connected (version 2.0, client Twisted_22.4.0)
[paramiko.transport ][INFO ] Authentication (password) successful!
Some text to return
This looks promising but the program execution hangs after this line. Do I have to close the connection from the server side after executing the command? How do I implement that properly?
In a traditional UNIX server, it's still the server's responsibility to close the connection if it was told to execute a command. It's up to the server's discretion where and how to do this.
I believe you just want to change protocol.session.conn.sendEOF(protocol.session) to protocol.loseConnection(). I apologize for not testing this myself to be sure, setting up an environment to properly test a full-size conch setup like this is a bit tedious (and the example isn't self-contained, requiring key generation and moduli, etc)
I'm new to python and currently researching its viability to be used as a soap server. I currently have a very rough application that uses the mysql blocking api, but would like to try twisted adbapi. I've successfully used twisted adbapi on regular twisted code using reactors, but can't seem to make it work with code below using ZSI framework. It's not returning anything from mysql. Anyone ever used twisted adbapi with ZSI?
import os
import sys
from dpac_server import *
from ZSI.twisted.wsgi import (SOAPApplication,
soapmethod,
SOAPHandlerChainFactory)
from twisted.enterprise import adbapi
import MySQLdb
def _soapmethod(op):
op_request = GED("http://www.example.org/dpac/", op).pyclass
op_response = GED("http://www.example.org/dpac/", op + "Response").pyclass
return soapmethod(op_request.typecode, op_response.typecode,operation=op, soapaction=op)
class DPACServer(SOAPApplication):
factory = SOAPHandlerChainFactory
#_soapmethod('GetIPOperation')
def soap_GetIPOperation(self, request, response, **kw):
dbpool = adbapi.ConnectionPool("MySQLdb", '127.0.0.1','def_user', 'def_pwd', 'def_db', cp_reconnect=True)
def _dbSPGeneric(txn, cmts):
txn.execute("call def_db.getip(%s)", (cmts, ))
return txn.fetchall()
def dbSPGeneric(cmts):
return dbpool.runInteraction(_dbSPGeneric, cmts)
def returnResults(results):
response.Result = results
def showError(msg):
response.Error = msg
response.Result = ""
response.Error = ""
d = dbSPGeneric(request.Cmts)
d.addCallbacks(returnResults, showError)
return request, response
def main():
from wsgiref.simple_server import make_server
from ZSI.twisted.wsgi import WSGIApplication
application = WSGIApplication()
httpd = make_server('127.0.0.1', 8080, application)
application['dpac'] = DPACServer()
print "listening..."
httpd.serve_forever()
if __name__ == '__main__':
main()
The code you posted creates a new ConnectionPool per (some kind of) request and it never stops the pool. This means you'll eventually run out of resources and you won't be able to service any more requests. "Eventually" is probably after one or two or three requests.
If you never get any responses perhaps this isn't the problem you've encountered. It will be a problem at some point though.
On closer inspection, I wonder if this code even runs the Twisted reactor at all. On first read, I thought you were using some ZSI Twisted integration to run your server. Now I see that you're using wsgiref.simple_server. I am moderately confident that this won't work.
You're already using Twisted, use Twisted's WSGI server instead.
Beyond that, verify that ZSI executes your callbacks in the correct thread. The default for WSGI applications is to run in a non-reactor thread. Twisted APIs are not thread-safe, so if ZSI doesn't do something to correct for this, you'll have bugs introduced by using un-thread-safe APIs in threads.
I have a twisted application which now needs to monitor processes running on several boxes. The way I manually do is 'ssh and ps', now I'd like my twisted application to do. I have 2 options.
Use paramiko or leverage the power of twisted.conch
I really want to use twisted.conch but my research led me to believe that its primarily intended to create SSHServers and SSHClients. However my requirement is a simple remoteExecute(some_cmd)
I was able to figure out how to do this using paramiko but I didnt want to stickparamiko in my twisted app before looking at how to do this using twisted.conch
Code snippets using twisted on how to run remote_cmds using ssh would be highly appreciated. Thanks.
Followup - Happily, the ticket I referenced below is now resolved. The simpler API will be included in the next release of Twisted. The original answer is still a valid way to use Conch and may reveal some interesting details about what's going on, but from Twisted 13.1 and on, if you just want to run a command and handle it's I/O, this simpler interface will work.
It takes an unfortunately large amount of code to execute a command on an SSH using the Conch client APIs. Conch makes you deal with a lot of different layers, even if you just want sensible boring default behavior. However, it's certainly possible. Here's some code which I've been meaning to finish and add to Twisted to simplify this case:
import sys, os
from zope.interface import implements
from twisted.python.failure import Failure
from twisted.python.log import err
from twisted.internet.error import ConnectionDone
from twisted.internet.defer import Deferred, succeed, setDebugging
from twisted.internet.interfaces import IStreamClientEndpoint
from twisted.internet.protocol import Factory, Protocol
from twisted.conch.ssh.common import NS
from twisted.conch.ssh.channel import SSHChannel
from twisted.conch.ssh.transport import SSHClientTransport
from twisted.conch.ssh.connection import SSHConnection
from twisted.conch.client.default import SSHUserAuthClient
from twisted.conch.client.options import ConchOptions
# setDebugging(True)
class _CommandTransport(SSHClientTransport):
_secured = False
def verifyHostKey(self, hostKey, fingerprint):
return succeed(True)
def connectionSecure(self):
self._secured = True
command = _CommandConnection(
self.factory.command,
self.factory.commandProtocolFactory,
self.factory.commandConnected)
userauth = SSHUserAuthClient(
os.environ['USER'], ConchOptions(), command)
self.requestService(userauth)
def connectionLost(self, reason):
if not self._secured:
self.factory.commandConnected.errback(reason)
class _CommandConnection(SSHConnection):
def __init__(self, command, protocolFactory, commandConnected):
SSHConnection.__init__(self)
self._command = command
self._protocolFactory = protocolFactory
self._commandConnected = commandConnected
def serviceStarted(self):
channel = _CommandChannel(
self._command, self._protocolFactory, self._commandConnected)
self.openChannel(channel)
class _CommandChannel(SSHChannel):
name = 'session'
def __init__(self, command, protocolFactory, commandConnected):
SSHChannel.__init__(self)
self._command = command
self._protocolFactory = protocolFactory
self._commandConnected = commandConnected
def openFailed(self, reason):
self._commandConnected.errback(reason)
def channelOpen(self, ignored):
self.conn.sendRequest(self, 'exec', NS(self._command))
self._protocol = self._protocolFactory.buildProtocol(None)
self._protocol.makeConnection(self)
def dataReceived(self, bytes):
self._protocol.dataReceived(bytes)
def closed(self):
self._protocol.connectionLost(
Failure(ConnectionDone("ssh channel closed")))
class SSHCommandClientEndpoint(object):
implements(IStreamClientEndpoint)
def __init__(self, command, sshServer):
self._command = command
self._sshServer = sshServer
def connect(self, protocolFactory):
factory = Factory()
factory.protocol = _CommandTransport
factory.command = self._command
factory.commandProtocolFactory = protocolFactory
factory.commandConnected = Deferred()
d = self._sshServer.connect(factory)
d.addErrback(factory.commandConnected.errback)
return factory.commandConnected
class StdoutEcho(Protocol):
def dataReceived(self, bytes):
sys.stdout.write(bytes)
sys.stdout.flush()
def connectionLost(self, reason):
self.factory.finished.callback(None)
def copyToStdout(endpoint):
echoFactory = Factory()
echoFactory.protocol = StdoutEcho
echoFactory.finished = Deferred()
d = endpoint.connect(echoFactory)
d.addErrback(echoFactory.finished.errback)
return echoFactory.finished
def main():
from twisted.python.log import startLogging
from twisted.internet import reactor
from twisted.internet.endpoints import TCP4ClientEndpoint
# startLogging(sys.stdout)
sshServer = TCP4ClientEndpoint(reactor, "localhost", 22)
commandEndpoint = SSHCommandClientEndpoint("/bin/ls", sshServer)
d = copyToStdout(commandEndpoint)
d.addErrback(err, "ssh command / copy to stdout failed")
d.addCallback(lambda ignored: reactor.stop())
reactor.run()
if __name__ == '__main__':
main()
Some things to note about it:
It uses the new endpoint APIs introduced in Twisted 10.1. It's possible to do this directly on reactor.connectTCP, but I did it as an endpoint to make it more useful; endpoints can be swapped easily without the code that actually asks for a connection knowing.
It does no host key verification at all! _CommandTransport.verifyHostKey is where you would implement that. Take a look at twisted/conch/client/default.py for some hints about what kinds of things you might want to do.
It takes $USER to be the remote username, which you may want to be a parameter.
It probably only works with key authentication. If you want to enable password authentication, you probably need to subclass SSHUserAuthClient and override getPassword to do something.
Almost all of the layers of SSH and Conch are visible here:
_CommandTransport is at the bottom, a plain old protocol that implements the SSH transport protocol. It creates a...
_CommandConnection which implements the SSH connection negotiation parts of the protocol. Once that completes, a...
_CommandChannel is used to talk to a newly opened SSH channel. _CommandChannel does the actual exec to launch your command. Once the channel is opened it creates an instance of...
StdoutEcho, or whatever other protocol you supply. This protocol will get the output from the command you execute, and can write to the command's stdin.
See http://twistedmatrix.com/trac/ticket/4698 for progress in Twisted on supporting this with less code.
I'd like to know why the following doesn't work.
from twisted internet import defer, reactor
from twisted.python.failure import Failure
import twisted.names.client
def do_lookup(do_lookup):
d = twisted.names.client.getHostByName(domain)
d.addBoth(lookup_done)
def lookup_done(result):
print 'result:', result
reactor.stop()
domain = 'twistedmatrix.com'
reactor.callLater(0, do_lookup, domain)
reactor.run()
Results in:
result: [Failure instance: Traceback
(failure with no frames): <class
'twisted.names.error.ResolverError'>:
Stuck at response without answers or
delegation ]
As of this writing, this fails on Windows, since it uses an invalid path for the windows host file (in twisted.names.client.createResolver. It uses 'c:\windows\hosts'. This was fine for windows versions 98 and Me (reference here), but would fail with a version as "modern" as XP.
Today, it should probably use something like:
hosts = os.path.join(
os.environ.get('systemroot','C:\\Windows'),
r'system32\drivers\etc\hosts'
)
I think this only partly resolves the issue though (or maybe this is a red herring).
This will only work now for names actually specified in this hosts file. What it likely needs to do is some sort of registry query for the DNS server, then query that for the actual DNS name lookup.
This recipe looks promising for getting the actual DNS server.
Correcting your example to the following, so that it is syntactically valid:
from twisted.internet import reactor
import twisted.names.client
def do_lookup(domain):
d = twisted.names.client.getHostByName(domain)
d.addBoth(lookup_done)
def lookup_done(result):
print 'result:', result
reactor.stop()
domain = 'twistedmatrix.com'
reactor.callLater(0, do_lookup, domain)
reactor.run()
I get:
$ python so-example.py
result: 66.35.39.65
So, to answer your question: your local DNS environment is broken, not twisted.names. Or maybe there's a bug. You'll need to track it down further.
I did some digging why an explicit client.createResolver(servers) wasn't working on our corporate windows machines. In the guts of Twisted's createResolver is a os-dependant branch:
def createResolver(servers=None, resolvconf=None, hosts=None):
...
from twisted.names import resolve, cache, root, hosts as hostsModule
if platform.getType() == 'posix':
if resolvconf is None:
resolvconf = '/etc/resolv.conf'
if hosts is None:
hosts = '/etc/hosts'
theResolver = Resolver(resolvconf, servers)
hostResolver = hostsModule.Resolver(hosts)
else:
if hosts is None:
hosts = r'c:\windows\hosts'
from twisted.internet import reactor
bootstrap = _ThreadedResolverImpl(reactor)
hostResolver = hostsModule.Resolver(hosts)
theResolver = root.bootstrap(bootstrap)
L = [hostResolver, cache.CacheResolver(), theResolver]
return resolve.ResolverChain(L)
The first warning sign for Windows is that argument servers is not used, so custom-DNS servers are ignored. Next is that the _ThreadedResolverImpl(), which uses platform specific code, is wrapped in a root.bootstrap() before being added to the resolver chain.
The purpose of root.bootstrap is to use the platform resolver to lookup a.root-servers.net, b.root-servers.net, etc. using the synchronous Windows platform resolver (which works - and returns IPs), and then do direct DNS queries against the root servers. The UDP packets fired off to root servers are then blocked by our corporate firewall - I see them in Wireshark.
The default getResolver() call used by names.client.getHostByName() invokes createResolver() directly, which may again result in this broken setup bootstrapped by a working DNS setup. For most environments I think adding the Resolver (with root or explicit servers) into the ResolverChain along with _ThreadedResolverImpl as a fallback would work - except that the platform resolver is a different interface.
Here's an example of detecting the local DNS servers asynchronously (by parsing the output of ipconfig) then installing a custom resolver to use them.
import sys
from twisted.python import log
from twisted.names import root, hosts, resolve, cache, client
from twisted.python.runtime import platform
from twisted.internet import reactor
from twisted.internet import utils
import re
def parseIpconfigDNSServers(output):
servers = []
found = False
for line in output.split('\n'):
if 'DNS Servers . . . . . . . . . . . :' in line or (found and not '. . . .' in line):
servers.append(line[38:].strip())
found = True
else:
found = False
log.msg( 'Windows: Detected DNS servers %s' % (str(servers)))
return servers
if platform.getType() != 'posix':
d = utils.getProcessOutput(os.path.join(os.environ['WINDIR'], 'system32', 'ipconfig.exe'), ["/all"])
d.addCallback(parseIpconfigDNSServers)
d.addCallback(lambda r: client.Resolver(servers=[(h, 53) for h in r]))
d.addErrback(log.msg)
theResolver = root.DeferredResolver(d)
client.theResolver = resolve.ResolverChain([cache.CacheResolver(), theResolver])
if __name__ == '__main__':
log.startLogging(sys.stdout)
def do_lookup(domain):
d = client.getHostByName(domain)
d.addBoth(log.msg)
from twisted.internet import reactor
reactor.callLater(0, do_lookup, 'example.com')
reactor.run()
I've tidied this up and posted it here https://gist.github.com/shuckc/af7490e1c4a2652ca740
I need to write a script that connects to a bunch of sites on our corporate intranet over HTTPS and verifies that their SSL certificates are valid; that they are not expired, that they are issued for the correct address, etc. We use our own internal corporate Certificate Authority for these sites, so we have the public key of the CA to verify the certificates against.
Python by default just accepts and uses SSL certificates when using HTTPS, so even if a certificate is invalid, Python libraries such as urllib2 and Twisted will just happily use the certificate.
How do I verify a certificate in Python?
I have added a distribution to the Python Package Index which makes the match_hostname() function from the Python 3.2 ssl package available on previous versions of Python.
http://pypi.python.org/pypi/backports.ssl_match_hostname/
You can install it with:
pip install backports.ssl_match_hostname
Or you can make it a dependency listed in your project's setup.py. Either way, it can be used like this:
from backports.ssl_match_hostname import match_hostname, CertificateError
...
sslsock = ssl.wrap_socket(sock, ssl_version=ssl.PROTOCOL_SSLv3,
cert_reqs=ssl.CERT_REQUIRED, ca_certs=...)
try:
match_hostname(sslsock.getpeercert(), hostname)
except CertificateError, ce:
...
You can use Twisted to verify certificates. The main API is CertificateOptions, which can be provided as the contextFactory argument to various functions such as listenSSL and startTLS.
Unfortunately, neither Python nor Twisted comes with a the pile of CA certificates required to actually do HTTPS validation, nor the HTTPS validation logic. Due to a limitation in PyOpenSSL, you can't do it completely correctly just yet, but thanks to the fact that almost all certificates include a subject commonName, you can get close enough.
Here is a naive sample implementation of a verifying Twisted HTTPS client which ignores wildcards and subjectAltName extensions, and uses the certificate-authority certificates present in the 'ca-certificates' package in most Ubuntu distributions. Try it with your favorite valid and invalid certificate sites :).
import os
import glob
from OpenSSL.SSL import Context, TLSv1_METHOD, VERIFY_PEER, VERIFY_FAIL_IF_NO_PEER_CERT, OP_NO_SSLv2
from OpenSSL.crypto import load_certificate, FILETYPE_PEM
from twisted.python.urlpath import URLPath
from twisted.internet.ssl import ContextFactory
from twisted.internet import reactor
from twisted.web.client import getPage
certificateAuthorityMap = {}
for certFileName in glob.glob("/etc/ssl/certs/*.pem"):
# There might be some dead symlinks in there, so let's make sure it's real.
if os.path.exists(certFileName):
data = open(certFileName).read()
x509 = load_certificate(FILETYPE_PEM, data)
digest = x509.digest('sha1')
# Now, de-duplicate in case the same cert has multiple names.
certificateAuthorityMap[digest] = x509
class HTTPSVerifyingContextFactory(ContextFactory):
def __init__(self, hostname):
self.hostname = hostname
isClient = True
def getContext(self):
ctx = Context(TLSv1_METHOD)
store = ctx.get_cert_store()
for value in certificateAuthorityMap.values():
store.add_cert(value)
ctx.set_verify(VERIFY_PEER | VERIFY_FAIL_IF_NO_PEER_CERT, self.verifyHostname)
ctx.set_options(OP_NO_SSLv2)
return ctx
def verifyHostname(self, connection, x509, errno, depth, preverifyOK):
if preverifyOK:
if self.hostname != x509.get_subject().commonName:
return False
return preverifyOK
def secureGet(url):
return getPage(url, HTTPSVerifyingContextFactory(URLPath.fromString(url).netloc))
def done(result):
print 'Done!', len(result)
secureGet("https://google.com/").addCallback(done)
reactor.run()
PycURL does this beautifully.
Below is a short example. It will throw a pycurl.error if something is fishy, where you get a tuple with error code and a human readable message.
import pycurl
curl = pycurl.Curl()
curl.setopt(pycurl.CAINFO, "myFineCA.crt")
curl.setopt(pycurl.SSL_VERIFYPEER, 1)
curl.setopt(pycurl.SSL_VERIFYHOST, 2)
curl.setopt(pycurl.URL, "https://internal.stuff/")
curl.perform()
You will probably want to configure more options, like where to store the results, etc. But no need to clutter the example with non-essentials.
Example of what exceptions might be raised:
(60, 'Peer certificate cannot be authenticated with known CA certificates')
(51, "common name 'CN=something.else.stuff,O=Example Corp,C=SE' does not match 'internal.stuff'")
Some links that I found useful are the libcurl-docs for setopt and getinfo.
http://curl.haxx.se/libcurl/c/curl_easy_setopt.html
http://curl.haxx.se/libcurl/c/curl_easy_getinfo.html
From release version 2.7.9/3.4.3 on, Python by default attempts to perform certificate validation.
This has been proposed in PEP 467, which is worth a read: https://www.python.org/dev/peps/pep-0476/
The changes affect all relevant stdlib modules (urllib/urllib2, http, httplib).
Relevant documentation:
https://docs.python.org/2/library/httplib.html#httplib.HTTPSConnection
This class now performs all the necessary certificate and hostname checks by default. To revert to the previous, unverified, behavior ssl._create_unverified_context() can be passed to the context parameter.
https://docs.python.org/3/library/http.client.html#http.client.HTTPSConnection
Changed in version 3.4.3: This class now performs all the necessary certificate and hostname checks by default. To revert to the previous, unverified, behavior ssl._create_unverified_context() can be passed to the context parameter.
Note that the new built-in verification is based on the system-provided certificate database. Opposed to that, the requests package ships its own certificate bundle. Pros and cons of both approaches are discussed in the Trust database section of PEP 476.
Or simply make your life easier by using the requests library:
import requests
requests.get('https://somesite.com', cert='/path/server.crt', verify=True)
A few more words about its usage.
Here's an example script which demonstrates certificate validation:
import httplib
import re
import socket
import sys
import urllib2
import ssl
class InvalidCertificateException(httplib.HTTPException, urllib2.URLError):
def __init__(self, host, cert, reason):
httplib.HTTPException.__init__(self)
self.host = host
self.cert = cert
self.reason = reason
def __str__(self):
return ('Host %s returned an invalid certificate (%s) %s\n' %
(self.host, self.reason, self.cert))
class CertValidatingHTTPSConnection(httplib.HTTPConnection):
default_port = httplib.HTTPS_PORT
def __init__(self, host, port=None, key_file=None, cert_file=None,
ca_certs=None, strict=None, **kwargs):
httplib.HTTPConnection.__init__(self, host, port, strict, **kwargs)
self.key_file = key_file
self.cert_file = cert_file
self.ca_certs = ca_certs
if self.ca_certs:
self.cert_reqs = ssl.CERT_REQUIRED
else:
self.cert_reqs = ssl.CERT_NONE
def _GetValidHostsForCert(self, cert):
if 'subjectAltName' in cert:
return [x[1] for x in cert['subjectAltName']
if x[0].lower() == 'dns']
else:
return [x[0][1] for x in cert['subject']
if x[0][0].lower() == 'commonname']
def _ValidateCertificateHostname(self, cert, hostname):
hosts = self._GetValidHostsForCert(cert)
for host in hosts:
host_re = host.replace('.', '\.').replace('*', '[^.]*')
if re.search('^%s$' % (host_re,), hostname, re.I):
return True
return False
def connect(self):
sock = socket.create_connection((self.host, self.port))
self.sock = ssl.wrap_socket(sock, keyfile=self.key_file,
certfile=self.cert_file,
cert_reqs=self.cert_reqs,
ca_certs=self.ca_certs)
if self.cert_reqs & ssl.CERT_REQUIRED:
cert = self.sock.getpeercert()
hostname = self.host.split(':', 0)[0]
if not self._ValidateCertificateHostname(cert, hostname):
raise InvalidCertificateException(hostname, cert,
'hostname mismatch')
class VerifiedHTTPSHandler(urllib2.HTTPSHandler):
def __init__(self, **kwargs):
urllib2.AbstractHTTPHandler.__init__(self)
self._connection_args = kwargs
def https_open(self, req):
def http_class_wrapper(host, **kwargs):
full_kwargs = dict(self._connection_args)
full_kwargs.update(kwargs)
return CertValidatingHTTPSConnection(host, **full_kwargs)
try:
return self.do_open(http_class_wrapper, req)
except urllib2.URLError, e:
if type(e.reason) == ssl.SSLError and e.reason.args[0] == 1:
raise InvalidCertificateException(req.host, '',
e.reason.args[1])
raise
https_request = urllib2.HTTPSHandler.do_request_
if __name__ == "__main__":
if len(sys.argv) != 3:
print "usage: python %s CA_CERT URL" % sys.argv[0]
exit(2)
handler = VerifiedHTTPSHandler(ca_certs = sys.argv[1])
opener = urllib2.build_opener(handler)
print opener.open(sys.argv[2]).read()
M2Crypto can do the validation. You can also use M2Crypto with Twisted if you like. The Chandler desktop client uses Twisted for networking and M2Crypto for SSL, including certificate validation.
Based on Glyphs comment it seems like M2Crypto does better certificate verification by default than what you can do with pyOpenSSL currently, because M2Crypto checks subjectAltName field too.
I've also blogged on how to get the certificates Mozilla Firefox ships with in Python and usable with Python SSL solutions.
Jython DOES carry out certificate verification by default, so using standard library modules, e.g. httplib.HTTPSConnection, etc, with jython will verify certificates and give exceptions for failures, i.e. mismatched identities, expired certs, etc.
In fact, you have to do some extra work to get jython to behave like cpython, i.e. to get jython to NOT verify certs.
I have written a blog post on how to disable certificate checking on jython, because it can be useful in testing phases, etc.
Installing an all-trusting security provider on java and jython.
http://jython.xhaus.com/installing-an-all-trusting-security-provider-on-java-and-jython/
The following code allows you to benefit from all SSL validation checks (e.g. date validity, CA certificate chain ...) EXCEPT a pluggable verification step e.g. to verify the hostname or do other additional certificate verification steps.
from httplib import HTTPSConnection
import ssl
def create_custom_HTTPSConnection(host):
def verify_cert(cert, host):
# Write your code here
# You can certainly base yourself on ssl.match_hostname
# Raise ssl.CertificateError if verification fails
print 'Host:', host
print 'Peer cert:', cert
class CustomHTTPSConnection(HTTPSConnection, object):
def connect(self):
super(CustomHTTPSConnection, self).connect()
cert = self.sock.getpeercert()
verify_cert(cert, host)
context = ssl.create_default_context()
context.check_hostname = False
return CustomHTTPSConnection(host=host, context=context)
if __name__ == '__main__':
# try expired.badssl.com or self-signed.badssl.com !
conn = create_custom_HTTPSConnection('badssl.com')
conn.request('GET', '/')
conn.getresponse().read()
pyOpenSSL is an interface to the OpenSSL library. It should provide everything you need.
I was having the same problem but wanted to minimize 3rd party dependencies (because this one-off script was to be executed by many users). My solution was to wrap a curl call and make sure that the exit code was 0. Worked like a charm.