Develop Reference

Mock DNS server using Twisted

I'm trying to write a Twisted-based mock DNS server to do some tests.
Taking inspiration from this guide I wrote a very simple server that just resolves everything to 127.0.0.1:
from twisted.internet import defer, reactor
from twisted.names import dns, error, server
class MockDNSResolver:
def _doDynamicResponse(self, query):
name = query.name.name
record = dns.Record_A(address=b"127.0.0.1")
answer = dns.RRHeader(name=name, payload=record)
authority = []
additional = []
return [answer], authority, additional
def query(self, query, timeout=None):
print("Incoming query for:", query.name)
if query.type == dns.A:
return defer.succeed(self._doDynamicResponse(query))
else:
return defer.fail(error.DomainError())
if __name__ == "__main__":
clients = [MockDNSResolver()]
factory = server.DNSServerFactory(clients=clients)
protocol = dns.DNSDatagramProtocol(controller=factory)
reactor.listenUDP(10053, protocol)
reactor.listenTCP(10053, factory)
reactor.run()
The above is working just fine with dig and nslookup (from a different terminal):
$ dig -p 10053 #localhost something.example.org A +short
127.0.0.1
$ nslookup something.else.example.org 127.0.0.1 -port=10053
Server: 127.0.0.1
Address: 127.0.0.1#10053
Non-authoritative answer:
Name: something.else.example.org
Address: 127.0.0.1
I'm also getting the corresponding hits on the terminal that's running the server:
Incoming query for: something.example.org
Incoming query for: something.else.example.org
Then, I wrote the following piece of code, based on this section about making requests and this section about installing a custom resolver:
from twisted.internet import reactor
from twisted.names.client import createResolver
from twisted.web.client import Agent
d = Agent(reactor).request(b'GET', b'http://does.not.exist')
reactor.installResolver(createResolver(servers=[('127.0.0.1', 10053)]))
def callback(result):
print('Result:', result)
d.addBoth(callback)
d.addBoth(lambda _: reactor.stop())
reactor.run()
But this fails (and I get no lines in the server terminal). It appears as if the queries are not going to the mock server, but to the system-defined server:
Result: [Failure instance: Traceback: <class 'twisted.internet.error.DNSLookupError'>: DNS lookup failed: no results for hostname lookup: does.not.exist.
/.../venv/lib/python3.6/site-packages/twisted/internet/_resolver.py:137:deliverResults
/.../venv/lib/python3.6/site-packages/twisted/internet/endpoints.py:921:resolutionComplete
/.../venv/lib/python3.6/site-packages/twisted/internet/defer.py:460:callback
/.../venv/lib/python3.6/site-packages/twisted/internet/defer.py:568:_startRunCallbacks
--- <exception caught here> ---
/.../venv/lib/python3.6/site-packages/twisted/internet/defer.py:654:_runCallbacks
/.../venv/lib/python3.6/site-packages/twisted/internet/endpoints.py:975:startConnectionAttempts
]
I'm using:
macOS 10.14.6 Python 3.6.6, Twisted 18.9.0
Linux Mint 19.1, Python 3.6.9, Twisted 19.7.0
I appreciate any help, please let me know if additional information is required.
Thanks!

The solution was:
(Client) swap the order of the lines that install the resolver and create the deferred for the request, as suggested by #Hadus. I thought this didn't matter, since the reactor was not running yet.
(Server) implement lookupAllRecords, reusing the existing _doDynamicResponse method.
# server
from twisted.internet import defer, reactor
from twisted.names import dns, error, server
class MockDNSResolver:
"""
Implements twisted.internet.interfaces.IResolver partially
"""
def _doDynamicResponse(self, name):
print("Resolving name:", name)
record = dns.Record_A(address=b"127.0.0.1")
answer = dns.RRHeader(name=name, payload=record)
return [answer], [], []
def query(self, query, timeout=None):
if query.type == dns.A:
return defer.succeed(self._doDynamicResponse(query.name.name))
return defer.fail(error.DomainError())
def lookupAllRecords(self, name, timeout=None):
return defer.succeed(self._doDynamicResponse(name))
if __name__ == "__main__":
clients = [MockDNSResolver()]
factory = server.DNSServerFactory(clients=clients)
protocol = dns.DNSDatagramProtocol(controller=factory)
reactor.listenUDP(10053, protocol)
reactor.listenTCP(10053, factory)
reactor.run()
# client
from twisted.internet import reactor
from twisted.names.client import createResolver
from twisted.web.client import Agent
reactor.installResolver(createResolver(servers=[('127.0.0.1', 10053)]))
d = Agent(reactor).request(b'GET', b'http://does.not.exist:8000')
def callback(result):
print('Result:', result)
d.addBoth(callback)
d.addBoth(lambda _: reactor.stop())
reactor.run()
$ python client.py
Result: <twisted.web._newclient.Response object at 0x101077f98>
(I'm running a simple web server with python3 -m http.server in another terminal, otherwise I get a reasonable twisted.internet.error.ConnectionRefusedError exception).

This is how you should make a request to it:
from twisted.internet import reactor
from twisted.names import client
resolver = client.createResolver(servers=[('127.0.0.1', 10053)])
d = resolver.getHostByName('does.not.exist')
def callback(result):
print('Result:', result)
d.addBoth(callback)
reactor.callLater(4, reactor.stop)
reactor.run()
And the MockDNSResolvershould be a subclass of twisted.names.common.ResolverBase and then you only have to implement _lookup.
class MockDNSResolver(common.ResolverBase):
def _lookup(self, name, cls, type, timeout):
print("Incoming query for:", name)
record = dns.Record_A(address=b"127.0.0.5")
answer = dns.RRHeader(name=name, payload=record)
return defer.succeed(([answer], [], []))
Full code: https://pastebin.com/MfGahtAV

Use TLS and Python for authentication

I want to make a little update script for a software that runs on a Raspberry Pi and works like a local server. That should connect to a master server in the web to get software updates and also to verify the license of the software.
For that I set up two python scripts. I want these to connect via a TLS socket. Then the client checks the server certificate and the server checks if it's one of the authorized clients. I found a solution for this using twisted on this page.
Now there is a problem left. I want to know which client (depending on the certificate) is establishing the connection. Is there a way to do this in Python 3 with twisted?
I'm happy with every answer.

In a word: yes, this is quite possible, and all the necessary stuff is
ported to python 3 - I tested all the following under Python 3.4 on my Mac and it seems to
work fine.
The short answer is
"use twisted.internet.ssl.Certificate.peerFromTransport"
but given that a lot of set-up is required to get to the point where that is
possible, I've constructed a fully working example that you should be able to
try out and build upon.
For posterity, you'll first need to generate a few client certificates all
signed by the same CA. You've probably already done this, but so others can
understand the answer and try it out on their own (and so I could test my
answer myself ;-)), they'll need some code like this:
# newcert.py
from twisted.python.filepath import FilePath
from twisted.internet.ssl import PrivateCertificate, KeyPair, DN
def getCAPrivateCert():
privatePath = FilePath(b"ca-private-cert.pem")
if privatePath.exists():
return PrivateCertificate.loadPEM(privatePath.getContent())
else:
caKey = KeyPair.generate(size=4096)
caCert = caKey.selfSignedCert(1, CN="the-authority")
privatePath.setContent(caCert.dumpPEM())
return caCert
def clientCertFor(name):
signingCert = getCAPrivateCert()
clientKey = KeyPair.generate(size=4096)
csr = clientKey.requestObject(DN(CN=name), "sha1")
clientCert = signingCert.signRequestObject(
csr, serialNumber=1, digestAlgorithm="sha1")
return PrivateCertificate.fromCertificateAndKeyPair(clientCert, clientKey)
if __name__ == '__main__':
import sys
name = sys.argv[1]
pem = clientCertFor(name.encode("utf-8")).dumpPEM()
FilePath(name.encode("utf-8") + b".client.private.pem").setContent(pem)
With this program, you can create a few certificates like so:
$ python newcert.py a
$ python newcert.py b
Now you should have a few files you can use:
$ ls -1 *.pem
a.client.private.pem
b.client.private.pem
ca-private-cert.pem
Then you'll want a client which uses one of these certificates, and sends some
data:
# tlsclient.py
from twisted.python.filepath import FilePath
from twisted.internet.endpoints import SSL4ClientEndpoint
from twisted.internet.ssl import (
PrivateCertificate, Certificate, optionsForClientTLS)
from twisted.internet.defer import Deferred, inlineCallbacks
from twisted.internet.task import react
from twisted.internet.protocol import Protocol, Factory
class SendAnyData(Protocol):
def connectionMade(self):
self.deferred = Deferred()
self.transport.write(b"HELLO\r\n")
def connectionLost(self, reason):
self.deferred.callback(None)
#inlineCallbacks
def main(reactor, name):
pem = FilePath(name.encode("utf-8") + b".client.private.pem").getContent()
caPem = FilePath(b"ca-private-cert.pem").getContent()
clientEndpoint = SSL4ClientEndpoint(
reactor, u"localhost", 4321,
optionsForClientTLS(u"the-authority", Certificate.loadPEM(caPem),
PrivateCertificate.loadPEM(pem)),
)
proto = yield clientEndpoint.connect(Factory.forProtocol(SendAnyData))
yield proto.deferred
import sys
react(main, sys.argv[1:])
And finally, a server which can distinguish between them:
# whichclient.py
from twisted.python.filepath import FilePath
from twisted.internet.endpoints import SSL4ServerEndpoint
from twisted.internet.ssl import PrivateCertificate, Certificate
from twisted.internet.defer import Deferred
from twisted.internet.task import react
from twisted.internet.protocol import Protocol, Factory
class ReportWhichClient(Protocol):
def dataReceived(self, data):
peerCertificate = Certificate.peerFromTransport(self.transport)
print(peerCertificate.getSubject().commonName.decode('utf-8'))
self.transport.loseConnection()
def main(reactor):
pemBytes = FilePath(b"ca-private-cert.pem").getContent()
certificateAuthority = Certificate.loadPEM(pemBytes)
myCertificate = PrivateCertificate.loadPEM(pemBytes)
serverEndpoint = SSL4ServerEndpoint(
reactor, 4321, myCertificate.options(certificateAuthority)
)
serverEndpoint.listen(Factory.forProtocol(ReportWhichClient))
return Deferred()
react(main, [])
For simplicity's sake we'll just re-use the CA's own certificate for the
server, but in a more realistic scenario you'd obviously want a more
appropriate certificate.
You can now run whichclient.py in one window, then python tlsclient.py a;
python tlsclient.py b in another window, and see whichclient.py print out
a and then b respectively, identifying the clients by the commonName
field in their certificate's subject.
The one caveat here is that you might initially want to put that call to
Certificate.peerFromTransport into a connectionMade method; that won't
work.
Twisted does not presently have a callback for "TLS handshake complete";
hopefully it will eventually, but until it does, you have to wait until you've
received some authenticated data from the peer to be sure the handshake has
completed. For almost all applications, this is fine, since by the time you
have received instructions to do anything (download updates, in your case) the
peer must already have sent the certificate.

twisted python daemonization and port bindings

I am using the following script from the Twisted tutorial (with slight modification):
from twisted.application import internet, service
from twisted.internet import reactor, protocol, defer
from twisted.protocols import basic
from twisted.web import client
class FingerProtocol(basic.LineReceiver):
def lineReceived(self, user):
d = self.factory.getUser(user)
def onError(err):
return "Internal server error"
d.addErrback(onError)
def writeResponse(message):
self.transport.write(message + "\r\n")
self.transport.loseConnection()
d.addCallback(writeResponse)
class FingerFactory(protocol.ServerFactory):
protocol = FingerProtocol
def __init__(self, prefix):
self.prefix = prefix
def getUser(self, user):
return client.getPage(self.prefix + user)
application = service.Application('finger', uid=1, gid=1)
factory = FingerFactory(prefix="http://livejournal.com/~")
internet.TCPServer(7979, factory).setServiceParent(
service.IServiceCollection(application))
which I save as finger_daemon.tac and run with
twistd -y finger_daemon.tac \
-l /home/me/twisted/finger.log \
--pidfile=/home/me/twisted/finger.pid
but of course it won't bind to 79, since it's a privileged port. I tried also running with sudo, no difference there.
I then tried changing the TCPServer port to 7979 and then connecting to the daemon once running with
telnet 127.0.0.1 7979
and I get Connection Refused error. What's going on here specifically? How is daemonizing supposed to work in Twisted?

When I run this code, I see the following log message:
2013-10-02 23:50:34-0700 [-] failed to set uid/gid 1/1 (are you root?) -- exiting.
and then twistd exits. So you'd need to do sudo twistd and then that adds a whole bunch of python path management problems...
Why are you setting the uid and gid arguments? You're trying to run it as the daemon user? You don't need to do that in order to daemonize. Just removing the uid=1, gid=1 arguments to Application makes it work for me.

Python CGIHTTPServer Default Directories

I've got the following minimal code for a CGI-handling HTTP server, derived from several examples on the inner-tubes:
#!/usr/bin/env python
import BaseHTTPServer
import CGIHTTPServer
import cgitb;
cgitb.enable() # Error reporting
server = BaseHTTPServer.HTTPServer
handler = CGIHTTPServer.CGIHTTPRequestHandler
server_address = ("", 8000)
handler.cgi_directories = [""]
httpd = server(server_address, handler)
httpd.serve_forever()
Yet, when I execute the script and try to run a test script in the same directory via CGI using http://localhost:8000/test.py, I see the text of the script rather than the results of the execution.
Permissions are all set correctly, and the test script itself is not the problem (as I can run it fine using python -m CGIHTTPServer, when the script resides in cgi-bin). I suspect the problem has something to do with the default CGI directories.
How can I get the script to execute?

My suspicions were correct. The examples from which this code is derived showed the wrong way to set the default directory to be the same directory in which the server script resides. To set the default directory in this way, use:
handler.cgi_directories = ["/"]
Caution: This opens up potentially huge security holes if you're not behind any kind of a firewall. This is only an instructive example. Use only with extreme care.

The solution doesn't seem to work (at least for me) if the .cgi_directories requires multiple layers of subdirectories ( ['/db/cgi-bin'] for instance). Subclassing the server and changing the is_cgi def seemed to work. Here's what I added/substituted in your script:
from CGIHTTPServer import _url_collapse_path
class MyCGIHTTPServer(CGIHTTPServer.CGIHTTPRequestHandler):
def is_cgi(self):
collapsed_path = _url_collapse_path(self.path)
for path in self.cgi_directories:
if path in collapsed_path:
dir_sep_index = collapsed_path.rfind(path) + len(path)
head, tail = collapsed_path[:dir_sep_index], collapsed_path[dir_sep_index + 1:]
self.cgi_info = head, tail
return True
return False
server = BaseHTTPServer.HTTPServer
handler = MyCGIHTTPServer

Here is how you would make every .py file on the server a cgi file (you probably don't want that for production/a public server ;):
import BaseHTTPServer
import CGIHTTPServer
import cgitb; cgitb.enable()
server = BaseHTTPServer.HTTPServer
# Treat everything as a cgi file, i.e.
# `handler.cgi_directories = ["*"]` but that is not defined, so we need
class Handler(CGIHTTPServer.CGIHTTPRequestHandler):
def is_cgi(self):
self.cgi_info = '', self.path[1:]
return True
httpd = server(("", 9006), Handler)
httpd.serve_forever()

Best way to run remote commands thru ssh in Twisted?

I have a twisted application which now needs to monitor processes running on several boxes. The way I manually do is 'ssh and ps', now I'd like my twisted application to do. I have 2 options.
Use paramiko or leverage the power of twisted.conch
I really want to use twisted.conch but my research led me to believe that its primarily intended to create SSHServers and SSHClients. However my requirement is a simple remoteExecute(some_cmd)
I was able to figure out how to do this using paramiko but I didnt want to stickparamiko in my twisted app before looking at how to do this using twisted.conch
Code snippets using twisted on how to run remote_cmds using ssh would be highly appreciated. Thanks.

Followup - Happily, the ticket I referenced below is now resolved. The simpler API will be included in the next release of Twisted. The original answer is still a valid way to use Conch and may reveal some interesting details about what's going on, but from Twisted 13.1 and on, if you just want to run a command and handle it's I/O, this simpler interface will work.
It takes an unfortunately large amount of code to execute a command on an SSH using the Conch client APIs. Conch makes you deal with a lot of different layers, even if you just want sensible boring default behavior. However, it's certainly possible. Here's some code which I've been meaning to finish and add to Twisted to simplify this case:
import sys, os
from zope.interface import implements
from twisted.python.failure import Failure
from twisted.python.log import err
from twisted.internet.error import ConnectionDone
from twisted.internet.defer import Deferred, succeed, setDebugging
from twisted.internet.interfaces import IStreamClientEndpoint
from twisted.internet.protocol import Factory, Protocol
from twisted.conch.ssh.common import NS
from twisted.conch.ssh.channel import SSHChannel
from twisted.conch.ssh.transport import SSHClientTransport
from twisted.conch.ssh.connection import SSHConnection
from twisted.conch.client.default import SSHUserAuthClient
from twisted.conch.client.options import ConchOptions
# setDebugging(True)
class _CommandTransport(SSHClientTransport):
_secured = False
def verifyHostKey(self, hostKey, fingerprint):
return succeed(True)
def connectionSecure(self):
self._secured = True
command = _CommandConnection(
self.factory.command,
self.factory.commandProtocolFactory,
self.factory.commandConnected)
userauth = SSHUserAuthClient(
os.environ['USER'], ConchOptions(), command)
self.requestService(userauth)
def connectionLost(self, reason):
if not self._secured:
self.factory.commandConnected.errback(reason)
class _CommandConnection(SSHConnection):
def __init__(self, command, protocolFactory, commandConnected):
SSHConnection.__init__(self)
self._command = command
self._protocolFactory = protocolFactory
self._commandConnected = commandConnected
def serviceStarted(self):
channel = _CommandChannel(
self._command, self._protocolFactory, self._commandConnected)
self.openChannel(channel)
class _CommandChannel(SSHChannel):
name = 'session'
def __init__(self, command, protocolFactory, commandConnected):
SSHChannel.__init__(self)
self._command = command
self._protocolFactory = protocolFactory
self._commandConnected = commandConnected
def openFailed(self, reason):
self._commandConnected.errback(reason)
def channelOpen(self, ignored):
self.conn.sendRequest(self, 'exec', NS(self._command))
self._protocol = self._protocolFactory.buildProtocol(None)
self._protocol.makeConnection(self)
def dataReceived(self, bytes):
self._protocol.dataReceived(bytes)
def closed(self):
self._protocol.connectionLost(
Failure(ConnectionDone("ssh channel closed")))
class SSHCommandClientEndpoint(object):
implements(IStreamClientEndpoint)
def __init__(self, command, sshServer):
self._command = command
self._sshServer = sshServer
def connect(self, protocolFactory):
factory = Factory()
factory.protocol = _CommandTransport
factory.command = self._command
factory.commandProtocolFactory = protocolFactory
factory.commandConnected = Deferred()
d = self._sshServer.connect(factory)
d.addErrback(factory.commandConnected.errback)
return factory.commandConnected
class StdoutEcho(Protocol):
def dataReceived(self, bytes):
sys.stdout.write(bytes)
sys.stdout.flush()
def connectionLost(self, reason):
self.factory.finished.callback(None)
def copyToStdout(endpoint):
echoFactory = Factory()
echoFactory.protocol = StdoutEcho
echoFactory.finished = Deferred()
d = endpoint.connect(echoFactory)
d.addErrback(echoFactory.finished.errback)
return echoFactory.finished
def main():
from twisted.python.log import startLogging
from twisted.internet import reactor
from twisted.internet.endpoints import TCP4ClientEndpoint
# startLogging(sys.stdout)
sshServer = TCP4ClientEndpoint(reactor, "localhost", 22)
commandEndpoint = SSHCommandClientEndpoint("/bin/ls", sshServer)
d = copyToStdout(commandEndpoint)
d.addErrback(err, "ssh command / copy to stdout failed")
d.addCallback(lambda ignored: reactor.stop())
reactor.run()
if __name__ == '__main__':
main()
Some things to note about it:
It uses the new endpoint APIs introduced in Twisted 10.1. It's possible to do this directly on reactor.connectTCP, but I did it as an endpoint to make it more useful; endpoints can be swapped easily without the code that actually asks for a connection knowing.
It does no host key verification at all! _CommandTransport.verifyHostKey is where you would implement that. Take a look at twisted/conch/client/default.py for some hints about what kinds of things you might want to do.
It takes $USER to be the remote username, which you may want to be a parameter.
It probably only works with key authentication. If you want to enable password authentication, you probably need to subclass SSHUserAuthClient and override getPassword to do something.
Almost all of the layers of SSH and Conch are visible here:
_CommandTransport is at the bottom, a plain old protocol that implements the SSH transport protocol. It creates a...
_CommandConnection which implements the SSH connection negotiation parts of the protocol. Once that completes, a...
_CommandChannel is used to talk to a newly opened SSH channel. _CommandChannel does the actual exec to launch your command. Once the channel is opened it creates an instance of...
StdoutEcho, or whatever other protocol you supply. This protocol will get the output from the command you execute, and can write to the command's stdin.
See http://twistedmatrix.com/trac/ticket/4698 for progress in Twisted on supporting this with less code.