I am using the following script from the Twisted tutorial (with slight modification):
from twisted.application import internet, service
from twisted.internet import reactor, protocol, defer
from twisted.protocols import basic
from twisted.web import client
class FingerProtocol(basic.LineReceiver):
def lineReceived(self, user):
d = self.factory.getUser(user)
def onError(err):
return "Internal server error"
d.addErrback(onError)
def writeResponse(message):
self.transport.write(message + "\r\n")
self.transport.loseConnection()
d.addCallback(writeResponse)
class FingerFactory(protocol.ServerFactory):
protocol = FingerProtocol
def __init__(self, prefix):
self.prefix = prefix
def getUser(self, user):
return client.getPage(self.prefix + user)
application = service.Application('finger', uid=1, gid=1)
factory = FingerFactory(prefix="http://livejournal.com/~")
internet.TCPServer(7979, factory).setServiceParent(
service.IServiceCollection(application))
which I save as finger_daemon.tac and run with
twistd -y finger_daemon.tac \
-l /home/me/twisted/finger.log \
--pidfile=/home/me/twisted/finger.pid
but of course it won't bind to 79, since it's a privileged port. I tried also running with sudo, no difference there.
I then tried changing the TCPServer port to 7979 and then connecting to the daemon once running with
telnet 127.0.0.1 7979
and I get Connection Refused error. What's going on here specifically? How is daemonizing supposed to work in Twisted?
When I run this code, I see the following log message:
2013-10-02 23:50:34-0700 [-] failed to set uid/gid 1/1 (are you root?) -- exiting.
and then twistd exits. So you'd need to do sudo twistd and then that adds a whole bunch of python path management problems...
Why are you setting the uid and gid arguments? You're trying to run it as the daemon user? You don't need to do that in order to daemonize. Just removing the uid=1, gid=1 arguments to Application makes it work for me.
Related
I implemented a Twisted SSH server to test a component that uses fabric to run commands on a remote machine via SSH. I have found this example but I don't understand how I have to implement the execCommand method to be compatible with fabric. Here is my implementation of the SSH server:
from pathlib import Path
from twisted.conch import avatar, recvline
from twisted.conch.insults import insults
from twisted.conch.interfaces import ISession
from twisted.conch.ssh import factory, keys, session
from twisted.cred import checkers, portal
from twisted.internet import reactor
from zope.interface import implementer
SSH_KEYS_FOLDER = Path(__file__).parent.parent / "resources" / "ssh_keys"
#implementer(ISession)
class SSHDemoAvatar(avatar.ConchUser):
def __init__(self, username: str):
avatar.ConchUser.__init__(self)
self.username = username
self.channelLookup.update({b"session": session.SSHSession})
def openShell(self, protocol):
pass
def getPty(self, terminal, windowSize, attrs):
return None
def execCommand(self, protocol: session.SSHSessionProcessProtocol, cmd: bytes):
protocol.write("Some text to return")
protocol.session.conn.sendEOF(protocol.session)
def eofReceived(self):
pass
def closed(self):
pass
#implementer(portal.IRealm)
class SSHDemoRealm(object):
def requestAvatar(self, avatarId, _, *interfaces):
return interfaces[0], SSHDemoAvatar(avatarId), lambda: None
def getRSAKeys():
with open(SSH_KEYS_FOLDER / "ssh_key") as private_key_file:
private_key = keys.Key.fromString(data=private_key_file.read())
with open(SSH_KEYS_FOLDER / "ssh_key.pub") as public_key_file:
public_key = keys.Key.fromString(data=public_key_file.read())
return public_key, private_key
if __name__ == "__main__":
sshFactory = factory.SSHFactory()
sshFactory.portal = portal.Portal(SSHDemoRealm())
users = {
"admin": b"aaa",
"guest": b"bbb",
}
sshFactory.portal.registerChecker(checkers.InMemoryUsernamePasswordDatabaseDontUse(**users))
pubKey, privKey = getRSAKeys()
sshFactory.publicKeys = {b"ssh-rsa": pubKey}
sshFactory.privateKeys = {b"ssh-rsa": privKey}
reactor.listenTCP(22222, sshFactory)
reactor.run()
Trying to execute a command via fabric yields the following output:
[paramiko.transport ][INFO ] Connected (version 2.0, client Twisted_22.4.0)
[paramiko.transport ][INFO ] Authentication (password) successful!
Some text to return
This looks promising but the program execution hangs after this line. Do I have to close the connection from the server side after executing the command? How do I implement that properly?
In a traditional UNIX server, it's still the server's responsibility to close the connection if it was told to execute a command. It's up to the server's discretion where and how to do this.
I believe you just want to change protocol.session.conn.sendEOF(protocol.session) to protocol.loseConnection(). I apologize for not testing this myself to be sure, setting up an environment to properly test a full-size conch setup like this is a bit tedious (and the example isn't self-contained, requiring key generation and moduli, etc)
I'm building a tester for a Python script which performs works with a RethinkDB database. As part of the setUp() method, I'm trying to make the tester start up the RethinkDB server on localhost on port 28016 in case that has not yet been done.
I'm using subprocess to start the server. The problem is that, according to https://docs.python.org/2/library/subprocess.html, subprocess waits for the command to complete. In this case, it seems that as long as the server is up and running, the process is not complete and testing does not continue beyond the setUp() stage.
Here is the script I'm trying:
import unittest
import rethinkdb as r
import subprocess
class TestController(unittest.TestCase):
HOST = "localhost"
PORT_OFFSET = 1
PORT = 28015 + PORT_OFFSET
DB = "ipercron"
TABLE = "sensor_data"
def setUp(self):
try:
self.conn = r.connect(self.HOST, self.PORT)
except r.ReqlDriverError:
print("The RethinkDB server is not yet ready. Starting it up...")
subprocess.call(["rethinkdb", "--port-offset", str(TestController.PORT_OFFSET)])
self.conn = r.connect(self.HOST, self.PORT)
if TestController.DB not in r.db_list().run(self.conn):
r.db_create(TestController.DB).run(self.conn)
self.conn.use(TestController.DB)
if TestController.TABLE not in r.table_list().run(self.conn):
r.table_create(TestController.TABLE).run(self.conn) # Create the table if it does not yet exist
r.table(TestController.TABLE).delete().run(self.conn) # Empty the table to start with a clean slate
def test_upper(self):
self.assertEqual('foo'.upper(), 'FOO')
suite = unittest.TestLoader().loadTestsFromTestCase(TestController)
unittest.TextTestRunner(verbosity=2).run(suite)
The subprocess is meant to perform the rethinkdb --port-offset 1 command at the command line and then continue with the script. However, when I run the script I get the usual message that the server is ready:
kurt#kurt-ThinkPad:~/dev/clones/ipercron-compose/controller$ python unittest_controller.py
test_upper (__main__.TestController) ... The RethinkDB server is not yet ready. Starting it up...
Running rethinkdb 2.3.5~0xenial (GCC 5.3.1)...
Running on Linux 4.4.0-42-generic x86_64
Loading data from directory /home/kurt/dev/clones/ipercron-compose/controller/rethinkdb_data
Listening for intracluster connections on port 29016
Listening for client driver connections on port 28016
Listening for administrative HTTP connections on port 8081
Listening on cluster addresses: 127.0.0.1, 127.0.1.1, ::1
Listening on driver addresses: 127.0.0.1, 127.0.1.1, ::1
Listening on http addresses: 127.0.0.1, 127.0.1.1, ::1
To fully expose RethinkDB on the network, bind to all addresses by running rethinkdb with the `--bind all` command line option.
Server ready, "kurt_ThinkPad_a0k" 07bb35f6-3a33-4e8b-9e9c-a78504457969
Without any further action. How can I make the unittest proceed with the testing?
The problem is as you said, that subprocess.call is blocking and will wait for the command to finish. For scenarios where you need to spawn a child process without waiting for it to finish, you can use subprocess.Popen:
process = subprocess.Popen(["rethinkdb", "--port-offset", str(TestController.PORT_OFFSET)])
This gives you back a Popen object that provides a whole bunch of very useful methods to communicate with the child process. For example, you will probably want to use process.kill() in your unit test's tearDown() function to shut down your database.
Try with the following code
subprocess.call(["rethinkdb", "--port-offset", str(TestController.PORT_OFFSET)], shell = True)
I want to run the demo that is part of open source project JInfinote. The demo server is written using Twisted Matrix library/server. However I have no idea how to run this and if it's standalone server that I need to download in order to run or is it just python or a library and how to configure the whole thing.
When I try to run this in python I'm getting some session exception - as if it was trying to literally execute the code function by function. I would appreciate any help with this.
I am sorry for the level of this question, but I'm not a python programmer and I'm just trying to understand the project JInfinote and this is a blocker.
Well, in order to run twisted matrix on a web-server, all you have to do is really just run a simple Python Script:
from twisted.web import server, resource
from twisted.internet import reactor
class HelloResource(resource.Resource):
isLeaf = True
numberRequests = 0
def render_GET(self, request):
self.numberRequests += 1
request.setHeader("content-type", "text/plain")
return "I am request #" + str(self.numberRequests) + "\n"
reactor.listenTCP(80, server.Site(HelloResource()))
reactor.run()
If you listen on port 80 then your server is open to the web. You can learn more about it from here.
I have a twisted application which now needs to monitor processes running on several boxes. The way I manually do is 'ssh and ps', now I'd like my twisted application to do. I have 2 options.
Use paramiko or leverage the power of twisted.conch
I really want to use twisted.conch but my research led me to believe that its primarily intended to create SSHServers and SSHClients. However my requirement is a simple remoteExecute(some_cmd)
I was able to figure out how to do this using paramiko but I didnt want to stickparamiko in my twisted app before looking at how to do this using twisted.conch
Code snippets using twisted on how to run remote_cmds using ssh would be highly appreciated. Thanks.
Followup - Happily, the ticket I referenced below is now resolved. The simpler API will be included in the next release of Twisted. The original answer is still a valid way to use Conch and may reveal some interesting details about what's going on, but from Twisted 13.1 and on, if you just want to run a command and handle it's I/O, this simpler interface will work.
It takes an unfortunately large amount of code to execute a command on an SSH using the Conch client APIs. Conch makes you deal with a lot of different layers, even if you just want sensible boring default behavior. However, it's certainly possible. Here's some code which I've been meaning to finish and add to Twisted to simplify this case:
import sys, os
from zope.interface import implements
from twisted.python.failure import Failure
from twisted.python.log import err
from twisted.internet.error import ConnectionDone
from twisted.internet.defer import Deferred, succeed, setDebugging
from twisted.internet.interfaces import IStreamClientEndpoint
from twisted.internet.protocol import Factory, Protocol
from twisted.conch.ssh.common import NS
from twisted.conch.ssh.channel import SSHChannel
from twisted.conch.ssh.transport import SSHClientTransport
from twisted.conch.ssh.connection import SSHConnection
from twisted.conch.client.default import SSHUserAuthClient
from twisted.conch.client.options import ConchOptions
# setDebugging(True)
class _CommandTransport(SSHClientTransport):
_secured = False
def verifyHostKey(self, hostKey, fingerprint):
return succeed(True)
def connectionSecure(self):
self._secured = True
command = _CommandConnection(
self.factory.command,
self.factory.commandProtocolFactory,
self.factory.commandConnected)
userauth = SSHUserAuthClient(
os.environ['USER'], ConchOptions(), command)
self.requestService(userauth)
def connectionLost(self, reason):
if not self._secured:
self.factory.commandConnected.errback(reason)
class _CommandConnection(SSHConnection):
def __init__(self, command, protocolFactory, commandConnected):
SSHConnection.__init__(self)
self._command = command
self._protocolFactory = protocolFactory
self._commandConnected = commandConnected
def serviceStarted(self):
channel = _CommandChannel(
self._command, self._protocolFactory, self._commandConnected)
self.openChannel(channel)
class _CommandChannel(SSHChannel):
name = 'session'
def __init__(self, command, protocolFactory, commandConnected):
SSHChannel.__init__(self)
self._command = command
self._protocolFactory = protocolFactory
self._commandConnected = commandConnected
def openFailed(self, reason):
self._commandConnected.errback(reason)
def channelOpen(self, ignored):
self.conn.sendRequest(self, 'exec', NS(self._command))
self._protocol = self._protocolFactory.buildProtocol(None)
self._protocol.makeConnection(self)
def dataReceived(self, bytes):
self._protocol.dataReceived(bytes)
def closed(self):
self._protocol.connectionLost(
Failure(ConnectionDone("ssh channel closed")))
class SSHCommandClientEndpoint(object):
implements(IStreamClientEndpoint)
def __init__(self, command, sshServer):
self._command = command
self._sshServer = sshServer
def connect(self, protocolFactory):
factory = Factory()
factory.protocol = _CommandTransport
factory.command = self._command
factory.commandProtocolFactory = protocolFactory
factory.commandConnected = Deferred()
d = self._sshServer.connect(factory)
d.addErrback(factory.commandConnected.errback)
return factory.commandConnected
class StdoutEcho(Protocol):
def dataReceived(self, bytes):
sys.stdout.write(bytes)
sys.stdout.flush()
def connectionLost(self, reason):
self.factory.finished.callback(None)
def copyToStdout(endpoint):
echoFactory = Factory()
echoFactory.protocol = StdoutEcho
echoFactory.finished = Deferred()
d = endpoint.connect(echoFactory)
d.addErrback(echoFactory.finished.errback)
return echoFactory.finished
def main():
from twisted.python.log import startLogging
from twisted.internet import reactor
from twisted.internet.endpoints import TCP4ClientEndpoint
# startLogging(sys.stdout)
sshServer = TCP4ClientEndpoint(reactor, "localhost", 22)
commandEndpoint = SSHCommandClientEndpoint("/bin/ls", sshServer)
d = copyToStdout(commandEndpoint)
d.addErrback(err, "ssh command / copy to stdout failed")
d.addCallback(lambda ignored: reactor.stop())
reactor.run()
if __name__ == '__main__':
main()
Some things to note about it:
It uses the new endpoint APIs introduced in Twisted 10.1. It's possible to do this directly on reactor.connectTCP, but I did it as an endpoint to make it more useful; endpoints can be swapped easily without the code that actually asks for a connection knowing.
It does no host key verification at all! _CommandTransport.verifyHostKey is where you would implement that. Take a look at twisted/conch/client/default.py for some hints about what kinds of things you might want to do.
It takes $USER to be the remote username, which you may want to be a parameter.
It probably only works with key authentication. If you want to enable password authentication, you probably need to subclass SSHUserAuthClient and override getPassword to do something.
Almost all of the layers of SSH and Conch are visible here:
_CommandTransport is at the bottom, a plain old protocol that implements the SSH transport protocol. It creates a...
_CommandConnection which implements the SSH connection negotiation parts of the protocol. Once that completes, a...
_CommandChannel is used to talk to a newly opened SSH channel. _CommandChannel does the actual exec to launch your command. Once the channel is opened it creates an instance of...
StdoutEcho, or whatever other protocol you supply. This protocol will get the output from the command you execute, and can write to the command's stdin.
See http://twistedmatrix.com/trac/ticket/4698 for progress in Twisted on supporting this with less code.
I'd like to know why the following doesn't work.
from twisted internet import defer, reactor
from twisted.python.failure import Failure
import twisted.names.client
def do_lookup(do_lookup):
d = twisted.names.client.getHostByName(domain)
d.addBoth(lookup_done)
def lookup_done(result):
print 'result:', result
reactor.stop()
domain = 'twistedmatrix.com'
reactor.callLater(0, do_lookup, domain)
reactor.run()
Results in:
result: [Failure instance: Traceback
(failure with no frames): <class
'twisted.names.error.ResolverError'>:
Stuck at response without answers or
delegation ]
As of this writing, this fails on Windows, since it uses an invalid path for the windows host file (in twisted.names.client.createResolver. It uses 'c:\windows\hosts'. This was fine for windows versions 98 and Me (reference here), but would fail with a version as "modern" as XP.
Today, it should probably use something like:
hosts = os.path.join(
os.environ.get('systemroot','C:\\Windows'),
r'system32\drivers\etc\hosts'
)
I think this only partly resolves the issue though (or maybe this is a red herring).
This will only work now for names actually specified in this hosts file. What it likely needs to do is some sort of registry query for the DNS server, then query that for the actual DNS name lookup.
This recipe looks promising for getting the actual DNS server.
Correcting your example to the following, so that it is syntactically valid:
from twisted.internet import reactor
import twisted.names.client
def do_lookup(domain):
d = twisted.names.client.getHostByName(domain)
d.addBoth(lookup_done)
def lookup_done(result):
print 'result:', result
reactor.stop()
domain = 'twistedmatrix.com'
reactor.callLater(0, do_lookup, domain)
reactor.run()
I get:
$ python so-example.py
result: 66.35.39.65
So, to answer your question: your local DNS environment is broken, not twisted.names. Or maybe there's a bug. You'll need to track it down further.
I did some digging why an explicit client.createResolver(servers) wasn't working on our corporate windows machines. In the guts of Twisted's createResolver is a os-dependant branch:
def createResolver(servers=None, resolvconf=None, hosts=None):
...
from twisted.names import resolve, cache, root, hosts as hostsModule
if platform.getType() == 'posix':
if resolvconf is None:
resolvconf = '/etc/resolv.conf'
if hosts is None:
hosts = '/etc/hosts'
theResolver = Resolver(resolvconf, servers)
hostResolver = hostsModule.Resolver(hosts)
else:
if hosts is None:
hosts = r'c:\windows\hosts'
from twisted.internet import reactor
bootstrap = _ThreadedResolverImpl(reactor)
hostResolver = hostsModule.Resolver(hosts)
theResolver = root.bootstrap(bootstrap)
L = [hostResolver, cache.CacheResolver(), theResolver]
return resolve.ResolverChain(L)
The first warning sign for Windows is that argument servers is not used, so custom-DNS servers are ignored. Next is that the _ThreadedResolverImpl(), which uses platform specific code, is wrapped in a root.bootstrap() before being added to the resolver chain.
The purpose of root.bootstrap is to use the platform resolver to lookup a.root-servers.net, b.root-servers.net, etc. using the synchronous Windows platform resolver (which works - and returns IPs), and then do direct DNS queries against the root servers. The UDP packets fired off to root servers are then blocked by our corporate firewall - I see them in Wireshark.
The default getResolver() call used by names.client.getHostByName() invokes createResolver() directly, which may again result in this broken setup bootstrapped by a working DNS setup. For most environments I think adding the Resolver (with root or explicit servers) into the ResolverChain along with _ThreadedResolverImpl as a fallback would work - except that the platform resolver is a different interface.
Here's an example of detecting the local DNS servers asynchronously (by parsing the output of ipconfig) then installing a custom resolver to use them.
import sys
from twisted.python import log
from twisted.names import root, hosts, resolve, cache, client
from twisted.python.runtime import platform
from twisted.internet import reactor
from twisted.internet import utils
import re
def parseIpconfigDNSServers(output):
servers = []
found = False
for line in output.split('\n'):
if 'DNS Servers . . . . . . . . . . . :' in line or (found and not '. . . .' in line):
servers.append(line[38:].strip())
found = True
else:
found = False
log.msg( 'Windows: Detected DNS servers %s' % (str(servers)))
return servers
if platform.getType() != 'posix':
d = utils.getProcessOutput(os.path.join(os.environ['WINDIR'], 'system32', 'ipconfig.exe'), ["/all"])
d.addCallback(parseIpconfigDNSServers)
d.addCallback(lambda r: client.Resolver(servers=[(h, 53) for h in r]))
d.addErrback(log.msg)
theResolver = root.DeferredResolver(d)
client.theResolver = resolve.ResolverChain([cache.CacheResolver(), theResolver])
if __name__ == '__main__':
log.startLogging(sys.stdout)
def do_lookup(domain):
d = client.getHostByName(domain)
d.addBoth(log.msg)
from twisted.internet import reactor
reactor.callLater(0, do_lookup, 'example.com')
reactor.run()
I've tidied this up and posted it here https://gist.github.com/shuckc/af7490e1c4a2652ca740