I've looked around quite a bit, but haven't found an error quite like this. When I execute my code (below), I get the exception ControllerFactory instance has no attribute 'startedConnecting'. I tried adding the method with the body just being pass, but that simply causes it to stall without transmitting anything, leading me to believe that the problem lies within the way I've set up the classes.
This code is based on code from the twisted website. It's meant to be able to transmit python files, which the server saves, then later transmit arguments for running the python file.
#!/usr/bin/env python
from twisted.internet import reactor, protocol
import argparse
file_header = "pfile:"
run_header = "runwith:"
class Controller(protocol.Protocol):
def sendMessage(self,message):
self.transport.write(message)
class ControllerFactory(protocol.Factory):
def buildProtocol(self, addr):
cont = Controller()
cont.factory = self
return cont
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument("--address")
parser.add_argument("--file")
parser.add_argument("--args")
args = parser.parse_args()
if(args.file and args.args):
raise Exception("Can't send file and args at same time.")
reactor.connectTCP(args.address, 1337, ControllerFactory())
reactor.run()
if(args.file):
print(args.file)
a = open(args.file)
factory.connectedProtocol.sendMessage(file_header + a.read())
a.close()
if(args.args):
print(args.args)
factory.connectedProtocol.sendMessage(run_header + args.args)
Use twisted.internet.protocol.ClientFactory for clients (instead of twisted.internet.protocol.Factory). Or use something from twisted.internet.endpoints instead of twisted.internet.reactor.connectTCP.
Also, note that reactor.run() blocks. All of your code that follows that line will not run in any useful way.
Related
When I use a standard Queue to send samples to a process, everything works fine. However, since my needs are simple, I tried to use a SimpleQueue and for some reason the 'empty' method doesn't work. Here's the details:
Error comes from the consumer process (when sample_queue is Queue, everything works, when sample_queue is SimpleQueue, things break):
def frame_update(i):
while not self.sample_queue.empty():
sample = self.sample_queue.get()
for line in lines:
While executing sample_queue.empty() -- SimpleQueue.empty(), from Python 3.6 on windows (queues.py) we get:
def empty(self):
return not self._poll()
Where self._poll() has been set in init by:
def __init__(self, *, ctx):
self._reader, self._writer = connection.Pipe(duplex=False)
self._rlock = ctx.Lock()
self._poll = self._reader.poll
if sys.platform == 'win32':
self._wlock = None
else:
self._wlock = ctx.Lock()
So I follow the self._reader which is set from connection.Pipe (connection.py):
...
c1 = PipeConnection(h1, writable=duplex)
c2 = PipeConnection(h2, readable=duplex)
Ok, great. The _reader is going to be a PipeConnection and pipe connection has this method:
def _poll(self, timeout):
if (self._got_empty_message or
_winapi.PeekNamedPipe(self._handle)[0] != 0):
return True
return bool(wait([self], timeout))
Alright -- So a couple of questions:
1) Shouldn't the init of SimpleQueue be assigning self.poll to self._reader._poll instead of self._reader.poll? Or am I missing something in the inheritance hierarchy?
2) The PipeConnection _poll routine takes a timeout parameter, so #1 shouldn't work...
*) -- Is there some other binding of PipeConnection _poll that I'm missing?
Am I missing something? I am using Python3.6, Windows, debugging in PyCharm and I follow all the paths and they're in the standard multiprocessing paths. I'd appreciate any help or advice. Thanks!
EDIT: After further review, I can see that PipeConnection is a subclass of _ConnectionBase which does indeed have a 'poll' method and it is bound with a default timeout parameter.
So the question is: When SimpleQueue is initializing and sets
self._poll = self._reader.poll
Why doesn't it go up the class hierarchy to grab that from _ConnectionBase?
After looking at why the Queue type works and why the SimpleQueue doesn't, I found that Queue sets the _poll method 'after_fork' as well as before. SimpleQueue doesn't. By changing the setstate method to add self._poll = self._reader.poll as follows (queues.py, line 338), SimpleQueue works
def __setstate__(self, state):
(self._reader, self._writer, self._rlock, self._wlock) = state
self._poll = self._reader.poll
Seems like a bug to me unless I'm really misunderstanding something. I'll submit a bug report and reference this post. Hope this helps someone!
http://bugs.python.org/issue30301
I want to use Luigi to manage workflows in Openstack. I am new to Luigi. For the starter, I just want to authenticate myself to Openstack and then fetch image list, flavor list etc using Luigi. Any help will be appreciable.
I am not good with python but I tried below code. I am also not able to list images. Error: glanceclient.exc.HTTPNotFound: The resource could not be found. (HTTP 404)
import luigi
import os_client_config
import glanceclient.v2.client as glclient
from luigi.mock import MockFile
import sys
import os
def get_credentials():
d = {}
d['username'] = 'X'
d['password'] = 'X'
d['auth_url'] = 'X'
d['tenant_name'] = 'X'
d['endpoint'] = 'X'
return d
class LookupOpenstack(luigi.Task):
d =[]
def requires(self):
pass
def output(self):
gc = glclient.Client(**get_credentials())
images = gc.images.list()
print("images", images)
for i in images:
print(i)
return MockFile("images", mirror_on_stderr=True)
def run(self):
pass
if __name__ == '__main__':
luigi.run(["--local-scheduler"], LookupOpenstack())
The general approach to this is just write python code to perform the tasks you want using the OpenStack API. https://docs.openstack.org/user-guide/sdk.html It looks like the error you are getting is addressed on the OpenStack site. https://ask.openstack.org/en/question/90071/glanceclientexchttpnotfound-the-resource-could-not-be-found-http-404/
You would then just wrap this code in luigi Tasks as appropriate- there's nothing special about doing with this OpenStack, except that you must define the output() of your luigi tasks to match up with an output that indicates the task is done. Right now it looks like the work is being done in the output() method, which should be in the run() method, the output method should just be what to look for to indicate that the run() method is complete so it doesn't run() when required by another task if it is already done.
It's really impossible to say more without understanding more details of your workflow.
I am using the cmd.Cmd class in Python to offer a simple readline interface to my program.
Self contained example:
from cmd import Cmd
class CommandParser(Cmd):
def do_x(self, line):
pass
def do_xy(self, line):
pass
def do_xyz(self, line):
pass
if __name__ == "__main__":
parser = CommandParser()
parser.cmdloop()
Pressing tab twice will show possibilities. Pressing tab again does the same.
My question is, how do I get the options to cycle on the third tab press? In readline terms I think this is called Tab: menu-complete, but I can't see how to apply this to a Cmd instance.
I already tried:
readline.parse_and_bind('Tab: menu-complete')
Both before and after instantiating the parser instance. No luck.
I also tried passing "Tab: menu-complete" to the Cmd constructor. No Luck here either.
Anyone know how it's done?
Cheers!
The easiest trick would be to add a space after menu-complete:
parser = CommandParser(completekey="tab: menu-complete ")
The bind expression that is executed
readline.parse_and_bind(self.completekey+": complete")
will then become
readline.parse_and_bind("tab: menu-complete : complete")
Everything after the second space is acutally ignored, so it's the same as tab: menu-complete.
If you don't want to rely on that behaviour of readline parsing (I haven't seen it documented) you could use a subclass of str that refuses to be extended as completekey:
class stubborn_str(str):
def __add__(self, other):
return self
parser = CommandParser(completekey=stubborn_str("tab: menu-complete"))
self.completekey+": complete" is now the same as self.completekey.
Unfortunately, it seems as though the only way around it is to monkey-patch the method cmdloop from the cmd.Cmd class, or roll your own.
The right approach is to use "Tab: menu-complete", but that's overriden by the class as shown in line 115: readline.parse_and_bind(self.completekey+": complete"), it is never activated. (For line 115, and the entire cmd package, see this: https://hg.python.org/cpython/file/2.7/Lib/cmd.py). I've shown an edited version of that function below, and how to use it:
import cmd
# note: taken from Python's library: https://hg.python.org/cpython/file/2.7/Lib/cmd.py
def cmdloop(self, intro=None):
"""Repeatedly issue a prompt, accept input, parse an initial prefix
off the received input, and dispatch to action methods, passing them
the remainder of the line as argument.
"""
self.preloop()
if self.use_rawinput and self.completekey:
try:
import readline
self.old_completer = readline.get_completer()
readline.set_completer(self.complete)
readline.parse_and_bind(self.completekey+": menu-complete") # <---
except ImportError:
pass
try:
if intro is not None:
self.intro = intro
if self.intro:
self.stdout.write(str(self.intro)+"\n")
stop = None
while not stop:
if self.cmdqueue:
line = self.cmdqueue.pop(0)
else:
if self.use_rawinput:
try:
line = raw_input(self.prompt)
except EOFError:
line = 'EOF'
else:
self.stdout.write(self.prompt)
self.stdout.flush()
line = self.stdin.readline()
if not len(line):
line = 'EOF'
else:
line = line.rstrip('\r\n')
line = self.precmd(line)
stop = self.onecmd(line)
stop = self.postcmd(stop, line)
self.postloop()
finally:
if self.use_rawinput and self.completekey:
try:
import readline
readline.set_completer(self.old_completer)
except ImportError:
pass
# monkey-patch - make sure this is done before any sort of inheritance is used!
cmd.Cmd.cmdloop = cmdloop
# inheritance of the class with the active monkey-patched `cmdloop`
class MyCmd(cmd.Cmd):
pass
Once you've monkey-patched the class method, (or implemented your own class), it provides the correct behavior (albeit without highlighting and reverse-tabbing, but these can be implemented with other keys as necessary).
I'll do my best to describe the issue I am having. I am building a Python program that is built on multiple classes and uses the unittest framework. In a nutshell, the Main.py file has a "ValidateDriver" class that defines a "driver" variable as an ElementTree type. If I point this directly to the XML file I need to parse, (i.e. driver = ElementTree.parse(rC:\test.xml)) then I can access it from another class. However, in reality I don't have the actual XML file that is passed in from the command-line until you get to the Main function in the ValidateDriver class. So under the ValidateDriver class driver would really be driver = ElementTree and then in the main function I would reassign that variable to ValidateDriver.driver = ElementTree.parse(args.driver). However, this is the crux. When I go to the other class and try to call ValidateDriver.driver I don't have the "findall" method/attribute available. Again, the only way it will work is to do something like: ElementTree.parse(rC:\test.xml)). If I did this in C# it would work, but I am new to Python and this is kicking my butt. Any help/suggestions is appreciated. I've included the code for both classes.
Main Function:
import sys
import argparse
import xml.etree.ElementTree as ElementTree
import unittest
import Tests.TestManufacturer
class ValidateDriver:
driver = ElementTree
def main(argv):
parser = argparse.ArgumentParser(description='Validation.')
parser.add_argument('-d', '--driver', help='Path and file name xml file', required=True)
parser.add_argument('-v', '--verbosity',
help='Verbosity for test output. 1 for terse, 2 for verbose. Default is verbose',
default=2, type=int)
#args = parser.parse_args()
args = r'C:\test.c4i'
#print ("Validate Driver: %s" % args.driver)
#print ("Verbosity Level: %s" % args.verbosity)
ValidateDriver.driver = ElementTree.parse(r'C:\test.c4i')
loader = unittest.TestLoader()
suite = loader.loadTestsFromModule(Tests.TestManufacturer)
runner = unittest.TextTestRunner(verbosity=2) # TODO Remove this...
# TODO Uncomment this...
runner = unittest.TextTestRunner(verbosity=args.verbosity)
result = runner.run(suite)
if __name__ == "__main__":
main(sys.argv[1:])
Other Class, Test Manufacturer:
import unittest
import Main
manufacturer = ['']
class Tests(unittest.TestCase):
# Test to see if Manufacturer exists.
def test_manufacturer_exists(self):
for m in Main.ValidateDriver.driver.findall('./manufacturer'):
print m.text
Producing the following error:
C:\Python27\python.exe C:\Users\test\PycharmProjects\Validator\Main.py
Traceback (most recent call last):
File "C:\Users\test\PycharmProjects\Validator\Main.py", line 22, in <module>
class ValidateDriver:
File "C:\Users\test\PycharmProjects\Validator\Main.py", line 65, in ValidateDriver
main(sys.argv[1:])
File "C:\Users\test\PycharmProjects\Validator\Main.py", line 36, in main
ValidateDriver.driver = ElementTree.parse(r'C:\test.c4i')
NameError: global name 'ValidateDriver' is not defined
Process finished with exit code 1
The main problem seems to be that your main script is wrapped in a class. There's really no reason for this, and is quite confusing.
if __name__ == "__main__":
main_object = ValidateDriver()
main_object.main(sys.argv[1:])
This should go outside the class definition
This has nothing to do with "findall" being available. The issue is that the class itself hasn't yet been completely declared at the time you try to access it. In python, the file is read top to bottom. For example, this is not allowed:
if __name__ == "__main__":
f()
def f():
...
The call to f must happen at the bottom of the file after it is declared.
What you're doing with ValidateDriver is similar, because the class isn't defined until the statements directly in its body are executed (this is different from functions, whose bodies of course aren't executed until they are called). You call main(sys.argv[1:]) inside the class body, which in turn tries to access ValidateDriver.driver, which doesn't exist yet.
Preferably, the main function, as well as the code which calls it, should be outside the class. As far as I can tell, the class doesn't need to exist at all (this isn't C# or Java -- you can put code directly at the module level without a class container). If you insist on putting it in a class as a static method, it must be defined as a class method:
#classmethod
def main(cls, argv):
...
which can then be called (outside the class definition) like:
ValidateDriver.main(sys.argv[1:])
But I stress that this is non-standard and should not be necessary.
I'm trying to use ldaptor to connect via startTLS to a LDAP server. Searching on internet and trying myself I arrived to this snippet of code:
from ldaptor.protocols.ldap import ldapclient, ldapsyntax, ldapconnector, distinguishedname
[...]
def main(base, serviceLocationOverrides):
c=ldapconnector.LDAPClientCreator(reactor, ldapclient.LDAPClient)
d = c.connect(base, serviceLocationOverrides)
d.addCallbacks(lambda proto: proto.startTLS(), error)
[...]
d.addErrback(error)
d.addBoth(lambda dummy: reactor.stop())
reactor.run()
but the code exits with an AssertionError:
[Failure instance: Traceback: <type 'exceptions.AssertionError'>:
/usr/lib/python2.7/dist-packages/twisted/internet/base.py:1167:mainLoop
/usr/lib/python2.7/dist-packages/twisted/internet/base.py:789:runUntilCurrent
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:361:callback
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:455:_startRunCallbacks
--- <exception caught here> ---
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:542:_runCallbacks
/usr/lib/pymodules/python2.7/ldaptor/protocols/ldap/ldapclient.py:239:_startTLS
/usr/lib/pymodules/python2.7/ldaptor/protocols/pureldap.py:1278:__init__
/usr/lib/pymodules/python2.7/ldaptor/protocols/pureldap.py:1144:__init__
]
I have tried to look in ldaptor code for the incriminated assertion but seems all ok.
Is there someone who succeded in using ldaptorClient startTLS?
A code snippet?
Thank you very much
Bye
I'm pretty certain that your problem is one I ran into a while back. In ldaptor/protocols/pureldap.py, line 1144 asserts that the LDAPExtendedRequest requestValue must be a string. But according to RFC 2251, that value is optional, and specifically should NOT be present in startTLS requests.
So your approach is correct; this is just a major bug in ldaptor. As far as I can tell, the author only tested using simple bind without TLS. You need to comment out that line in pureldap.py. If you're deploying this with the expectation that users will download or easy-install ldaptor, then you'll need to create a fixed copy of the LDAPExtendedRequest class in your own code, and sub it in at run-time.
Having had to maintain a project using ldaptor for several years, I would strongly urge you to switch to python-ldap if at all possible. Since it wraps the OpenLDAP libs, it can be much more difficult to build, especially with full support for SSL/SASL. But it's well worth it, because ldaptor has a lot more problems than just the one you ran across.
Using ldaptor 0.0.54 from https://github.com/twisted/ldaptor, I had no problems using StartTLS.
Here is the code:
#! /usr/bin/env python
from twisted.internet import reactor, defer
from ldaptor.protocols.ldap import ldapclient, ldapsyntax, ldapconnector
#defer.inlineCallbacks
def example():
serverip = 'your.server.name.or.ip'
basedn = 'o=Organization'
binddn = 'cn=admin,o=Organization'
bindpw = 'Sekret'
query = '(uid=jetsong)'
c = ldapconnector.LDAPClientCreator(reactor, ldapclient.LDAPClient)
overrides = {basedn: (serverip, 389)}
client = yield c.connect(basedn, overrides=overrides)
client = yield client.startTLS()
yield client.bind(binddn, bindpw)
o = ldapsyntax.LDAPEntry(client, basedn)
results = yield o.search(filterText=query)
for entry in results:
print entry
if __name__ == '__main__':
df = example()
df.addErrback(lambda err: err.printTraceback())
df.addCallback(lambda _: reactor.stop())
reactor.run()