I am currently having some trouble accessing content hosted via https using the twisted python library. I am new to this library, and am assuming there is some concept I am missing that's causing the issue, but perhaps not based upon the example.
Here is a link to the page in which I gathered the example:
https://twistedmatrix.com/documents/current/web/howto/client.html
Under the heading HTTP over SSL
from twisted.python.log import err
from twisted.web.client import Agent
from twisted.internet import reactor
from twisted.internet.ssl import optionsForClientTLS
def display(response):
print("Received response")
print(response)
def main():
contextFactory = optionsForClientTLS(u"https://example.com/")
agent = Agent(reactor, contextFactory)
d = agent.request("GET", "https://example.com/")
d.addCallbacks(display, err)
d.addCallback(lambda ignored: reactor.stop())
reactor.run()
if __name__ == "__main__":
main()
When running this code, it straight up fails. I get an error that looks like this:
Traceback (most recent call last):
File "https.py", line 19, in <module>
main()
File "https.py", line 11, in main
contextFactory = optionsForClientTLS(u"https://example.com/")
File "/home/amaricich/.local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1336, in optionsForClientTLS
return ClientTLSOptions(hostname, certificateOptions.getContext())
File "/home/amaricich/.local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1198, in __init__
self._hostnameBytes = _idnaBytes(hostname)
File "/home/amaricich/.local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 86, in _idnaBytes
return idna.encode(text)
File "/usr/local/lib/python2.7/dist-packages/idna/core.py", line 355, in encode
result.append(alabel(label))
File "/usr/local/lib/python2.7/dist-packages/idna/core.py", line 276, in alabel
check_label(label)
File "/usr/local/lib/python2.7/dist-packages/idna/core.py", line 253, in check_label
raise InvalidCodepoint('Codepoint {0} at position {1} of {2} not allowed'.format(_unot(cp_value), pos+1, repr(label)))
idna.core.InvalidCodepoint: Codepoint U+003A at position 6 of u'https://example' not allowed
This error lead me to believe the parameter being passed into optionsForClientTLS were incorrect. It calls for a hostname and not a full url, so I shortened the parameter to simply example.com. Once that change was made, the function completed successfully.
Unfortunately though, after making the change, the script now failed at the line invoking agent.request. The error it supplied was this:
Traceback (most recent call last):
File "https.py", line 19, in <module>
main()
File "https.py", line 13, in main
d = agent.request("GET", "https://example.com/")
File "/home/amaricich/.local/lib/python2.7/site-packages/twisted/web/client.py", line 1596, in request
endpoint = self._getEndpoint(parsedURI)
File "/home/amaricich/.local/lib/python2.7/site-packages/twisted/web/client.py", line 1580, in _getEndpoint
return self._endpointFactory.endpointForURI(uri)
File "/home/amaricich/.local/lib/python2.7/site-packages/twisted/web/client.py", line 1456, in endpointForURI
uri.port)
File "/home/amaricich/.local/lib/python2.7/site-packages/twisted/web/client.py", line 982, in creatorForNetloc
context = self._webContextFactory.getContext(hostname, port)
AttributeError: 'ClientTLSOptions' object has no attribute 'getContext'
This error leads me to believe that the object being produced by optionsForClientTLS is not the object type that is expected to be passed into the Agent upon creation. A function is trying to be invoked that does not exist. With all that said, I have two questions.
Is this example deprecated? The previous examples that make http requests all work like a charm. Am I doing something wrong, or is the example no longer valid?
I am only looking for a simple way to retrieve data from a server using HTTPS. If doing things this way is not the solution, is anyone familiar with how HTTPS requests can be made using twisted?
Yes you're absolutely correct that the example on the docs is wrong. I noticed the bug while working w/ treq. Try following this example from v14. With that being said, you should use treq as opposed to trying to use Twisted directly. Most of the heavy lifting has been taken care of for you. Here's an simple conversion of your example:
from __future__ import print_function
import treq
from twisted.internet import defer, task
from twisted.python.log import err
#defer.inlineCallbacks
def display(response):
content = yield treq.content(response)
print('Content: {0}'.format(content))
def main(reactor):
d = treq.get('https://twistedmatrix.com')
d.addCallback(display)
d.addErrback(err)
return d
task.react(main)
As you can see treq takes care of the SSL stuff for you. The display() callback function can be used to extract various components of the HTTP response, such as headers, status codes, body, etc. If you only need a single component, such as the response body, then you can simplify further like so:
def main(reactor):
d = treq.get('https://twistedmatrix.com')
d.addCallback(treq.content) # get response content when available
d.addErrback(err)
d.addCallback(print)
return d
task.react(main)
Related
I'm trying to get ladon working, however, I don't seem to be able to define the service properly.
Specifically, even with a minimal test case, it's throwing
Traceback (most recent call last):
File "C:\Python33\lib\site-packages\ladon\server\wsgi_application.py", line 332, in __call__
self.import_services(self.service_list)
File "C:\Python33\lib\site-packages\ladon\server\wsgi_application.py", line 288, in import_services
__import__(service)
File "D:\Workspaces\Python\SOAPManager.py", line 20, in <module>
#ladonize(PORTABLE_STRING, PORTABLE_STRING, rtype=PORTABLE_STRING)
File "C:\Python33\lib\site-packages\ladon\ladonizer\decorator.py", line 118, in decorator
injector.__doc__ = ladon_method_info._doc
AttributeError: 'NoneType' object has no attribute '_doc'
My Run.py contains:
from ladon.server.wsgi import LadonWSGIApplication
from os.path import abspath, dirname
from wsgiref.simple_server import make_server
application = LadonWSGIApplication(
['SOAPManager'],
[dirname(abspath(__file__))],
catalog_name='API',
catalog_desc='API Description')
httpd = make_server('0.0.0.0', 8004, application)
print("Listening on port 8004...")
# Respond to requests until process is killed
httpd.serve_forever()
And the minimal test case in SOAPManager.py:
from ladon.ladonizer import ladonize
from ladon.types.ladontype import LadonType
from ladon.compat import PORTABLE_STRING
#ladonize(PORTABLE_STRING, PORTABLE_STRING, rtype=PORTABLE_STRING)
def Authenticate(Username, Password):
return "Test"
The error is being raised from within the ladonize decorator. It seems to be occuring when trying to build up the definition of the service. Specifically, in ladon the decorator calls collection.add_service_method which returns None instead of a method. I think it's failing a line-number check.
firstlineno = f.__code__.co_firstlineno
# get an ast-analyzed object of the source file
sinfo = self.source_info(src_fname)
...
for clsname,v in sinfo.items():
if firstlineno>v[0] and firstlineno<=v[1]:
For some reason that check is failing so the method defaults to returning None.
I am trying to connect to some router using the above code and I am using juniperj2320.py module and in test file I have
am using code base from this http://subversion.assembla.com/svn/clauzsistel08/trunk/dist/
#!/usr/bin/python
from localconf import *
from juniperj2320 import *
import sys
import traceback
SERIALDEVICE = '/dev/ttyUSB0'
# Now configure the router
try:
router = JuniperJ2320(SERIALDEVICE)
except RouterConfigurationException, err:
print "Router configuration error: ", err
print "Please try again."
sys.exit(1)
but am getting this following error
./test.py
> /root/pyserial-2.6/examples/serialrouter.py(37)__init__()
-> serial.Serial.__init__(self, serialdevice, baudrate=baudrate, \
(Pdb) c
Traceback (most recent call last):
File "./test.py", line 30, in
router = JuniperJ2320(SERIALDEVICE)
File "/root/pyserial-2.6/examples/juniperj2320.py", line 32, in __init__
BYTESIZE, PARITY, STOPBITS, TIMEOUT)
File "/root/pyserial-2.6/examples/serialrouter.py", line 44, in __init__
fdpexpect.fdspawn.__init__(self, self.fileno())
File "/usr/lib/python2.6/site-packages/fdpexpect.py", line 40, in __init__
spawn.__init__(self, None, args, timeout, maxread, searchwindowsize, logfile )
File "/usr/lib/python2.6/site-packages/pexpect.py", line 412, in __init__
self.closed = True # File-like object.
AttributeError: can't set attribute
and absolutely clue less on wat happening here ! any help ll be greatly appreciated
thanks
This is a little bit of a shot in the dark, because I'm not familiar with the modules you're using, but based on the traceback, it looks like the constructor is expecting a file-like object, not just a file path. Try this instead.
SERIALDEVICE = '/dev/ttyUSB0'
# Now configure the router
try:
router = JuniperJ2320(open(SERIALDEVICE))
Instead of writing your own serial communications program wouldn't be easier to have pexpect drive a serial program like minicom?
I wanted to write an iptables script in Python. Rather than calling iptables itself I wanted to use the python-iptables package. However I'm having a hard time getting some basic rules setup. I wanted to use the filter chain to accept incoming TCP traffic on port 1234. So I wrote this:
import iptc
chain = iptc.Chain(iptc.TABLE_FILTER,"INPUT")
rule = iptc.Rule()
target = iptc.Target(rule,"ACCEPT")
match = iptc.Match(rule,'tcp')
match.dport='1234'
rule.add_match(match)
rule.target = target
chain.insert_rule(rule)
However when I run this I get this thrown back at me:
Traceback (most recent call last):
File "testing.py", line 9, in <module>
chain.insert_rule(rule)
File "/usr/local/lib/python2.6/dist-packages/iptc/__init__.py", line 1133, in insert_rule
self.table.insert_entry(self.name, rbuf, position)
File "/usr/local/lib/python2.6/dist-packages/iptc/__init__.py", line 1166, in new
obj.refresh()
File "/usr/local/lib/python2.6/dist-packages/iptc/__init__.py", line 1230, in refresh
self._free()
File "/usr/local/lib/python2.6/dist-packages/iptc/__init__.py", line 1224, in _free
self.commit()
File "/usr/local/lib/python2.6/dist-packages/iptc/__init__.py", line 1219, in commit
raise IPTCError("can't commit: %s" % (self.strerror()))
iptc.IPTCError: can't commit: Invalid argument
Exception AttributeError: "'NoneType' object has no attribute 'get_errno'" in <bound method Table.__del__ of <iptc.Table object at 0x7fcad56cc550>> ignored
Does anyone have experience with python-iptables that could enlighten on what I did wrong?
Oh, just noticed this one. Can you give the latest head from github a shot? I've fixed tons of bugs and updated python-iptables to work with the latest iptables version. If you still experience issues please open a ticket on github.
One thing that sure is not quite right is that you don't set the protocol in the rule:
import iptc
chain = iptc.Chain(iptc.TABLE_FILTER,"INPUT")
rule = iptc.Rule()
Set protocol e.g. here:
rule.protocol = 'tcp'
and then you should be all right:
target = iptc.Target(rule,"ACCEPT")
match = iptc.Match(rule,'tcp')
match.dport='1234'
rule.add_match(match)
rule.target = target
chain.insert_rule(rule)
I am attempting to build a download accelerator for Linux. My program utilizes gevent, os, and urllib2. My program receives a URL and attempts to download the file concurrently. All of my code is valid. My only problem is that urllib2.urlopen.read() is blocking me from running the .read() function concurrently.
This is the exception thats thrown at me.
Traceback (most recent call last):
File "/usr/lib/pymodules/python2.7/gevent/greenlet.py", line 405, in run
result = self._run(*self.args, **self.kwargs)
File "gevent_concurrent_downloader.py", line 94, in childTasklet
_tempRead = handle.read(divisor) # Read/Download part
File "/usr/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
File "/usr/lib/python2.7/httplib.py", line 561, in read
s = self.fp.read(amt)
File "/usr/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
File "/usr/lib/pymodules/python2.7/gevent/socket.py", line 407, in recv
wait_read(sock.fileno(), timeout=self.timeout, event=self._read_event)
File "/usr/lib/pymodules/python2.7/gevent/socket.py", line 153, in wait_read
assert event.arg is None, 'This event is already used by another greenlet: %r' % (event.arg, )
AssertionError: This event is already used by another greenlet: (<Greenlet at 0x2304958: childTasklet(<__main__.NewFile object at 0x22c4390>, 4595517, <addinfourl at 37154616 whose fp = <socket._fileob, 459551, 1)>, timeout('timed out',))
<Greenlet at 0x2304ea8: childTasklet(<__main__.NewFile object at 0x22c4390>,4595517, <addinfourl at 37154616 whose fp = <socket._fileob, 7, -1)failed with AssertionError
My program works by getting the file byte size from the URL by invoking:
urllib2.urlopen(URL).info().get("Content-Length")
and dividing the file size by a divisor and thus breaking the download process into parts. In this example i am breaking the download into 10 parts.
Each greenlet runs a command in this fassion:
urllib2.urlopen(URL).read(offset)
Here's a link to my code hosted on pastie: http://pastie.org/3253705
Thank you for the help!
FYI: I am running on Ubuntu 11.10.
You're trying to read a response to a single request from different greenlets.
If you'd like to download the same file using several concurrent connections then you could use Range http header if the server supports it (you get 206 status instead of 200 for the request with Range header). See HTTPRangeHandler.
the argument to read is a number of bytes, not an offset.
It seems gevent will let you call urllib asynchronously, but not let you access the same resource from multiple greenlets.
Furthermore, since it is using wait_read, the effect will still be a synchronous, sequential read from the file (The complete opposite of what you wanted to achieve).
I'd suggest you might need to go lower than, or use a different library from, urllib2.
I'm using Python 2.6 and suds 0.3.7 to interact with JIRA 4.0.
When I connect to the JIRA server, I get information on all the issues just fine.
However, when I want to update an issue, I get a SAXException from suds (presumably):
WebFault: Server raised fault:
org.xml.sax.SAXException: Found character data inside an array element while deserializing
I'm following the steps described here: http://confluence.atlassian.com/display/JIRA/Creating+a+SOAP+Client
only replacing SOAPpy calls with suds.
My attempt to update an issue looks like this, complete with exceptions:
>>> w="http://bugs/rpc/soap/jirasoapservice-v2?wsdl"
>>> from suds.client import Client
>>> client = Client(w)
>>> auth = client.service.login("myname","mypass")
>>> issue = client.service.getIssue(auth,"BUG-30112")
>>> issue.summary
This is the original summary for BUG-30112
>>>
>>> client.service.updateIssue(auth,"BUG-30112",[
... {"id":"summary","values":["My new summary"]}])
Traceback (most recent call last):
File "<interactive input>", line 2, in <module>
File "C:\Python26\lib\suds\client.py", line 535, in __call__
return client.invoke(args, kwargs)
File "C:\Python26\lib\suds\client.py", line 595, in invoke
result = self.send(msg)
File "C:\Python26\lib\suds\client.py", line 630, in send
result = self.failed(binding, e)
File "C:\Python26\lib\suds\client.py", line 681, in failed
r, p = binding.get_fault(reply)
File "C:\Python26\lib\suds\bindings\binding.py", line 235, in get_fault
raise WebFault(p, faultroot)
WebFault: Server raised fault: 'org.xml.sax.SAXException: Found character data inside an array element while deserializing'
>>>
Has anyone seen a problem like this?
This will be solved if you switch to suds 3.0.9 (beta) ... the only one to have the fix.
How about increasing the verbosity to see what is being sent? Or use wireshark. You could also do the same with SOAPpy and compare exactly what is sent. Debugging soap errors is usually like this for me :-/
~Matt
Actually, by just changing the library from suds to SOAPpy, everything started working with no other modifications. Kind of annoying. I skipped SOAPpy because it seemed to have been abandoned and more complex to install, compared to suds. But SOAPpy works!
Thanks, all.