Develop Reference

python, suds and client certificate: SAXParseException: not well-formed (invalid token)

I have implementing a SOAP client with python (2.6.6) and suds. The server needs a certificate from the client for authentication. For implementing this in python and with suds I have use this answer from Andre Miras. This seem's also to work because I can access and get the WSDL from the server. But I have another problem. While parsing the WSDL I get a xml.sax._exceptions.SAXParseException: <unknown>:1:1: not well-formed (invalid token) error.
To check if I can realy get the WSDL I have manipulate the "open" method (please note the "print"s)
def open(self, request):
"""
Fetches the WSDL using cert.
"""
print "11 "
# self.addcredentials(request)
resp = requests.get(request.url, data=request.message,
headers=request.headers, cert=self.cert)
result = io.StringIO(resp.content.decode('utf-8'))
print str(result.getvalue())
print "<<WSDL END>>"
return result
If I run the script I get the output:
11
<?xml version='1.0' encoding='UTF-8'?><wsdl:definitions xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:tns="http://webService.net.app.my.com/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:ns1="http://schemas.xmlsoap.org/soap/http" name="CStatisticService" targetNamespace="http://webService.net.app.my.com/">
<wsdl:types>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:tns="http://webService.net.app.my.com/" elementFormDefault="unqualified" targetNamespace="http://webService.net.app.my.com/" version="1.0">
<xs:element name="getFileActionsNmbr" type="tns:getFileActionsNmbr"/>
...
...
...
</wsdl:service>
</wsdl:definitions>
<<WSDL END>>
Traceback (most recent call last):
File "./c-App_jvmThreads", line 63, in <module>
client = suds.client.Client(wsdl_url, headers=headers, transport=t)
File "/usr/lib/python2.6/site-packages/suds/client.py", line 112, in __init__
self.wsdl = reader.open(url)
File "/usr/lib/python2.6/site-packages/suds/reader.py", line 152, in open
d = self.fn(url, self.options)
File "/usr/lib/python2.6/site-packages/suds/wsdl.py", line 136, in __init__
d = reader.open(url)
File "/usr/lib/python2.6/site-packages/suds/reader.py", line 79, in open
d = self.download(url)
File "/usr/lib/python2.6/site-packages/suds/reader.py", line 101, in download
return sax.parse(string=content)
File "/usr/lib/python2.6/site-packages/suds/sax/parser.py", line 136, in parse
sax.parse(source)
File "/usr/lib64/python2.6/xml/sax/expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib64/python2.6/xml/sax/xmlreader.py", line 123, in parse
self.feed(buffer)
File "/usr/lib64/python2.6/xml/sax/expatreader.py", line 211, in feed
self._err_handler.fatalError(exc)
File "/usr/lib64/python2.6/xml/sax/handler.py", line 38, in fatalError
raise exception
xml.sax._exceptions.SAXParseException: <unknown>:1:1: not well-formed (invalid token)
I have cut the WSDL because clarity. But I have also a Java client which use the SOAP service and can read and parse the WSDL successfully.
Anybody knows what's can be wrong?

I found I have to replace the following lines. The commented out code is the old code, the uncommented line the new new code:
#import io
import StringIO
...
# result = io.StringIO(resp.content.decode('utf-8'))
result = StringIO.StringIO(resp.content)
Not sure why, but this works for me (changing only io.StringIO(resp.content) doesn't work).

how to prevent SUDS from checking WSDL

Now I am writing simple OpenERP module for my customer. I use SUDS to connect with bank, to get Statements.
I wrote xml request, that works without problem. I get response from bank, that also looks ok. Problem is that response from bank use type, that is not defined in WSDL (I wrote to bank support, that they have a bug).
Traceback (most recent call last):
File "example3.py", line 112, in <module>
wiadomosc = client.service.GetStatement(__inject={'msg': xml})
File "/usr/lib/python2.7/dist-packages/suds/client.py", line 542, in __call__
return client.invoke(args, kwargs)
File "/usr/lib/python2.7/dist-packages/suds/client.py", line 773, in invoke
return self.send(msg)
File "/usr/lib/python2.7/dist-packages/suds/client.py", line 647, in send
result = self.succeeded(binding, reply.message)
File "/usr/lib/python2.7/dist-packages/suds/client.py", line 684, in succeeded
reply, result = binding.get_reply(self.method, reply)
File "/usr/lib/python2.7/dist-packages/suds/bindings/binding.py", line 156, in get_reply
result = self.replycomposite(rtypes, nodes)
File "/usr/lib/python2.7/dist-packages/suds/bindings/binding.py", line 230, in replycomposite
sobject = unmarshaller.process(node, resolved)
File "/usr/lib/python2.7/dist-packages/suds/umx/typed.py", line 66, in process
return Core.process(self, content)
File "/usr/lib/python2.7/dist-packages/suds/umx/core.py", line 48, in process
return self.append(content)
File "/usr/lib/python2.7/dist-packages/suds/umx/core.py", line 63, in append
self.append_children(content)
File "/usr/lib/python2.7/dist-packages/suds/umx/core.py", line 140, in append_children
cval = self.append(cont)
File "/usr/lib/python2.7/dist-packages/suds/umx/core.py", line 61, in append
self.start(content)
File "/usr/lib/python2.7/dist-packages/suds/umx/typed.py", line 80, in start
raise TypeNotFound(content.node.qname())
suds.TypeNotFound: Type not found: 'ns29:BkToCstmrStmt'
And types, they provide
....
ns37:BaselineStatus3Code
ns32:BatchBookingIndicator
ns33:BatchBookingIndicator
ns36:BatchBookingIndicator
ns30:BatchInformation1
ns31:BatchInformation2
ns29:BatchInformation2
ns39:BilateralLimitDetails3
ns27:BkToCstmrCardRptType
ns27:BkTxCdType
ns27:BookgDtType
ns10:BookingDate
ns4:Bool
ns16:Bool
ns2:Bool
ns15:Bool
ns17:BouldingNumber
....
So there is no BkToCstmrStmt. How can I make suds to just get response from server, and not analyse it? Just build tree?
Thank you

Still I don't know what is wrong with my code. I solved it in that way:
class MessageInterceptor(suds.plugin.MessagePlugin):
def __init__(self, *args, **kwargs):
self.message = None
def received(self, context):
#recieved xml as a string
#print "%s bytes received" % len(context.reply)
self.message = context.reply
#clean up reply to prevent parsing
context.reply = ""
return context
message_interceptor = MessageInterceptor()
client = Client('https://some-adress-to?wsdl',plugins=[message_interceptor])
So now I can call client method
xml = Raw("""
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
<soapenv:Header>
...
</soapenv:Header>
<soapenv:Body>
<urn1:GetStatement>
...
</urn1:GetStatement>
</soapenv:Body>
</soapenv:Envelope>
""")
response = client.service.GetStatement(__inject={'msg': xml})
Now suds thinks that got nothing from server. But message we recieved is stored in
message_interceptor.message
Now to get normal dict object from message I do it like that:
import xmltodict
message_interceptor.message = message_interceptor.message.replace('ns17:','')
message_interceptor.message = message_interceptor.message.replace('ns40:','')
message_interceptor.message = message_interceptor.message.replace('soap:','')
response = xmltodict.parse(message_interceptor.message)['Envelope']['Body']['GetStatementResponse']['Document']
Now I can use response as normal response from suds.

Running Bram Cohen's original BitTorrent sources (Python)

I found an old source of Bram Cohen's original BitTorrent here:
http://bittorrent.cvs.sourceforge.net/viewvc/bittorrent/?hideattic=0
(says here: Where to find BitTorrent source code? that it's version 3.x)
and I'm trying to run it on my Mac (10.7) and my Python version is 2.7
If you would try and download the source, you can try running the btdownloadcurses.py or the btdownloadheadless.py
I tried running:
$ ./btdownloadcurses.py --url http://sometorrenthost/somefile.torrent
Ok, I'll be more specific. This is what I did:
$ ./btdownloadcurses.py --url http://torcache.net/torrent/848A6A0EC6C85507B8370E979B133214E5B5A6D4.torrent
And this is what I got:
Traceback (most recent call last):
File "./btdownloadcurses.py", line 243, in <module>
run(mainerrlist, argv[1:])
File "./btdownloadcurses.py", line 186, in run
download(params, d.chooseFile, d.display, d.finished, d.error, mainkillflag, fieldw)
File "/Users/joal21/Desktop/BitTorrent/BitTorrent/download.py", line 120, in download
h = urlopen(config['url'])
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 517, in http_response
code, msg, hdrs = response.code, response.msg, response.info()
AttributeError: addinfourldecompress instance has no attribute 'msg'
When I searched for that AttributeError, I came to:
http://mail.python.org/pipermail/python-bugs-list/2005-May/028824.html
I think the second comment has something to do with my problem. But I don't know how else to procees from there. Am I simply just passing the wrong url? Does it have something to do with the Python version? Or of the BitTorrent source being old. Or is there something new in present .torrent files. What am I missing? Not doing?
Forgive my ignorance. I'm really at a loss here.

Bram worked against an older version of Python, one where the urllib2 code did not add .msg and .code attributes to addinfourl objects. Specifically, the Python version he developed with did not have this change applied.
The workaround is to copy those attributes yourself from the original addinfourl object in the HTTPContentEncodingHandler class found in the original zurllib.py file:
class HTTPContentEncodingHandler(HTTPHandler):
"""Inherit and add gzip/deflate/etc support to HTTP gets."""
def http_open(self, req):
# add the Accept-Encoding header to the request
# support gzip encoding (identity is assumed)
req.add_header("Accept-Encoding","gzip")
req.add_header('User-Agent', 'BitTorrent/' + version)
if DEBUG:
print "Sending:"
print req.headers
print "\n"
fp = HTTPHandler.http_open(self,req)
headers = fp.headers
if DEBUG:
pprint.pprint(headers.dict)
url = fp.url
resp = addinfourldecompress(fp, headers, url)
resp.code = fp.code
resp.msg = fp.msg
return resp

Nested text encodings in suds requests

Environment: Python 2.7.4 (partly on Windows, partly on Linux, see below), suds (SVN HEAD with minor modifications)
I need to call into a web service that takes a single argument, which is an XML string (yes, I know…), i.e. the request is declared in the WSDL with the following type:
<s:complexType>
<s:sequence>
<s:element minOccurs="0" maxOccurs="1" name="actionString" type="s:string"/>
</s:sequence>
</s:complexType>
I'm using cElementTree to construct this inner XML document, then I pass it as the only parameter to the client.service.ProcessAction(request) method that suds generates.
For a while, this worked okay:
root = ET.Element(u'ActionCommand')
value = ET.SubElement(root, u'value')
value.text = saxutils.escape(complex_value)
request = u'<?xml version="1.0" encoding="utf-8"?>\n' + ET.tostring(root, encoding='utf-8')
client.service.ProcessAction(request)
The saxutils.escape, I had added at some point to fix the first encoding problems, pretty much without being able to understand why exactly I need it and what difference it makes.
Now (possibly due to the first occurence of the pound sign), I suddenly got the following exception:
Traceback (most recent call last):
File "/app/module.py", line 135, in _process_web_service_call
request = u'<?xml version="1.0" encoding="utf-8"?>\n' + ET.tostring(root, encoding='utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 137: ordinal not in range(128)
The position 137 here corresponds to the location of the special characters inside the inner XML request. Apparently, cElementTree.tostring() returns a 'str' type, not a 'unicode' even when an encoding is given. So Python tries to decode this string str into unicode (why with 'ascii'?), so that it can concatenate it with the unicode literal. This fails (of course, because the str is actually encoded in UTF-8, not ASCII).
So I figured, fine, I'll decode it to unicode myself then:
root = ET.Element(u'ActionCommand')
value = ET.SubElement(root, u'value')
value.text = saxutils.escape(complex_value)
request_encoded_str = ET.tostring(root, encoding='utf-8')
request_unicode = request_encoded_str.decode('utf-8')
request = u'<?xml version="1.0" encoding="utf-8"?>\n' + request_unicode
client.service.ProcessClientAction(request)
Except that now, it blows up inside suds, which tries to decode the outer XML request for some reason:
Traceback (most recent call last):
File "/app/module.py", line 141, in _process_web_service_call
raw_response = client.service.ProcessAction(request)
File "/app/.heroku/python/lib/python2.7/site-packages/suds/client.py", line 542, in __call__
return client.invoke(args, kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/suds/client.py", line 602, in invoke
result = self.send(soapenv)
File "/app/.heroku/python/lib/python2.7/site-packages/suds/client.py", line 643, in send
reply = transport.send(request)
File "/app/.heroku/python/lib/python2.7/site-packages/suds/transport/https.py", line 64, in send
return HttpTransport.send(self, request)
File "/app/.heroku/python/lib/python2.7/site-packages/suds/transport/http.py", line 118, in send
return self.invoke(request)
File "/app/.heroku/python/lib/python2.7/site-packages/suds/transport/http.py", line 153, in invoke
u2response = urlopener.open(u2request, timeout=tm)
File "/app/.heroku/python/lib/python2.7/urllib2.py", line 404, in open
response = self._open(req, data)
File "/app/.heroku/python/lib/python2.7/urllib2.py", line 422, in _open
'_open', req)
File "/app/.heroku/python/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/app/.heroku/python/lib/python2.7/urllib2.py", line 1222, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/app/.heroku/python/lib/python2.7/urllib2.py", line 1181, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "/app/.heroku/python/lib/python2.7/httplib.py", line 973, in request
self._send_request(method, url, body, headers)
File "/app/.heroku/python/lib/python2.7/httplib.py", line 1007, in _send_request
self.endheaders(body)
File "/app/.heroku/python/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/app/.heroku/python/lib/python2.7/httplib.py", line 827, in _send_output
msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 565: ordinal not in range(128)
The position 565 here again corresponds with the same character as above, except this time it's the location of my inner XML request embedded into the outer XML request (SOAP) created by suds.
I'm confused. Can anyone help me out of this mess? :)
To make matters worse, all of this only happens on the server under Linux. None of these raises an exception in my development environment on Windows. (Bonus points for an explanation as to why that is, just because I'm curious. I suspect it has to do with a different default encoding.) However, they all are not accepted by the server. What does work on Windows is if I drop the saxutils.escape and then hand a proper unicode object to suds. This however still results in the same UnicodeDecodeError on Linux.
Update: I started debugging this on Windows (where it works fine), and in the line 827 of httplib.py, it indeed tries to concatenate the unicode object msg (containing the HTTP headers) and the str object message_body, leading to the implicit unicode decoding with the incorrect encoding. I guess it just happens to not fail on Windows for some reason. I don't understand why suds tries to send a str object when I put a unicode object in at the top.

This turned out to be more than absurd. I'm still understanding only small parts of the whole problem and situation, but I managed to solve my problem.
So let's trace it back: my last attempt was the most sane one, I believe. So let's start there:
msg += message_body
That line in Python's httplib.py tries to concatenate a unicode and a str object, which leads to an implicit .decode('ascii') of the str, even though the str is UTF8-encoded. Why is that? Because msg is a unicode object.
msg = "\r\n".join(self._buffer)
self._buffer is a list of HTTP headers. Inspecting that, only one header in there was unicode, 'infecting' the resulting string: the action and endpoint.
And there's the problem: I'm using unicode_literals from __future__ (makes it more future-proof, right? right???) and I'm passing my own endpoint into suds.
By just doing an .encode('utf-8') on the URL, all my problems went away. Even the whole saxutils.escape was no longer needed (even though it weirdly also didn't hurt).
tl;dr: make sure you're not passing any unicode objects anywhere into httplib or suds, I guess.
root = ET.Element(u'ActionCommand')
value = ET.SubElement(root, u'value')
value.text = complex_value)
request = ET.tostring(root, encoding='utf-8').decode('utf-8')
client.service.ProcessAction(request)

Python BadStatusLine error on certain DELETE requests

I am trying to use the python-rest-client ( http://code.google.com/p/python-rest-client/wiki/Using_Connection ) to perform testing of some RESTful webservices. Since I'm just learning, I've been pointing my tests at the sample services provided at http://www.predic8.com/rest-demo.htm.
I have no problems with creating entries, updating entries, or retrieving entries (POST and GET requests). When I try make a DELETE request, it fails. I can use the Firefox REST Client to perform DELETE requests and they work. I can also make DELETE requests on other services, but I've been driving myself crazy trying to figure out why it doesn't work in this case. I'm using Python 3 with updated Httplib2, but I also tried Python 2.5 so that I could use the python-rest-client with the included version of Httplib2. I see the same problem in either case.
The code is simple, matching the documented use:
from restful_lib import Connection
self.base_url = "http://www.thomas-bayer.com"
self.conn = Connection(self.base_url)
response = self.conn.request_delete('/sqlrest/CUSTOMER/85')
I've looked at the resulting HTTP requests from the browser tool and from my code and I can't see why one works and the other doesn't. This is the trace I receive:
Traceback (most recent call last):
File "/home/fmk/python/rest-client/src/TestExampleService.py", line 68, in test_CRUD
self.Delete()
File "/home/fmk/python/rest-client/src/TestExampleService.py", line 55, in Delete
response = self.conn.request_delete('/sqlrest/CUSTOMER/85')
File "/home/fmk/python/rest-client/src/restful_lib.py", line 64, in request_delete
return self.request(resource, "delete", args, headers=headers)
File "/home/fmk/python/rest-client/src/restful_lib.py", line 138, in request
resp, content = self.h.request("%s://%s%s" % (self.scheme, self.host, '/'.join(request_path)), method.upper(), body=body, headers=headers )
File "/home/fmk/python/rest-client/src/httplib2/__init__.py", line 1175, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/home/fmk/python/rest-client/src/httplib2/__init__.py", line 931, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/home/fmk/python/rest-client/src/httplib2/__init__.py", line 897, in _conn_request
response = conn.getresponse()
File "/usr/lib/python3.2/http/client.py", line 1046, in getresponse
response.begin()
File "/usr/lib/python3.2/http/client.py", line 346, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.2/http/client.py", line 316, in _read_status
raise BadStatusLine(line)
http.client.BadStatusLine: ''
What's breaking? What do I do about it? Actually, I'd settle for advice on debugging it. I've changed the domain in my script and pointed it at my own machine so I could view the request. I've viewed/modified the Firefox requests in BurpProxy to make them match my script requests. The modified Burp requests still work and the Python requests still don't.

Apparently the issue is that the server expects there to be some message body for DELETE requests. That's an unusual expectation for a DELETE, but by specifying Content-Length:0 in the headers, I'm able to successfully perform DELETEs.
Somewhere along the way (in python-rest-client or httplib2), the Content-Length header is wiped out if I try to do:
from restful_lib import Connection
self.base_url = "http://www.thomas-bayer.com"
self.conn = Connection(self.base_url)
response = self.conn.request_delete('/sqlrest/CUSTOMER/85', headers={'Content-Length':'0'})
Just to prove the concept, I went to the point in the stack trace where the request was happening:
File "/home/fmk/python/rest-client/src/httplib2/__init__.py", line 897, in _conn_request
response = conn.getresponse()
I printed the headers parameter there to confirm that the content length wasn't there, then I added:
if(method == 'DELETE'):
headers['Content-Length'] = '0'
before the request.
I think the real answer is that the service is wonky, but at least I got to know httplib2 a little better. I've seen some other confused people looking for help with REST and Python, so hopefully I'm not the only one who got something out of this.

The following script correctly produces 404 response from the server:
#!/usr/bin/env python3
import http.client
h = http.client.HTTPConnection('www.thomas-bayer.com', timeout=10)
h.request('DELETE', '/sqlrest/CUSTOMER/85', headers={'Content-Length': 0})
response = h.getresponse()
print(response.status, response.version)
print(response.info())
print(response.read()[:77])
python -V => 3.2
curl -X DELETE http://www.thomas-bayer.com/sqlrest/CUSTOMER/85
curl: (52) Empty reply from server
Status-Line is not optional; HTTP server must return it. Or at least send 411 Length Required response.
curl -H 'Content-length: 0' -X DELETE \
http://www.thomas-bayer.com/sqlrest/CUSTOMER/85
Returns correctly 404.