I'm trying to POST data to a URL using Pythons requests library.
If I try and do this by setting a multiline string variable which contains the post data in my script, everything works fine.
If I try to read in an external file with the same data in, the request fails on the application server I'm posting to, because it thinks there is invalid XML.
For example:
This works
starturl="http://myserver.example.com/location/where/I/post"
username=user
password=mypassword
# Set the XML data
xmldata="""<?xml version="1.0" encoding="utf-8"?>
(Lots more xml)
"""
# POST the job data
session = requests.Session()
request = session.post(starturl, auth=(username,password), data=xmldata, headers=post_headers)
Server side application processes the request just fine. However, if the only change I make is to read the xml data from an external file, this no longer works.
This does not work
xmlfile="/path/to/my/xmldata.xml"
xmldata = open(xmlfile,'r')
session = requests.Session()
request = session.post(start_url, auth=(username,password), data=xmldata.read(), headers=post_headers)
The server side application, then errors with:
"Data at the root level is invalid. Line 1, position 1"
When inspecting with wireshark I can see there is a difference in the request body of my POST. Three little dots are appearing from somewhere
When it works:
Content-Type: application/xml
Authorization: Basic c3BvdGFkbTpQQHNzdzByZA==
<?xml version="1.0" encoding="utf-8"?>
When it fails:
Content-Type: application/xml
Authorization: Basic c3BvdGFkbTpQQHNzdzByZA==
...<?xml version="1.0" encoding="utf-8"?>
I'm not sure what's causing the 3 leading dots to appear in the request body. I've inspected the source XML file, tried stripping newlines from it. Nothing seems to do the trick?
It's impossible to tell for sure without having your xml file, but you might have a BOM at the beginning of your file. Microsoft is notably (in)famous for insisting on putting useless BOM on all utf-8 files.
You can check the first three characters of your file for the codecs.BOM_UTF8 sequence ('\xef\xbb\xbf') and strip it out if it's there.
Related
TPA.wsdl https://pastebin.com/7DBhCHbv
DataService.xsd https://pastebin.com/AFhg64hH
from zeep import Client
import base64
from requests import Session
from zeep.wsse.username import UsernameToken
from zeep.transports import Transport
from zeep.exceptions import Fault
Username = '....'
Password = '....'
sendFile = 'V07_220110.ffdata'
session = Session()
session.verify = False
try:
wsdl = 'TPA.wsdl'
# initialize zeep client
client = Client(
wsdl=wsdl,
wsse=UsernameToken(Username, Password),
transport=Transport(session=session)
)
with open(sendFile, "rb") as pdf_file:
encoded_string = base64.b64encode(pdf_file.read())
with client.options(raw_response=True):
node = client.service.uploadEdasDraft(sendFile, encoded_string )
print(node.content)
except Fault as fault:
parsed_fault_detail = client.wsdl.types.deserialize(fault.detail[0])
Always getting Response
I got error ORA-31011: XML parsing failed
From SOAPUI everything sending ok with Enable MTOM settings
So how to make request to it, and how debug sending requests?
Based on that WSDL file, a code like this:
from zeep import Client
from zeep.wsse.username import UsernameToken
username = 'username'
password = 'p#$$word'
file_name = 'test.txt'
client = Client(wsdl = 'TPA.wsdl',
wsse = UsernameToken(username, password))
with open(file_name, "rb") as f:
content = f.read()
client.service.uploadEdasDraft(file_name, content)
should produce something like this:
<soap-env:Envelope xmlns:soap-env="http://www.w3.org/2003/05/soap-envelope">
<soap-env:Header>
<wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
<wsse:UsernameToken>
<wsse:Username>username</wsse:Username>
<wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">p#$$word</wsse:Password>
</wsse:UsernameToken>
</wsse:Security>
</soap-env:Header>
<soap-env:Body>
<ns0:uploadEdasDraft xmlns:ns0="http://types.data.external.ws.edas.sodra.epr.lt">
<fileName>test.txt</fileName>
<document>dGhpcyBpcyBhIHRlc3QNCnRoaXMgaXMgYSB0ZXN0DQp0aGlzIGlzIGEgdGVzdA0KdGhpcyBpcyBhIHRlc3QNCg==</document>
</ns0:uploadEdasDraft>
</soap-env:Body>
</soap-env:Envelope>
Since the document type is marked as xsd:base64Binary zeep should handle the base64 encoding for you (your code seems to be doing the encoding twice).
In the example above I'm using a text file, but I assume your file named V07_220110.ffdata is an XML file, since that's what this attribute says: xmime:expectedContentTypes="application/xml". Server will probably complain if you don't send a file with this content type. This may also be a possible cause for that "ORA-31011: XML parsing failed" message, together with the double encoding (server is expecting XML in the document but finds another base64 string).
From SOAPUI everything sending ok with Enable MTOM settings
When using MTOM, your file is not encoded within the SOAP message as text, but it's attached as binary next to it, and you get references to that part of the message. See an explanation here: How does MTOM work?
Your document element might change to something like:
<document>
<inc:Include href="cid:123456789" xmlns:inc="http://www.w3.org/2004/08/xop/include"/>
</document>
where 123456789 is a reference to the content id in the multipart message.
Does your call work from SoapUI only if you enable MTOM? Have you tried sending a base64 encoded file string within the SOAP message using SoapUI?
If your call only works with MTOM then you have a problem, because I'm not sure zeep can handle it out of the box.
The documentation (https://docs.python-zeep.org/en/master/attachments.html) mentions only a multipart response and how you can read the file from the response, but says nothing about making requests. See for example these items:
https://github.com/mvantellingen/python-zeep/issues/599
https://github.com/mvantellingen/python-zeep/pull/314
https://github.com/remaudcorentin-dev/python-zeep-adv
The last project might help you with some code samples (https://github.com/remaudcorentin-dev/python-zeep-adv/blob/master/src/zeep/transport_with_attach.py) but do consider the caveat:
This has been developed for a specific usage and this code should probably not be used (has it) for other puposes.
(or at your own risks ;) )
So it seems you might have to build your own multipart request from scratch or from that sample in the project.
[...] and how debug sending requests?
You might use the HistoryPlugin just to see what messages get exchanged between your client and server, but since you might need to see all of the request, I suggest Wireshark, TcpMon (old, defunct, but still useful), or SoapUI with TcpMon.
This might not be the answer you are looking for, but hope that at least it leaves you more equipped in figuring out how to make the call. Zeep unfortunately is a small fish client in a large pond of WS specifications.
I am able to send a simple SOAP request with Zeep.
with client.settings(strict=False):
resp = client.service.demandeFicheProduit(
demandeur=self.xxx, motDePasse=self.yyy,
ean13s="foo",
multiple=False)
However, I need to give multiple times the ean13s argument, which is not possible in a Python function call, so I figured I need to build the XML myself.
With Zeep's debug on, I see that the XML sent is this:
<?xml version='1.0' encoding='utf-8'?>
<soap-env:Envelope xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/">
<soap-env:Body>
<ns0:demandeFicheProduit xmlns:ns0="http://fel.ws.accelya.com/">
<demandeur>xxx
</demandeur>
<motDePasse>yyy
</motDePasse>
<ean13s>foo
</ean13s>
<multiple>false
</multiple>
</ns0:demandeFicheProduit>
</soap-env:Body>
</soap-env:Envelope>
So I only need to replicate the
<ean13s>foo
</ean13s>
part.
Looking into Zeep, I see a Transport.post_xml method: https://github.com/mvantellingen/python-zeep/blob/da8a88b9f5/src/zeep/transports.py#L86 which takes an lxml tree as parameter. (doc)
def post_xml(self, address, envelope, headers):
"""Post the envelope xml element to the given address with the headers.
This method is intended to be overriden if you want to customize the
serialization of the xml element. By default the body is formatted
and encoded as utf-8. See ``zeep.wsdl.utils.etree_to_string``.
"""
message = etree_to_string(envelope)
return self.post(address, message, headers)
I tried a post_raw_xml method, without etree_to_string:
def post_raw_xml(self, address, raw_envelope, headers):
return self.post(address, raw_envelope, headers)
I call it with the above XML
transport = zeep.Transport()
transport.post_raw_xml("adress", my_xml, {}) # {}: headers?
and the response status is OK (200), however the service answers it is an invalid request.
Are there XML / SOAP intricacies I didn't pay attention to? Encoding? Headers? (here {})
edit: after spying a bit more the Zeep internals, the headers it sends are
{'SOAPAction': '""', 'Content-Type': 'text/xml; charset=utf-8'}
So I thought I could simply use requests.post, to no avail yet. To use requests, see Sending SOAP request using Python Requests
How to otherwise build an XML manually?
Any more idea on how I can duplicate the eans13 argument?
Thank you.
I solved my problem without Zeep.
I send a raw XML string with requests, I don't construct an XML manually with lxml or xsd.
What helped me was print-debugging the internals of Zeep's transport.py/post method (so the headers are {'SOAPAction': '""', 'Content-Type': 'text/xml; charset=utf-8'}
),
this other question: Sending SOAP request using Python Requests to send a post request:
requests.post(url, data=xml, headers=headers)
and being careful on the XML string (yes there is a newline character after the closing <?xml> tag).
The downside is to parse the XML back.
Here is my FileUploadView class to handle POST request of an uploaded file. The file I am expecting are XML Files in which I use ElementTree to parse through it in fileHandler(). However, when using Postman to send a file through using ('form-data'), I realized that it is attaching some type of header to my uploaded file, which in turn causes the tree parse() to have a syntax error since its reading something that is not of an XML format.
I tried using HTTPie to send the file through, which worked with no issue, the XML Parser parsed it correctly and entered the data into the expected Object.
I then tried to do some TestCases with Django, and tried to test the fileupload. Which caused the parser to have a syntax error again due to having a header attached to the file once more.
class UploadTest(APITestCase):
def test_file_upload(self):
c = Client()
with open("/Users/Ren/Desktop/Capstone/Backend/projectB/VMA/testing/Test.xml") as fp:
c.post('/upload/TestXML.xml', {'filename' : 'Test.xml', 'attachment': fp})
My question is: What is causing that header to pop up/be added onto the uploaded file. I'm guessing it has something to do with how I am sending the post request through Postman and the Django TestCase which is different to HTTPie
view.py
class FileUploadView(APIView):
parser_classes = (FileUploadParser,)
def post(self, request, filename, format=None):
print(request.FILES)
file_obj = request.FILES['file']
fileHandler(file_obj)
return Response(status=204)
FileReader.py
def fileHandler(file):
filepath = file.temporary_file_path()
print(file.read())
tree = ET.parse(filepath)
root = tree.getroot()
XML File and output when calling file.read()
XML I need to read in (Expected Output):
<site host="192.168.212.4" name="http://192.168.212.4" port="80" ssl="false"><alerts><alertitem>\n <pluginid>10021</pluginid>\n <alert>X-Content-Type-Options header missing</alert>\n <riskcode>1</riskcode>\n <reliability>2</reliability>\n <riskdesc>Low (Warning)</riskdesc>\n <desc>The Anti-MIME-Sniffing header X-Content-Type-Options was not set to \'nosniff\'.\n\tThis allows older versions of Internet Explorer and Chrome to perform MIME-sniffing on the response body, potentially causing the response body to be interpreted and displayed as a content type other than the declared content type.\n\tCurrent (early 2014) and legacy versions of Firefox will use the declared content type (if one is set), rather than performing MIME-sniffing.\n\t</desc>\n <uri>http://192.168.212.4/</uri>\n <param/>\n <attack/>\n <otherinfo/>\n <solution>Ensure that the application/web server sets the Content-Type header appropriately, and that it sets the X-Content-Type-Options header to \'nosniff\' for all web pages.\n\tIf possible, ensure that the end user uses a standards-compliant and modern web browser that does not perform MIME-sniffing at all, or that can be directed by the web application/web server to not perform MIME-sniffing.\n\t</solution>\n <reference>\n\t</reference>\n</alertitem>
The Output when running request.FILES['file'].read() --- Current Output
b'----------------------------507481440966899800347275\r\nContent-Disposition: form-data; name=""; filename="sampleXML.xml"\r\nContent-Type: application/xml\r\n\r\n<site host="192.168.212.4" name="http://192.168.212.4" port="80" ssl="false"><alerts><alertitem>\n <pluginid>10021</pluginid>\n <alert>X-Content-Type-Options header missing</alert>\n <riskcode>1</riskcode>\n <reliability>2</reliability>\n <riskdesc>Low (Warning)</riskdesc>\n <desc>The Anti-MIME-Sniffing header X-Content-Type-Options was not set to \'nosniff\'.\n\tThis allows older versions of Internet Explorer and Chrome to perform MIME-sniffing on the response body, potentially causing the response body to be interpreted and displayed as a content type other than the declared content type.\n\tCurrent (early 2014) and legacy versions of Firefox will use the declared content type (if one is set), rather than performing MIME-sniffing.\n\t</desc>\n <uri>http://192.168.212.4/</uri>\n <param/>\n <attack/>\n <otherinfo/>\n <solution>Ensure that the application/web server sets the Content-Type header appropriately, and that it sets the X-Content-Type-Options header to \'nosniff\' for all web pages.\n\tIf possible, ensure that the end user uses a standards-compliant and modern web browser that does not perform MIME-sniffing at all, or that can be directed by the web application/web server to not perform MIME-sniffing.\n\t</solution>\n <reference>\n\t</reference>\n</alertitem>\n\n \r\n----------------------------507481440966899800347275--\r\n'
Contains the unnecessary: b'----------------------------507481440966899800347275\r\nContent-Disposition: form-data; name=""; filename="sampleXML.xml"\r\nContent-Type: application/xml\r\n\r\n
I played around with code for a bit and made a tiny change into the testCase:
class UploadTest(APITestCase):
def test_file_upload(self):
c = Client()
with open("/Users/Ren/Desktop/Capstone/Backend/projectB/VMA/testing/Test.xml") as fp:
c.post('/upload/TestXML.xml', {'filename' : 'Test.xml', 'attachment': fp})
I changed the
{'filename' : 'Test.xml', 'attachment': fp}
to
{'filename' : b'Test.xml', 'attachment': fp}
I remember reading it somewhere, unfortunately I do not remember where but... turning the file into "bytes" fixed it...
I'm trying to make a POST request that sends XML data to a 3rd party server, it works with cmd line curl but not the python requests library (which I need to work). I've omitted out the username, password, and domain urls. I'm using requests==2.2.1
CURL EXAMPLE (WORKS):
Request:
$ curl --data '<?xml version="1.0" encoding="UTF-8"?><request><authentication partnerid="12345" password="mypass1234" /><method name="GetTestList" /></request>' 'http://dev.expertrating.com/mydomain/webservices/'
Response:
<response><result name="GetTestList"><records><record test_id="6683" test_name="Demo Adobe Photoshop CS3 Test" coverage="Layers ;Type ;Automating Tasks and Keyboard Shortcuts ;Workspace ;Working with Images, Retouching and Transforming ;Color Management ;Color and Tonal Adjustments ;Using Selection and Drawing ;Filters and Saving Images ;Working With Web, Video and Animation" total_questions="10" duration="10" passing_marks="60" category="Demo Tests" /><record test_id="6684" test_name="Demo Programming with C++ Test" coverage="Classes ;Constructors ;Variables and Datatypes" total_questions="10" duration="10" passing_marks="60" category="Demo Tests" /><record test_id="6685" test_name="Demo HTML 4.01 Test" coverage="Advanced Tags ;Fundamentals ;Tags ;Tables ;Links and Images ;Forms and Frames" total_questions="10" duration="10" passing_marks="60" category="Demo Tests" /><record test_id="6682" test_name="Demo Editing Skills Certification" coverage="Prepositions ;Parallel Construction ;Comma Usage ;Punctuation ;Misplaced Modifiers ;Apostrophe Usage ;Sentence Structure ;Pronoun Usage ;Tenses ;Article Usage" total_questions="10" duration="10" passing_marks="60" category="Demo Tests" /></records></result></response>
REQUESTS EXAMPLE: (FAILS) (via ipython/django)
IPython:
In [1]: import requests
In [2]: payload = '<?xml version="1.0" encoding="UTF-8"?><request><authentication partnerid="12345" password="mypass1234" /><method name="GetTestList" /></request>'
In [3]: r = requests.post('http://dev.expertrating.com/mydomain/webservices/', data=payload, headers={'Content-Type': 'application/xml'})
In [4]: r.text
Out[4]: u"<?xml version='1.0' encoding='UTF-8'?><response><error id='101' message='Invalid Request'><info>You have not posted a valid request.</info></error></response>"
Am I encoding the requests string wrong? Including the wrong headers? I literally copy/pasted the urls and payload xml. I also tried POSTing with urllib2 with similar failing results.
Any help will be greatly appreciated, thanks in advance to anyone who reads this.
It worked by setting the Content-Type header to Content-Type: application/x-www-form-urlencoded instead of Content-Type: application/xml.
I'm trying to replace curl with Python & the requests library. With curl, I can upload a single XML file to a REST server with the curl -T option. I have been unable to do the same with the requests library.
A basic scenario works:
payload = '<person test="10"><first>Carl</first><last>Sagan</last></person>'
headers = {'content-type': 'application/xml'}
r = requests.put(url, data=payload, headers=headers, auth=HTTPDigestAuth("*", "*"))
When I change payload to a bigger string by opening an XML file, the .put method hangs (I use the codecs library to get a proper unicode string). For example, with a 66KB file:
xmlfile = codecs.open('trb-1996-219.xml', 'r', 'utf-8')
headers = {'content-type': 'application/xml'}
content = xmlfile.read()
r = requests.put(url, data=content, headers=headers, auth=HTTPDigestAuth("*", "*"))
I've been looking into using the multipart option (files), but the server doesn't seem to like that.
So I was wondering if there is a way to simulate curl -T behaviour in Python requests library.
UPDATE 1:
The program hangs in textmate, but throws an UnicodeEncodeError error on the commandline. Seems that must be the problem. So the question would be: is there a way to send unicode strings to a server with the requests library?
UPDATE 2:
Thanks to the comment of Martijn Pieters the UnicodeEncodeError went away, but a new issue turned up.
With a literal (ASCII) XML string, logging shows the following lines:
2012-11-11 15:55:05,154 INFO Starting new HTTP connection (1): my.ip.address
2012-11-11 15:55:05,294 DEBUG "PUT /v1/documents?uri=/example/test.xml HTTP/1.1" 401 211
2012-11-11 15:55:05,430 DEBUG "PUT /v1/documents?uri=/example/test.xml HTTP/1.1" 201 0
Seems the server always bounces the first authentication attempt (?) but then accepts the second one.
With a file object (open('trb-1996-219.xml', 'rb')) passed to data, the logfile shows:
2012-11-11 15:50:54,309 INFO Starting new HTTP connection (1): my.ip.address
2012-11-11 15:50:55,105 DEBUG "PUT /v1/documents?uri=/example/test.xml HTTP/1.1" 401 211
2012-11-11 15:51:25,603 WARNING Retrying (0 attempts remain) after connection broken by 'BadStatusLine("''",)': /v1/documents?uri=/example/test.xml
So, first attempt is blocked as before, but no second attempt is made.
According to Martijn Pieters (below), the second issue can be explained by a faulty server (empty line).
I will look into this, but if someone has a workaround (apart from using curl) I wouldn't mind hearing it.
And I am still surprised that the requests library behaves so differently for small string and file object. Isn't the file object serialized before it gets to the server anyway?
To PUT large files, don't read them into memory. Simply pass the file as the data keyword:
xmlfile = open('trb-1996-219.xml', 'rb')
headers = {'content-type': 'application/xml'}
r = requests.put(url, data=xmlfile, headers=headers, auth=HTTPDigestAuth("*", "*"))
Moreover, you were opening the file as unicode (decoding it from UTF-8). As you'll be sending it to a remote server, you need raw bytes, not unicode values, and you should open the file as a binary instead.
Digest authentication always requires you to make at least two request to the server. The first request doesn't contain any authentication data. This first request will fail with a 401 "Authorization required" response code and a digest challenge (called a nounce) to be used for hashing your password etc. (the exact details don't matter here). This is used to make a second request to the server containing your credentials hashed with the challenge.
The problem is in the this two step authentication: your large file was already send with the first unauthorized request (send in vain) but on the second request the file object is already at the EOF position. Since the file size was also send in the Content-length header of the second request, this causes the server to wait for a file that will never be send.
You could solve it using a requests Session and first make a simple request for authentication purposes (say a GET request). Then make a second PUT request containing the actual payload using the same digest challenge form the first request.
sess = requests.Session()
sess.auth = HTTPDigestAuth("*", "*")
sess.get(url)
headers = {'content-type': 'application/xml'}
with codecs.open('trb-1996-219.xml', 'r', 'utf-8') as xmlfile:
sess.put(url, data=xmlfile, headers=headers)
i used requests in python to upload an XML file using the commands.
first to open the file use open()
file = open("PIR.xsd")
fragment = file.read()
file.close()
copy the data of XML file in the payload of the requests and post it
payload = {'key':'PFAkrzjmuZR957','xmlFragment':fragment}
r = requests.post(URL,data=payload)
to check the html validation code
print (r.text)