I have a problem with getting my test running using Robot Framework and robotframework-requests. I need to send a POST request and a binary data in the body. I looked at this question already, but it's not really answered. Here's how my test case looks like:
Upload ${filename} file
Create Session mysession http://${ADDRESS}
${data} = Get Binary File ${filename}
&{headers} = Create Dictionary Content-Type=application/octet-stream Accept=application/octet-stream
${resp} = Post Request mysession ${CGIPath} data=${data} headers=&{headers}
[Return] ${resp.status_code} ${resp.text}
The problem is that my binary data is about 250MB. When the data is read with Get Binary File I see that memory consumption goes up to 2.x GB. A few seconds later when the Post Request is triggered my test is killed by OOM. I already looked at files parameter, but it seems it uses multipart encoding upload, which is not what I need.
My other thought was about passing open file handler directly to underlying requests library, but I guess that would require robotframework-request modification. Another idea is to fall back to curl for this test only.
Am I missing something in my test? What is the better way to address this?
I proceeded with the idea of robotframework-request modification and added this method
def post_request_binary(
self,
alias,
uri,
path=None,
params=None,
headers=None,
allow_redirects=None,
timeout=None):
session = self._cache.switch(alias)
redir = True if allow_redirects is None else allow_redirects
self._capture_output()
method_name = "post"
method = getattr(session, method_name)
with open(path, 'rb') as f:
resp = method(self._get_url(session, uri),
data=f,
params=self._utf8_urlencode(params),
headers=headers,
allow_redirects=allow_redirects,
timeout=self._get_timeout(timeout),
cookies=self.cookies,
verify=self.verify)
self._print_debug()
# Store the last session object
session.last_resp = resp
self.builtin.log(method_name + ' response: ' + resp.text, 'DEBUG')
return resp
I guess I can improve it a bit and create a pull request.
Related
I'm in the process of writing a very simple Python application for a friend that will query a service and get some data in return. I can manage the GET requests easily enough, but I'm having trouble with the POST requests. Just to get my feet wet, I've only slightly modified their example JSON data, but when I send it, I get an error. Here's the code (with identifying information changed):
import urllib.request
import json
def WebServiceClass(Object):
def __init__(self, user, password, host):
self.user = user
self.password = password
self.host = host
self.url = "https://{}/urldata/".format(self.host)
mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
mgr.add_password(None, "https://{}".format(self.host), self.user, self.password)
self.opener = urllib.request.build_opener(urllib.request.HTTPDigestAuthHandler(mgr))
username = "myusername"
password = "mypassword"
service_host = "thisisthehostinfo"
web_service_object = WebServiceClass(username, password, service_host)
user_query = {"searchFields":
{
"First Name": "firstname",
"Last Name": "lastname"
},
"returnFields":["Entity ID","First Name","Last Name"]
}
user_query = json.dumps(user_query)
user_query = user_query.encode("ascii")
the_url = web_service_object.url + "modules/Users/search"
try:
user_data = web_service_object.opener.open(the_url, user_query)
user_data.read()
except urllib.error.HTTPError as e:
print(e.code)
print(e.read())
I got the class data from their API documentation.
As I said, I can do GET requests fine, but this POST request gives me a 500 error with the following text:
Content type 'application/x-www-form-urlencoded;charset=UTF-8' not supported
In researching this error, my assumption has become that the above error means I need to submit the data as multipart/form-data. Whether or not that assumption is correct is something I would like to test, but stock Python doesn't appear to have any easy way to create multipart/form-data - there are modules out there, but all of them appear to take a file and convert it to multipart/form-data, whereas I just want to convert this simple JSON data to test.
This leads me to two questions: does it seem as though I'm correct in my assumption that I need multipart/form-data to get this to work correctly? And if so, do I need to put my JSON data into a text file and use one of those modules out there to turn it into multipart, or is there some way to do it without creating a file?
Maybe you want to try the requests lib, You can pass a files param, then requests will send a multipart/form-data POST instead of an application/x-www-form-urlencoded POST. You are not limited to using actual files in that dictionary, however:
import requests
response = requests.post('http://httpbin.org/post', files=dict(foo='bar'))
print response.status_code
If you want to know more about the requests lib, and specially in sending multipart forms take a look at this:
http://docs.python-requests.org/en/master/
and
http://docs.python-requests.org/en/master/user/advanced/?highlight=Multipart-Encoded%20Files
In my application, I'm rendering a PDF file and pass it back as a response. For this purpose I'm using flask_weasyprint's render_pdf, which does exactly this:
def render_pdf(html, stylesheets=None, download_filename=None):
if not hasattr(html, 'write_pdf'):
html = HTML(html)
pdf = html.write_pdf(stylesheets=stylesheets)
response = current_app.response_class(pdf, mimetype='application/pdf')
if download_filename:
response.headers.add('Content-Disposition', 'attachment', filename=download_filename)
return response
I now need to render a template + returning the rendered pdf as a download. Something like
#app.route("/view")
def view() :
resp1 = render_pdf(HTML(string="<p>Render me!</p>"), download_filename = "test.pdf")
resp2 = render_template("test.html")
return resp1, resp2
Is there any way to achieve this? Any workaround?
I am not sure if this is solvable in the backend, you want to send two http responses following one request. Should that be possible? (I really don't know) Shouldn't the client make two responses? (javascript).
An option would be, javascript datablob returned in your render_template call.
Maybe something like this? (untested):
fileData = new Blob([pdf_data_here], { type: 'application/pdf' });
fileUrl = URL.createObjectURL(fileData);
window.location.assign(fileUrl);
Or maybe just use the window.location.assign() function to generate the second request.
Or put the data base64 encoded in a href attribute?
I am trying to upload a file to JIRA via its REST API using the python lib found here: jira python documentation
It seems pretty straight forward I wrote a method that allows me to pass an issue and then it attaches a filename. and one that lets me retrieve an issue from JIRA.
from jira.client import JIRA
class JIRAReport (object):
def attach(self,issue):
print 'Attaching... '
attachment = self.jira.add_attachment(issue, attachment=self.reportpath, filename='Report.xlsx')
print 'Success!'
def getissue(self):
if not self.issue == None:
return self.jira.issue(self.issue)
return None
then in my main script I am getting the issue and attaching the file to an issue I retrieved from JIRA
report = JiraReport()
report.issue = 'ProjectKey-1'
report.reportpath = '../report_upload/tmp/' + filename
issue = report.getissue()
if not issue == None:
report.attach(issue)
else:
print "No Issue with Key Found"
I am able to get the issue/create issues if needed but when using the self.jira.add_attachment() method I am getting 405 Method Not Allowed.
The file exists and is able to be opened.
Here is the add_attachment() method from the source code:
def add_attachment(self, issue, attachment, filename=None):
"""
Attach an attachment to an issue and returns a Resource for it.
The client will *not* attempt to open or validate the attachment; it expects a file-like object to be ready
for its use. The user is still responsible for tidying up (e.g., closing the file, killing the socket, etc.)
:param issue: the issue to attach the attachment to
:param attachment: file-like object to attach to the issue, also works if it is a string with the filename.
:param filename: optional name for the attached file. If omitted, the file object's ``name`` attribute
is used. If you aquired the file-like object by any other method than ``open()``, make sure
that a name is specified in one way or the other.
:rtype: an Attachment Resource
"""
if isinstance(attachment, string_types):
attachment = open(attachment, "rb")
# TODO: Support attaching multiple files at once?
url = self._get_url('issue/' + str(issue) + '/attachments')
fname = filename
if not fname:
fname = os.path.basename(attachment.name)
content_type = mimetypes.guess_type(fname)[0]
if not content_type:
content_type = 'application/octet-stream'
files = {
'file': (fname, attachment, content_type)
}
r = self._session.post(url, files=files, headers=self._options['headers'])
raise_on_error(r)
attachment = Attachment(self._options, self._session, json.loads(r.text)[0])
return attachment
It is mentioned in documentation that as a argument they expect file-like object.
Try to do something like :
file_obj = open('test.txt','rb')
jira.add_attachment(issue,file_obj,'test.txt')
file_obj.close()
Check that the URL that you are specifying for JIRA (if using the on-demand service) is https://instance.atlassian.net.
I just hit this as well, and it sends a POST request to http://instance.atlassian.net and gets redirected to https://instance.atlassian.net, but the client sends a GET request to the redirected address (see: https://softwareengineering.stackexchange.com/questions/99894/why-doesnt-http-have-post-redirect for more information)
I have a small flask application which takes some images for upload and converts them into a multipage tiff. Nothing special.
But how do I test the upload of multiple files and the file download?
My Testclient:
class RestTestCase(unittest.TestCase):
def setUp(self):
self.dir = os.path.dirname(__file__)
rest = imp.load_source('rest', self.dir + '/../rest.py')
rest.app.config['TESTING'] = True
self.app = rest.app.test_client()
def runTest(self):
with open(self.dir + '/img/img1.jpg', 'rb') as img1:
img1StringIO = StringIO(img1.read())
response = self.app.post('/convert',
content_type='multipart/form-data',
data={'photo': (img1StringIO, 'img1.jpg')},
follow_redirects=True)
assert True
if __name__ == "__main__":
unittest.main()
The application sends back the file with
return send_file(result, mimetype='image/tiff', \
as_attachment=True)
I want to read the file sent in the response and compare it with another file. How do I get the file from the response object?
I think maybe the confusion here is that response is a Response object and not the data downloaded by the post request. This is because an HTTP response has other attributes that are often useful to know, for example http status code returned, the mime-type of the response, etc... The attribute names to access these are listed in the link above.
The response object has an attribute called 'data', so response.data will contain the data downloaded from the server. The docs I linked to indicate that data is soon to be deprecated, and the get_data() method should be used instead, but the testing tutorial still uses data. Test on your own system to see what works.Assuming you want to test a round trip of the data,
def runTest(self):
with open(self.dir + '/img/img1.jpg', 'rb') as img1:
img1StringIO = StringIO(img1.read())
response = self.app.post('/convert',
content_type='multipart/form-data',
data={'photo': (img1StringIO, 'img1.jpg')},
follow_redirects=True)
img1StringIO.seek(0)
assert response.data == imgStringIO.read()
I have the following Werkzeug application for returning a file to the client:
from werkzeug.wrappers import Request, Response
#Request.application
def application(request):
fileObj = file(r'C:\test.pdf','rb')
response = Response( response=fileObj.read() )
response.headers['content-type'] = 'application/pdf'
return response
The part I want to focus on is this one:
response = Response( response=fileObj.read() )
In this case the response takes about 500 ms (C:\test.pdf is a 4 MB file. Web server is in my local machine).
But if I rewrite that line to this:
response = Response()
response.response = fileObj
Now the response takes about 1500 ms. (3 times slower)
And if write it like this:
response = Response()
response.response = fileObj.read()
Now the response takes about 80 seconds (that's right, 80 SECONDS).
Why is there that much difference between the 3 methods?
And why is the third method sooooo slow?
The answer to that is pretty simple:
x.read() <- reads the whole file into memory, inefficient
setting response to a file: very inefficient as the protocol for that object is an iterator. So you will send the file line by line. If it's binary you will send it with random chunk sizes even.
setting response to a string: bad idea. It's an iterator as mentioned before, so you are now sending each character in the string as a separate packet.
The correct solution is to wrap the file in the file wrapper provided by the WSGI server:
from werkzeug.wsgi import wrap_file
return Response(wrap_file(environ, yourfile), direct_passthrough=True)
The direct_passthrough flag is required so that the response object does not attempt to iterate over the file wrapper but leaves it untouched for the WSGI server.
After some testing I think I've figure out the mistery.
#Armin already explained why this...
response = Response()
response.response = fileObj.read()
...is so slow. But that doesn't explain why this...
response = Response( response=fileObj.read() )
...is so fast. They appear to be the same thing, but obviously they are not. Otherwise there wouldn't be that tremendous difference is speed.
The key here is in this part of the docs: http://werkzeug.pocoo.org/docs/wrappers/
Response can be any kind of iterable or string. If it’s a string it’s considered being an iterable with one item which is the string passed.
i.e. when you give a string to the constructor, it's converted to an iterable with the string being it's only element. But when you do this: response.response = fileObj.read(), the string is treated as is.
So to make it behave like the constructor, you have to do this:
response.response = [ fileObj.read() ]
and now the file is sent as fast as possible.
I can't give you a precise answer as to why this occurs, however http://werkzeug.pocoo.org/docs/wsgi/#werkzeug.wsgi.wrap_file may help address your underling problem.