Develop Reference

Robot Framework: send binary data in POST request body with

I have a problem with getting my test running using Robot Framework and robotframework-requests. I need to send a POST request and a binary data in the body. I looked at this question already, but it's not really answered. Here's how my test case looks like:
Upload ${filename} file
Create Session mysession http://${ADDRESS}
${data} = Get Binary File ${filename}
&{headers} = Create Dictionary Content-Type=application/octet-stream Accept=application/octet-stream
${resp} = Post Request mysession ${CGIPath} data=${data} headers=&{headers}
[Return] ${resp.status_code} ${resp.text}
The problem is that my binary data is about 250MB. When the data is read with Get Binary File I see that memory consumption goes up to 2.x GB. A few seconds later when the Post Request is triggered my test is killed by OOM. I already looked at files parameter, but it seems it uses multipart encoding upload, which is not what I need.
My other thought was about passing open file handler directly to underlying requests library, but I guess that would require robotframework-request modification. Another idea is to fall back to curl for this test only.
Am I missing something in my test? What is the better way to address this?

I proceeded with the idea of robotframework-request modification and added this method
def post_request_binary(
self,
alias,
uri,
path=None,
params=None,
headers=None,
allow_redirects=None,
timeout=None):
session = self._cache.switch(alias)
redir = True if allow_redirects is None else allow_redirects
self._capture_output()
method_name = "post"
method = getattr(session, method_name)
with open(path, 'rb') as f:
resp = method(self._get_url(session, uri),
data=f,
params=self._utf8_urlencode(params),
headers=headers,
allow_redirects=allow_redirects,
timeout=self._get_timeout(timeout),
cookies=self.cookies,
verify=self.verify)
self._print_debug()
# Store the last session object
session.last_resp = resp
self.builtin.log(method_name + ' response: ' + resp.text, 'DEBUG')
return resp
I guess I can improve it a bit and create a pull request.

How can I send data from HDFS through Django backend to Angular 5 frontend?

I am downloading file from Hadoop to Django backend and storing the file using the code below:
import shutil
import requests
url = 'http://112.138.0.12:9870/webhdfs/v1/user/username/1.jpg?op=OPEN&user.name=username'
response = requests.get(url, stream=True)
with open('img.png', 'wb') as out_file:
shutil.copyfileobj(response.raw, out_file)
del response
I don't need to store the file in backend local system since I want to send this file to Angular 5 frontend where user will save this file in their local system. I'm getting the following error
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position
0: invalid start byte.
Can someone suggest me what would be the right way to do download large files in a short time?
DJANGO:
views.py:
class DownloadFileView(GenericAPIView):
serializer_class = UserNameSerializer
def get(self, request):
key = request.META.get('HTTP_AUTHORIZATION').split()[1]
user_id = Token.objects.get(key=key).user_id
user_name = User.objects.get(id=user_id).username
response = download_files(user_name)
return Response(response)
def download_files(user_name):
response = requests.get('http://112.138.0.12:9870/webhdfs/v1/user/' + user_name + '/1.jpg?op=OPEN&user.name=username', stream=True)
return response.raw
ANGULAR:
DownloadFile(){
this.userService.DownloadFiles().subscribe((data : any) => {
const blob = new Blob([data], { type: 'application/octet-stream'});
fileUrl = this.sanitizer.bypassSecurityTrustResourceUrl(window.URL.createObjectURL(blob));
}
}
DownloadFiles() {
this.token = localStorage.getItem('userToken')
var reqHeader = new HttpHeaders({ 'Content-Type': 'application/octet-stream', 'Authorization': 'token ' + this.token });
console.log(reqHeader)
return this.http.get(this.rootURL + 'download/', { headers: reqHeader});
}

To begin with your unicode error, it's because:
HttpResponse.init(content='', content_type=None, status=200,
reason=None, charset=None)
Instantiates an HttpResponse
object with the given page content and content type.
content should be an iterator or a string. If it’s an iterator, it
should return strings, and those strings will be joined together to
form the content of the response. If it is not an iterator or a
string, it will be converted to a string when accessed.
I do believe django is having trouble converting the binary data in the file to string. A more common approach when dealing with file downloads is:
response = HttpResponse(content_type="application/jpeg")
response.write(binary_data)
This works because there is a call to make_bytes behind the scenes which handles the binary data correctly.
Having said that, this is not the most efficient way to go about it. Your web app makes a request to a remote server using requests, and then passes that onto the client. Why not get your angular code to fetch the data directly from the end point?
Can't do that because you want authentication you say? Ok, How about checking the authentiation and then sending an HttpResponseDirect like this:
return HttpResponseRedirect('http://112.138.0.12:9870/webhdfs/v1/user/' + user_name + '/1.jpg?op=OPEN&user.name=username')

Django & Ajax: How can I create a csv file download from posting data through AJAX to Django?

I'm trying to allow the click trigger to do some backend dynamic csv creation, and then return it back to the user as a csv download file. I guess I'm not sure how I should write out the return statement other than just putting return response. I'v come across some other posts saying that I would need to set my url to a hidden iframe?? Not sure what this means though. Any tips?
Ajax looks like this:
$('#download-maxes').on('click', function(){
$.ajax({
type: "POST",
url: "{{request.path}}download/",
dataType: 'json',
async: false,
data: JSON.stringify(workouts),
success: function(workoutData) {
console.log(workoutData);
},
error:function(error){
console.log(error);
}
});
});
And my django view looks like this:
def download(request):
#(... a lot of mongo stuff here & other things defined)
workouts = json.load(request.body)
response = HttpResponse(content_type='text/xlsx')
response['Content-Disposition'] = 'attachment; filename="team_maxes.xlsx"'
writer = csv.writer(response)
writer.writerow(['Name', 'Date', 'Workout', 'Max'])
for member in team_members.all():
for wo in workouts:
wo_data = db.activity_types.find_one({"name": wo["name"]})
best_wo = db.activity.find_one({"u_id": member.user.id, "a_t": str(wo_data["_id"]), "is_last": 1}) or 0
member_name = member.user.first_name + ' ' + member.user.last_name
try:
max_stat = best_wo["y_ts"]
except:
max_stat = 0
try:
date = best_wo["e_d"]
except:
date = ""
workout_name = wo_data["name"]
writer.writerow([member_name, date, workout_name, max_stat])
return response

You don't need to use ajax. Since you are POSTing some json data to your view, just make a form with a hidden text input, and set it's value to the json data. Then make a regular submit button in the form.
When the form gets submitted, and the server responds with Content-Disposition: attachment; filename="team_maxes.xlsx", your browser will automatically trigger a download.
If you decide to go this route, Keep in mind:
You are using a regular html form now, with the POST method, so you must remember to use django's {% csrf_token %} tag inside of it.
You will probably have set the input's value to your json string, right before submitting the form.
Your workouts json gets sent in a form input. So, assuming you named your input "workouts", in your view you would do something like:
workouts = json.loads(request.POST.get('workouts'))
Plus a bunch of error checking, of course.

Python server cgi.FieldStorage parsing multipart/form-data

so I have been writing a simple web server in Python, and right now I'm trying to handle multipart/form-data POST requests. I can already handle application/x-www-form-urlencoded POST requests, but the same code won't work for the multipart. If it looks like I am misunderstanding anything, please call me out, even if it's something minor. Also if you guys have any advice on making my code better please let me know as well :) Thanks!
When the request comes in, I first parse it, and split it into a dictionary of headers and a string for the body of the request. I use those to then construct a FieldStorage form, which I can then treat like a dictionary to pull the data out:
requestInfo = ''
while requestInfo[-4:] != '\r\n\r\n':
requestInfo += conn.recv(1)
requestSplit = requestInfo.split('\r\n')[0].split(' ')
requestType = requestSplit[0]
url = urlparse.urlparse(requestSplit[1])
path = url[2] # Grab Path
if requestType == "POST":
headers, body = parse_post(conn, requestInfo)
print "!!!Request!!! " + requestInfo
print "!!!Body!!! " + body
form = cgi.FieldStorage(headers = headers, fp = StringIO(body), environ = {'REQUEST_METHOD':'POST'}, keep_blank_values=1)
Here's my parse_post method:
def parse_post(conn, headers_string):
headers = {}
headers_list = headers_string.split('\r\n')
for i in range(1,len(headers_list)-2):
header = headers_list[i].split(': ', 1)
headers[header[0]] = header[1]
content_length = int(headers['Content-Length'])
content = conn.recv(content_length)
# Parse Content differently if it's a multipart request??
return headers, content
So for an x-www-form-urlencoded POST request, I can treat FieldStorage form like a dictionary, and if I call, for example:
firstname = args['firstname'].value
print firstname
It will work. However, if I instead send a multipart POST request, it ends up printing nothing.
This is the body of the x-www-form-urlencoded request:
firstname=TEST&lastname=rwar
This is the body of the multipart request:
--070f6a3146974d399d97c85dcf93ed44
Content-Disposition: form-data; name="lastname"; filename="lastname"
rwar
--070f6a3146974d399d97c85dcf93ed44
Content-Disposition: form-data; name="firstname"; filename="firstname"
TEST
--070f6a3146974d399d97c85dcf93ed44--
So here's the question, should I manually parse the body for the data in parse_post if it's a multipart request?
Or is there a method that I need/can use to parse the multipart body?
Or am I doing this wrong completely?
Thanks again, I know it's a long read but I wanted to make sure my question was comprehensive

So I solved my problem, but in a totally hacky way.
Ended up manually parsing the body of the request, here's the code I wrote:
if("multipart/form-data" in headers["Content-Type"]):
data_list = []
content_list = content.split("\r\n\r\n")
for i in range(len(content_list) - 1):
data_list.append("")
data_list[0] += content_list[0].split("name=")[1].split(";")[0].replace('"','') + "="
for i,c in enumerate(content_list[1:-1]):
key = c.split("name=")[1].split(";")[0].replace('"','')
data_list[i+1] += key + "="
value = c.split("\r\n")
data_list[i] += value[0]
data_list[-1] += content_list[-1].split("\r\n")[0]
content = "&".join(data_list)
If anybody can still solve my problem without having to manually parse the body, please let me know!

There's the streaming-form-data project that provides a Python parser to parse data that's multipart/form-data encoded. It's intended to allow parsing data in chunks, but since there's no chunk size enforced, you could just pass your entire input at once and it should do the job. It should be installable via pip install streaming_form_data.
Here's the source code - https://github.com/siddhantgoel/streaming-form-data
Documentation - https://streaming-form-data.readthedocs.io/en/latest/
Disclaimer: I'm the author. Of course, please create an issue in case you run into a bug. :)

How to download several files with GAE Python

I'd like to download several files with GAE Python code.
My current code is like below
import webapp2, urllib
url1 = 'http://dummy/sample1.jpg'
url2 = 'http://dummy/sample2.jpg'
class DownloadHandler(webapp2.RequestHandler):
def get(self):
#image1
self.response.headers['Content-Type'] = 'application/octet-stream'
self.response.headers['Content-Disposition'] = 'attachment; filename="' + 'sample1.jpg' + '"'
f = urllib.urlopen(url1)
data = f.read()
self.response.out.write(data)
#image2
self.response.headers['Content-Type'] = 'application/octet-stream'
self.response.headers['Content-Disposition'] = 'attachment; filename="' + 'sample2.jpg' + '"'
f = urllib.urlopen(url2)
data = f.read()
self.response.out.write(data)
app = webapp2.WSGIApplication([('/.*', DownloadHandler)],
debug=True)
I expected to occur download dialogue twice with this code, but actually occurred once, and only sample2.jpg was downloaded.
How can you handle download dialogue several times?
I'd actually like to realize some other functions adding above as well.
To display progressing message on the browser such as
sample1.jpg was downloaded
sample2.jpg was downloaded
sample3.jpg was downloaded ...
And redirect to the other page after downloading files.
When I wrote a code such as
self.redirect('/otherpage')
after
self.response.out.write(data)
Only redirect had happened and didn't occur download procedure.
Would you give me any ideas to solve it please.
I'm using python2.7

Two things.
You cannot write two files in one response that has a Content-Type of application/octet-stream. To stuff multiple files in in the response, you would have to encode your response with multipart/form-data or multipart/mixed and hope that the client would understand that and parse it and show two download dialogues
Once you've already called self.response.out.write(…), you shouldn't be setting any more headers.
To me it seems that the most foolproof option would be to serve an HTML file that contains something like:
<script>
window.open('/path/to/file/1.jpg');
window.open('/path/to/file/1.jpg');
</script>
… and then handle those paths using different handlers.
Another option would be to zip the two files and serve the zipfile to the client, though it may or may not be preferable in your case.

I reached the goal what I wanted to do.
As user interaction, generating html sources include below
<script type="text/javascript">
window.open("/download?url=http://dummy/sample1.jpg")
window.open("/download?url=http://dummy/sample2.jpg")
</script>
then created new windows are handled with this code.
class DownloadHandler(webapp2.RequestHandler):
def get(self):
url = self.request.get('url')
filename = str(os.path.basename(url))
self.response.headers['Content-Type'] ='application/octet-stream'
self.response.headers['Content-Disposition'] = 'attachment; filename="%s"' % (filename)
data = urllib.urlopen(url).read()
self.response.out.write(data)
app = webapp2.WSGIApplication([('/download', DownloadHandler)], debug=True)
Thank you, Attila.