Develop Reference

InMemoryUploadedFile corrupted in server

I am creating an API with django REST Framework and I am having this problem when creating and endpoint.
I am getting the file with file_details = request.FILES.get('file').
When I run the code on my machine, the file_details.size is 12679, but when I upload the code on the server the file_details.size is 21370 with the exact same file!
Can anyone tell me why is this happening?
Thanks in advance.
EDIT:
I found the solution.
The problem was with the swagger. I added this lines at the generate_swagger_apigw.py and everything worked:
BINARY_CONTENT_TYPES = {
"x-amazon-apigateway-binary-media-types": ['application/msword', 'image/tiff', 'video/mp4', 'video/avi',
'video/ogm', 'video/mov', 'audio/mp3', 'audio/ogg', 'audio/wav', 'image/gif', 'image/jpg',
'image/png', 'application/pdf', "multipart/form-data"
]
}
......
json_data.update(BINARY_CONTENT_TYPES)
where json_data is the swagger configuration (please read the swagger documentation).
(api_link, connection_id, uri) = process_arguments(sys.argv)
response = requests.get(api_link)
json_data = json.JSONDecoder(object_pairs_hook=collections.OrderedDict).decode(response.text)

Keep getting an unexpected token k in JSON at position 0 when sending a python request

There is this game API I'm trying to develop a client for, and sending a PUT request to change the profile pic (even with the same structure as the games source code) returns an unexpected token k in JSON at position 0, the code I used is below:
with open("img.jpg", "rb") as image:
f = image.read()
bytess = str(bytearray(f))
imageParam = {
"key": "profile_image",
"data": bytess,
"filename": "profilepic.jpg",
"mimeType": "image/jpg"
}
newUrl = "https://bwsecondary.ddns.net:8080/api/v1/current_user/profile_world"
Confirm = requests.put(newUrl, data=imageParam, headers=headers)
print(Confirm.text)
It isn't the server since the game doesn`t receive the error when making a put request

Ok so, I had to parse it as Json instead of data, which is a rookie mistake, oh well...

Unable to generate a file to download with Django

Part which is working fine :
I have made a <form> whose submit call makes an ajax request.Code for the same :
$.ajax({
type: 'POST',
url: '/uploaded_proto_file/',
data: formdata,
processData: false,
contentType: false,
success: function (data) {
console.log("in success")
console.log("data is --->"+data)
return;
},
error: function (data) {
console.log("in error")
return;
}
});
I am able to receive the call for the same on the function below. Now i need to send a file (which is in my document structure) that needs to auto downloaded in my browser. I execute this function based on this answer
def uploaded_proto_file(request):
with open(
'/Users/metal/Documents/test_pb2.py','rb') as text_file:
response = HttpResponse(FileWrapper(text_file.getvalue()), content_type='application/zip')
response['Content-Disposition'] = 'attachment; filename=test_pb2.py'
return response
The above code wasn't working fine, so I debugged it and found this error on the filewrapper statement
Now I changed the solution a bit but the file sent back in response is not getting auto downloaded , it is getting printed in the console (as my success block in ajax function has console.log(data))
Changed solution :
with open(
'/Users/metal/Documents/test_pb2.py','rb') as text_file:
response = HttpResponse(text_file, content_type='application/zip')
response['Content-Disposition'] = 'attachment; filename=test_pb2.py'
return response
As you can see above I removed FileWrapper and was atleast able to send the file back to ajax request. But the problem still persists as the file is not getting auto downloaded.
P.S. : I have also tried opening the file in different modes like 'r','rb'.
Any help would be appreciated !

You can't return a file to an ajax request and expect it to be downloaded. What you need is to separate into a few steps:
Your ajax request triggers the file creation process
Django puts that file somewhere where it can be downloaded and returns the URL to the file as ajax response
Your js response handler can now do two things:
Generate a button in the HTML for the user to click and download the file (using the URL from the response)
Create the button and auto-click it (call .click() on it) and then delete it again. A bit like in this example

eBay LMS uploadFile - Cannot specify file with valid format

I am attempting to use the eBay Large Merchant Services API to upload calls in bulk. This API and documentation are notoriously bad (I found a blog post that explains it and rails against it here). The closest I can find to a working Python solution is based on this Question/Answer and his related Github version which is no longer maintained.
I have followed these examples very closely and as best I can tell they never fully worked in the first place. I emailed the original author and he says it was never put in production. However it should be very close to working.
All of my attempts result in a error 11 Please specify a File with Valid Format message.
I have tried various ways to construct this call, using both the github method, or a method via the email.mime package, and via Requests library, and the http.client library. The output below is what I believe is the closest I have to a what it should be. The file attached is using the Github's sample XML file, and it is read and gzipped using the same methods as in the Github repo.
Any help on this would be appreciated. I feel like I have exhausted my possibilities.
POST https://storage.sandbox.ebay.com/FileTransferService
X-EBAY-SOA-SERVICE-VERSION: 1.1.0
Content-Type: multipart/related; boundary=MIME_boundary; type="application/xop+xml"; start="<0.urn:uuid:86f4bbc4-cfde-4bf5-a884-cbdf3c230bf2>"; start-info="text/xml"
User-Agent: python-requests/2.5.1 CPython/3.3.4 Darwin/14.1.0
Accept: */*
Accept-Encoding: gzip, deflate
X-EBAY-SOA-SECURITY-TOKEN: **MYSANDBOXTOKEN**
X-EBAY-SOA-SERVICE-NAME: FileTransferService
Connection: keep-alive
X-EBAY-SOA-OPERATION-NAME: uploadFile
Content-Length: 4863
--MIME_boundary
Content-Type: application/xop+xml; charset=UTF-8; type="text/xml; charset=UTF-8"
Content-Transfer-Encoding: binary
Content-ID: <0.urn:uuid:86f4bbc4-cfde-4bf5-a884-cbdf3c230bf2>
<uploadFileRequest xmlns:sct="http://www.ebay.com/soaframework/common/types" xmlns="http://www.ebay.com/marketplace/services">
<taskReferenceId>50009042491</taskReferenceId>
<fileReferenceId>50009194541</fileReferenceId>
<fileFormat>gzip</fileFormat>
<fileAttachment>
<Size>1399</Size>
<Data><xop:Include xmlns:xop="http://www.w3.org/2004/08/xop/include" href="cid:urn:uuid:1b449099-a434-4466-b8f6-3cc2d59797da"/></Data>
</fileAttachment>
</uploadFileRequest>
--MIME_boundary
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
Content-ID: <urn:uuid:1b449099-a434-4466-b8f6-3cc2d59797da>
b'\x1f\x8b\x08\x08\x1a\x98\xdaT\x02\xffuploadcompression.xml\x00\xcdW[W\xdb8\x10~\x0e\xbfB\x9b}\x858\xd0\xb4\x14\x8e\xebnn\x94l\xe3\x96M\xccv\xfb\xd4#\xec!\xd1V\x96\\I\x06\xfc\xefw$\xdbI\x9cK\x0f\xec\xbe,\xe5P{f\xbe\xb9i\xe6\xb3\xed\xbf\x7fJ9y\x00\xa5\x99\x14\xef\xda\xa7\x9dn\x9b\x80\x88e\xc2\xc4\xe2]\xfb6\xba:y\xdb~\x1f\x1c\xf9\x83\x9c\x7f\x1fQC\xc7O\xf1\x92\x8a\x05\xcc\xe0G\x0e\xdah\x82p\xa1\xdf\xb5s%.\xe1\x8e\x16\x974c\xfa\x12\x06\xd3\x01\xd50\x94i&\x05\x08\xa3\xdb\xc1Q\xcb\xbf\x06\x9a\x80\xc2\xab\x96\xffg\x1908\xef\x9d\xfb^}c\x15sf`2\nN\xbb]\xdf\xab\xae\x11\xe9\xad\xa1-\xbf\x9f$\x13\x03i\x95\xc1\x0b\x12 \x04\xd1c\xa5\xa4\x9ab\t9]#\x00\xe2\xdb\xed\xdc\xf7\x9a\xc2\xca\xf2\x0bU\x02\xbb0\x85\x07\xe0\xc15[,}\xaf!\xaa\xcc\x0e\x95\xd2\xf2C\xd0\x1a\xfda\t\x11&\x8a8\xf2\t\x1e\xc9\xdc({y%UJ\x8d\xef\xad\x8d\x10\x83\x0e}[\x9b\xc3\xb7\xfc\x88\x19\x0e\xc1\xf5\xefC2\xffz\x12\xf6\xff\xc2\xff\xec\xdf\xc9\x84\x9c\x91\xeb\xf14\x1cG\xa4\xff)\xba\x9e\xf5\x87\x93hLB/\x1c\x8f&\xb7\xa1\xef\x95\xb8\xd2\xc7\x08t\xacXflV\xd1\x92i\x82\xbf\x94\xc4Rr\xb2\x04\x9e\x829&\xccX\xe1\x9d\xa2"!\x023\xbc+\x88Y\x02y\xa4\xc5/\xbe\xb7\x89/=\xde(\x96RU\x0c\xa9\x81\x85TE)m\xf9\xf5=V\xf2\xe6\xbcw\xe1{\x1b\x82\x12\xe8\xedE\xfasC\x95AU\x0c\xc1\xc5E\xe7\x02\x91\x1b\x92\xd2d(E\xc2l\n\xe5h\xe0llJ\x8e\x1a\xf1C\x9ae\xd8\xe0>\xe7\xf2\x11\x92 R9\xacs\xd9R\xd6\xdesa0\x1d;\t\xf5u\xa5\xc9\x95\xc2m\xb0\xaa\x11\xea\xea\xbb\xaa\xb3Lg\xd4\xc4\xcb\x88\xa5\x10\xd2\xa7\xe0\x156kKT\x1aN\x99;\xfdQ\xae\xa8k\xe3\x88\x16\xfa\xdb)\x16\xb1\xadh\x98GE\x06\xc1\x15{\x82\xc4u\xc2\x8e\xc5\n\xe1t\xd5i\xd0"\xc5\x01\x0f\xc1,e\xa2\x03\xbc\xbd\xa1\x1c[\xdd\x14\xaflQ9N)\xe3\xb8D\n\'/\xb0+\xf3\x9bb\xb8\\:a:\xb6\xd5wb\x99:\x07\xdb\xb6\x95\x13\x16\x9b\\\xc1\x08\x0c\xaat}\xfa\x95\xf4v6\r\x96\xc6d\x97\x9e\x07\x9d]\xb7\x1e\x9e\xff\x02\xb4\x87#\xedu\xcf\xce\xce\xbbo\xbd\xd7\xe7oN^\xbf9\xfd\xa6\xd3\xce\xdf\xd9\x02\xe3\xae\x1d\xd5S\xb3\'\xa0\x7f#\xb5\xa1|(\x13\x08z\x17\xbd\xb3\x1e\x9a\xad%\xa5\xc9\x1f9\x15\x86\x99"\xb0\xad^\xdd\x94\xba\x19\xa0Kq#9\x8bW\x03<\x83\xfb\\$\x9f\xcbQ\xafy\xce\xf7\x1a\xe2\x86\xe9\x8e\xd1Zm\xbd\xeb/\xcc,\x99\xa8\x90\xe5\xa1\xf7\xac\xe9\xaer\x1f.8\xed\x11\x0b\xdaBl\xd9\xf6\xe3\x182\x03u~[\xd2\x15v\xcbl\xbf\x8f\x1aM\x0e\xc2k\xe0\x0e)\xb4\xeaV \xb9( \x19\xa8\x94\x19\x04\x1c\x13x\xb2P"\x05\x01\x0e\xb1QR\xb0\x18\x19\x07R}L,\xe1\xa49r\xf8\x1d\x10&p\x9dqK\x13\xf2\xe8\xea$X~\x82\xe5\x13yO\x14\xc4\x80\xd1:$\x04e\xc3\xe0H\xc1\x06\xd0\x91V\\\x13\x82\xc3\x13\xca9\xc9h\xfc\x9d.p]\x8eIJENy\x152\xa3\x98\xdf\xa3T\xdf\x11khl\x9c0\x17\x94\x1bP\x90tH$m\xd6\xae\x9cc\x92q\xc0\x07\x89u\xefLc\x8c*SPD\x83z\xc0\xb5$F\x96\xe9=\x00\xd2\xea,\xec\xff\xea\xbc\xc9\\\xad|\x90{\xa4\xfa\x0e\x19O\xc7\xc3h\xf6\xf9\xd3dH\x90\xad\xc3\xf91Ir\x07G\xaee\xe8/\x83\x98QN\x04\xb5\xc3Nb*\x84t\xf5\xd5n\x12\xeb\x07\x9d\x17\x18\x8fj\xac\xd3\xc6\xb1\xcd\xd6\x92\x03/0\xc3\x07\x9b>I\x18\xe6c\xb8\xe5p%\xf3\xc5\xb2\xf2\x8f\x1b\x8c\x11\x8c\xcd\xd36\xe3\x9e\xba\xa5R\xbaS\x1d\xe9.\xd1#3/\x99\xa3\xcb!\xae\xd6\re\xc9\xa0\xa8\xe6g\x90\x17\xa0\x90\xa7\x0f\xe9\x0f\xe2\x0f#\xebm\xdf\xdd\xcc\x95\x9b\r\x06X\xc9\xe6\xe51\x94qKr\xd8\xd6\x05\xf5}I\x86\xf8p\x11\tU\xc9:\x89\xda\xae\x19\xad\x92\xda\x0c\x83n\xa7\xbbc\xee\x14{!\xc8\x97n\x12-\x1b\x1d\x00o\x99\xecu\x83\xb4/\x95\xe3\xaf\x1d\xf8JU\x02\xc7O\x19\xa0?H\xeaJ\xeeq\xd6\x91\x95v\xe4\xae=W\n\xa0\xf6\x17\x18\xf7xl\x88l{\xbd\xc3\xfd\xf5\'\x02\xf7D\xd02\xfd\xbdv\x07\x8e\xa1j|\x03\xbf\xff\x14\xf6\x1d\x02\xae^\xf9\xf8\x9d\x8c\xf0\xc5t>j\x07g\xf8\xb6\xf0\xf6\xf0\xb9\x1c\xec\xe7\xd9\xcf\xfb\xe9p\x91\x9c\xca\xb8|*\x0f\xfb\xa5\xfd\x86\xc8\xb5\xe8y}\xf8\xff\xb4\xeb\xd5\xbfl\xd7\xab\x97\xb5\xab\x8f\xec\xc8b\xaa\xf75m\xc7x\x9c+\x99\xc1\xb3L\xfb\x9a\xd1\xe7\x19\xde\xfe\xa7\xf3\xaa5\xe5\xfb\x17\xb7\xef\xe8\r\xd1\x91[\xb8\x98\xe7\tlE\x99~\xb4+\xb7Os\x183\x19\xbd\x1c\x13~}9\xe6\xc3\xe0\xe5\x98\xf9\x87\x9fa\xe6\xc09\xa8\xbdz}\xa3\xe0\x1e\xec\xf4AE0\xcf4>\xda`\x9e\xe6\xfb\x9e\xfd\x16\x0c`#\x8bP\x1a\xa9t\xf9qX\xeb>\xde\xba\xaf\x02\xfc9\xb1\xbb\x8d\xb7n.\xbc\xfaS\xca\xf7\x9a\xdf\x8cVv\xe4{\x87\xbeiQ]\xff\xfb\x07\xe0\xf2E>\x1f\x0f\x00\x00'
--MIME_boundary--
EDIT: Showing the methods I have tried to actually generate the binary data:
with open('testpayload.xml', mode='rt') as myfile:
full_xml = myfile.read() # replacing my full_xml with testpayload.xml, from Github repo
# METHOD 1: Simple gzip.compress()
payload = gzip.compress(bytes(full_xml, 'UTF-8'))
# METHOD 2: Implementing https://stackoverflow.com/a/8507012/1281743 which should produce a gzip-compatible string with header
import io
out = io.BytesIO()
with gzip.GzipFile(fileobj=out, mode="w") as f:
f.write(bytes(full_xml, 'UTF-8'))
payload = out.getvalue()
# METHOD 3: Simply reading in a pre-compressed version of same testpayload.xml file. Compressed on commandline with 'gzip testpayload.xml'
with open('testpayload.xml.gz', mode='rb') as myfile:
payload = myfile.read()
# METHOD 4: Adoping the _generate_date() function from github eBay-LMS-API repo http://goo.gl/YgFyBi
mybuffer = io.BytesIO()
fp = open('testpayload.xml', 'rb') # This file is from https://github.com/wrhansen/eBay-LMS-API/
# Create a gzip object that reads the compression to StringIO buffer
gzipbuffer = gzip.GzipFile('uploadcompression.xml.gz', 'wb', 9, mybuffer)
gzipbuffer.writelines(fp)
gzipbuffer.close()
fp.close()
mybuffer.seek(0)
payload = mybuffer.read()
mybuffer.close()
# DONE: send payload to API call
r = ebay.fileTransfer_upload_file(file_id='50009194541', task_id='50009042491', gzip_data=payload)
EDIT2, THE QUASI-SOLUTION: Python was not actually sending binary data, but rather a string representation of binary, which did not work. I ended up having to base64-encode the file first. I had tried that before, but have been waived off of that by a suggestion on the eBay forums. I just changed one of the last headers to Content-Transfer-Encoding: base64 and I got a success response.
EDIT3, THE REAL SOLUTION: Base 64 encoding only solved my problems with the example XML file. I found that my own XML payload was NOT sending successfully (same error as before), but that the script on Github could send that same file just fine. The difference was that the Github script is Python2 and able to mix string and binary data without differentiation. To send binary data in Python3 I just had to make a few simple changes to change all the strings in the request_part and binary_part into bytes, so that the bytes from the gzip payload would concatenate with it. Looks like this:
binary_part = b'\r\n'
binary_part += b'--MIME_boundary\r\n'
binary_part += b'Content-Type: application/octet-stream\r\n'
binary_part += b'Content-Transfer-Encoding: binary\r\n'
binary_part += b'Content-ID: <' + URN_UUID_ATTACHMENT.encode('utf-8') + b'>\r\n\r\n'
binary_part += gzip_data + b'\r\n'
binary_part += b'--MIME_boundary--'
So, no base64 encoding, just needed to figure out how to make Python send real binary data. And the original github repo does work as advertised, in python2.

So after scanning all my code about this, it looks like it's setup right. So if your error is a good error then it means your binary data is wrong. Can you show how you're gzipping and then reading that data back in? Here's what I'm doing
$dir = '/.../ebayUpload.gz';
if(is_file($dir)) unlink($dir);
$gz = gzopen($dir,'w9');
gzwrite($gz, $xmlFile);
chmod($dir, 0777);
gzclose($gz);
// open that file as data;
$handle = fopen($dir, 'r');
$fileData = fread($handle, filesize($dir));
fclose($handle);
And in your case the $xmlFile should be the string found in https://github.com/wrhansen/eBay-LMS-API/blob/master/examples/AddItem.xml
I should add that this is how I'm using $fileData
$binaryPart = '';
$binaryPart .= "--" . $boundry . $CRLF;
$binaryPart .= 'Content-Type: application/octet-stream' . $CRLF;
$binaryPart .= 'Content-Transfer-Encoding: binary' . $CRLF;
$binaryPart .= 'Content-ID: <urn:uuid:'.$uuid_attachment.'>' . $CRLF . $CRLF;
$binaryPart .= $fileData . $CRLF;
$binaryPart .= "--" . $boundry . "--";

Needed advice to automate REST services test

I am kind of newbie to REST and testing dept. I needed to write automation scripts to test our REST services.We are planning to run these scripts from a Jenkins CI job regularly. I prefer writing these in python as we already have UI functionality testing scripts in python generated by selenium IDE, but I am open to any good solution.I checked httplib,simplejson and Xunit, but looking for better solutions available out there.
And also, I would prefer to write a template and generate actual script for each REST API by reading api info from xml or something. Advance thanks to all advices.

I usually use Cucumber to test my restful APIs. The following example is in Ruby, but could easily be translated to python using either the rubypy gem or lettuce.
Start with a set of RESTful base steps:
When /^I send a GET request for "([^\"]*)"$/ do |path|
get path
end
When /^I send a POST request to "([^\"]*)" with the following:$/ do |path, body|
post path, body
end
When /^I send a PUT request to "([^\"]*)" with the following:$/ do |path, body|
put path, body
end
When /^I send a DELETE request to "([^\"]*)"$/ do |path|
delete path
end
Then /^the response should be "([^\"]*)"$/ do |status|
last_response.status.should == status.to_i
end
Then /^the response JSON should be:$/ do |body|
JSON.parse(last_response.body).should == JSON.parse(body)
end
And now we can write features that test the API by actually issuing the requests.
Feature: The users endpoints
Scenario: Creating a user
When I send a POST request to "/users" with the following:
"""
{ "name": "Swift", "status": "awesome" }
"""
Then the response should be "200"
Scenario: Listing users
Given I send a POST request to "/users" with the following:
"""
{ "name": "Swift", "status": "awesome" }
"""
When I send a GET request for "/users"
Then the response should be "200"
And the response JSON should be:
"""
[{ "name": "Swift", "status": "awesome" }]
"""
... etc ...
These are easy to run on a CI system of your choice. See these links for references:
http://www.anthonyeden.com/2010/11/testing-rest-apis-with-cucumber-and-rack-test/
http://jeffkreeftmeijer.com/2011/the-pain-of-json-api-testing/
http://www.cheezyworld.com/2011/08/09/running-your-cukes-in-jenkins/

import openpyxl
import requests
import json
from requests.auth import HTTPBasicAuth
urlHead='https://IP_ADDRESS_HOST:PORT_NUMBER/'
rowStartAt=2
apiColumn=2
#payloadColumn=3
responseBodyColumn=12
statusCodeColumn=13
headerTypes = {'Content-Type':'application/json',
'Accept':'application/json',
'Authorization': '23324'
}
wb = openpyxl.load_workbook('Excel_WORKBOOK.xlsx')
# PROCESS EACH SHEET
for sheetName in (wb.get_sheet_names()):
print ('Sheet Name = ' + sheetName)
flagVar = input('Enter N To avoid APIs Sheets')
if (flagVar=='N'):
print ('Sheet got skipped')
continue
#get a sheet
sheetObj = wb.get_sheet_by_name(sheetName)
#for each sheet iterate the API's
for i in range(2, sheetObj.max_row+1):
#below is API with method type
apiFromSheet = (sheetObj.cell(row=i, column=apiColumn).value)
if apiFromSheet is None:
continue
#print (i, apiFromSheet)
#Let's split the api
apiType = apiFromSheet.split()[0]
method = apiFromSheet.split()[1]
if (apiType!='GET'):
continue
#lets process GET API's
absPath = urlHead + method
print ("REQUESTED TYPE AND PATH = ", apiType, absPath)
print('\n')
res = requests.get(absPath, auth=HTTPBasicAuth(user, pwd), verify=False, headers=headerTypes)
#LET's write res body into relevant cell
sheetObj.cell(row=i, column=responseBodyColumn).value = (res.text)
sheetObj.cell(row=i, column=statusCodeColumn).value = (res.status_code)
wb.save('Excel_WORKBOOK.xlsx')
`#exit(0)`