I'm trying to send raw response (which is a jpeg image from my laptop camera) in python using flask, this is the particular snippet:
#app.route("/")
def stream():
frm = imencode('.jpg',cap.read()[1])[1].tobytes()
resp = Response()
resp.set_data(value="HTTP/1.1 200 Ok\nContent-Type:image/jpeg\nContent-Length:"+str(len(frm))+"\n\n"+str(frm))
return resp
My browser seem to display it as text nonetheless. If initialize
Response(frm.tobytes(), headers={"Content-Type":"image/jpeg"})
then then browser decodes the image ok. I'm not very good with web stuff, but from what I've found so far response consists of first line specifying the http version, response status code and respective message. Then come the header fields each on new line. then a clean line separates metadata from the body. I've read that the bare minimum for a simple response are Content-Type and Content-Length headers. Some sources also mention using \r in combination with \n to separate the lines but i didn't find the example so far and this source didn't specify where exactly \r should be added.
Related
Lets say i have this endpoint in flask.
#app.route('/video.mp4', methods=['GET'])
def video_send():
return flask.send_file('video.mp4')
This sends the video file raw with a 200 status. The client thats making this request does not support range byte requests with 206 statuses. However, i can still make a request to the endpoint with some url parameter such as /video.mp4?skip_bytes=50000. The video file was encoded with handbrake and "web optimizations" where enabled. Is it possible to be able to send a certain range of bytes of that file and still have it be completely playable?
I have a python program in which I am trying to do a post sending file contents as a string. I am doing the following:
header 'content-type':'multipart/form-data'
data {file1: filedata}
filedata is a string I build by reading a file putting the contents in the string
when I make the post call I get a 500 back and looking in the catalina log I see the error "couldn't determine the boundary from the message".
the thing is when I put this in insomnia it works properly and the catalina log shows the header had ; boundary=X-INSOMNIA-BOUNDARY appended to the content-type I defined
Why would this work for insomnia but not when I do it in python using requests? This is my requests call (auth is set to None)
response = requests.post(url, data=data, headers=headers, auth=auth, timeout=REQUEST_TIMEOUT, verify=False)
headers = 'content-type':'multipart/form-data
data = {'timepunch': 'datastring'}
I found my problem - when sending a file I need the request to be files-data instead of data=data - not sure exactly what happens in the background but this resolved my issue
I am trying to use the requests library in Python to upload a file into Fedora commons repository on localhost. I'm fairly certain my main problem is not understanding open() / read() and what I need to do to send data with an http request.
def postBinary(fileName,dirPath,url):
path = dirPath+'/'+fileName
print('to ' + url + '\n' + path)
openBin = {'file':(fileName,open(path,'rb').read())}
headers = {'Slug': fileName} #not important
r = requests.put(url, files=openBin,headers=headers, auth=HTTPBasicAuth('username', 'pass'))
print(r.text)
print("and the url used:")
print(r.url)
This will successfully upload a file in the repository, but it will be slightly larger and corrupted after. For example an image that was 6.6kb became 6.75kb and was not openable anymore.
So how should I properly open and upload a file using put in python?
###Extra details:###
When I replace files=openBin with data=openBin I end up with my dictionary and I presume the data as a string. I don't know if that information is helpful or not.
"file=FILE_NAME.extension&file=TYPE89a%24%02Q%03%E7%FF%00E%5B%19%FC%....
and the size of the file increases to a number of megabytes
I am using specifically put because the Fedora RESTful HTTP API end point says to use put.
The following command does work:
curl -u username:password -H "Content-Type: text/plain" -X PUT -T /path/to/someFile.jpeg http://localhost:8080/fcrepo/rest/someFile.jpeg
Updated
Using requests.put() with the files parameter sends a multipart/form-data encoded request which the server does not seem to be able to handle without corrupting the data, even when the correct content type is declared.
The curl command simply performs a PUT with the raw data contained in the body of the request. You can create a similar request by passing the file data in the data parameter. Specify the content type in the header:
headers = {'Content-type': 'image/jpeg', 'Slug': fileName}
r = requests.put(url, data=open(path, 'rb'), headers=headers, auth=('username', 'pass'))
You can vary the Content-type header to suit the payload as required.
Try setting the Content-type for the file.
If you are sure that it is a text file then try text/plain which you used in your curl command - even though you would appear to be uploading a jpeg file? However, for a jpeg image, you should use image/jpeg.
Otherwise for arbitrary binary data you can use application/octet-stream:
openBin = {'file': (fileName, open(path,'rb'), 'image/jpeg' )}
Also it is not necessary to explicitly read the file contents in your code, requests will do that for you, so just pass the open file handle as shown above.
I'm trying to replace curl with Python & the requests library. With curl, I can upload a single XML file to a REST server with the curl -T option. I have been unable to do the same with the requests library.
A basic scenario works:
payload = '<person test="10"><first>Carl</first><last>Sagan</last></person>'
headers = {'content-type': 'application/xml'}
r = requests.put(url, data=payload, headers=headers, auth=HTTPDigestAuth("*", "*"))
When I change payload to a bigger string by opening an XML file, the .put method hangs (I use the codecs library to get a proper unicode string). For example, with a 66KB file:
xmlfile = codecs.open('trb-1996-219.xml', 'r', 'utf-8')
headers = {'content-type': 'application/xml'}
content = xmlfile.read()
r = requests.put(url, data=content, headers=headers, auth=HTTPDigestAuth("*", "*"))
I've been looking into using the multipart option (files), but the server doesn't seem to like that.
So I was wondering if there is a way to simulate curl -T behaviour in Python requests library.
UPDATE 1:
The program hangs in textmate, but throws an UnicodeEncodeError error on the commandline. Seems that must be the problem. So the question would be: is there a way to send unicode strings to a server with the requests library?
UPDATE 2:
Thanks to the comment of Martijn Pieters the UnicodeEncodeError went away, but a new issue turned up.
With a literal (ASCII) XML string, logging shows the following lines:
2012-11-11 15:55:05,154 INFO Starting new HTTP connection (1): my.ip.address
2012-11-11 15:55:05,294 DEBUG "PUT /v1/documents?uri=/example/test.xml HTTP/1.1" 401 211
2012-11-11 15:55:05,430 DEBUG "PUT /v1/documents?uri=/example/test.xml HTTP/1.1" 201 0
Seems the server always bounces the first authentication attempt (?) but then accepts the second one.
With a file object (open('trb-1996-219.xml', 'rb')) passed to data, the logfile shows:
2012-11-11 15:50:54,309 INFO Starting new HTTP connection (1): my.ip.address
2012-11-11 15:50:55,105 DEBUG "PUT /v1/documents?uri=/example/test.xml HTTP/1.1" 401 211
2012-11-11 15:51:25,603 WARNING Retrying (0 attempts remain) after connection broken by 'BadStatusLine("''",)': /v1/documents?uri=/example/test.xml
So, first attempt is blocked as before, but no second attempt is made.
According to Martijn Pieters (below), the second issue can be explained by a faulty server (empty line).
I will look into this, but if someone has a workaround (apart from using curl) I wouldn't mind hearing it.
And I am still surprised that the requests library behaves so differently for small string and file object. Isn't the file object serialized before it gets to the server anyway?
To PUT large files, don't read them into memory. Simply pass the file as the data keyword:
xmlfile = open('trb-1996-219.xml', 'rb')
headers = {'content-type': 'application/xml'}
r = requests.put(url, data=xmlfile, headers=headers, auth=HTTPDigestAuth("*", "*"))
Moreover, you were opening the file as unicode (decoding it from UTF-8). As you'll be sending it to a remote server, you need raw bytes, not unicode values, and you should open the file as a binary instead.
Digest authentication always requires you to make at least two request to the server. The first request doesn't contain any authentication data. This first request will fail with a 401 "Authorization required" response code and a digest challenge (called a nounce) to be used for hashing your password etc. (the exact details don't matter here). This is used to make a second request to the server containing your credentials hashed with the challenge.
The problem is in the this two step authentication: your large file was already send with the first unauthorized request (send in vain) but on the second request the file object is already at the EOF position. Since the file size was also send in the Content-length header of the second request, this causes the server to wait for a file that will never be send.
You could solve it using a requests Session and first make a simple request for authentication purposes (say a GET request). Then make a second PUT request containing the actual payload using the same digest challenge form the first request.
sess = requests.Session()
sess.auth = HTTPDigestAuth("*", "*")
sess.get(url)
headers = {'content-type': 'application/xml'}
with codecs.open('trb-1996-219.xml', 'r', 'utf-8') as xmlfile:
sess.put(url, data=xmlfile, headers=headers)
i used requests in python to upload an XML file using the commands.
first to open the file use open()
file = open("PIR.xsd")
fragment = file.read()
file.close()
copy the data of XML file in the payload of the requests and post it
payload = {'key':'PFAkrzjmuZR957','xmlFragment':fragment}
r = requests.post(URL,data=payload)
to check the html validation code
print (r.text)
I'm using the python poster library to try to upload a form containing including an image to a servlet. Locally, it runs fine, but when I deploy to app engine, it doesn't recognize it as multipart content.
ServletFileUpload.isMultipartContent(request) returns false
Here's how I'm using the poster library:
register_openers()
datagen, headers = multipart_encode({"image": open(filename)})
request = urllib2.Request(url, datagen, headers)
The servlet checks to make sure it is Multipart, but it fails that check. What can I do to further debug?
Thanks,
jean
*******update*********
printing out the stack trace...here's what i get. It complains the content type header isnull
org.apache.commons.fileupload.FileUploadBase$InvalidContentTypeException: the request doesn't contain a multipart/form-data or multipart/mixed stream, content type header is null
at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.(FileUploadBase.java:885)
at org.apache.commons.fileupload.FileUploadBase.getItemIterator(FileUploadBase.java:331)
at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:349)
at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126)
If you're on Windows (or a pedant;-), open(filename) is the wrong way to open a binary file and might mess things up -- use open(filename, 'rb'). Apart from that, assuming of course that you continue with a urllib2.urlopen(request) which you've omitted, that your imports are correct, and that filename and url are properly set previously, then your code seems legit.