I have a falcon server that I am trying to port to django. One of the falcon endpoints processes a request that contains a PNG file sent with content_type = 'application/octet-stream'. It writes the data to a file maintaining the correct PNG structure.
The falcon code does this:
form = cgi.FieldStorage(fp=req.stream, environ=req.env)
and then writes the png like this:
fd.write(form[key].file.read())
I cannot figure out how to do the same thing in django. When my view is called the data in request.POST[key] has already been decoded to unicode text and it's no longer valid png data.
How can I do this with django? Should/can I use cgi.FieldStorage? The request I get (of type django.core.handlers.wsgi.WSGIRequest) does not have a stream method. I'm sure there's some way to do this, but I have not come up with anything googling.
I solved this by changing the client to set the file and filename fields each part of the multipart and then I was able it iterate through request.FILES and successfully write the files as PNG.
Related
I need to send an encoded and decoded image along with some metadata via HTTP.
I would like to send the images as binary data instead of encoding them as base64 as encoding & decoding adds unnecessary latency.
So for example, the encoded image may look like this:
img = open(img_file, 'rb').read()
and the decoded image may look like this:
img = cv2.imread(img_file)
Assume I also need to send some additional information in POST request, such as the image name for example.
What is the most efficient way to send these? What would the code look like in Python? What content-type or other headers would I need to use?
I've found some examples like this online, but they only send a single image and therefore set the content-type as image/jpeg, but I'm wondering what happens when you have additional fields to send.
If you want to send additional fields you have a few options:
Base64 encode the image data and embed it in a json string with all your extra data
Add custom HTTP headers with your fields in
Add your fields to the image metadata itself
I know you said you didn't want to do 1, but how do you know it is adding unnecessary latency if you've never tried it? I expect it's far less than the latency of an HTTP request. Option 2 is risky as the headers can get stripped or changed by network infrastructure and your users might not expect to find data in the headers. Option 3 depends a bit what the data is and whether it makes sense for it to be inside the image (and again whether your users know to look for it there)
I need some examples with file operations using in memory and temporary files.
I searched a lot for a good example /tutorial and I found just basic read/write/append operations.
I need to understand how can I read in Django a file that is uploaded(an image) before the save(post) is finished.
Because Django is Python, I think is better to understand first in python.
I checked the Django documentation/examples about, but it is not very clear so I need to understand first in Python then in Django how the operations are working, not just copy and paste.
I know how to use ImageFields, upload default operation, I'm interested only in using "in memory and temporary files."
I want to use this in combination with a crop function. So the user can upload 1, 2, 3... images, and with a javascript crop script I get the coordinates in a hidden field. After javascript simulated crop I show the user a thumbnail with the crop, how will look like ratio
The user can change his mind and can edit/update or delete a file before saving.
Now depending on the file size it can be keep in memory or write.
When a file is uploaded Django will do one of two things: store it in memory if the file is small (< 2 MB last time I checked), or store it as a temporary file on disk if it's large. This behavior is configurable via the FILE_UPLOAD_HANDLERS setting. So, your web server and Django take care of the actual upload and storage, but it's your job to process the file before the request is over, otherwise the file is deleted.
Uploaded files are accessible through the request.FILES property. Each key in FILES will match the name of the file input on your <form>. The value is an UploadedFile object, which is a stream you can use to read the file data.
For example, say you have an <input name="img" type="file" /> and you want to detect if the image is completely white. You don't need to store the file for this, you just need to load it into memory, process it to get the result and then let it be discarded.
from PIL import Image
def some_view(request):
if request.method == 'POST':
img_file = request.FILES['img']
if img_file.size > 2000000:
return HttpResponseBadRequest()
img = Image.open(img_file)
# analyze the image...
Another possibility is that someone is uploading a backup file that is quite large (lets say 2 GB), and you need to store it somewhere. It's effectively the same thing, except we read the file into memory in chunks, then write each chunk to disk somewhere else so that it's saved after the request finishes.
def some_view(request):
if request.method == 'POST':
backup_file = request.FILES['backup_file']
with open('some/file/name.bak', 'wb+') as destination:
for chunk in backup_file.chunks():
destination.write(chunk)
# file is saved
When the request is over, the uploaded file is stored at some/file/name.bak.
Whether it's in memory or a temporary file is usually not important because the interface is the same. You can read a temporary file just like you can read an in memory file.
You can set which upload handler is used for a Django form: https://docs.djangoproject.com/en/1.11/ref/files/uploads/#module-django.core.files.uploadhandler
There are two built in options:
in memory
temporary file
The way Django seems to work is by walking through the list of all upload handlers: https://docs.djangoproject.com/en/1.11/topics/http/file-uploads/#upload-handlers
For each upload handler it will check for a specific condition and if that condition is true, it will activate and use that upload handler.
The InMemoryUploadHandler, for example, is activated and used when files are below a certain size. The TemporaryFileUploadHandler is used when files are very large.
You will be able to access the data and files from the request object.
If you're looking at Python specific (not Django), then it sounds like you may be interested in the tempfile module:
https://docs.python.org/3/library/tempfile.html
Adapting the example from the document above:
import tempfile
fp = tempfile.TemporaryFile()
fp.write(b'Hello world!')
# Closing automatically deletes the tempfile
fp.close()
You can see this works fairly similarly to generic read/write file operations.
If I understand correctly you are looking for a way to access the uploaded file before it is saved using instance.save() or similar.
If that is the case, you might try to read the file directly from request:
if my_form.is_valid():
data = request.FILES['myfile'].read()
I'm getting an Image from URL with Pillow, and creating an stream (BytesIO/StringIO).
r = requests.get("http://i.imgur.com/SH9lKxu.jpg")
stream = Image.open(BytesIO(r.content))
Since I want to upload this image using an <input type="file" /> with selenium WebDriver. I can do something like this to upload a file:
self.driver.find_element_by_xpath("//input[#type='file']").send_keys("PATH_TO_IMAGE")
I would like to know If its possible to upload that image from a stream without having to mess with files / file paths... I'm trying to avoid filesystem Read/Write. And do it in-memory or as much with temporary files. I'm also Wondering If that stream could be encoded to Base64, and then uploaded passing the string to the send_keys function you can see above :$
PS: Hope you like the image :P
You seem to be asking multiple questions here.
First, how do you convert a a JPEG without downloading it to a file? You're already doing that, so I don't know what you're asking here.
Next, "And do it in-memory or as much with temporary files." I don't know what this means, but you can do it with temporary files with the tempfile library in the stdlib, and you can do it in-memory too; both are easy.
Next, you want to know how to do a streaming upload with requests. The easy way to do that, as explained in Streaming Uploads, is to "simply provide a file-like object for your body". This can be a tempfile, but it can just as easily be a BytesIO. Since you're already using one in your question, I assume you know how to do this.
(As a side note, I'm not sure why you're using BytesIO(r.content) when requests already gives you a way to use a response object as a file-like object, and even to do it by streaming on demand instead of by waiting until the full content is available, but that isn't relevant here.)
If you want to upload it with selenium instead of requests… well then you do need a temporary file. The whole point of selenium is that it's scripting a web browser. You can't just type a bunch of bytes at your web browser in an upload form, you have to select a file on your filesystem. So selenium needs to fake you selecting a file on your filesystem. This is a perfect job for tempfile.NamedTemporaryFile.
Finally, "I'm also Wondering If that stream could be encoded to Base64".
Sure it can. Since you're just converting the image in-memory, you can just encode it with, e.g., base64.b64encode. Or, if you prefer, you can wrap your BytesIO in a codecs wrapper to base-64 it on the fly. But I'm not sure why you want to do that here.
I am writing a bit of code that will take a CSV file input and perform an operation based on its contents. In the admin panel I am designing, the admin should be able to select a CSV file on their local system which my application will then read. The application does not need to store the CSV file, just read from it for a one-time operation.
Any ideas on how to best handle this in Pyramid?
What you want is essentially a file upload, followed by additional processing on the uploaded data. You can create input elements of type "file" in HTML forms to allow uploading of files.
Refer to the cookbook in the Pyramid documentation on file uploads for how to handle the uploaded data on the server side (summarized: use the file-like object request.POST[ field_name ].file).
I want to display the png or svg barcode generated by pyBarcode on a webpage. But I'm thinking it would be better if I don't have to save the image on the server. Is there anyway I can do this with pyBarcode? or with any other barcode image generation method with python?
I'm using pyramid as the web framework, if that changes anything.
Write the barcode straight out to the response.body_file:
barcode.writer import ImageWriter
ean = barcode.get_barcode('ean', '123456789102', writer=ImageWriter())
response = request.response
response.content_type = 'image/png'
ean.write(response.body_file)
I do not see how you can serve a file thats not even created . Consider saving it in /tmp if it won't be required for a long time but you will create and serve it once.
You should be able to do this by creating a HTTP response that returns the file object.
So that the request to "http://mysite.com/someimage.png" returns the file contents.
Try writing to the pyramid.response.Response object's .body_file.