Develop Reference

Python web crawling/scraping - Download diagram(PDF or TIFF) from Webpage and save to Local machine

I have one website which has search button and i need to give some numeric value and give enter button. It will go to another page and it display some content in which there are some URL, if i click that URL, it will ask to save diagram and the diagram is either tiff format or PDF.
To download Tiff format diagram, i am using swift plugin in internet explore and save to my machine
Here i am doing this work manually, just i want to do automate this whole process.
Steps:
Using python request module and pass the URL with numeric value to post method
save response content to variable
perform pattern matching and fetch url
click the url but i am stuck with this part to save the diagram local since it is tiff.
is there any module to download tiff based diagram and save to local machine?

Just I want to share How i resolved the issue for the above question and it might be useful for others.
Since tiff image needs to be downloaded from web, so I used python request module with pillow module as below,
from PIL import image
import requests
tiffURL='https://***.tif'
img=Image.open(requests.get(tiffURL,stream=True).raw)
img.save('imagename.jpg')
#img.save('imagename.jpg',quality=95)
Note:
tiff image can not be viewed by normal editor , so i converted to jpg
if you want high resoultion, you can pass quality=95 to save method

receiving PNG file in django

I have a falcon server that I am trying to port to django. One of the falcon endpoints processes a request that contains a PNG file sent with content_type = 'application/octet-stream'. It writes the data to a file maintaining the correct PNG structure.
The falcon code does this:
form = cgi.FieldStorage(fp=req.stream, environ=req.env)
and then writes the png like this:
fd.write(form[key].file.read())
I cannot figure out how to do the same thing in django. When my view is called the data in request.POST[key] has already been decoded to unicode text and it's no longer valid png data.
How can I do this with django? Should/can I use cgi.FieldStorage? The request I get (of type django.core.handlers.wsgi.WSGIRequest) does not have a stream method. I'm sure there's some way to do this, but I have not come up with anything googling.

I solved this by changing the client to set the file and filename fields each part of the multipart and then I was able it iterate through request.FILES and successfully write the files as PNG.

Upload image with an in-memory stream to input using Pillow + WebDriver?

I'm getting an Image from URL with Pillow, and creating an stream (BytesIO/StringIO).
r = requests.get("http://i.imgur.com/SH9lKxu.jpg")
stream = Image.open(BytesIO(r.content))
Since I want to upload this image using an <input type="file" /> with selenium WebDriver. I can do something like this to upload a file:
self.driver.find_element_by_xpath("//input[#type='file']").send_keys("PATH_TO_IMAGE")
I would like to know If its possible to upload that image from a stream without having to mess with files / file paths... I'm trying to avoid filesystem Read/Write. And do it in-memory or as much with temporary files. I'm also Wondering If that stream could be encoded to Base64, and then uploaded passing the string to the send_keys function you can see above :$
PS: Hope you like the image :P

You seem to be asking multiple questions here.
First, how do you convert a a JPEG without downloading it to a file? You're already doing that, so I don't know what you're asking here.
Next, "And do it in-memory or as much with temporary files." I don't know what this means, but you can do it with temporary files with the tempfile library in the stdlib, and you can do it in-memory too; both are easy.
Next, you want to know how to do a streaming upload with requests. The easy way to do that, as explained in Streaming Uploads, is to "simply provide a file-like object for your body". This can be a tempfile, but it can just as easily be a BytesIO. Since you're already using one in your question, I assume you know how to do this.
(As a side note, I'm not sure why you're using BytesIO(r.content) when requests already gives you a way to use a response object as a file-like object, and even to do it by streaming on demand instead of by waiting until the full content is available, but that isn't relevant here.)
If you want to upload it with selenium instead of requests… well then you do need a temporary file. The whole point of selenium is that it's scripting a web browser. You can't just type a bunch of bytes at your web browser in an upload form, you have to select a file on your filesystem. So selenium needs to fake you selecting a file on your filesystem. This is a perfect job for tempfile.NamedTemporaryFile.
Finally, "I'm also Wondering If that stream could be encoded to Base64".
Sure it can. Since you're just converting the image in-memory, you can just encode it with, e.g., base64.b64encode. Or, if you prefer, you can wrap your BytesIO in a codecs wrapper to base-64 it on the fly. But I'm not sure why you want to do that here.

How to return image with GAE?

Each time user accesses http://www.example.com/some-random-symbols-1x1.png, I should return transparent image 1x1px. I've created according file, but how should I read it in my code to return to the user? I know how to display the image from the datastore or blobstore. But have no idea how to read and return binary file.
It can not be static file due to the following reasons:
url will contain some-random-symbols;
once url is accessed, prior to displaying images, I would like to log that somebody accessed the file.

A 1x1 transparent PNG (or GIF) is small enough that you can hard-code the base64 representation directly and emit it directly via self.response.write() (after decoding).
Reading from disk every time is relatively expensive. If you want to go that route, lazily initialize a global variable.

In a more general case, I'd use the blobstore and the BlobstoreDownloadHandler, but for a tiny gif that will definitely fit into memory, something like this to read the file's content:
with open('path/to/file.gif') as f:
img_content = f.read()
I'd put this outside of my handler, so it was done once per instance. If you're using 2.5, then you'll need to import 'with' from future, or open and close the file yourself.
then in your handler, assuming webapp2:
self.response.content_type = 'image/gif'
self.response.write(img_content)

Django - Creating & Store PDF Files using XHTML2PDF

As of now we are using XHTML2PDF to dynamically generate PDFs and outputting to browser whenever required. Now our requirements is changed to generate the PDF only once and store it in the server. The link should be displayed to user to view the PDF. Could you please point out any resources or snippets to achieve this?

This is pretty easy to do. Observe:
from django.core.files.base import ContentFile
# get_pdf_contents should return the binary information for
# a properly formed pdf doc.
pdf_contents = get_pdf_contents()
file_to_be_saved = ContentFile(pdf_contents)
item = Item.objects.get(pk=1)
item.myfilefield.save('blarg.pdf', file_to_be_saved)
The get_pdf_contents function shouldn't be too hard to write - basically take whatever function you have already and chop it off before it funnels the results into an HttpResponse object. If you need help with that, post the code you have already.