I have a few different applications that require me to POST a file FROM Google App Engine to a remote site. I've tried a few approaches with urllib2, but I've run into problems with each approach as I have moved the code into GAE.
What is the simplest way to post a file (csv, zip, etc.) from Google App Engine to a remote website? Once I can post an existing file, I can move on to posting files from the datastore.
Have you looked at urlfetch. Example from docs.
import urllib
from google.appengine.api import urlfetch
with open('/file', 'r') as f:
data = f.read()
result = urlfetch.fetch(url=url,
payload=data,
method=urlfetch.POST,
headers={'Content-Type': 'application/x-www-form-urlencoded'})
From the reference,
payload: Body content for a POST or PUT request.
So just load the contents of the file and set the payload a la
with open(filename, 'r') as fh:
payload = fh.read()
response = urlfetch.fetch(url, payload=payload, method='POST')
and do what you would with response.
This would work in the exact same fashion with a string from a datastore object.
EDIT: filename will likely be a path relative to your project. So if your project lives in /home/dinosaurs/sinclair on your local machine and you have /home/dinosaurs/sinclair/stuff/contents.xml in your project, then your relative path that will work in production on App Engine is stuff/contents.xml.
Related
I need to connect an API on azurewebsites using Python to download a JSON file automatically.
I can access the website and download a JSON file manually.
I tried to connect using:
url = 'https://myplatformconnectiot.azurewebsites.net/swagger/index.html'
r = requests.get(url, headers={"Authentication": " application/json"},cookies={},auth=('user#example.com', 'password'),)
r.json()
Do you know how to download a JSON file in azurewebsites using Python?
You need to use the kudu console url to a get particular file download from a web app.
By using the below python code you can download the file form the web app
import json
import requests
url = 'https://<webappname>.scm.azurewebsites.net/wwwroot/wwwroot/css/site.css'
r = requests.get(url,auth=('username','urlpassword'))
with open(r'C:\Users\name.json','wb') as f:
f.write(r.content)
username & password will be from publish profile credentials file of a web app. you can get the publish profile credentials from portal as shown in below image
Kudu is the engine behind a number of features in Azure App Service related to source control based deployment, and other deployment methods like Dropbox and OneDrive sync.
for more information about kudu you can refer the below document
I'm trying to enable the downloading of previously uploaded files in Django, here's the code I'm using so far:
def downloadview(request):
path=os.path.join('media', 'files', '5560026113', '20180412231515.jpg' )
response = HttpResponse()
response['Content-Type']=''
response['Content-Disposition'] = "attachment; filename='Testname'"
response['X-Sendfile']=smart_str(os.path.join(path))
return response
The inspiration for this trial comes from this thread but I don't get it working. An empty txt file is downloaded instead of the image that is stored on the server.
In this trial code the exact filename and extension is hard coded in the path variable.
Here is a way you can serve a file through Django (although it is usually not a good approach, the better one is to serve files with a webserver like nginx etc - for performance reasons):
from mimetypes import guess_type
from django.http import HttpResponse
file_path=os.path.join('media', 'files', '5560026113', '20180412231515.jpg' )
with open(file_path, 'rb') as f:
response = HttpResponse(f, content_type=guess_type(file_path)[0])
response['Content-Length'] = len(response.content)
return response
guess_type infers the content_type from the file extension.
https://docs.python.org/3/library/mimetypes.html
more on HttpResponse here: https://docs.djangoproject.com/en/2.0/ref/request-response/#django.http.HttpResponse
And here is why it is not recommended to serve files through Django, although not recommended just means that you should probably understand what you are doing:
https://docs.djangoproject.com/en/2.0/howto/static-files/deployment/
I am trying to get Google App Engine to gunzip my .gz blob file (single file compressed) automatically by setting the response headers as follows:
class download(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
resource = str(urllib.unquote(resource))
blob_info = blobstore.BlobInfo.get(resource)
self.response.headers['Content-Encoding'] = str('gzip')
# self.response.headers['Content-type'] = str('application/x-gzip')
self.response.headers['Content-type'] = str(blob_info.content_type)
self.response.headers['Content-Length'] = str(blob_info.size)
cd = 'attachment; filename=%s' % (blob_info.filename)
self.response.headers['Content-Disposition'] = str(cd)
self.response.headers['Cache-Control'] = str('must-revalidate, post-check=0, pre-check=0')
self.response.headers['Pragma'] = str(' public')
self.send_blob(blob_info)
When this runs, the file is downloaded without the .gz extension. However, the downloaded file is still gzipped. The file size of the downloaded data match the .gz file size on the server. Also, I can confirm this by manually gunzipping the downloaded file. I am trying to avoid the manual gunzip step.
I am trying to get the blob file to automatically gunzip during the download. What am I doing wrong?
By the way, the gzip file contains only a single file. On my self-hosted (non Google) server, I could accomplish the automatic gunzip by setting same response headers; albeit, my code there is written in PHP.
UPDATE:
I rewrote the handler to serve data from the bucket. However, this generates HTML 500 error. The file is partially downloaded before the failure. The rewrite is as follows:
class download(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
resource = str(urllib.unquote(resource))
blob_info = blobstore.BlobInfo.get(resource)
file = '/gs/mydatabucket/%s' % blob_info.filename
print file
self.response.headers['Content-Encoding'] = str('gzip')
self.response.headers['Content-Type'] = str('application/x-gzip')
# self.response.headers['Content-Length'] = str(blob_info.size)
cd = 'filename=%s' % (file)
self.response.headers['Content-Disposition'] = str(cd)
self.response.headers['Cache-Control'] = str('must-revalidate, post-check=0, pre-check=0')
self.response.headers['Pragma'] = str(' public')
self.send_blob(file)
This downloads 540,672 bytes of the 6,094,848 bytes file to the client before the server terminate and issued a 500 error. When I issue 'file' on the partially downloaded file from the command line, Mac OS seems to correctly identify the file format as 'SQLite 3.x database' file. Any idea of why the 500 error on the server? How can I fix the problem?
You should first check to see if your requesting client supports gzipped content. If it does support gzip content encoding, then you may pass the gzipped blob as is with the proper content-encoding and content-type headers, otherwise you need to decompress the blob for the client. You should also verify that your blob's content_type isn't gzip (this depends on how you created your blob to begin with!)
You may also want to look at Google Cloud Storage as this automatically handles gzip transportation so long as you properly compress the data before storing it with the proper content-encoding and content-type metadata.
See this SO question: Google cloud storage console Content-Encoding to gzip
Or the GCS Docs: https://cloud.google.com/storage/docs/gsutil/addlhelp/WorkingWithObjectMetadata#content-encoding
You may use GCS as easily (if not more easily) as you use the blobstore in AppEngine and it seems to be the preferred storage layer to use going forward. I say this because the File API has been deprecated which made blobstore interaction easier and great efforts and advancements have been made to the GCS libraries making the API similar to the base python file interaction API
UPDATE:
Since the objects are stored in GCS, you can use 302 redirects to point users to files rather than relying on the Blobstore API. This eliminates any unknown behavior of the Blobstore API and GAE delivering your stored objects with the content-type and content-encoding you intended to use. For objects with a public-read ACL, you may simply direct them to either storage.googleapis.com/<bucket>/<object> or <bucket>.storage.googleapis.com/<object>. Alternatively, if you'd like to have application logic dictate access, you should keep the ACL to the objects private and can use GCS Signed URLs to create short lived URLs to use when doing a 302 redirect.
Its worth noting that if you want users to be able to upload objects via GAE, you'd still use the Blobstore API to handle storing the file in GCS, but you'd have to modify the object after it was uploaded to ensure proper gzip compressing and content-encoding meta data is used.
class legacy_download(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
filename = str(urllib.unquote(resource))
url = 'https://storage.googleapis.com/mybucket/' + filename
self.redirect(url)
GAE already serves everything using gzip if the client supports it.
So I think what's happening after your update is that the browser expects there to be more of the file, but GAE thinks it's already at the end of the file since it's already gzipped. That's why you get the 500.
(if that makes sense)
Anyway, since GAE already handles compression for you, the easiest way is probably to put non compressed files in GCS and let the Google infrastructure handle the compression automatically for you when you serve them.
I'm trying to upload files to the blobstore in my Google App without using a form. But I'm stuck at how to get the app to read my local file. I'm pretty new to python and Google Apps but after some cut and pasting I've ended up with this:
import webapp2
import urllib
import os
from google.appengine.api import files
from poster.encode import multipart_encode
class Upload(webapp2.RequestHandler):
def get(self):
# Create the file in blobstore
file_name = files.blobstore.create(mime_type='application/octet-stream')
# Get the local file path from an url param
file_path = self.request.get('file')
# Read the file
file = open(file_path, "rb")
datagen, headers = multipart_encode({"file": file})
data = str().join(datagen) # this is supposedly memory intense and slow
# Open the blobstore file and write to it
with files.open(file_name, 'a') as f:
f.write(data)
# Finalize the file. Do this before attempting to read it.
files.finalize(file_name)
# Get the file's blob key
blob_key = files.blobstore.get_blob_key(file_name)
The problem now is I don't really know how to get hold of the local file
You can't read from the local file system from within the application itself, you will need to use http POST to send the file to the app.
You can certainly do this from within another application - you just need to create the mime multipart message with the file content and POST it to your app, the sending application will just have to create the http request that you will post to the app manually. You should have a read on how to create a mime mulitpart message using c#.
I use python Appengine. I'm trying to create a link on a webpage, which a user can click to download a csv file. How can I do this?
I've looked at csv module, but it seems to want to open a file on the server, but appengine doesn't allow that.
I've looked at remote_api, but it seems that its only for uploading or downloading using app config, and from account owner's terminal.
Any help thanks.
Pass a StringIO object as the first parameter to csv.writer; then set the content-type and content-disposition on the response appropriately (probably "text/csv" and "attachment", respectively) and send the StringIO as the content.
I used this code:
self.response.headers['Content-Type'] = 'application/csv'
writer = csv.writer(self.response.out)
writer.writerow(['foo','foo,bar', 'bar'])
Put it in your handler's get method. When user requests it, user's browser will download the list content automatically.
Got from: generating a CSV file online on Google App Engine