I've been googling for a solution for some time and tried a couple of ways to solve this.
In short:
I used the sample from https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/getstarted to create my own uploader, but it dies on the error, mentioned here:
No api proxy found for service "app_identity_service" when running GAE script
So, from what I understand, the script needs to be uploaded to google app engine and run from there using App Engine console. But even if it's possible, how do I automate it ?
Or maybe there are other solutions I'm missing. I looked through appcfg.pya but didn't find such an option as well.
You are following a sample to upload from GAE to Cloud Storage. If your only goal is to upload files to Cloud storage, then simply use gsutil. You can easily script with gsutil, do streaming copy, copy full directories and rsync a file system.
Why you need GAE in your solution ?
Google App Engine allows you to easily upload static files and serve them but if you simply just want a place to store static files then Google Cloud Storage is the way to go. It's much easier to use the gsutil tool to automate uploading your content than deploying using the App Engine SDK. The infrastructure serving the Cloud Storage files is the same as App Engine so there's really no advantage to using App Engine's static files feature.
Also, If you need a way to set up a custom domain, index page, and/or error pages you may want to check out the guide on Configuring a Bucket as a Website.
Related
Does anyone have any good suggestions for where I can store my custom Python modules on Google Cloud Platform?
I have a bunch of modules that I would like to access from the different GCP services I am using (App Engine, Compute Engine, Cloud Functions etc), without having to copy the Python files and upload to the service's Python environment each time.
I was thinking GCS could be an option but then I am not sure how I would then get the module into, say Cloud Functions or App Engine?
Any ideas?
The code will eventually need to be written to your service's local storage. Python does not access code remotely during execution unless you write your code to do so (download the module and then execute). Package your code as modules and publish to PyPI and then add them as dependencies. When you deploy a service, your modules will be downloaded.
How might one send data from Twitter directly to Google Cloud data storage. Would like to skip the step of first downloading it down to my local machine and then uploading it up to the cloud. It would run once. Not looking for full code, but any pointers or tutorials that someone might have learned from. Using python to interact with google-cloud and storage.
Any help would be appreciated.
Here's a blog post which describes the following architecture:
Run a Python script on Compute Engine
Moving your data to BigQuery for storage
Here's another one that describes a somewhat more complex architecture, including the ability to analyze tweets:
Use Google Cloud Dataflow templates
Launch Dataflow pipelines from a Google App Engine (GAE) app
In order to support MapReduce jobs
For a project, our team is using the app engine python environment to host several scripts that are scraping a website to store data in the form of various json files and directories of images. We want to expose these directories to a url (Ex: /img/01.jpg in the app engine directory to "sample.appspot.com/img/01.jpg"). The reason is that we want to be able to download these files directly to a react-native mobile app using the fetch api. Is this feasable, efficient, and quick using app engine and how? If not what combination of google cloud services could we use to achieve the same functionality and how?
You could use Google Cloud Storage to store your files:
(for flexible environment) Application code
(for standard environment) Writing to Cloud Storage
Once stored they're pretty much static files, so for serving them you have 2 options:
serve them as static content, directly from GCS, see, for example Serving from Cloud Storage.
I'd suspect this would be faster. Your app's environment doesn't matter, since it's not even involved.
serve them dynamically, through your app's URLs & handlers, with your app reading them from GCS. See, for example:
(for flexible environment) Serving from your application
(for standard environment) Reading from Cloud Storage
Referencing Manage Google Drive files through App Engine, I have kind of the same question about App Engine w.r.t. Google Code Project Hosting, i.e., is there a Python API to interact with the latter using the former. Thanks!
It's possible to programatically upload files to a Google Code project, see the script at:
https://code.google.com/p/support/wiki/ScriptedUploads
What else were you looking to automate?
I have developed an app using Flask and App Engine. I was able to upload file in google cloud storage, but not able to delete file.
I am using: files.delete('/gs/bucket/file.pdf') to delete a file.
App executes this line, but file is not deleted from cloud storage.
According to https://developers.google.com/appengine/docs/python/googlestorage/functions#delete, it should work.
It is not even working in local development server also.
Please let me know, what I am missing or is there any better to achieve this?