Develop Reference

Retrive endpoint url of deployed app from google cloud run with Python

I want to send requests to a deployed app on a cloud run with python, but inside the test file, I don't want to hardcode the endpoint; how can I get the URL of the deployed app with python script inside the test file so that I can send requests to that URL?

You can use gcloud to fetch the url of the service like this
gcloud run services describe SERVICE_NAME
--format="value(status.url)"

In a pure Python way, you can use Google's API Client Library for Run.
To my knowledge, there isn't a Cloud Client Library
The method is namespaces.services.get and it is documented by APIs Explorer namespaces.services.get.
One important fact with Cloud Run is that the API endpoint differs by Cloud Run region.
See service endpoint. You will need to override the client configuration (using ClientOptions) with the correct (region-specific) api_endpoint.
The following is from-memory! I've not run this code but it should be (nearly) correct:
import google.auth
import os
from googleapiclient import discovery
from google.api_core.client_options import ClientOptions
creds, project = google.auth.default()
REGION = os.getenv("REGION")
SERVICE = os.getenv("SERVICE")
# Must override the default run.googleapis.com endpoint
# with region-specific endpoint
api_endpoint = "https://{region}-run.googleapis.com".format(
region=REGION
)
options = ClientOptions(
api_endpoint=api_endpoint
)
service = discovery.build("run", "v1",
client_options=options,
credentials=creds
)
name = "namespaces/{namespace}/services/{service}".format(
namespace=project,
service=SERVICE
)
rqst = service.namespaces().services().get(name=name)
resp = rqst.execute()
The resp will be Service and you can grab its ServiceStatus url.

Is it possible to download file to Google Cloud Storage via API call (with Python) using Google App Engine (without Google Compute Engine)

I wrote a python program which connected various platforms' API for file downloading purposes here. The program is currently running on my local machine (laptop) without a problem (all downloaded files saved to my local drive of course).
Here is my real question, without Google Compute Engine, is it possible to deploy the very same python program using Google App Engine? If yes, how could I save my files (via API calls) to Google Cloud Storage here?
Thanks.

Is this a Web App? If so you deploy it using GOOGLE APP ENGINE standard or flexible.
In order to send files to Cloud Storage, try the example in the python-docs-samples repo (folder appengine/flexible/storage/):
# [START upload]
#app.route('/upload', methods=['POST'])
def upload():
"""Process the uploaded file and upload it to Google Cloud Storage."""
uploaded_file = request.files.get('file')
if not uploaded_file:
return 'No file uploaded.', 400
# Create a Cloud Storage client.
gcs = storage.Client()
# Get the bucket that the file will be uploaded to.
bucket = gcs.get_bucket(CLOUD_STORAGE_BUCKET)
# Create a new blob and upload the file's content.
blob = bucket.blob(uploaded_file.filename)
blob.upload_from_string(
uploaded_file.read(),
content_type=uploaded_file.content_type
)
# The public URL can be used to directly access the uploaded file via HTTP.
return blob.public_url
# [END upload]

How to access a remote datastore when running dev_appserver.py?

I'm attempting to run a localhost web server that has remote api access to a remote datastore using the remote_api_stub method ConfigureRemoteApiForOAuth.
I have been using the following Google doc for reference but find it rather sparse:
https://cloud.google.com/appengine/docs/python/tools/remoteapi
I believe I'm missing the authentication bit, but can't find a concrete resource to guide me. What would be the easiest way, given the follow code example, to access a remote datastore while running dev_appserver.py?
import webapp2
from google.appengine.ext import ndb
from google.appengine.ext.remote_api import remote_api_stub
class Topic(ndb.Model):
created_by = ndb.StringProperty()
subject = ndb.StringProperty()
#classmethod
def query_by_creator(cls, creator):
return cls.query(Topic.created_by == creator)
class MainPage(webapp2.RequestHandler):
def get(self):
remote_api_stub.ConfigureRemoteApiForOAuth(
'#####.appspot.com',
'/_ah/remote_api'
)
topics = Topic.query_by_creator('bill')
self.response.headers['Content-Type'] = 'text/plain'
self.response.out.write('<html><body>')
self.response.out.write('<h1>TOPIC SUBJECTS:<h1>')
for topic in topics.fetch(10):
self.response.out.write('<h3>' + topic.subject + '<h3>')
self.response.out.write('</body></html>')
app = webapp2.WSGIApplication([
('/', MainPage)
], debug=True)

This get's asked a lot, simply because you can't use app engines libraries outside of the SDK. However, there is also an easier way to do it from within the App Engine SDK as well.
I would use gcloud for this. Here's how to set it up:
If you want to interact with google cloud storage services inside or outside of the App Engine environment, you may use Gcloud (https://googlecloudplatform.github.io/gcloud-python/stable/) to do so.
You need a service account on your application as well as download the JSON credentials file. You do this on the app engine console under the authentication tab. Create it, and then download it. Call it client_secret.json or something.
With those, once you install the proper packages for gcloud with pip, you'll be able to make queries as well as write data.
Here is an example of authenticating yourself to use the library:
from gcloud import datastore
# the location of the JSON file on your local machine
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/location/client_secret.json"
# project ID from the Developers Console
projectID = "THE_ID_OF_YOUR_PROJECT"
os.environ["GCLOUD_TESTS_PROJECT_ID"] = projectID
os.environ["GCLOUD_TESTS_DATASET_ID"] = projectID
client = datastore.Client(dataset_id=projectID)
Once that's done, you can make queries like this:
query = client.query(kind='Model').fetch()
It's actually super easy. Any who, that's how I would do that! Cheers.

Deploying Flask app to via Elastic Beanstalk which interacts with S3

I have a Flask app which looks like this:
from flask import Flask
import boto3
application = Flask(__name__)
#application.route("/")
def home():
return "Server successfully loaded"
#application.route("/app")
def frontend_from_aws():
s3 = boto3.resource("s3")
frontend = s3.Object(bucket_name = "my_bucket", key = "frontend.html")
return frontend.get()["Body"].read()
if __name__ == "__main__":
application.debug = True
application.run()
Everything works perfectly when I test locally, but when I deploy the app to Elastic Beanstalk the second endpoint gives an internal server error:
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
I didn't see anything alarming in the logs, though I'm not completely sure I'd know where to look. Any ideas?
Update: As a test, I moved frontend.html to a different bucket and modified the "/app" endpoint accordingly, and mysteriously it worked fine. So apparently this has something to do with the settings for the original bucket. Does anybody know what the right settings might be?

I found a quick and dirty solution: IAM policies (AWS console -> Identity & Access Management -> Policies). There was an existing policy called AmazonS3FullAccess, and after I attached aws-elasticbeanstalk-ec2-role to it my app was able to read and write to S3 at will. I'm guessing that more subtle access management can be achieved by creating custom roles and policies, but this was good enough for my purposes.

Did you set up your AWS credentials on your Elastc Beanstalk instance as they are on your local machine (i.e. in ~/.aws/credentials)?

How does GCS know which project am I using?

A simple Python code to list obj in certain folder in Google Cloud Storage:
from apiclient import discovery
import apiclient
import json
client = discovery.build('storage', 'v1beta2')
request = client.objects().list(
bucket = 'mybucket',
prefix = 'myfolder1/myfolder',
key = 'A0rsER3odwksawsesse3Dw_d3Ks') # my API key
try:
response = request.execute()
print json.dumps(response, indent = 2)
except apiclient.errors.HttpError, e:
print e
Then I got below failure message:
https://www.googleapis.com/storage/v1beta2/b/mybucket/o?prefix=myfolder1%2Fmyfolder&alt=json&key=A0rsER3odwksawsesse3Dw_d3Ks returned "*Access Not Configured. Please use Google Developers Console to activate the API for your project.*">
I have enabled the API for my project, so it should not be the problem as the return message said, maybe it jump to incorrect project? As I know the flow should be -> project in GCS -> bucket -> root-folder -> folder-1.
So my question is how do this code know which project I'm using? Do I miss any code here?
Thanks for all the kind help!

API key alone is only good for accessing data in public buckets. GCS is able to map API usage back to your project by looking at the API key, which is unique. For anything else than public data access you need proper authentication, via OAuth2, for example. More info here:
https://developers.google.com/storage/docs/json_api/v1/how-tos/authorizing
Your code looks OK and it works with public buckets, which I've verified. Ensure you have both Google Cloud Storage and Google Cloud Storage JSON API enabled in Developers Console.