I'm using the Firebase Admin Python SDK to read/write data to Firestore. I've created a service account with the necessary permissions and saved the credentials .json file in the source code (I know this isn't the most secure, but I want to get the thing running before fixing security issues). When testing the integration locally, it works flawlessly. But after deploying to GCP, where our service is hosted, calls to Firestore don't work properly and retry for a while before throwing 503 Deadline Exceeded errors. However, SSHing into a GKE pod and calling the SDK manually works without issues. It's just when the SDK is used in code flow that causes problems.
Our service runs in Google Kubernetes Engine in one project (call it Project A), but the Firestore database is in another project (call it project B). The service account that I'm trying to use is owned by Project B, so it should still be able to access the database even when it is being initialized from inside Project A.
Here's how I'm initiating the SDK:
from firebase_admin import get_app
from firebase_admin import initialize_app
from firebase_admin.credentials import Certificate
from firebase_admin.firestore import client
from google.api_core.exceptions import AlreadyExists
credentials = Certificate("/path/to/credentials.json")
try:
app = initialize_app(credential=credentials, name="app_name")
except ValueError:
app = get_app(name="app_name")
client = client(app=app)
Another wrinkle is that another part of our code is able to successfully use the same service account to produce Firebase Access Tokens. The successful code is:
import firebase_admin
from firebase_admin import auth as firebase_admin_auth
if "app_name" in firebase_admin._apps:
# Already initialized
app = firebase_admin.get_app(name="app_name")
else:
# Initialize
credentials = firebase_admin.credentials.Certificate("/path/to/credentials.json")
app = firebase_admin.initialize_app(credential=credentials, name="app_name")
firebase_token = firebase_admin_auth.create_custom_token(
uid="id-of-user",
developer_claims={"admin": is_admin, "site_slugs": read_write_site_slugs},
app=app,
)
Any help appreciated.
Turns out that the problem here was a conflict between gunicorn's gevents and the SDK's use of gRCP. Something related to websockets. I found the solution here. I added the following code to our Django app's settings:
import grpc.experimental.gevent as grpc_gevent
grpc_gevent.init_gevent()
Related
I want to send requests to a deployed app on a cloud run with python, but inside the test file, I don't want to hardcode the endpoint; how can I get the URL of the deployed app with python script inside the test file so that I can send requests to that URL?
You can use gcloud to fetch the url of the service like this
gcloud run services describe SERVICE_NAME
--format="value(status.url)"
In a pure Python way, you can use Google's API Client Library for Run.
To my knowledge, there isn't a Cloud Client Library
The method is namespaces.services.get and it is documented by APIs Explorer namespaces.services.get.
One important fact with Cloud Run is that the API endpoint differs by Cloud Run region.
See service endpoint. You will need to override the client configuration (using ClientOptions) with the correct (region-specific) api_endpoint.
The following is from-memory! I've not run this code but it should be (nearly) correct:
import google.auth
import os
from googleapiclient import discovery
from google.api_core.client_options import ClientOptions
creds, project = google.auth.default()
REGION = os.getenv("REGION")
SERVICE = os.getenv("SERVICE")
# Must override the default run.googleapis.com endpoint
# with region-specific endpoint
api_endpoint = "https://{region}-run.googleapis.com".format(
region=REGION
)
options = ClientOptions(
api_endpoint=api_endpoint
)
service = discovery.build("run", "v1",
client_options=options,
credentials=creds
)
name = "namespaces/{namespace}/services/{service}".format(
namespace=project,
service=SERVICE
)
rqst = service.namespaces().services().get(name=name)
resp = rqst.execute()
The resp will be Service and you can grab its ServiceStatus url.
I am currently in the process of creating a Web App (Flask) where users can log in and upload photos to Google Firebase. At one point in the code I was initially saving user uploaded photos to a local folder, but when deployed this doesn't work correctly so I decided to temporarily store it in Google Storage, analyze the faces in it, then delete it. However, I am unable to delete it from Google Storage for some reason.
Firebase Initilizations:
import pyrebase
from pyrebase.pyrebase import storage
import firebase_admin
from firebase_admin import storage, credentials
firebase = pyrebase.initialize_app(json.load(open('firebase/firebaseConfig.json')))
auth = firebase.auth()
db = firebase.database()
storage = firebase.storage()
I have not needed to delete the photos in storage before, but I am able to store Images as well as retrieve their URLs for download as seen below. I am certain the image is stored in my Google Storage and the try fails when I attempt the storage.delete()
try:
storage.child("images/temp/" + filename).put(image, userIdToken)
current_app.logger.info("[UPLOAD-IMAGE] Photo saved, grabbing url")
imageURL = storage.child("images/temp/" + filename).get_url(None)
anazlyzeInfo = recognize.facialRecognition(imageURL)
delete_temp_image_path = "images/temp/" + filename
#storage.delete(imageURL) # same error happens when URL is passed
storage.delete(delete_temp_image_path)
The error described in the exception is: 'Storage' object has no attribute 'bucket'
I looked into this for a while and tried other solutions like StorageRef
and was met with the error 'Storage' has no attribute 'ref'.
I also tried A Service Account following the Google Admin SDK setup but am not sure what to do now that I have:
cred = credentials.Certificate(json.load(open('firebase/fiddl-dev-firebase-adminsdk-stuffdnsnfsnfk.json')))
admin = firebase_admin.initialize_app(cred)
I tried working with this for a while but I could not figure out what was callable with admin.
Was I on the correct path with either of the two fixes I attempted? My use of Firebase was pretty low level before and I would think that deleting would be the same. Thanks!
I’m the OP and I figured out my issue! This GitHub post helped me learn that you need to "add a service account to the config" when getting the 'storage' has not 'bucket' error.
To do this I followed the Firebase Admin Documentation which was pretty straight forward.
However there were 2 main fixes I needed. I fixed this using this Stackoverflow post as a guide.
The first was adding my storageBucket for my app which I was missing above.
admin = firebase_admin.initialize_app(cred, {
'storageBucket': 'fiddl-dev.appspot.com'})
The second issues was when I was trying the bucket = storage.bucket() seen in the same [Stackoverflow] post I was getting the error that storage didn’t have an attribute bucket. This I couldn’t find anything on and was why I made the post.
At the top of my file.py I import:
import pyrebase
from pyrebase.pyrebase import storage
import firebase_admin
from firebase_admin import storage as admin_storage, credentials, firestore
The key being that I added import storage as admin_storage rather than what I had import storage. Essentially I was importing a module named storage twice and it was getting confused.
With that last change I was now able to test the following code that deleted the image from the filepath in Google Firebase Storage specified.
bucket = admin_storage.bucket()
blob = bucket.blob('images/temp/pumpkin.jpg')
print(blob)
blob.delete()
I'm using the python google.cloud api
For example using the metrics module
from google.cloud import monitoring
client = monitoring.Client()
client.query(my/gcp/metric, minutes=10)
For my GOOGLE_APPLICATION_CREDENTIALS im using a service account that has specific access to a gcp project.
Does google.cloud have any modules that can let me derive the project from the service account (like get what project the service account is in)?
This would be convenient because each service account only has access to a single project, so I could set my service account and be able to reference that project in code.
Not sure if this will work, you may need to tweak it:
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
credentials = GoogleCredentials.get_application_default()
service = discovery.build('yourservicename', credentials=credentials)
request = service.projects().list()[0]
Google Cloud Identity and Access Management (IAM) API has ‘serviceAccounts.get’ method and which shows the projects associated with a service account as shown here. You need to have proper permissions on the projects for the API to work.
The method google.auth.default return a tuple (project_id, credentials) if that information is available on the environment.
Also, the client object knows to which project it is linked from (either client.project or client.project_id, I'm not sure which one for the Monitoring API).
If you set the service account manually with the GOOGLE_APPLICATION_CREDENTIALS env var, you can open the file and load its json. One of the parameters in a service account key file is the project id.
I'm trying to get a list of Drive files on my App Engine. The app is not for consumers other than the app so I'm not implementing OAuth2 for user sign in.
I have enabled Drive in the GCP project and the compute instance is an owner. However, I receive the following error:
Traceback (most recent call last):
File "test.py", line 5, in <module>
credentials = app_engine.Credentials(scopes=SCOPES)
File "/usr/local/lib/python2.7/site-packages/google/auth/app_engine.py", line 105, in __init__
'The App Engine APIs are not available.')
EnvironmentError: The App Engine APIs are not available.
Here is my code (based off this tutorial: https://developers.google.com/api-client-library/python/auth/service-accounts):
from google.auth import app_engine
import googleapiclient.discovery
SCOPES = ['https://www.googleapis.com/auth/drive']
credentials = app_engine.Credentials(scopes=SCOPES)
project_id = 'project-id'
service = googleapiclient.discovery.build('drive', 'v3',credentials=credentials)
bucket = service.files().list(project=project_id).execute()
print bucket
Something is going on with my credentials and I'm unsure what it is.
Tldr; How do I authorize my App Engine Flask App (Standard) to consume Google APIs, including Google Sheets, Drive, BigQuery and others. Keep in mind, this is a server-to-server job. No users are logging in other than the app itself.
Ultimately, I'm trying to have values in Google Sheets synced with BigQuery (the app needs Drive and BQ permissions) that will then get pushed to DataStore.
If you have suggestions on how to fix it, ELI5.
Looking at the tutorial that you are following there is a note that says if you need to test you application locally you should use other credential mechanism. In this case you have to obtain and provide the service account credentials manually.
You need to get a JSON file with the service account key and then on your code use the service account credentials by specifying the private key file.
There is another good tutorial that maybe can be useful for setting up authentication for server to server production applications.
I am trying to set up a django app which needs access to my own (not a user's) google calendar. I'd rather not have to go through the whole oauth process just to do this, so I set up my authentication in a file (called calendar.dat) in the same directory as the django view which uses my calendar. The code to authorize my account is as follows:
import os.path
import httplib2
from oauth2client.file import Storage
module_dir = os.path.dirname(__file__)
file_path = os.path.join(module_dir, 'calendar.dat')
storage = Storage(file_path)
credentials = storage.get()
http = hhtplib2.Http()
http = credentials.authorize(http)
When this code runs through a call from the browser I get the error that credentials = None. However, if I put this code in a normal python file in the same directory and just run it in the terminal it works. Is there something about django that is messing this up and is there a way I can fix this? Somewhat of a beginner, sorry if this is a novice question.