I've been making api with flask on Google App Engine and When I send request to this app from browser after deploy, I got 502 error. I'm sure this error is caused by credential of GCP by "gcloud app logs tail -s test" but The path of credential Json file and file name seems OK . I have googled and I tried every articles I have found there but could not solve.
I have already done export GOOGLE_APPLICATION_CREDENTIALS="/home/user/secret_key/bq.json"
Could anyone tell me the solution??
If there is lack of any info , please let me know . Thank you .
besides, my api function is getting luid parameter over http request and run SQL with that luid and if the row of the luid has data in cv_date column in BigQuery, it returns True to client.
【The result of "gcloud app logs tail -s test"】
File "/env/lib/python3.7/site-packages/google/auth/_default.py", line 97, in load_credentials_from_file "File {} was not found.".format(filename) google.auth.exceptions.DefaultCredentialsError: File /home/user/secret_key/bq.json was not found.
【/home/user/api_dev/main.py】
from flask import Flask,request
from google.cloud import bigquery
import os
credentials_json = '/home/user/secret_key/bq.json'
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = credentials_json
client = bigquery.Client()
app = Flask(__name__)
#app.route('/')
def get_request():
request_luid = request.args.get('luid') or ''
query = """
SELECT EXISTS(SELECT cv_date FROM `test-266110.conversion_log.conversion_log_202008*` t WHERE request_luid = p.luid)
"""
query_res = client.query(query)
return query_res
if __name__ == "__main__":
app.run()
【Remove the codes for BigQuery except import library and variables】
*This code works well and returns luid you input on url parameter
from flask import Flask, request
from google.cloud import bigquery
import os
credentials_json = '/home/user/secret_key/bq.json'
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = credentials_json
app = Flask(__name__)
#app.route('/')
def get_request():
request_luid = request.args.get('luid') or ''
return request_luid
if __name__ == "__main__":
app.run()
I'd recommend reading through the auth docs.
https://cloud.google.com/docs/authentication/production talks about service account interactions in a bit more detail. You likely don't need to pass in your credentials in the live app. You can simply set the GOOGLE_APPLICATION_CREDENTIALS when you're running locally to use the credentials, but you don't need to set it in production.
The issue is that the path you've specified (/home/user/secret_key/bq.json) is only valid for your development environment, and either not included in your production deployment at all or the absolute path to the file in the deployed app is different.
Related
I want to set my google json credential file at run time through postman. I have made a bigquery rest api. Right now I am passing it in my code like this:
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = 'C:/Users/Documents/bigQuery/service.json'
#app.route('/', methods=['GET', 'POST'])
def get_request():
query:
.
.
.
return results
if __name__ == "__main__":
app.run()
I tried taking the input from form data in postman approach but the code throws compile time error asking for credential file to be set up first.
compilation error:
DefaultCredentialsError: Could not automatically determine credentials.
Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see
https://cloud.google.com/docs/authentication/getting-started
Setting the GOOGLE_APPLICATION_CREDENTIALS is needed for the code to fetch the big query data.
I want postman for passing the key.json file to the code at run time.
Instead of setting the env variable, there are 2 other options:
Create the client using client = bigquery.Client.from_service_account_json(json_credentials_path) as per documentation.
Create the client using client = bigquery.Client.from_service_account_info(json_object) as per documentation.
Either way will allow you to pass the credentials at runtime. See the example below for the second option. The first option should be quite evident.
As you can see in the code below, the credentials get set on the client side, not on the server side.
server.py
from google.cloud import bigquery
from flask import Flask, request
import json
app = Flask(__name__)
def querysomething(json_object):
# https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.client.Client.html#google.cloud.bigquery.client.Client.from_service_account_info
client = bigquery.Client.from_service_account_info(json_object)
# example below stolen from:
# https://cloud.google.com/bigquery/docs/reference/libraries#using_the_client_library
query = """
SELECT name, SUM(number) as total_people
FROM `bigquery-public-data.usa_names.usa_1910_2013`
WHERE state = 'TX'
GROUP BY name, state
ORDER BY total_people DESC
LIMIT 20
"""
query_job = client.query(query) # Make an API request.
print("The query data:")
for row in query_job:
# Row values can be accessed by field name or index.
print("name={}, count={}".format(row[0], row["total_people"]))
#app.route("/api/query", methods=["POST"])
def api_query():
print(request.is_json)
json_object = json.loads(request.get_json())
print(json_object)
querysomething(json_object)
return "ok"
if __name__ == "__main__":
app.run(debug=True)
client.py
import requests
import json
with open("credentials.json") as infile:
credentials = json.load(infile)
target = "http://127.0.0.1:5000/api/query"
asjson = json.dumps(credentials)
response = requests.post(target, json=asjson)
print(response, response.text)
I'm having difficulty with my Cloud Function in GCP that is simply supposed to return the raw XML stored in a GCS Bucket when invoked with a basic GET request. It works fine without any type of authentication, however since I added the Flask-HTTPAuth package to the mix in order to add some measure of security before exposing the endpoint, the application deploys fine, but crashes without any sort of hint as to why as soon as it is invoked. The error in SD Logging is as follows:
severity: "DEBUG"
textPayload: "Function execution took 1847 ms, finished with status: 'crash'"
timestamp: "2020-07-15T17:22:15.158036700Z"
The function in question (anonymized):
from flask import Flask, request, jsonify, make_response, abort
from flask_httpauth import HTTPBasicAuth
from google.cloud import storage, secretmanager
import google.cloud.logging
import logging
import sys
app = Flask(__name__)
auth = HTTPBasicAuth()
PROJECT_ID = 'example_project'
GCS_BUCKET = 'example_bucket'
users = ['example_user']
# Instantiate logger
client = google.cloud.logging.Client()
client.get_default_handler()
client.setup_logging()
#auth.verify_password
def verify_password(username, password):
# Instantiate the Secret Manager client.
sm_client = secretmanager.SecretManagerServiceClient()
# Load secrets
name = sm_client.secret_version_path(PROJECT_ID, 'example_secrets_ref', 1)
secrets_pass = sm_client.access_secret_version(name)
passwords = [secrets_pass]
if username in users and password in passwords:
logging.info('auth success')
return username
logging.info('auth fail')
return abort(403)
#app.route('/')
#auth.login_required
def latest_xml():
try:
request_json = request.get_json()#silent=True)
storage_client = storage.Client(project=PROJECT_ID)
bucket = storage_client.get_bucket(GCS_BUCKET)
blob = bucket.get_blob('latest_pull.xml')
latest_xml = blob.download_as_string()
logging.info('Loaded blob from GCS')
return(latest_xml)
except exception as e:
logging.error(str(e))
logging.error("Failed to load blob from GCS")
sys.exit(1)
if __name__ == '__main__':
app.run()
I've tried setting the entrypoint as both the main function as well as the auth function to no avail. My question is: is it possible to even use basic auth in a GCP Cloud Function or am I barking up the wrong tree here?
Your function doesn't enforce the standard signature for http function
def latest_xml(request):
...
Here you use a flask web server, which is not need, and not used by Cloud Functions. However, I recommend you to have a look to Cloud Run, and to add a simple and generic Dockerfile to deploy . You can deploy your "function" as-is in a container and to have the same behavior as Cloud Functions.
EDIT
When you use flask, the request object is global for each request. You use it like this:
request_json = request.get_json()#silent=True)
With Cloud Functions, this object is caught by the Cloud Functions platform and passed in parameter to your function.
In the request object, you have the body of the request, useless in GET for example. But also, all the request context: headers, user agent, source ip,...
I am new to API, and get a tasks of creating POST API. I have created a code somehow.
I want to add data to the hello.txt through post API, So how will I do it?
Here is my code:
import flask
from flask import request, jsonify
app = flask.Flask(__name__)
app.config["DEBUG"] = True
#app.route('/api/v1/resources/messages', methods = ['POST'])
def api_message():
if request.headers['Content-Type'] == 'text/plain':
return "Text Message: " + request.data
elif request.headers['Content-Type'] == 'application/octet-stream':
return "Binary message written!"
elif request.headers['Content-Type'] == 'application/json':
f = open('F:\Asif_Ahmed\Projects\api\hello.txt',"w")
f.write(request.data)
f.close()
return "JSON Message: " + json.dumps(request.json)
else:
return "415 Unsupported Media Type ;)"
app.run()
from flask import Flask, jsonify, render_template, request #import flask library
from flask_basicauth import BasicAuth # import flask library for create basic authentication if needed
from flask_cors import CORS # import flask library Cross-Origin Resource Sharing that is a mechanism that uses additional HTTP headers to tell a browser to let a web application running at one origin (domain) have permission to access selected resources from a server at a different origin
app = Flask(__name__)
CORS(app) #set-up cors for my app
#if you want use basic authentication you need set-up username and password
app.config['BASIC_AUTH_USERNAME'] = 'admin'
app.config['BASIC_AUTH_PASSWORD'] = 'password'
basic_auth = BasicAuth(app)#set-up username and password for my app but in this case I'm not specifying yet in which API use them
#app.route('/api/v1/resources/add_messages', methods=['POST'])#create my POST api
#basic_auth.required# set-up basic authentication for this API, comment out if not needed
def update_credential ():
json_credential=request.get_json()#get the JSON sent via API
print (json_credential["message"])#get the node "message" of my JSON
###########
#code to write in your file, you need write the json_credential["message"]
###########
return ("ok")
if __name__ == '__main__':
app.run(host='0.0.0.0', port=1024, threaded=True)#start my flask app with local_host IP and specific port, if you don't specify the port it will run in the default port
In this case the JSON Input should be:
{"message":"your text"}
Please let me know if something is not clear, I even try this code on my local and the JSON is passed without problems.....
So you need run your python script and see that the API is running, if you had no JSON to send and was just a simple API that give back information you should have used even Chrome but in this case that you need send some JSON data I would advice you to use Postman.
See screenshot example:
I built a simple python application to be run on the Google App Engine. Code:
import webapp2
from oauth2client.contrib.appengine import AppAssertionCredentials
from apiclient.discovery import build
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
class MainPage(webapp2.RequestHandler):
def get(self):
self.response.headers['Content-Type'] = 'text/plain'
self.response.write('BigQuery App')
credentials = AppAssertionCredentials(
'https://www.googleapis.com/auth/sqlservice.admin')
service = discovery.build('bigquery', 'v2', credentials=credentials)
projectId = '<Project-ID>'
query_request_body = {
"query": "SELECT a from Data.test LIMIT 10"
}
request = service.jobs().query(projectId=projectId, body=query_request_body)
response = request.execute()
self.response.write(response)
app = webapp2.WSGIApplication([
('/', MainPage),
], debug=True)
I am able to deploy this code locally (http://localhost:8080) and everything works correctly, however I get the following error 500 Server Error when I try to deploy it to GAE using:
appcfg.py -A <Project-Id> -V v1 update .
This is the error I get from the Error Report Console:
error: An error occured while connecting to the server: DNS lookup failed for URL:http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/https://www.googleapis.com/auth/sqlservice.admin/?recursive=True
I believe it is an auth issue and to make sure my service account was authorized I went through the gcloud authentification for service accounts and I also set the set environment variables from the SDK.
I have been trying to get around this for a while, any pointers are very appreciated. Thank you.
Also, I have been using Service Account Auth by following these docs: https://developers.google.com/identity/protocols/OAuth2ServiceAccount where it says that I shouldn't be able to run AppAsseritionCredenitals locally, which adds to my confusion because I actually can with no errors.
EDIT:
After reuploading and reauthorizing my service account I was able to connect to the server. However, the authorization error continues with this:
HttpError: <HttpError 403 when requesting https://www.googleapis.com/bigquery/v2/projects/sqlserver-1384/queries?alt=json returned "Insufficient Permission">
To fix the "error while connecting to the server", follow the instructions listed in this answer: https://stackoverflow.com/questions/31651973/default-credentials-in-google-app-engine-invalid-credentials-error#=
and then re-upload the app
Then, to fix the HttpError 403 when requesting ... returned "Insufficient Permission", you have to change the scope you were requesting. In my case I was requesting:
credentials = AppAssertionCredentials(
'https://www.googleapis.com/auth/sqlservice.admin')
however, the correct scope for Google BigQuery is: https://www.googleapis.com/auth/bigquery. Which looks like this:
credentials = AppAssertionCredentials(
'https://www.googleapis.com/auth/bigquery')
If you are using a different API, use whichever scope is outlined in the documentations.
I am a first time user of pythonanywhere
I first started by doing a git clone of my code from github through the bash console. I did not use a virtual environment. My WSGI app was invoked in my app.py file. Also, my code uses sqlalchemy to interact with my database.
Basically, the flask app was like a custom api that returned JSON for GET and POST requests and I am having trouble viewing the JSON output. I am not sure what exactly I am doing wrong or missing.
Code in app.py file:
#!flask/bin/python
from flask import Flask
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy_declarative import Base, Quote
from flask import request
from flask import abort
import json
#connect to database
engine = create_engine("sqlite:///quotes.db")
Base.metadata.bind = engine
DBSession = sessionmaker(bind=engine)
session = DBSession()
app = Flask(__name__)
#app.route("/trumptext/api/quotes", methods=["GET"])
def get_quotes():
quoteList = session.query(Quote).all()
result = []
for q in quoteList:
my_dict = {}
my_dict["id"] = q.id
my_dict["quote"] = q.quote
result.append(my_dict)
return json.dumps(result,ensure_ascii=False).encode('utf8')
#app.route("/trumptext/api/quotes", methods=["POST"])
def add_quote():
if not request.json or not "quote" in request.json:
abort(400)
new_quote = request.json["quote"]
q = Quote(quote=new_quote)
session.add(q)
session.commit()
quoteList = session.query(Quote).all()
last = quoteList[-1]
result = []
my_dict = {}
my_dict["id"] = last.id
my_dict["quote"] = last.quote
result.append(my_dict)
return json.dumps(result,ensure_ascii=False).encode('utf8'), 201
if __name__ == "__main__":
app.run()
Also, code in /var/www/nnelson_pythonanywhere_com_wsgi.py:
import os
import sys
path = '/home/nnelson/trumptextapi'
if path not in sys.path:
sys.path.append(path)
from app import app as application
If I enter something like :
http://nnelson.pythonanywhere.com/trumptext/api/quotes (to perform a GET request)
It should ideally return all the quotes stored in the quotes.db database in JSON format, however all I get it output that looks like this: [] I tested my code on localhost using the curl tool and it works just fine. I am having trouble hosting it though
Any help is appreciated.
You're using a relative path to your database, so it's probably looking at a database that you don't expect. Use a full path to the database or make it relative to the path of your app.py file so that you know where it's getting the database from.