Develop Reference

Getting trouble in connecting with Google BigQuery

I've been making api with flask on Google App Engine and When I send request to this app from browser after deploy, I got 502 error. I'm sure this error is caused by credential of GCP by "gcloud app logs tail -s test" but The path of credential Json file and file name seems OK . I have googled and I tried every articles I have found there but could not solve.
I have already done export GOOGLE_APPLICATION_CREDENTIALS="/home/user/secret_key/bq.json"
Could anyone tell me the solution??
If there is lack of any info , please let me know . Thank you .
besides, my api function is getting luid parameter over http request and run SQL with that luid and if the row of the luid has data in cv_date column in BigQuery, it returns True to client.
【The result of "gcloud app logs tail -s test"】
File "/env/lib/python3.7/site-packages/google/auth/_default.py", line 97, in load_credentials_from_file "File {} was not found.".format(filename) google.auth.exceptions.DefaultCredentialsError: File /home/user/secret_key/bq.json was not found.
【/home/user/api_dev/main.py】
from flask import Flask,request
from google.cloud import bigquery
import os
credentials_json = '/home/user/secret_key/bq.json'
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = credentials_json
client = bigquery.Client()
app = Flask(__name__)
#app.route('/')
def get_request():
request_luid = request.args.get('luid') or ''
query = """
SELECT EXISTS(SELECT cv_date FROM `test-266110.conversion_log.conversion_log_202008*` t WHERE request_luid = p.luid)
"""
query_res = client.query(query)
return query_res
if __name__ == "__main__":
app.run()
【Remove the codes for BigQuery except import library and variables】
*This code works well and returns luid you input on url parameter
from flask import Flask, request
from google.cloud import bigquery
import os
credentials_json = '/home/user/secret_key/bq.json'
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = credentials_json
app = Flask(__name__)
#app.route('/')
def get_request():
request_luid = request.args.get('luid') or ''
return request_luid
if __name__ == "__main__":
app.run()

I'd recommend reading through the auth docs.
https://cloud.google.com/docs/authentication/production talks about service account interactions in a bit more detail. You likely don't need to pass in your credentials in the live app. You can simply set the GOOGLE_APPLICATION_CREDENTIALS when you're running locally to use the credentials, but you don't need to set it in production.
The issue is that the path you've specified (/home/user/secret_key/bq.json) is only valid for your development environment, and either not included in your production deployment at all or the absolute path to the file in the deployed app is different.

why is this flask not showing standard message according to my code?

my rest service sits at http://127.0.0.1:5000, but when i launch it, it gives me 404:
Not Found
The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
why is that? I want my server to show some status msg like 'service ready'.
The actual function that i will use is accessible and works, when i press 'http://127.0.0.1:5000/parser/tengrinews' and hit enter it outputs the msg i coded in the function in my flask app:
[
"parsing this website :",
"tengrinews"
]
the main code:
from flask import Flask
import requests
from datetime import datetime
from flask import jsonify
app = Flask(__name__)
#this is my std method i can't see
#app.route("/http://127.0.0.1:5000/", methods = ['GET'])
def main():
return jsonify('service is ready')
#app.route("/parser/<string:website>", methods = ['GET'])
def parse(website):
return jsonify("parsing this website :", website )
if __name__ == "__main__":
app.run(debug=True)

Change this line -
#app.route("/http://127.0.0.1:5000/", methods = ['GET'])
to
#app.route("/", methods = ['GET']).
Because you have to specify only the extended URL that will be used. The #app.route decorator handles the rest for us
Note* (Don't do this. For fun only) -
If you wish to continue to use #app.route("/http://127.0.0.1:5000/", methods = ['GET']) then access the endpoint with the url - http://localhost:5000/http://127.0.0.1:5000/. You will get the response as "service is ready"

Unable to perform Cloud Function trigger a HTTP triggered Cloud Function that doesn't allow unauthenticated invocations?

I have a situation where I am trying to create two Cloud Functions namely CF1 & CF2 and I have one Cloud Scheduler. Both cloud functions are having authenticated invocation enabled. My flow is Cloud Scheduler will trigger CF1. On completion of CF1, the CF1 will trigger CF2 as a http call. I have referred Cannot invoke Google Cloud Function from GCP Scheduler to access authenticated CF1 from Cloud Scheduler and able to access CF1. But I am getting problem when accessing CF2 from CF1. The CF1 does not trigger CF2 and also not giving any error message. Do we need to follow any other technique when accessing authenticated Cloud Function from another authenticated Cloud Function.
CF1 code:
import json
import logging
from requests_futures.sessions import FuturesSession
def main(request):
# To read parameter values from request (url arguments or Json body).
raw_request_data = request.data
string_request_data = raw_request_data.decode("utf-8")
request_json: dict = json.loads(string_request_data)
request_args = request.args
if request_json and 'cf2_endpoint' in request_json:
cf2_endpoint = request_json['cf2_endpoint']
elif request_args and 'cf2_endpoint' in request_args:
cf2_endpoint = request_args['cf2_endpoint']
else:
cf2_endpoint = 'Invalid endpoint for CF2'
logger = logging.getLogger('test')
try:
session = FuturesSession()
session.get("{}".format(cf2_endpoint))
logger.info("First cloud function executed successfully.")
except RuntimeError:
logger.error("Exception occurred {}".format(RuntimeError))
CF2 code:
import logging
def main(request):
logger = logging.getLogger('test')
logger.info("second cloud function executed successfully.")
Current output logs:
First cloud function executed successfully.
Expected output logs:
First cloud function executed successfully.
second cloud function executed successfully.
Note: Same flow is working if I use unauthenticated access to the both cloud functions.

Two things are happening here:
You're not using request-futures entirely correctly. Since the request is made asynchronously, you need to block on the result before the function implicitly returns, otherwise it might return before your HTTP request completes (although it probably is in this example):
session = FuturesSession()
future = session.get("{}".format(cf2_endpoint))
resp = future.result() # Block on the request completing
The request you're making to the second function is not actually an authenticated request. Outbound requests from a Cloud Function are not authenticated by default. If you looked at what the actual response is above, you would see:
>>> resp.status_code
403
>>> resp.content
b'\n<html><head>\n<meta http-equiv="content-type" content="text/html;charset=utf-8">\n<title>403 Forbidden</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Forbidden</h1>\n<h2>Your client does not have permission to get URL <code>/function_two</code> from this server.</h2>\n<h2></h2>\n</body></html>\n'
You could jump through a lot of hoops to properly authenticate this request, as detailed in the docs: https://cloud.google.com/functions/docs/securing/authenticating#function-to-function
However, a better alternative would be to make your second function a "background" function and invoke it via a PubSub message published from the first function instead:
from google.cloud import pubsub
publisher = pubsub.PublisherClient()
topic_name = 'projects/{project_id}/topics/{topic}'.format(
project_id=<your project id>,
topic='MY_TOPIC_NAME', # Set this to something appropriate.
)
def function_one(request):
message = b'My first message!'
publisher.publish(topic_name, message)
def function_two(event, context):
message = event['data'].decode('utf-8')
print(message)
As long as your functions have the permissions to publish PubSub messages, this avoids the need to add authorization to the HTTP requests, and also ensures at-least-once delivery.

Google Cloud Function provide REST API interface what include call method that can be used in another Cloud Function HTTP invokation.
Although the documentation mention using Google-provided client libraries there is still non one for Cloud Function on Python.
And instead you need to use general Google API Client Libraries. [This is the python one].3
Probably, the main difficulties while using this approach is an understanding of authentification process.
Generally you need provide two things to build a client service:
credentials ans scopes.
The simpliest way to get credentials is relay on Application Default Credentials (ADC) library. The rigth documentation about that are:
https://cloud.google.com/docs/authentication/production
https://github.com/googleapis/google-api-python-client/blob/master/docs/auth.md
The place where to get scopes is the each REST API function documentation page.
Like, OAuth scope: https://www.googleapis.com/auth/cloud-platform
The complete code example of calling 'hello-world' clound fucntion is below.
Before run:
Create default Cloud Function on GCP in your project.
Keep and notice the default service account to use
Keep the default body.
Notice the project_id, function name, location where you deploy function.
If you will call function outside Cloud Function environment (locally for instance) setup the environment variable GOOGLE_APPLICATION_CREDENTIALS according the doc mentioned above
If you will call actualy from another Cloud Function you don't need to configure credentials at all.
from googleapiclient.discovery import build
from googleapiclient.discovery_cache.base import Cache
import google.auth
import pprint as pp
def get_cloud_function_api_service():
class MemoryCache(Cache):
_CACHE = {}
def get(self, url):
return MemoryCache._CACHE.get(url)
def set(self, url, content):
MemoryCache._CACHE[url] = content
scopes = ['https://www.googleapis.com/auth/cloud-platform']
# If the environment variable GOOGLE_APPLICATION_CREDENTIALS is set,
# ADC uses the service account file that the variable points to.
#
# If the environment variable GOOGLE_APPLICATION_CREDENTIALS isn't set,
# ADC uses the default service account that Compute Engine, Google Kubernetes Engine, App Engine, Cloud Run,
# and Cloud Functions provide
#
# see more on https://cloud.google.com/docs/authentication/production
credentials, project_id = google.auth.default(scopes)
service = build('cloudfunctions', 'v1', credentials=credentials, cache=MemoryCache())
return service
google_api_service = get_cloud_function_api_service()
name = 'projects/{project_id}/locations/us-central1/functions/function-1'
body = {
'data': '{ "message": "It is awesome, you are develop on Stack Overflow language!"}' # json passed as a string
}
result_call = google_api_service.projects().locations().functions().call(name=name, body=body).execute()
pp.pprint(result_call)
# expected out out is:
# {'executionId': '3h4c8cb1kwe2', 'result': 'It is awesome, you are develop on Stack Overflow language!'}

Post API-- what are the steps that need to be followed once the code is created so that I can add data into a txt file through this API

I am new to API, and get a tasks of creating POST API. I have created a code somehow.
I want to add data to the hello.txt through post API, So how will I do it?
Here is my code:
import flask
from flask import request, jsonify
app = flask.Flask(__name__)
app.config["DEBUG"] = True
#app.route('/api/v1/resources/messages', methods = ['POST'])
def api_message():
if request.headers['Content-Type'] == 'text/plain':
return "Text Message: " + request.data
elif request.headers['Content-Type'] == 'application/octet-stream':
return "Binary message written!"
elif request.headers['Content-Type'] == 'application/json':
f = open('F:\Asif_Ahmed\Projects\api\hello.txt',"w")
f.write(request.data)
f.close()
return "JSON Message: " + json.dumps(request.json)
else:
return "415 Unsupported Media Type ;)"
app.run()

from flask import Flask, jsonify, render_template, request #import flask library
from flask_basicauth import BasicAuth # import flask library for create basic authentication if needed
from flask_cors import CORS # import flask library Cross-Origin Resource Sharing that is a mechanism that uses additional HTTP headers to tell a browser to let a web application running at one origin (domain) have permission to access selected resources from a server at a different origin
app = Flask(__name__)
CORS(app) #set-up cors for my app
#if you want use basic authentication you need set-up username and password
app.config['BASIC_AUTH_USERNAME'] = 'admin'
app.config['BASIC_AUTH_PASSWORD'] = 'password'
basic_auth = BasicAuth(app)#set-up username and password for my app but in this case I'm not specifying yet in which API use them
#app.route('/api/v1/resources/add_messages', methods=['POST'])#create my POST api
#basic_auth.required# set-up basic authentication for this API, comment out if not needed
def update_credential ():
json_credential=request.get_json()#get the JSON sent via API
print (json_credential["message"])#get the node "message" of my JSON
###########
#code to write in your file, you need write the json_credential["message"]
###########
return ("ok")
if __name__ == '__main__':
app.run(host='0.0.0.0', port=1024, threaded=True)#start my flask app with local_host IP and specific port, if you don't specify the port it will run in the default port
In this case the JSON Input should be:
{"message":"your text"}
Please let me know if something is not clear, I even try this code on my local and the JSON is passed without problems.....
So you need run your python script and see that the API is running, if you had no JSON to send and was just a simple API that give back information you should have used even Chrome but in this case that you need send some JSON data I would advice you to use Postman.
See screenshot example:

Accessing session object during Unit test of Flask application

I know it is possible to create session object using session_transaction() method. However, is there a way to access the current session object which gets created when for example "/" route gets hit? I did from flask import session to access the session but it's empty. Let me know if it is possible. Thanks.

This is what you're looking for. As it says however, you'd have to use the instantiation you create in your with statement.
with app.test_client() as c:
with c.session_transaction() as sess:
sess['a_key'] = 'a value'
# once this is reached the session was stored
result = app.test_client.get('/a_url')
# NOT part of the 2nd context
Note that this won't work if you run your test within the scope of the with c.session_transaction() as sess statement, it needs to be run after that block.

If you want to read the session data written in your view from the test, one way is to mock the session view as a dict and verify the session in your test. Here's an example using Python's unittest.mock:
app.py
from flask import Flask, session, request
app = Flask(__name__)
app.config["SECRET_KEY"] = "my secret key"
#app.route("/", methods=["POST"])
def index():
session["username"] = request.form["username"]
return "Username saved in session"
test_index.py
from unittest.mock import patch
from app import app
def test_index():
with patch("app.session", dict()) as session:
client = app.test_client()
response = client.post("/", data={
"username": "test"
})
assert session.get("username") == "test"
assert response.data == b"Username saved in session"
You can use any mocking solution you prefer, of course.