Develop Reference

Tracing a Python gRPC server deployed on Cloud Run with OpenTelemetry

I'm running a Python gRPC server on Cloud Run and attempting to add instrumentation to capture trace information. I have a basic setup currently, however I'm having trouble making use of propagation as shown in the OpenTelemetry docs.
Inbound requests have the x-cloud-trace-context header, and I can log the header value in the gRPC method I've been working with, however the traces created by the OpenTelemetry library always have a different ID than the trace ID from the request header.
This is the simple tracing.py module I've created to provide configuration and access to the current Tracer instance:
"""Utility functions for tracing."""
import opentelemetry.exporter.cloud_trace as cloud_trace
import opentelemetry.propagate as propagate
import opentelemetry.propagators.cloud_trace_propagator as cloud_trace_propagator
import opentelemetry.trace as trace
from opentelemetry.sdk import trace as sdk_trace
from opentelemetry.sdk.trace import export
import app_instance
def get_tracer() -> trace.Tracer:
"""Function that provides an object for tracing.
Returns:
trace.Tracer instance.
"""
return trace.get_tracer(__name__)
def configure_tracing() -> None:
trace.set_tracer_provider(sdk_trace.TracerProvider())
if app_instance.IS_LOCAL:
print("Configuring local tracing.")
span_exporter: export.SpanExporter = export.ConsoleSpanExporter()
else:
print(f"Configuring cloud tracing in environment {app_instance.ENVIRONMENT}.")
span_exporter = cloud_trace.CloudTraceSpanExporter()
propagate.set_global_textmap(cloud_trace_propagator.CloudTraceFormatPropagator())
trace.get_tracer_provider().add_span_processor(export.SimpleSpanProcessor(span_exporter))
This configure_tracing function is called by the entrypoint script run on container start, so it executes before any requests are handled. When running in Google Cloud, the CloudTraceFormatPropagator should be what's required to ensure trace propagation, however it doesn't seem to be working for me.
This is the simple gRPC method I've been implementing with:
import grpc
from opentelemetry import trace
import stripe
from common import cloud_logging, datastore_utils, proto_helpers, tracing
from services.payment_service import payment_service_pb2
from third_party import stripe_client
def GetStripeInvoice(
self, request: payment_service_pb2.GetStripeInvoiceRequest, context: grpc.ServicerContext
) -> payment_service_pb2.StripeInvoiceResponse:
tracer: trace.Tracer = tracing.get_tracer()
with tracer.start_as_current_span('GetStripeInvoice'):
print(f"trace ID from header: {dict(context.invocation_metadata()).get('x-cloud-trace-context')}")
cloud_logging.info(f"Getting Stripe invoice.")
order = datastore_utils.get_pb_with_pb_key(request.order)
try:
invoice: stripe.Invoice = stripe_client.get_invoice(
invoice_id=order.stripe_invoice_id
)
cloud_logging.info(f"Retrieved Stripe invoice. Amount due: {invoice['amount_due']}")
except stripe.error.StripeError as e:
cloud_logging.error(
f"Failed to retrieve invoice: {e}"
)
context.abort(code=grpc.StatusCode.INTERNAL, details=str(e))
return payment_service_pb2.StripeInvoiceResponse(
invoice=proto_helpers.create_struct(invoice)
)
I've even gone as far as adding the x-cloud-trace-context header to local client requests, to no avail - the included value isn't used when starting traces.
I'm not sure what I'm missing here - I can see traces in the Cloud Trace dashboard so I believe the basic instrumentation is correct, however there's obviously something going on with the configuration/usage of the CloudTraceFormatPropagator.

It turns out that my configuration wasn't correct - or, I should say, it wasn't complete. I'd followed this basic example from the docs for the Google Cloud OpenTelemetry library, but I didn't realize that manually instrumenting wasn't needed.
I removed the call to tracer.start_as_current_span in my gRPC method, installed the gRPC instrumentation package (opentelemetry-instrumentation-grpc), and added it to the tracing configuration step during startup of my gRPC server, which now looks something like this:
from opentelemetry.instrumentation import grpc as grpc_instrumentation
from common import tracing # from my original question
def main():
"""Starts up GRPC server."""
# Set up tracing
tracing.configure_tracing()
grpc_instrumentation.GrpcInstrumentorServer().instrument()
# Set up the gRPC server
server = grpc.server(futures.ThreadPoolExecutor(max_workers=100))
# set up services & start
This approach has solved the issue described in my question - my log messages are now threaded in the expected manner
As someone new to telemetry & instrumentation, I didn't realize that I'd need to take an extra step since I'm tracing gRPC requests, but it makes sense now.
I ended up finding some helpful examples in a different set of docs - I'm not sure why these are separate from the docs linked earlier in this answer.
EDIT: Ah, I believe the gRPC instrumentation, and thus the related docs, are part of a separate but related project wherein contributors can add packages that instrument libraries of interest (i.e. gRPC, redis, etc). It'd be helpful if it was unified, which is the topic of this issue in the main OpenTelemetry Python repo.

While reviewing Google Documentation of OpenTelemetry using Python, I found some configurations that could help with the issue of tracing the correct ID. Additionally, there is a troubleshooting document to view traces in your Google Cloud Project when you expect trace data to be present.
Python-OpenTelemetry - https://cloud.google.com/trace/docs/setup/python-ot
Google Cloud Trace Troubleshooting - https://cloud.google.com/trace/docs/troubleshooting
For secure channels, you need to pass in chanel_type=’secure’. It is explained in the following link: https://github.com/open-telemetry/opentelemetry-python-contrib/issues/365
You need to use the x-cloud-trace-context header to ensure your traces use the same trace ID as the load balancer and AppServer on Google Cloud Run, and all link up in Google Trace.
The code below works to see you logs alongside traces in Google Trace’s Trace List view:
from opentelemetry import trace
from opentelemetry.trace.span import get_hexadecimal_trace_id, get_hexadecimal_span_id
current_span = trace.get_current_span()
if current_span:
trace_id = current_span.get_span_context().trace_id
span_id = current_span.get_span_context().span_id
if trace_id and span_id:
logging_fields['logging.googleapis.com/trace'] = f"projects/{self.gce_project}/traces/{get_hexadecimal_trace_id(trace_id)}"
logging_fields['logging.googleapis.com/spanId'] = f"{get_hexadecimal_span_id(span_id)}"
logging_fields['logging.googleapis.com/trace_sampled'] = True
The documentation and code above were tested using Flask Framework.

How to call a Python Bottle API from within that application

I have created an API using the Bottle Library in Python. It endpoints look like this
#app.get('/api/users/<id>', name='user')
def get_user(model, id):
user = model.get_user(id)
if not user:
return HTTPError(404, 'User not found')
else:
response.content_type = "application/json"
return json.dumps(user)
I want to call the API in other functions within the same app
#app.route('/users/<id>')
def users (id=1):
user = request.get("http://localhost:8001/api/user/1")
return template('user', user=user)
However, this is showing no results. The request get timed out each time
So my question is, how to call a bottle API from within that app using Requests library or through any other means.

Are you running Bottle in single-threaded mode (the default)? If so, then your internal get request will hang forever. This is because your server can serve only one request at a time, and you are asking it to handle two at once: the first call to /users/<id>, and then the second call to /api/users/<id>.
A band-aid fix would be to run the server in asynchronous mode. Try this method and see if your timeouts go away:
run(host='0.0.0.0', port=YOUR_PORT_NUMBER, server='gevent')
However: You shouldn't be designing your application this way in the first place. Instead, refactor your code so that both methods can call a function that returns the JSON representation of a user. Then one api call can return that raw json object, while the other api call can present it as HTML. NOTE: that's not how I would design this API, but it's the answer that's the shortest distance from how you've structured your application so far.

how can I make an outbound SIP call using Twilio?

I am using a Galaxy 8S android phone with the Samsung SIP settings. I have successfully registered a (Twilio) SIP account on the phone. I want to make an outbound call to an international PSTN Number NOT to another sip address.
My SIP doman, on Twilio, points to my heroku app.
The code is:
#application.route("/makesip", methods=['GET', 'POST'])
def makesip():
to_number=request.values.get('To', None),
client = Client(ACCOUNT_SID, AUTH_TOKEN)
call = client.calls.create(to=to_number, from_="+1415792xxxx", url="https://myapp.herokuapp.com/sipout", method="POST")
return call.sid
#return "OK"
#application.route("/sipout", methods=['GET', 'POST'])
def sipout():
response = VoiceResponse()
to_number = request.values.get('To', None)
dial = Dial(caller_id='+1415792xxxx')
dial.number(to_number)
response.append(dial)
return str(response)
When I make the call from my cell phone it hangs up almost immediately and says "Server Error try again later". The Twilio error log says:
We don't support dialing out to global twilio domains (domainname.sip.twilio.com). Only regional (domainname.sip.us1.twilio.com) ones are supported.
I think that I am making a very fundamental error here but I simply cannot identify it. Can anybody help please? Should I, for example, set the "from_" parameter as "sip:my_sip_address.domainname.sip.us1.twilio.com"?

I'm not a heroku expert, but your code looks similar enough to the php I have running which works fine for this.
In your phone settings is your SIP server set as user#domainname.sip.twilio.com or as user#domainname.sip.us1.twilio.com? It should be the latter. I seem to remember getting caught out by something like this when I was trying to get things working
EDIT
Just had another play with mine and I figured it out. You have to dial the number from your phone as phonenumber#yourdomain.sip.twilio.com, then twilio will return to as sip:phonenumber#yourdomain.sip.twilio.com
You need to change this line of your code to strip out just the number
to_number=request.values.get('To', None),
My php line is substr(strtok($to, '#'), 4); so whatever your equivalent of that is.

What I think is probably happening is that the "To" that is hitting your Heroku app is in the format "sip:+12312123123#yoursipdomain.sip.twilio.com;user=phone" and you're trying to inject that directly into the "dial" verb.
What you actually want is to strip it down to the bare number in E.194 format (with the leading +).
I'd suggest starting by testing using a quick TwiML Bin as per the Twilio SIP Registration docs, rather than your Heroku app.
TwiML Bins are basically static TwiML but with a tiny bit of intelligence in special tags. It's like the Twilio equivalent of Mail Merge, if you've ever used that in Microsoft Word.
(Twilio recently updated the SIP Registration docs. They're much better now.)
Use a TwiML Bin for initial testing otherwise you risk spending time fixing an otherwise working Heroku app because the problem is your phone/account.
Go to "Twilio Docs > API Reference > Twilio Voice API > SIP Registration".
Scroll down to "Using Enhanced TwiML Bin Templates to Call a mobile/landline on the Public Telephone Network" and follow that.
See if that works.
If it doesn't, my suspicion is your Samsung is actually spitting out something daft due to something dial plan related. (Dial plan is the conversion of +12345645642 into a SIP URI. You might find it's doing something like +012345645642 instead.)
If it does work, great. If you want to get your Heroku app working, compare the working response body to the one your Heroku app is spitting out. Post both, and we'll figure out what's going wrong.
Just to check, you are specifying a region in your Domain and Registration Server settings on the Samsung, yeah? The "yoursipdomain.sip.us1.twilio.com" that miknik talked about?

How to submit a POST request to the Calls resource? (making outgoing calls in Twilio)

I'm having trouble with making outgoing calls using Twilio and I believe I skipped this step where I should submit a POST request to the Calls resource. I don't really understand what to do with this and I need someone to explain it for me since I'm just a beginner. And by the way, my Twilio account is just free trial. Does this have something to do as to why I can't make local calls?
Thanks.

Twilio developer evangelist here.
Best place for you to start would be the Python making calls quick start. Follow that tutorial through and you should understand how to make a call with Python.
Though, as you say, the key is the POST request to the calls resource. It's easiest to perform this using our Python helper library and looks a bit like this:
from twilio.rest import TwilioRestClient
# Get these credentials from http://twilio.com/user/account
account_sid = "ACXXXXXXXXXXXXXXXXX"
auth_token = "YYYYYYYYYYYYYYYYYY"
client = TwilioRestClient(account_sid, auth_token)
# Make the call
call = client.calls.create(to="+14085551234", # Any phone number
from_="+12125551234", # Must be a valid Twilio number
url="http://twimlets.com/holdmusic?Bucket=com.twilio.music.ambient") # Just plays hold music
Let me know if that helps at all.

Can I persist an http connection (or other data) across Flask requests?

I'm working on a Flask app which retrieves the user's XML from the myanimelist.net API (sample), processes it, and returns some data. The data returned can be different depending on the Flask page being viewed by the user, but the initial process (retrieve the XML, create a User object, etc.) done before each request is always the same.
Currently, retrieving the XML from myanimelist.net is the bottleneck for my app's performance and adds on a good 500-1000ms to each request. Since all of the app's requests are to the myanimelist server, I'd like to know if there's a way to persist the http connection so that once the first request is made, subsequent requests will not take as long to load. I don't want to cache the entire XML because the data is subject to frequent change.
Here's the general overview of my app:
from flask import Flask
from functools import wraps
import requests
app = Flask(__name__)
def get_xml(f):
#wraps(f)
def wrap():
# Get the XML before each app function
r = requests.get('page_from_MAL') # Current bottleneck
user = User(data_from_r) # User object
response = f(user)
return response
return wrap
#app.route('/one')
#get_xml
def page_one(user_object):
return 'some data from user_object'
#app.route('/two')
#get_xml
def page_two(user_object):
return 'some other data from user_object'
if __name__ == '__main__':
app.run()
So is there a way to persist the connection like I mentioned? Please let me know if I'm approaching this from the right direction.

I think you aren't approaching this from the right direction because you place your app too much as a proxy of myanimelist.net.
What happens when you have 2000 users? Your app end up doing tons of requests to myanimelist.net, and a mean user could definitely DoS your app (or use it to DoS myanimelist.net).
This is a much cleaner way IMHO :
Server side :
Create a websocket server (ex: https://github.com/aaugustin/websockets/blob/master/example/server.py)
When a user connects to the websocket server, add the client to a list, remove it from the list on disconnect.
For every connected users, do frequently check myanimelist.net to get the associated xml (maybe lower the frequence the more online users you get)
for every xml document, make a diff with your server local version, and send that diff to the client using the websocket channel (assuming there is a diff).
Client side :
on receiving diff : update the local xml with the differences.
disconnect from websocket after n seconds of inactivity + when disconnected add a button on the interface to reconnect
I doubt you can do anything much better assuming myanimelist.net doesn't provide a "push" API.