When my app crashes, the client receives a response with:
response.status_code = 500
response.text =
500
Internal Server Error Internal Server Error The
server encountered an internal error and was unable to complete your
request. Either the server is overloaded or there is an error in the
application.
I would like to also show the error trace, to inform the client about what has gone wrong. How can I do that?
Related
Similar situation to what was asked here, however the solution described there do not work in my case. I want to run a grpc server on google cloud run, and be able to call it. I have a basic python grpc server that works fine and is able to be called by my client when running the gcr.io image on localhost with docker run. However, after successfully deploying to google cloud run, and with the server listening on port 8080 (I logged it), I am no longer able to reach the server with my client code.
I receive the following:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "DNS resolution failed"
debug_error_string = "{"created":"#1591245267.073965000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3981,"referenced_errors":[{"created":"#1591245267.073946000","description":"Resolver transient failure","file":"src/core/ext/filters/client_channel/resolving_lb_policy.cc","file_line":214,"referenced_errors":[{"created":"#1591245267.073945000","description":"DNS resolution failed","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":357,"grpc_status":14,"referenced_errors":[{"created":"#1591245267.073882000","description":"C-ares status is not ARES_SUCCESS: Domain name not found","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244,"referenced_errors":[{"created":"#1591245267.059636000","description":"C-ares status is not ARES_SUCCESS: Domain name not found","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244}]}]}]}]}"
>
my server code is as follows:
def serve():
# expected port from google cloud run is 8080
port = os.getenv('PORT')
if not port:
port = server_port
server.add_insecure_port('[::]:{port}'.format(port=port))
server.start()
logger.info('listening on port {port}'.format(port=port))
server.wait_for_termination()
my client code, I have tried both
channel = grpc.insecure_channel('<the .run.app url on my cloud run service>:8080')
stub = my_service_pb2_grpc.MyServiceStub(channel)
request = my_service_pb2.MyEndpointRequest(
test='hello'
)
response = stub.MyEndpoint(request)
print(response)
and
with grpc.secure_channel('<the .run.app url on my cloud run service>:8080', grpc.ssl_channel_credentials()) as channel:
stub = my_service_pb2_grpc.MyServiceStub(channel)
request = my_service_pb2.MyEndpointRequest(
test='hello'
)
response = stub.MyEndpoint(request)
print(response)
both of which do not work and gives me the same error that I wrote above.
Furthermore, when I ping <the .run.app url on my cloud run service> on the command line I get Name or service not known.
Any help would be greatly appreciated!
Managed to solve it, the solution was to remove the https:// prefix from the url shown on the cloud run service and use the 443 port instead of 8080
with grpc.secure_channel(
'<url without https:// prefix>:443',
grpc.ssl_channel_credentials(),
) as channel:
stub = my_service_pb2_grpc.MyServiceStub(channel)
request = my_service_pb2.MyEndpointRequest(
test='hello'
)
response = stub.MyEndpoint(request)
print(response)
We have host python service on the server (no public accessible) which allows HTTPS and Web Socket requests on 4431.To connect to the python service, we have implemented an Azure App Gateway (bind with the domain) with rule - domain:4431 and which will be redirected to the server.
So, the problem is we are facing the following error
502 Bad Gateway - Microsoft-Azure-Application-Gateway/v2
We have observed that request from the browser comes to the AppGW but on server, we are receiving request intermittently.
socketio.run(app, host='0.0.0.0', port=4431, certfile=certs["crt"], keyfile=certs["key"], ssl_version=ssl.PROTOCOL_TLSv1_2)
Our Architecture
Error
Found out, what was causing the issue, we haven't provided root CA Certificate to run server.
socketio.run(app, host='0.0.0.0', port=4431, certfile=certs["crt"], keyfile=certs["key"], ca_certs=certs["CA crt"], ssl_version=ssl.PROTOCOL_TLSv1_2)
My web-application makes HTTP requests to an Extensible Service Proxy (ESP), which in-turn, delegates to a gRPC server (written in Python). Ignoring Android and iOS clients, the architecture is:
The ESP is a nginx reverse proxy.
The gRPC server ("Your code" in the reference architecture) may raise an exception, in which case I use context.abort to raise an exception and terminate the RPC with a non-OK status:
try:
# Do something that could fail.
except ValueError as e:
context.abort(grpc.StatusCode.DATA_LOSS, str(e))
While it is possible to use set_code and set_details, they still result in a HTTP status of 200 OK.
There are two problems:
The gRPC status codes are translated by the ESP container (nginx proxy) to a a generic 500 Internal Server Error.
The accompanying details are stripped-out.
and 2. combined means the web client has, at most, a 500 Internal Server Error for all exceptions raised by the gRPC server.
Ultimately, I don't understand how more informative (ideally, custom) errors can be returned to web-clients.
grpc Status code::DATA_LOSS, are translated to HTTP code 500. The code is here
The grpc status detail (status code and error message) are send back in the response body in JSON format. The code is here
How can I determine when a flask server is ready to receive traffic?
For example, I have server A, inside a dockerfile, that sends a request to an already running server B, announcing its existence upon the container being run.
However once the already running server (B) recieves that request, it tries to send data to one of server A's routes, receiving the error: Failed to establish a new connection: [Errno 111]
Is there a flask or external resource that can send a request in the server when it is ready to receive resulting traffic?
I just pull the server state by sending a simple GET request:
#app.route('/srv_status', methods=['GET'])
def get_srv_status():
return 'OK', 200 # just means we're on air
If it replied, that means the server is ready to process the requests.
Additionally, it can serve for monitoring purposes.
PS: it's been a while after your question. It would be interesting to learn how you've solved the problem.
Flask server usually crashes after the Exception:
Exception happened during processing of request from ('x.x.x.x', 12554)
This exception usually comes when some client make certain request and closes its connection before receiving the response. Meanwhile, server tries writing to the pipe for the response, which fails as connection is already closed by the client. And, the web server crashes.
Is there any way we can catch this exception, and stops the web server from crashing?