My web-application makes HTTP requests to an Extensible Service Proxy (ESP), which in-turn, delegates to a gRPC server (written in Python). Ignoring Android and iOS clients, the architecture is:
The ESP is a nginx reverse proxy.
The gRPC server ("Your code" in the reference architecture) may raise an exception, in which case I use context.abort to raise an exception and terminate the RPC with a non-OK status:
try:
# Do something that could fail.
except ValueError as e:
context.abort(grpc.StatusCode.DATA_LOSS, str(e))
While it is possible to use set_code and set_details, they still result in a HTTP status of 200 OK.
There are two problems:
The gRPC status codes are translated by the ESP container (nginx proxy) to a a generic 500 Internal Server Error.
The accompanying details are stripped-out.
and 2. combined means the web client has, at most, a 500 Internal Server Error for all exceptions raised by the gRPC server.
Ultimately, I don't understand how more informative (ideally, custom) errors can be returned to web-clients.
grpc Status code::DATA_LOSS, are translated to HTTP code 500. The code is here
The grpc status detail (status code and error message) are send back in the response body in JSON format. The code is here
Related
So i want to send (through a proxy) a request to a website.. The script looks like this and its made with the socket library in python:
import socket
TargetDomainName="www.stackoverflow.com"
TargetIP="151.101.65.69"
TargetPort=80
ProxiesIP=["107.151.182.247"]
ProxiesPort=[80]
Connect=f"CONNECT {TargetDomainName} HTTP/1.1"
Connection=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
Connection.connect((ProxiesIP[0],ProxiesPort[0]))
Connection.sendto(str.encode(Connect),(TargetIP, TargetPort))
Connection.sendto(("GET /" + TargetIP + " HTTP/1.1\r\n").encode('ascii'), (TargetIP, TargetPort))
Connection.sendto(("Host: " + ProxiesIP[0] + "\r\n\r\n").encode('ascii'), (TargetIP, TargetPort))
print (Connection.recv(1028))
Connection.close()
My question is why i get the 400 bad request error?
You did not indicate whether the 400 reply is coming from the proxy or the target server. But both of your commands are malformed.
Your CONNECT command is missing a port number, a Host header since you are requesting HTTP 1.1, and trailing line breaks to terminate the command properly.
Your GET command is sent to the target server (if CONNECT is successful) and should not be requesting a resource by IP address. It is also sending the wrong value for the Host header. The command is relative to the target server, so it needs to specify the target server's host name.
Also, you should be using send()/sendall() instead of sendto().
Try something more like this instead:
import socket
TargetDomainName="www.stackoverflow.com"
TargetIP="151.101.65.69"
TargetPort=80
ProxiesIP=["107.151.182.247"]
ProxiesPort=[80]
Connection=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
Connection.connect((ProxiesIP[0], ProxiesPort[0]))
Connection.sendall((f"CONNECT {TargetDomainName}:{TargetPort} HTTP/1.1\r\n").encode("ascii"))
Connection.sendall((f"Host: {TargetDomainName}:{TargetPort}\r\n\r\n").encode("ascii"))
print (Connection.recv(1028))
Connection.sendall(("GET / HTTP/1.1\r\n").encode('ascii'))
Connection.sendall((f"Host: {TargetDomainName}\r\n").encode('ascii'))
Connection.sendall(("Connection: close\r\n\r\n").encode('ascii'))
print (Connection.recv(1028))
Connection.close()
You really need to read the proxy's reply before sending the GET command. The proxy will send its own HTTP reply indicating whether it successfully connected to the target server or not.
You really should not be implementing HTTP manually though, there are plenty of HTTP libraries for Python that can handle these details for you. Python even has one built-in: http.client
I'm trying to make JavaScript client to a Python websocket server through an Apache2 proxy.
The client is dead simple:
const socket = io({
transports: ['websocket']
});
I have a NodeJS websocket server and a working Apache2 reverse proxy setup.
Now I want to replace the NodeJS server with a Python server - but none of the example implementations from socket.io works. With each of the my client reports an "error 400" when setting up the websocket connection.
The Python server examples come from here:
https://github.com/miguelgrinberg/python-socketio/tree/master/examples/server
Error 400 stands for "Bad Request" - but I know that my requests are fine because my NodeJS server understands them.
When not running behind a proxy then all Python examples work fine.
What could be the problem?
I found the solution - all the Python socket.io server examples that I refered to are not configured to run behind a reverse proxy. The reason is, that the socket.io server is managing a list of allowed request origins and the automatic list creation is failing in the reverse proxy situation.
This function creates the automatic list of allowed origins (engineio/asyncio_server.py):
def _cors_allowed_origins(self, environ):
default_origins = []
if 'wsgi.url_scheme' in environ and 'HTTP_HOST' in environ:
default_origins.append('{scheme}://{host}'.format(
scheme=environ['wsgi.url_scheme'], host=environ['HTTP_HOST']))
if 'HTTP_X_FORWARDED_HOST' in environ:
scheme = environ.get(
'HTTP_X_FORWARDED_PROTO',
environ['wsgi.url_scheme']).split(',')[0].strip()
default_origins.append('{scheme}://{host}'.format(
scheme=scheme, host=environ['HTTP_X_FORWARDED_HOST'].split(
',')[0].strip()))
As you can see, it only adds URLs with {scheme} as a protocol. When behind a reverse proxy, {scheme} will always be "http". So if the initial request was HTTPS based, it will not be in the list of allowed origins.
The solution to this problem is very simple: when creating the socket.io server, you have to either tell him to allow all origins or specify your origin:
import socketio
sio = socketio.AsyncServer(cors_allowed_origins="*") # allow all
# or
sio = socketio.AsyncServer(cors_allowed_origins="https://example.com") # allow specific
When my app crashes, the client receives a response with:
response.status_code = 500
response.text =
500
Internal Server Error Internal Server Error The
server encountered an internal error and was unable to complete your
request. Either the server is overloaded or there is an error in the
application.
I would like to also show the error trace, to inform the client about what has gone wrong. How can I do that?
Flask server usually crashes after the Exception:
Exception happened during processing of request from ('x.x.x.x', 12554)
This exception usually comes when some client make certain request and closes its connection before receiving the response. Meanwhile, server tries writing to the pipe for the response, which fails as connection is already closed by the client. And, the web server crashes.
Is there any way we can catch this exception, and stops the web server from crashing?
Suppose that I have http server located at (host:port) and I'm trying to use HTTPClient example from the tornado docs to fetch some data from that server, so I have a code below
from tornado import httpclient
http_client = httpclient.HTTPClient()
try:
response = http_client.fetch("http://host:port", connect_timeout=2)
print response.body
except httpclient.HTTPError as e:
print e.code
print e.message
http_client.close()
This works fine and in case everything goes well (host is available, server is working, etc.) my client successfully receives data from the server. Now I want to handle errors that can occur during connection process and handle them depending on the error code. I'm trying to get error code by using e.code in my exception handler, but it contains HTTP error codes only and I need some lower level error codes, for instance:
1) If server is not avaliable on the destination host and port it prints out
599
HTTP 599: [Errno 111] Connection refused
2) If it is impossible to reach the host (for example due to network connectivity problems), I receive the same HTTP 599 error code:
599
HTTP 599: [Errno 101] Connection refused
So I need to get Errno in my exception handler that would indicate actual lower level socket error and the only solution I see for now is to grep it from the text message I receive (e.message) but this solution looks rough and ugly.
I think your only option is to pull it out of e.message:
import re
m = re.search( "\[Errno (\d+)\]", e.message)
if m:
errno = m.group(1)
On my platform (Linux), I actually get a socket.gaierror exception when I take your code example and use a bogus URL, which provides direct access to errno. But if you're dealing with tornado.httpclient.HTTPError, I don't think there's any way for you to get at it directly.