My streaming dataflow job (from Pub/Sub source) throws multiple error messages from the worker:
Failed to read inputs in the data plane. Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/data_plane.py", line 581, in _read_inputs for elements in elements_iterator: File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 416, in __next__ return self._next() File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 706, in _next raise self grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "keepalive watchdog timeout" debug_error_string = "{"created":"#1619783831.069194941","description":"Error received from peer ipv6:[::1]:12371","file":"src/core/lib/surface/call.cc","file_line":1061,"grpc_message":"keepalive watchdog timeout","grpc_status":14}" >
which leads to another error message:
Python sdk harness failed: Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker_main.py", line 155, in main SdkHarness( File "/usr/local/lib/python3.8/site-packages/apache_beam/runners/worker/sdk_worker.py", line 256, in run for work_request in self._control_stub.Control(get_responses()): File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 416, in __next__ return self._next() File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 689, in _next raise self grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "keepalive watchdog timeout" debug_error_string = "{"created":"#1619786284.366500317","description":"Error received from peer ipv6:[::1]:12371","file":"src/core/lib/surface/call.cc","file_line":1061,"grpc_message":"keepalive watchdog timeout","grpc_status":14}" >
judging from "description":"Error received from peer ipv6:[::1]:12371","file":"src/core/lib/surface/call.cc" line, it seems that for some reason, the job cannot reach the grpc backend of the dataflow service.
Is that correct or any other reason for that error message? I use Apache Beam SDK 2.28 for python 3.8. I also use VPC settings that set using --subnetwork=https://www.googleapis.com/compute/v1/projects/.../regions/.../subnetworks/... pipeline option if that worth mentioning
Related
For my python (python 3.10) based project, we were using aioredis (aioredis 2.0.1) to connect to redis cache and all of a sudden all the requests accessing redis cache started failing. After analysis, we found that Aioredis is now in redis-py. Post that we removed aioredis and added redis (redis 4.5.1) as a dependency in pipfile.
I didn't added any extra code just changed the imports from
import aioredis
to
from redis import asyncio as aioredis
But that didn't resolve the issue completely, now half of the requests are failing with the error code as below.(In an hour 145 requests were success while 79 failed)
redis.exceptions.ConnectionError: Error UNKNOWN while writing to
socket. Connection lost
we use aioredis.Redis for connection
aioredis.Redis(
host=redis_hostname,
port=redis_port,
db=db_name,
password=redis_password,
ssl=true,
connection_pool=aioredis.ConnectionPool.from_url(
f"{redis_protocol}://:{redis_password}#{redis_hostname}:{redis_port}/{db_name}",
connection_class=aioredis.Connection,
max_connections=redis_pool_size,
)
Below is the error trace
Traceback (most recent call last): File
/usr/local/lib/python3.10/site-packages/redis/asyncio/connection.py,
line 788, in send_packed_command await self._writer.drain()
File /usr/local/lib/python3.10/asyncio/streams.py, line 371, in
drain await self._protocol._drain_helper()
Traceback (most recent call last): File
/usr/local/lib/python3.10/site-packages/ddtrace/contrib/asgi/middleware.py,
line 173, in call return await self.app(scope, receive,
wrapped_send)
File /usr/local/lib/python3.10/asyncio/streams.py, line 167, in
_drain_helper raise ConnectionResetError('Connection
lost')ConnectionResetError: Connection lost The above exception was
the direct cause of the following exception:
.....
File /usr/local/lib/python3.10/site-packages/redis/asyncio/client.py,
line 487, in _send_command_parse_response await
conn.send_command(*args)redis.exceptions.ConnectionError: Error
UNKNOWN while writing to socket. Connection lost.
WARNING: Invalid HTTP request received.
Traceback (most recent call last):
File "/home/ubuntu/MySQL_UI_Backend/venv/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 136, in handle_events
event = self.conn.next_event()
File "/home/ubuntu/MySQL_UI_Backend/venv/lib/python3.8/site-packages/h11/_connection.py", line 432, in next_event
raise RemoteProtocolError(
h11._util.RemoteProtocolError: Receive buffer too long
WARNING:uvicorn.error:Invalid HTTP request received.
Traceback (most recent call last):
File "/home/ubuntu/MySQL_UI_Backend/venv/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 136, in handle_events
event = self.conn.next_event()
File "/home/ubuntu/MySQL_UI_Backend/venv/lib/python3.8/site-packages/h11/_connection.py", line 432, in next_event
raise RemoteProtocolError(
h11._util.RemoteProtocolError: Receive buffer too long
The APIs are working fine locally but when I run it on EC2, I get the above error, the ports I have used are 7879 for the API server, 6869 for the frontend UI running on react
1.Make sure that the ports that you use are open at your security group settings.
2.Try changing the url from https to http
I have a server running locally. When I run it on AWS EC2 and send a request from outside on port 8000, I get the following error:
$ uvicorn sql_app.main:app --host="0.0.0.0" --port=8000
INFO: Started server process [9806]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
WARNING: Invalid HTTP request received.
Traceback (most recent call last):
File "/home/ec2-user/.local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 170, in handle_events
event = self.conn.next_event()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 443, in next_event
exc._reraise_as_remote_protocol_error()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_util.py", line 76, in _reraise_as_remote_protocol_error
raise self
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 425, in next_event
event = self._extract_next_receive_event()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 367, in _extract_next_receive_event
event = self._reader(self._receive_buffer)
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_readers.py", line 73, in maybe_read_from_IDLE_client
request_line_re, lines[0], "illegal request line: {!r}", lines[0]
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_util.py", line 88, in validate
raise LocalProtocolError(msg)
h11._util.RemoteProtocolError: illegal request line: bytearray(b'\x16\x03\x01\x02\x00\x01\x00\x01\xfc\x03\x03\x91\xa5\xe2Y\xf0\xa1\xdd\x1d+\x08\x1c\r\x15X\x1d#\x1e/\xb1N\x00\xb5\xe5\xec\xf3F\x1fm\x03\xa1{> \xa80\xb4\x14\x1aUs\xaa\xcd\xc3<s\xcd\xd1\x17\xdf3\x0e\xdbh\xd1c\x88}\x8c\x1f\xa5\x15\x9aa\x14I\x00 ')
WARNING: Invalid HTTP request received.
Traceback (most recent call last):
File "/home/ec2-user/.local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 170, in handle_events
event = self.conn.next_event()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 443, in next_event
exc._reraise_as_remote_protocol_error()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_util.py", line 76, in _reraise_as_remote_protocol_error
raise self
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 425, in next_event
event = self._extract_next_receive_event()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 367, in _extract_next_receive_event
event = self._reader(self._receive_buffer)
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_readers.py", line 68, in maybe_read_from_IDLE_client
raise LocalProtocolError("illegal request line")
h11._util.RemoteProtocolError: illegal request line
WARNING: Invalid HTTP request received.
Traceback (most recent call last):
File "/home/ec2-user/.local/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 170, in handle_events
event = self.conn.next_event()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 443, in next_event
exc._reraise_as_remote_protocol_error()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_util.py", line 76, in _reraise_as_remote_protocol_error
raise self
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 425, in next_event
event = self._extract_next_receive_event()
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_connection.py", line 367, in _extract_next_receive_event
event = self._reader(self._receive_buffer)
File "/home/ec2-user/.local/lib/python3.7/site-packages/h11/_readers.py", line 68, in maybe_read_from_IDLE_client
raise LocalProtocolError("illegal request line")
h11._util.RemoteProtocolError: illegal request line
WARNING: Invalid HTTP request received.
It would be very nice if you could tell me how to do it on port 80.
I was getting the same arcane WARNING: Invalid HTTP request received. error with an unhelpful stack trace. I tried all of the environment variable tweaks recommended and none worked (see FastAPI issue #680, uvicorn issue #441).
My issue was that when I was calling my FastAPI microservice I was using https when my microservice did not have HTTPS support. I changed the url from https to http and it started working as expected.
Note that if your service requires HTTPS support you can add HTTPS support as Ilgizar Murzakov suggests.
Had the same issue. Disabling http2 on application load balancer helped me.
I had the same issue and solved it by adding HTTPS support. I configured NGINX with LetsEncrypt certificate. Make sure that the ports that you use are open at your security group settings.
this is because uvicorn updated to display detailed error. uvicorn #886
then, use uvicorn 0.13.1 which is before the update. this version won't display the error.
P.S: this update is to fix this error. and uvicorn haven't fixed the error for now. uvicorn #1296
I have a migration script that I use to load data into Grakn via the Python driver with Grakn Core 1.6.2. This works.
I have recently downloaded 1.7.1, but when I run the same migration script I get the following error:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grakn/service/Session/TransactionService.py", line 161, in send
response = next(self._response_iterator)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 388, in __next__
return self._next()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 382, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = ""
debug_error_string = "{"created":"#1589227241.242279000","description":"Error received from peer ipv6:[::1]:48555","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"","grpc_status":3}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "migrator.py", line 240, in <module>
insertSchema(URI, KEYSPACE)
File "/Users/johnnie/Documents/grain/insert.py", line 21, in insertSchema
write_transaction.query(schema)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grakn/client.py", line 131, in query
return self._tx_service.query(query, infer)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grakn/service/Session/TransactionService.py", line 49, in query
response = self._communicator.send(request)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grakn/service/Session/TransactionService.py", line 165, in send
raise GraknError("Server/network error: {0}\n\n generated from request: {1}".format(e, request))
grakn.exception.GraknError.GraknError: Server/network error: <_Rendezvous of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = ""
debug_error_string = "{"created":"#1589227241.242279000","description":"Error received from peer ipv6:[::1]:48555","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"","grpc_status":3}"
>
generated from request: query_req {
[SCHEMA IS SHOWN HERE]
}
Any help is greatly appreciated.
Are you using the latest release of both the Python Grakn client and Grakn server? Grakn core 1.7 is not compatible with pre-1.7 clients.
I have the same issue when using session.transaction().read(), when it call TransactionService.py
Becaues Grakn Core 1.7.1 does not support KGMS
Detail
You need to write
pip install --upgrade grakn-client
Sources: python client api docs and pip install docs
(the --upgrade flag because it is already installed.)
I'm in the process of writing a minimal websocket server with Python 3. I am using flask, socketio, and eventlet per the instructions on the latest docs. The problem is that when the webpage with the socket connection is reloaded, the server throws the following exception:
Traceback (most recent call last):
File "C:\Users\Noah\AppData\Local\Programs\Python\Python35-32\lib\site-packages\eventlet\greenpool.py", line 88, in _spawn_n_impl
func(*args, **kwargs)
File "C:\Users\Noah\AppData\Local\Programs\Python\Python35-32\lib\site-packages\eventlet\wsgi.py", line 734, in process_request
proto.__init__(sock, address, self)
File "C:\Users\Noah\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 686, in __init__
self.finish()
File "C:\Users\Noah\AppData\Local\Programs\Python\Python35-32\lib\site-packages\eventlet\wsgi.py", line 651, in finish
greenio.shutdown_safe(self.connection)
File "C:\Users\Noah\AppData\Local\Programs\Python\Python35-32\lib\site-packages\eventlet\greenio\base.py", line 479, in shutdown_safe
return sock.shutdown(socket.SHUT_RDWR)
OSError: [WinError 10038] An operation was attempted on something that is not a socket
I took a look at the source, and it seems like shutdown_safe is supposed to just catch any exceptions while shutting down a connection. In short, it seems like the author of this part of the library didn't foresee Windows throwing an OSError on shutdown.
Although this is a benign issue, I was wondering if there are any existing fixes/tweaks, and if not, whether I should submit this to the python-socketio GitHub issues list.