I'm encountering a circumstance where a demonstrably known hash in db2 of our redis cache dies when being requested with .hgetall(key). I'm hoping for some insight! Thank you.
Right, so... first, a sliver of code:
def from_cache(self, cachekey):
""" pull oft needed material from our persistent redis memory cache, ensuring of course that we have a connection """
try:
log.debug('trying to get \'%s\' from cache' % cachekey)
return self.redis.hgetall(cachekey)
except Exception, e:
self.connect_to_cache()
return self.redis.get(cachekey)
resulting in:
2013-05-21 14:45:26,035 23202 DEBUG trying to get 'fax:1112223333' from cache
2013-05-21 14:45:26,036 23202 DEBUG initializing connection to redis/cache memory localhost, port 6379, db 2...
2013-05-21 14:45:26,039 23202 ERROR stopping with an exception
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/simpledaemon/base.py", line 165, in start
self.run()
File "newgov.py", line 51, in run
if self.ready_for_queue(fax):
File "newgov.py", line 61, in ready_for_queue
if self.too_many_already_queued(fax):
File "newgov.py", line 116, in too_many_already_queued
rules = self.from_cache(key)
File "newgov.py", line 142, in from_cache
return self.redis.get(cachekey)
File "/usr/lib/python2.6/site-packages/redis/client.py", line 588, in get
return self.execute_command('GET', name)
File "/usr/lib/python2.6/site-packages/redis/client.py", line 378, in execute_command
return self.parse_response(connection, command_name, **options)
File "/usr/lib/python2.6/site-packages/redis/client.py", line 388, in parse_response
response = connection.read_response()
File "/usr/lib/python2.6/site-packages/redis/connection.py", line 309, in read_response
raise response
ResponseError: Operation against a key holding the wrong kind of value
And here is what is in redis:
$ redis-cli
redis 127.0.0.1:6379> SELECT 2
OK
redis 127.0.0.1:6379[2]> type fax:1112223333
hash
redis 127.0.0.1:6379[2]> hgetall fax:1112223333
1) "delay"
2) "0"
3) "concurrent"
4) "20"
5) "queued"
6) "20"
7) "exclude"
8) ""
Look at your Python stack trace: it fails on "return self.execute_command('GET', name)". It means that:
the hgetall command failed (probably because the connection was not established before)
an exception was raised and caught in your method
the Redis connection is established (I suppose by calling connect_to_cache())
then you try to run "self.redis.get(cachekey)"
it fails of course because the content of cachekey is the key of a hash (not a string)
(here I imagine you should use hgetall instead)
an other exception is raised - the Redis error is a type error (Operation against a key holding the wrong kind of value)
With redis-cli, try to run "GET fax:1112223333", you will have the same error.
There are different types of data when you set and hset. In hset your value is hash map. And when you try get hash map you have same Error. Just try hgetall and redis will return hashmap, or try hget with cachekey and any key of setted hashmap and enjoy
Related
Inserting via debezium connector to mysql database brought up via docker container.
Trying to query and it is working fine until some number of hours. But, after that, same query is throwing below exception.
export JAVA_HOME=/tmp/tests/artifacts/java-17/jdk-17; export PATH=$PATH:/tmp/tests/artifacts/java-17/jdk-17/bin; docker exec -i mysql_be1e6a mysql --user=demo --password=demo -D demo -e "select count(k) from test_cdc_f0bf84 where uuid = 'd1e5cd6d-8f7a-457c-b2ea-880c2be52f69'"
2023-01-02 16:27:43,812:ERROR: failed to execute query MySQL rows count by uuid:
Traceback (most recent call last):
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/env/lib/python3.11/site-packages/paramiko/channel.py", line 699, in recv
out = self.in_buffer.read(nbytes, self.timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/env/lib/python3.11/site-packages/paramiko/buffered_pipe.py", line 164, in read
raise PipeTimeout()
paramiko.buffered_pipe.PipeTimeout
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/suites/cdc/abstract.py", line 667, in try_query
res = query_function()
^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/suites/cdc/test_cdc.py", line 635, in <lambda>
query = lambda: self.mysql_query(
^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/suites/cdc/abstract.py", line 544, in mysql_query
result = self.ssh.exec_on_host(host, [
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/main/connection.py", line 335, in exec_on_host
return self._exec_on_host(host, commands, fetch, timeout=timeout, limit_output=limit_output)[host]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/main/connection.py", line 321, in _exec_on_host
res = list(out)
^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/env/lib/python3.11/site-packages/paramiko/file.py", line 125, in __next__
line = self.readline()
^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/env/lib/python3.11/site-packages/paramiko/file.py", line 291, in readline
new_data = self._read(n)
^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/env/lib/python3.11/site-packages/paramiko/channel.py", line 1361, in _read
return self.channel.recv(size)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/workspace/stress_tests/run_test_with_universe/src/env/lib/python3.11/site-packages/paramiko/channel.py", line 701, in recv
raise socket.timeout()
TimeoutError
After some time, logged manually to machine and tried to read, it still reads fine. Not sure, what does this issue mean.
As explained, tried querying from database via python. Expected it will return count of rows, which it was happening until certain time, but after that, it threw timeout error and socket error.
Trying to query and it is working fine until some number of hours. But, after that, same query is throwing below exception.
The default value for interactive_timeout and wait_timeout is 28880 seconds (8 hours). you can disable this behavior by setting this system variable to zero in your MySQL config.
source: Configuring session timeouts
I'm trying to setup a GRPC client in Python to hit a particular server. The server is setup to require authentication via access token. Therefore, my implementation looks like this:
def create_connection(target, access_token):
credentials = composite_channel_credentials(
ssl_channel_credentials(),
access_token_call_credentials(access_token))
target = target if target else DEFAULT_ENDPOINT
return secure_channel(target = target, credentials = credentials)
conn = create_connection(svc = "myservice", session = Session(client_id = id, client_secret = secret)
stub = FakeStub(conn)
stub.CreateObject(CreateObjectRequest())
The issue I'm having is that, when I attempt to use this connection I get the following error:
File "<stdin>", line 1, in <module>
File "\anaconda3\envs\test\lib\site-packages\grpc\_interceptor.py", line 216, in __call__
response, ignored_call = self._with_call(request,
File "\anaconda3\envs\test\lib\site-packages\grpc\_interceptor.py", line 257, in _with_call
return call.result(), call
File "anaconda3\envs\test\lib\site-packages\grpc\_channel.py", line 343, in result
raise self
File "\anaconda3\envs\test\lib\site-packages\grpc\_interceptor.py", line 241, in continuation
response, call = self._thunk(new_method).with_call(
File "\anaconda3\envs\test\lib\site-packages\grpc\_interceptor.py", line 266, in with_call
return self._with_call(request,
File "\anaconda3\envs\test\lib\site-packages\grpc\_interceptor.py", line 257, in _with_call
return call.result(), call
File "\anaconda3\envs\test\lib\site-packages\grpc\_channel.py", line 343, in result
raise self
File "\anaconda3\envs\test\lib\site-packages\grpc\_interceptor.py", line 241, in continuation
response, call = self._thunk(new_method).with_call(
File "\anaconda3\envs\test\lib\site-packages\grpc\_channel.py", line 957, in with_call
return _end_unary_response_blocking(state, call, True, None)
File "\anaconda3\envs\test\lib\site-packages\grpc\_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{
"created":"#1633399048.828000000",
"description":"Failed to pick subchannel",
"file":"src/core/ext/filters/client_channel/client_channel.cc",
"file_line":3159,
"referenced_errors":[
{
"created":"#1633399048.828000000",
"description":
"failed to connect to all addresses",
"file":"src/core/lib/transport/error_utils.cc",
"file_line":147,
"grpc_status":14
}
]
}"
I looked up the status code associated with this response and it seems that the server is unavailable. So, I tried waiting for the connection to be ready:
channel_ready_future(conn).result()
but this hangs. What am I doing wrong here?
UPDATE 1
I converted the code to use the async connection instead of the synchronous connection but the issue still persists. Also, I saw that this question had also been posted on SO but none of the solutions presented there fixed the problem I'm having.
UPDATE 2
I assumed that this issue was occurring because the client couldn't find the TLS certificate issued by the server so I added the following code:
def _get_cert(target: str) -> bytes:
split_around_port = target.split(":")
data = ssl.get_server_certificate((split_around_port[0], split_around_port[1]))
return str.encode(data)
and then changed ssl_channel_credentials() to ssl_channel_credentials(_get_cert(target)). However, this also hasn't fixed the problem.
The issue here was actually fairly deep. First, I turned on tracing and set GRPC log-level to debug and then found this line:
D1006 12:01:33.694000000 9032 src/core/lib/security/transport/security_handshaker.cc:182] Security handshake failed: {"created":"#1633489293.693000000","description":"Cannot check peer: missing selected ALPN property.","file":"src/core/lib/security/security_connector/ssl_utils.cc","file_line":160}
This lead me to this GitHub issue, which stated that the issue was with grpcio not inserting the h2 protocol into requests, which would cause ALPN-enabled servers to return that specific error. Some further digging led me to this issue, and since the server I connected to also uses Envoy, it was just a matter of modifying the envoy deployment file so that:
clusters:
- name: my-server
connect_timeout: 10s
type: strict_dns
lb_policy: round_robin
http2_protocol_options: {}
hosts:
- socket_address:
address: python-server
port_value: 1337
tls_context:
common_tls_context:
tls_certificates:
alpn_protocols: ["h2"] <====== Add this.
I'm trying to build a list of files in a particular directory on an SFTP server and capture some of the attributes of said files. There's an issue that has been inconsistently popping up when connecting to the server, and I've been unable to find a solution. I say the issue is inconsistent because I can run my Databricks notebook one minute and have it return this particular error but then run it a few minutes later and have it complete successfully with absolutely no changes made to the notebook at all.
from base64 import decodebytes
import paramiko
import pysftp
keydata=b"""host key here"""
key = paramiko.RSAKey(data=decodebytes(keydata))
cnopts = pysftp.CnOpts()
cnopts.hostkeys.add('123.456.7.890', 'ssh-rsa', key)
hostname = "123.456.7.890"
user = "username"
pw = "password"
with pysftp.Connection(host=hostname, username=user, password=pw, cnopts=cnopts) as sftp:
* actions once the connection has been established *
I get the below error message (when it does error out), and it flags the final line of code where I establish the SFTP connection as the culprit. I am unable to reproduce this error on demand. As I said, the code will sometimes run flawlessly and other times return the below error, even though I'm making no changes to the code between runs whatsoever.
Unknown exception: from_buffer() cannot return the address of the raw string within a bytes or unicode object
Traceback (most recent call last):
File "/local_disk0/pythonVirtualEnvDirs/virtualEnv-a488e5a9-de49-48a7-b684-893822004827/lib/python3.5/site-packages/paramiko/transport.py", line 2075, in run
self.kex_engine.parse_next(ptype, m)
File "/local_disk0/pythonVirtualEnvDirs/virtualEnv-a488e5a9-de49-48a7-b684-893822004827/lib/python3.5/site-packages/paramiko/kex_curve25519.py", line 64, in parse_next
return self._parse_kexecdh_reply(m)
File "/local_disk0/pythonVirtualEnvDirs/virtualEnv-a488e5a9-de49-48a7-b684-893822004827/lib/python3.5/site-packages/paramiko/kex_curve25519.py", line 128, in _parse_kexecdh_reply
self.transport._verify_key(peer_host_key_bytes, sig)
File "/local_disk0/pythonVirtualEnvDirs/virtualEnv-a488e5a9-de49-48a7-b684-893822004827/lib/python3.5/site-packages/paramiko/transport.py", line 1886, in _verify_key
if not key.verify_ssh_sig(self.H, Message(sig)):
File "/local_disk0/pythonVirtualEnvDirs/virtualEnv-a488e5a9-de49-48a7-b684-893822004827/lib/python3.5/site-packages/paramiko/rsakey.py", line 134, in verify_ssh_sig
msg.get_binary(), data, padding.PKCS1v15(), hashes.SHA1()
File "/local_disk0/pythonVirtualEnvDirs/virtualEnv-a488e5a9-de49-48a7-b684-893822004827/lib/python3.5/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 474, in verify
self._backend, data, algorithm
File "/local_disk0/pythonVirtualEnvDirs/virtualEnv-a488e5a9-de49-48a7-b684-893822004827/lib/python3.5/site-packages/cryptography/hazmat/backends/openssl/utils.py", line 41, in _calculate_digest_and_algorithm
hash_ctx.update(data)
File "/local_disk0/pythonVirtualEnvDirs/virtualEnv-a488e5a9-de49-48a7-b684-893822004827/lib/python3.5/site-packages/cryptography/hazmat/primitives/hashes.py", line 93, in update
self._ctx.update(data)
File "/local_disk0/pythonVirtualEnvDirs/virtualEnv-a488e5a9-de49-48a7-b684-893822004827/lib/python3.5/site-packages/cryptography/hazmat/backends/openssl/hashes.py", line 50, in update
data_ptr = self._backend._ffi.from_buffer(data)
TypeError: from_buffer() cannot return the address of the raw string within a bytes or unicode object
I'm getting the following error in one of my Celery workers:
2015-07-21T15:02:04.010066+00:00 app[worker.1]: Traceback (most recent call last):
2015-07-21T15:02:04.010069+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 296, in trace_task
2015-07-21T15:02:04.010070+00:00 app[worker.1]: on_chord_part_return(task, state, R)
2015-07-21T15:02:04.010073+00:00 app[worker.1]: deps.delete()
2015-07-21T15:02:04.010074+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/result.py", line 773, in delete
2015-07-21T15:02:04.010071+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 587, in on_chord_part_return
2015-07-21T15:02:04.010078+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 329, in delete_group
2015-07-21T15:02:04.010076+00:00 app[worker.1]: (backend or self.app.backend).delete_group(self.id)
2015-07-21T15:02:04.010079+00:00 app[worker.1]: return self._delete_group(group_id)
2015-07-21T15:02:04.010081+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 499, in _delete_group
2015-07-21T15:02:04.010082+00:00 app[worker.1]: self.delete(self.get_key_for_group(group_id))
2015-07-21T15:02:04.010083+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 172, in delete
2015-07-21T15:02:04.010084+00:00 app[worker.1]: self.client.delete(key)
2015-07-21T15:02:04.010085+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 824, in delete
2015-07-21T15:02:04.010087+00:00 app[worker.1]: return self.execute_command('DEL', *names)
2015-07-21T15:02:04.010088+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 565, in execute_command
2015-07-21T15:02:04.010089+00:00 app[worker.1]: return self.parse_response(connection, command_name, **options)
2015-07-21T15:02:04.010090+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 579, in parse_response
2015-07-21T15:02:04.010091+00:00 app[worker.1]: return self.response_callbacks[command_name](response, **options)
2015-07-21T15:02:04.010093+00:00 app[worker.1]: ValueError: invalid literal for int() with base 10: 'QUEUED'
What I find weird is that I see no call to int in the last line of the stack trace. QUEUED probably came in as a worker's status. I'm using it as a custom worker status like this:
#before_task_publish.connect
def update_sent_state(sender=None, body=None, **kwargs):
# the task may not exist if sent using `send_task` which
# sends tasks by name, so fall back to the default result backend
# if that is the case.
task = current_app.tasks.get(sender)
backend = task.backend if task else current_app.backend
logging.debug("Setting status for %s" % body["id"])
backend.store_result(body['id'], None, "QUEUED")
What could be the issue here?
In case it's relevant, here's the code for my task. I only call fetch directly is fetch.
#app.task
def fetch(url_or_urls, subscribe=None):
"""This fetches a (list of) podcast(s) and stores it in the db. It assumes that it only gets called
by Podcast.get_by_url, or some other method that knows whether a given podcast has
already been fetched.
If *subscribe* is given, it should be a User instance to be subscribed to the given podcasts."""
if isinstance(url_or_urls, basestring):
url_or_urls = [url_or_urls]
body = _store_podcasts.s()
if subscribe:
body.link(_subscribe_user.s(user=subscribe))
return chord([_fetch_podcast_data.s(url) for url in url_or_urls])(body)
#app.task
def _fetch_podcast_data(url):
return do_fetch(url) # This function returns a dict of podcast data.
#app.task
def _store_podcasts(podcasts_data):
"""Given a list of dictionaries representing podcasts, store them all in the database."""
podcasts = [Podcast(**pdata) for pdata in podcasts_data]
return Podcast.objects.insert(podcasts)
#app.task
def _subscribe_user(podcasts, user):
"""Subscribe the given users to all the podcasts in the list."""
return user.subscribe_multi(podcasts)
Is there anything else that could be relevant here?
Library versions as shown by pip freeze:
redis==2.10.3
celery==3.1.18
It is hard to debug such a bug without working code. Here is what i think it could be.
Lets start here:
http://celery.readthedocs.org/en/latest/_modules/celery/backends/base.html#BaseBackend.store_result
def store_result(self, task_id, result, status,
traceback=None, request=None, **kwargs):
"""Update task state and result."""
result = self.encode_result(result, status)
self._store_result(task_id, result, status, traceback,
request=request, **kwargs)
return result
It calls ecnode_result. Lets check that out
def encode_result(self, result, status):
if status in self.EXCEPTION_STATES and isinstance(result, Exception):
return self.prepare_exception(result)
else:
return self.prepare_value(result)
It looks like "status" is expected to be something from predefined STATE constants.
Its code is here
http://celery.readthedocs.org/en/latest/_modules/celery/states.html#state
And docs here
http://celery.readthedocs.org/en/latest/reference/celery.states.html
That does not look like they expect to see something like "QUEUED" there. Try one of the predefined.
The redis python packages expects the response from the DEL action to always be an integer, which I assume is the count of deleted rows.
The call to int happens in the last line (return self.response_callbacks[command_name](response, **options)) where self.response_callbacks['DEL'] is equal to int.
As a workaround, you could subclass the redis.client.StrictRedis and set the DEL response callback to something other than int, just make sure you're familiar with the implications.
I got the same error these days. And founded my QUEUED response comes from redis MULTI commands. See https://redis.io/topics/transactions#usage.
It maybe that your are reading response from wrong connection. Maybe in multi-processing / multi-threading / eventlet, etc. Not sure.
I am trying to get my bottle server so that when one person in a game logs out, everyone can immediately see it. As I am using long polling, there is a request open with all the users.
The bit I am having trouble with is catching the exception that is thrown when the user leaves the page from the long polling that can no longer connect to the page. The error message is here.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 438, in handle_one_response
self.run_application()
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 425, in run_application
self.process_result()
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 416, in process_result
self.write(data)
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 373, in write
self.socket.sendall(msg)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 509, in sendall
data_sent += self.send(_get_memory(data, data_sent), flags)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 483, in send
return sock.send(data, flags)
error: [Errno 32] Broken pipe
<WSGIServer fileno=3 address=0.0.0.0:8080>: Failed to handle request:
request = GET /refreshlobby/1 HTTP/1.1 from ('127.0.0.1', 53331)
application = <bottle.Bottle object at 0x7f9c05672750>
127.0.0.1 - - [2013-07-07 10:59:30] "GET /refreshlobby/1 HTTP/1.1" 200 160 6.038377
The function to handle that page is this.
#route('/refreshlobby/<id>')
def refreshlobby(id):
while True:
yield lobby.refresh()
gevent.sleep(1)
I tried catching the exception within the function, and in a decorator which I put to wrap #route, neither of which worked. I tried making an #error(500) decorator, but that didn't trigger, either. It seems that this is to do with the internals of bottle.
Edit: I know now that I need to be catching socket.error, but I don't know whereabouts in my code
The WSGI runner
Look closely at the traceback: this in not happening in your function, but in the WSGI runner.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 438, in handle_one_response
self.run_application()
The way the WSGI runner works, in your case, is:
Receives a request
Gets a partial response from your code
Sends it to the client (this is where the exception is raised)
Repeats steps 2-3
You can't catch this exception
This error is not raised in your code.
It happens when you try to send a response to a client that closed the connection.
You'll therefore not be able to catch this error from within your code.
Alternate solutions
Unfortunately, it's not possible to tell from within the generator (your code) when it stops being consumed.
It's also not a good idea to rely on your generator being garbage collected.
You have a couple other solutions.
"Last seen"
Another way to know when an user disconnects would probably be to record a "last seen", after your yield statement.
You'll be able to identify clients that disconnected if their last seen is far in the past.
Other runner
Another, non-WSGI runner, will be more appropriate for a realtime application. You could give tornado a try.