Description:
I want to have a cached value (let's call it a flag) to know when a celery task finishes execution.
I have a view for the frontend to poll this flag until it turns to False.
Code:
settings.py:
...
MEMCACHED_URL = os.getenv('MEMCACHED_URL', None) # Cache of devel or production
if MEMCACHED_URL:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
'LOCATION': MEMCACHED_URL,
}
}
else:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'unique-snowflake',
}
}
api/views.py:
def a_view(request):
# Do some stuff
cache.add(generated_flag_key, True)
tasks.my_celery_task.apply_async([argument_1, ..., generated_flag_key])
# Checking here with cache.get(generated_flag_key), the value is True.
# Do other stuff.
tasks.py:
#shared_task
def my_celery_task(argument_1, ..., flag_cache_key):
# Do stuff
cache.set(flag_cache_key, False) # Checking here with
# cache.get(flag_cache_key),the
# flag_cache_key value is False
views.py:
def get_cached_value(request, cache_key):
value = cache_key.get(cache_key) # This remains True until the cache key
# expires.
Problem:
If I run the task synchronously everything works as expected. When I run the task asynchronously, the cache key stays the same (as expected) and it is correctly passed around through those 3 methods, but the cached value doesn't seem to be updated between the task and the view.
If you run your tasks asynchronously, they are part of different processes which means that because of the LocMemCache backend, the task and the view will not use the same storage (each has its own memory).
Since #Linovia's answer and a dive in Django's documentation, I am now using django-redis as a workaround for my case.
The only thing that needs to change is the CACHES settings (and an active Redis server of course!):
settings.py:
CACHES = {
"default": {
"BACKEND": "django_redis.cache.RedisCache",
"LOCATION": 'redis://127.0.0.1:6379/1',
"OPTIONS": {
"CLIENT_CLASS": "django_redis.client.DefaultClient",
}
}
}
Now the cache storage is singular.
django-redis is a well-documented library and one can follow the instructions to make it work.
Related
#db_task()
def test_db_access(tenant_id, batch_obj):
print('DBAccess')
print(tenant_id)
print(batch_obj.id)
files = File.objects.filter(batch_id=batch_obj.id)
print(files)
If I run this in django without django-huey, I get a filtered queryset but if I start using django-huey, I'm always getting an empty queryset. Only 'DBAccess' is getting printed and files is always '[]'.
Do I have to add other settings in settings.py?
This is my current huey settings
# Huey - Task Queue
HUEY = {
'name': 'appname',
'consumer': {
'workers': 4,
'worker_type': 'thread'
},
'immediate': False,
'connection': {
'host': RedisConfig.HOST,
'port': RedisConfig.PORT,
},
}
You're trying to pass an object as an argument to that function and it's probably not getting serialized - huey uses pickle to serialize function call data that is passed to the consumer. Instead, change your function to accept a batch_obj identifier like this:
#db_task()
def test_db_access(tenant_id, batch_obj_id):
print('DBAccess')
print(tenant_id)
print(batch_obj_id)
files = File.objects.filter(batch_id=batch_obj_id)
print(files)
and pass in batch_obj_id=batch_obj.id when you're calling test_db_access. Alternatively, you can write a custom serializer, but it should be much simpler to just pass the numerical identifier.
I have a multi-database configuration of Django 1.10 and json fixtures for connections. For example my configuration looks like
DATABASES = {
'default': {
'NAME': 'default',
...,
'TEST': {
'NAME': 'test_default',
}
},
'second': {
'NAME': 'second',
...,
'TEST': {
'NAME': 'test_second',
'MIRROR': 'default',
}
}
}
When Django bootstrap testing environment it loads TestCase.fixtures into not-mirrored connections (in my case only into test_default).
When test case next tries to get model placed in second connection it fails with DoesNotExists.
This happens because fixtures are loaded into first connection that is not committed because of using savepoints between running test cases.
Consequently all tests that assume presence of data in mirroring connections like in master connection will fail!
This looks like a problem with Django test bootstrap algorithm.
Also possible that i do something completely wrong.
Why Django not loads fixtures also into mirrored connections?
-- OR --
Why Django not starts transactions after loading fixtures?
I am trying to make a cache flow, that on a user request it cache a big dict of 870 records and it should stay in cache for some time. When the defined time pass on next request the dict should be updated in na cache memory.
So I have created such a function:
from django.core.cache import get_cache
def update_values_mapping():
cache_en = get_cache('en')
values_dict = get_values_dict() <- this make a request to obtain the dict with values
cache_en.set_many(values_dict, 120) # 120s for testing
cache_en.set('expire', datetime.datetime.now() + datetime.timedelta(seconds=120))
Then in the second function I try to get values from cache
from django.core.cache import get_cache
def get_value_details(_id):
cache = get_cache('en')
details = cache.get(_id, {}) # Values in cache has expire date so they should eventually be gone
expire = cache.get('expire', None)
if not details and expire and expire < datetime.now():
update_values_mapping()
value = cache.get(_id, {})
return details
During rendering a view get_value_details() is called many times to obtain all needed values.
The problem is that some of the values are missing e.g. cache.get('b', {}) return {} even if the value 'b' was saved to cache (and expire date does not pass yet). The missing values are changing, sometimes it is 'a', sometimes 'b', other time 'c' etc.
I have been testing it on LocMemCache and DummyCache so far.
My example cache settings:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'cache-default'
},
'en': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'cache-en'
},
'pl': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'cache-pl'
}
}
When I was playing with that in a console some of the values was disappearing from cache after next call of update_values_mapping(), but some were missing from the beginning.
Does anyone have any clue what it could be ?
Or maybe how to solve described flow in another way ?
LocMemCache is exactly that - a local memory cache. That means it's local to the particular server process, and won't be visible either in other processes or in the console.
If you need something that is shared across all processes, you should use a proper cache backend like memcached or redis.
I found solution for connect django view to redis:
import redis
cacheDB = redis.StrictRedis()
cacheDB.sadd("new_post", post.id)
But when I add code to the view, my page loads with ~2 sec delay. Did it creates a new connect session on every request? Or maybe it's because of my Win7 test platform...
My modules: redis, redis_cache, django_redis.
in settings.py:
CACHES = {
"default": {
"BACKEND": "redis_cache.cache.RedisCache",
"LOCATION": "127.0.0.1:6379:1",
"OPTIONS": {
"CLIENT_CLASS": "redis_cache.client.DefaultClient",
}
}
}
SESSION_ENGINE = 'redis_sessions.session' # for djcelery
And nothing for redis in INSTALLED_APPS, mb i missed something here?
I have some custom flask methods in an eve app that need to communicate with a telnet device and return a result, but I also want to pre-populate data into some resources after retrieving data from this telnet device, like so:
#app.route("/get_vlan_description", methods=['POST'])
def get_vlan_description():
switch = prepare_switch(request)
result = dispatch_switch_command(switch, 'get_vlan_description')
# TODO: populate vlans resource with result data and return status
My settings.py looks like this:
SERVER_NAME = '127.0.0.1:5000'
DOMAIN = {
'vlans': {
'id': {
'type': 'integer',
'required': True,
'unique': True
},
'subnet': {
'type': 'string',
'required': True
},
'description': {
'type': 'boolean',
'default': False
}
}
}
I'm having trouble finding docs or source code for how to access a mongo resource directly and insert this data.
Have you looked into the on_insert hook? From the documentation:
When documents are about to be stored in the database, both on_insert(resource, documents) and on_insert_<resource>(documents) events are raised. Callback functions could hook into these events to arbitrarily add new fields, or edit existing ones. on_insert is raised on every resource being updated while on_insert_<resource> is raised when the <resource> endpoint has been hit with a POST request. In both circumstances, the event will be raised only if at least one document passed validation and is going to be inserted. documents is a list and only contains documents ready for insertion (payload documents that did not pass validation are not included).
So, if I get what you want to achieve, you could have something like this:
def telnet_service(resource, documents):
"""
fetch data from telnet device;
update 'documents' accordingly
"""
pass
app = Eve()
app.on_insert += telnet_service
if __name__ == "__main__":
app.run()
Note that this way you don't have to mess with the database directly as Eve will take care of that.
If you don't want to store the telnet data but only send it back along with the fetched documents, you can hook to on_fetch instead.
Lastly, if you really want to use the data layer you can use app.data.driveras seen in this example snippet.
use post_internal
Usage example:
from run import app
from eve.methods.post import post_internal
payload = {
"firstname": "Ray",
"lastname": "LaMontagne",
"role": ["contributor"]
}
with app.test_request_context():
x = post_internal('people', payload)
print(x)