Develop Reference

Celery results not updating after dockerization

I'm currently putting my Django app on Docker. I successfully dockerized Django, gunicorn, nginx and Celery, however when I run a Celery task, even though it's executed (displayed in the logs), the results are not stored to my database.
That worked before dockerizing everything so that probably comes from my docker configurations, but I didn't manage to find which part was incorrect/incomplete.
Also, I'm using the default sqlite3 Django database as I don't need to store a huge amount of data.
Here is my docker-compose.yml:
version: '3.8'
services:
django_gunicorn:
volumes:
- db:/db.sqlite3
- static:/static
- media:/media
env_file:
- env
build:
context: .
ports:
- "8000:8000"
command: sh -c "python manage.py migrate && python manage.py collectstatic --no-input && gunicorn main.wsgi:application --bind 0.0.0.0:8000"
nginx:
build: ./nginx
volumes:
- static:/static
- media:/media
ports:
- "80:80"
depends_on:
- django_gunicorn
rabbitmq3:
image: rabbitmq:3-alpine
ports:
- 5672:5672
celery:
restart: always
build:
context: .
command: celery -A main worker -P eventlet -c 100 -l INFO
env_file:
- env
depends_on:
- rabbitmq3
- django_gunicorn
volumes:
- db:/db.sqlite3
volumes:
db:
static:
media:
Dockerfile
FROM python:3.10.5-alpine
ENV PYTHONUNBEFFERED = 1
RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip install -r requirements.txt
COPY ./src /app
WORKDIR /app
celery.py
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'main.settings')
app = Celery('main', backend='django-db')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
Maybe it's because I'm using sqlite3 ? I saw everyone using PostgreSQL during my search for a solution. Should I switch to a PostgreSQL ? Or else, what should I change in order to get my tasks results in my database ?
EDIT : settings.py
from pathlib import Path
import os
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = os.getenv('SECRET_KEY')
DEBUG = os.getenv('DEBUG')
# SECURITY WARNING: don't run with debug turned on in production!
#DEBUG = True
ALLOWED_HOSTS = ["127.0.0.1"]
#CSRF_COOKIE_SECURE = True
#SESSION_COOKIE_SECURE = True
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django_celery_results',
'django_celery_beat',
'crispy_forms',
'crispy_bootstrap5',
'fontawesomefree',
'bootstrap_datepicker_plus',
'scripts.apps.ScriptsConfig',
'django_cleanup.apps.CleanupConfig',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
ROOT_URLCONF = 'main.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [BASE_DIR / 'templates'],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
WSGI_APPLICATION = 'main.wsgi.application'
# Database
# https://docs.djangoproject.com/en/3.2/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'db.sqlite3',
}
}
# Password validation
# https://docs.djangoproject.com/en/3.2/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
# Internationalization
# https://docs.djangoproject.com/en/3.2/topics/i18n/
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC'
USE_I18N = True
USE_L10N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/3.2/howto/static-files/
STATIC_ROOT = BASE_DIR / 'static'
STATIC_URL = '/static/'
# Media files (scripts, results)
MEDIA_ROOT = BASE_DIR
MEDIA_URL = 'scripts/'
# Default primary key field type
# https://docs.djangoproject.com/en/3.2/ref/settings/#default-auto-field
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
# Crispy with Bootstrap
CRISPY_TEMPLATE_PACK = 'bootstrap5'
CRISPY_ALLOWED_TEMPLATE_PACKS = "bootstrap5"
CELERY_BROKER_URL="amqp://guest:guest#rabbitmq3:5672/"

Why it shows Unknown command: 'collectstatic', when I try to collect-static

I am trying to deploy my Django project on Digital Ocean. I created my droplet and spaces on Digital Ocean and created a static folder to store my static files. I pulled my code from my github-repo. then I installed all requirements and tried to collect static files with command
python3 manage.py collectstatic
but it shows
Unknown command: 'collectstatic'
Type 'manage.py help' for usage.
what should I do here?
I checked my manage.py helper but it has no command as collectstatic
check,
compilemessages,
createcachetable,
dbshell,
diffsettings,
dumpdata,
flush,
inspectdb,
loaddata,
makemessages,
makemigrations,
migrate,
runserver,
sendtestemail,
shell,
showmigrations,
sqlflush,
sqlmigrate,
sqlsequencereset,
squashmigrations,
startapp,
startproject,
test,
testserver,
these are the commands in manage.py helper.
And my settings.py is the following
import os
from pathlib import Path
from decouple import config
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
DEBUG = config('DEBUG', default=False, cast=bool)
SECRET_KEY = config("SECRET_KEY")
ALLOWED_HOSTS = ["134.209.153.105",]
ROOT_URLCONF = f'{config("PROJECT_NAME")}.urls'
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.sites',
'crispy_forms',
'accounts',
'adminn',
'student',
'union',
'chat',
'channels',
'allauth',
'allauth.account',
'allauth.socialaccount',
'allauth.socialaccount.providers.google',
]
AWS_ACCESS_KEY_ID = config('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = config('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = config('AWS_STORAGE_BUCKET_NAME')
AWS_S3_ENDPOINT_URL = config('AWS_S3_ENDPOINT_URL')
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_LOCATION = config('AWS_LOCATION')
STATICFILES_DIRS = [
os.path.join(BASE_DIR, 'static'),
]
STATIC_ROOT = os.path.join(BASE_DIR, 'static/')
STATIC_URL = 'https://%s/%s/' % (AWS_S3_ENDPOINT_URL, AWS_LOCATION)
TEMP = os.path.join(BASE_DIR, 'temp')
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
BASE_URL = "http://134.209.153.105"
here, in static url AWS_STORAGE_BUCKET_NAME, AWS_S3_ENDPOINT_UR, AWS_LOCATION are following...
AWS_STORAGE_BUCKET_NAME=studentcricle
AWS_S3_ENDPOINT_URL=https://sfo3.digitaloceanspaces.com
AWS_LOCATION=studentcircle-static

Thank you for those who checked my question.
My problem is solved as I run the following code.
export DJANGO_SETTINGS_MODULE=mysite.settings
I found it from Django documentation.
But I still did not find out what was the real problem.
It was something about my settings file. or multiple settings files
so, if anyone know the details, please describe it here. or in personal.

I think you should uncomment STATIC_ROOT inside your settings.py file and try this:
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')

Try with replacing STATIC_ROOT in settings.py file with the following:
STATIC_ROOT = '/static/'
and whenever you run the python3 manage.py collectstatic command, make sure you are in the base directory where the manage.py file is located and there is a folder named static.

First of all please check if there is a folder in base directory named 'static'; if yes, then make a little change in the following code, remove'/' after static:
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
Secondly, make sure you have registered all the apps in the installed apps menu.
If the problem still persists, then try to run the following command: python manage.py shell
It should let you know where the problem is, if it's in the settings.

Django staticfiles of app does not copying in production after manage.py collectstatic

I'm using Django + Nginx + uwsgi
Settings.py :
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'main',
]
if DEBUG:
STATIC_ROOT = os.path.join(BASE_DIR, "static/")
else:
STATIC_ROOT = "/var/www/mysite/static/"
STATIC_URL = '/static/'
if DEBUG:
MEDIA_ROOT = os.path.join(BASE_DIR, "media/")
else:
MEDIA_ROOT = "/var/www/mysite/media/"
MEDIA_URL = '/media/'
STATICFILES_FINDERS = [
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
]
After update a code (in DEBUG = True all works good, static files are loading correct) I'm using python manage.py collectstatic on DEBUG = False in production server and it's collect admin static, but it's not collecting static for "main" app.
In debug mode all working good.
Checked serving by Nginx - it's working correct.
Why "collectstatic" doesn't collect static from "main" app?
UPD:
I'm an idiot. Forgot about add directory with static files in git. Sry.

Deploying my Django website to Heroku with DEBUG=False doesn't work

When deploying django onto either localhost or heroku with DEBUG=False, it throws an error saying
C:\Users\krish\Envs\PRREMIA\lib\site-packages\whitenoise\base.py:105: UserWarning: No directory at: c:\Users\krish\Documents\python\PRREMIA\staticfiles\
warnings.warn(u'No directory at: {}'.format(root))
[28/Jul/2019 16:05:43] "GET / HTTP/1.1" 500 27
When DEBUG=True, it works fine.
STATIC SETTINGS
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATIC_URL = '/static/'
STATICFILES_DIRS = (
os.path.join(BASE_DIR, 'static'),
)
On Github
Why? And how do I stop and fix this?
Note:
Removing whitenose middleware from MIDDLEWARE and changing STATICFILES_STORAGE
# STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage'
removed the 500 Error, but the staticfiles are still not found.

Configure Django / Create-React-App / Whitenoise to run git deploy to Heroku App with DEBUG = False
Big thank-you to Whitenoise developers first off to make this all possible!
In your settings.py
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [os.path.join(BASE_DIR, 'staticfiles')],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
STATIC_URL = '/static/'
STATICFILES_DIRS = [
os.path.join(BASE_DIR, 'build/static'), os.path.join(BASE_DIR, 'build')
]
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'
In views.py create a template view of index.html
from django.views.generic import TemplateView
from django.views.decorators.cache import never_cache
# Serve Single Page Application
index = never_cache(TemplateView.as_view(template_name='index.html'))
In urls.py add path to your new view
from django.urls import path, include, re_path
from .views import index
urlpatterns += [
re_path('.*', index)
]
Run npm run build and then python manage.py collectstatic before pushing to heroku. This will build your application into a build folder, and then collect these static build files into a staticfiles folder in your project root.
If python manage.py collectstatic fails, create an empty directory called staticfiles in the project root first.
The point of the index re_path view is to always point your application back to index.html when reloading a page at a path like '/login'.
git add .
git commit -m "blah"
git push heroku master
This worked for me, hopefully it helps people somewhat in the future.
If this does not work, my advice is to investigate by running python manage.py runserver locally and messing around with your build folder, staticfiles folder, and settings.py to figure out what is going on. I would also recommend logging to find the error, although I did not do this myself.

Your STATICFILES_DIRS setting is incorrect:
STATICFILES_DIRS = (
os.path.join(BASE_DIR, 'static'),
)
This refers to a directory called static in the root of your project, but there is no such directory.
It looks like your static files are inside the business application, in which case they will be picked up automatically so you can just remove the STATICFILES_DIRS setting altogether.

Hosting an already made Django website with Heroku

I have already finished my website made with Django and would like to put it up. I found Heroku and saw that is free for my basic needs and would like to host it there. My site uses SQLite not PostgreSQL so I would like to know how to use that since it doesn't support sqlite. I found the getting started with Django page on Heroku but I didn't use pip or virtualenv to set it up.
What steps should I follow to get my site up? Just FYI I am using latest Django dev branch 1.8 and Python 3.4.1 and it is a personal website.
Here is my settings.py file
"""
Django settings for mysite project.
For more information on this file, see
https://docs.djangoproject.com/en/dev/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/dev/ref/settings/
"""
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
import os
BASE_DIR = os.path.dirname(os.path.dirname(__file__))
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/dev/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '0r58%!j00w2q1faj*57=d)*fv^=ai#-wgnakj91^--z5f(ohq1'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
TEMPLATE_DEBUG = True
ALLOWED_HOSTS = []
# Application definition
INSTALLED_APPS = (
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
)
MIDDLEWARE_CLASSES = (
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
)
ROOT_URLCONF = 'mysite.urls'
WSGI_APPLICATION = 'mysite.wsgi.application'
# Database
# https://docs.djangoproject.com/en/dev/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
# Internationalization
# https://docs.djangoproject.com/en/dev/topics/i18n/
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC'
USE_I18N = True
USE_L10N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/dev/howto/static-files/
STATIC_URL = '/static/'
TEMPLATE_DIRS = (
os.path.join(os.path.dirname(BASE_DIR), "static", "templates"),
)
if DEBUG:
MEDIA_URL = '/media/'
STATIC_ROOT = os.path.join(os.path.dirname(BASE_DIR), "static", "static-only")
MEDIA_ROOT = os.path.join(os.path.dirname(BASE_DIR), "static", "media")
STATICFILES_DIRS = (
os.path.join(os.path.dirname(BASE_DIR), "static", "static"),
)

You need a requirements.txt with all your dependencies, because heroku uses that file to provide you the services you need.
After you added the postgreSQL add-on to your heroku project, heroku generates you a DATABASE_URL environment variable. you can parse that in your settings.py according to heroku's tutorial:
# Parse database configuration from $DATABASE_URL
import dj_database_url
DATABASES['default'] = dj_database_url.config()
the basic wsgi.py is also provided in the tutorial, you can just use that.
from django.core.wsgi import get_wsgi_application
from dj_static import Cling
application = Cling(get_wsgi_application())
commit the application to your local git repository and push it to the heroku remote repository. if you do that on the console you'll get good feedback about your deployment.
basically you need to follow the getting started tutorial. skipping the virtual environment stuff just makes generating the requirements file more complicated, but it is possible.