I'm currently putting my Django app on Docker. I successfully dockerized Django, gunicorn, nginx and Celery, however when I run a Celery task, even though it's executed (displayed in the logs), the results are not stored to my database.
That worked before dockerizing everything so that probably comes from my docker configurations, but I didn't manage to find which part was incorrect/incomplete.
Also, I'm using the default sqlite3 Django database as I don't need to store a huge amount of data.
Here is my docker-compose.yml:
version: '3.8'
services:
django_gunicorn:
volumes:
- db:/db.sqlite3
- static:/static
- media:/media
env_file:
- env
build:
context: .
ports:
- "8000:8000"
command: sh -c "python manage.py migrate && python manage.py collectstatic --no-input && gunicorn main.wsgi:application --bind 0.0.0.0:8000"
nginx:
build: ./nginx
volumes:
- static:/static
- media:/media
ports:
- "80:80"
depends_on:
- django_gunicorn
rabbitmq3:
image: rabbitmq:3-alpine
ports:
- 5672:5672
celery:
restart: always
build:
context: .
command: celery -A main worker -P eventlet -c 100 -l INFO
env_file:
- env
depends_on:
- rabbitmq3
- django_gunicorn
volumes:
- db:/db.sqlite3
volumes:
db:
static:
media:
Dockerfile
FROM python:3.10.5-alpine
ENV PYTHONUNBEFFERED = 1
RUN pip install --upgrade pip
COPY ./requirements.txt .
RUN pip install -r requirements.txt
COPY ./src /app
WORKDIR /app
celery.py
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'main.settings')
app = Celery('main', backend='django-db')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
Maybe it's because I'm using sqlite3 ? I saw everyone using PostgreSQL during my search for a solution. Should I switch to a PostgreSQL ? Or else, what should I change in order to get my tasks results in my database ?
EDIT : settings.py
from pathlib import Path
import os
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = os.getenv('SECRET_KEY')
DEBUG = os.getenv('DEBUG')
# SECURITY WARNING: don't run with debug turned on in production!
#DEBUG = True
ALLOWED_HOSTS = ["127.0.0.1"]
#CSRF_COOKIE_SECURE = True
#SESSION_COOKIE_SECURE = True
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django_celery_results',
'django_celery_beat',
'crispy_forms',
'crispy_bootstrap5',
'fontawesomefree',
'bootstrap_datepicker_plus',
'scripts.apps.ScriptsConfig',
'django_cleanup.apps.CleanupConfig',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
ROOT_URLCONF = 'main.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [BASE_DIR / 'templates'],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
WSGI_APPLICATION = 'main.wsgi.application'
# Database
# https://docs.djangoproject.com/en/3.2/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'db.sqlite3',
}
}
# Password validation
# https://docs.djangoproject.com/en/3.2/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
# Internationalization
# https://docs.djangoproject.com/en/3.2/topics/i18n/
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC'
USE_I18N = True
USE_L10N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/3.2/howto/static-files/
STATIC_ROOT = BASE_DIR / 'static'
STATIC_URL = '/static/'
# Media files (scripts, results)
MEDIA_ROOT = BASE_DIR
MEDIA_URL = 'scripts/'
# Default primary key field type
# https://docs.djangoproject.com/en/3.2/ref/settings/#default-auto-field
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
# Crispy with Bootstrap
CRISPY_TEMPLATE_PACK = 'bootstrap5'
CRISPY_ALLOWED_TEMPLATE_PACKS = "bootstrap5"
CELERY_BROKER_URL="amqp://guest:guest#rabbitmq3:5672/"
Related
I created a simple note with django & react app.
And I am trying to deploy to Heroku.
I tried heroku open at terminal, but this error page came out.
enter image description here
I don't have a folder named frontend.
Maybe it's a problem to try to find index.html from there.
Here is my environment and things i've tried.
File structure)
project-folder
api
env
build
index.html (Here)
public
src
package.json
mynote (my Django main app)
settings.py
manage.py
Procfile
requirements.txt
runtime.txt
requirements.txt)
asgiref==3.5.2
dj-database-url==0.5.0
Django==4.0.5
django-cors-headers==3.13.0
djangorestframework==3.13.1
gunicorn==20.1.0
psycopg2-binary==2.9.3
pytz==2022.1
sqlparse==0.4.2
tzdata==2022.1
whitenoise==6.2.0
runtime.txt)
python-3.10.0
Procfile)
release: python manage.py migrate
web: gunicorn djreact.wsgi --log-file -
settings.py)
"""
Django settings for mynotes project.
"""
from pathlib import Path
import os
BASE_DIR = Path(__file__).resolve().parent.parent
# for heroku deploy
SECRET_KEY = os.environ.get('DJANGO_SECRET_KEY', '~')
# for heroku deploy
DEBUG = bool( os.environ.get('DJANGO_DEBUG', True) )
ALLOWED_HOSTS = [
'[here is my project name!].herokuapp.com',
'127.0.0.1'
]
# Application definition
INSTALLED_APPS = [
~
'rest_framework',
'corsheaders',
'api',
]
MIDDLEWARE = [
'whitenoise.middleware.WhiteNoiseMiddleware',
'django.middleware.security.SecurityMiddleware',
'corsheaders.middleware.CorsMiddleware',
~
]
CORS_ALLOW_ALL_ORIGINS = True
ROOT_URLCONF = 'mynotes.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [
os.path.join(BASE_DIR, 'build')
],
~
},
]
WSGI_APPLICATION = 'mynotes.wsgi.application'
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'db.sqlite3',
}
}
AUTH_PASSWORD_VALIDATORS = [
~
]
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC'
USE_I18N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/4.0/howto/static-files/
STATIC_URL = '/static/'
STATICFILES_DIRS = [
os.path.join(BASE_DIR, 'build/static'),
]
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATICFILES_STORAGE = 'whitenoise.django.GzipManifestStaticFilesStorage'
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
# Heroku: Update database configuration from $DATABASE_URL.
import dj_database_url
db_from_env = dj_database_url.config(conn_max_age=500)
DATABASES['default'].update(db_from_env)
mynotes/urls.py)
from django.contrib import admin
from django.urls import path, include, re_path
from django.views.generic import TemplateView
urlpatterns = [
path('admin/', admin.site.urls),
path('api/', include('api.urls')),
re_path('^.*', TemplateView.as_view(template_name='index.html')),
]
output of termial after execute heroku open)
enter image description here
In 3000 port, 8000 port, my operation test worked as intended. (CRUD with RESTFUL api)
And i did the following in heroku, there was no error.
Except the heroku open
heroku login
heroku create [my server name]
git push heroku main
heroku run python manage.py migrate
And i installed buildpack.
heroku buildpacks:set heroku/python
heroku config:set DISABLE_COLLECTSTATIC=1
Thanks you
I've got a project that runs on Heroku from a Dockerfile and heroku.yml.
The site generally works, but I am having trouble with static files.
collectstatic is run when building the pack.
If I set DEBUG to True, it finds the files.
I'm trying to use whitenoise but not sure why it's not working. It sounds so simple so I'm sure it's something silly.
heroku.yml
setup:
addons:
- plan: heroku-postgresql
build:
docker:
web: Dockerfile
release:
image: web
command:
- python manage.py collectstatic --noinput
run:
web: gunicorn records_project.wsgi
settings.py
MIDDLEWARE = [
'django.middleware.cache.UpdateCacheMiddleware',
'django.middleware.security.SecurityMiddleware',
'whitenoise.middleware.WhiteNoiseMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'django.contrib.sites.middleware.CurrentSiteMiddleware',
]
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'whitenoise.runserver_nostatic',
'django.contrib.staticfiles',
'django.contrib.sites',
... more stuff here...
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/3.0/howto/static-files/
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
# for referencing files with a URL
STATIC_URL = '/static/'
# where to find static files when local
STATICFILES_DIRS = [os.path.join(BASE_DIR, 'static'),]
# location of satatic files for production
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
# how Django should look for static file directories; below is default
STATICFILES_FINDERS = [
# defaults
"django.contrib.staticfiles.finders.FileSystemFinder",
"django.contrib.staticfiles.finders.AppDirectoriesFinder",
]
# This gives me a 500 error
# STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
urls.py
urlpatterns here...
...
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
Any chance you are referencing a static file (e.g., a CSS file or image) in your template that doesn't exist? I was having this same problem because I had the following in my base template:
<link rel="stylesheet" href="{% static 'css/styles.css' %}">
But no styles.css file in my css folder.
I had similar issue because static files, collected on release stage, were missing on run. So i changed my code to:
heroku.yml
setup:
addons:
- plan: heroku-postgresql
build:
docker:
web: Dockerfile
release:
image: web
command:
- chmod u+x entrypoint_heroku.sh
run:
web: ./entrypoint_heroku.sh
Dockerfile
FROM python:3.7
WORKDIR /usr/src/app
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
RUN apt-get update
RUN apt install -y netcat
RUN pip install pipenv
COPY Pipfile* /usr/src/app/
RUN pipenv install --system --dev
COPY . /usr/src/app/
RUN mkdir -p /storage/static/
entrypoint_heroku.sh
#!/bin/sh
python manage.py migrate --noinput
python manage.py collectstatic --noinput
gunicorn app.wsgi
settings.py
STATIC_URL = '/static/'
MEDIA_ROOT = '/storage/'
STATIC_ROOT = MEDIA_ROOT + 'static/'
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
Not the best solution. But it helped me to run server with Debug=False
For what it's worth, I never found a way to get WhiteNoise to serve those static files. I swear it's worked in the past with a similar set up, so that will remain a mystery.
I received a tip from Matt from justdjango.com that Heroku doesn't want to serve static files from that same server. Once I moved my static files over to an AWS S3 bucket, all was well.
I also faced this issue. Although successfully collecting static files on release stage, they weren't present when dyno was up (thus Django missing manifest errors).
This setup worked:
heroku.yml
build:
docker:
web: Dockerfile
run:
web: ./heroku_entrypoint.sh
Dockerfile
--- some code ---
RUN chmod +x heroku_entrypoint.sh
RUN mkdir -p /app/static/
--- other code ---
heroku_entrypoint.sh
python manage.py collectstatic --noinput
python manage.py migrate
gunicorn app.wsgi:application --bind 0.0.0.0:$PORT
I have running on my windows pc a django app with following structure
this app also runs locally and works with this settings
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django_mysql',
'users',
'posts',
'comments',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
# 'chango2.middelware.ProfileCompletionMiddelware',
]
libraries: {
'tags': 'tags',
}
ROOT_URLCONF = 'chango2.urls'
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [os.path.join(BASE_DIR,'templates')],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
this app work as expected when its runned locally with python manage.py runserver
now i trying to deploy this app on a ec2 , i have configurated the security gropus to allow port 80 **and **8000 for inbound and outbound
on ec2 i run with
(env) ubuntu#ip-172-31-3-242:~$ gunicorn --bind 0.0.0.0:8000 chango2.wsgi:application
app starts listening on port 8000 got the first page but for other link
then i recive the
the only thing changed between prod and local was
ALLOWED_HOSTS = [my ec2 dns]
and DATABASES (but that its working)
I am new to Django (but not to Python). I've installed Postgresql 9.3, PgAdmin3 and Psycopg2 successfully in Ubuntu Linux and tested Postgresql manually by creating a database called mysite and a few tables and it's working fine.
I've also tested Django from Python (2.7.6) and it displays the correct version of Django too (1.6).
I am using Django's Official Documentation for setting up everything. I was able to execute
django-admin.py startproject mysite and python manage.py runserver on localhost, port 8000 without any errors. However, whenever I run python manage.py syncdb, it draws a blank forever. It would be really appreciated if someone could help me out with this problem, been stuck for more than a week!
Here's my settings.py:
"""
Django settings for mysite project.
For more information on this file, see
https://docs.djangoproject.com/en/1.6/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.6/ref/settings/
"""
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
import os
BASE_DIR = os.path.dirname(os.path.dirname(__file__))
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.6/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'mduc6%9mt+ca_ir_g6gq8nd(piu90cdtn^fn=u#2om8t=a8!en'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
TEMPLATE_DEBUG = True
ALLOWED_HOSTS = []
# Application definition
INSTALLED_APPS = (
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
)
MIDDLEWARE_CLASSES = (
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
)
ROOT_URLCONF = 'mysite.urls'
WSGI_APPLICATION = 'mysite.wsgi.application'
# Database
# https://docs.djangoproject.com/en/1.6/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'HOST': '127.0.0.1',
'NAME': 'mysite',
'USER': 'akshat',
'PASSWORD': 'password',
'PORT': 8000
}
}
# Internationalization
# https://docs.djangoproject.com/en/1.6/topics/i18n/
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC'
USE_I18N = True
USE_L10N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.6/howto/static-files/
STATIC_URL = '/static/'
And here's the directory hierarchy when viewed from ~/Programming/Django/mysite:
├── __init__.py
├── manage.py
└── mysite
├── __init__.py
├── __init__.pyc
├── settings.py
├── settings.py~
├── settings.pyc
├── urls.py
├── urls.pyc
├── wsgi.py
└── wsgi.pyc
Edit: Also, it doesn't show the error (mentioned in the comments) when the development server is ran, but it does with python manage.py syncdb.
Based on your current error of-
django.db.utils.OperationalError: FATAL: password authentication failed for user "akshat"
FATAL: password authentication failed for user "akshat"
-you need to set up permissions for Postgres to use by editing the file /etc/postgresql/9.1/main/pg_hba.conf.
Just add the following line to the end of the file:
local database_name user_name md5
Be sure to replace database_name with your database name as well as user_name with the actual user name.
I have already finished my website made with Django and would like to put it up. I found Heroku and saw that is free for my basic needs and would like to host it there. My site uses SQLite not PostgreSQL so I would like to know how to use that since it doesn't support sqlite. I found the getting started with Django page on Heroku but I didn't use pip or virtualenv to set it up.
What steps should I follow to get my site up? Just FYI I am using latest Django dev branch 1.8 and Python 3.4.1 and it is a personal website.
Here is my settings.py file
"""
Django settings for mysite project.
For more information on this file, see
https://docs.djangoproject.com/en/dev/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/dev/ref/settings/
"""
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
import os
BASE_DIR = os.path.dirname(os.path.dirname(__file__))
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/dev/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '0r58%!j00w2q1faj*57=d)*fv^=ai#-wgnakj91^--z5f(ohq1'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
TEMPLATE_DEBUG = True
ALLOWED_HOSTS = []
# Application definition
INSTALLED_APPS = (
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
)
MIDDLEWARE_CLASSES = (
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
)
ROOT_URLCONF = 'mysite.urls'
WSGI_APPLICATION = 'mysite.wsgi.application'
# Database
# https://docs.djangoproject.com/en/dev/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
# Internationalization
# https://docs.djangoproject.com/en/dev/topics/i18n/
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC'
USE_I18N = True
USE_L10N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/dev/howto/static-files/
STATIC_URL = '/static/'
TEMPLATE_DIRS = (
os.path.join(os.path.dirname(BASE_DIR), "static", "templates"),
)
if DEBUG:
MEDIA_URL = '/media/'
STATIC_ROOT = os.path.join(os.path.dirname(BASE_DIR), "static", "static-only")
MEDIA_ROOT = os.path.join(os.path.dirname(BASE_DIR), "static", "media")
STATICFILES_DIRS = (
os.path.join(os.path.dirname(BASE_DIR), "static", "static"),
)
You need a requirements.txt with all your dependencies, because heroku uses that file to provide you the services you need.
After you added the postgreSQL add-on to your heroku project, heroku generates you a DATABASE_URL environment variable. you can parse that in your settings.py according to heroku's tutorial:
# Parse database configuration from $DATABASE_URL
import dj_database_url
DATABASES['default'] = dj_database_url.config()
the basic wsgi.py is also provided in the tutorial, you can just use that.
from django.core.wsgi import get_wsgi_application
from dj_static import Cling
application = Cling(get_wsgi_application())
commit the application to your local git repository and push it to the heroku remote repository. if you do that on the console you'll get good feedback about your deployment.
basically you need to follow the getting started tutorial. skipping the virtual environment stuff just makes generating the requirements file more complicated, but it is possible.