Develop Reference

Can't connect to Postgres from Django using a connection service file (on Windows)

[NOTE: This is using Django 4.0.2, Python 3.8.2, and Postgres 14.2.]
I have successfully set up Django and Postgres, and I can get them to work together when I put all of the Postgres parameters in the Django settings.py file. However, as shown in the Django documentation, I want to use a Postgres connection service file instead. I've created a service file (C:\Program Files\PostgreSQL\14\etc\pg_service.conf) that looks like this:
[test_svc_1]
host=localhost
user=django_admin
dbname=MyDbName
port=5432
Launching Postgres from the command line with this file seems to work fine, as it prompts me for the password:
> psql service=test_svc_1
Password for user django_admin:
However, when I try to make migrations with Django, I get the following error:
Traceback (most recent call last):
File "C:\...\django\db\backends\base\base.py", line 219, in ensure_connection
self.connect()
File "C:\...\django\utils\asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "C:\...\django\db\backends\base\base.py", line 200, in connect
self.connection = self.get_new_connection(conn_params)
File "C:\...\django\utils\asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "C:\...\django\db\backends\postgresql\base.py", line 187, in get_new_connection
connection = Database.connect(**conn_params)
File "C:\Users\...\psycopg2\__init__.py",line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: definition of service "test_svc_1" not found
There were other exceptions related to this one, such as:
django.db.utils.OperationalError: definition of service "test_svc_1" not found
but they all pointed back to not finding the service "test_svc_1".
Here is an excerpt from my Django settings.py file. Adding the NAME parameter got me a little further along, but I shouldn't need to include it once Django(?) finds the connection service file.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'HOST': 'localhost',
'NAME': 'MyDbName',
'OPTIONS': {
'service': 'test_svc_1',
'passfile': '.my_pgpass',
},
},
}
Any thoughts as to what I'm missing? Worst case, I guess that I can revert to using environment variables and have the settings.py file refer to them. But I'd like to understand what I'm doing wrong rather than giving up.
Thanks for any guidance.

In case anyone runs across this question, I finally got it to work with service and password files thanks to some guidance from Ken Whitesell on the Django forum. Here is what I needed to do:
I had to create Windows environment variables for PGPASSFILE and PSERVICEFILE that point to the two files. I placed them both in C:\Program Files\PostgreSQL\14\etc, although I’m guessing that they can go into other directories as long as the environment variables point to them.
Next, I had to modify my database record in Django's settings.py as follows (which is different from what the Django docs specify):
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'HOST': 'localhost',
'NAME': 'MyDbName',
'OPTIONS': {
'service': 'test_svc_1',
},
},
}
The service name in OPTIONS needs to match the service name in the PGSERVICEFILE.
Two differences from the Django docs:
I needed to include the database name.
I couldn’t get it to work with a ‘passfile’ entry in the OPTIONS dictionary, but it does work when I delete that. Perhaps I was doing something wrong, but I tried several different options and folders.
EDIT: Let me add some explanation points to avoid misunderstanding.
Django documentation is correct, but I would add one note about defining path on Windows (with forward slashes /) as they noted in other documents, where need to write path to file.
Your way above (without passfile key is working because you added PGPASSFILE environment variable and psycopg2 reads path from it.
But you can specify path to pgpass.conf directly (or retrieve it from environment variable, see commented line below).
Also DB host and DB name should be placed in .pg_service.conf file. See all parameter key words in PostgreSQL docs.
Considering this, the following files example should work (taking your settings values):
# C:\Program Files\PostgreSQL\14\etc\.pg_service.conf
[test_svc_1]
host=localhost
user=django_admin
dbname=MyDbName
port=5432
# Django `settings.py`
import os
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'OPTIONS': {
'service': 'test_svc_1',
# 'passfile': os.getenv("PGPASSFILE"),
'passfile': "C:/Program Files/PostgreSQL/14/etc/pgpass.conf",
},
}
}
And of course, line with correct credentials in pgpass.conf must be present:
#hostname:port:database:username:password
localhost:5432:MyDbName:django_admin:your_secret

Turns out that the service name on the ~/.pg_service.conf file must be the database name, i.e: [djangodb] . After that it was not necessary to pass DBNAME. And the environment variable should call the service name only, not the file path. I also was having problems in Django 4 to find my env vars on the settings.py file with os.environ.get('VAR'). After changing to str(os.getenv('VAR')) things got smooth. Bye.

Using Process Monitor I found that the location of the file is in %APPDATA%\Roaming\postgresql\.pg_service.conf
Putting the file in that location didn't have any problems and I didn't have to set any system variables.
In the file I put the host, port, user and password and in django settings I only specified the database and the service.
The postgresql documentation is apparently not correct.

Django(djongo) can't connect to MondoDB Atlas after Heroku deployment

I managed to get it working locally (different cluster, separate settings.py), but not after when deployed to Heroku.
Heroku - automatically adds DATABASE_URL config var with a postgresql, and I cannot remove/edit it.
MongoDB Atlas - I've set the MongoDB Atlas cluster to allow IPs from everywhere. And the password has no funny characters.
django production settings.py
DATABASES = {
'default': {
'ENGINE': 'djongo',
'NAME': 'DBProd',
'CLIENT': {
'host': "mongodb+srv://XXX:YYY#ZZZ.pc4rx.mongodb.net/DBProd?retryWrites=true&w=majority",
}
}
}
I ran migrate straight after the deployment and it's all green OKs
heroku run python manage.py migrate
Everything works functional wise, just that the data are not stored in the MongoDB Atlas cluster. There are lots of posts from various sites on this, but they all have different instructions... Some of the posts I tried to follow:
https://developer.mongodb.com/how-to/use-atlas-on-heroku/
Django + Heroku + MongoDB Atlas (Djongo) = DatabaseError with No Exception
Connecting Heroku App to Atlas MongoDB Cloud service
-- A very confused beginner

For whoever lands on this. Add 'authMechanism': 'SCRAM-SHA-1', below 'host' will fix the issue.

This happened to me as well, in my case, somehow Heroku added postgresql as an add-on for my app so have to delete that first.
Check Heroku documentation about postgresql and how to destroy add-on:
https://devcenter.heroku.com/changelog-items/438
https://devcenter.heroku.com/articles/managing-add-ons
Then you need to configure MongoDB as your database by adding a MONGODB_URI config var with your database url. You should be ready to connect to your MongoDB now!
The following doc also helps:
https://www.mongodb.com/developer/how-to/use-atlas-on-heroku/

I've been having the same issue. Everything works fine locally. The problem is when deploying on Heroku. I have added 'authMechanism': 'SCRAM-SHA-1' and I have also configured MongoDB as my database by adding a MONGODB_URI config var. Heroku still autoconfigures DATABASE_URL config var with a postgresql, and I cannot remove/edit it.
In my Javascript code I used fetch('127.0.0.1:8000/<something>') which I specified in my urls.py and that way views.py read data from pymongo and returned it as a response.
After deploying my app to Heroku (and hardcoding the 127.0.0.1:8000 to <myappname>.heroku.com) the same fetch method seems to return [] instead of json containing values from MongoDB.
This is the most identical issue I found in a post I hope I'm not out of subject.

Pythonanywhere MySQL connection

I'm new to pythonanywhere and am currently deploying my first app with it and the bottle framework. I have created a db with the online console but I don't know the syntax for accessing it. Is it the same syntax as when deploying locally? Or is it something else? MySQLdb has been imported... Thanks for any help.

Here is page on using MySQL at PythonAnywhere. It suggests to use the following configuration for django:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': '<your_username>$<your_database_name>',
'USER': '<your_username>',
'PASSWORD': '<your_mysql_password>',
'HOST': 'mysql.server',
}
}
And I'm almost entirely sure that these credentials will be acceped by MySQLdb driver:
db=MySQLdb.connect(
host='mysql.server',
user='<your_username>',
passwd='<your_mysql_password>',
db='<your_username>$<your_database_name>')

git merge conflicts with our database file (multiple developers)

A buddy and I are developing a Django app and are using git.
As we work, we make fake accounts on our site, login, and upload content to the database, etc..for testing purposes. Every time we merge branches, we get merge conflicts in our database file. The database file is in the repository, and, since we're testing separately, the local copies of the file develop differently.
How do I prevent the database file from being tracked, so we can each keep our local copies?
With the following, we've been able to avoid using a local path:
## settings.py
from os.path import dirname, join
PROJECT_DIR = dirname(__file__)
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': join(PROJECT_DIR, 'foo.db'),
'USER': '',
'PASSWORD': '',
'HOST': '',
'PORT': '',
}
}
What would be ideal, is something like:
## settings.py
from os.path import dirname, join
PROJECT_DIR = dirname(__file__)
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': join('../../../', PROJECT_DIR, 'foo.db'), # this path is outside the repository (ie, 'Users/sgarza62/foo.db')
'USER': '',
'PASSWORD': '',
'HOST': '',
'PORT': '',
}
}
How can we keep our database files from being committed?

Add your database file to .gitignore. Then you can keep it in its current location, but it will not be under version control.

First off, you'll want to remove the database file from your git repository.
git rm <database_file>
To prevent the file from being added to your repository, create a file named ".gitignore" inside your checkout of the repository, add the database file to .gitignore, and add .gitignore to your repository. (Documentation)
To prevent conflicts with settings.py, I also add settings.py to .gitignore. I then create a file called "settings.production.py", which contains all of the settings for the production server, and add it to the repository. On my local checkout, I simply copy this file into settings.py and modify variables as needed. On my production server, I make a symlink to settings.production.py.
ln -s settings.production.py settings.py
WARNING:
If your repository is public, it should never store secret keys, passwords, certificates, etc. You don't want others to have access to these files.
You should also verify that your web server does not serve ".git" folders. A hacker could gain access to your source code if http://example.com/.git is accessible.

When you work on projects with other ppl sharing repo, you have to make local_settings.py and keep there all local settings :) Then in settings.py just add from local_settings import *.
And add local_settings.py and database file to .gitignore file.
In example if your file name is database.db then in directory with this file create file with name .gitignore and write in it database.db or *.db to ignore all db files.

This is a common problem. I would recommend not checking in the database and loading and saving data fixtures as needed. (https://docs.djangoproject.com/en/dev/howto/initial-data/)
Create a test_data directory and run the following commands to export your database to a database agnostic json file:
./manage.py dumpdata > test_data/test_file_1.json
Check that file in to source. At any point if you want to restore the database to that point simply run:
./manage.py loaddata test_data/test_file_1.json
This also has the advantage of being used for unit tests (read Loading fixtures in django unit tests)
from django.test import TestCase
class MyTestCase(TestCase):
fixtures = ['/myapp/fixtures/dump.json',]

Local server address for multiple users on different computers? (Python)

I'm working with some friends to build a PostgreSQL/SQLAlchemy Python app and have the following line:
engine = create_engine('postgresql+pg8000://oldmba#localhost/helloworld')
Newbie question: Instead of having to edit in "oldmba" (my username) all the time whenever I git pull someone else's code, what's the simple way to make that line equally applicable to all users so we don't have to constantly edit it? Thanks in advance!

have a config file with your settings.
It can store data in python config dictionary or variables
The config file can import from a local_settings.py file. This file can be ignored in your gitignore. It can contain your individdual settings , username , password, database urls, pretty much anything that you need to configure and that may differ depending on your enviornment (production vs devel)
This is how settings in django projects are usually handled. It allows for multiple users to devlop on the same project with different settings. You might want a 'database_url' field or something too so on production if you need to set your database to a different server but on development you use 'localhost'
# config.py
database = {
'username': 'production_username',
'password': 'production_password'
}
try:
from local_config import *
catch ImportError:
pass
# local_config.py
database = {
'username': 'your_username',
'password': 'your_password'
}
from config import *
engine = create_engine('postgresql+pg8000://{0}#localhost/helloworld'.format(database['username']))