I'm working with some friends to build a PostgreSQL/SQLAlchemy Python app and have the following line:
engine = create_engine('postgresql+pg8000://oldmba#localhost/helloworld')
Newbie question: Instead of having to edit in "oldmba" (my username) all the time whenever I git pull someone else's code, what's the simple way to make that line equally applicable to all users so we don't have to constantly edit it? Thanks in advance!
have a config file with your settings.
It can store data in python config dictionary or variables
The config file can import from a local_settings.py file. This file can be ignored in your gitignore. It can contain your individdual settings , username , password, database urls, pretty much anything that you need to configure and that may differ depending on your enviornment (production vs devel)
This is how settings in django projects are usually handled. It allows for multiple users to devlop on the same project with different settings. You might want a 'database_url' field or something too so on production if you need to set your database to a different server but on development you use 'localhost'
# config.py
database = {
'username': 'production_username',
'password': 'production_password'
}
try:
from local_config import *
catch ImportError:
pass
# local_config.py
database = {
'username': 'your_username',
'password': 'your_password'
}
from config import *
engine = create_engine('postgresql+pg8000://{0}#localhost/helloworld'.format(database['username']))
Related
I have a flask application where I login to another service for which I need login data. So I have my endpoint in the flask application /service and this endpoint uses a username and password which I currently have in clear text, meaning
#app.route('/service'), methods = ['GET','POST'])
def access_service(test: str):
username = 'user1'
password = 'passwordincleartext'
req = 'https://anotherservice.com/'
headers = {'Content-type': 'application/json'}
HTTPAUTH = HTTPBasicAuth(username, password)
my_data = '''{"myjsonfield":''' + test + '''}'''
requests.get(req,headers=headers,data=my_data,auth=HTTPAUTH)
My problem is that I can not provide the username and password with the request because another program is using my flask application and this program is an external one where I can not manipulate the request on /service. Is there a way to use a username and password securely, meaning not in clear text, in flask, without having to create a database?
Your passwords or any login credentials should not be included in your code, for that it's preferable and more secure to use something like dot.env, and you'll keep this based to where you project is and not upload this file any way, not even your github repo. please check the following it's a simple and clear explanation of how you can use dot.env
https://dev.to/emma_donery/python-dotenv-keep-your-secrets-safe-4ocn
I suggest you create 2 files, where one will be local to each machine running the code and one will be pushed to github with your code where it shows only the variable names, and example bellow:
# .env file (local machine specific)
USERNAME=user1
PASSWORD=passwordincleartext
# example.env file (pushed with your code)
USERNAME=<ask-from-maintainer>
PASSWORD=<ask-from-maintainer>
NOTE: example.env file will not be used in your code, but you will need it if you are running the code on a different machine, this way all you need is to copy and paste the file, rename it to .env and replace the variables values with the right credentials. This way when you run your code it will work on the new environment without any issue
[NOTE: This is using Django 4.0.2, Python 3.8.2, and Postgres 14.2.]
I have successfully set up Django and Postgres, and I can get them to work together when I put all of the Postgres parameters in the Django settings.py file. However, as shown in the Django documentation, I want to use a Postgres connection service file instead. I've created a service file (C:\Program Files\PostgreSQL\14\etc\pg_service.conf) that looks like this:
[test_svc_1]
host=localhost
user=django_admin
dbname=MyDbName
port=5432
Launching Postgres from the command line with this file seems to work fine, as it prompts me for the password:
> psql service=test_svc_1
Password for user django_admin:
However, when I try to make migrations with Django, I get the following error:
Traceback (most recent call last):
File "C:\...\django\db\backends\base\base.py", line 219, in ensure_connection
self.connect()
File "C:\...\django\utils\asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "C:\...\django\db\backends\base\base.py", line 200, in connect
self.connection = self.get_new_connection(conn_params)
File "C:\...\django\utils\asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "C:\...\django\db\backends\postgresql\base.py", line 187, in get_new_connection
connection = Database.connect(**conn_params)
File "C:\Users\...\psycopg2\__init__.py",line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: definition of service "test_svc_1" not found
There were other exceptions related to this one, such as:
django.db.utils.OperationalError: definition of service "test_svc_1" not found
but they all pointed back to not finding the service "test_svc_1".
Here is an excerpt from my Django settings.py file. Adding the NAME parameter got me a little further along, but I shouldn't need to include it once Django(?) finds the connection service file.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'HOST': 'localhost',
'NAME': 'MyDbName',
'OPTIONS': {
'service': 'test_svc_1',
'passfile': '.my_pgpass',
},
},
}
Any thoughts as to what I'm missing? Worst case, I guess that I can revert to using environment variables and have the settings.py file refer to them. But I'd like to understand what I'm doing wrong rather than giving up.
Thanks for any guidance.
In case anyone runs across this question, I finally got it to work with service and password files thanks to some guidance from Ken Whitesell on the Django forum. Here is what I needed to do:
I had to create Windows environment variables for PGPASSFILE and PSERVICEFILE that point to the two files. I placed them both in C:\Program Files\PostgreSQL\14\etc, although I’m guessing that they can go into other directories as long as the environment variables point to them.
Next, I had to modify my database record in Django's settings.py as follows (which is different from what the Django docs specify):
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'HOST': 'localhost',
'NAME': 'MyDbName',
'OPTIONS': {
'service': 'test_svc_1',
},
},
}
The service name in OPTIONS needs to match the service name in the PGSERVICEFILE.
Two differences from the Django docs:
I needed to include the database name.
I couldn’t get it to work with a ‘passfile’ entry in the OPTIONS dictionary, but it does work when I delete that. Perhaps I was doing something wrong, but I tried several different options and folders.
EDIT: Let me add some explanation points to avoid misunderstanding.
Django documentation is correct, but I would add one note about defining path on Windows (with forward slashes /) as they noted in other documents, where need to write path to file.
Your way above (without passfile key is working because you added PGPASSFILE environment variable and psycopg2 reads path from it.
But you can specify path to pgpass.conf directly (or retrieve it from environment variable, see commented line below).
Also DB host and DB name should be placed in .pg_service.conf file. See all parameter key words in PostgreSQL docs.
Considering this, the following files example should work (taking your settings values):
# C:\Program Files\PostgreSQL\14\etc\.pg_service.conf
[test_svc_1]
host=localhost
user=django_admin
dbname=MyDbName
port=5432
# Django `settings.py`
import os
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'OPTIONS': {
'service': 'test_svc_1',
# 'passfile': os.getenv("PGPASSFILE"),
'passfile': "C:/Program Files/PostgreSQL/14/etc/pgpass.conf",
},
}
}
And of course, line with correct credentials in pgpass.conf must be present:
#hostname:port:database:username:password
localhost:5432:MyDbName:django_admin:your_secret
Turns out that the service name on the ~/.pg_service.conf file must be the database name, i.e: [djangodb] . After that it was not necessary to pass DBNAME. And the environment variable should call the service name only, not the file path. I also was having problems in Django 4 to find my env vars on the settings.py file with os.environ.get('VAR'). After changing to str(os.getenv('VAR')) things got smooth. Bye.
Using Process Monitor I found that the location of the file is in %APPDATA%\Roaming\postgresql\.pg_service.conf
Putting the file in that location didn't have any problems and I didn't have to set any system variables.
In the file I put the host, port, user and password and in django settings I only specified the database and the service.
The postgresql documentation is apparently not correct.
I managed to get it working locally (different cluster, separate settings.py), but not after when deployed to Heroku.
Heroku - automatically adds DATABASE_URL config var with a postgresql, and I cannot remove/edit it.
MongoDB Atlas - I've set the MongoDB Atlas cluster to allow IPs from everywhere. And the password has no funny characters.
django production settings.py
DATABASES = {
'default': {
'ENGINE': 'djongo',
'NAME': 'DBProd',
'CLIENT': {
'host': "mongodb+srv://XXX:YYY#ZZZ.pc4rx.mongodb.net/DBProd?retryWrites=true&w=majority",
}
}
}
I ran migrate straight after the deployment and it's all green OKs
heroku run python manage.py migrate
Everything works functional wise, just that the data are not stored in the MongoDB Atlas cluster. There are lots of posts from various sites on this, but they all have different instructions... Some of the posts I tried to follow:
https://developer.mongodb.com/how-to/use-atlas-on-heroku/
Django + Heroku + MongoDB Atlas (Djongo) = DatabaseError with No Exception
Connecting Heroku App to Atlas MongoDB Cloud service
-- A very confused beginner
For whoever lands on this. Add 'authMechanism': 'SCRAM-SHA-1', below 'host' will fix the issue.
This happened to me as well, in my case, somehow Heroku added postgresql as an add-on for my app so have to delete that first.
Check Heroku documentation about postgresql and how to destroy add-on:
https://devcenter.heroku.com/changelog-items/438
https://devcenter.heroku.com/articles/managing-add-ons
Then you need to configure MongoDB as your database by adding a MONGODB_URI config var with your database url. You should be ready to connect to your MongoDB now!
The following doc also helps:
https://www.mongodb.com/developer/how-to/use-atlas-on-heroku/
I've been having the same issue. Everything works fine locally. The problem is when deploying on Heroku. I have added 'authMechanism': 'SCRAM-SHA-1' and I have also configured MongoDB as my database by adding a MONGODB_URI config var. Heroku still autoconfigures DATABASE_URL config var with a postgresql, and I cannot remove/edit it.
In my Javascript code I used fetch('127.0.0.1:8000/<something>') which I specified in my urls.py and that way views.py read data from pymongo and returned it as a response.
After deploying my app to Heroku (and hardcoding the 127.0.0.1:8000 to <myappname>.heroku.com) the same fetch method seems to return [] instead of json containing values from MongoDB.
This is the most identical issue I found in a post I hope I'm not out of subject.
I have created an own python package (let's call it OwnRep) within which I need to use config settings like
database access address, credentials, etc.
I want to use the own package in other projects. Now, what I want to avoid is that my
credentials are hardcoded within OwnRep.Therefore, I have started to think about intialising
my OwnRep package with some sort of settings setup similar to Django and SciPy, so that
the package's setup are set dynamically.
Any solution proposals for this?
A conceptual picture is attached as a link
enter image description here
Put the credentials in your environment. It is safer. Then create a file called settings.py or anything you want and do something like this:
DATABASES = {
'default': {
'ENGINE': 'your_engine',
'USER': os.environ.get('DB_USER'),
'PASSWORD': os.environ.get('DB_PASSWORD'),
'HOST': '127.0.0.1',
'PORT': '5432',
'NAME': os.environ.get('DB_NAME'),
}
}
To achieve this you need to make sure all required credentials (api_key, username, password) is supplied during instantiation of your own package. This can be done by passing actual information to your own package or by reading from environment variable.
Let's take few real world example:
Sendgrid:
import sendgrid
import os
from sendgrid.helpers.mail import *
sg = sendgrid.SendGridAPIClient(api_key=os.environ.get('SENDGRID_API_KEY'))
from_email = Email("test#example.com")
to_email = To("test#example.com")
subject = "Sending with SendGrid is Fun"
content = Content("text/plain", "and easy to do anywhere, even with Python")
mail = Mail(from_email, to_email, subject, content)
response = sg.client.mail.send.post(request_body=mail.get())
In snippet above, for us to be able to access sendgrid API we need to provide api_key. The api_key can be received from environment variable called SENDGRID_API_KEY even though the implementation behind also do exact same thing. please check https://github.com/sendgrid/sendgrid-python/blob/master/sendgrid/sendgrid.py for more implementation details.
Oy-Client (Python package that i created):
from oy import build_client
oy_client = build_client("https://sandbox.oyindonesia.com/staging/partner", "username", "api-key")
oy_client.get_balance()
In snippet above, for us to be able to access the API we need to provide base_url (where were point it) and api_key.
With both of this example, we can achieve portability and security on our own code according to https://12factor.net/config. And there's no need to hardcoded the configuration / credentials inside your own package.
I was wondering about the safety of some thing in my app.py flask app. First the database, I'm using mysql and currently I am connecting to it in the following way:
# Config MySQL
app.config['MYSQL_HOST'] = 'localhost'
app.config['MYSQL_USER'] = 'root'
app.config['MYSQL_PASSWORD'] = 'password'
app.config['MYSQL_DB'] = 'databasename'
app.config['MYSQL_CURSORCLASS'] = 'DictCursor'
And to me this feels very weird, just putting in your password in plain text etc. I've been searching online but have not found any other way of doing this other than putting it in a seperate python file and just importing it. Which kinda feels like doing nothing at all.. Is there a better way to do this security wise?
Then the secret key I use for password encoding. Which is also just stored in plain text in my code, is there also a way to make this more secure or make it less obvious?
Thanks in advance!
The computer which runs your code needs to know the password, so you can't secure against the owner of the computer (if that's not you). But if you are having the password in the sourcecode it can easily happen that you put it into version control and if you use a public github it can easily happen that you publish your key.
As alternative you can put the password in a config file (take care to not put it into version control e.g. via .gitignore) or you can use environmental variables.
I would suggest to store the credentials in the OS environment.
app.config['MYSQL_HOST'] = os.environ.get('HOST')
app.config['MYSQL_USER'] = os.environ.get('USER')
app.config['MYSQL_PASSWORD'] = os.environ.get('PASSWORD')
app.config['MYSQL_DB'] = os.environ.get('DB')
app.config['MYSQL_CURSORCLASS'] = 'DictCursor'
It will help you to get those information from a standalone application or as a dockerized application (using docker file).
Another way is .env file
pip install python-dotenv
from dotenv import load_dotenv
load_dotenv()
class Config:
SECRET_KEY = os.getenv("SECRET_KEY")
Remember to gitignore .env as well