Probably this will not be difficult question for python experts, so please help. I want to quickly list all settings of my django project. I want to have a simple python script for that (in a separate file). Here is how I started:
from django.conf import settings
settings.configure()
settings_list = dir(settings)
for i in settings_list:
settings_name = i
print settings_name
In this way I get names of all settings. However after each settings_name I want to print its value. Tried many ways. Looks like those settings are actually empty. For example:
print settings.INSTALLED_APPS
returns empty list. I execute the script from django root directory and inside project's virtual environment.
Please suggest the right method to print out all settings for my Django project.
You can call Django's built-in diffsettings:
from django.core.management.commands import diffsettings
output = diffsettings.Command().handle(default=None, output="hash", all=False)
desensitized = []
for line in output.splitlines():
if "SECRET" in line or "KEY" in line:
continue
desensitized.append(line)
print("\n".join(desensitized))
There are two problems which has to be answered: 1) settings are empty 2) how to iterate over attributes and values in settings object.
Regarding empty settings - referencing to django documenation
from django.conf import settings
settings.configure()
print settings.SECRET_KEY
should work, BUT for some reason it didn't in my case. So instead below code worked for me:
from django.conf import settings
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'my_django_project.settings'
print settings.SECRET_KEY
Then in order to collect attributes and values from settings object, I used below code, which I actually borrowed from django-print-settings:
a_dict = {}
for attr in dir(settings):
value = getattr(settings, attr)
a_dict[attr] = value
for key, value in a_dict.items():
print('%s = %r' % (key, value))
To summarize, my full code in my print_settings.py file now looks:
from django.conf import settings
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'my_django_project.settings'
a_dict = {}
for attr in dir(settings):
value = getattr(settings, attr)
a_dict[attr] = value
for key, value in a_dict.items():
print('%s = %r' % (key, value))
This is not the answer I am expecting, but I found another good solution how to print all settings of Django project.
This can be done by installing python package django-print-settings:
pip install django-print-settings
I found it from here https://readthedocs.org/projects/django-print-settings/. Please refer to that site for setup and usage.
import django, os
from django.conf import settings
os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings' # Settings will pick this up on lazy init.
for attr in dir(settings):
print "%-40s: %s" % (attr, getattr(settings, attr))
Related
How can I get the value of an environment variable in Python?
Environment variables are accessed through os.environ:
import os
print(os.environ['HOME'])
To see a list of all environment variables:
print(os.environ)
If a key is not present, attempting to access it will raise a KeyError. To avoid this:
# Returns `None` if the key doesn't exist
print(os.environ.get('KEY_THAT_MIGHT_EXIST'))
# Returns `default_value` if the key doesn't exist
print(os.environ.get('KEY_THAT_MIGHT_EXIST', default_value))
# Returns `default_value` if the key doesn't exist
print(os.getenv('KEY_THAT_MIGHT_EXIST', default_value))
To check if the key exists (returns True or False)
'HOME' in os.environ
You can also use get() when printing the key; useful if you want to use a default.
print(os.environ.get('HOME', '/home/username/'))
where /home/username/ is the default
Here's how to check if $FOO is set:
try:
os.environ["FOO"]
except KeyError:
print "Please set the environment variable FOO"
sys.exit(1)
Actually it can be done this way:
import os
for item, value in os.environ.items():
print('{}: {}'.format(item, value))
Or simply:
for i, j in os.environ.items():
print(i, j)
For viewing the value in the parameter:
print(os.environ['HOME'])
Or:
print(os.environ.get('HOME'))
To set the value:
os.environ['HOME'] = '/new/value'
You can access the environment variables using
import os
print os.environ
Try to see the content of the PYTHONPATH or PYTHONHOME environment variables. Maybe this will be helpful for your second question.
As for the environment variables:
import os
print os.environ["HOME"]
Import the os module:
import os
To get an environment variable:
os.environ.get('Env_var')
To set an environment variable:
# Set environment variables
os.environ['Env_var'] = 'Some Value'
import os
for a in os.environ:
print('Var: ', a, 'Value: ', os.getenv(a))
print("all done")
That will print all of the environment variables along with their values.
If you are planning to use the code in a production web application code, using any web framework like Django and Flask, use projects like envparse. Using it, you can read the value as your defined type.
from envparse import env
# will read WHITE_LIST=hello,world,hi to white_list = ["hello", "world", "hi"]
white_list = env.list("WHITE_LIST", default=[])
# Perfect for reading boolean
DEBUG = env.bool("DEBUG", default=False)
NOTE: kennethreitz's autoenv is a recommended tool for making project-specific environment variables. For those who are using autoenv, please note to keep the .env file private (inaccessible to public).
There are also a number of great libraries. Envs, for example, will allow you to parse objects out of your environment variables, which is rad. For example:
from envs import env
env('SECRET_KEY') # 'your_secret_key_here'
env('SERVER_NAMES',var_type='list') #['your', 'list', 'here']
You can also try this:
First, install python-decouple
pip install python-decouple
Import it in your file
from decouple import config
Then get the environment variable
SECRET_KEY=config('SECRET_KEY')
Read more about the Python library here.
Edited - October 2021
Following #Peter's comment, here's how you can test it:
main.py
#!/usr/bin/env python
from os import environ
# Initialize variables
num_of_vars = 50
for i in range(1, num_of_vars):
environ[f"_BENCHMARK_{i}"] = f"BENCHMARK VALUE {i}"
def stopwatch(repeat=1, autorun=True):
"""
Source: https://stackoverflow.com/a/68660080/5285732
stopwatch decorator to calculate the total time of a function
"""
import timeit
import functools
def outer_func(func):
#functools.wraps(func)
def time_func(*args, **kwargs):
t1 = timeit.default_timer()
for _ in range(repeat):
r = func(*args, **kwargs)
t2 = timeit.default_timer()
print(f"Function={func.__name__}, Time={t2 - t1}")
return r
if autorun:
try:
time_func()
except TypeError:
raise Exception(f"{time_func.__name__}: autorun only works with no parameters, you may want to use #stopwatch(autorun=False)") from None
return time_func
if callable(repeat):
func = repeat
repeat = 1
return outer_func(func)
return outer_func
#stopwatch(repeat=10000)
def using_environ():
for item in environ:
pass
#stopwatch
def using_dict(repeat=10000):
env_vars_dict = dict(environ)
for item in env_vars_dict:
pass
python "main.py"
# Output
Function=using_environ, Time=0.216224731
Function=using_dict, Time=0.00014206099999999888
If this is true ... It's 1500x faster to use a dict() instead of accessing environ directly.
A performance-driven approach - calling environ is expensive, so it's better to call it once and save it to a dictionary. Full example:
from os import environ
# Slower
print(environ["USER"], environ["NAME"])
# Faster
env_dict = dict(environ)
print(env_dict["USER"], env_dict["NAME"])
P.S- if you worry about exposing private environment variables, then sanitize env_dict after the assignment.
For Django, see Django-environ.
$ pip install django-environ
import environ
env = environ.Env(
# set casting, default value
DEBUG=(bool, False)
)
# reading .env file
environ.Env.read_env()
# False if not in os.environ
DEBUG = env('DEBUG')
# Raises Django's ImproperlyConfigured exception if SECRET_KEY not in os.environ
SECRET_KEY = env('SECRET_KEY')
You should first import os using
import os
and then actually print the environment variable value
print(os.environ['yourvariable'])
of course, replace yourvariable as the variable you want to access.
The tricky part of using nested for-loops in one-liners is that you have to use list comprehension. So in order to print all your environment variables, without having to import a foreign library, you can use:
python -c "import os;L=[f'{k}={v}' for k,v in os.environ.items()]; print('\n'.join(L))"
You can use python-dotenv module to access environment variables
Install the module using:
pip install python-dotenv
Then import the module into your Python file
import os
from dotenv import load_dotenv
# Load the environment variables
load_dotenv()
# Access the environment variable
print(os.getenv("BASE_URL"))
How can I get the value of an environment variable in Python?
Environment variables are accessed through os.environ:
import os
print(os.environ['HOME'])
To see a list of all environment variables:
print(os.environ)
If a key is not present, attempting to access it will raise a KeyError. To avoid this:
# Returns `None` if the key doesn't exist
print(os.environ.get('KEY_THAT_MIGHT_EXIST'))
# Returns `default_value` if the key doesn't exist
print(os.environ.get('KEY_THAT_MIGHT_EXIST', default_value))
# Returns `default_value` if the key doesn't exist
print(os.getenv('KEY_THAT_MIGHT_EXIST', default_value))
To check if the key exists (returns True or False)
'HOME' in os.environ
You can also use get() when printing the key; useful if you want to use a default.
print(os.environ.get('HOME', '/home/username/'))
where /home/username/ is the default
Here's how to check if $FOO is set:
try:
os.environ["FOO"]
except KeyError:
print "Please set the environment variable FOO"
sys.exit(1)
Actually it can be done this way:
import os
for item, value in os.environ.items():
print('{}: {}'.format(item, value))
Or simply:
for i, j in os.environ.items():
print(i, j)
For viewing the value in the parameter:
print(os.environ['HOME'])
Or:
print(os.environ.get('HOME'))
To set the value:
os.environ['HOME'] = '/new/value'
You can access the environment variables using
import os
print os.environ
Try to see the content of the PYTHONPATH or PYTHONHOME environment variables. Maybe this will be helpful for your second question.
As for the environment variables:
import os
print os.environ["HOME"]
Import the os module:
import os
To get an environment variable:
os.environ.get('Env_var')
To set an environment variable:
# Set environment variables
os.environ['Env_var'] = 'Some Value'
import os
for a in os.environ:
print('Var: ', a, 'Value: ', os.getenv(a))
print("all done")
That will print all of the environment variables along with their values.
If you are planning to use the code in a production web application code, using any web framework like Django and Flask, use projects like envparse. Using it, you can read the value as your defined type.
from envparse import env
# will read WHITE_LIST=hello,world,hi to white_list = ["hello", "world", "hi"]
white_list = env.list("WHITE_LIST", default=[])
# Perfect for reading boolean
DEBUG = env.bool("DEBUG", default=False)
NOTE: kennethreitz's autoenv is a recommended tool for making project-specific environment variables. For those who are using autoenv, please note to keep the .env file private (inaccessible to public).
There are also a number of great libraries. Envs, for example, will allow you to parse objects out of your environment variables, which is rad. For example:
from envs import env
env('SECRET_KEY') # 'your_secret_key_here'
env('SERVER_NAMES',var_type='list') #['your', 'list', 'here']
You can also try this:
First, install python-decouple
pip install python-decouple
Import it in your file
from decouple import config
Then get the environment variable
SECRET_KEY=config('SECRET_KEY')
Read more about the Python library here.
Edited - October 2021
Following #Peter's comment, here's how you can test it:
main.py
#!/usr/bin/env python
from os import environ
# Initialize variables
num_of_vars = 50
for i in range(1, num_of_vars):
environ[f"_BENCHMARK_{i}"] = f"BENCHMARK VALUE {i}"
def stopwatch(repeat=1, autorun=True):
"""
Source: https://stackoverflow.com/a/68660080/5285732
stopwatch decorator to calculate the total time of a function
"""
import timeit
import functools
def outer_func(func):
#functools.wraps(func)
def time_func(*args, **kwargs):
t1 = timeit.default_timer()
for _ in range(repeat):
r = func(*args, **kwargs)
t2 = timeit.default_timer()
print(f"Function={func.__name__}, Time={t2 - t1}")
return r
if autorun:
try:
time_func()
except TypeError:
raise Exception(f"{time_func.__name__}: autorun only works with no parameters, you may want to use #stopwatch(autorun=False)") from None
return time_func
if callable(repeat):
func = repeat
repeat = 1
return outer_func(func)
return outer_func
#stopwatch(repeat=10000)
def using_environ():
for item in environ:
pass
#stopwatch
def using_dict(repeat=10000):
env_vars_dict = dict(environ)
for item in env_vars_dict:
pass
python "main.py"
# Output
Function=using_environ, Time=0.216224731
Function=using_dict, Time=0.00014206099999999888
If this is true ... It's 1500x faster to use a dict() instead of accessing environ directly.
A performance-driven approach - calling environ is expensive, so it's better to call it once and save it to a dictionary. Full example:
from os import environ
# Slower
print(environ["USER"], environ["NAME"])
# Faster
env_dict = dict(environ)
print(env_dict["USER"], env_dict["NAME"])
P.S- if you worry about exposing private environment variables, then sanitize env_dict after the assignment.
For Django, see Django-environ.
$ pip install django-environ
import environ
env = environ.Env(
# set casting, default value
DEBUG=(bool, False)
)
# reading .env file
environ.Env.read_env()
# False if not in os.environ
DEBUG = env('DEBUG')
# Raises Django's ImproperlyConfigured exception if SECRET_KEY not in os.environ
SECRET_KEY = env('SECRET_KEY')
You should first import os using
import os
and then actually print the environment variable value
print(os.environ['yourvariable'])
of course, replace yourvariable as the variable you want to access.
The tricky part of using nested for-loops in one-liners is that you have to use list comprehension. So in order to print all your environment variables, without having to import a foreign library, you can use:
python -c "import os;L=[f'{k}={v}' for k,v in os.environ.items()]; print('\n'.join(L))"
You can use python-dotenv module to access environment variables
Install the module using:
pip install python-dotenv
Then import the module into your Python file
import os
from dotenv import load_dotenv
# Load the environment variables
load_dotenv()
# Access the environment variable
print(os.getenv("BASE_URL"))
I have the following structure of my project
project
--project
----settings
------base.py
------development.py
------testing.py
------secrets.json
--functional_tests
--manage.py
development.py and testing.py 'inherit' from base.py
from .base import *
So, where I have problems
I have the SECRET_KEY for Django in secrets.json, which is stored in settings folder
I load this key like this (saw this in "Two scoops of Django")
import json
from django.core.exceptions import ImproperlyConfigured
key = "secrets.json"
with open(key) as f:
secrets = json.loads(f.read())
def get_secret(setting, secret=secrets):
try:
return secrets[setting]
except KeyError:
error_msg = "Set the {} environment variable".format(setting)
raise ImproperlyConfigured(error_msg)
SECRET_KEY = get_secret("SECRET_KEY")
But when I run python manage.py runserver
Blah-blah-blah
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.
After some investigations I got the following
If I put print(os.getcwd()) inside base.py I get /media/grimel/Home/project/ instead of /media/grimel/Home/project/project/settings/
This code works only if I replace:
key = "secrets.json"
by
key = "project/settings/secrets.json"
Personally, I don't like this solution.
So, questions:
Why, for base.py current working directory is so confusing?
What's a better approach in solving this problem?
The working directory is based on how you run the program, in your case python manage.py runserver hints that your working directory is the one containing manage.py. Beware that this can vary when run as WSGI script or otherwise, so your concern with using key = "project/settings/secrets.json" is valid.
One solution is to use the value of __file__ in base.py, likely to be "project/settings/base.py". I would use something like
import os
BASE_DIR = os.path.dirname(__file__)
key = os.path.join(BASE_DIR, "secrets.json")
To make life simpler why not move secrets.json to your project root and reference
import os
key = os.path.join(BASE_DIR, "secrets.json")
directly. This is platform independent saving you the need to override BASE_DIR at all in your settings file. Don't forget to add your settings file to version control.
I'm trying to develop my Scrapy application using multiple configurations depending on my environment (e.g. development, production). My problem is that there are some settings that I'm not sure how to set them. For example, if I have to setup my database, in development should be "localhost" and in production has to be another one.
How can I specify these settings when I'm doing scrapy deploy ? Can I set them with a variable in command-line?
You should set the deploy options in your scrapy.cfg file. For example:
[deploy:dev]
url = http://dev_url/
[deploy:production]
url = http://production_url/
With that, you could do:
scrapyd-deploy def
or
scrapyd-deploy production
You can refer to the answer in the following link :
https://alanbuxton.wordpress.com/2018/10/09/using-local-settings-in-a-scrapy-project/
I copy here for quick reference:
Edit the settings.py file so it would read from additional settings files depending on a SCRAPY_ENV environment variable
Move all the settings files to a separate config directory (and change scrapy.cfg so it knew where to look
The magic happens at the end of settings.py：
from importlib import import_module
from scrapy.utils.log import configure_logging
import logging
import os
SCRAPY_ENV=os.environ.get('SCRAPY_ENV',None)
if SCRAPY_ENV == None:
raise ValueError("Must set SCRAPY_ENV environment var")
logger = logging.getLogger(__name__)
configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})
# Load if file exists; incorporate any names started with an
# uppercase letter into globals()
def load_extra_settings(fname):
if not os.path.isfile("config/%s.py" % fname):
logger.warning("Couldn't find %s, skipping" % fname)
return
mdl=import_module("config.%s" % fname)
names = [x for x in mdl.__dict__ if x[0].isupper()]
globals().update({k: getattr(mdl,k) for k in names})
load_extra_settings("secrets")
load_extra_settings("secrets_%s" % SCRAPY_ENV)
load_extra_settings("settings_%s" % SCRAPY_ENV)
Then in the python file you want to get the variables defined in the setting, use the following code
from scrapy.utils.project import get_project_settings
settings = get_project_settings()
env_variable = settings.get('ENV_VARIABLE')
Is there a way for me to configure PyCharm to run shell_plus instead of the default shell?
I've tried putting the text of the manage command in the 'Starting script' but then I get the folloiwing
django_manage_shell.run("/Users/cmason/counsyl/code/website/counsyl/product")
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings")
# The new Django 1.4 default manage.py wants "from django..." before
# importing settings, but we usually tinker with sys.path in
# settings_local.py, which is called from settings.py. Importing
# settings.py works but does mean some double importing. Luckily that
# module does very little work.
import settings
# appease pyflakes; don't ever do this in
# non-super-meta-namespace-trickery code
settings
from django.core.management import execute_from_command_line
execute_from_command_line("shellplus")
and it hasn't really run shell_plus.
It seems like the 'Starting script' happens in addition to rather than instead of the default.
Shell_plus automatically imports all Django model classes, among other things.
I got the model objects auto-loading by hooking into the shell_plus code. I appended this to the default startup script in Preferences > Build, Execution, Deployment > Console > Django Console:
from django_extensions.management import shells
from django.core.management.color import color_style
imported_items = shells.import_objects({}, color_style())
for k, v in imported_items.items():
globals()[k] = v
This was on PyCharm 2018.3.3 Pro
For completeness, this was the full content of starting script:
import sys; print('Python %s on %s' % (sys.version, sys.platform))
import django; print('Django %s' % django.get_version())
sys.path.extend([WORKING_DIR_AND_PYTHON_PATHS])
if 'setup' in dir(django): django.setup()
import django_manage_shell; django_manage_shell.run(PROJECT_ROOT)
from django_extensions.management import shells
from django.core.management.color import color_style
imported_items = shells.import_objects({}, color_style())
for k, v in imported_items.items():
globals()[k] = v
I've been looking for a solution to the same problem, and I ended up here. I tried solutions proposed by others, but none of those appeared to solve this issue. So I decided to find another solution. This is what I came up with:
The code block below is the original Django Console starting script of PyCharm 2019.2:
import sys, django
print('Python %s on %s' % (sys.version, sys.platform))
print('Django %s' % django.get_version())
sys.path.extend([WORKING_DIR_AND_PYTHON_PATHS])
if 'setup' in dir(django):
django.setup()
import django_manage_shell
django_manage_shell.run(PROJECT_ROOT)
Installing IPython and changing the last two lines as below gets it done in the most proper way:
from IPython.core.getipython import get_ipython
ipython = get_ipython()
from django_extensions.management.notebook_extension import load_ipython_extension
load_ipython_extension(ipython)
To make it work: open PyCharm settings (CTRL+S) and head to Django Console section. Then make changes in Starting script window and apply. Finally, start the new Python Console instance.
I looked at the source code of shell_plus, and noticed you could use a method on a Command class named get_imported_objects({})
In PyCharm, go to: Build, Execution, Deployment > Console > Django Console > Starting script
Add this to the existing code in that box:
from django_extensions.management.commands.shell_plus import Command
globals().update(Command().get_imported_objects({}))
Note: you may have to restart PyCharm to see the effect.
One way to solve this is to create a new Python run configuration. Set the target to module, and select the manage.py file for the project. Then put shell_plus in the Parameters field. Set the Working Directory to the project directory. Then lastly, set the Execution to Run with Python Console. Apply the changes, then run the new configuration.
This isn't a complete answer, but I found this script that at least loads up all the app models. Put this in Settings > Console > Django Console > Starting script:
import sys
import logging
logging.basicConfig(format="%(levelname)-8s %(asctime)s %(name)s %(message)s", datefmt='%m/%d/%y %H:%M:%S', stream=sys.stdout )
log = logging.getLogger("root")
from django.db.models import get_models
from django.conf import settings
from django.core.exceptions import ObjectDoesNotExist, MultipleObjectsReturned
logging.config.dictConfig(settings.LOGGING)
log.debug("Logging has been initialized at DEBUG")
log.setLevel( logging.DEBUG)
log.disabled = False
for _class in get_models():
if _class.__name__.startswith("Historical"): continue
log.debug("Registering model {}".format(_class.__name__))
globals()[_class.__name__] = _class
def debug_sql():
from debug_toolbar.management.commands import debugsqlshell
return
I also submitted this a feature request to JetBrains.
In Django 1.7, following script can be used as a workaround with PyCharm 3.4:
File -> Settings -> Console -> Django Console and manage.py options
In Starting script, put:
import sys
import django
django.setup()
from django.db.models import get_models
for _class in get_models():
globals()[_class.__name__] = _class
This configuration works for me
As django.db.models.get_models no longer exists, here's an updated version that will accomplish the same as Christopher Mason's version.
import sys; print('Python %s on %s' % (sys.version, sys.platform))
import django; print('Django %s' % django.get_version())
import logging
logging.basicConfig(format="%(levelname)-8s %(asctime)s %(name)s %(message)s", datefmt='%m/%d/%y %H:%M:%S', stream=sys.stdout )
log = logging.getLogger("root")
from django.apps import apps
from django.conf import settings
from django.core.exceptions import ObjectDoesNotExist, MultipleObjectsReturned
logging.config.dictConfig(settings.LOGGING)
log.debug("Logging has been initialized at DEBUG")
log.setLevel( logging.DEBUG)
log.disabled = False
for _configs in apps.get_app_configs():
for _class in _configs.get_models():
if _class.__name__.startswith("Historical"): continue
log.debug("Registering model {}".format(_class.__name__))
globals()[_class.__name__] = apps.get_model(_configs.label, _class.__name__)
def debug_sql():
from debug_toolbar.management.commands import debugsqlshell
return