Develop Reference

os.environ.setdefault('DJANGO_SETTINGS_MODULE') doesn't work

I'm trying to run a standalone script that uses the Django models for accessing the database.
The script is very simple, see below:
import sys
from manager.models import Playlist
from manager.utils import clean_up_playlist, add_record_to_playlist
def main(playlist_id, username):
playlist = Playlist.objects.get(playlists=playlist_id)
# the script does other stuff
if __name__ == "__main__":
playlist_id = sys.argv[1]
username = sys.argv[2]
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'SpotifyPlaylistManager.settings')
import django
django.setup()
main(playlist_id, username)
The script is in the top folder of the Django folder
SpotifyPlaylistManager/
|-SpotifyPlaylistManager/
|-settings.py
|-venv
|-manage.py
|-my_script.py
For some reason, if I try to run it with the command below I got the error
raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
The actual command I need to launch
source /home/nicola/PycharmProjects/SpotifyPlaylistManager/venv/bin/activate && python /home/nicola/PycharmProjects/SpotifyPlaylistManager/scheduler.py 6tIMeXF1Q9bB7KDywBhG2P nicoc && deactivate
I can't find the issue

Moving the Django include inside the main worked
if __name__ == "__main__":
playlist_id = sys.argv[1]
username = sys.argv[2]
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'SpotifyPlaylistManager.settings')
import django
django.setup()
from manager.models import Playlist
from manager.utils import clean_up_playlist, add_record_to_playlist
main(playlist_id, username)

Django tests manage.py

To run my Django's tests, I was using the manage.py file.
But I decided to create my own file runtests.py.
As specified in the Django's doc, the manage.py file will execute all methods whose name is starting like "test" and whose class inherits TestCase !
My aim is to use this searching method to display all possible tests (not running them).
So do you know where I could get this "searching method" ?
Thank you !

What you're looking for is called "test discovery", and you don't need to do it manually.
If you want to write a standalone script that runs your tests, you can simply invoke Django's test runner manually from Python code after configuring. The process for this is:
# First you need settings.
# You can do this by setting DJANGO_SETTINGS_MODULE:
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'path.to.your.settings')
# Or you can do it manually:
SETTINGS_DICT = {
# Put all the settings you'll need in here...
}
from django.conf import settings
settings.configure(**SETTINGS_DICT)
import django
django.setup()
import sys
# Now you can run tests:
from django.test.utils import get_runner
TestRunner = get_runner(settings)
test_runner = TestRunner(verbosity=2, interactive=True)
failures = test_runner.run_tests(
['test_module_to_look_in']
)
sys.exit(bool(failures))

Shebang line #!/usr/bin/python3 preventing server run

Here's the code I have. Basically I have the Shebang line in there because the psycopg2 wasn't working without it.
But now when I have this line in there it doesn't allow me to run the database, it just says "no module named 'flask'"
#!/usr/bin/python3.4
#
# Small script to show PostgreSQL and Pyscopg together
#
from flask import Flask, render_template
from flask import request
from flask import *
from datetime import datetime
from functools import wraps
import time
import csv
import psycopg2
app = Flask(__name__)
app.secret_key ='lukey'
def getConn():
connStr=("dbname='test' user='lukey' password='lukey'")
conn=psycopg2.connect(connStr)
return conn
#app.route('/')
def home():
return render_template(index.html)
#app.route('/displayStudent', methods =['GET'])
def displayStudent():
residence = request.args['residence']
try:
conn = None
conn = getConn()
cur = conn.cursor()
cur.execute('SET search_path to public')
cur.execute('SELECT stu_id,student.name,course.name,home_town FROM student,\
course WHERE course = course_id AND student.residence = %s',[residence])
rows = cur.fetchall()
if rows:
return render_template('stu.html', rows = rows, residence = residence)
else:
return render_template('index.html', msg1='no data found')
except Exception as e:
return render_template('index.html', msg1='No data found', error1 = e)
finally:
if conn:
conn.close()
##app.route('/addStudent, methods =['GET','POST']')
#def addStudent():
if __name__ == '__main__':
app.run(debug = True)

This is an environment problem, not a flask, postgres or shebang problem. A specific version of Python is being called, and it is not being given the correct path to its libraries.
Depending on what platform you are on, changing you shebang to #! /usr/bin/env python3 can fix the problem, but if not (very likely not, though using env is considered better/portable practice these days), then you may need to add your Python3 libs location manually in your code.
sys.path.append("/path/to/your/python/libs")
If you know where your Python libs are (or maybe flask is installed somewhere peculiar?) then you can add that to the path and imports following the line where you added to the path will include it in their search for modules.

Using environment dependent django settings inside fabric task

I'm trying to use the database configuration set on settings files to make a database dump using fabric.
There's more than one settings file, so I'd like to be able to do so based on the environment I choose.
by now, my task is like this
def dump_database():
with cd('~/project_folder'), prefix(WORKON_VIRTUALENV):
django.settings_module(env.settings)
from django.conf import settings
dbname = settings.DATABASES['default']['NAME']
dbuser = settings.DATABASES['default']['USER']
dbpassword = settings.DATABASES['default']['PASSWORD']
fname = '/tmp/{0}-backup-{1}.sql.gz'.format(
dbname,
time.strftime('%Y%m%d%H%M%S')
)
run('mysqldump -u %s -p=%s %s | gzip -9 /tmp/backup-%s.sql.gz' % (
dbuser,
dbpassword,
dbname,
fname))
But I'm getting an ImportError:
ImportError: Could not import settings 'project.settings.production'
I've tried to use shell_env() to set the DJANGO_SETTINGS_MODULE instead of django.settings_module(env.settings), with the same result.
I use a task to change the environment based on a environment dict:
def environment(name):
env.update(environments[name])
env.environment = name
This way, I want to be able to create a dump from multiple hosts like:
fab environment:live dump_database
fab environment:otherhost dump_database
Without having to reproduce database settings from all hosts on fabfile.

Importing your Django settings file in fabric is explained here.
http://fabric.readthedocs.org/en/1.3.3/api/contrib/django.html
Quoting from the above link:
from fabric.api import run
from fabric.contrib import django
django.settings_module('myproject.settings')
from django.conf import settings
def dump_production_database():
run('mysqldump -u %s -p=%s %s > /tmp/prod-db.sql' % (
settings.DATABASE_USER,
settings.DATABASE_PASSWORD,
settings.DATABASE_NAME
))

NOTE: I don't answer the question but offer a different solution
I had the same problem.. so I did custom .py script like that:
I created a file named dump_db.py (placed next to fabfile.py for example, that is on the remote machine)
import os
import sys
from datetime import datetime
from django.conf import settings
def dump_mysql():
os.environ.setdefault("DJANGO_SETTINGS_MODULE", SETTINGS_MODULE)
DB_NAME = settings.DATABASES['default']['NAME']
DB_USER = settings.DATABASES['default']['USER']
DB_PASSWORD = settings.DATABASES['default']['PASSWORD']
dump_file_name = '{time}_{db_name}.sql'.format(
time=datetime.now().strftime('%Y_%m_%d'),
db_name=DB_NAME,
)
os.system('mysqldump -u {db_user} -p{db_password} {db_name} > {to_file}'.format(
db_user=DB_USER,
db_password=DB_PASSWORD,
db_name=DB_NAME,
to_file=dump_file_name,
))
return dump_file_name
if __name__ == '__main__':
try:
SETTINGS_MODULE = sys.argv[1:].pop()
except IndexError:
SETTINGS_MODULE = 'project_name.settings'
print dump_mysql()
As you see sys.argv[1:].pop() tries to take optional argument (the setting module in this case).
So in my fabfile:
import os
from fabric.api import env, local, run, prefix, cd
.....
def dump():
current_dir = os.getcwd()
with prefix('source {}bin/activate'.format(env.venv)), cd('{}'.format(env.home)):
dumped_file = run('python dump_db.py {}'.format(env.environment)) # the optional argument given
file_path = os.path.join(env.home, dumped_file)
copy_to = os.path.join(current_dir, dumped_file)
scp(file_path, copy_to)
def scp(file_path, copy_to):
local('scp {}:{} {}'.format(env.host, file_path, copy_to))
where env.environment = 'project_name.settings.env_module'
And this is how I dump my DB and copy it back to me.
Hope it comes handy to someone! :)

Run shell_plus through PyCharm?

Is there a way for me to configure PyCharm to run shell_plus instead of the default shell?
I've tried putting the text of the manage command in the 'Starting script' but then I get the folloiwing
django_manage_shell.run("/Users/cmason/counsyl/code/website/counsyl/product")
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings")
# The new Django 1.4 default manage.py wants "from django..." before
# importing settings, but we usually tinker with sys.path in
# settings_local.py, which is called from settings.py. Importing
# settings.py works but does mean some double importing. Luckily that
# module does very little work.
import settings
# appease pyflakes; don't ever do this in
# non-super-meta-namespace-trickery code
settings
from django.core.management import execute_from_command_line
execute_from_command_line("shellplus")
and it hasn't really run shell_plus.
It seems like the 'Starting script' happens in addition to rather than instead of the default.
Shell_plus automatically imports all Django model classes, among other things.

I got the model objects auto-loading by hooking into the shell_plus code. I appended this to the default startup script in Preferences > Build, Execution, Deployment > Console > Django Console:
from django_extensions.management import shells
from django.core.management.color import color_style
imported_items = shells.import_objects({}, color_style())
for k, v in imported_items.items():
globals()[k] = v
This was on PyCharm 2018.3.3 Pro
For completeness, this was the full content of starting script:
import sys; print('Python %s on %s' % (sys.version, sys.platform))
import django; print('Django %s' % django.get_version())
sys.path.extend([WORKING_DIR_AND_PYTHON_PATHS])
if 'setup' in dir(django): django.setup()
import django_manage_shell; django_manage_shell.run(PROJECT_ROOT)
from django_extensions.management import shells
from django.core.management.color import color_style
imported_items = shells.import_objects({}, color_style())
for k, v in imported_items.items():
globals()[k] = v

I've been looking for a solution to the same problem, and I ended up here. I tried solutions proposed by others, but none of those appeared to solve this issue. So I decided to find another solution. This is what I came up with:
The code block below is the original Django Console starting script of PyCharm 2019.2:
import sys, django
print('Python %s on %s' % (sys.version, sys.platform))
print('Django %s' % django.get_version())
sys.path.extend([WORKING_DIR_AND_PYTHON_PATHS])
if 'setup' in dir(django):
django.setup()
import django_manage_shell
django_manage_shell.run(PROJECT_ROOT)
Installing IPython and changing the last two lines as below gets it done in the most proper way:
from IPython.core.getipython import get_ipython
ipython = get_ipython()
from django_extensions.management.notebook_extension import load_ipython_extension
load_ipython_extension(ipython)
To make it work: open PyCharm settings (CTRL+S) and head to Django Console section. Then make changes in Starting script window and apply. Finally, start the new Python Console instance.

I looked at the source code of shell_plus, and noticed you could use a method on a Command class named get_imported_objects({})
In PyCharm, go to: Build, Execution, Deployment > Console > Django Console > Starting script
Add this to the existing code in that box:
from django_extensions.management.commands.shell_plus import Command
globals().update(Command().get_imported_objects({}))
Note: you may have to restart PyCharm to see the effect.

One way to solve this is to create a new Python run configuration. Set the target to module, and select the manage.py file for the project. Then put shell_plus in the Parameters field. Set the Working Directory to the project directory. Then lastly, set the Execution to Run with Python Console. Apply the changes, then run the new configuration.

This isn't a complete answer, but I found this script that at least loads up all the app models. Put this in Settings > Console > Django Console > Starting script:
import sys
import logging
logging.basicConfig(format="%(levelname)-8s %(asctime)s %(name)s %(message)s", datefmt='%m/%d/%y %H:%M:%S', stream=sys.stdout )
log = logging.getLogger("root")
from django.db.models import get_models
from django.conf import settings
from django.core.exceptions import ObjectDoesNotExist, MultipleObjectsReturned
logging.config.dictConfig(settings.LOGGING)
log.debug("Logging has been initialized at DEBUG")
log.setLevel( logging.DEBUG)
log.disabled = False
for _class in get_models():
if _class.__name__.startswith("Historical"): continue
log.debug("Registering model {}".format(_class.__name__))
globals()[_class.__name__] = _class
def debug_sql():
from debug_toolbar.management.commands import debugsqlshell
return
I also submitted this a feature request to JetBrains.

In Django 1.7, following script can be used as a workaround with PyCharm 3.4:
File -> Settings -> Console -> Django Console and manage.py options
In Starting script, put:
import sys
import django
django.setup()
from django.db.models import get_models
for _class in get_models():
globals()[_class.__name__] = _class

This configuration works for me

As django.db.models.get_models no longer exists, here's an updated version that will accomplish the same as Christopher Mason's version.
import sys; print('Python %s on %s' % (sys.version, sys.platform))
import django; print('Django %s' % django.get_version())
import logging
logging.basicConfig(format="%(levelname)-8s %(asctime)s %(name)s %(message)s", datefmt='%m/%d/%y %H:%M:%S', stream=sys.stdout )
log = logging.getLogger("root")
from django.apps import apps
from django.conf import settings
from django.core.exceptions import ObjectDoesNotExist, MultipleObjectsReturned
logging.config.dictConfig(settings.LOGGING)
log.debug("Logging has been initialized at DEBUG")
log.setLevel( logging.DEBUG)
log.disabled = False
for _configs in apps.get_app_configs():
for _class in _configs.get_models():
if _class.__name__.startswith("Historical"): continue
log.debug("Registering model {}".format(_class.__name__))
globals()[_class.__name__] = apps.get_model(_configs.label, _class.__name__)
def debug_sql():
from debug_toolbar.management.commands import debugsqlshell
return