Gitlab CI with Python flask app that use config.py? - python

I use a config.py in my flask project to handle login to databases, store secret keys, and such.
example to the the config.py file:
import urllib.parse
import os
class Config:
SECRET_KEY = '123'
#staticmethod
def init_app(app):
pass
class DevelopmentConfig(Config):
DEV = True
# Configure Database URI:
params = urllib.parse.quote_plus("DRIVER={ODBC};" +
"SERVER=azure.com;" +
"DATABASE=db;" +
"UID=po;" +
"PWD=12345")
SQLALCHEMY_DATABASE_URI = "mssql+pyodbc:///?odbc_connect=%s" % params
and I use it as such:
def create_app(config_name):
app_path = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
template_path = os.path.join(app_path, 'server/templates/')
static_path = os.path.join(app_path, 'server/static/')
app = Flask(__name__, template_folder=template_path, static_folder=static_path)
app.config.from_object(config[config_name])
config[config_name].init_app(app)
db.init_app(app)
mail.init_app(app)
login_manager.init_app(app)
app.register_blueprint(bp_front)
app.register_blueprint(bp_back)
app.register_blueprint(auth_blueprint, url_prefix='/auth')
return app
Now everything works as expected, but I want to use Gitlab CI for testing, and since I have a test DB which I content to. So how do I get access to settings for connecting without committing the config.py to the repo?
How\Should I configure flask to use environment variables if the tests are executed by Gitlab CI?

Related

How can I run the main file with flask?

I am trying to run the server of this repository (it's about oauth2): https://github.com/lepture/flask-oauthlib/blob/master/tests/oauth2/server.py
The main file looks like this:
if __name__ == '__main__':
from flask import Flask
app = Flask(__name__)
app.debug = True
app.secret_key = 'development'
app.config.update({
'SQLALCHEMY_DATABASE_URI': 'sqlite:///test.sqlite'
})
app = create_server(app)
app.run()
However, I am getting this error:
Error: Failed to find Flask application or factory in module 'hello'. Use 'FLASK_APP=hello:name' to specify one.
I executed the following commands in terminal:
export FLASK_APP=server.py` and
export FLASK_APP=main.py
After that, I tried rerunning with flask run
Again, I am getting this error:
Error: Failed to find Flask application or factory in module 'main'. Use 'FLASK_APP=main:name' to specify one.
Try this code
from flask import Flask
app = Flask(__name__)
app.debug = True
app.secret_key = 'development'
app.config.update({
'SQLALCHEMY_DATABASE_URI': 'sqlite:///test.sqlite'
})
if __name__ == '__main__':
app.run(host="0.0.0.0", port="5000", debug=True)
You should run it directly
python server.py
And if you want to use flask run then you would have to put all (except app.run()) before if __name__ == '__main__': because flask run will import this file and import will skip code inside if __name__ == '__main__':
# ... other code ...
from flask import Flask
app = Flask(__name__)
app.debug = True
app.secret_key = 'development'
app.config.update({
'SQLALCHEMY_DATABASE_URI': 'sqlite:///test.sqlite'
})
app = create_server(app)
if __name__ == '__main__':
app.run()
And it will need export FLASK_APP=server:app because you want to run file server.py and use instance of Flask() with name app
export FLASK_APP=server:app
flask run
Because code uses standard name app so you could skip it in export
export FLASK_APP=server
flask run
You can also run without export
flask --app server run
Try this one:
But I recommend you to start with the basics before work with databases. Try to render HTML templates, use bootstrap, etc. You are running here. Anyway, this boilerplate works for me.
import os
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
app = Flask(__name__)
# Key for Forms
app.config['SECRET_KEY'] = 'mysecretkey'
# SQL DATABASE AND MODELS
basedir = os.path.abspath(os.path.dirname(__file__))
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + os.path.join(basedir, 'data.sqlite')
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
db = SQLAlchemy(app)
Migrate(app, db)
class Puppy(db.Model):
__tablename__ = 'puppies'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.Text)
def __init__(self, name):
self.name = name
#app.route('/')
def index():
return '<h1>Hello Puppy!</h1>'
if __name__ == '__main__':
app.run(debug=True)

create_all() not working when deploying flask app with Zappa

I'm trying to deploy a flask app on AWS Lambda using Zappa. However, when I call zappa deploy dev, I'm getting an error. After examining the logs, I've found that the error is being caused when the code runs db.create_all() within my create_database() function. The database isn't being created which is strange because when I run it locally, it works fine. I'm including the code for my application and a stack trace of the error below.
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from os import path
from flask_login import LoginManager
db = SQLAlchemy()
DB_NAME = 'database.db'
def create_app():
app = Flask(__name__)
app.config['SECRET_KEY'] = 'asdfghjkl'
app.config['SQLALCHEMY_DATABASE_URI'] = f'sqlite:///{DB_NAME}'
db.init_app(app)
from .views import views
app.register_blueprint(views, url_prefix = '/')
from .models import User
create_database(app)
login_manager = LoginManager()
login_manager.login_view = 'views.home'
login_manager.init_app(app)
#login_manager.user_loader
def load_user(id):
return User.query.get(int(id))
return app
def create_database(app):
if not path.exists('website/' + DB_NAME):
db.create_all(app = app) #where error is occuring
print('Created Database')
After running zappa deploy/update dev:
Any help would be appreciated!

flask wsgi file configuration

I have created a flask app.I want to run this both development and production server.
This is my app.py file:
app = Flask(__name__)
db = SQLAlchemy(app)
app.config.from_object(Config)
db.init_app(app)
if __name__ == "__main__":
app.run()
this is my wsgi.py file:
from app import app as application
app = application
if __name__ == '__main__':
app.run(debug=True, host='0.0.0.0', port=5000)
I want to know if my wsgi configurtion is ok or is it the right way to do it.How can i make sure my production server will run wsgi.py file with the production database configuration?In local i am getting this warning Use a production WSGI server instead.the way i configured willl it ensure it will run on wsgi server in prodcution?While i run this command:
uwsgi --wsgi-file app.py --http :5000
i am getting internal server error in localhost:5000.
One way of handling this is by having a configuration file (config.py) that contains classes for each environment. Then within each of your environments, you'd have a .env file that contains environment-specific variables, such as database URIs, secret keys, etc.
Example config file:
from os import environ, path
from dotenv import load_dotenv
basedir = path.abspath(path.dirname(__file__))
load_dotenv(path.join(basedir, '.env'))
class Config:
"""Base config."""
SECRET_KEY = environ.get('SECRET_KEY')
SESSION_COOKIE_NAME = environ.get('SESSION_COOKIE_NAME')
STATIC_FOLDER = 'static'
TEMPLATES_FOLDER = 'templates'
class ProdConfig(Config):
FLASK_ENV = 'production'
DEBUG = False
TESTING = False
DATABASE_URI = environ.get('PROD_DATABASE_URI')
class DevConfig(Config):
FLASK_ENV = 'development'
DEBUG = True
TESTING = True
DATABASE_URI = environ.get('DEV_DATABASE_URI')
Then, in your application, you can specify which config class you'd like to load.
ENV = environ.get("environment")
# Using a production configuration
if ENV=='PROD':
app.config.from_object('config.ProdConfig')
# Using a development configuration
if ENV=='PROD':
app.config.from_object('config.DevConfig')
For more details, Flask has some docs on this situation

Flask: get error 404 when I try to access index

when I run the Flask Server with flask run, I get error 404 in the Index Page.
* Serving Flask app "sf.py"
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [18/Feb/2021 10:25:56] "GET / HTTP/1.1" 404 -
Not Found
The requested URL was not found on the server. If you entered the URL
manually please check your spelling and try again.
Project Structure
.
├── app
│   ├── models.py
│   ├── routes.py
│   └── __init__.py
├── clients
│   └── client.py
├── migrations
├── tests
│   ├── conftest.py
│   ├── test_models.py
│   ├── test_client.py
│   └── __init__.py
├── publisher.py
├── manage.py
├── run_client.py
├── requirements.txt
└── sf.py
/sf.py
from app import create_app
create_app()
/app/__init__.py
from flask import Flask
from . models import db
POSTGRES = {
'user': 'sf',
'pw': 'sf',
'db': 'sf',
'host': 'localhost',
'port': '5432',
}
def create_app():
app = Flask(__name__)
app.config['DEBUG'] = True
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://%(user)s:%(pw)s#%(host)s:%(port)s/%(db)s' % POSTGRES
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
db.init_app(app)
return app
from app import routes
/app/routes.py
from app import create_app
from app.models import Area, Sensor, Monitoring
from flask import request, jsonify
from flask.views import MethodView
app = create_app()
#app.route('/')
def hello_world():
return 'Hello, World!'
...
I need to use the create_app() because I need an that the /clients/client.py use the app.
/clients/client.py
from paho.mqtt.client import Client
import json
from app import create_app
from app.models import db
from app.models import Monitoring
app = create_app()
class CustomClient(Client):
def add_reading(self, reading):
with app.app_context():
db.session.add(reading)
db.session.commit()
def on_connect(self, client, userdata, flags, rc):
print(
"Connected:",
str(client._host) + ":" + str(client._port)
)
def on_subscribe(self, mqttc, obj, mid, granted_qos):
print(
"Subscribed:",
str(mid), str(granted_qos)
)
def on_message(self, client, userdata, message):
msg = message.payload.decode()
print(message.topic, msg)
data = json.loads(msg)
reading = Monitoring(**data)
self.add_reading(reading)
def run(self):
self.connect("localhost", 1883, 60)
self.subscribe("Main/#", 0)
self.loop_forever()
But in this way I get the 404 error. And I'm not sure that I'm using the app properly. It would be fine to have an app and a db session separate, to test models and client without care the app configuration (probably I need to create a separate config for test?). What I've missed?
You are creating three instances of the Flask() object. One is created in sf.py, the others in routes.py and client.py. The first one is used to serve the site, and so doesn't have your route, because the route is registered with the instance created in routes.py. The 3rd instance, in client.py is independent and isn't further altered, so is not an issue here; more on that below.
Don't create multiple copies, at least not and alter the registrations on one and expect those to be available on the other. Instead, use blueprints to register your views, and then register the blueprint with the Flask() object in your create_app() function. That way you can decouple registration of your routes from creating the Flask() object, and still get your routes registered centrally.
In your routes.py, use:
from app.models import Area, Sensor, Monitoring
from flask import Blueprint, request, jsonify
from flask.views import MethodView
bp = Blueprint('main', __name__)
#bp.route('/')
def hello_world():
return 'Hello, World!'
# ...
and then import that blueprint in create_app():
def create_app():
app = Flask(__name__)
app.config['DEBUG'] = True
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://%(user)s:%(pw)s#%(host)s:%(port)s/%(db)s' % POSTGRES
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
db.init_app(app)
from . import routes
app.register_blueprint(routes.bp)
return app
The reason you want to do the import in create_app() is that in most Flask applications you'll also be using one or more Flask extensions that are generally created outside of create_app() so your views can import them. You'd get a circular import if you tried to import one of those objects in your routes module if your routes module was imported into app.py at the top level.
With this change (to using a blueprint), you avoid creating a separate Flask() instance with registrations that the main instance, used for serving your site, won't see. Even your client.py process will be able to access those routes now, should there be a need (e.g. if you need to generate URLs with url_for()).
Here is an example from an in-production Flask project I built for a client recently, the app.py module contains, in part, the following code:
from flask import Flask
from flask_babel import Babel
from flask_marshmallow import Marshmallow
from flask_migrate import Migrate
from flask_security import Security, SQLAlchemyUserDatastore
from flask_sqlalchemy import SQLAlchemy
babel = Babel()
db = SQLAlchemy()
ma = Marshmallow()
migrate = Migrate()
security = Security()
_app_init_hooks = []
app_init_hook = _app_init_hooks.append
def create_app():
app = Flask(__name__)
for f in _app_init_hooks:
f(app)
return app
#app_init_hook
def _configure(app):
"""Load Flask configurations"""
app.config.from_object(f"{__package__}.config")
# optional local overrides
app.config.from_pyfile("settings.cfg", silent=True)
app.config.from_envvar("PROJECT_NAME_SETTINGS", silent=True)
#app_init_hook
def _init_extensions(app):
"""Initialise Flask extensions"""
if app.env != "production":
# Only load and enable when in debug mode
from flask_debugtoolbar import DebugToolbarExtension
DebugToolbarExtension(app)
# Python-level i18n
babel.init_app(app)
# Database management (models, migrations, users)
from .models import Role, User
db.init_app(app)
migrate.init_app(app, db)
user_datastore = SQLAlchemyUserDatastore(db, User, Role)
security.init_app(app, user_datastore)
# Marshmallow integration (must run after db.init_app())
ma.init_app(app)
#app_init_hook
def _setup_blueprints(app):
"""Import and initialise blueprints"""
from . import users
from .sections import BLUEPRINTS
for blueprint in (*BLUEPRINTS, users.bp):
app.register_blueprint(blueprint)
return app
I've broken up the various components into separate functions to ease readability and maintainability, there are separate blueprints used for distinct site functions (which drives some automation in the UI).
At the top of the module are several Flask extensions that various routes and other modules need access to without having to worry about circular imports, so the blueprints are imported separately inside of the _setup_blueprints() hook function that is called from create_app().
Your use of create_app() in client.py should be fine because it doesn't add any new configuration to the Flask() instance that you'd want to have access to elsewhere, and presumably client.py is used outside of the Flask webserver process. But I would, personally, just make the result of create_app()an instance attribute of your Client instance. You don't need a global there, you only need it to access the database session easily for when add_reading() is called:
class CustomClient(Client):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs):
# Create a Flask context so we can access the SQLAlchemy session
self._app = create_app()
def add_reading(self, reading):
with self._app.app_context():
db.session.add(reading)
db.session.commit()
# ...
If add_reading() is called very frequently, you could consider making app an instance attribute of CustomClient():

Python ModuleNotFoundError during gunicorn start

I want to start gunicorn with >> gunicorn wsgi:app.
But I get an error saying ModuleNotFoundError: No module named 'default_config. My virtuel env is activated. I spent hours on google but couldn't find an answer. Even hints are very much appreciated.
Folder structure:
- App
- flask_app
- __init__.py
- factory.py
- default_config.py
- venv (virtual environment)
- wsgi.py
__init__.py => is empty
### wsgi.py ###
from flask_app.factory import create_app
app = create_app('development')
.
### default_config.py ###
import os
basedir = os.path.abspath(os.path.dirname(__file__))
class Config:
SECRET_KEY = 'development key'
class DevelopmentConfig(Config):
DEBUG = True
class ProductionConfig(Config):
DEBUG = False
config = {
'development': DevelopmentConfig,
'production': ProductionConfig,
'default': DevelopmentConfig
}
.
### factory.py ###
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from default_config import config
db = SQLAlchemy()
def create_app(config_name):
app = Flask(__name__, instance_relative_config=True)
app.config.from_object(config[config_name])
app.config.from_pyfile('config.py')
db.init_app(app)
return app
This is because your wsgi file doesn't know the app location by default. You can use relative path like the comment above or you can just add app to your environment path.

Categories

Resources