Develop Reference

cannot patch class because of __init__.py

I'm writing unit tests for a codebase. My issue is I need to patch a database abstraction class but it is masked by the __init__.py file. My file structure is:
.
├── __init__.py
├── tools
│   ├── __init__.py
│   ├── test
│   │   ├── __init__.py
│   │   └── test_tool1.py
│   ├── tool1.py
│   └── tool2.py
└── utils
├── __init__.py
└── sql_client.py
Now the content of the files:
# tools/__init__.py
from .tool1 import * # `SQLClient` is imported here as part of tool1
from .tool2 import * # `SQLClient` is imported here again but now it's part of tool2
# tools/tool1.py
from utils import SQLClient
class A(object):
...
def run(self, **kwargs):
# the function I want to test
sql = SQLClient("some id")
# tools/tool2.py
from utils import SQLClient
...
# utils/__init__.py
from sql_client import *
# utils/sql_client.py
class SQLClient(object):
# The sql abstraction class that needs to be patched
In my test file, I'm creating a mock class to be used as the patch. I'm using absolute imports in my tests because later I want to move all the tests outside of the source folders.
# tools/test/test_tool1.py
from unittest.mock import MagicMock
from utils import SQLClient
from tools import A
class MockSQLClient(MagicMock):
def __init__(self, *args, **kwargs):
super().__init__(spec=SQLClient)
self._mocks = {"select *": "rows"}
def make_query(query)
return self._mocks[query]
def test_run_func(monkeypatch):
monkeypatch.setattr("tools.SQLClient", MockSQLClient)
a = A()
a.run()
# rest of the test
Now, the issue is that tools/__init__.py imports everything from all the submodules so SQLClient from tool1 is being masked by SQLClient from tool2. As a result, my monkeypatch is patching tool2.SQLClient. I cannot patch the tool1 part directly by monkeypatch.setattr("tools.tool1.SQLClient") because of tools/__init__.py which throws an error complaining there is no tool1 in tools.
EDIT
changed question title, more detail on where the test is.

Error in models folder "doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS"

I can't find a solution for the following error:
RuntimeError: Model class users.models.User.User doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.
The INSTALLED_APPS:
LOCAL_APPS = [
"api.users",
"api.achievements",
"api.others",
]
# https://docs.djangoproject.com/en/dev/ref/settings/#installed-apps
INSTALLED_APPS = DJANGO_APPS + THIRD_PARTY_APPS + LOCAL_APPS
Models folder
users/models/__init__.py:
from .User import * # noqa
from .Profile import * # noqa
users/models/User.py:
""" User related models """
from django.contrib.auth.models import AbstractUser
from django.db import models
from django.urls import reverse
from django_countries.fields import CountryField
from api.others.constants import GENDER
class User(AbstractUser):
"""
Default custom user model for Dezumi API.
If adding fields that need to be filled at user signup,
check forms.SignupForm and forms.SocialSignupForms accordingly.
"""
birth_date = models.DateField(null=True, blank=True)
country = CountryField(null=True, blank=True, blank_label='Select country')
gender = models.CharField(choices=GENDER, max_length=6, null=True, blank=True)
is_verified = models.BooleanField(
default=False,
)
def get_absolute_url(self):
"""Get url for user's detail view.
Returns:
str: URL for user detail.
"""
return reverse("users:detail", kwargs={"username": self.username})
The app is on installed_apps, so what could it be? Something with the init.py?
Directory Structure:
api
├── api
│ ├── __init__.py
│ ├── conftest.py
│ ├── users
│ │ ├── admin
│ │ ├── api
│ │ │ ├── serializers.py
│ │ │ └── views.py
│ │ ├── migrations
│ │ ├── models
│ │ │ ├── __init__.py
│ │ │ ├── Profile.py
│ │ │ └── User.py
│ │ ├── __init__.py
│ │ ├── apps.py
│ │ └── ...
│ └── ...
├── config
│ ├── settings
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── local.py
│ │ ├── production.py
│ │ └── test.py
│ ├── __init__.py
│ └── ...
├── requirements
├── utility
├── manage.py
└── ...
Log:
File "path\api\api\users\admin\__init__.py", line 1, in <module>
from .Profile import * # noqa
File "path\api\api\users\admin\Profile.py", line 3, in <module>
from users.models.Profile import Profile
File "path\api\api\users\models\__init__.py", line 1, in <module>
from .User import * # noqa
File "path\api\api\users\models\User.py", line 11, in <module>
class User(AbstractUser):
File "path\lib\site-packages\django\db\models\base.py", line 113, in __new__
raise RuntimeError(
RuntimeError: Model class users.models.User.User doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.
apps.py from users
from django.apps import AppConfig
from django.utils.translation import gettext_lazy as _
class UsersConfig(AppConfig):
name = "api.users"
verbose_name = _("Users")
def ready(self):
try:
import api.users.signals # noqa F401
except ImportError:
pass

Explicitly referencing the Users app configuration class might work:
LOCAL_APPS = [
"api.users.apps.UsersConfig",
...
]

variable defined in __init__.py can not be imported

I am trying to follow the instructions in http://flask.pocoo.org/docs/0.12/patterns/celery/ so I can perform flask/socketIO operations in celery tasks. My directory structure is a bit different however and I'm not having any luck with imports.
My directory structure is as follows:
├── app
│   ├── __init__.py
│   ├── __pycache__
│   ├── auth.py
│   ├── ctasks.py
│   ├── helper.py
│   ├── saml.py
│   ├── socks.py
│   ├── templates
│   ├── threads.py
│   └── views.py
├── app.py
├── config.py
├── requirements.txt
└── saml
   ├── dev
   └── prod
I call the app from app.py
from app import socketio, app
if __name__ == '__main__':
socketio.run(app, debug=True, port=443, ssl_context='adhoc')
__init__.py
from flask import Flask, request
from flask_socketio import SocketIO
from .ctasks import subtaskcaller, make_celery
from .helper import wait_to_finish
async_mode = None
app = Flask(__name__)
app.config.from_object('config')
socketio = SocketIO(app, async_mode=async_mode)
cel = make_celery(app)
from .auth import SamlManager
saml_manager = SamlManager()
saml_manager.init_app(app)
from app import views, socks, saml, helper, ctasks
ctasks.py
from celery import Celery
from config import *
from .helper import wait_to_finish, emitter
import time
from app import cel
def make_celery(app):
c = Celery(app.import_name, backend=CELERY_RESULT_BACKEND, broker=CELERY_BROKER_URL)
c.conf.update(app.config)
taskbase = c.Task
class ContextTask(taskbase):
abstract = True
def __call__(self, *args, **kwargs):
with app.app_context():
return taskbase.__call__(self, *args, **kwargs)
c.Task = ContextTask
return c
#cel.task(name='tasks.tester', serializer='pickle')
def tester():
emitter('emit from subsubtask')
for i in range(1, 50):
time.sleep(1)
print('test {0}'.format(i))
x = True
return x
#cel.task(name='task.subtaskcaller', serializer='pickle')
def subtaskcaller():
emitter('emit from subtask')
finished = tester.delay()
wait_to_finish(finished)
return finished
I am getting an error when trying to import cel from app in ctasks.py:
ImportError: cannot import name 'cel'

In your __init__.py you only have cel. You don't have an object called celery in your __init__ file, so it can't be imported to some other file. You can try from app import cel.˛
EDIT:
in __init__ you from .ctasks import subtaskcaller, make_celery
but in ctasks you import cel from app (which doesn't exist yet at that point, only Flask, request, and SocketIO exist at that point in time).
So you need to put your #cel decorated functions in yet another script, which you can import all the way at the bottom of __init__

Organizing a Flask project

This is my first time creating a project using python and flask. I intend to use SQLAlchemy models along too. and this is a fairly bigger project. As of now, I have divided the project in 2 Blueprints : site and the api. After organizing the project, I am confused as to how can I connnect these models with the database and do I need to re-organize the structure as I am not fully aware of nature of flask.
so this is the directory structure of the dir app/ in my base repository:
`
.
├── Blueprints
│   ├── __init__.py
│   ├── __pycache__
│   │   └── __init__.cpython-36.pyc
│   ├── api
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   ├── __init__.cpython-36.pyc
│   │   │   └── routes.cpython-36.pyc
│   │   └── routes.py
│   ├── config.py
│   └── site
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-36.pyc
│   │   └── routes.cpython-36.pyc
│   ├── operations.py
│   ├── routes.py
│   ├── static
│   └── templates
│   ├── about.html
│   ├── contact.html
│   ├── home.html
│   ├── login.html
│   ├── services.html
│   └── stories.html
├── __main__.py
├── __pycache__
│   └── __main__.cpython-36.pyc
└── models
├── Attendance.py
├── Batch.py
├── Course.py
├── Module.py
├── Student.py
├── Test.py
└── __init__.py
`
Please ignore Pycache, as this is auto generated.
now I cannot figure out a way as to how to import and use these models in api and site, neither I can understand as to how am I supposed to fetch the db object created in /Blueprints/__init__.py to all the models.
I understand that this question is not upto the standards of stack overflow questions, BUT I personally feel that organizing a flask project is itself very confusing with each tutorial or forum I see, having their own perspectives of organizing it.

There's several ways to organize a project, but the __init__.py file contained inside the app/ folder is what links a lot of it together. Here's the contents of one of my project's __init__.py file:
from werkzeug.contrib.fixers import ProxyFix
from flask import Flask, session
from app.config import (PERMANENT_SESSION_LIFETIME_MS, Time_Before_Warning,
Min_Ping_Interval)
import datetime
app = Flask(__name__)
app.wsgi_app = ProxyFix(app.wsgi_app)
# Setup the app with the config.py file
app.config.from_pyfile('config.py')
# Setup the logger
from app.logger_setup import logger, log_view
# Setup the database
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy(app)
#setup zipcode database
from pyzipcode import ZipCodeDatabase
zdb = ZipCodeDatabase()
# Setup the mail server
from flask.ext.mail import Mail
mail = Mail(app)
# Setup the debug toolbar
#from flask_debugtoolbar import DebugToolbarExtension
#app.config['DEBUG_TB_TEMPLATE_EDITOR_ENABLED'] = False
#app.config['DEBUG_TB_PROFILER_ENABLED'] = False
#toolbar = DebugToolbarExtension(app)
# Setup the password crypting
from flask.ext.bcrypt import Bcrypt
bcrypt = Bcrypt(app)
# Import the views
from app.views import (main, user, error, request, upload, dashboard, org,
msgs, notifications, download, reports,
direct_send,provider,utils)
app.register_blueprint(user.userbp)
app.register_blueprint(request.requestbp)
app.register_blueprint(upload.uploadbp)
app.register_blueprint(dashboard.dashboardbp)
app.register_blueprint(org.orgbp)
app.register_blueprint(msgs.msgbp)
app.register_blueprint(notifications.notificationsbp)
app.register_blueprint(download.downloadbp)
app.register_blueprint(reports.reportsbp)
app.register_blueprint(direct_send.directsendbp)
app.register_blueprint(provider.providerbp)
app.register_blueprint(utils.utilsbp)
# Setup the user login process
from flask.ext.login import LoginManager, current_user
from app.models import User, View
login_manager = LoginManager()
login_manager.init_app(app)
login_manager.login_view = 'userbp.signin'
#login_manager.user_loader
def load_user(email):
return User.query.filter(User.email == email).first()
from flask.ext.principal import identity_loaded, RoleNeed, UserNeed
#identity_loaded.connect_via(app)
def on_identity_loaded(sender, identity):
# Set the identity user object
identity.user = current_user
# Add the UserNeed to the identity
if hasattr(current_user, 'id'):
identity.provides.add(UserNeed(current_user.id))
# Assuming the User model has a list of roles, update the
# identity with the roles that the user provides
if hasattr(current_user, 'roles'):
identity.provides.add(RoleNeed(current_user.roles.type))
from flask.ext.principal import Principal
# load the extension
principals = Principal(app)
# Create a permission with a single Need, in this case a RoleNeed.
#from app import admin
#app.before_request
def make_session_permanent():
session.permanent = True
lt = PERMANENT_SESSION_LIFETIME_MS / (60*1000)
app.permanent_session_lifetime = datetime.timedelta(minutes=lt)
#app.context_processor
def add_session_config():
"""
Add current_app.permanent_session_lifetime converted to milliseconds
to context.
"""
return {
'PERMANENT_SESSION_LIFETIME_MS': PERMANENT_SESSION_LIFETIME_MS,
'Time_Before_Warning': Time_Before_Warning,
'Min_Ping_Interval': Min_Ping_Interval,
}
And then inside one of the blueprints:
from flask import (Blueprint, render_template, redirect, url_for,
abort, flash, request)
from flask.ext.login import login_required, current_user
from app import app, models, db, log_view, config
from app.models import (Groups, Organizations, OrgHasOwner, UserHasGroups,
GroupHasOwner, User, Fax, FavoriteGroups)
from app.forms import org as org_forms
from app.toolbox import email, misc, s3, fax
from sqlalchemy.sql import func
from werkzeug import secure_filename
from uuid import uuid4
import datetime
import string
import os
# Create a user blueprint
orgbp = Blueprint('orgbp', __name__, url_prefix='/org')
#orgbp.route('/invite_user', methods=['GET','POST'])
#login_required
def invite_user():
[stuff goes here]

Python function that similar to bash find command

I have a dir structure like the following:
[me#mypc]$ tree .
.
├── set01
│   ├── 01
│   │   ├── p1-001a.png
│   │   ├── p1-001b.png
│   │   ├── p1-001c.png
│   │   ├── p1-001d.png
│   │   └── p1-001e.png
│   ├── 02
│   │   ├── p2-001a.png
│   │   ├── p2-001b.png
│   │   ├── p2-001c.png
│   │   ├── p2-001d.png
│   │   └── p2-001e.png
I would like to write a python script to rename all *a.png to 01.png, *b.png to 02.png, and so on. Frist I guess I have to use something similar to find . -name '*.png', and the most similar thing I found in python was os.walk. However, in os.walk I have to check every file, if it's png, then I'll concatenate it with it's root, somehow not that elegant. I was wondering if there is a better way to do this? Thanks in advance.

For a search pattern like that, you can probably get away with glob.
from glob import glob
paths = glob('set01/*/*.png')

You can use os.walk to traverse the directory tree.
Maybe this works?
import os
for dpath, dnames, fnames in os.walk("."):
for i, fname in enumerate([os.path.join(dpath, fname) for fname in fnames]):
if fname.endswith(".png"):
#os.rename(fname, os.path.join(dpath, "%04d.png" % i))
print "mv %s %s" % (fname, os.path.join(dpath, "%04d.png" % i))
For Python 3.4+ you may want to use pathlib.glob() with a recursive pattern (e.g., **/*.png) instead:
Recursively iterate through all subdirectories using pathlib
https://docs.python.org/3/library/pathlib.html#pathlib.Path.glob
https://docs.python.org/3/library/pathlib.html#pathlib.Path.rglob

Check out genfind.py from David M. Beazley.
# genfind.py
#
# A function that generates files that match a given filename pattern
import os
import fnmatch
def gen_find(filepat,top):
for path, dirlist, filelist in os.walk(top):
for name in fnmatch.filter(filelist,filepat):
yield os.path.join(path,name)
# Example use
if __name__ == '__main__':
lognames = gen_find("access-log*","www")
for name in lognames:
print name

These days, pathlib is a convenient option.