I'm writing unit tests for a codebase. My issue is I need to patch a database abstraction class but it is masked by the __init__.py file. My file structure is:
.
├── __init__.py
├── tools
│ ├── __init__.py
│ ├── test
│ │ ├── __init__.py
│ │ └── test_tool1.py
│ ├── tool1.py
│ └── tool2.py
└── utils
├── __init__.py
└── sql_client.py
Now the content of the files:
# tools/__init__.py
from .tool1 import * # `SQLClient` is imported here as part of tool1
from .tool2 import * # `SQLClient` is imported here again but now it's part of tool2
# tools/tool1.py
from utils import SQLClient
class A(object):
...
def run(self, **kwargs):
# the function I want to test
sql = SQLClient("some id")
# tools/tool2.py
from utils import SQLClient
...
# utils/__init__.py
from sql_client import *
# utils/sql_client.py
class SQLClient(object):
# The sql abstraction class that needs to be patched
In my test file, I'm creating a mock class to be used as the patch. I'm using absolute imports in my tests because later I want to move all the tests outside of the source folders.
# tools/test/test_tool1.py
from unittest.mock import MagicMock
from utils import SQLClient
from tools import A
class MockSQLClient(MagicMock):
def __init__(self, *args, **kwargs):
super().__init__(spec=SQLClient)
self._mocks = {"select *": "rows"}
def make_query(query)
return self._mocks[query]
def test_run_func(monkeypatch):
monkeypatch.setattr("tools.SQLClient", MockSQLClient)
a = A()
a.run()
# rest of the test
Now, the issue is that tools/__init__.py imports everything from all the submodules so SQLClient from tool1 is being masked by SQLClient from tool2. As a result, my monkeypatch is patching tool2.SQLClient. I cannot patch the tool1 part directly by monkeypatch.setattr("tools.tool1.SQLClient") because of tools/__init__.py which throws an error complaining there is no tool1 in tools.
EDIT
changed question title, more detail on where the test is.
Related
I have a Class that is importing many methods from submodules. Currently Sphinx is setup to organize 'bysource' so they're at least sorting by the order in which the submodules are imported.
What I would like though, is some kind of header or searchable text for the title of the file they come from.
Current Directory Structure:
my_project
├── setup.py
└── my_package
├── __init__.py # Classes for subpackage_1 and subpackage_2 and imports occur here
├── subpackage_1
│ ├── __init__.py
│ ├── _submodule_a.py
│ └── _submodule_b.py
└── subpackage_2
├── __init__.py
├── _submodule_c.py
└── _submodule_d.py
Sphinx module rst file:
Module contents
---------------
.. automodule:: my_package
:members:
:undoc-members:
:show-inheritance:
:member-order: bysource
In the my_package init.py there are there parent Classes defined where all the submodules/methods are imported to their related Class.
class MyClass_1():
...
from .subpackage_1._submodule_a import method_a
from .subpackage_1._submodule_b import method_b, method_c
class MyClass_2():
...
from .subpackage_2._submodule_c import method_d, method_e
from .subpackage_2._submodule_d import method_f
In the resulting Sphinx method documentation I see the methods under each class, but the desire is to be able to see which submodule file the methods sourced from. Doesn't have to be a subsection, merely including a header/note etc. would be nice without having to resort to manually listing all the modlues/methods in the Sphinx file.
There's hundreds of functions in the real package so it would be helpful for the user to be able to quickly discern what submodule file a method came from when viewing the documentation.
Current Sphinx Output:
Desired Sphinx Output:
Class MyClass_1...
Class MyClass_1...
submodule_a
* method_a
* method_a
submodule_b
* method_b
* method_b
* method_c
* method_c
Class MyClass_2...
Class MyClass_2...
submodule_c
* method_d
* method_d
* method_d
* method_e
submodule_d
* method_f
* method_f
I see worse case scenario, putting the submodule filename in docstring for each method within the file, but would love if someone's figured this out in a more automated fashion
I have the following directory and file structure in my current directory:
├── alpha
│ ├── A.py
│ ├── B.py
│ ├── Base.py
│ ├── C.py
│ └── __init__.py
└── main.py
Each file under the alpha/ directory is it's own class and each of those classes inheirts the Base class in Base.py. Right now, I can do something like this in main.py:
from alpha.A import *
from alpha.B import *
from alpha.C import *
A()
B()
C()
And it works fine. However, if I wanted to add a file and class "D" and then use D() in main.py, I'd have to go into my main.py and do "from alpha.D import *". Is there anyway to do an import in my main file so that it imports EVERYTHING under the alpha directory?
depens what you are trying to do with the objects, one possible solution could be:
import importlib
import os
for file in os.listdir("alpha"):
if file.endswith(".py") and not file.startswith("_") and not file.startswith("Base"):
class_name = os.path.splitext(file)[0]
module_name = "alpha" + '.' + class_name
loaded_module = importlib.import_module(module_name)
loaded_class = getattr(loaded_module, class_name)
class_instance = loaded_class()
Importing everything with * is not a good practice, so if your files have only one class, importing this class is "cleaner" ( class_name is in your case)
I am trying to follow the instructions in http://flask.pocoo.org/docs/0.12/patterns/celery/ so I can perform flask/socketIO operations in celery tasks. My directory structure is a bit different however and I'm not having any luck with imports.
My directory structure is as follows:
├── app
│ ├── __init__.py
│ ├── __pycache__
│ ├── auth.py
│ ├── ctasks.py
│ ├── helper.py
│ ├── saml.py
│ ├── socks.py
│ ├── templates
│ ├── threads.py
│ └── views.py
├── app.py
├── config.py
├── requirements.txt
└── saml
├── dev
└── prod
I call the app from app.py
from app import socketio, app
if __name__ == '__main__':
socketio.run(app, debug=True, port=443, ssl_context='adhoc')
__init__.py
from flask import Flask, request
from flask_socketio import SocketIO
from .ctasks import subtaskcaller, make_celery
from .helper import wait_to_finish
async_mode = None
app = Flask(__name__)
app.config.from_object('config')
socketio = SocketIO(app, async_mode=async_mode)
cel = make_celery(app)
from .auth import SamlManager
saml_manager = SamlManager()
saml_manager.init_app(app)
from app import views, socks, saml, helper, ctasks
ctasks.py
from celery import Celery
from config import *
from .helper import wait_to_finish, emitter
import time
from app import cel
def make_celery(app):
c = Celery(app.import_name, backend=CELERY_RESULT_BACKEND, broker=CELERY_BROKER_URL)
c.conf.update(app.config)
taskbase = c.Task
class ContextTask(taskbase):
abstract = True
def __call__(self, *args, **kwargs):
with app.app_context():
return taskbase.__call__(self, *args, **kwargs)
c.Task = ContextTask
return c
#cel.task(name='tasks.tester', serializer='pickle')
def tester():
emitter('emit from subsubtask')
for i in range(1, 50):
time.sleep(1)
print('test {0}'.format(i))
x = True
return x
#cel.task(name='task.subtaskcaller', serializer='pickle')
def subtaskcaller():
emitter('emit from subtask')
finished = tester.delay()
wait_to_finish(finished)
return finished
I am getting an error when trying to import cel from app in ctasks.py:
ImportError: cannot import name 'cel'
In your __init__.py you only have cel. You don't have an object called celery in your __init__ file, so it can't be imported to some other file. You can try from app import cel.˛
EDIT:
in __init__ you from .ctasks import subtaskcaller, make_celery
but in ctasks you import cel from app (which doesn't exist yet at that point, only Flask, request, and SocketIO exist at that point in time).
So you need to put your #cel decorated functions in yet another script, which you can import all the way at the bottom of __init__
This is my first time creating a project using python and flask. I intend to use SQLAlchemy models along too. and this is a fairly bigger project. As of now, I have divided the project in 2 Blueprints : site and the api. After organizing the project, I am confused as to how can I connnect these models with the database and do I need to re-organize the structure as I am not fully aware of nature of flask.
so this is the directory structure of the dir app/ in my base repository:
`
.
├── Blueprints
│ ├── __init__.py
│ ├── __pycache__
│ │ └── __init__.cpython-36.pyc
│ ├── api
│ │ ├── __init__.py
│ │ ├── __pycache__
│ │ │ ├── __init__.cpython-36.pyc
│ │ │ └── routes.cpython-36.pyc
│ │ └── routes.py
│ ├── config.py
│ └── site
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── __init__.cpython-36.pyc
│ │ └── routes.cpython-36.pyc
│ ├── operations.py
│ ├── routes.py
│ ├── static
│ └── templates
│ ├── about.html
│ ├── contact.html
│ ├── home.html
│ ├── login.html
│ ├── services.html
│ └── stories.html
├── __main__.py
├── __pycache__
│ └── __main__.cpython-36.pyc
└── models
├── Attendance.py
├── Batch.py
├── Course.py
├── Module.py
├── Student.py
├── Test.py
└── __init__.py
`
Please ignore Pycache, as this is auto generated.
now I cannot figure out a way as to how to import and use these models in api and site, neither I can understand as to how am I supposed to fetch the db object created in /Blueprints/__init__.py to all the models.
I understand that this question is not upto the standards of stack overflow questions, BUT I personally feel that organizing a flask project is itself very confusing with each tutorial or forum I see, having their own perspectives of organizing it.
There's several ways to organize a project, but the __init__.py file contained inside the app/ folder is what links a lot of it together. Here's the contents of one of my project's __init__.py file:
from werkzeug.contrib.fixers import ProxyFix
from flask import Flask, session
from app.config import (PERMANENT_SESSION_LIFETIME_MS, Time_Before_Warning,
Min_Ping_Interval)
import datetime
app = Flask(__name__)
app.wsgi_app = ProxyFix(app.wsgi_app)
# Setup the app with the config.py file
app.config.from_pyfile('config.py')
# Setup the logger
from app.logger_setup import logger, log_view
# Setup the database
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy(app)
#setup zipcode database
from pyzipcode import ZipCodeDatabase
zdb = ZipCodeDatabase()
# Setup the mail server
from flask.ext.mail import Mail
mail = Mail(app)
# Setup the debug toolbar
#from flask_debugtoolbar import DebugToolbarExtension
#app.config['DEBUG_TB_TEMPLATE_EDITOR_ENABLED'] = False
#app.config['DEBUG_TB_PROFILER_ENABLED'] = False
#toolbar = DebugToolbarExtension(app)
# Setup the password crypting
from flask.ext.bcrypt import Bcrypt
bcrypt = Bcrypt(app)
# Import the views
from app.views import (main, user, error, request, upload, dashboard, org,
msgs, notifications, download, reports,
direct_send,provider,utils)
app.register_blueprint(user.userbp)
app.register_blueprint(request.requestbp)
app.register_blueprint(upload.uploadbp)
app.register_blueprint(dashboard.dashboardbp)
app.register_blueprint(org.orgbp)
app.register_blueprint(msgs.msgbp)
app.register_blueprint(notifications.notificationsbp)
app.register_blueprint(download.downloadbp)
app.register_blueprint(reports.reportsbp)
app.register_blueprint(direct_send.directsendbp)
app.register_blueprint(provider.providerbp)
app.register_blueprint(utils.utilsbp)
# Setup the user login process
from flask.ext.login import LoginManager, current_user
from app.models import User, View
login_manager = LoginManager()
login_manager.init_app(app)
login_manager.login_view = 'userbp.signin'
#login_manager.user_loader
def load_user(email):
return User.query.filter(User.email == email).first()
from flask.ext.principal import identity_loaded, RoleNeed, UserNeed
#identity_loaded.connect_via(app)
def on_identity_loaded(sender, identity):
# Set the identity user object
identity.user = current_user
# Add the UserNeed to the identity
if hasattr(current_user, 'id'):
identity.provides.add(UserNeed(current_user.id))
# Assuming the User model has a list of roles, update the
# identity with the roles that the user provides
if hasattr(current_user, 'roles'):
identity.provides.add(RoleNeed(current_user.roles.type))
from flask.ext.principal import Principal
# load the extension
principals = Principal(app)
# Create a permission with a single Need, in this case a RoleNeed.
#from app import admin
#app.before_request
def make_session_permanent():
session.permanent = True
lt = PERMANENT_SESSION_LIFETIME_MS / (60*1000)
app.permanent_session_lifetime = datetime.timedelta(minutes=lt)
#app.context_processor
def add_session_config():
"""
Add current_app.permanent_session_lifetime converted to milliseconds
to context.
"""
return {
'PERMANENT_SESSION_LIFETIME_MS': PERMANENT_SESSION_LIFETIME_MS,
'Time_Before_Warning': Time_Before_Warning,
'Min_Ping_Interval': Min_Ping_Interval,
}
And then inside one of the blueprints:
from flask import (Blueprint, render_template, redirect, url_for,
abort, flash, request)
from flask.ext.login import login_required, current_user
from app import app, models, db, log_view, config
from app.models import (Groups, Organizations, OrgHasOwner, UserHasGroups,
GroupHasOwner, User, Fax, FavoriteGroups)
from app.forms import org as org_forms
from app.toolbox import email, misc, s3, fax
from sqlalchemy.sql import func
from werkzeug import secure_filename
from uuid import uuid4
import datetime
import string
import os
# Create a user blueprint
orgbp = Blueprint('orgbp', __name__, url_prefix='/org')
#orgbp.route('/invite_user', methods=['GET','POST'])
#login_required
def invite_user():
[stuff goes here]
I am planning to split my one big test file to smaller test based on the part of code it tests. And I have a custom assert function for one of my tests. If I split them out to a separate file, how should I import them to other test files.
TestSchemacase:
class TestSchemaCase(unittest.TestCase):
"""
This will test our schema against the JSONTransformer output
just to make sure the schema matches the model
"""
# pylint: disable=too-many-public-methods
_base_dir = os.path.realpath(os.path.dirname(__file__))
def assertJSONValidates(self, schema, data):
"""
This function asserts the validation works as expected
Args:
schema(dict): The schema to test against
data(dict): The data to validate using the schema
"""
# pylint: disable=invalid-name
validator = jsonschema.Draft4Validator(schema)
self.assertIsNone(validator.validate(data))
def assertJSONValidateFails(self, schema, data):
"""
This function will assertRaises an ValidationError Exception
is raised.
Args:
schema(dict): The schema to validate from
data(dict): The data to validate using the schema
"""
# pylint: disable=invalid-name
validator = jsonschema.Draft4Validator(schema)
with self.assertRaises(jsonschema.ValidationError):
validator.validate(data)
My question is,1. When I try to import them I get an Import error with No module name found. I am breaking the TestValidation to mentioned small files. 2. I know I can raise Validation error in the assertJSONValidateFails but what should I return in case of validation pass.
tests/schema
├── TestSchemaCase.py
├── TestValidation.py
├── __init__.py
└── models
├── Fields
│ ├── TestImplemen.py
│ ├── TestRes.py
│ └── __init__.py
├── Values
│ ├── TestInk.py
│ ├── TestAlue.py
│ └── __init__.py
└── __init__.py
3.And is this how we should inherit them?
class TestRes(unittest.TestCase, TestSchemaCase):
Thanks for your time. Sorry for the big post
I did see the post, But that doesn't solve the problem.
I would suggest using a test framework that doesn't force you to put your tests in classes, like pytest.