Develop Reference

Celery with memory backend hanging

I'm developing a testing suite for a flask app using celery for processing background tasks.
I am working on integration tests and have been trying to configure a embedded live worker as per the documentation (https://docs.celeryproject.org/en/latest/userguide/testing.html)
conftest.py
#pytest.fixture(scope='session')
def celery_config():
return {
'broker_url': 'memory://localhost/',
'result_backend': 'memory://localhost/',
}
#pytest.fixture(scope='module')
def create_flask_app():
#drop all records in testDatabase before strting new test module
db = connect(host=os.environ["MONGODB_SETTINGS_TEST"], alias="testConnect")
for collection in db["testDatabase"].list_collection_names():
db["testDatabase"].drop_collection(collection)
db.close()
# Create a test client using the Flask application configured for testing
flask_app = create_app()
return flask_app
#pytest.fixture(scope='function')
def test_client(create_flask_app):
"""
Establish a test client for use within each test module
"""
with create_flask_app.test_client() as testing_client:
with create_flask_app.app_context():
yield testing_client
#pytest.fixture(scope='function')
def celery_app(create_flask_app):
from celery.contrib.testing import tasks
from app import celery
return celery
I'm trying to run the tests using local memory as the backend. Yet, the tasks hang and the test suite never finishes executing.
When I run the tests with a redis backend (and initialize redis in my development machine) everything works fine. But I'd like to not be dependent on redis when running the tests.
Am I doing something wrong with the setup? Does anyone have any idea on why the tasks are hanging?

Run pytest test suite against multiple database versions

I build an application that uses a database in the backend. For integration tests, I start the database in Docker and run a test suite with pytest.
I use a session scoped fixture with autouse=True to start the Docker container:
#pytest.fixture(scope='session', autouse=True)
def run_database():
# setup code skipped ...
# start container with docker-py
container.start()
# yield container to run tests
yield container
# stop container afterwards
container.stop()
I pass the database connection to the test functions with another session scoped fixture:
#pytest.fixture(scope='session')
def connection():
return Connection(...)
Now I can run a test function:
def test_something(connection):
result = connection.run(...)
assert result == 'abc'
However, I would like to run my test functions against multiple different versions of the database.
I could run multiple Docker containers in the run_database() fixture. How can I parametrize my test functions so that they run for two different connection() fixtures?

The answer by #Guy works!
I found another solution for the problem. It is possible to parametrize a fixture. Every test function that uses the fixture will run multiple times: https://docs.pytest.org/en/latest/fixture.html#parametrizing-fixtures
Thus, I parametrized the connection() function:
#pytest.fixture(scope='session', params=['url_1', 'url_2'])
def connection(request):
yield Connection(url=request.param)
Now every test function that uses the connection fixture runs twice. The advantage is that you do not have to change/adapt/mark the existing test functions.

You can send a function that yields connections and pass it with #pytest.mark.parametrize. If you change the the scope of run_database() to class it will run for every test
def data_provider():
connections = [Connection(1), Connection(2), Connection(3)]
for connection in connections:
yield connection
#pytest.fixture(scope='class', autouse=True)
def run_database():
container.start()
yield container
container.stop()
#pytest.mark.parametrize('connection', data_provider())
#pytest.mark.testing
def test_something(connection):
result = connection.run()
assert result == 'abc'
If you add #pytest.mark.parametrize('connection', data_provider()) to run_database() the connection will be passed to there as well.

Flask-SQLAlchemy integration tests can't find a way to rollback changes

I'm trying to learn flask technology stack and for my application I'm using Flask-SQLAlchemy. Everything works perfect, but I'm struggling with writing integration tests. I don't want to use SQLite since on production I'm using PostgreSQL and putting tons of mocks will actually more test my own implementation not the logic itself.
So, after some research I decided to implement tests that will write data in the test database and after each tests rollback the changes (for performance sake). Actually, I'm trying to implement something similar to this approach: http://sontek.net/blog/detail/writing-tests-for-pyramid-and-sqlalchemy.
My problem is creating correct transaction and being able to rollback it. Here is the code of my base class:
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy()
class MyAppIntegrationTestCase(unittest.TestCase):
#classmethod
def setUpClass(cls):
app.config['TESTING'] = True
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql+psycopg2:///db_test'
init_app()
db.app = app
db.create_all(app=app)
#classmethod
def tearDownClass(cls):
db.drop_all(app=app)
def setUp(self):
db.session.rollback()
self.trans = db.session.begin(subtransactions=True)
def tearDown(self):
self.trans.rollback()
When I'm trying to execute tests I got a following error:
Traceback (most recent call last):
File "myapp/src/core/tests/__init__.py", line 53, in tearDown
self.trans.rollback()
File "myapp/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 370, in rollback
self._assert_active(prepared_ok=True, rollback_ok=True)
File "myapp/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 203, in _assert_active
raise sa_exc.ResourceClosedError(closed_msg)
ResourceClosedError: This transaction is closed
I bet that this is the problem with scoped_session and that when I'm running tests it reuse one global session for all tests, but my knowledge in SQLAlchemy is not deep enough yet.
Any help will be highly appreciated!
Thanks!

You're tearDownClass and setUpClass are causing the issues.
The setUpClass is called once before all the tests, and the tearDownClass is after all the tests in the class.
So if you have 3 tests.
setUpClass is called
setUp is called
tearDown is called (You rollback, but you don't begin a session, this throws an error)
setUp is called (another rollback that's going to error)
etc...
Add a db.session.begin to your tearDown and you'll be fine.
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy()
class MyAppIntegrationTestCase(unittest.TestCase):
#classmethod
def setUpClass(cls):
app.config['TESTING'] = True
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql+psycopg2:///db_test'
init_app()
db.app = app
db.create_all(app=app)
#classmethod
def tearDownClass(cls):
db.drop_all(app=app)
def setUp(self):
db.session.rollback()
self.trans = db.session.begin(subtransactions=True)
def tearDown(self):
self.trans.rollback()
db.session.begin()

I wrote a blog post on how to set this up... in short, you have to create a nested transaction so that any session.commit() calls inside your application don't break your isolation. Then apply a listener to the inner transaction to restart it anytime someone tries to commit it or roll it back. Setup Flask-Sqlalchemy Transaction Test Case

A possible solution to your question:
If data size of your database is not very large and you want keep data unchanged, you can do a backup (by write straight sql sentenses) in set up
"CREATE TABLE {0}_backup SELECT * FROM {0}".format(table_name)
and do recover in teardown
"DROP TABLE {0}".format(table_name)
"RENAME TABLE {0}_backup TO {0}".format(table_name)

Python: block network connections for testing purposes?

I'm trying to test a package that provides interfaces to a few web services. It has a test suite that is supposed to test most functions without connecting to the internet. However, there are some lingering tests that may attempt to connect to the internet / download data, and I'd like to prevent them from doing so for two reasons: first, to make sure my test suite works if no network connection is available; second, so that I'm not spamming the web services with excess queries.
An obvious solution is to unplug my machine / turn off wireless, but when I'm running tests on a remote machine that obviously doesn't work.
So, my question: Can I block network / port access for a single python process? ("sandbox" it, but just blocking network connections)
(afaict, pysandbox doesn't do this)
EDIT: I'm using py.test so I need a solution that will work with py.test, in case that affects any proposed answers.

Monkey patching socket ought to do it:
import socket
def guard(*args, **kwargs):
raise Exception("I told you not to use the Internet!")
socket.socket = guard
Make sure this runs before any other import.

Update: There is now a pytest plugin that does the same thing as this answer! You can read the answer just to see how things work, but I strongly recommend using the plugin instead of copying-pasting my answer :-) See here: https://github.com/miketheman/pytest-socket
I found Thomas Orozco's answer to be very helpful. Following on keflavich, this is how I integrated into my unit test suite. This works for me with thousands of very different unit test-cases (<100 that need socket though) ... and in and out of doctests.
I posted it here. Including below for convenience. Tested with Python 2.7.5, pytest==2.7.0. (To test for yourself, run py.test --doctest-modules in directory with all 3 files cloned.)
_socket_toggle.py
from __future__ import print_function
import socket
import sys
_module = sys.modules[__name__]
def disable_socket():
""" disable socket.socket to disable the Internet. useful in testing.
.. doctest::
>>> enable_socket()
[!] socket.socket is enabled.
>>> disable_socket()
[!] socket.socket is disabled. Welcome to the desert of the real.
>>> socket.socket(socket.AF_INET, socket.SOCK_STREAM)
Traceback (most recent call last):
...
RuntimeError: I told you not to use the Internet!
>>> enable_socket()
[!] socket.socket is enabled.
>>> enable_socket()
[!] socket.socket is enabled.
>>> disable_socket()
[!] socket.socket is disabled. Welcome to the desert of the real.
>>> socket.socket(socket.AF_INET, socket.SOCK_STREAM)
Traceback (most recent call last):
...
RuntimeError: I told you not to use the Internet!
>>> enable_socket()
[!] socket.socket is enabled.
"""
setattr(_module, '_socket_disabled', True)
def guarded(*args, **kwargs):
if getattr(_module, '_socket_disabled', False):
raise RuntimeError("I told you not to use the Internet!")
else:
# SocketType is a valid public alias of socket.socket,
# we use it here to avoid namespace collisions
return socket.SocketType(*args, **kwargs)
socket.socket = guarded
print(u'[!] socket.socket is disabled. Welcome to the desert of the real.')
def enable_socket():
""" re-enable socket.socket to enable the Internet. useful in testing.
"""
setattr(_module, '_socket_disabled', False)
print(u'[!] socket.socket is enabled.')
conftest.py
# Put this in the conftest.py at the top of your unit tests folder,
# so it's available to all unit tests
import pytest
import _socket_toggle
def pytest_runtest_setup():
""" disable the interet. test-cases can explicitly re-enable """
_socket_toggle.disable_socket()
#pytest.fixture(scope='function')
def enable_socket(request):
""" re-enable socket.socket for duration of this test function """
_socket_toggle.enable_socket()
request.addfinalizer(_socket_toggle.disable_socket)
test_example.py
# Example usage of the py.test fixture in tests
import socket
import pytest
try:
from urllib2 import urlopen
except ImportError:
import urllib3
urlopen = urllib.request.urlopen
def test_socket_disabled_by_default():
# default behavior: socket.socket is unusable
with pytest.raises(RuntimeError):
urlopen(u'https://www.python.org/')
def test_explicitly_enable_socket(enable_socket):
# socket is enabled by pytest fixture from conftest. disabled in finalizer
assert socket.socket(socket.AF_INET, socket.SOCK_STREAM)

Building on the very helpful answers from Thomas Orozco and driftcatcher here is a variant that works with Python's unittest and (after a small change) Django.
All you need to do is inherit your test case class from the enhanced NoSocketTestCase class and any access to the network will be detected and raises the SocketAccessError exception.
And this approach also works with Django. You only need to change the NoSocketTestCase class to inherit from django.test.TestCase instead of unittest.TestCase.
While not strictly answering OP's question I think this might be helpful for anyone who wants to block network access in unit tests.
no_sockets.py
import socket
from unittest import TestCase
class SocketAccessError(Exception):
pass
class NoSocketsTestCase(TestCase):
"""Enhancement of TestCase class that prevents any use of sockets
Will throw the exception SocketAccessError when any code tries to
access network sockets
"""
#classmethod
def setUpClass(cls):
cls.socket_original = socket.socket
socket.socket = cls.guard
return super().setUpClass()
#classmethod
def tearDownClass(cls):
socket.socket = cls.socket_original
return super().tearDownClass()
#staticmethod
def guard(*args, **kwargs):
raise SocketAccessError('Attempted to access network')
test_no_sockets.py
import urllib.request
from .no_sockets import NoSocketsTestCase, SocketAccessError
class TestNoSocketsTestCase(NoSocketsTestCase):
def test_raises_exception_on_attempted_network_access(self):
with self.assertRaises(SocketAccessError):
urllib.request.urlopen('https://www.google.com')

A simple way to put a gag on the requests library:
from unittest import mock
requests_gag = mock.patch(
'requests.Session.request',
mock.Mock(side_effect=RuntimeError(
'Please use the `responses` library to mock HTTP in your tests.'
))
)
with requests_gag:
... # no Internet here

httpretty is a small library that solves this problem.
If you are using Django test runner, write a custom test runner where you disable all 3rd party API calls.
# common/test_runner.py
import httpretty
from django.test.runner import DiscoverRunner
class CustomTestRunner(DiscoverRunner):
def run_tests(self, *args, **kwargs):
with httpretty.enabled(allow_net_connect=False):
return super().run_tests(*args, **kwargs)
add this new test runner to your settings
TEST_RUNNER = "common.test_runner.CustomTestRunner"
And from now on all external API calls have to be mocked or httpretty.errors.UnmockedError will be raised.
If you are using pytest, this fixture should work.
#pytest.fixture
def disable_external_api_calls():
httpretty.enable()
yield
httpretty.disable()

I have a pytest solution. pytest-network lybrary help me on this.
# conftest.py
import pytest
import socket
_original_connect = socket.socket.connect
def patched_connect(*args, **kwargs):
...
# It depends on your testing purpose
# You may want a exception, add here
# If you test unconnectable situations
# it can stay like this
#pytest.fixture
def enable_network():
socket.socket.connect = _original_connect
yield
socket.socket.connect = patched_connect
#pytest.fixture
def disable_network():
socket.socket.connect = patched_connect
yield
socket.socket.connect = _original_connect
# test_internet.py
def test_your_unconnectable_situation(disable_network):
response = request.get('http://stackoverflow.com/')
response.status_code == 400

Best practice of testing django-rq ( python-rq ) in Django

I'll start using django-rq in my project.
Django integration with RQ, a Redis based Python queuing library.
What is the best practice of testing django apps which is using RQ?
For example, if I want to test my app as a black box, after User makes some actions I want to execute all jobs in current Queue, and then check all results in my DB. How can I do it in my django-tests?

I just found django-rq, which allows you to spin up a worker in a test environment that executes any tasks on the queue and then quits.
from django.test impor TestCase
from django_rq import get_worker
class MyTest(TestCase):
def test_something_that_creates_jobs(self):
... # Stuff that init jobs.
get_worker().work(burst=True) # Processes all jobs then stop.
... # Asserts that the job stuff is done.

I separated my rq tests into a few pieces.
Test that I'm correctly adding things to the queue (using mocks).
Assume that if something gets added to the queue, it will eventually be processed. (rq's test suite should cover this).
Test, given the correct input, my tasks work as expected. (normal code tests).
Code being tested:
def handle(self, *args, **options):
uid = options.get('user_id')
# ### Need to exclude out users who have gotten an email within $window
# days.
if uid is None:
uids = User.objects.filter(is_active=True, userprofile__waitlisted=False).values_list('id', flat=True)
else:
uids = [uid]
q = rq.Queue(connection=redis.Redis())
for user_id in uids:
q.enqueue(mail_user, user_id)
My tests:
class DjangoMailUsersTest(DjangoTestCase):
def setUp(self):
self.cmd = MailUserCommand()
#patch('redis.Redis')
#patch('rq.Queue')
def test_no_userid_queues_all_userids(self, queue, _):
u1 = UserF.create(userprofile__waitlisted=False)
u2 = UserF.create(userprofile__waitlisted=False)
self.cmd.handle()
self.assertItemsEqual(queue.return_value.enqueue.mock_calls,
[call(ANY, u1.pk), call(ANY, u2.pk)])
#patch('redis.Redis')
#patch('rq.Queue')
def test_waitlisted_people_excluded(self, queue, _):
u1 = UserF.create(userprofile__waitlisted=False)
UserF.create(userprofile__waitlisted=True)
self.cmd.handle()
self.assertItemsEqual(queue.return_value.enqueue.mock_calls, [call(ANY, u1.pk)])

I commited a patch that lets you do:
from django.test impor TestCase
from django_rq import get_queue
class MyTest(TestCase):
def test_something_that_creates_jobs(self):
queue = get_queue(async=False)
queue.enqueue(func) # func will be executed right away
# Test for job completion
This should make testing RQ jobs easier. Hope that helps!

Just in case this would be helpful to anyone. I used a patch with a custom mock object to do the enqueue that would run right away
#patch django_rq.get_queue
with patch('django_rq.get_queue', return_value=MockBulkJobGetQueue()) as mock_django_rq_get_queue:
#Perform web operation that starts job. In my case a post to a url
Then the mock object just had one method:
class MockBulkJobGetQueue(object):
def enqueue(self, f, *args, **kwargs):
# Call the function
f(
**kwargs.pop('kwargs', None)
)

what I've done for this case is to detect if I'm testing, and use fakeredis during tests. finally, in the test itself, I enqueue the redis worker task in synch mode:
first, define a function that detects if you're testing:
TESTING = len(sys.argv) > 1 and sys.argv[1] == 'test'
def am_testing():
return TESTING
then in your file that uses redis to queue up tasks, manage the queue this way.
you could extend get_queue to specify a queue name if needed:
if am_testing():
from fakeredis import FakeStrictRedis
from rq import Queue
def get_queue():
return Queue(connection=FakeStrictRedis())
else:
import django_rq
def get_queue():
return django_rq.get_queue()
then, enqueue your task like so:
queue = get_queue()
queue.enqueue(task_mytask, arg1, arg2)
finally, in your test program, run the task you are testing in synch mode, so that it runs in the same process as your test. As a matter of practice, I first clear the fakeredis queue, but I don't think its necessary since there are no workers:
from rq import Queue
from fakeredis import FakeStrictRedis
FakeStrictRedis().flushall()
queue = Queue(async=False, connection=FakeStrictRedis())
queue.enqueue(task_mytask, arg1, arg2)
my settings.py has the normal django_redis settings, so django_rq.getqueue() uses these when deployed:
RQ_QUEUES = {
'default': {
'HOST': env_var('REDIS_HOST'),
'PORT': 6379,
'DB': 0,
# 'PASSWORD': 'some-password',
'DEFAULT_TIMEOUT': 360,
},
'high': {
'HOST': env_var('REDIS_HOST'),
'PORT': 6379,
'DB': 0,
'DEFAULT_TIMEOUT': 500,
},
'low': {
'HOST': env_var('REDIS_HOST'),
'PORT': 6379,
'DB': 0,
}
}

None of the answers above really solved how to test without having redis installed and using django settings. I found including the following code in the tests does not impact the project itself yet gives everything needed.
The code uses fakeredis to pretend there is a Redis service available, set up the connection before RQ Django reads the settings.
The connection must be the same because in fakeredis connections
do not share the state. Therefore, it is a singleton object to reuse it.
from fakeredis import FakeStrictRedis, FakeRedis
class FakeRedisConn:
"""Singleton FakeRedis connection."""
def __init__(self):
self.conn = None
def __call__(self, _, strict):
if not self.conn:
self.conn = FakeStrictRedis() if strict else FakeRedis()
return self.conn
django_rq.queues.get_redis_connection = FakeRedisConn()
def test_case():
...

You'll need your tests to pause while there are still jobs in the queue. To do this, you can check Queue.is_empty(), and suspend execution if there are still jobs in the queue:
import time
from django.utils.unittest import TestCase
import django_rq
class TestQueue(TestCase):
def test_something(self):
# simulate some User actions which will queue up some tasks
# Wait for the queued tasks to run
queue = django_rq.get_queue('default')
while not queue.is_empty():
time.sleep(5) # adjust this depending on how long your tasks take to execute
# queued tasks are done, check state of the DB
self.assert(.....)

I came across the same issue. In addition, I executed in my Jobs e.g. some mailing functionality and then wanted to check the Django test mailbox if there were any E-Mail. However, since the with Django RQ the jobs are not executed in the same context as the Django test, the emails that are sent do not end up in the test mailbox.
Therefore I need to execute the Jobs in the same context. This can be achieved by:
from django_rq import get_queue
queue = get_queue('default')
queue.enqueue(some_job_callable)
# execute input watcher
jobs = queue.get_jobs()
# execute in the same context as test
while jobs:
for job in jobs:
queue.remove(job)
job.perform()
jobs = queue.get_jobs()
# check no jobs left in queue
assert not jobs
Here you just get all the jobs from the queue and execute them directly in the test. One can nicely implement this in a TestCase Class and reuse this functionality.