Develop Reference

Fixture found but not called

I am testing a bot in python-telegram-bot using pyrogram to simulate inputs. I have to set a cryptography key in my environment so the code being tested can access it. I tried to accomplish this using the setup-teardown concept inside pytest.fixtures.
For that, i created a file where all my fixtures are created, and one of them is set like #pytest.fixture(autouse=True, scope="session")
The fixture is being found, but not executed. Why?
File structure:
--root
|-tests/
|- conftest.py
|- test_something1.py
|- test_something2.py
I am executing pytest -s --fixtures from the root folder
Here is conftest.py
# coding:utf-8
import os
import pytest
from pyrogram import Client
from telegram.ext import Application
from config import BOT_ID
from contrib.consts import CRYPT_KEY_EDK
from tests.config import TESTER_USERNAME, API_ID, API_HASH, CRYPT_KEY
#pytest.fixture
async def bot():
async with Application.builder().token(BOT_ID).build() as bot:
yield bot
#pytest.fixture
async def pyro_client():
async with Client(name=TESTER_USERNAME, api_id=API_ID, api_hash=API_HASH) as pyro_client:
yield pyro_client
#pytest.fixture(autouse=True, scope="session")
def set_crypt_key():
print("Entered set_crypt_key fixture")
os.environ[CRYPT_KEY_EDK] = CRYPT_KEY
And here is 'test_can_access_netbox.py':
# coding: utf-8
from unittest import mock
import pynetbox
import pytest
import requests
from pynetbox import RequestError
from telegram.ext import ContextTypes
from contrib.consts import NETBOX_CONFIG_CDK
from contrib.datawrappers.netboxconfig import NetboxConfig
from tests.config import NETBOX_TEST_TOKEN, NETBOX_TEST_URL
chat_data_mock = {
NETBOX_CONFIG_CDK: NetboxConfig(token=NETBOX_TEST_TOKEN, url=NETBOX_TEST_URL)
}
#mock.patch("telegram.ext.ContextTypes.DEFAULT_TYPE", chat_data=chat_data_mock)
def test_can_access_netbox(context: ContextTypes.DEFAULT_TYPE):
"""TC002"""
try:
nb = pynetbox.api(**context.chat_data[NETBOX_CONFIG_CDK].to_dict())
nb.status()
except RequestError as e:
pytest.fail("Error requesting connection to Netbox: {}".format(e))
except requests.exceptions.ConnectionError as e:
pytest.fail("Connection error: {}".format(e))
if __name__ == '__main__':
test_can_access_netbox()
And when i run my tests with pytest -s --fixtures, i get this error:
------------------------------------------------------------- fixtures defined from tests.fixtures -------------------------------------------------------------
bot -- tests/fixtures.py:19
pyro_client -- tests/fixtures.py:29
set_crypt_key [session scope] -- tests/fixtures.py:39
============================================================================ ERRORS ============================================================================
_______________________________________________________ ERROR collecting tests/test_can_access_netbox.py _______________________________________________________
tests/test_can_access_netbox.py:16: in <module>
NETBOX_CONFIG_CDK: NetboxConfig(token=NETBOX_TEST_TOKEN, url=NETBOX_TEST_URL)
contrib/datawrappers/netboxconfig.py:8: in __init__
super().__init__()
contrib/classesbehaviors/cryptographs.py:13: in __init__
self.crypt = Fernet(os.environ[CRYPT_KEY_EDK].encode())
/usr/lib/python3.8/os.py:675: in __getitem__
raise KeyError(key) from None
E KeyError: 'TELOPS_CRYPT_KEY'
=================================================================== short test summary info ====================================================================
ERROR tests/test_can_access_netbox.py - KeyError: 'TELOPS_CRYPT_KEY'
======================================================================= 1 error in 0.36s =======================================================================
Notice that my print() did not appear, despite of me using -s on command.
Libs i am using:
pytest==7.2.1
pytest-asyncio==0.20.3
Python is 3.8.10
Here are the first lines of the command output, i hope they help:
platform linux -- Python 3.8.10, pytest-7.2.1, pluggy-1.0.0
rootdir: /home/joao/Desktop/enterprise/project_name
plugins: asyncio-0.20.3, anyio-3.6.2
asyncio: mode=strict

The error messages you pasted don't quite match the snippets you included, so it is hard to see the precise reason for the failures.
For example, if we look at the confest.py snippet you included, it will definitely fail, but with a different exception - NameError because it will not be able to resolve the names CRYPT_KEY_EDK or CRYPT_KEY values.
Secondly the error message talks about a file test_can_access_netbox.py for which we cannot see here.
This being said, here is a simple example to show how one can use a fixture to set an environment variable:
constants.py
CRYPT_KEY_EDK = "CRYPT-KEY-EDK-VALUE"
CRYPT_KEY = "SOME-CRYPT-KEY-VALUE"
test_fixture_executed.py
import pytest
import os
import constants
#pytest.mark.asyncio
async def test_env_var_is_present():
assert constants.CRYPT_KEY_EDK in os.environ
conftest.py
import pytest
import os
import constants
#pytest.fixture(autouse=True, scope="session")
def set_crypt_key():
print("Entered set_crypt_key fixture")
os.environ[CRYPT_KEY_EDK] = CRYPT_KEY

Okay, i figured it out (thanks to willcode.io asking for the test file):
In my test file i had some lines out of any function. The problem was: they were being executed when pytest was trying to fetch all the tests. But these lines needed to be executed after the fixtures.
Solution: As these floating lines were doing some setup process, i just inserted them into a fixture and everything is working now.

pytest fixtures: testing pandas dataframe

I have some scripts in package directory and some tests in tests directory, along with a CSV file containing a dataframe that i want to use for testing purposes.
main_directory/
|
|- package/
| |- foo.py
| |- bar.py
|
|- tests/
|- conftest.py
|- test1.py
|- test.csv
I am using pytest and i have defined a conftest.py that contains a fixture that i want to use for the whole test session, that should return a pandas test dataframe imported from a csv file, as in the following:
#conftest.py
import pytest
from pandas import read_csv
path="test.csv"
#pytest.fixture(scope="session")
def test_data():
return read_csv(path)
I have been trying to use the fixture to return the test dataframe for the test_functions.
The original test functions were a bit more complex, calling pandas groupby on the object returned by the fixture. I kept on getting the error 'TestStrataFrame' object has no attribute 'groupby' so i simplified the test to the test below and, as I was still getting errors, I realized that i am probably missing something.
My test is the following:
#test1.py
import unittest
import pytest
class TestStrataFrame(unittest.TestCase):
def test_fixture(test_data):
assert isinstance(test_data,pd.DataFrame) is True
The above test_fixture returns:
=============================================== FAILURES ================================================
_____________________________________ TestStrataFrame.test_fixture ______________________________________
test_data = <tests.test_data.TestStrataFrame testMethod=test_fixture>
def test_fixture(test_data):
ciao=test_data
> assert isinstance(ciao,pd.DataFrame) is True
E AssertionError: assert False is True
E + where False = isinstance(<tests.test_data.TestStrataFrame testMethod=test_fixture>, <class 'pandas.core.frame.DataFrame'>)
E + where <class 'pandas.core.frame.DataFrame'> = pd.DataFrame
tests/test_data.py:23: AssertionError
=========================================== warnings summary ============================================
../../../../../opt/miniconda3/envs/geo/lib/python3.7/importlib/_bootstrap.py:219
/opt/miniconda3/envs/geo/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
return f(*args, **kwds)
-- Docs: https://docs.pytest.org/en/stable/warnings.html
======================================== short test summary info ========================================
FAILED tests/test_data.py::TestStrataFrame::test_fixture - AssertionError: assert False is True
================================ 1 failed, 4 passed, 1 warning in 12.82s ================================
How can i do this correctly?
PS : At the moment i would not focus on the RuntimeWarning. I am getting since after I have started trying to solve this issue, but i am quite sure the tests were failing even before I got that warning - so they are probably unrelated. I reinstalled the environment and the warning persists, hopefully might go away with solving the issue...

this works for me:
isinstance(type(my_pd_df),type(pandas.core.frame.DataFrame) )

The error is coming as test_data is not been passed to test_fixture method. for example below are two ways you can tweak your Class and its method.
import unittest
import pytest
import pandas as pd
class TestStrataFrame(unittest.TestCase):
test_data=pd.DataFrame()
def test_fixture(self):
test_data=pd.DataFrame()
assert isinstance(test_data,pd.DataFrame) is True
def test_fixture_1(self):
assert isinstance(TestStrataFrame.test_data,pd.DataFrame) is True
and run from terminal : pytest test_sample.py

This is the expected behavior if you take note of this page here. That page clearly states:
The following pytest features do not work, and probably never will due to different design philosophies:
1. Fixtures (except for autouse fixtures, see below);
2. Parametrization;
3. Custom hooks;
You can modify your code to the following to work.
# conftest.py
from pathlib import Path
import pytest
from pandas import read_csv
CWD = Path(__file__).resolve()
FIN = CWD.parent / "test.csv"
#pytest.fixture(scope="class")
def test_data(request):
request.cls.test_data = read_csv(FIN)
# test_file.py
import unittest
import pytest
import pandas as pd
#pytest.mark.usefixtures("test_data")
class TestStrataFrame(unittest.TestCase):
def test_fixture(self):
assert hasattr(self, "test_data")
assert isinstance(self.test_data, pd.DataFrame)
==>pytest tests/
============================= test session starts ==============================
platform darwin -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/***/Desktop/scripts/stackoverflow
collected 1 item
tests/test_file.py . [100%]
============================== 1 passed in 0.03s ===============================
You can see more about mixing fixtures with the unittest framework here.

Make a python py.test unit test run independantly of the location where py.test in executed?

Lets say my code looks like this
import pytest
import json
#pytest.fixture
def test_item():
test_item = json.load(open('./directory/sample_item_for_test.json','rb'))
return test_item
def test_fun(test_document):
assert type(test_item.description[0]) == unicode
And I would like to run this test via Py.Test
If I run Py.test from the directory that it is in, it is fine. BUT if I run it from an above directory, it fails due to not being able to find 'sample_item_for_test.json'. Is there a way to make this test run correctly no matter where I execute Py.test?

The magic attribute __file__ is the path to the python file on the filesystem. So, you can use that with some magic from os to get the current directory...
import pytest
import json
import os
_HERE = os.path.dirname(__file__)
_TEST_JSON_FILENAME = os.path.join(_HERE, 'directory', 'sample_item_for_test.json')
#pytest.fixture
def test_item():
with open(_TEST_JSON_FILENAME, 'rb') as file_input:
return json.load(file_input)

When I migrated to py.test, I had a large set of legacy tests that were accustomed to being executed in the directory where the test file lives. Instead of tracking down every test failure, I added a pytest hook to my conftest.py to chdir to the test directory before each test starts:
import os
import functools
def pytest_runtest_setup(item):
"""
Execute each test in the directory where the test file lives.
"""
starting_directory = os.getcwd()
test_directory = os.path.dirname(str(item.fspath))
os.chdir(test_directory)
teardown = functools.partial(os.chdir, starting_directory)
# There's probably a cleaner way than accessing a private member.
item.session._setupstate.addfinalizer(teardown, item)

How to run unittest discover from "python setup.py test"?

I'm trying to figure out how to get python setup.py test to run the equivalent of python -m unittest discover. I don't want to use a run_tests.py script and I don't want to use any external test tools (like nose or py.test). It's OK if the solution only works on python 2.7.
In setup.py, I think I need to add something to the test_suite and/or test_loader fields in config, but I can't seem to find a combination that works correctly:
config = {
'name': name,
'version': version,
'url': url,
'test_suite': '???',
'test_loader': '???',
}
Is this possible using only unittest built into python 2.7?
FYI, my project structure looks like this:
project/
package/
__init__.py
module.py
tests/
__init__.py
test_module.py
run_tests.py <- I want to delete this
setup.py
Update: This is possible with unittest2 but I want find something equivalent using only unittest
From https://pypi.python.org/pypi/unittest2
unittest2 includes a very basic setuptools compatible test collector. Specify test_suite = 'unittest2.collector' in your setup.py. This starts test discovery with the default parameters from the directory containing setup.py, so it is perhaps most useful as an example (see unittest2/collector.py).
For now, I'm just using a script called run_tests.py, but I'm hoping I can get rid of this by moving to a solution that only uses python setup.py test.
Here's the run_tests.py I'm hoping to remove:
import unittest
if __name__ == '__main__':
# use the default shared TestLoader instance
test_loader = unittest.defaultTestLoader
# use the basic test runner that outputs to sys.stderr
test_runner = unittest.TextTestRunner()
# automatically discover all tests in the current dir of the form test*.py
# NOTE: only works for python 2.7 and later
test_suite = test_loader.discover('.')
# run the test suite
test_runner.run(test_suite)

If you use py27+ or py32+, the solution is pretty simple:
test_suite="tests",

From Building and Distributing Packages with Setuptools (emphasis mine):
test_suite
A string naming a unittest.TestCase subclass (or a package or module
containing one or more of them, or a method of such a subclass), or naming
a function that can be called with no arguments and returns a unittest.TestSuite.
Hence, in setup.py you would add a function that returns a TestSuite:
import unittest
def my_test_suite():
test_loader = unittest.TestLoader()
test_suite = test_loader.discover('tests', pattern='test_*.py')
return test_suite
Then, you would specify the command setup as follows:
setup(
...
test_suite='setup.my_test_suite',
...
)

You don't need config to get this working. There are basically two main ways to do it:
The quick way
Rename your test_module.py to module_test.py (basically add _test as a suffix to tests for a particular module), and python will find it automatically. Just make sure to add this to setup.py:
from setuptools import setup, find_packages
setup(
...
test_suite = 'tests',
...
)
The long way
Here's how to do it with your current directory structure:
project/
package/
__init__.py
module.py
tests/
__init__.py
test_module.py
run_tests.py <- I want to delete this
setup.py
Under tests/__init__.py, you want to import the unittest and your unit test script test_module, and then create a function to run the tests. In tests/__init__.py, type in something like this:
import unittest
import test_module
def my_module_suite():
loader = unittest.TestLoader()
suite = loader.loadTestsFromModule(test_module)
return suite
The TestLoader class has other functions besides loadTestsFromModule. You can run dir(unittest.TestLoader) to see the other ones, but this one is the simplest to use.
Since your directory structure is such, you'll probably want the test_module to be able to import your module script. You might have already done this, but just in case you didn't, you could include the parent path so that you can import the package module and the module script. At the top of your test_module.py, type:
import os, sys
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
import unittest
import package.module
...
Then finally, in setup.py, include the tests module and run the command you created, my_module_suite:
from setuptools import setup, find_packages
setup(
...
test_suite = 'tests.my_module_suite',
...
)
Then you just run python setup.py test.
Here is a sample someone made as a reference.

One possible solution is to simply extend the test command for distutilsand setuptools/distribute. This seems like a total kluge and way more complicated than I would prefer, but seems to correctly discover and run all the tests in my package upon running python setup.py test. I'm holding off on selecting this as the answer to my question in hopes that someone will provide a more elegant solution :)
(Inspired by https://docs.pytest.org/en/latest/goodpractices.html#integrating-with-setuptools-python-setup-py-test-pytest-runner)
Example setup.py:
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
def discover_and_run_tests():
import os
import sys
import unittest
# get setup.py directory
setup_file = sys.modules['__main__'].__file__
setup_dir = os.path.abspath(os.path.dirname(setup_file))
# use the default shared TestLoader instance
test_loader = unittest.defaultTestLoader
# use the basic test runner that outputs to sys.stderr
test_runner = unittest.TextTestRunner()
# automatically discover all tests
# NOTE: only works for python 2.7 and later
test_suite = test_loader.discover(setup_dir)
# run the test suite
test_runner.run(test_suite)
try:
from setuptools.command.test import test
class DiscoverTest(test):
def finalize_options(self):
test.finalize_options(self)
self.test_args = []
self.test_suite = True
def run_tests(self):
discover_and_run_tests()
except ImportError:
from distutils.core import Command
class DiscoverTest(Command):
user_options = []
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
discover_and_run_tests()
config = {
'name': 'name',
'version': 'version',
'url': 'http://example.com',
'cmdclass': {'test': DiscoverTest},
}
setup(**config)

Another less than ideal solution slightly inspired by http://hg.python.org/unittest2/file/2b6411b9a838/unittest2/collector.py
Add a module that returns a TestSuite of discovered tests. Then configure setup to call that module.
project/
package/
__init__.py
module.py
tests/
__init__.py
test_module.py
discover_tests.py
setup.py
Here's discover_tests.py:
import os
import sys
import unittest
def additional_tests():
setup_file = sys.modules['__main__'].__file__
setup_dir = os.path.abspath(os.path.dirname(setup_file))
return unittest.defaultTestLoader.discover(setup_dir)
And here's setup.py:
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
config = {
'name': 'name',
'version': 'version',
'url': 'http://example.com',
'test_suite': 'discover_tests',
}
setup(**config)

Python's standard library unittest module supports discovery (in Python 2.7 and later, and Python 3.2 and later). If you can assume those minimum versions, then you can just add the discover command line argument to the unittest command.
Only a small tweak is needed to setup.py:
import setuptools.command.test
from setuptools import (find_packages, setup)
class TestCommand(setuptools.command.test.test):
""" Setuptools test command explicitly using test discovery. """
def _test_args(self):
yield 'discover'
for arg in super(TestCommand, self)._test_args():
yield arg
setup(
...
cmdclass={
'test': TestCommand,
},
)

This won't remove run_tests.py, but will make it work with setuptools. Add:
class Loader(unittest.TestLoader):
def loadTestsFromNames(self, names, _=None):
return self.discover(names[0])
Then in setup.py: (I assume you're doing something like setup(**config))
config = {
...
'test_loader': 'run_tests:Loader',
'test_suite': '.', # your start_dir for discover()
}
The only downside I see is it's bending the semantics of loadTestsFromNames, but the setuptools test command is the only consumer, and calls it in a specified way.

Django and tests in docfiles

I'm having a small problem with my test suite with Django.
I'm working on a Python package that can run in both Django and Plone (http://pypi.python.org/pypi/jquery.pyproxy).
All the tests are written as doctests, either in the Python code or in separate docfiles (for example the README.txt).
I can have those tests running fine but Django just do not count them:
[vincent ~/buildouts/tests/django_pyproxy]> bin/django test pyproxy
...
Creating test database for alias 'default'...
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
But if I had some failing test, it will appear correctly:
[vincent ~/buildouts/tests/django_pyproxy]> bin/django test pyproxy
...
Failed example:
1+1
Expected nothing
Got:
2
**********************************************************************
1 items had failures:
1 of 44 in README.rst
***Test Failed*** 1 failures.
Creating test database for alias 'default'...
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
This is how my test suite is declared right now:
import os
import doctest
from unittest import TestSuite
from jquery.pyproxy import base, utils
OPTIONFLAGS = (doctest.ELLIPSIS |
doctest.NORMALIZE_WHITESPACE)
__test__ = {
'base': doctest.testmod(
m=base,
optionflags=OPTIONFLAGS),
'utils': doctest.testmod(
m=utils,
optionflags=OPTIONFLAGS),
'readme': doctest.testfile(
"../../../README.rst",
optionflags=OPTIONFLAGS),
'django': doctest.testfile(
"django.txt",
optionflags=OPTIONFLAGS),
}
I guess I'm doing something wrong when declaring the test suite but I don't have a clue what it is exactly.
Thanks for your help,
Vincent

I finally solved the problem with the suite() method:
import os
import doctest
from django.utils import unittest
from jquery.pyproxy import base, utils
OPTIONFLAGS = (doctest.ELLIPSIS |
doctest.NORMALIZE_WHITESPACE)
testmods = {'base': base,
'utils': utils}
testfiles = {'readme': '../../../README.rst',
'django': 'django.txt'}
def suite():
return unittest.TestSuite(
[doctest.DocTestSuite(mod, optionflags = OPTIONFLAGS)
for mod in testmods.values()] + \
[doctest.DocFileSuite(f, optionflags = OPTIONFLAGS)
for f in testfiles.values()])
Apparently the problem when calling doctest.testfile or doctest.testmod is that the tests are directly ran.
Using DocTestSuite/DocFileSuite builds the list and then the test runner runs them.