I have an AWS S3 directory containing several JSON files, which are used as test inputs.
I've created a PyTest module that downloads all JSON files once using a module wide fixture, and then runs several test functions - each being parameterized over the set of JSONs:
import pytest
import os
from tempfile import mkdtemp, TemporaryDirectory
from glob import glob
JSONS_AWS_PATH = 's3://some/path/'
def download_jsons():
temp_dir = mkdtemp()
aws_sync_dir(JSONS_AWS_PATH, temp_dir)
json_filenames = glob(os.path.join(local_path, "*.json"))
return json_filenames
#pytest.fixture(scope='module', params=download_jsons()) #<-- Invoking download_jsons() directly
def json_file(request):
return request.param
def test_something(json_file):
# Open the json file and test something
def test_something_else(json_file):
# Open the json file and test something else
def test_another_thing(json_file):
# you got the point...
This test module in itself works - the only pain point is how to cleanup the temp_dir at the end of the module\session.
Since download_jsons() is being invoked directly, before json_file fixture is even started - it has no context of its own. So I can't make it clean temp_dir after all the tests are done.
I would like to make download_jsons() a module\session scope fixture in itself. Something like:
fixture(scope='module')
def download_jsons():
temp_dir = mkdtemp()
# Download and as glob, as above
yield json_filenames
shutil.rmtree(temp_dir)
or
fixture(scope='module')
def download_jsons(tmpdir_factory):
#...
as #Gabriela Melo has suggested.
The question is how to make the json_file fixture parameterized over the list returned by download_jsons(), without invoking it directly?
I've tried implementing this solution with either mark.parametrize, setup_module(), or pytest_generate_tests(metafunc) - but wasn't able to implement the exact functionality I was looking for.
This seems to be what you're looking for: https://docs.pytest.org/en/latest/tmpdir.html#the-tmpdir-factory-fixture
(Using Pytest's tmpdir_factory fixture and setting the scope of your json_file function to session instead of module)
If you want to use a resource for parametrization, it can't be returned by a fixture (at least with the current version of pytest). However, you can move the setup/teardown code out to the hooks - this will also enable parametrizing via pytest_generate_tests hook. Example: in the root dir of your project, create a file named conftest.py with the following contents:
from tempfile import TemporaryDirectory
from pathlib import Path
def pytest_sessionstart(session):
# this is where we create the temp dir and download the files
session.config._tempdir = TemporaryDirectory()
d = session.config._tempdir.name
aws_sync_dir(JSONS_BLOBBY_PATH, d)
session.config._json_files = Path(d).glob("*.json")
def pytest_sessionfinish(session):
# this is where we delete the created temp dir
session.config._tempdir.cleanup()
def pytest_generate_tests(metafunc):
if "file" in metafunc.fixturenames:
# parametrize the file argument with the downloaded files
files = metafunc.config._json_files
metafunc.parametrize("file", files)
You can now use the file argument in tests as usual, e.g.
def test_ends_with_json(file):
assert file.suffix == ".json"
Related
I have just started a python project. The directory structure is as follows:
/algorithms
----/__init__.py
----/linkedlist
--------/__init__.py
--------/file1.py
--------/file2.py
/tests
----/test_linkedlist
You can also check the Github repository.
In each of the sub folders under algorithms, in the __init__ file I am including the following for all the files one by one:
from .file1 import *
from .file2 import *
And so on.
The task that I am trying to achieve is running all tests together using the query:
python3 -m unittest discover tests
Each file in the tests directory starts as follows:
from algorithms.linkedlist import *
import unittest
Right now if I want to add a new file to the linkedlist directory, I create the file and then add another from .filename import * in the __init__ file.
How do I write a script in the __init__ file so that each time I create a new file, I do not have to manually insert the import command?
So the __init__ is in the same folder? As the docs say The import statement is syntactic sugar for the __import__ function.
So we can use:
import importlib
import glob
for file in glob.iglob('*.py'):
importlib.__import__(file)
Some reasons why this does not work:
You want to load the functions in the module - the import * from syntax. With this code you can only run file1.test.
You run the script loading from another directory, which confuses glob. We have to specify the actual working directory.
__import__ prefers to know the module name.
To find the solution I combine the import * from function from this answer with pkgutil.walk_packages from this blog.
import importlib
import pkgutil
def custom_import_all(module_name):
""" Use to dynamically execute from module_name import * """
# get a handle on the module
mdl = importlib.import_module(module_name)
# is there an __all__? if so respect it
if "__all__" in mdl.__dict__:
names = mdl.__dict__["__all__"]
else:
# otherwise we import all names that don't begin with _
names = [x for x in mdl.__dict__ if not x.startswith("_")]
# now drag them in
globals().update({k: getattr(mdl, k) for k in names})
__path__ = pkgutil.extend_path(__path__, __name__)
for importer, modname, ispkg in pkgutil.walk_packages(path=__path__, prefix=__name__+'.'):
custom_import_all(modname)
I have a proprietary repository format and I'm trying to develop a Python module to process these repositories. Repo format goes as:
/home/X/
|
+ alpha/
|
+ beta/
|
+ project.conf
Here, X is a project. alpha and beta are folders inside this project and they represent groups in this project. A group is a container in this repo and what it represents is really not relevant for this question. The repo X also has files in its root level; project.conf is an example of such a file.
I have a class called Project that abstracts projects such as X. The Project class has a method load() that builds an in-memory representation.
class Project(object):
def load(self):
for entry in os.listdir(self.root):
path = os.path.join(self.root, entry)
if os.path.isdir(path):
group = Group(path)
self.groups.append(group)
group.load()
else:
# process files ...
To unit test the load() method by mocking the file system, I have:
import unittest
from unittest import mock
import Project
class TestRepo(unittest.TestCase):
def test_load_project(self):
project = Project("X")
with mock.patch('os.listdir') as mocked_listdir:
mocked_listdir.return_value = ['alpha', 'beta', 'project.conf']
project.load()
self.assertEqual(len(project.groups), 2)
This does mock os.listdir successfully. But I can't trick Python to treat mocked_listdir.return_value as consisting of files and directories.
How do I mock either os.listdir or os.path.isdir, in the same test, such that the test will see alpha and beta as directories and project.conf as a file?
You could use pyfakefs (source, docs), a very handy lib to test operations on a fake file system.
If you use pytest, it has a plugin, all file system functions already get patched, you just need to use its fs fixture:
import os
def test_foo(fs):
fs.CreateFile('/home/x/alpha/1')
fs.CreateFile('/home/x/beta/2')
fs.CreateFile('/home/x/p.conf')
assert os.listdir('/home/x/') == ['alpha', 'beta', 'p.conf']
or if you prefer unittest:
import os
import unittest
from pyfakefs import fake_filesystem_unittest
class TestRepo(fake_filesystem_unittest.TestCase):
def setUp(self):
self.setUpPyfakefs()
def test_foo(self):
os.makedirs('/home/x/alpha')
os.makedirs('/home/x/beta')
with open('/home/x/p.conf', 'w') as f:
f.write('foo')
self.assertEqual(os.listdir('/home/x/'), ['alpha', 'beta', 'p.conf'])
if __name__ == "__main__":
unittest.main()
I managed to achieve the desired behavior by passing an iterable to the side_effect attribute of the mocked isdir object.
import unittest
from unittest import mock
import Project
class TestRepo(unittest.TestCase):
def test_load_project(self):
project = Project("X")
with mock.patch('os.listdir') as mocked_listdir:
with mock.patch('os.path.isdir') as mocked_isdir:
mocked_listdir.return_value = ['alpha', 'beta', 'project.conf']
mocked_isdir.side_effect = [True, True, False]
project.load()
self.assertEqual(len(project.groups), 2)
The key is the mocked_isdir.side_effect = [True, True, False] line. The boolean values in the iterable should match the order of directory and file entries passed to the mocked_listdir.return_value attribute.
It will depend, of course, on exactly which os functions you use, but it looks like mock.patch.multiple on os is just what you need. (Note that you may not need to patch path; many of its functions are lexical-only and do not care about the actual filesytem.)
I am dynamically loading filters to my app. Now I want to be able to extend existing filters:
In my Main.py I do:
spec = importlib.util.spec_from_file_location(filter, file)
inst = importlib.util.module_from_spec(spec)
spec.loader.exec_module(inst)
I have a file called varianceFilter.py that loads and runs fine:
varianceFilter.py:
def run(images):
#do something
return result
Now I want to reuse and extend this filter like so:
testFilter.py:
import varianceFilter as vf
def run(images):
ret = vf.run(images)
#do something with ret
return ret
However as soon as I try to import varianceFilter.py the exception
No module named 'varianceFilter'
is thrown. both files are in the same directory.
What am I doing wrong?
EDIT:
My directory structure is:
main.py
filters/varianceFilter.py
filters/testFilter.py
After creating a copy of varianceFilter in the main directory, testFilter.py works fine.
Adding the empty file
filters/__init__.py
and changing the import statement in testFilter.py to
import filters.varianceFilter as vf
fixed the problem.
Lets say my code looks like this
import pytest
import json
#pytest.fixture
def test_item():
test_item = json.load(open('./directory/sample_item_for_test.json','rb'))
return test_item
def test_fun(test_document):
assert type(test_item.description[0]) == unicode
And I would like to run this test via Py.Test
If I run Py.test from the directory that it is in, it is fine. BUT if I run it from an above directory, it fails due to not being able to find 'sample_item_for_test.json'. Is there a way to make this test run correctly no matter where I execute Py.test?
The magic attribute __file__ is the path to the python file on the filesystem. So, you can use that with some magic from os to get the current directory...
import pytest
import json
import os
_HERE = os.path.dirname(__file__)
_TEST_JSON_FILENAME = os.path.join(_HERE, 'directory', 'sample_item_for_test.json')
#pytest.fixture
def test_item():
with open(_TEST_JSON_FILENAME, 'rb') as file_input:
return json.load(file_input)
When I migrated to py.test, I had a large set of legacy tests that were accustomed to being executed in the directory where the test file lives. Instead of tracking down every test failure, I added a pytest hook to my conftest.py to chdir to the test directory before each test starts:
import os
import functools
def pytest_runtest_setup(item):
"""
Execute each test in the directory where the test file lives.
"""
starting_directory = os.getcwd()
test_directory = os.path.dirname(str(item.fspath))
os.chdir(test_directory)
teardown = functools.partial(os.chdir, starting_directory)
# There's probably a cleaner way than accessing a private member.
item.session._setupstate.addfinalizer(teardown, item)
This question already has answers here:
How to import a module in Python with importlib.import_module
(3 answers)
Closed 8 years ago.
I have the following directory structure
+ code
|
--+ plugins
|
-- __init__.py
-- test_plugin.py (has a class TestPlugin)
-- another_test_plugin.py (has a class AnotherTestPlugin)
--+ load.py
--+ __init__.py
In load.py, I want to be able to initialize only those classes that the user specifies. For example, lets say I do something like
$ python load.py -c test_plugin # Should only import test_plugin.py and initialize an object of the TestPlugin class
I am having trouble trying to use the "imp" module to do it. It keeps on saying "No such file or directory". My understanding is that it is somehow not understanding the path properly. Can someone help me out with this?
ok, your problem is a path related problem. You expect that the script is being run in the same directory as where load.py is, where it is not the case.
what you have to do is something like:
import imp, os, plugins
path = os.path.dirname(plugins.__file__)
imp.load_source('TestPlugin', os.path.join(path, 'test_plugin.py')
where plugins is the module containing all your plugins (i.e. just the empty __init__.py), that will help you get the full path to your plugin modules' files.
Another solution, if you want a "plugins" discovery tool:
import imp, os
import glob
def load_plugins(path):
"""
Assuming `path` is the only directory in which you store your plugins,
and assuming each name follows the syntax:
plugin_file.py -> PluginFile
Please note that we don't import files starting with an underscore.
"""
plugins = {}
plugin_files = glob.glob(path + os.sep + r'[!_]*.py')
for plugin_path in plugin_files:
module_name, ext = os.path.splitext(plugin_path)
module_name = os.path.basename(module_name)
class_name = module_name.title().replace('_', '')
loaded_module = imp.load_source(class_name, plugin_path) # we import the plugin
plugins[module_name] = getattr(loaded_module, class_name)
return plugins
plugins = load_plugins(your_path_here)
plugin_name = sys.argv[3]
plugin = plugins.get(plugin_name)
if not plugin:
# manage a not existing plugin
else:
plugin_instance = plugin() # creates an instance of your plugin
This way, you can also specify different names by changing your keys, e.g., 'test_plugins' => 'tp'. You don't have to initialize your plugins, but you can still run this function whenever you want to load your plugins at runtime.
exec('import ' + sys.argv[2])
obj = test_plugin.TestPlugin()
Here sys.argv[2] is 'test_plugin' string from command line arguments.
EDIT: Another way to avoid using exec:
import importlib
mod = importlib.import_module(sys.argv[2])