I have a class that stores a function and a method that tries to run the function in parallel using the multiprocessing module. I'm testing the class with the nose module and the nose.main() function hangs (nothing happens, no output is displayed, the kernel must be manually told to stop and restart) when run. My directory looks like:
Parent/
test.py
Code/
__init__.py
code.py
tests/
test_code.py
code.py contains:
import multiprocessing as mp
import numpy as np
import itertools
def target(): pass
def target_star(args): return target(*args)
class FuncWrapper():
def __init__(self, fun):
self.fun = fun
def run(self, X):
try:
arg_list_objects = []
arg_list_inputs = []
for i in range(len(X)):
arg_list_objects.append(self.fun.im_self)
arg_list_inputs.append(X[i])
target.__code__ = self.fun.im_func.__code__
pool = mp.Pool(processes=mp.cpu_count()-1)
F = np.array(pool.map(target_star, itertools.izip(arg_list_objects, arg_list_inputs))).reshape(X.shape)
pool.close()
pool.join()
except:
F = self.fun(X)
return F
Most of the code in the try: block is used to make the pool.map function work with class methods.
test_code.py contains:
import numpy as np
from unittest import TestCase
import unittest
from Code.code import FuncWrapper
class TestCode(TestCase):
def f(self, x):
return x
def test_FuncWrapper(self):
x = np.random.uniform(0, 1, (50, 1))
return FuncWrapper(self.f).run(x)
if __name__ == '__main__':
unittest.main()
test.py contains:
import nose
nose.main()
When I run test_code.py from the Parent folder, it successfully performs the test but when test.py is run, nothing happens. How should I use the nose.main function so that it doesn't hang, or is there another function in the nose module that would work here?
I'm using Ubuntu 14.04 LTS 64-bit with the Enthought Python distribution (Python 2.7.11 64-bit) and their Canopy IDE.
Related
I have a simple Python script which runs a parallel pool:
import multiprocessing as mp
def square(x):
return x**2
if __name__ == '__main__':
pool = mp.Pool(4)
results=pool.map(square,range(1,20))
It works fine and as expected. However, if I import any simple custom class, such as the code below, it doesn't work any more. I start the script execution, but the script does not terminate. This is weird, as I do not use the imported class.
import multiprocessing as mp
from Person import Person
def square(x):
return x**2
if __name__ == '__main__':
pool = mp.Pool(4)
results=pool.map(square,range(1,20))
The class Person is very simple:
class Person:
def __init__(self, id):
self.id = id
What is the reason for this behavior and how can I fix it?
EDIT: I am using Windows 10
so i've got a problem with my code.
File 1:
class Abc(object):
...
def function1(self):
#do something
def function2(self):
x = input()
return x+1
and now i'm trying to test function 2 so i wrote a test for it and i don't know what i am doing wrong:
from unittest.mock import patch
import unittest
from file1 import *
class TestBackend(unittest.TestCase):
def test_mode_first(self):
self.assertEqual(Abc().funcion1(), 30)
#patch('funcion2.input', create=True)
def test_mode_second(self, mocked_input):
mocked_input.side_effect = ["QWE"]
result = Abc().funcion2()
self.assertEqual(result, 10)
if __name__ == '__main__':
unittest.main()
i get ModuleNotFoundError: No module named 'function2'
so what i am doing wrong in here?
thanks for your help :)
You get ModuleNotFoundError because funcion2 is not a module. patch doc is clear about this:
target should be a string in the form 'package.module.ClassName'. The
target is imported and the specified object replaced with the new
object, so the target must be importable from the environment you are
calling patch() from. The target is imported when the decorated
function is executed, not at decoration time.
This works for me when executed with python3 -m unittest discover from the directory the files are in.
BTW you have a couple of typos in your example, e.g. Abc().funcion2(), note the missing t in funcion2.
Also, try not to use from … import *: https://docs.quantifiedcode.com/python-anti-patterns/maintainability/from_module_import_all_used.html#using-wildcard-imports-from-import
# file1.py
class Abc(object):
def function1(self):
return 30
def function2(self):
x = input()
return x + "1"
# test_file1.py
import unittest
from unittest.mock import patch
from file1 import Abc
class TestBackend(unittest.TestCase):
def test_mode_first(self):
self.assertEqual(Abc().function1(), 30)
#patch('builtins.input')
def test_mode_second(self, mocked_input):
mocked_input.return_value = "QWE"
result = Abc().function2()
self.assertEqual(result, "QWE1")
I would like to ensure with tests that:
- the application cannot be imported
- the application can be started as a real application (i.e: python src.py)
I'm interested about that, why the following is not working:
src.py
class A:
def x(self):
print('this is x')
if __name__ == '__main__':
A().x()
test.py (snippet)
class Test(unittest.TestCase):
#mock.patch('src.A.x')
def test_main(self, mock_x):
import src
mock_x.assert_any_call()
This test fails... why?
Because the name of the module when imported is src, not __main__.
The easiest solution would be to move that code into a function:
def main():
A().x()
if __name__ == '__main__':
main()
and in your test, you would invoke src.main()
#mock.patch('src.A.x')
def test_main(self, mock_x):
import src
src.main()
mock_x.assert_any_call()
To test that a module is not importable you do not need to use mocks.
See assertRaises.
Just check if an error is thrown on import od module.
with self.assertRaises(...):
...
In python 2.6, I've been reading through the unittest documentation. But I still haven't found this answer.
What function does pyton -m unittest execute?
For example, how would I modify this code such that just executing python -m unittest would detect it and run the test?
import random
import unittest
class TestSequenceFunctions(unittest.TestCase):
def setUp(self):
self.seq = range(10)
def test_shuffle(self):
# make sure the shuffled sequence does not lose any elements
random.shuffle(self.seq)
self.seq.sort()
self.assertEqual(self.seq, range(10))
def test_choice(self):
element = random.choice(self.seq)
self.assertTrue(element in self.seq)
def test_sample(self):
self.assertRaises(ValueError, random.sample, self.seq, 20)
for element in random.sample(self.seq, 5):
self.assertTrue(element in self.seq)
if __name__ == '__main__':
unittest.main()
EDIT:
Note this is just a example, I'm actually trying to get it to detect and run multiple tests as a suite, here is my starting point - but python -m unittest doesn't detect it nor does python -m unittest discovery work with it. I have to call python discovery.py to execute it.
discovery.py:
import os
import unittest
def makeSuite():
"""Function stores all the modules to be tested"""
modules_to_test = []
test_dir = os.listdir('.')
for test in test_dir:
if test.startswith('test') and test.endswith('.py'):
modules_to_test.append(test.rstrip('.py'))
all_tests = unittest.TestSuite()
for module in map(__import__, modules_to_test):
module.testvars = ["variables you want to pass through"]
all_tests.addTest(unittest.findTestCases(module))
return all_tests
if __name__ == '__main__':
unittest.main(defaultTest='makeSuite')
python -m something executes the module something as a script. i.e. from python --help:
-m mod : run library module as a script (terminates option list)
The unittest module can be run as a script -- And the arguments that you pass to it determine which files it tests. The commandline interface is also documented in the language reference.
I have started using nose to run tests. I discovered the multiprocessing plugin has a timeout, that I can change on the command line.
Is there a way to extend the timeout for individual tests (in the test code) so I don't have a massive global timeout?
I haven't any experience with the multiprocessing plugin but if you subclass the plugin with something like this:
from nose.plugins.multiprocess import MultiProcess
PLUGIN = None
class TimeoutMultiProcess(MultiProcess):
def configure(self, options, conf):
global PLUGIN
PLUGIN = self
super(TimeoutMultiProcess, self).configure(options, conf)
if not self.enabled:
return
then you can create your own test running script like:
import unittest
class TestA(unittest.TestCase):
def setUp(self):
from runtest import PLUGIN
print PLUGIN.config.multiprocess_timeout
def test_a(self):
pass
def test_b(self):
pass
if __name__ == '__main__':
from runtest import TimeoutMultiProcess
import nose
nose.main(addplugins=[TimeoutMultiProcess()], defaultTest="./test.py")
You'll be able to change the config.multiprocess_timeout to different values within your tests. I'm not sure if it will work for you, but it's worth a shot.