I have a Python project with a bunch of tests that have already been implemented, and I'd like to begin benchmarking them so I can compare performance of the code, servers, etc over time. Locating the files in a manner similar to Nose was no problem because I have "test" in the names of all my test files anyway. However, I'm running into some trouble in attempting to dynamically execute these tests.
As of right now, I'm able to run a script that takes a directory path as an argument and returns a list of filepaths like this:
def getTestFiles(directory):
fileList = []
print "Searching for 'test' in " + directory
if not os.path.isdir(os.path.dirname(directory)):
# throw error
raise InputError(directory, "Not a valid directory")
else:
for root, dirs, files in os.walk(directory):
#print files
for f in files:
if "test" in f and f.endswith(".py"):
fileList.append(os.path.join(root, f))
return fileList
# returns a list like this:
# [ 'C:/Users/myName/Desktop/example1_test.py',
# 'C:/Users/myName/Desktop/example2_test.py',
# 'C:/Users/myName/Desktop/folder1/example3_test.py',
# 'C:/Users/myName/Desktop/folder2/example4_test.py'... ]
The issue is that these files can have different syntax, which I'm trying to figure out how to handle. For example:
TestExampleOne:
import dummy1
import dummy2
import dummy3
class TestExampleOne(unittest.TestCase):
#classmethod
def setUpClass(cls):
# set up
def test_one(self):
# test stuff
def test_two(self):
# test stuff
def test_three(self):
# test stuff
# etc...
TestExampleTwo:
import dummy1
import dummy2
import dummy3
def setup(self):
try:
# config stuff
except Exception as e:
logger.exception(e)
def test_one():
# test stuff
def test_two():
# test stuff
def test_three():
# test stuff
# etc...
TestExampleThree:
import dummy1
import dummy2
import dummy3
def setup(self):
try:
# config stuff
except Exception as e:
logger.exception(e)
class TestExampleTwo(unittest.TestCase):
def test_one(self):
# test stuff
def test_two(self):
# test stuff
# etc...
class TestExampleThree(unittest.TestCase):
def test_one(self):
# test stuff
def test_two(self):
# test stuff
# etc...
# etc...
I would really like to be able to write one module that searches a directory for every file containing "test" in its name, and then executes every unit test in each file, providing execution time for each test. I think something like NodeVisitor is on the right track, but I'm not sure. Even an idea of where to start would be greatly appreciated. Thanks
Using nose test runner would help to discover the tests, setup/teardown functions and methods.
nose-timer plugin would help with benchmarking:
A timer plugin for nosetests that answers the question: how much time
does every test take?
Demo:
imagine you have a package named test_nose with the following scripts inside:
test1.py:
import time
import unittest
class TestExampleOne(unittest.TestCase):
#classmethod
def setUpClass(cls):
cls.value = 1
def test_one(self):
time.sleep(1)
self.assertEqual(1, self.value)
test2.py:
import time
value = None
def setup():
global value
value = 1
def test_one():
time.sleep(2)
assert value == 1
test3.py:
import time
import unittest
value = None
def setup():
global value
value = 1
class TestExampleTwo(unittest.TestCase):
def test_one(self):
time.sleep(3)
self.assertEqual(1, value)
class TestExampleThree(unittest.TestCase):
def test_one(self):
time.sleep(4)
self.assertEqual(1, value)
install nose test runner:
pip install nose
install nose-timer plugin:
pip install nose-timer
run the tests:
$ nosetests test_nose --with-timer
....
test_nose.test3.TestExampleThree.test_one: 4.0003s
test_nose.test3.TestExampleTwo.test_one: 3.0010s
test_nose.test2.test_one: 2.0011s
test_nose.test1.TestExampleOne.test_one: 1.0005s
----------------------------------------------------------------------
Ran 4 tests in 10.006s
OK
The result is actually conveniently highlighted:
The coloring can be controlled by --timer-ok and --timer-warning arguments.
Note that time.sleep(n) calls were added for making the manual slowdowns to see the impact clearly. Also note that value variable is set to 1 in "setup" functions and methods, then in test function and methods the value is asserted to be 1 - this way you can see the work of setup functions.
UPD (running nose with nose-timer from script):
from pprint import pprint
import nose
from nosetimer import plugin
plugin = plugin.TimerPlugin()
plugin.enabled = True
plugin.timer_ok = 1000
plugin.timer_warning = 2000
plugin.timer_no_color = False
nose.run(plugins=[plugin])
result = plugin._timed_tests
pprint(result)
Save it into the test.py script and pass a target directory to it:
python test.py /home/example/dir/tests --with-timer
The result variable would contain:
{'test_nose.test1.TestExampleOne.test_one': 1.0009748935699463,
'test_nose.test2.test_one': 2.0003929138183594,
'test_nose.test3.TestExampleThree.test_one': 4.000233173370361,
'test_nose.test3.TestExampleTwo.test_one': 3.001115083694458}
Related
I am somewhat of a beginner in python, i am currently writing a suite of test cases with selenium webdriver using unittest; i have also found a lot of useful answers here, but it's time a ask my first question, i have struggled a lot with this and cannot find a proper answer, so any help is greatly appreciated:
For short, i have a suite of multiple tests cases, and in each case the first step is always ".get('Some URL')"; i have written these test cases for a single environment, but i would like to be able to select the URL on which all tests will be executed. In the example below i called the "access_url" method with a specific environment, but i need to do this for all of my scenarios at once, is it possible to do this from where i execute the .py file (e.g. "python example.py")? or to pass it in the .run() method when i select what suite to run?
import HTMLTestRunner
from selenium import webdriver
import unittest
This is a custom class used to create the 'access_url' method
def MyClass(object):
def __init__(self, driver):
self.driver = driver
def access_url(self, URL):
if URL == 'environment 1':
self.driver.get('https://www.google.com/')
elif URL == 'environment 2':
self.driver.get('https://example.com/')
In the classes i use to write test cases the first step is always 'access URL'
class TestScenario01(unittest.TestCase):
def setUp(self):
[...]
def test_01_access(self):
MyClass(self.driver).access_url(URL='environment 2')
def test_02(self):
[...]
def test_03(self):
[...]
In order to run the tests i place them all in a suite and use .run() on them
tc_scenario01 = unittest.TestLoader().loadTestsFromTestCase(TestScenario01)
test_suite = unittest.TestSuite([tc_scenario01])
HTMLReporterCustom.HTMLTestRunner().run(test_suite)
Finally, in order to execute the script i type the follwoing line in CMD: 'python example_file.py
As i mentioned above, all i want to do is to be able to somehow pass the URL one time to all test cases that call the "access_url()" method. Thanks!
You can maintain environment properties in separate config file.
config.py
DEFAULT_ENVIRONMENT='environment1'
URL = {
'environment1': 'https://www.google.com/',
'environment2': 'https://example.com/'
}
Your Class,
from package import config
def MyClass(object):
def __init__(self, driver):
self.driver = driver
def access_url(self):
self.driver.get(config.URL[config.DEFAULT_ENVIRONMENT])
Then test class will be as expected,
class TestScenario01(unittest.TestCase):
def setUp(self):
[...]
def test_01_access(self):
MyClass(self.driver).access_url()
def test_02(self):
[...]
def test_03(self):
[...]
While running test you can change,
main.py
from package import config
config.DEFAULT_ENVIRONMENT = 'enviroment2'
tc_scenario01 = unittest.TestLoader().loadTestsFromTestCase(TestScenario01)
test_suite = unittest.TestSuite([tc_scenario01])
HTMLReporterCustom.HTMLTestRunner().run(test_suite)
You can also pass the environment name while running python main.py.
main.py
if __name__ == '__main__':
config.DEFAULT_ENVIRONMENT = sys.argv[1] if len(sys.argv) > 2 else 'dev'
tc_scenario01 = unittest.TestLoader().loadTestsFromTestCase(TestScenario01)
test_suite = unittest.TestSuite([tc_scenario01])
HTMLReporterCustom.HTMLTestRunner().run(test_suite)
Ok please look at my example code
My ConnectBase class:
class ConnectBase(unittest.TestCase):
def setUp(self):
desired_caps = {}
desired_caps['platformName'] = 'Android'
desired_caps['deviceName'] = '4cfe4b59'
desired_caps['platformVersion'] = '5.0'
desired_caps['appPackage'] = 'com.xyz.bookshelf'
desired_caps['appActivity'] = 'com.xyz.bookshelf.MainActivity'
desired_caps['noReset'] = False
self.driver_android = webdriver.Remote('http://localhost:4723/wd/hub', desired_caps)
self.driver_android.implicitly_wait(30)
And my main file with tests:
import unittest
from connect import ConnectBase
from main_screen import MainScreenCheck
class MainTests(ConnectBase, MainScreenCheck):
with open('report.txt', 'w') as f:
f.write("----------Bookshelf----------\n")
def test_bookshelf_tutorial(self):
self.addToFile("Test Tutorial")
self.driver_android.orientation = 'LANDSCAPE'
super(MainTests, self).logout_screen_check()
def test_bookshelf_2(self):
self.addToFile("Test 2")
super(MainTests, self).login_screen_check()
def test_bookshelf_3(self):
self.addToFile("Test 3")
super(MainTests, self).loading_screen_check()
def test_bookshelf_4(self):
self.addToFile("Test 4")
super(MainTests, self).list_check()
if __name__ == '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(MainTests)
unittest.TextTestRunner(verbosity=2).run(suite)
I run script -> it is connecting
It starts "test_bookshelf_tutorial()"
Test passed and i would like to continue with "test_bookshelf_2()" but the app is restarting... and i have to go throught tutorial screen again...
The problem is that every unittest "def test_xyz(self)" application is restarting so I can't use unittest function that shows the passed test in report becouse, each test I must go through everything that I made in tests before
I created my way to make a test report -> I'm adding each test result to txt file... but I wonder if there is a possibility to turn off this app restarting and use normal unittest reports?
Or maybe there is another great way to do reports of automation tests?
Try to make an order to your test cases, some times test depend from each other
in the first step open the application and close it only in the last step:
class MainTests(ConnectBase, TestCase):
def step1(self):
#open the application
def step2(self):
...
def steps(self):
for name in sorted(dir(self)):
if name.startswith("step"):
yield name, getattr(self, name)
def test_steps(self):
for name, step in self.steps():
try:
step()
except Exception as e:
self.fail("{} failed ({}: {})".format(step, type(e), e)
I suggest that you use a Test framework like 'TESTNG' to define test priority to manage tests order but be sure that the first test is always executed to open the application ;)
I wrote the following code as below:
#base.py
import sys
import unittest
class BaseClass(unittest.TestCase):
def setUp():
print"in base script"
def tearDown():
print"in teardown"
This is the test script:
#test.py
import sys
import unittest
from base import *
class demo_script(BaseClass):
def setUp(self):
self.flag = False
self.abc = None
super(demo_script, self).setUp()
def test_before(self):
self.abc = 5
## reboot occurs here and system context is saved
def test_after(self):
if self.abc == 5:
print"Pass"
else:
print"fail"
def tearDown(self):
print"clean"
The test is failing as it is unable to access the variable self.abc.
How can the local variable be accessed in both the tests i.e. test_before() and test_after()?
Test runners such as Nose don't ensure test method running sequence. So if you set self.abc in test_before it's not sure that test_after is run after test_before.
Anyway, sharing state between test methods other than defined in setUp is a bad idea -- the tests should be isolated.
So merge the test methods into one.
I am currently attempting to write unit tests for my Main.py's main() function
Here is a simplified version of my Main.py:
from Configuration import Configuration # Configuration.py is a file in the same dir
def main():
try:
Configuration('settings.ini')
except:
sys.exit(1) # Test path1
sys.exit(0) # Test path2
if __name__ == '__main__':
main()
In my Unit Tests\MainUnitTests.py I want to import ..\Main.py and fake the Configuration class in such a way that I can hit Test path1 and Test path2
I found that i can assert sys.exit() with the following:
with self.assertRaises(SystemExit) as cm:
main()
self.assertEqual(cm.exception.code, 1)
but I am having trouble overriding the from Configuration import Configuration
Thoughts?
So far I have tried the following within Unit Tests\MainUnitTests.py:
class FakeFactory(object):
def __init__(self, *a):
pass
sys.modules['Configuration'] = __import__('FakeFactory')
class Configuration(FakeFactory):
pass
Another example for demonstration:
foo.py:
from bar import a,b
x = a()
class a(object):
def __init__(self):
self.q = 2
y = a()
print x.q, y.q # prints '1 2' as intended
b() # I want this to print 2 without modifying bar.py
bar.py:
class a(object):
def __init__(self):
self.q = 1
def b():
t = a()
print t.q
when you use the import
from bar import a
it import the name directly into the module, so monkeypatching bar won't help, you need to override a directly in the main file:
def fake_a():
print("OVERRIDEN!")
main.a = fake_a
Do know that unittest has helper functions for this in the mock subpackage, I believe you could do something like:
from unittest.mock import patch
...
with patch("main.a", fake_a) as mock_obj: #there are additional things you can do with the mock_obj
do_stuff()
This would work in your first example with configuration since the class that needs to be patched is not used in the global scope although foo uses bar.a as soon as it is loaded so you would need to patch it before even loading foo:
from unittest.mock import patch
...
with patch("bar.a", fake_a) as mock_obj: #there are additional things you can do with the mock_obj
import foo #now when it loads it will be loaded with the patched name
However in this case foo.a would not be reverted at the end of the with block because it can't be caught by unittest... I really hope your actual use case doesn't use the stuff to be patched at module level.
I cannot figure out how to run single unit tests from within a python script and collect the results.
Scenario: I have a battery of tests that check various methods producing various statistical distributions of different objects. The tests sometimes fail, as they should given that I am basically checking for particular kinds of randomness. I would like to run the tests repeatedly from a script or even from the interpreter and collect results for further analysis.
Suppose I have a module myTest.py with:
class myTest(unittest.TestCase):
def setup(self):
...building objects, etc....
def testTest1(self):
..........
def testTest2(self):
..........
Basically I need to:
run the setup method
run testTest1 (say), 100 times
collect the failures
return the failures
The closest I got to was (using code from a similar question):
from unittest import TextTestRunner, TestSuite
runner = TextTestRunner(verbosity = 2)
tests = ['testTest1']
suite = unittest.TestSuite(map(myTest, tests))
runner.run(suite)
But this does not work, because:
runner.run(suite) does not run the setup method
and
I cannot catch the exception it throws when testTest1 fails
you simply need to add the test that you want to run multiple times to the suite.
Here is a complete code example. You can also see this code running in an interactive Python console to prove that it does actually work.
import unittest
import random
class NullWriter(object):
def write(*_, **__):
pass
def flush(*_, **__):
pass
SETUP_COUNTER = 0
class MyTestCase(unittest.TestCase):
def setUp(self):
global SETUP_COUNTER
SETUP_COUNTER += 1
def test_one(self):
self.assertTrue(random.random() > 0.3)
def test_two(self):
# We just want to make sure this isn't run
self.assertTrue(False, "This should not have been run")
def suite():
tests = []
for _ in range(100):
tests.append('test_one')
return unittest.TestSuite(map(MyTestCase, tests))
results = unittest.TextTestRunner(stream=NullWriter()).run(suite())
print dir(results)
print 'setUp was run', SETUP_COUNTER, 'times'