Develop Reference

Multiple pytest sessions during tests run

I am writing tests using pytest and pytest-xdist and I want to run pytest_sessionstart before all workers start running and pytest_sessionfinish when they are done.
I found this solution: link, but this is not working as expected. There are multiple sessions starting and finishing during test run. Hence multiple cleanups are done, which cause the tests to fail (it cleans tmp directory and tests fails with FileNotFoundError).
I added code to write to file when session is started and once it is finished. The log looks like this:
init 0x00007f988f5ee120
worker init gw0
...
worker init gw7
init 0x00007f229cdac2e0
cleanup 0x00007f229cdac2e0 0
init 0x00007f1a31a4e2e0
cleanup 0x00007f1a31a4e2e0 0
worker done gw0
...
worker done gw4
cleanup 0x00007f988f5ee120 1
As you can see there are some session starting after all workers started and before they are done.
My code looks like this:
def pytest_sessionstart(session: pytest.Session):
if hasattr(session.config, 'workerinput'):
# log to file 'worker init {id}'
return
# log to file 'init {sess id}'
# do some stuff
def pytest_sessionfinish(session: pytest.Session, exitstatus: int):
if hasattr(session.config, 'workerinput'):
# log to file 'worker done {id}'
return
# log to file 'cleanup {sess id} {exitstatus}'
# do some stuff

It turned out that vs-code starts pytest in background with --collect-only argument. Those session were not filtered as they were not worker sessions and they performed init / cleanup.
The solution is to add checking if argument --collect-only is present.
Code:
def pytest_sessionfinish(session: pytest.Session, exitstatus: int):
if hasattr(session.config, 'workerinput'):
return
if '--collect-only' in session.config.invocation_params.args:
return
# do some stuff

Pytest parametrize run after the last iteration

When using #pytest.mark.parametrize('arg', param) is there a way to find out if the last item in param is being run? The reason I'm asking is I want to run a cleanup function unique to that test that should only run after the very last iteration of param.
param = [(1, 'James'), (2, 'John')]
#pytest.mark.parametrize('id, user', param)
def test_myfunc(id, user):
# Do something
print(user)
# Run this only after the last param which would be after (2, 'John')
print('All done!')
I can run a conditional which checks for the value of param but I was just wondering if pytest has a way for this.

You'll need to perform this logic within a pytest hook, specifically the pytest_runtest_teardown hook.
Assuming your test looks like the following,
import pytest
param = [(1, 'James'), (2, 'John')]
#pytest.mark.parametrize('id, user', param)
def test_myfunc(id, user):
print(f"Iteration number {id}")
In the root of your test folder, create a conftest.py file and place the following,
func_of_interest = "test_myfunc"
def pytest_runtest_teardown(item, nextitem):
curr_name = item.function.__qualname__
# check to see it is the function we want
if curr_name == func_of_interest:
# check to see if there are any more functions after this one
if nextitem is not None:
next_name = nextitem.function.__qualname__
else:
next_name = "random_name"
# check to see if the next item is a different function
if curr_name != next_name:
print("\nPerform some teardown once")
Then when we run it, it produces the following output,
===================================== test session starts ======================================
platform darwin -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
cachedir: .pytest_cache
rootdir: ***
collected 2 items
test_grab.py::test_myfunc[1-James] Iteration number 1
PASSED
test_grab.py::test_myfunc[2-John] Iteration number 2
PASSED
Perform some teardown once
As we can see, the teardown logic was called exactly once, after the final iteration of the test call.

fabric 2.3 parallel execution of sudo commands

I am trying to use fabric 2.3 to run few tasks that require sudo on some servers . My main goal here was to parallelize the operation so i thought of using ThreadingGroup class of fabric api however it does not support sudo.
Below is my code for sake of clarity
#!/usr/bin/env python
from fabric import ThreadingGroup, Config
from getpass import getpass
sudo_pass = getpass("Enter your sudo password: ")
sudo_config = Config(overrides={'sudo': {'password': sudo_pass}})
server_pool = ThreadingGroup("test1", "test2",
config=sudo_config)
result = server_pool.sudo('cat /etc/shadow', hide='stderr')
print(result)
Now this does not work as mentioned above because ThreadingGroup does not support all of the methods that Connection class support.
I can run sudo on multiple servers by iterating over the individual hosts and then creating connection for each but that isn't efficient.
So is there a way to make this parallel with fabric 2.3 ? I have gone through the official documentation as well but did not find anything.
Further i did some more testing on it following the official documentation and it seems like ThreadingGroup achieves parallelism only if you run it like below
fabric.ThreadingGroup('test1', 'test2').run('uname -s')
however if you run it like below it does not run in parallel
def run_task(c):
c.run('uname -s')
for cxn in fabric.ThreadingGroup('test1', 'test2'):
run_task(cxn)
So it looks like there isn't much flexible support for parallelism in fabric 2.3 as of now and i might have to switch back to fabric version 1 only.

It seems some functions are not implemented in current version (2.4 so far).
An optional solution is to add some code in its source file.
You can find the install path of fabric, and edit group.py.
First, add this function in group.py:
def thread_worker_sudo(cxn, queue, args, kwargs):
result = cxn.sudo(*args, **kwargs)
# TODO: namedtuple or attrs object?
queue.put((cxn, result))
and then add sudo function in class ThreadingGroup:
class ThreadingGroup(Group):
.... original ThreadingGroup
def sudo(self, *args, **kwargs):
results = GroupResult()
queue = Queue()
threads = []
for cxn in self:
my_kwargs = dict(cxn=cxn, queue=queue, args=args, kwargs=kwargs)
thread = ExceptionHandlingThread(
target=thread_worker_sudo, kwargs=my_kwargs
)
threads.append(thread)
for thread in threads:
thread.start()
for thread in threads:
# TODO: configurable join timeout
# TODO: (in sudo's version) configurability around interactive
# prompting resulting in an exception instead, as in v1
thread.join()
# Get non-exception results from queue
while not queue.empty():
# TODO: io-sleep? shouldn't matter if all threads are now joined
cxn, result = queue.get(block=False)
# TODO: outstanding musings about how exactly aggregate results
# ought to ideally operate...heterogenous obj like this, multiple
# objs, ??
results[cxn] = result
# Get exceptions from the threads themselves.
# TODO: in a non-thread setup, this would differ, e.g.:
# - a queue if using multiprocessing
# - some other state-passing mechanism if using e.g. coroutines
# - ???
excepted = False
for thread in threads:
wrapper = thread.exception()
if wrapper is not None:
# Outer kwargs is Thread instantiation kwargs, inner is kwargs
# passed to thread target/body.
cxn = wrapper.kwargs["kwargs"]["cxn"]
results[cxn] = wrapper.value
excepted = True
if excepted:
raise GroupException(results)
return results
I just copy the code of function run and replace this line
thread = ExceptionHandlingThread(
target=thread_worker, kwargs=my_kwargs
)
with
thread = ExceptionHandlingThread(
target=thread_worker_sudo, kwargs=my_kwargs
)
It works for me like that:
def test_sudo(group):
group.sudo('whoami', user='root')
$ python fabfile.py
root
root
root
root
root
However, I am not sure, it will work well for all situations.

Python - CherryPy testing - set session data?

When running a pytest unit test against a CherryPy server, using a cherrypy.helper.CPWebCase sub-class, how can I set data for the session object? I tried just calling cherrypy.session['foo']='bar' like I would if I was really in a cherrypy call, but that just gave an "AttributeError: '_Serving' object has no attribute 'session'"
For reference, a test case might look something like this (pulled from the CherryPy Docs with minor edits):
import cherrypy
from cherrypy.test import helper
from MyApp import Root
class SimpleCPTest(helper.CPWebCase):
def setup_server():
cherrypy.tree.mount(Root(), "/", {'/': {'tools.sessions.on': True}})
setup_server = staticmethod(setup_server)
def check_two_plus_two_equals_four(self):
#<code to set session variable to 2 here>
# This is the question: How do I set a session variable?
self.getPage("/")
self.assertStatus('200 OK')
self.assertHeader('Content-Type', 'text/html;charset=utf-8')
self.assertBody('4')
And the handler might look something like this (or anything else, it makes no difference whatsoever):
class Root:
#cherrypy.expose
def test_handler(self):
#get a random session variable and do something with it
number_var=cherrypy.session.get('Number')
# Add two. This will fail if the session variable has not been set,
# Or is not a number
number_var = number_var+2
return str(number_var)
It's safe to assume that the config is correct, and sessions work as expected.
I could, of course, write a CherryPy page that takes a key and value as arguments, and then sets the specified session value, and call that from my test code (EDIT: I've tested this, and it does work). That, however, seems kludgy, and I'd really want to limit it to testing only somehow if I went down that road.

What you are trying to achieve is usually referred as mocking.
While running tests you'd usually want to 'mock' some of resources you access with dummy objects having same interface (duck typing). This may be achieved with monkey patching. To simplify this process you may use unittest.mock.patch as either context manager or method/function decorator.
Please find below the working example with context manager option:
==> MyApp.py <==
import cherrypy
class Root:
_cp_config = {'tools.sessions.on': True}
#cherrypy.expose
def test_handler(self):
# get a random session variable and do something with it
number_var = cherrypy.session.get('Number')
# Add two. This will fail if the session variable has not been set,
# Or is not a number
number_var = number_var + 2
return str(number_var)
==> cp_test.py <==
from unittest.mock import patch
import cherrypy
from cherrypy.test import helper
from cherrypy.lib.sessions import RamSession
from MyApp import Root
class SimpleCPTest(helper.CPWebCase):
#staticmethod
def setup_server():
cherrypy.tree.mount(Root(), '/', {})
def test_check_two_plus_two_equals_four(self):
# <code to set session variable to 2 here>
sess_mock = RamSession()
sess_mock['Number'] = 2
with patch('cherrypy.session', sess_mock, create=True):
# Inside of this block all manipulations with `cherrypy.session`
# actually access `sess_mock` instance instead
self.getPage("/test_handler")
self.assertStatus('200 OK')
self.assertHeader('Content-Type', 'text/html;charset=utf-8')
self.assertBody('4')
Now you may safely run test as follows:
$ py.test -sv cp_test.py
============================================================================================================ test session starts =============================================================================================================
platform darwin -- Python 3.5.2, pytest-2.9.2, py-1.4.31, pluggy-0.3.1 -- ~/.pyenv/versions/3.5.2/envs/cptest-pyenv-virtualenv/bin/python3.5
cachedir: .cache
rootdir: ~/src/cptest, inifile:
collected 2 items
cp_test.py::SimpleCPTest::test_check_two_plus_two_equals_four PASSED
cp_test.py::SimpleCPTest::test_gc <- ../../.pyenv/versions/3.5.2/envs/cptest-pyenv-virtualenv/lib/python3.5/site-packages/cherrypy/test/helper.py PASSED

py.test logging messages and test results/assertions into a single file

I am starting to work with py.test at the moment for a new project. We are provisioning Linux servers and I need to write a script to check the setup and configuration of these servers. I thought that py.test is a good way to implement these tests and it is working quite fine until now.
The problem I face right now is that I need a log file at the end of these tests showing some log messages for each test and the result of the test. For the log messages I use logger:
logging.basicConfig(filename='config_check.log', level=logging.INFO)
pytest.main()
logging.info('all done')
As an example test I have this:
def test_taintedKernel():
logging.info('checking for tainted kernel')
output = runcmd('cat /proc/sys/kernel/tainted')
assert output == '0', 'tainted kernel found'
So in my logfile I would like an output like that:
INFO:root:checking for tainted kernel
ERROR:root:tainted kernel found
INFO:root:next test
INFO:root:successful
INFO:root:all done
But I cannot get the test results into the logfile, instead I get the standard output on stdout after the tests:
======================================= test session starts =======================================
platform linux2 -- Python 2.6.8 -- py-1.4.22 -- pytest-2.6.0
collected 14 items
test_basicLinux.py .............F
============================================ FAILURES =============================================
_______________________________________ test_taintedKernel ________________________________________
def test_taintedKernel():
logging.info('checking for tainted kernel')
output = runcmd('cat /proc/sys/kernel/tainted')
> assert output == '0', 'tainted kernel found'
E AssertionError: tainted kernel found
test_basicLinux.py:107: AssertionError
=============================== 1 failed, 13 passed in 6.07 seconds ===============================
This may be quite confusing for the users of my script. I tried to get into logger and pytest_capturelog since it was mentioned here quite often but I am for sure doing something wrong since I just don't get it. Maybe just a lack of understanding how this really works. Hope you can give me some hints on this. Please let me know if anything is missing here.
Thanks in advance for your help,
Stephan

pytest's job is to capture output and present it to the operator. So, rather than trying to get pytest to do the logging the way you want it, you can build the logging into your tests.
Python's assert command just takes a truth value, and a message. So, instead of using a bare assert in your tests, you can write a small function that does the logging if the value is false (which is the same condition that triggers the assert to fail), then calls the assert, so that you get the logging you want, plus the assert-driven behavior that creates the console output.
Here's a small test file using such a function:
# test_foo.py
import logging
def logAssert(test,msg):
if not test:
logging.error(msg)
assert test,msg
def test_foo():
logging.info("testing foo")
logAssert( 'foo' == 'foo', "foo is not foo")
def test_foobar():
logging.info("testing foobar")
logAssert( 'foobar' == 'foo', "foobar is not foo")
Here's the test runner, very similar to yours:
# runtests.py
import logging
import pytest
logging.basicConfig(filename='config_check.log', level=logging.INFO)
logging.info('start')
pytest.main()
logging.info('done')
Here's the output:
# python runtests.py
==== test session starts ========================
platform linux2 -- Python 2.6.6 -- py-1.4.22 -- pytest-2.6.0
collected 2 items
test_foo.py .F
========== FAILURES ============================
________ test_foobar __________________________
def test_foobar():
logging.info("testing foobar")
> logAssert( 'foobar' == 'foo', "foobar is not foo")
test_foo.py:14:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test = False, msg = 'foobar is not foo'
def logAssert(test,msg):
if not test:
logging.error(msg)
> assert test,msg
E AssertionError: foobar is not foo
test_foo.py:6: AssertionError ==== 1 failed, 1 passed in 0.02 seconds =======
And here's the log that gets written:
# cat config_check.log
INFO:root:start
INFO:root:testing foo
INFO:root:testing foobar
ERROR:root:foobar is not foo
INFO:root:done

Since version 3.3, pytest supports live logging to terminal and file. Example test module:
import logging
import os
def test_taintedKernel():
logging.info('checking for tainted kernel')
output = os.system('cat /proc/sys/kernel/tainted')
assert output == 0, 'tainted kernel found'
Configure of logging to file can be done in pytest.ini:
[pytest]
log_file = my.log
log_file_level = DEBUG
log_file_format = %(asctime)s [%(levelname)8s] %(message)s (%(filename)s:%(lineno)s)
log_file_date_format=%Y-%m-%d %H:%M:%S
Running the test yields as usual:
$ pytest
======================================================= test session starts ========================================================
...
collected 1 item
test_spam.py . [100%]
===================================================== 1 passed in 0.01 seconds =====================================================
Now check the written log file:
$ cat my.log
2019-07-12 23:51:41 [ INFO] checking for tainted kernel (test_spam.py:6)
For more examples of emitting live logs to both terminal and log file, check out my answer to Logging within py.test tests
.
Reference: Live Logs section in pytest docs.