I am new the usage of more "advanced" python. I decided to learn and implements unit testing for all my scripts.
Here is my issue :
I have a function from an external package I have made myself called gendiag. This package has a function "notify" that will send an email to a set recipient, defined in a config file, but leave the subject and the message as parameters :
gendiag.py :
import subprocess
#...
try:
configfile = BASE_PACKAGE_DIR.joinpath('config.yml')
with open(configfile, "r") as f:
config = yaml.safe_load(f)
except Exception as e:
print("Uh Oh!")
def notify(subject,message):
adress = config['mail']['nipt']
command = f'echo -e "{message}" | mail -s "{subject}" "{adress}"'
subprocess.run(command, shell=True)
In an other project called watercheck which import gendiag, I am using this function to get some info from every directory and send it as an email :
watercheck.py :
import gendiag as gdl
#...
def scan_water_metrics(indir_list):
for dir in indir_list:
#Do the things, this is a dummy example to simplify
some_info = os.path.basename(dir)
subject="Houlala"
message="I have parsed this directory, amazing.\n"
message += some_info
gdl.notify(subject,message)
Now in my test_watercheck.py, I would like to test that this function works with already created dummy data. But of course, I don't want to send an email to the rest of the world everytime I use pytest to see if email sending works. Instead, I was thinking I would create the following mock function in conftest.py :
conftest.py :
import gendiag
#pytest.fixture
def mock_scan_water_metrics():
mck = gendiag
mck.notify = mock.Mock(
return_value=subprocess.check_output(
"echo 'Hmm interesting' | mail -s Test my_own_email#gmule.com", shell=True
)
)
return mck
And then pass this mock to my test function in test_watercheck.py :
test_watercheck.py :
import gendiag
import pytest
from unittest import mock
from src import watercheck
def test_scan_water_metrics(mock_scan_water_metrics):
indir_list = ["tests/outdir/CORRECT_WATERCHECK","tests/outdir/BADWATER_WATERCHECK"]
water_check.scan_water_metrics(indir_list)
So this works in the sense that I am able to overwrite the email, but I would still like to test that some_info is collected properly, and for that I need to pass subject and message to the mock function. And this is the very confusing part for me. I don't doubt the answer is probably out there, but my understanding of the topic is too limited for me to find it out, or even formulate properly my question.
I have tried to read more about the object mock.Mock to see if I could collect the parameters somewhere, I have tried the following to see if I could access the parameters :
My attempt in conftest.py :
#pytest.fixture
#mock.patch("gendiag.notify_nipt")
def mock_scan_water_metrics(event):
print("Event : "+event)
args, kwargs = event.call_args
print("Args : "+args)
print("Kwargs : "+kwargs)
mck = gendiag
mck.notify = mock.Mock(
return_value=subprocess.check_output(
"echo 'Hmm interesting' | mail -s Test my_own_email#gmule.com", shell=True
)
)
return mck
I was hoping somewhere in args, I would find my two parameters, but when starting pytest, I have an error that the module "gendiag" does not exists, even though I had imported it everywhere just to be sure. I imagine the line causing it is the decorator here : #mock.patch("gendiag.notify_nipt"). I have tried with #mock.patch("gdl.notify_nipt") as well, as it is how it is called in the main function, with no success.
To be honest, I am really not sure where to go from here, it's getting too complex for me for now. How can I simply access to the parameters given to the function before it is decorated by pytest ?
Related
So I m loading test data from a different file based on the environment I'm meant to run the tests:
TestData/DevTestData.py contains:
data = {"accessToken": "Random Access Token"}
Then I have set up in conftest.py file:
To get the CLI parameter:
def pytest_addoption(parser):
parser.addoption('--environment', action='store')
Then to load the data I use LazySettings from simple-settings as a fixture:
#pytest.fixture
def testData(request):
return LazySettings("TestData." + request.config.getoption("--environment") + "TestData")
The test class looks like this:
class Test_User_Current():
userCurrentFacadeInstance = userCurrentGetAPI_Facade.User_Current_API_Facade()
def test_succesfull_request(self, environmentConfigs, testData):
self.userCurrentFacadeInstance.getRequest_user_current_API(environmentConfigs, testData).\
validateSuccessfullStatusCode().\
validateJsonContents()
CLI is:
py.test --environment Dev
My problem is, I have to pass "testData" for every test method rather then passing it to User_Current_API_Facade()'s constructor, and I cant do that for some reason, if I'm passing it to the constructor and not the test method (test_succesfull_request()) it does not work.
Do you guys have any idea on how to do this in a better way?
We have unit tests running via Pytest, which use a custom decorator to start up a context-managed mock echo server before each test, and provide its address to the test as an extra parameter. This works on Python 2.
However, if we try to run them on Python 3, then Pytest complains that it can't find a fixture matching the name of the extra parameter, and the tests fail.
Our tests look similar to this:
#with_mock_url('?status=404&content=test&content-type=csv')
def test_file_not_found(self, url):
res_id = self._test_resource(url)['id']
result = update_resource(None, res_id)
assert not result, result
self.assert_archival_error('Server reported status error: 404 Not Found', res_id)
With a decorator function like this:
from functools import wraps
def with_mock_url(url=''):
"""
Start a MockEchoTestServer and call the decorated function with the server's address prepended to ``url``.
"""
def decorator(func):
#wraps(func)
def decorated(*args, **kwargs):
with MockEchoTestServer().serve() as serveraddr:
return func(*(args + ('%s/%s' % (serveraddr, url),)), **kwargs)
return decorated
return decorator
On Python 2 this works; the mock server starts, the test gets a URL similar to "http://localhost:1234/?status=404&content=test&content-type=csv", and then the mock is shut down afterward.
On Python 3, however, we get an error, "fixture 'url' not found".
Is there perhaps a way to tell Python, "This parameter is supplied from elsewhere and doesn't need a fixture"? Or is there, perhaps, an easy way to turn this into a fixture?
You can use url as args parameter
#with_mock_url('?status=404&content=test&content-type=csv')
def test_file_not_found(self, *url):
url[0] # the test url
Looks like Pytest is content to ignore it if I add a default value for the injected parameter, to make it non-mandatory:
#with_mock_url('?status=404&content=test&content-type=csv')
def test_file_not_found(self, url=None):
The decorator can then inject the value as intended.
consider separating the address from the service of the url. Using marks and changing fixture behavior based on the presence of said marks is clear enough. Mock should not really involve any communication, but if you must start some service, then make it separate from
with_mock_url = pytest.mark.mock_url('http://www.darknet.go')
#pytest.fixture
def url(request):
marker = request.get_closest_marker('mock_url')
if marker:
earl = marker.args[0] if args else marker.kwargs['fake']
if earl:
return earl
try:
#
earl = request.param
except AttributeError:
earl = None
return earl
#fixture
def server(request):
marker = request.get_closest_marker('mock_url')
if marker:
# start fake_server
#with_mock_url
def test_resolve(url, server):
server.request(url)
As per the pytest documentation, it possible to override the default temporary directory setting as follows:
py.test --basetemp=/base_dir
When the tmpdir fixture is then used in a test ...
def test_new_base_dir(tmpdir):
print str(tmpdir)
assert False
... something like the following would then be printed to the screen:
/base_dir/test_new_base_dir_0
This works as intended and for certain use cases can be very useful.
However, I would like to be able to change this setting on a per-test (or perhaps I should say a "per-fixture") basis. Is such a thing possible?
I'm close to just rolling my own tmpdir based on the code for the original, but would rather not do this -- I want to build on top of existing functionality where I can, not duplicate it.
As an aside, my particular use case is that I am writing a Python module that will act on different kinds of file systems (NFS4, etc), and it would be nice to be able to yield the functionality of tmpdir to be able to create the following fixtures:
def test_nfs3_stuff(nfs3_tmpdir):
... test NFS3 functionality
def test_nfs4_stuff(nfs4_tmpdir):
... test NFS4 functionality
In the sources of TempdirFactory the .config.option.basetemp is used as the attribute to store the basetemp. So you can directly set it before the usage:
import pytest
import time
import os
def mktemp_db(tmpdir_factory, db):
basetemp = None
if 'PYTEST_TMPDIR' in os.environ:
basetemp = os.environ['PYTEST_TMPDIR']
if basetemp:
tmpdir_factory.config.option.basetemp = basetemp
if db == "db1.db":
tmpdb = tmpdir_factory.mktemp('data1_').join(db)
elif db == "db2.db":
tmpdb = tmpdir_factory.mktemp('data2_').join(db)
return tmpdb
#pytest.fixture(scope='session')
def empty_db(tmpdir_factory):
tmpdb = mktemp_db(tmpdir_factory, 'db1.db')
print("* " + str(tmpdb))
time.sleep(5)
return tmpdb
#pytest.fixture(scope='session')
def empty_db2(tmpdir_factory):
tmpdb = mktemp_db(tmpdir_factory, 'db2.db')
print("* " + str(tmpdb))
time.sleep(5)
return tmpdb
def test_empty_db(empty_db):
pass
def test_empty_db2(empty_db2):
pass
-
>set PYTEST_TMPDIR=./tmp
>python.exe -m pytest -q -s test_my_db.py
* c:\tests\tmp\data1_0\db1.db
.* c:\tests\tmp\data2_0\db2.db
.
2 passed in 10.03 seconds
There didn't appear to be a nice solution to the problem as posed in the question so I settled on making two calls to py.test:
Passing in a different --basetemp for each.
Marking (using #pytest.mark.my_mark) which tests needed the special treatment of using a non-standard basetemp.
Passing -k my_mark or -k-my_mark into each call.
I don't know why I'm just not getting this, but I want to use mock in Python to test that my functions are calling functions in ftplib.FTP correctly. I've simplified everything down and still am not wrapping my head around how it works. Here is a simple example:
import unittest
import ftplib
from unittest.mock import patch
def download_file(hostname, file_path, file_name):
ftp = ftplib.FTP(hostname)
ftp.login()
ftp.cwd(file_path)
class TestDownloader(unittest.TestCase):
#patch('ftplib.FTP')
def test_download_file(self, mock_ftp):
download_file('ftp.server.local', 'pub/files', 'wanted_file.txt')
mock_ftp.cwd.assert_called_with('pub/files')
When I run this, I get:
AssertionError: Expected call: cwd('pub/files')
Not called
I know it must be using the mock object since that is a fake server name, and when run without patching, it throws a "socket.gaierror" exception.
How do I get the actual object the fuction is running? The long term goal is not having the "download_file" function in the same file, but calling it from a separate module file.
When you do patch(ftplib.FTP) you are patching FTP constructor. dowload_file() use it to build ftp object so your ftp object on which you call login() and cmd() will be mock_ftp.return_value instead of mock_ftp.
Your test code should be follow:
class TestDownloader(unittest.TestCase):
#patch('ftplib.FTP', autospec=True)
def test_download_file(self, mock_ftp_constructor):
mock_ftp = mock_ftp_constructor.return_value
download_file('ftp.server.local', 'pub/files', 'wanted_file.txt')
mock_ftp_constructor.assert_called_with('ftp.server.local')
self.assertTrue(mock_ftp.login.called)
mock_ftp.cwd.assert_called_with('pub/files')
I added all checks and autospec=True just because is a good practice
Like Ibrohim's answer, I prefer pytest with mocker.
I have went a bit further and have actually wrote a library which helps me to mock easily. Here is how to use it for your case.
You start by having your code and a basic pytest function, with the addition of my helper library to generate mocks to modules and the matching asserts generation:
import ftplib
from mock_autogen.pytest_mocker import PytestMocker
def download_file(hostname, file_path, file_name):
ftp = ftplib.FTP(hostname)
ftp.login()
ftp.cwd(file_path)
def test_download_file(mocker):
import sys
print(PytestMocker(mocked=sys.modules[__name__],
name=__name__).mock_modules().prepare_asserts_calls().generate())
download_file('ftp.server.local', 'pub/files', 'wanted_file.txt')
When you run the test for the first time, it would fail due to unknown DNS, but the print statement which wraps my library would give us this valuable input:
...
mock_ftplib = mocker.MagicMock(name='ftplib')
mocker.patch('test_29817963.ftplib', new=mock_ftplib)
...
import mock_autogen
...
print(mock_autogen.generator.generate_asserts(mock_ftplib, name='mock_ftplib'))
I'm placing this in the test and would run it again:
def test_download_file(mocker):
mock_ftplib = mocker.MagicMock(name='ftplib')
mocker.patch('test_29817963.ftplib', new=mock_ftplib)
download_file('ftp.server.local', 'pub/files', 'wanted_file.txt')
import mock_autogen
print(mock_autogen.generator.generate_asserts(mock_ftplib, name='mock_ftplib'))
This time the test succeeds and I only need to collect the result of the second print to get the proper asserts:
def test_download_file(mocker):
mock_ftplib = mocker.MagicMock(name='ftplib')
mocker.patch(__name__ + '.ftplib', new=mock_ftplib)
download_file('ftp.server.local', 'pub/files', 'wanted_file.txt')
mock_ftplib.FTP.assert_called_once_with('ftp.server.local')
mock_ftplib.FTP.return_value.login.assert_called_once_with()
mock_ftplib.FTP.return_value.cwd.assert_called_once_with('pub/files')
If you would like to keep using unittest while using my library, I'm accepting pull requests.
I suggest using pytest and pytest-mock.
from pytest_mock import mocker
def test_download_file(mocker):
ftp_constructor_mock = mocker.patch('ftplib.FTP')
ftp_mock = ftp_constructor_mock.return_value
download_file('ftp.server.local', 'pub/files', 'wanted_file.txt')
ftp_constructor_mock.assert_called_with('ftp.server.local')
assert ftp_mock.login.called
ftp_mock.cwd.assert_called_with('pub/files')
I am developing ryu app. The app is basically a python script. The ryu apps are invoked by ryu-manager like this
ryu-manager {filename}
There are certain parameters that are taken by ryu-manager. I want to know if there is a way i could pass arguments to my file?
argparse module of python to parse command line options is there but am not sure it will work as all arguments I provide are used by ryu-manager not my script.
Any help would be appreciated.
I haven't tried this, but:
"Ryu currently uses oslo.config.cfg for command-line parsing.
(ryu/contrib/oslo/config).
There are several examples in the tree. ryu/app/tunnel_port_updater.py"
from
http://comments.gmane.org/gmane.network.ryu.devel/2709
see also
https://github.com/osrg/ryu/blob/master/ryu/app/tunnel_port_updater.py
The Ryu 'getting started' page simply suggests:
ryu-manager [--flagfile <path to configuration file>] [generic/application specific options…]
http://www.osrg.net/ryu/_sources/getting_started.txt
Doing so is a 4-step process. I'll show an example where you read parameters and then print them, but you could assign them to variables or do whatever you would like to by referencing this process.
Create a .conf file (e.g. params.conf)
#Example Conf File
[DEFAULT]
param1_int = 42
param2_str = "You read my data :)"
param3_list = 1,2,3
param4_float = 3.14
Add the following code to your __init__ method. I did this to the simple_switch_13.py which comes with ryu:
from ryu import cfg
:
:
class SimpleSwitch13(app_manager.RyuApp):
def __init__(self, *args, **kwargs):
super(SimpleSwitch13, self).__init__(*args, **kwargs)
:
CONF = cfg.CONF
CONF.register_opts([
cfg.IntOpt('param1_int', default=0, help = ('The ultimate answer')),
cfg.StrOpt('param2_str', default='default', help = ('A string')),
cfg.ListOpt('param3_list', default = None, help = ('A list of numbers')),
cfg.FloatOpt('param4_float', default = 0.0, help = ('Pi? Yummy.'))])
print 'param1_int = {}'.format(CONF.param1_int))
print 'param2_str = {}'.format(CONF.param2_str))
print 'param3_list = {}'.format(CONF.param3_list))
print 'param4_float = {}'.format(CONF.param4_float))
Run Script
ryu-manager paramtest.py --config-file [PATH/TO/FILE/params.conf]
Profit
I referenced the following when putting together my answer, they can provide more detail (such as the oslo.config stuff, which I had never heard of prior to running into this issue).
More info on oslo.config: http://www.giantflyingsaucer.com/blog/?p=4822
Ryu email chain on this issue: https://sourceforge.net/p/ryu/mailman/message/33410077/
I have not found a way to pass arguments to a ryu controller. One way that I have used to get around this is to pass arguments as an environment variable. For example, I have a program which invokes ryu-manager and needs to pass a parameter to the app. I do this as follows: ARG=value ryu-manager app.py