Related
I'm trying to test some code that operates on a file, and I can't seem to get my head around how to replace using a real file with mock and io.StringIO
My code is pretty much the following:
class CheckConfig(object):
def __init__(self, config):
self.config = self._check_input_data(config)
def _check_input_data(self, data):
if isinstance(data, list):
return self._parse(data)
elif os.path.isfile(data):
with open(data) as f:
return self._parse(f.readlines())
def _parse(self, data):
return data
I have a class that can take either a list or a file, if it's a file it opens it and extracts the contents into a list, and then does what it needs to do to the resulting list.
I have a working test as follows:
def test_CheckConfig_with_file():
config = 'config.txt'
expected = parsed_file_data
actual = CheckConfig(config).config
assert expected == actual
I want to replace the call to the filesystem. I have tried replacing the file with io.StringIO but I get a TypeError from os.path.isfile() as it's expecting either a string, bytes or int. I also tried mocking the isfile method like so:
#mock.patch('mymodule.os.path')
def test_CheckConfig_with_file(mock_path):
mock_path.isfile.return_value = True
config = io.StringIO('data')
expected = parsed_file_data
actual = CheckConfig(config).config
assert expected == actual
but I still get the same TypeError as the _io.StringIO type is causing the exception before isfile gets a chance to return something.
How can I get os.path.isfile to return True, when I pass it a fake file? Or is this a suggestion I should change my code?
Just mock out both os.path.isfile and the open() call, and pass in a fake filename (you are not expected to pass in an open file, after all).
The mock library includes a utility for the latter: mock_open():
#mock.patch('os.path.isfile')
def test_CheckConfig_with_file(mock_isfile):
mock_isfile.return_value = True
config_data = mock.mock_open(read_data='data')
with mock.patch('mymodule.open', config_data) as mock_open:
expected = parsed_file_data
actual = CheckConfig('mocked/filename').config
assert expected == actual
This causes the if isinstance(data, list): test to be false (because data is a string instead), followed by the elif os.path.isfile(data): returning True, and the open(data) call to use your mocked data from the mock_open() result.
You can use the mock_open variable to assert that open() was called with the right data (mock_open. assert_called_once_with('mocked/filename') for example).
Demo:
>>> import os.path
>>> from unittest import mock
>>> class CheckConfig(object):
... def __init__(self, config):
... self.config = self._check_input_data(config)
... def _check_input_data(self, data):
... if isinstance(data, list):
... return self._parse(data)
... elif os.path.isfile(data):
... with open(data) as f:
... return self._parse(f.readlines())
... def _parse(self, data):
... return data
...
>>> with mock.patch('os.path.isfile') as mock_isfile:
... mock_isfile.return_value = True
... config_data = mock.mock_open(read_data='line1\nline2\n')
... with mock.patch('__main__.open', config_data) as mock_open:
... actual = CheckConfig('mocked/filename').config
...
>>> actual
['line1\n', 'line2\n']
>>> mock_open.mock_calls
[call('mocked/filename'),
call().__enter__(),
call().readlines(),
call().__exit__(None, None, None)]
In case you end up here wondering how to solve this using the pytest-mock library, here is how you do it:
def test_open(mocker):
m = mocker.patch('builtins.open', mocker.mock_open(read_data='bibble'))
with open('foo') as h:
result = h.read()
m.assert_called_once_with('foo')
assert result == 'bibble'
This code example was found (but had to be adjusted) here.
Q: Is is possible to create a format string using Python 3.5's string formatting syntax to left truncate?
Basically what I want to do is take a git SHA:
"c1e33f6717b9d0125b53688d315aff9cf8dd9977"
And using only a format string, get the display only the right 8 chars:
"f8dd9977"
Things Ive tried:
Invalid Syntax
>>> "{foo[-8:]}".format(foo="c1e33f6717b9d0125b53688d315aff9cf8dd9977")
>>> "{foo[-8]}".format(foo="c1e33f6717b9d0125b53688d315aff9cf8dd9977")
>>> "{:8.-8}".format("c1e33f6717b9d0125b53688d315aff9cf8dd9977")
Wrong Result
### Results in first 8 not last 8.
>>> "{:8.8}".format("c1e33f6717b9d0125b53688d315aff9cf8dd9977")
Works but inflexible and cumbersome
### solution requires that bar is always length of 40.
>>> bar="c1e33f6717b9d0125b53688d315aff9cf8dd9977"
>>> "{foo[32]}{foo[33]}{foo[34]}{foo[35]}{foo[36]}{foo[37]}{foo[38]}{foo[39]}".format(foo=bar)
A similar question was asked, but never answered. However mine differs in that I am limited to using only format string, I don't have the ability to change the range of the input param. This means that the following is an unacceptable solution:
>>> bar="c1e33f6717b9d0125b53688d315aff9cf8dd9977"
>>> "{0}".format(bar[-8:])
One more aspect I should clarify... the above explains the simplest form of the problem. In actual context, the problem is expressed more correctly as:
>>> import os
>>> "foo {git_sha}".format(**os.environ)
Where I want to left_truncate "git_sha" environment variable. Admittedly this is a tad more complex than simplest form, but if I can solve the simplest - I can find a way to solve the more complex.
So here is my solution, with thanks to #JacquesGaudin and folks on #Python for providing much guidance...
class MyStr(object):
"""Additional format string options."""
def __init__(self, obj):
super(MyStr, self).__init__()
self.obj = obj
def __format__(self, spec):
if spec.startswith("ltrunc."):
offset = int(spec[7:])
return self.obj[offset:]
else:
return self.obj.__format__(spec)
So this works when doing this:
>>> f = {k: MyStr(v) for k, v in os.environ.items()}
>>> "{PATH:ltrunc.-8}".format(**f)
Subclassing str and overriding the __format__ method is an option:
class CustomStr(str):
def __format__(self, spec):
if spec == 'trunc_left':
return self[-8:]
else:
return super().__format__(spec)
git_sha = 'c1e33f6717b9d0125b53688d315aff9cf8dd9977'
s = CustomStr(git_sha)
print('{:trunc_left}'.format(s))
Better though, you can create a custom Formatter which inherits from string.Formatter and will provide a format method. By doing this, you can override a number of methods used in the process of formatting strings. In your case, you want to override format_field:
from string import Formatter
class CustomFormatter(Formatter):
def format_field(self, value, format_spec):
if format_spec.startswith('trunc_left.'):
char_number = int(format_spec[len('trunc_left.'):])
return value[-char_number:]
return super().format_field(value, format_spec)
environ = {'git_sha': 'c1e33f6717b9d0125b53688d315aff9cf8dd9977'}
fmt = CustomFormatter()
print(fmt.format('{git_sha:trunc_left.8}', **environ))
Depending on the usage, you could put this in a context manager and temporarily shadow the builtin format function:
from string import Formatter
class CustomFormat:
class CustomFormatter(Formatter):
def format_field(self, value, format_spec):
if format_spec.startswith('trunc_left.'):
char_number = int(format_spec[len('trunc_left.'):])
return value[-char_number:]
return super().format_field(value, format_spec)
def __init__(self):
self.custom_formatter = self.CustomFormatter()
def __enter__(self):
self.builtin_format = format
return self.custom_formatter.format
def __exit__(self, exc_type, exc_value, traceback):
# make sure global format is set back to the original
global format
format = self.builtin_format
environ = {'git_sha': 'c1e33f6717b9d0125b53688d315aff9cf8dd9977'}
with CustomFormat() as format:
# Inside this context, format is our custom formatter's method
print(format('{git_sha:trunc_left.8}', **environ))
print(format) # checking that format is now the builtin function
After submitting queries to a service, I get a dictionary or a list back and I want to make sure it's not empty. I using Python 2.7.
I am surprised of not having any assertEmpty method for the unittest.TestCase class instance.
The existing alternatives just don't look right:
self.assertTrue(bool(d))
self.assertNotEqual(d,{})
self.assertGreater(len(d),0)
Is this kind of a missing method in the Python unittest framework? If yes, what would be the most pythonic way to assert that an iterable is not empty?
Empty lists/dicts evaluate to False, so self.assertTrue(d) gets the job done.
Depends exactly what you are looking for.
If you want to make sure the object is an iterable and it is not empty:
# TypeError: object of type 'NoneType' has no len()
# if my_iterable is None
self.assertTrue(len(my_iterable))
If it is OK for the object being tested to be None:
self.assertTrue(my_maybe_iterable)
"Falsy" values in Python
A falsy (sometimes written falsey) value is a value that is considered false when encountered in a Boolean context.
According to the official doc, the following built-in types evaluate to false:
constants defined to be false: None and False.
zero of any numeric type: 0, 0.0, 0j, Decimal(0), Fraction(0, 1)
empty sequences and collections: '', (), [], {}, set(), range(0)
Therefore, it's possible to check for
non-emptiness with assertTrue() and for
emptiness with assertFalse().
(The official doc has a full list of all available assert methods.)
Clean Code
All those assertTrue() and assertFalse() calls are kind of misleading as we wanted to check for emptiness and one needs to know which types evaluate to false to properly understand what's happening in the test.
So, for the sake of clean code and for better readability, we can simply define our own assertEmpty() and assertNotEmpty() methods like so:
def assertEmpty(self, obj):
self.assertFalse(obj)
def assertNotEmpty(self, obj):
self.assertTrue(obj)
Maybe:
self.assertRaises(StopIteration, next(iterable_object))
All the credit for this goes to winklerrr, I am just extending his idea: have importable mixins for when you need assertEmpty or assertNotEmpty:
class AssertEmptyMixin( object ):
def assertEmpty(self, obj):
self.assertFalse(obj)
class AssertNotEmptyMixin( object ):
def assertNotEmpty(self, obj):
self.assertTrue(obj)
Caveat, mixins should go on the left:
class MyThoroughTests( AssertNotEmptyMixin, TestCase ):
def test_my_code( self ):
...
self.assertNotEmpty( something )
Based on #winklerr's answer and #Merk's comment, I extended the idea for checking whether the given object is a Container in the first place.
from typing import Container
def assertContainerEmpty(self, obj: Container) -> None:
"""Asserts whether the given object is an empty container."""
self.assertIsInstance(obj, Container)
self.assertFalse(obj)
def assertContainerNotEmpty(self, obj: Container) -> None:
"""Asserts whether the given object is a non-empty container."""
self.assertIsInstance(obj, Container)
self.assertTrue(obj)
This means that assertEmpty and assertNotEmpty will always fail if the given object is e.g. a float, or an instance of an user-defined class - no matter if it would properly evaluate to True/False.
A slightly different answer to those already proposed... If specific named assertions are absolutely required, you could subclass TestCase and add methods for new assertions there.
from pathlib import Path
from typing import Container
from unittest import TestCase
class BaseTestCase(TestCase):
def assertIsFile(self, path: str, msg: str=None) -> None:
default_msg = 'File does not exist: {0}'.format(path)
msg = msg if msg is not None else default_msg
if not Path(path).resolve().is_file():
raise AssertionError(msg)
def assertIsEmpty(self, obj: Container, msg: str=None) -> None:
default_msg = '{0} is not empty.'.format(obj)
msg = msg if msg is not None else default_msg
self.assertIsInstance(obj, Container, '{0} is not a container.'.format(obj))
if len(obj) > 0:
raise AssertionError(msg)
def assertIsNotEmpty(self, obj: Container, msg: str=None) -> None:
default_msg = '{0} is empty.'.format(obj)
msg = msg if msg is not None else default_msg
self.assertIsInstance(obj, Container, '{0} is not a container.'.format(obj))
if obj is None or len(obj) == 0:
raise AssertionError(msg)
And then subclass the new BaseTestCase class to use the new assertion methods.
class TestApplicationLoadBalancer(_BaseTestCase):
def setUp(self) -> None:
# These assertions will fail.
self.assertIsFile('does-not-exist.txt')
self.assertIsEmpty(['asdf'])
self.assertIsNotEmpty([])
Just like the built-in unittest assertions, you can pass an error message to these if desired.
class TestApplicationLoadBalancer(_BaseTestCase):
def setUp(self) -> None:
# These assertions will fail.
self.assertIsFile('does-not-exist.txt', 'Foo')
self.assertIsEmpty(['asdf'], 'Bar')
self.assertIsNotEmpty([], 'Baz')
I'm a python noob and I'm trying to solve my problems the 'pythonic' way. I have a class, who's __init__ method takes 6 parameters. I need to validate each param and throw/raise an Exception if any fails to validate.
Is this the right way?
class DefinitionRunner:
def __init__(self, canvasSize, flightId, domain, definitionPath, harPath):
self.canvasSize = canvasSize
self.flightId = flightId
self.domain = domain
self.harPath = harPath
self.definitionPath = definitionPath
... bunch of validation checks...
... if fails, raise ValueError ...
If you want the variables to be settable independently of __init__, you could use properties to implement validations in separate methods.
They work only for new style classes though, so you need to define the class as class DefinitionRunner(object)
So for example,
#property
def canvasSize(self):
return self._canvasSize
#canvasSize.setter
def canvasSize(self, value):
# some validation here
self._canvasSize = value
Broadly speaking, that looks like the way you'd do it. Though strictly speaking, you might as well do validation before rather than after assignment, especially if assignment could potentially be time or resource intensive. Also, style convention says not to align assignment blocks like you are.
I would do it like you did it. Except the validating stuff. I would validate in a setter method and use it to set the attributes.
You could do something like this. Make a validator for each type of input. Make a helper function to run validation:
def validate_and_assign(obj, items_d, validators):
#validate all entries
for key, validator in validators.items():
if not validator[key](items_d[key]):
raise ValueError("Validation for %s failed" % (key,))
#set all entries
for key, val in items_d.items():
setattr(obj, key, val)
Which you'd use like this:
class DefinitionRunner:
validators = {
'canvasSize': canvasSize_validator,
'flightId': flightId_validator,
'domain': domain_validator,
'definitionPath': definitionPath_validator,
'harPath': harPath_validator,
}
def __init__(self, canvasSize, flightId, domain, definitionPath, harPath):
validate_and_assign(self, {
'canvasSize': canvasSize,
'flightId': flightId,
'domain': domain,
'definitionPath': definitionPath,
'harPath': harPath,
}, DefinitionRunner.validators)
The validators might be the same function, of course, if the data type is the same.
I'm not sure if this is exactly "Pythonic", but I've defined a function decorator called require_type. (To be honest, I think I found it somewhere online.)
def require_type(my_arg, *valid_types):
'''
A simple decorator that performs type checking.
#param my_arg: string indicating argument name
#param valid_types: list of valid types
'''
def make_wrapper(func):
if hasattr(func, 'wrapped_args'):
wrapped = getattr(func, 'wrapped_args')
else:
body = func.func_code
wrapped = list(body.co_varnames[:body.co_argcount])
try:
idx = wrapped.index(my_arg)
except ValueError:
raise(NameError, my_arg)
def wrapper(*args, **kwargs):
def fail():
all_types = ', '.join(str(typ) for typ in valid_types)
raise(TypeError, '\'%s\' was type %s, expected to be in following list: %s' % (my_arg, all_types, type(arg)))
if len(args) > idx:
arg = args[idx]
if not isinstance(arg, valid_types):
fail()
else:
if my_arg in kwargs:
arg = kwargs[my_arg]
if not isinstance(arg, valid_types):
fail()
return func(*args, **kwargs)
wrapper.wrapped_args = wrapped
return wrapper
return make_wrapper
Then, to use it:
class SomeObject(object):
#require_type("prop1", str)
#require_type("prop2", numpy.complex128)
def __init__(self, prop1, prop2):
pass
Let's say I have a class that has a member called data which is a list.
I want to be able to initialize the class with, for example, a filename (which contains data to initialize the list) or with an actual list.
What's your technique for doing this?
Do you just check the type by looking at __class__?
Is there some trick I might be missing?
I'm used to C++ where overloading by argument type is easy.
A much neater way to get 'alternate constructors' is to use classmethods. For instance:
>>> class MyData:
... def __init__(self, data):
... "Initialize MyData from a sequence"
... self.data = data
...
... #classmethod
... def fromfilename(cls, filename):
... "Initialize MyData from a file"
... data = open(filename).readlines()
... return cls(data)
...
... #classmethod
... def fromdict(cls, datadict):
... "Initialize MyData from a dict's items"
... return cls(datadict.items())
...
>>> MyData([1, 2, 3]).data
[1, 2, 3]
>>> MyData.fromfilename("/tmp/foobar").data
['foo\n', 'bar\n', 'baz\n']
>>> MyData.fromdict({"spam": "ham"}).data
[('spam', 'ham')]
The reason it's neater is that there is no doubt about what type is expected, and you aren't forced to guess at what the caller intended for you to do with the datatype it gave you. The problem with isinstance(x, basestring) is that there is no way for the caller to tell you, for instance, that even though the type is not a basestring, you should treat it as a string (and not another sequence.) And perhaps the caller would like to use the same type for different purposes, sometimes as a single item, and sometimes as a sequence of items. Being explicit takes all doubt away and leads to more robust and clearer code.
Excellent question. I've tackled this problem as well, and while I agree that "factories" (class-method constructors) are a good method, I would like to suggest another, which I've also found very useful:
Here's a sample (this is a read method and not a constructor, but the idea is the same):
def read(self, str=None, filename=None, addr=0):
""" Read binary data and return a store object. The data
store is also saved in the interal 'data' attribute.
The data can either be taken from a string (str
argument) or a file (provide a filename, which will
be read in binary mode). If both are provided, the str
will be used. If neither is provided, an ArgumentError
is raised.
"""
if str is None:
if filename is None:
raise ArgumentError('Please supply a string or a filename')
file = open(filename, 'rb')
str = file.read()
file.close()
...
... # rest of code
The key idea is here is using Python's excellent support for named arguments to implement this. Now, if I want to read the data from a file, I say:
obj.read(filename="blob.txt")
And to read it from a string, I say:
obj.read(str="\x34\x55")
This way the user has just a single method to call. Handling it inside, as you saw, is not overly complex
with python3, you can use Implementing Multiple Dispatch with Function Annotations as Python Cookbook wrote:
import time
class Date(metaclass=MultipleMeta):
def __init__(self, year:int, month:int, day:int):
self.year = year
self.month = month
self.day = day
def __init__(self):
t = time.localtime()
self.__init__(t.tm_year, t.tm_mon, t.tm_mday)
and it works like:
>>> d = Date(2012, 12, 21)
>>> d.year
2012
>>> e = Date()
>>> e.year
2018
Quick and dirty fix
class MyData:
def __init__(string=None,list=None):
if string is not None:
#do stuff
elif list is not None:
#do other stuff
else:
#make data empty
Then you can call it with
MyData(astring)
MyData(None, alist)
MyData()
A better way would be to use isinstance and type conversion. If I'm understanding you right, you want this:
def __init__ (self, filename):
if isinstance (filename, basestring):
# filename is a string
else:
# try to convert to a list
self.path = list (filename)
You should use isinstance
isinstance(...)
isinstance(object, class-or-type-or-tuple) -> bool
Return whether an object is an instance of a class or of a subclass thereof.
With a type as second argument, return whether that is the object's type.
The form using a tuple, isinstance(x, (A, B, ...)), is a shortcut for
isinstance(x, A) or isinstance(x, B) or ... (etc.).
You probably want the isinstance builtin function:
self.data = data if isinstance(data, list) else self.parse(data)
OK, great. I just tossed together this example with a tuple, not a filename, but that's easy. Thanks all.
class MyData:
def __init__(self, data):
self.myList = []
if isinstance(data, tuple):
for i in data:
self.myList.append(i)
else:
self.myList = data
def GetData(self):
print self.myList
a = [1,2]
b = (2,3)
c = MyData(a)
d = MyData(b)
c.GetData()
d.GetData()
[1, 2]
[2, 3]
My preferred solution is:
class MyClass:
_data = []
__init__(self,data=None):
# do init stuff
if not data: return
self._data = list(data) # list() copies the list, instead of pointing to it.
Then invoke it with either MyClass() or MyClass([1,2,3]).
Hope that helps. Happy Coding!
Why don't you go even more pythonic?
class AutoList:
def __init__(self, inp):
try: ## Assume an opened-file...
self.data = inp.read()
except AttributeError:
try: ## Assume an existent filename...
with open(inp, 'r') as fd:
self.data = fd.read()
except:
self.data = inp ## Who cares what that might be?