I am running nosetests across multiple files and getting an error relating to the importing of a specific file, well I'm not actually sure what the error is related to, I think it is either something up with the import or something up with the patching of it. The error itself looks like:
(I'm getting one of these errors for each test function that uses an #patch decorator)
Error
Traceback (most recent call last):
File "/home/user/Documents/venvs/migration/local/lib/python2.7/site-packages/unittest2/case.py", line 67, in testPartExecutor
yield
File "/home/user/Documents/venvs/migration/local/lib/python2.7/site-packages/unittest2/case.py", line 625, in run
testMethod()
File "/home/user/Documents/venvs/migration/local/lib/python2.7/site-packages/mock/mock.py", line 1297, in patched
arg = patching.__enter__()
File "/home/user/Documents/venvs/migration/local/lib/python2.7/site-packages/mock/mock.py", line 1353, in __enter__
self.target = self.getter()
File "/home/user/Documents/venvs/migration/local/lib/python2.7/site-packages/mock/mock.py", line 1523, in <lambda>
getter = lambda: _importer(target)
File "/home/user/Documents/venvs/migration/local/lib/python2.7/site-packages/mock/mock.py", line 1210, in _importer
thing = _dot_lookup(thing, comp, import_path)
File "/home/user/Documents/venvs/migration/local/lib/python2.7/site-packages/mock/mock.py", line 1200, in _dot_lookup
return getattr(thing, comp)
AttributeError: 'module' object has no attribute 'utils'
The package structure looks like this:
my_package
- my_module
- __init__.py
- utils.py
- other.py
- tests
- test_utils.py
- test_other.py
The nosetests command:
nosetests -e unit --with-coverage --cover-package=my_package --cover-erase --cover-xml --with-xunit tests --nocapture
So the weird thing is, if I run nosetests only on the utils test class itself, it runs fine, all imports work and all patches work, no errors, all tests pass.
Here's what the test_utils.py file looks like:
from my_module.utils import *
class TestBusinessProcess(unittest2.TestCase):
#patch('my_module.utils.something')
def test_some_utils_function(self, something_mock):
# test implementation..
# this function will throw:
# AttributeError: 'module' object has no attribute 'utils'
# when running whole tests folder and not on individual test file
pass
#patch('my_module.utils.something_else')
def test_some_other_utils_function(self, something_else_mock):
# test implementation..
# same as above
pass
An example of a test in the other test file that has no issues when ran either way:
from my_module.other import *
class TestBusinessProcess(unittest2.TestCase):
#patch('my_module.other.something')
def test_some_function(self, something_mock):
# test implementation..
# no issues!
pass
#patch('my_module.other.something_else')
def test_some_other_function(self, something_else_mock):
# test implementation..
# no issues!
pass
Any help greatly appreciated.
I still have no idea what was wrong, but it seemed to be something to do with the importing.
Anyway, this workaround solved the problem, but not sure why exactly.
The __init__.py in my_module was empty initially. Then I edited it to expose the individual functions of utils.py:
__init__.py
from utils import test_some_utils_function, test_some_other_utils_function
__all__ = [
"test_some_utils_function",
"test_some_other_utils_function"
]
Related
I've been trying to implement the __getattr__ function as in the following example:
PEP 562 -- Module __getattr__ and __dir__
And I don't get why this simple piece of code:
# lib.py
def __getattr__(name):
print(name)
# main.py
from lib import test
outputs:
__path__
test
test
What is __path__ ? Why is it sent to __getattr__ ? Why is test sent 2 times ?
TL;DR the first "test" printed is a side-effect of the "from import" implementation, i.e. it's printed during creation of lib module. The second "test" is from subsequent access of dynamic attribute on the module directly.
Knowing that importlib is implemented in Python code, modify your lib.py slightly to also dump a trace:
# lib.py
from traceback import print_stack
def __getattr__(name):
print_stack()
print(name)
print("-" * 80)
This gives the hint to pinpoint the library location in importlib which triggers double attribute access:
$ python3 main.py
File "main.py", line 3, in <module>
from lib import test
File "<frozen importlib._bootstrap>", line 1019, in _handle_fromlist
File "/private/tmp/lib.py", line 5, in __getattr__
print_stack()
__path__
--------------------------------------------------------------------------------
File "main.py", line 3, in <module>
from lib import test
File "<frozen importlib._bootstrap>", line 1032, in _handle_fromlist
File "/private/tmp/lib.py", line 5, in __getattr__
print_stack()
test
--------------------------------------------------------------------------------
File "main.py", line 3, in <module>
from lib import test
File "/private/tmp/lib.py", line 5, in __getattr__
print_stack()
test
--------------------------------------------------------------------------------
Now we can find the answer easily by RTFS (below I use Python v3.7.6, switch on git to the exact tag you use in case of different version). Look in importlib._bootstrap. _handle_fromlist at the indicated line numbers.
_handle_fromlist is a helper intended to load package submodules in a from import. Step 1 is to see if the module is a package at all:
if hasattr(module, '__path__'):
The __path__ access comes there, on line 1019. Because your __getattr__ returns None for all inputs, hasattr returns True here, so your module looks like a package, and the code continues on. (If hasattr had returned False, _handle_fromlist would abort at this point.)
The "fromlist" here will have the name you requested, ["test"], so we go into the for-loop with x="test" and on line 1032 there is the "extra" invocation:
elif not hasattr(module, x):
from lib import test will only attempt to load a lib.test submodule if lib does not already have a test attribute. This check is testing whether the attribute exists, to see if _handle_fromlist needs to attempt to load a submodule.
Should you return different values for the first and second invocation of __getattr__ with name "test", then the second value returned is the one which will actually be received within main.py.
I've written a bunch of tests for my Python application, but they've suddenly appeared to no longer be working properly!
I've created a bunch of tests inside a tests module:
Inside of tests.__main__.py, I've included the following code to load my test suite:
import unittest
if __name__ == "__main__":
loader = unittest.TestLoader()
start_dir = "."
suite = loader.discover(start_dir=start_dir, pattern='*_test.py')
runner = unittest.TextTestRunner()
runner.run(suite)
I know I am going to sound like a complete noob, but these tests were working perfectly for me about an hour ago. I would issue them by typing python3 -m tests from my base directory. However, I'm now receiving a really strange TypeError:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/yuchen/Desktop/myproj/myapp/tests/__main__.py", line 6, in <module>
suite = loader.discover(start_dir=start_dir, pattern='*_test.py')
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 341, in discover
tests = list(self._find_tests(start_dir, pattern))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 406, in _find_tests
yield from self._find_tests(full_path, pattern, namespace)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 406, in _find_tests
yield from self._find_tests(full_path, pattern, namespace)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 398, in _find_tests
full_path, pattern, namespace)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 452, in _find_test_path
return self.loadTestsFromModule(module, pattern=pattern), False
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 123, in loadTestsFromModule
tests.append(self.loadTestsFromTestCase(obj))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 92, in loadTestsFromTestCase
loaded_suite = self.suiteClass(map(testCaseClass, testCaseNames))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/suite.py", line 24, in __init__
self.addTests(tests)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/suite.py", line 57, in addTests
for test in tests:
TypeError: __init__() takes 0 positional arguments but 2 were given
I've followed the stack trace down to what I believe is the appropriate file causing the issue, the suite.py file of unittest:
"""TestSuite"""
import sys
from . import case
from . import util
__unittest = True
def _call_if_exists(parent, attr):
func = getattr(parent, attr, lambda: None)
func()
class BaseTestSuite(object):
"""A simple test suite that doesn't provide class or module shared fixtures.
"""
_cleanup = True
def __init__(self, tests=()):
self._tests = []
self._removed_tests = 0
print("Tests", tests) # <--- this was added by me
self.addTests(tests)
However, the __init__() signature doesn't appear to require 0 positional arguments. Moreover, it's clearly executing, since after I added in print("Tests", tests) in the code above, I saw the following output before the error was thrown:
Tests: []
Tests: <map object at 0x11033c0b8>
Tests: <map object at 0x11033c0b8>
Tests: <map object at 0x11033c0b8>
Tests: [<unittest.suite.TestSuite tests=[... a list of my tests]]
Tests: <map object at 0x110483978>
Tests: <map object at 0x110483978>
Traceback (most recent call last):
I'm pretty at a loss for what is going on. The error printed in the stack trace doesn't appear to correspond at all with what I'm seeing in the source code.
Edit (Resolution):
The accepted answer was spot on. I had written another test class, and decided to simply add in a stub for the __init__ method:
class MyUnfinishedTest(BaseTest):
def __init__():
pass
Then I had forgotten about it and worked on other stuff, and ran the tests. That's when I started seeing my error. After removing this class from my test module, the tests ran smoothly.
One of the reasons that the traceback seems a little confusing is that the line:
for test in tests
is actually hitting a generator, which is discovering and loading the tests on the fly.
So I'm guessing that this:
TypeError: __init__() takes 0 positional arguments but 2 were given
actually means that the discovery process has found a test class it cannot load, possibly because that class has its own __init__ method with a different number of arguments to what discover expects.
I would double-check your test classes for __init__ methods, and possibly also change this line:
start_dir = "."
to:
start_dir = os.path.dirname(__file__)
That would mean that the discovery process would start looking from the tests folder down, which is safer than "." (current working directory) because the current working directory might change depending upon where you start the tests from (and may end up picking up tests that you weren't expecting).
let's say I have a directory with 3 test*.py files.
__init__.py
lib1.py
lib2.py
test_01.py
test_02_badimport.py
test_03.py
nosetest by itself just runs all the tests.
Now, let's say also that one of the tests, test_02_badimport.py has an issue with an import. This is the code for it, though it's not super-important:
import unittest
#this module is missing, so it will trigger an ImportError
import missing_module
class TestStringMethods(unittest.TestCase):
def test_upper(self):
self.assertEqual('foo'.upper(), 'FOO')
if __name__ == '__main__':
unittest.main()
This is what I get from doing a nosetests:
(env) me#test_nose$ nosetests
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
test_basic (test_nose.test_01.MyTest) ... ok
test_something (test_nose.test_01.MyTest) ... ok
Failure: ImportError (No module named missing_module) ... ERROR
test_basic (test_nose.test_03.MyTest) ... ok
test_something (test_nose.test_03.MyTest) ... ok
======================================================================
ERROR: Failure: ImportError (No module named missing_module)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/me/env/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/Users/me/env/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/Users/me/env/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/Users/me/wk/explore/test_nose/test_02_badimport.py", line 3, in <module>
import missing_module
ImportError: No module named missing_module
----------------------------------------------------------------------
Ran 5 tests in 0.007s
FAILED (errors=1)
Look at Failure: ImportError (No module named missing_module) ... ERROR. Now, I know that it it's caused by import missing_module.
But it requires a fair bit of tracking to figure out that the problem lies with test_02_badimport.py, because that information is not communicated to the test run output, it's only part of the stack trace dump.
My actual case was more complicated, it's due to a circular import issue in the context of a django.setup() call, so my own code appears both interleaved with nose and unittest code, as well as Django files, in the midst of a huge stack trace, rather than at the bottom as in this example.
Is there any way to achieve something like:
(test_nose.test_02_badimport) Failure: ImportError (No module named missing_module) ... ERROR in the output?
Or do I just have to pay attention to the next line after the nose load_module call in the stack trace?
File "/Users/me/env/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
Following the above, test_02_badimport.py is the guilty file:
File "/Users/me/wk/explore/test_nose/test_02_badimport.py", line 3, in <module>
import missing_module
My workflow:
Nosetest, especially with the -x (stop on first error) flag allows me to quickly run all tests. And, once I know which test_xxx.py file has a problem I can run it from the command line python test_02_badimport.py directly and fix it. In most cases, that works very well, but ImportError somewhat hides the filename.
Is there a commandline switch? (I tried --collect-only but that didn't help)? Neither did nosetests --debug=nose,nose.importer --debug-log=nose_debug.
Or a plugin?
Failing that, is the next line after mod = load_module always going to show my problem test file?
I am trying to Use PyUnit within PyDev for the first time. I created a unittest.py module. When I did run as -> Python unit-test, I got the following error:
Finding files... done.
Importing test modules ... done.
======================================================================
Traceback (most recent call last):
File "/Applications/eclipse/plugins/org.python.pydev_2.7.5.2013052819/pysrc/runfiles.py", line 163, in
main()
File "/Applications/eclipse/plugins/org.python.pydev_2.7.5.2013052819/pysrc/runfiles.py", line 77, in main
pydev_runfiles.main(configuration)
File "/Applications/eclipse/plugins/org.python.pydev_2.7.5.2013052819/pysrc/pydev_runfiles.py", line 761, in main
PydevTestRunner(configuration).run_tests()
File "/Applications/eclipse/plugins/org.python.pydev_2.7.5.2013052819/pysrc/pydev_runfiles.py", line 747, in run_tests
runner.run(test_suite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/runner.py", line 158, in run
result.printErrors()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/runner.py", line 108, in printErrors
self.printErrorList('ERROR', self.errors)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/runner.py", line 114, in printErrorList
self.stream.writeln("%s: %s" % (flavour,self.getDescription(test)))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/runner.py", line 46, in getDescription
return '\n'.join((str(test), doc_first_line))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 1060, in __str__
self._testFunc.__name__)
AttributeError: 'str' object has no attribute '__name__'
My unit test is just the PyDev-generated default:
import unittest
class Test(unittest.TestCase):
def setUp(self):
pass
def tearDown(self):
pass
def testName(self):
pass
print "hello test"
if __name__ == "__main__":
#import sys;sys.argv = ['', 'Test.testName']
unittest.main()
I expected it to print hello test. What am I missing?
Ok, I realized your problem, after having a similar issue for several minutes. You need to rename your file. Name it anything other than unittest.py. This causes the compiler to confuse between your module and the unittest module that you are importing.
Second, even after you change the name of your file, you may run into the same error. That is caused by the unittest.pyc file not being removed from your project. Rename the file, delete the previously generated unittest.pyc file, and the test should run just fine.
I am trying to get started with using nosegae, however I run into the issue that I can't seem to get it to pass even the simplest of cases when using django.
when running without the --without-sandbox flag both the following tests fail
def test_import_django ():
import django
def test_import_django_http ():
import django.http
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\nose-1.1.2-py2.7.egg\nose\case.py", line 1
97, in runTest
self.test(*self.arg)
File "C:\Users\User\Desktop\TDD_GAE\myproj\tests.py", line 2, in test_import_d
jango
import django
File "C:\Python27\lib\site-packages\nosegae-0.1.9-py2.7.egg\nosegae.py", line
207, in find_module
return super(HookMixin, self).find_module(fullname, path)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 1505, in Decorate
return func(self, *args, **kwargs)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 1998, in find_module
search_path)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 1505, in Decorate
return func(self, *args, **kwargs)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 2119, in FindModuleRestricted
result = self.FindPathHook(submodule, submodule_fullname, path_entry)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 2219, in FindPathHook
return self._imp.find_module(submodule, [path_entry])
Howevere if I do use --without-sandbox at least the first test passes
myproj.tests.test_import_django ... ok
myproj.tests.test_import_django_http ... ERROR
======================================================================
ERROR: myproj.tests.test_import_django_http
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\nose-1.1.2-py2.7.egg\nose\case.py", line 1
97, in runTest
self.test(*self.arg)
File "C:\Users\User\Desktop\TDD_GAE\myproj\tests.py", line 5, in test_import_d
jango_http
import django.http
File "C:\Program Files (x86)\Google\google_appengine\lib\django_1_2\django\htt
p\__init__.py", line 9, in <module>
from mod_python.util import parse_qsl
File "C:\Python27\lib\site-packages\nosegae-0.1.9-py2.7.egg\nosegae.py", line
199, in find_module
mod_path = self.find_mod_path(fullname)
File "C:\Python27\lib\site-packages\nosegae-0.1.9-py2.7.egg\nosegae.py", line
251, in find_mod_path
_sf, path, _desc= self._imp.find_module(top, None)
AttributeError: 'str' object has no attribute 'find_module'
Has anyone encountered and know how I can go about past this?
Edit
It seems that the issue is recursive imports
def test_import_pdb ():
import pdb
pdb.set_trace ()
part of the stack trace is
File "C:\Python27\lib\pdb.py", line 72, in __init__
import readline
notice that an import in __init__ of django.http is also part of the stack trace
Read https://docs.djangoproject.com/en/dev/topics/testing/ about Django testing.
As I know it's better to use unittest or doctest shipped with django as it have several improvements for django-specific testing like form field output testing and some database features. Hovewer it's not essential and if you want to continue using nose - think you missed django environment setup:
from django.test.utils import setup_test_environment
setup_test_environment()
This lines needed to run your tests outside of ./manage.py --test
UPD
Yeah my previous thought's were wrong. So I just digged into sources of nose and nose-gae, and what I think - check HardenedModulesHook definition in your nose version, cause in trunk of nose I've found following:
class HardenedModulesHook(object):
...
def __init__(self,
module_dict,
imp_module=imp,
os_module=os,
dummy_thread_module=dummy_thread,
pickle_module=pickle):
...
That gives following - when noseGAE plugin begin() method is executed -> there self._install_hook(dev_appserver.HardenedModulesHook) is called which declares mixed-hook class and creates it's instance like self.hook = Hook(sys.modules, self._path). <- There is HardenedModulesHook.__init__ called with second argument as mystic '_path' however in NOSE this argument should be 'imp' module by default -> That makes an exception you've got:
_sf, path, _desc= self._imp.find_module(top, None)
AttributeError: 'str' object has no attribute 'find_module'
So I think it might be a problem with nose-gae :(