Develop Reference

Python unittest loader throws TypeError during an __init__() call

I've written a bunch of tests for my Python application, but they've suddenly appeared to no longer be working properly!
I've created a bunch of tests inside a tests module:
Inside of tests.__main__.py, I've included the following code to load my test suite:
import unittest
if __name__ == "__main__":
loader = unittest.TestLoader()
start_dir = "."
suite = loader.discover(start_dir=start_dir, pattern='*_test.py')
runner = unittest.TextTestRunner()
runner.run(suite)
I know I am going to sound like a complete noob, but these tests were working perfectly for me about an hour ago. I would issue them by typing python3 -m tests from my base directory. However, I'm now receiving a really strange TypeError:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/yuchen/Desktop/myproj/myapp/tests/__main__.py", line 6, in <module>
suite = loader.discover(start_dir=start_dir, pattern='*_test.py')
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 341, in discover
tests = list(self._find_tests(start_dir, pattern))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 406, in _find_tests
yield from self._find_tests(full_path, pattern, namespace)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 406, in _find_tests
yield from self._find_tests(full_path, pattern, namespace)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 398, in _find_tests
full_path, pattern, namespace)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 452, in _find_test_path
return self.loadTestsFromModule(module, pattern=pattern), False
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 123, in loadTestsFromModule
tests.append(self.loadTestsFromTestCase(obj))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 92, in loadTestsFromTestCase
loaded_suite = self.suiteClass(map(testCaseClass, testCaseNames))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/suite.py", line 24, in __init__
self.addTests(tests)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/suite.py", line 57, in addTests
for test in tests:
TypeError: __init__() takes 0 positional arguments but 2 were given
I've followed the stack trace down to what I believe is the appropriate file causing the issue, the suite.py file of unittest:
"""TestSuite"""
import sys
from . import case
from . import util
__unittest = True
def _call_if_exists(parent, attr):
func = getattr(parent, attr, lambda: None)
func()
class BaseTestSuite(object):
"""A simple test suite that doesn't provide class or module shared fixtures.
"""
_cleanup = True
def __init__(self, tests=()):
self._tests = []
self._removed_tests = 0
print("Tests", tests) # <--- this was added by me
self.addTests(tests)
However, the __init__() signature doesn't appear to require 0 positional arguments. Moreover, it's clearly executing, since after I added in print("Tests", tests) in the code above, I saw the following output before the error was thrown:
Tests: []
Tests: <map object at 0x11033c0b8>
Tests: <map object at 0x11033c0b8>
Tests: <map object at 0x11033c0b8>
Tests: [<unittest.suite.TestSuite tests=[... a list of my tests]]
Tests: <map object at 0x110483978>
Tests: <map object at 0x110483978>
Traceback (most recent call last):
I'm pretty at a loss for what is going on. The error printed in the stack trace doesn't appear to correspond at all with what I'm seeing in the source code.
Edit (Resolution):
The accepted answer was spot on. I had written another test class, and decided to simply add in a stub for the __init__ method:
class MyUnfinishedTest(BaseTest):
def __init__():
pass
Then I had forgotten about it and worked on other stuff, and ran the tests. That's when I started seeing my error. After removing this class from my test module, the tests ran smoothly.

One of the reasons that the traceback seems a little confusing is that the line:
for test in tests
is actually hitting a generator, which is discovering and loading the tests on the fly.
So I'm guessing that this:
TypeError: __init__() takes 0 positional arguments but 2 were given
actually means that the discovery process has found a test class it cannot load, possibly because that class has its own __init__ method with a different number of arguments to what discover expects.
I would double-check your test classes for __init__ methods, and possibly also change this line:
start_dir = "."
to:
start_dir = os.path.dirname(__file__)
That would mean that the discovery process would start looking from the tests folder down, which is safer than "." (current working directory) because the current working directory might change depending upon where you start the tests from (and may end up picking up tests that you weren't expecting).

what's a good strategy for identifying the problematic file on a nosetest ImportError?

let's say I have a directory with 3 test*.py files.
__init__.py
lib1.py
lib2.py
test_01.py
test_02_badimport.py
test_03.py
nosetest by itself just runs all the tests.
Now, let's say also that one of the tests, test_02_badimport.py has an issue with an import. This is the code for it, though it's not super-important:
import unittest
#this module is missing, so it will trigger an ImportError
import missing_module
class TestStringMethods(unittest.TestCase):
def test_upper(self):
self.assertEqual('foo'.upper(), 'FOO')
if __name__ == '__main__':
unittest.main()
This is what I get from doing a nosetests:
(env) me#test_nose$ nosetests
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
test_basic (test_nose.test_01.MyTest) ... ok
test_something (test_nose.test_01.MyTest) ... ok
Failure: ImportError (No module named missing_module) ... ERROR
test_basic (test_nose.test_03.MyTest) ... ok
test_something (test_nose.test_03.MyTest) ... ok
======================================================================
ERROR: Failure: ImportError (No module named missing_module)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/me/env/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/Users/me/env/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/Users/me/env/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/Users/me/wk/explore/test_nose/test_02_badimport.py", line 3, in <module>
import missing_module
ImportError: No module named missing_module
----------------------------------------------------------------------
Ran 5 tests in 0.007s
FAILED (errors=1)
Look at Failure: ImportError (No module named missing_module) ... ERROR. Now, I know that it it's caused by import missing_module.
But it requires a fair bit of tracking to figure out that the problem lies with test_02_badimport.py, because that information is not communicated to the test run output, it's only part of the stack trace dump.
My actual case was more complicated, it's due to a circular import issue in the context of a django.setup() call, so my own code appears both interleaved with nose and unittest code, as well as Django files, in the midst of a huge stack trace, rather than at the bottom as in this example.
Is there any way to achieve something like:
(test_nose.test_02_badimport) Failure: ImportError (No module named missing_module) ... ERROR in the output?
Or do I just have to pay attention to the next line after the nose load_module call in the stack trace?
File "/Users/me/env/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
Following the above, test_02_badimport.py is the guilty file:
File "/Users/me/wk/explore/test_nose/test_02_badimport.py", line 3, in <module>
import missing_module
My workflow:
Nosetest, especially with the -x (stop on first error) flag allows me to quickly run all tests. And, once I know which test_xxx.py file has a problem I can run it from the command line python test_02_badimport.py directly and fix it. In most cases, that works very well, but ImportError somewhat hides the filename.
Is there a commandline switch? (I tried --collect-only but that didn't help)? Neither did nosetests --debug=nose,nose.importer --debug-log=nose_debug.
Or a plugin?
Failing that, is the next line after mod = load_module always going to show my problem test file?

PYTHON: nosetests import file path with multiple modules/files

I'm currently working through LearnPythonTheHardWay and have reached Exercise 48 which details Nosetests. I am able to perform a unit testing as long as all of the code is in a single python.py file. However if I include other files as part of a program, i.e. use import and then attempt to nosetest such a project I am getting an error, as follows:
======================================================================
ERROR: Failure: ImportError (No module named 'temp')
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/nose/failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
File "/usr/local/lib/python3.4/dist-packages/nose/loader.py", line 414, in loadTestsFromName ## ##
addr.filename, addr.module)
File "/usr/local/lib/python3.4/dist-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/usr/local/lib/python3.4/dist-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/usr/lib/python3.4/imp.py", line 235, in load_module
return load_source(name, filename, file)
File "/usr/lib/python3.4/imp.py", line 171, in load_source
module = methods.load()
File "", line 1220, in load
File "", line 1200, in _load_unlocked
File "", line 1129, in _exec
File "", line 1471, in exec_module
File "", line 321, in _call_with_frames_removed
File "/home/user/LEARNPYTHONTHEHARDWAY/ex48/tests/scanner_tests.py", line 6, in
from ex48.scanner import lexicon
File "/home/user/LEARNPYTHONTHEHARDWAY/ex48/ex48/scanner.py", line 6, in
import temp
ImportError: No module named 'temp'
Ran 1 test in 0.028s
FAILED (errors=1)
The structure of my project directories are as follows:
ex48/
ex48/
scanner.py
temp.py
__pycache__/
tests/
__init__.py
scanner_tests.py
Screenshot of my directory::
Screen shot of files themselves::
My scanner_tests.py file is as follows:
from nose.tools import *
from ex48.scanner import lexicon
from ex48 import temp
def test_directions():
assert_equal(lexicon.scan("north"),[('direction','north')])
result = lexicon.scan("north south east")
assert_equal(result, [('direction', 'north'),
('direction', 'south'),
('direction', 'east')])
My scanner.py file is as follows:
import temp
class lexicon:
def scan(val):
if(val == "north"):
return [('direction', 'north')]
else:
return [('direction', 'north'),
('direction', 'south'),
('direction', 'east')]
runner = temp.temp("hello")
And finally my temp.py file is as follows:
class temp(object):
def __init__(self,name):
self.name = name
def run(self):
print "Your name is; %s" % self.name
runner.run()
My question is how to overcome the ImportError: No Module named 'temp' because it seems as if I have imported the temp.py file in both the scanner.py file and the scanner_tests.py file but nose does not seem to be able to import it when it runs. Nosetests works fine when its just the single scanner.py file but not when importing. Is there a special syntax for importing into a unit test for nose? The script also works fine when run and imports properly at the command line.
*Note: I'm running python off a limited account off an online server so some admin privileges are not available.
**Note below are entirely different screenshots from another project with the exact same error:
Directory Layout:
Game.py:
Otherpy.py - the imported file:
the Nose test script file:
And finally the nosetests importerror:

Everything needs to be with respect to your execution point. You are running your nose command from the root of ex48, therefore all your imports need to be with respect to that location.
Therefore, in game.py you should be importing with respect to ex48. Therefore:
from ex48.otherpy import House
The same logic should be applied to your example referencing the temp folder.
from ex48.temp import temp

The only solution I have found is to post the following to the top of the main file:
try:
# This handles imports when running .py files from inside app directory
from file_to_import.py import class_instance
except:
# This handles imports when running nosetests from top-level (above app)
# directory
from directory_containing_app_files.file_to_import import class_instance
I am supremely interested in an alternative solution.

How can I make a shortcut to a module that runs unittest.main() and then allows the Python interpreter to remain open?

I am developing a package that uses unittest for its tests. The tests are all in a submodule called tests. At the bottom of the submodule is this:
if __name__ == '__main__':
unittest.main()
But if you run the module, Windows closes the command window before one can read the output. OK, this is not a new problem. There are ways around this, for example making a shortcut that looks something like:
cmd /k "python -m mypackage.tests"
This works, now you get to see the output. But then you are dumped back at the C:\Windows\System32 command prompt. It would be nicer to be able to still be in the Python interpreter, so that I can play around in Python if something occurs to me to check after I saw the tests. So you try something like this:
cmd /k "python -i -m mypackage.tests"
But, whoa, what's this? after the test output, you see
Traceback (most recent call last):
File "C:\Python33\lib\runpy.py", line 160, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "C:\Python33\lib\runpy.py", line 73, in _run_code
exec(code, run_globals)
File "D:\Docs\programs\python\mypackage\tests.py", line 274, in <module>
unittest.main()
File "C:\Python33\lib\unittest\main.py", line 125, in __init__
self.runTests()
File "C:\Python33\lib\unittest\main.py", line 267, in runTests
sys.exit(not self.result.wasSuccessful())
SystemExit: True
>>>
You are still in the Python interpreter, but now for some reason there are several lines of dumb noise in between the test results and the new prompt. No-one wants to see that. It is presumably because unittest.main() tries to exit the interpreter but the interpreter doesn't let it happen because you used the -i option. Which is good in a way, but the traceback isn't wanted.
You look at the documentation for unittest to see if there is a way to make Python stick around after running the tests. There are no command line switches for unittest that have this effect, but there is a keyword argument exit that could be used to prevent unittest from trying to close Python. So we could change the end of tests.py to this:
if __name__ == '__main__':
unittest.main(exit = False)
The problem with this is that sometimes you might want the default behaviour of closing the interpreter, like say if you had your version control software run the tests module automatically. So it would be best to have some way of conditionally disabling the exit. Well, it seems like by implementing a rudimentary check for command line options this could be done. So we try changing it to
if __name__ == '__main__':
import sys
unittest.main(exit = 'noexit' not in ''.join(sys.argv[1:]))
and the shortcut to
cmd /k "python -i -m mypackage.tests --noexit"
But now when you run it the tests don't even run, instead you see a big wall of complaints about how there's no option called "noexit":
Usage: tests.py [options]
tests.py: error: no such option: --noexit
Traceback (most recent call last):
File "C:\Python33\lib\optparse.py", line 1391, in parse_args
stop = self._process_args(largs, rargs, values)
File "C:\Python33\lib\optparse.py", line 1431, in _process_args
self._process_long_opt(rargs, values)
File "C:\Python33\lib\optparse.py", line 1484, in _process_long_opt
opt = self._match_long_opt(opt)
File "C:\Python33\lib\optparse.py", line 1469, in _match_long_opt
return _match_abbrev(opt, self._long_opt)
File "C:\Python33\lib\optparse.py", line 1674, in _match_abbrev
raise BadOptionError(s)
optparse.BadOptionError: no such option: --noexit
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python33\lib\runpy.py", line 160, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "C:\Python33\lib\runpy.py", line 73, in _run_code
exec(code, run_globals)
File "D:\Docs\programs\python\mypackage\tests.py", line 275, in <module>
unittest.main(exit = 'noexit' not in ''.join(sys.argv[1:]))
File "C:\Python33\lib\unittest\main.py", line 124, in __init__
self.parseArgs(argv)
File "C:\Python33\lib\unittest\main.py", line 148, in parseArgs
options, args = parser.parse_args(argv[1:])
File "C:\Python33\lib\optparse.py", line 1393, in parse_args
self.error(str(err))
File "C:\Python33\lib\optparse.py", line 1573, in error
self.exit(2, "%s: error: %s\n" % (self.get_prog_name(), msg))
File "C:\Python33\lib\optparse.py", line 1563, in exit
sys.exit(status)
SystemExit: 2
Well I think I should be the one who decides what command line options there are, but maybe I haven't gone through the correct formalities to declare what command line options I'll accept. But actually, no, that isn't the case, because if I comment out the line that calls unittest.main():
if __name__ == '__main__':
import sys
#unittest.main(exit = 'noexit' not in ''.join(sys.argv[1:]))
then there is no complaining at all, just a >>> prompt! From which I can only deduce that unittest.main() is for some reason inspecting the command line arguments to my script, and throwing a hissy fit when they don't meet its standards - this despite the fact that they aren't directed at it in the first place. This wouldn't be so much of a problem if there were a command line switch that made it hang around instead of exiting (like the exit keyword argument does), but there isn't.
What's the answer? Every option seems unsatisfactory in one way or another.

Can't import django packages with nosegae

I am trying to get started with using nosegae, however I run into the issue that I can't seem to get it to pass even the simplest of cases when using django.
when running without the --without-sandbox flag both the following tests fail
def test_import_django ():
import django
def test_import_django_http ():
import django.http
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\nose-1.1.2-py2.7.egg\nose\case.py", line 1
97, in runTest
self.test(*self.arg)
File "C:\Users\User\Desktop\TDD_GAE\myproj\tests.py", line 2, in test_import_d
jango
import django
File "C:\Python27\lib\site-packages\nosegae-0.1.9-py2.7.egg\nosegae.py", line
207, in find_module
return super(HookMixin, self).find_module(fullname, path)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 1505, in Decorate
return func(self, *args, **kwargs)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 1998, in find_module
search_path)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 1505, in Decorate
return func(self, *args, **kwargs)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 2119, in FindModuleRestricted
result = self.FindPathHook(submodule, submodule_fullname, path_entry)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\de
v_appserver.py", line 2219, in FindPathHook
return self._imp.find_module(submodule, [path_entry])
Howevere if I do use --without-sandbox at least the first test passes
myproj.tests.test_import_django ... ok
myproj.tests.test_import_django_http ... ERROR
======================================================================
ERROR: myproj.tests.test_import_django_http
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\nose-1.1.2-py2.7.egg\nose\case.py", line 1
97, in runTest
self.test(*self.arg)
File "C:\Users\User\Desktop\TDD_GAE\myproj\tests.py", line 5, in test_import_d
jango_http
import django.http
File "C:\Program Files (x86)\Google\google_appengine\lib\django_1_2\django\htt
p\__init__.py", line 9, in <module>
from mod_python.util import parse_qsl
File "C:\Python27\lib\site-packages\nosegae-0.1.9-py2.7.egg\nosegae.py", line
199, in find_module
mod_path = self.find_mod_path(fullname)
File "C:\Python27\lib\site-packages\nosegae-0.1.9-py2.7.egg\nosegae.py", line
251, in find_mod_path
_sf, path, _desc= self._imp.find_module(top, None)
AttributeError: 'str' object has no attribute 'find_module'
Has anyone encountered and know how I can go about past this?
Edit
It seems that the issue is recursive imports
def test_import_pdb ():
import pdb
pdb.set_trace ()
part of the stack trace is
File "C:\Python27\lib\pdb.py", line 72, in __init__
import readline
notice that an import in __init__ of django.http is also part of the stack trace

Read https://docs.djangoproject.com/en/dev/topics/testing/ about Django testing.
As I know it's better to use unittest or doctest shipped with django as it have several improvements for django-specific testing like form field output testing and some database features. Hovewer it's not essential and if you want to continue using nose - think you missed django environment setup:
from django.test.utils import setup_test_environment
setup_test_environment()
This lines needed to run your tests outside of ./manage.py --test
UPD
Yeah my previous thought's were wrong. So I just digged into sources of nose and nose-gae, and what I think - check HardenedModulesHook definition in your nose version, cause in trunk of nose I've found following:
class HardenedModulesHook(object):
...
def __init__(self,
module_dict,
imp_module=imp,
os_module=os,
dummy_thread_module=dummy_thread,
pickle_module=pickle):
...
That gives following - when noseGAE plugin begin() method is executed -> there self._install_hook(dev_appserver.HardenedModulesHook) is called which declares mixed-hook class and creates it's instance like self.hook = Hook(sys.modules, self._path). <- There is HardenedModulesHook.__init__ called with second argument as mystic '_path' however in NOSE this argument should be 'imp' module by default -> That makes an exception you've got:
_sf, path, _desc= self._imp.find_module(top, None)
AttributeError: 'str' object has no attribute 'find_module'
So I think it might be a problem with nose-gae :(