I have my Python unittest script like below. It takes an argument '-a' to determine whether the testcases should load the base module from foo_PC_A.py or foo_PC_B.py. I use shutil.move() to rename either .py file to foo.py, so all the testcase modules (e.g. tm1.py, tm2.py) can simply import foo. Though this looks like a workaround and not Pythonic. Is there any better way to do this? Or a better design to fundamentally resolve this issue.
(run_all_unittest.py)
if sys.argv[1] = '-a':
shutil.move('foo_PC_A.py', 'foo.py')
else:
shutil.move('foo_PC_B.py', 'foo.py')
test_module_list = ['tm1', 'tm2', ...]
for test_module_name in test_module_list:
test_module = __import__(test_module_name)
test_suites.append(unittest.TestLoader().loadTestsFromModule(test_module))
alltests = unittest.TestSuite(test_suites)
unittest.TextTestRunner().run(alltests)
if sys.argv[1] = '-a':
shutil.move('foo.py', 'foo_PC_A.py')
else:
shutil.move('foo.py', 'foo_PC_B.py')
(tm1.py)
from foo import MyTestCase
...
(foo_PC_A.py)
import <some module only available on PC A>
class MyTestCase(unittest.TestCase):
...
(foo_PC_B.py)
# since the PC A modules are not available on PC B,
# just call the pre-built executable via subprocess
import subprocess
class MyTestCase(unittest.TestCase):
...
def test_run(self):
subprocess.call(...)
You can fool Python into thinking the module has already been loaded. Just import the module dynamically and use sys.modules:
import sys
import importlib
if sys.argv[1] = '-a':
sys.modules['foo'] = importlib.import_module('foo_PC_A')
else:
sys.modules['foo'] = importlib.import_module('foo_PC_A')
When any module runs import foo or from foo import ..., Python will use that path.
Note that if foo is moved to a package, the full Python path must be specified, as in:
sys.modules['path.to.foo'] = ...
Lets say my code looks like this
import pytest
import json
#pytest.fixture
def test_item():
test_item = json.load(open('./directory/sample_item_for_test.json','rb'))
return test_item
def test_fun(test_document):
assert type(test_item.description[0]) == unicode
And I would like to run this test via Py.Test
If I run Py.test from the directory that it is in, it is fine. BUT if I run it from an above directory, it fails due to not being able to find 'sample_item_for_test.json'. Is there a way to make this test run correctly no matter where I execute Py.test?
The magic attribute __file__ is the path to the python file on the filesystem. So, you can use that with some magic from os to get the current directory...
import pytest
import json
import os
_HERE = os.path.dirname(__file__)
_TEST_JSON_FILENAME = os.path.join(_HERE, 'directory', 'sample_item_for_test.json')
#pytest.fixture
def test_item():
with open(_TEST_JSON_FILENAME, 'rb') as file_input:
return json.load(file_input)
When I migrated to py.test, I had a large set of legacy tests that were accustomed to being executed in the directory where the test file lives. Instead of tracking down every test failure, I added a pytest hook to my conftest.py to chdir to the test directory before each test starts:
import os
import functools
def pytest_runtest_setup(item):
"""
Execute each test in the directory where the test file lives.
"""
starting_directory = os.getcwd()
test_directory = os.path.dirname(str(item.fspath))
os.chdir(test_directory)
teardown = functools.partial(os.chdir, starting_directory)
# There's probably a cleaner way than accessing a private member.
item.session._setupstate.addfinalizer(teardown, item)
I need to run a script foo.py, but I need to also insert some debugging lines to run before the code in foo.py. Currently I just put those lines in foo.py and I'm careful not to commit that to Git, but I don't like this solution.
What I want is a separate file bar.py that I don't commit to Git. Then I want to run:
python /somewhere/bar.py /somewhere_else/foo.py
What I want this to do is first run some lines of code in bar.py, and then run foo.py as __main__. It should be in the same process that the bar.py lines ran in, otherwise the debugging lines won't help.
Is there a way to make bar.py do this?
Someone suggested this:
import imp
import sys
# Debugging code here
fp, pathname, description = imp.find_module(sys.argv[1])
imp.load_module('__main__', fp, pathname, description)
The problem with that is that because it uses import machinery, I need to be on the same folder as foo.py to run that. I don't want that. I want to simply put in the full path to foo.py.
Also: The solution needs to work with .pyc files as well.
Python has a mechanism for running code at startup; the site module.
"This module is automatically imported during initialization."
The site module will attempt to import a module named sitecustomize before __main__ is imported.
It will also attempt to import a module named usercustomize if your environment instructs it to.
For example, you could put a sitecustomize.py file in your site-packages folder that contains this:
import imp
import os
if 'MY_STARTUP_FILE' in os.environ:
try:
file_path = os.environ['MY_STARTUP_FILE']
folder, file_name = os.path.split(file_path)
module_name, _ = os.path.splitext(file_name)
fp, pathname, description = imp.find_module(module_name, [folder])
except Exception as e:
# Broad exception handling since sitecustomize exceptions are ignored
print "There was a problem finding startup file", file_path
print repr(e)
exit()
try:
imp.load_module(module_name, fp, pathname, description)
except Exception as e:
print "There was a problem loading startup file: ", file_path
print repr(e)
exit()
finally:
# "the caller is responsible for closing the file argument" from imp docs
if fp:
fp.close()
Then you could run your script like this:
MY_STARTUP_FILE=/somewhere/bar.py python /somewhere_else/foo.py
You could run any script before foo.py without needing to add code to reimport __main__.
Run export MY_STARTUP_FILE=/somewhere/bar.py and not need to reference it every time
You can use execfile() if the file is .py and uncompyle2 if the file is .pyc.
Let's say you have your file structure like:
test|-- foo.py
|-- bar
|--bar.py
foo.py
import sys
a = 1
print ('debugging...')
# run the other file
if sys.argv[1].endswith('.py'): # if .py run right away
execfile(sys.argv[1], globals(), locals())
elif sys.argv[1].endswith('.pyc'): # if .pyc, first uncompyle, then run
import uncompyle2
from StringIO import StringIO
f = StringIO()
uncompyle2.uncompyle_file(sys.argv[1], f)
f.seek(0)
exec(f.read(), globals(), locals())
bar.py
print a
print 'real job'
And in test/, if you do:
$ python foo.py bar/bar.py
$ python foo.py bar/bar.pyc
Both, outputs the same:
debugging...
1
real job
Please also see this answer.
You probably have something along the lines of:
if __name__ == '__main__':
# some code
Instead, write your code in a function main() in foo and then do:
if __name__ == '__main__':
main()
Then, in bar, you can import foo and call foo.main().
Additionaly, if you need to change the working directory, you can use the os.chdir(path) method, e.g. os.chdir('path/of/bar') .
bar.py would have to behave like the Python interpreter itself and run foo.py (or foo.pyc, as you asked for that) as if it was the main script. This is surprisingly hard. My 90% solution to do that looks like so:
def run_script_as_main(cmdline):
# It is crucial to import locally what we need as we
# later remove everything from __main__
import sys, imp, traceback, os
# Patch sys.argv
sys.argv = cmdline
# Clear the __main__ namespace and set it up to run
# the secondary program
maindict = sys.modules["__main__"].__dict__
builtins = maindict['__builtins__']
maindict.clear()
maindict['__file__'] = cmdline[0]
maindict['__builtins__'] = builtins
maindict['__name__'] = "__main__"
maindict['__doc__'] = None
# Python prepends a script's location to sys.path
sys.path[0] = os.path.dirname(os.path.abspath(cmdline[0]))
# Treat everything as a Python source file, except it
# ends in '.pyc'
loader = imp.load_source
if maindict["__file__"].endswith(".pyc"):
loader = imp.load_compiled
with open(cmdline[0], 'rb') as f:
try:
loader('__main__', maindict["__file__"], f)
except Exception:
# In case of an exception, remove this script from the
# stack trace; if you don't care seeing some bar.py in
# traceback, you can leave that out.
ex_type, ex_value, ex_traceback = sys.exc_info()
tb = traceback.extract_tb(ex_traceback, None)[1:]
sys.stderr.write("Traceback (most recent call last):\n")
for line in traceback.format_list(tb):
sys.stderr.write(line)
for line in traceback.format_exception_only(ex_type, ex_value):
sys.stderr.write(line)
This mimics the default behavior of the Python interpreter
relatively closely. It sets system globals __name__, __file__,
sys.argv, inserts the script location into sys.path, clears the global namespace etc.
You would copy that code to bar.py or import it from somewhere and use it like so:
if __name__ == "__main__":
# You debugging setup goes here
...
# Run the Python program given as argv[1]
run_script_as_main(sys.argv[1:])
Then, given this:
$ find so-foo-bar
so-foo-bar
so-foo-bar/debugaid
so-foo-bar/debugaid/bar.py
so-foo-bar/foo.py
and foo.py looking like this:
import sys
if __name__ == "__main__":
print "My name is", __name__
print "My file is", __file__
print "My command line is", sys.argv
print "First 2 path items", sys.path[:2]
print "Globals", globals().keys()
raise Exception("foo")
... running foo.py via bar.py gives you that:
$ python so-foo-bar/debugaid/bar.py so-foo-bar/foo.py
My name is __main__
My file is so-foo-bar/foo.py
My command line is ['so-foo-bar/foo.py']
First 2 path items ['~/so-foo-bar', '/usr/local/...']
Globals ['__builtins__', '__name__', '__file__', 'sys', '__package__', '__doc__']
Traceback (most recent call last):
File "so-foo-bar/foo.py", line 9, in
raise Exception("foo")
Exception: foo
While I'd guess that run_script_as_main is lacking in some interesting details, it's pretty close to the way Python would run foo.py.
The foo.py need not be on the same folder as the main folder. Use
export PYTHONPATH=$HOME/dirWithFoo/:$PYTHONPATH
To add the path that foo resides to Python path. Now you can
from foo import myfunc
myfunc()
And make myfunc do whatever u want
Have you tried this?
bar.py
imp.load_source('__main__', '../../foo.py')
for me it runs whatever is in foo.py (before and inside main)
After, maybe what you do in bar.py might be more tricky that my basic test.
The good thing with load_source is that it imports the .pyc if it already exists or initiates a "compile" of the .py and then imports the .pyc.
This solution is what ended up working well for me:
#!/usr/bin/env python
import imp
import sys
import os.path
file_path = sys.argv.pop(1)
assert os.path.exists(file_path)
folder, file_name = os.path.split(file_path)
###############################################################################
# #
# Debugging cruft here
# #
###############################################################################
module_name, _ = os.path.splitext(file_name)
sys.path.append(folder)
imp.load_module('__main__', *imp.find_module(module_name))
I am using jython with a third party application. The third party application has some builtin libraries foo. To do some (unit) testing we want to run some code outside of the application. Since foo is bound to the application we decided to write our own mock implementation.
However there is one issue, we implemented our mock class in python while their class is in java. Thus to use their code one would do import foo and foo is the mock class afterwards. However if we import the python module like this we get the module attached to the name, thus one has to write foo.foo to get to the class.
For convenience reason we would love to be able to write from ourlib.thirdparty import foo to bind foo to the foo-class. However we would like to avoid to import all the classes in ourlib.thirdparty directly, since the loading time for each file takes quite a while.
Is there any way to this in python? ( I did not get far with Import hooks I tried simply returning the class from load_module or overwriting what I write to sys.modules (I think both approaches are ugly, particularly the later))
edit:
ok: here is what the files in ourlib.thirdparty look like simplified(without magic):
foo.py:
try:
import foo
except ImportError:
class foo
....
Actually they look like this:
foo.py:
class foo
....
__init__.py in ourlib.thirdparty
import sys
import os.path
import imp
#TODO: 3.0 importlib.util abstract base classes could greatly simplify this code or make it prettier.
class Importer(object):
def __init__(self, path_entry):
if not path_entry.startswith(os.path.join(os.path.dirname(__file__), 'thirdparty')):
raise ImportError('Custom importer only for thirdparty objects')
self._importTuples = {}
def find_module(self, fullname):
module = fullname.rpartition('.')[2]
try:
if fullname not in self._importTuples:
fileObj, self._importTuples[fullname] = imp.find_module(module)
if isinstance(fileObj, file):
fileObj.close()
except:
print 'backup'
path = os.path.join(os.path.join(os.path.dirname(__file__), 'thirdparty'), module+'.py')
if not os.path.isfile(path):
return None
raise ImportError("Could not find dummy class for %s (%s)\n(searched:%s)" % (module, fullname, path))
self._importTuples[fullname] = path, ('.py', 'r', imp.PY_SOURCE)
return self
def load_module(self, fullname):
fp = None
python = False
print fullname
if self._importTuples[fullname][1][2] in (imp.PY_SOURCE, imp.PY_COMPILED, imp.PY_FROZEN):
fp = open( self._importTuples[fullname][0], self._importTuples[fullname][1][1])
python = True
try:
imp.load_module(fullname, fp, *self._importTuples[fullname])
finally:
if python:
module = fullname.rpartition('.')[2]
#setattr(sys.modules[fullname], module, getattr(sys.modules[fullname], module))
#sys.modules[fullname] = getattr(sys.modules[fullname], module)
if isinstance(fp, file):
fp.close()
return getattr(sys.modules[fullname], module)
sys.path_hooks.append(Importer)
As others have remarked, it is such a plain thing in Python that the import statement iself has a syntax for that:
from foo import foo as original_foo, for example -
or even import foo as module_foo
Interesting to note is that the import statemente binds a name to the imported module or object ont he local context - however, the dictionary sys.modules (on the moduels sys of course), is a live reference to all imported modules, using their names as a key. This mechanism plays a key role in avoding that Python re-reads and re-executes and already imported module , when running (that is, if various of yoru modules or sub-modules import the samefoo` module, it is just read once -- the subsequent imports use the reference stored in sys.modules).
And -- besides the "import...as" syntax, modules in Python are just another object: you can assign any other name to them in run time.
So, the following code would also work perfectly for you:
import foo
original_foo = foo
class foo(Mock):
...
I am attempting to run some unit tests in python from what I believe is a module. I have a directory structure like
TestSuite.py
UnitTests
|__init__.py
|TestConvertStringToNumber.py
In testsuite.py I have
import unittest
import UnitTests
class TestSuite:
def __init__(self):
pass
print "Starting testting"
suite = unittest.TestLoader().loadTestsFromModule(UnitTests)
unittest.TextTestRunner(verbosity=1).run(suite)
Which looks to kick off the testing okay but it doesn't pick up any of the test in TestConvertNumberToString.py. In that class I have a set of functions which start with 'test'.
What should I be doing such that running python TestSuite.py actually kicks off all of my tests in UnitTests?
Here is some code which will run all the unit tests in a directory:
#!/usr/bin/env python
import unittest
import sys
import os
unit_dir = sys.argv[1] if len(sys.argv) > 1 else '.'
os.chdir(unit_dir)
suite = unittest.TestSuite()
for filename in os.listdir('.'):
if filename.endswith('.py') and filename.startswith('test_'):
modname = filename[:-2]
module = __import__(modname)
suite.addTest(unittest.TestLoader().loadTestsFromModule(module))
unittest.TextTestRunner(verbosity=2).run(suite)
If you call it testsuite.py, then you would run it like this:
testsuite.py UnitTests
Using Twisted's "trial" test runner, you can get rid of TestSuite.py, and just do:
$ trial UnitTests.TestConvertStringToNumber
on the command line; or, better yet, just
$ trial UnitTests
to discover and run all tests in the package.