Develop Reference

Dynamically importing modules as global variables

I am trying to dynamically import modules and get it as global variable.
I am using maya 2020 python interpreter (Python 2.7)
I have a test module called "trigger_test_script.py" under "/home/arda.kutlu/Downloads/" folder.
When I dont import any custom class and run this:
###########################################################################[START]
import sys
import imp
class TestClass(object):
def __init__(self):
self.filePath = None
self.asName = None
def action(self):
exec("global %s" % self.asName, globals())
foo = "imp.load_source('%s', '/home/arda.kutlu/Downloads/trigger_test_script.py')" %self.asName
cmd = "{0}={1}".format(self.asName, foo)
exec(cmd, globals())
###########################################################################[END]
test = TestClass()
test.filePath = "/home/arda.kutlu/Downloads/trigger_test_script.py"
test.asName = "supposed_to_be_global"
test.action()
print(supposed_to_be_global)
I get the exact result that I want:
<module 'trigger_test_script' from '/home/arda.kutlu/Downloads/trigger_test_script.pyc'>
However, when I save TestClass (the part between hashes) into a file and import it like this:
import testClass
test = testClass.TestClass()
test.filePath = "/home/arda.kutlu/Downloads/trigger_test_script.py"
test.asName = "supposed_to_be_global"
test.action()
print(supposed_to_be_global)
the variable which 'supposed_to_be_global' is not becoming global and I get the NameError.
I always assumed that these two usages should return the same result but clearly I am missing something.
I appreciate any help, thanks.

I don't quite understand your last comment about having several modules with different action() methods is a problem. So ignoring that, here's how to make what's in your question work, The part in the hashes will work both in-line or if put in a separate module and imported.
###########################################################################[START]
import imp
class TestClass(object):
def __init__(self):
self.filePath = None
self.asName = None
def action(self):
foo = imp.load_source(self.asName, self.filePath)
return foo
###########################################################################[END]
#from testclass import TestClass
test = TestClass()
test.filePath = "/home/arda.kutlu/Downloads/trigger_test_script.py"
test.asName = "supposed_to_be_global"
supposed_to_be_global = test.action()
print(supposed_to_be_global)

Python - unittest, mock, patch, input

so i've got a problem with my code.
File 1:
class Abc(object):
...
def function1(self):
#do something
def function2(self):
x = input()
return x+1
and now i'm trying to test function 2 so i wrote a test for it and i don't know what i am doing wrong:
from unittest.mock import patch
import unittest
from file1 import *
class TestBackend(unittest.TestCase):
def test_mode_first(self):
self.assertEqual(Abc().funcion1(), 30)
#patch('funcion2.input', create=True)
def test_mode_second(self, mocked_input):
mocked_input.side_effect = ["QWE"]
result = Abc().funcion2()
self.assertEqual(result, 10)
if __name__ == '__main__':
unittest.main()
i get ModuleNotFoundError: No module named 'function2'
so what i am doing wrong in here?
thanks for your help :)

You get ModuleNotFoundError because funcion2 is not a module. patch doc is clear about this:
target should be a string in the form 'package.module.ClassName'. The
target is imported and the specified object replaced with the new
object, so the target must be importable from the environment you are
calling patch() from. The target is imported when the decorated
function is executed, not at decoration time.
This works for me when executed with python3 -m unittest discover from the directory the files are in.
BTW you have a couple of typos in your example, e.g. Abc().funcion2(), note the missing t in funcion2.
Also, try not to use from … import *: https://docs.quantifiedcode.com/python-anti-patterns/maintainability/from_module_import_all_used.html#using-wildcard-imports-from-import
# file1.py
class Abc(object):
def function1(self):
return 30
def function2(self):
x = input()
return x + "1"
# test_file1.py
import unittest
from unittest.mock import patch
from file1 import Abc
class TestBackend(unittest.TestCase):
def test_mode_first(self):
self.assertEqual(Abc().function1(), 30)
#patch('builtins.input')
def test_mode_second(self, mocked_input):
mocked_input.return_value = "QWE"
result = Abc().function2()
self.assertEqual(result, "QWE1")

How do I override imports of other files for unit testing

I am currently attempting to write unit tests for my Main.py's main() function
Here is a simplified version of my Main.py:
from Configuration import Configuration # Configuration.py is a file in the same dir
def main():
try:
Configuration('settings.ini')
except:
sys.exit(1) # Test path1
sys.exit(0) # Test path2
if __name__ == '__main__':
main()
In my Unit Tests\MainUnitTests.py I want to import ..\Main.py and fake the Configuration class in such a way that I can hit Test path1 and Test path2
I found that i can assert sys.exit() with the following:
with self.assertRaises(SystemExit) as cm:
main()
self.assertEqual(cm.exception.code, 1)
but I am having trouble overriding the from Configuration import Configuration
Thoughts?
So far I have tried the following within Unit Tests\MainUnitTests.py:
class FakeFactory(object):
def __init__(self, *a):
pass
sys.modules['Configuration'] = __import__('FakeFactory')
class Configuration(FakeFactory):
pass
Another example for demonstration:
foo.py:
from bar import a,b
x = a()
class a(object):
def __init__(self):
self.q = 2
y = a()
print x.q, y.q # prints '1 2' as intended
b() # I want this to print 2 without modifying bar.py
bar.py:
class a(object):
def __init__(self):
self.q = 1
def b():
t = a()
print t.q

when you use the import
from bar import a
it import the name directly into the module, so monkeypatching bar won't help, you need to override a directly in the main file:
def fake_a():
print("OVERRIDEN!")
main.a = fake_a
Do know that unittest has helper functions for this in the mock subpackage, I believe you could do something like:
from unittest.mock import patch
...
with patch("main.a", fake_a) as mock_obj: #there are additional things you can do with the mock_obj
do_stuff()
This would work in your first example with configuration since the class that needs to be patched is not used in the global scope although foo uses bar.a as soon as it is loaded so you would need to patch it before even loading foo:
from unittest.mock import patch
...
with patch("bar.a", fake_a) as mock_obj: #there are additional things you can do with the mock_obj
import foo #now when it loads it will be loaded with the patched name
However in this case foo.a would not be reverted at the end of the with block because it can't be caught by unittest... I really hope your actual use case doesn't use the stuff to be patched at module level.

How to benchmark unit tests in Python without adding any code

I have a Python project with a bunch of tests that have already been implemented, and I'd like to begin benchmarking them so I can compare performance of the code, servers, etc over time. Locating the files in a manner similar to Nose was no problem because I have "test" in the names of all my test files anyway. However, I'm running into some trouble in attempting to dynamically execute these tests.
As of right now, I'm able to run a script that takes a directory path as an argument and returns a list of filepaths like this:
def getTestFiles(directory):
fileList = []
print "Searching for 'test' in " + directory
if not os.path.isdir(os.path.dirname(directory)):
# throw error
raise InputError(directory, "Not a valid directory")
else:
for root, dirs, files in os.walk(directory):
#print files
for f in files:
if "test" in f and f.endswith(".py"):
fileList.append(os.path.join(root, f))
return fileList
# returns a list like this:
# [ 'C:/Users/myName/Desktop/example1_test.py',
# 'C:/Users/myName/Desktop/example2_test.py',
# 'C:/Users/myName/Desktop/folder1/example3_test.py',
# 'C:/Users/myName/Desktop/folder2/example4_test.py'... ]
The issue is that these files can have different syntax, which I'm trying to figure out how to handle. For example:
TestExampleOne:
import dummy1
import dummy2
import dummy3
class TestExampleOne(unittest.TestCase):
#classmethod
def setUpClass(cls):
# set up
def test_one(self):
# test stuff
def test_two(self):
# test stuff
def test_three(self):
# test stuff
# etc...
TestExampleTwo:
import dummy1
import dummy2
import dummy3
def setup(self):
try:
# config stuff
except Exception as e:
logger.exception(e)
def test_one():
# test stuff
def test_two():
# test stuff
def test_three():
# test stuff
# etc...
TestExampleThree:
import dummy1
import dummy2
import dummy3
def setup(self):
try:
# config stuff
except Exception as e:
logger.exception(e)
class TestExampleTwo(unittest.TestCase):
def test_one(self):
# test stuff
def test_two(self):
# test stuff
# etc...
class TestExampleThree(unittest.TestCase):
def test_one(self):
# test stuff
def test_two(self):
# test stuff
# etc...
# etc...
I would really like to be able to write one module that searches a directory for every file containing "test" in its name, and then executes every unit test in each file, providing execution time for each test. I think something like NodeVisitor is on the right track, but I'm not sure. Even an idea of where to start would be greatly appreciated. Thanks

Using nose test runner would help to discover the tests, setup/teardown functions and methods.
nose-timer plugin would help with benchmarking:
A timer plugin for nosetests that answers the question: how much time
does every test take?
Demo:
imagine you have a package named test_nose with the following scripts inside:
test1.py:
import time
import unittest
class TestExampleOne(unittest.TestCase):
#classmethod
def setUpClass(cls):
cls.value = 1
def test_one(self):
time.sleep(1)
self.assertEqual(1, self.value)
test2.py:
import time
value = None
def setup():
global value
value = 1
def test_one():
time.sleep(2)
assert value == 1
test3.py:
import time
import unittest
value = None
def setup():
global value
value = 1
class TestExampleTwo(unittest.TestCase):
def test_one(self):
time.sleep(3)
self.assertEqual(1, value)
class TestExampleThree(unittest.TestCase):
def test_one(self):
time.sleep(4)
self.assertEqual(1, value)
install nose test runner:
pip install nose
install nose-timer plugin:
pip install nose-timer
run the tests:
$ nosetests test_nose --with-timer
....
test_nose.test3.TestExampleThree.test_one: 4.0003s
test_nose.test3.TestExampleTwo.test_one: 3.0010s
test_nose.test2.test_one: 2.0011s
test_nose.test1.TestExampleOne.test_one: 1.0005s
----------------------------------------------------------------------
Ran 4 tests in 10.006s
OK
The result is actually conveniently highlighted:
The coloring can be controlled by --timer-ok and --timer-warning arguments.
Note that time.sleep(n) calls were added for making the manual slowdowns to see the impact clearly. Also note that value variable is set to 1 in "setup" functions and methods, then in test function and methods the value is asserted to be 1 - this way you can see the work of setup functions.
UPD (running nose with nose-timer from script):
from pprint import pprint
import nose
from nosetimer import plugin
plugin = plugin.TimerPlugin()
plugin.enabled = True
plugin.timer_ok = 1000
plugin.timer_warning = 2000
plugin.timer_no_color = False
nose.run(plugins=[plugin])
result = plugin._timed_tests
pprint(result)
Save it into the test.py script and pass a target directory to it:
python test.py /home/example/dir/tests --with-timer
The result variable would contain:
{'test_nose.test1.TestExampleOne.test_one': 1.0009748935699463,
'test_nose.test2.test_one': 2.0003929138183594,
'test_nose.test3.TestExampleThree.test_one': 4.000233173370361,
'test_nose.test3.TestExampleTwo.test_one': 3.001115083694458}

Is there a way to get the name of the 'parent' module from an imported module?

I'm curious if it's possible to access the name of the parent module that some other module is imported into.
For instance, if I have a module (moduleA) and a parent is module, foo.py, into which it will be imported into, is it possible for moduleA to know where foo is located ?
ModuleA
def print_parent_module():
os.path.asbpath(#somehow access filename of parent module)
foo.py
import moduleA
print moduleA.print_parent_module()
>>> "foo.py"

I ran into a similar issue. You could make use of __name__:
parent_name = '.'.join(__name__.split('.')[:-1])
Or if you try to access the module directly (not the OP's question but related), see my answer in Is there a way to access parent modules in Python

No. Imported modules do not hold any form of state which stores data related to how they are imported.
I think the better question is, why would you even want to do it this way? You're aware that if foo.py is indeed your __main__ then you can easily get the name foo.py out of it (by using sys.argv)? And if foo.py is an imported module instead, then you obviously already know the name of foo.py and its location, etc. at the time that you import it.

Here is something I came with to store a parent's method name and variables to be recalled later in the class with a decorator.
import inspect
from functools import wraps
def set_reuse_vars(method):
#wraps(method)
def _impl(self, *method_args, **method_kwargs):
func_current = inspect.currentframe()
self.recall_func = dict()
self.recall_func['method_kwargs'] = func_current.f_locals['method_kwargs']
self.recall_func['method_name'] = func_current.f_locals['method'].__name__
return method(self, *method_args, **method_kwargs)
return _impl
class APIData(object):
def __init__(self):
self.client = None
self.response = None
self.recall_func = None
def get_next_page(self):
# Takes a pageToken to return the next result
get_func = getattr(self, self.recall_func['method_name'])
self.recall_func['method_kwargs']['pageToken'] = self.response['nextPageToken']
return get_func(**self.recall_func['method_kwargs'])
#set_reuse_vars
def search_list_by_keyword(self, **kwargs):
self.response = self.client.search().list(
**kwargs
).execute()
return self.response
script to run it.
api_data = APIData()
results = api_data.search_list_by_keyword(
part='snippet',
maxResults=25,
order='date')
print(results)
resultsTwo = api_data.get_next_page()
print(resultsTwo)

Figured it out!
A little import statement inside of the function can give access to the module name.
moduleA
def print_module_name():
import sys
return sys.argv[0]
Then, inside the "parent" module
# foo.py
import os
import moduleA
if __name__ == '__main__':
print os.path.split(moduleA.print_module_name())[-1]
Gives:
>>> 'foo.py'