Develop Reference

Annotate a function argument as being a specific module

I have a pytest fixture that imports a specific module. This is needed as importing the module is very expensive, so we don't want to do it on import-time (i.e. during pytest test collection). This results in code like this:
#pytest.fixture
def my_module_fix():
import my_module
yield my_module
def test_something(my_module_fix):
assert my_module_fix.my_func() = 5
I am using PyCharm and would like to have type-checking and autocompletion in my tests. To achieve that, I would somehow have to annotate the my_module_fix parameter as having the type of the my_module module.
I have no idea how to achieve that. All I found is that I can annotate my_module_fix as being of type types.ModuleType, but that is not enough: It is not any module, it is always my_module.

If I get your question, you have two (or three) separate goals
Deferred import of slowmodule
Autocomplete to continue to work as if it was a standard import
(Potentially?) typing (e.g. mypy?) to continue to work
I can think of at least five different approaches, though I'll only briefly mention the last because it's insane.
Import the module inside your tests
This is (by far) the most common and IMHO preferred solution.
e.g. instead of
import slowmodule
def test_foo():
slowmodule.foo()
def test_bar():
slowmodule.bar()
you'd write:
def test_foo():
import slowmodule
slowmodule.foo()
def test_bar():
import slowmodule
slowmodule.bar()
[deferred importing] Here, the module will be imported on-demand/lazily. So if you have pytest setup to fail-fast, and another test fails before pytest gets to your (test_foo, test_bar) tests, the module will never be imported and you'll never incur the runtime cost.
Because of Python's module cache, subsequent import statements won't actually re-import the module, just grab a reference to the already-imported module.
[autocomplete/typing] Of course, autocomplete will continue to work as you expect in this case. This is a perfectly fine import pattern.
While it does require adding potentially many additional import statements (one inside each test function), it's immediately clear what is going on (regardless of whether it's clear why it's going on).
[3.7+] Proxy your module with module __getattr__
If you create a module (e.g. slowmodule_proxy.py) with the contents like:
def __getattr__(name):
import slowmodule
return getattr(slowmodule, name)
And in your tests, e.g.
import slowmodule
def test_foo():
slowmodule.foo()
def test_bar():
slowmodule.bar()
instead of:
import slowmodule
you write:
import slowmodule_proxy as slowmodule
[deferred import] Thanks to PEP-562, you can "request" any name from slowmodule_proxy and it will fetch and return the corresponding name from slowmodule. Just as above, including the import inside the function will cause slowmodule to be imported only when the function is called and executed instead of on module load. Module caching still applies here of course, so you're only incurring the import penalty once per interpreter session.
[autocomplete] However, while deferred importing will work (and your tests run without issue), this approach (as stated so far) will "break" autocomplete:
Now we're in the realm of PyCharm. Some IDEs will perform "live" analysis of modules and actually load up the module and inspect its members. (PyDev had this option). If PyCharm did this, implementing module.__dir__ (same PEP) or __all__ would allow your proxy module to masquerade as the actual slowmodule and autocomplete would work.† But, PyCharm does not do this.
Nonetheless, you can fool PyCharm into giving you autocomplete suggestions:
if False:
import slowmodule
else:
import slowmodule_proxy as slowmodule
The interpreter will only execute the else branch, importing the proxy and naming it slowmodule (so your test code can continue to reference slowmodule unchanged).
But PyCharm will now provide autocompletion for the underlying module:
† While live-analysis can be an incredibly helpful, there's also a (potential) security concern that comes with it that static syntax analysis doesn't have. And the maturation of type hinting and stub files has made it less of an issue still.
Proxy slowmodule explicitly
If you really hated the dynamic proxy approach (or the fact that you have to fool PyCharm in this way), you could proxy the module explicitly.
(You'd likely only want to consider this if the slowmodule API is stable.)
If slowmodule has methods foo and bar you'd create a proxy module like:
def foo(*args, **kwargs):
import slowmodule
return slowmodule.foo(*args, **kwargs)
def bar(*args, **kwargs):
import slowmodule
return slowmodule.bar(*args, **kwargs)
(Using args and kwargs to pass arguments through to the underlying callables. And you could add type hinting to these functions to mirror the slowmodule functions.)
And in your test,
import slowmodule_proxy as slowmodule
Same as before. Importing inside the method gives you the deferred importing you want and the module cache takes care of multiple import calls.
And since it's a real module whose contents can be statically analyzed, there's no need to "fool" PyCharm.
So the benefit of this solution is that you don't have a bizarre looking if False in your test imports. This, however, comes at the (substantial) cost of having to maintain a proxy file alongside your module -- which could prove painful in the case that slowmodule's API wasn't stable.
[3.5+] Use importlib's LazyLoader instead of a proxy module
Instead of the proxy module slowmodule_proxy, you could follow a pattern similar to the one shown in the importlib docs
>>> import importlib.util
>>> import sys
>>> def lazy_import(name):
... spec = importlib.util.find_spec(name)
... loader = importlib.util.LazyLoader(spec.loader)
... spec.loader = loader
... module = importlib.util.module_from_spec(spec)
... sys.modules[name] = module
... loader.exec_module(module)
... return module
...
>>> lazy_typing = lazy_import("typing")
>>> #lazy_typing is a real module object,
>>> #but it is not loaded in memory yet.
You'd still need to fool PyCharm though, so something like:
if False:
import slowmodule
else:
slowmodule = lazy_import('slowmodule')
would be necessary.
Outside of the single additional level of indirection on module member access (and the two minor version availability difference), it's not immediately clear to me what, if anything, there is to be gained from this approach over the previous proxy module method, however.
Use importlib's Finder/Loader machinery to hook import (don't do this)
You could create a custom module Finder/Loader that would (only) hook your slowmodule import and, instead load, for example your proxy module.
Then you could just import that "importhook" module before you imported slowmode in your tests, e.g.
import myimporthooks
import slowmodule
def test_foo():
...
(Here, myimporthooks would use importlib's finder and loader machinery to do something simlar to the importhook package but intercept and redirect the import attempt rather than just serving as an import callback.)
But this is crazy. Not only is what you want (seemingly) achievable through (infinitely) more common and supported methods, but it's incredibly fragile, error-prone and, without diving into the internals of PyTest (which may mess with module loaders itself), it's hard to say whether it'd even work.

When Pytest collects files to be tested, modules are only imported once, even if the same import statement appears in multiple files.
To observe when my_module is imported, add a print statement and then use the Pytest -s flag (short for --capture=no), to ensure that all standard output is displayed.
my_module.py
answer: int = 42
print("MODULE IMPORTED: my_module.py")
You could then add your test fixture to a conftest.py file:
conftest.py
import pytest
#pytest.fixture
def my_module_fix():
import my_module
yield my_module
Then in your test files, my_module.py may be imported to add type hints:
test_file_01.py
import my_module
def test_something(my_module_fix: my_module):
assert my_module_fix.answer == 42
test_file_02.py
import my_module
def test_something2(my_module_fix: my_module):
assert my_module_fix.answer == 42
Then run Pytest to display all standard output and verify that the module is only imported once at runtime.
pytest -s ./
Output from Pytest
platform linux -- Python 3.9.7, pytest-6.2.5
rootdir: /home/your_username/your_repo
collecting ... MODULE IMPORTED: my_module.py <--- Print statement executed once
collected 2 items
test_file_01.py .
test_file_02.py .

This is quick and naive, but can you possibly annotate your tests with the "quote-style" annotation that is meant for different purposes, but may suit here by skipping the import at runtime but still help your editor?
def test_something(my_module_fix: "my_module"):
In a quick test, this seems to accomplish it at least for my setup.

Although it might not be considered a 'best practice', to keep your specific use case simple, you could just lazily import the module directly in your test where you need it.
def test_something():
import my_module
assert my_module.my_func() = 5
I believe Pytest will only import the module when the applicable tests run. Pytest should also 'cache' the import as well so that if multiple tests import the module, it is only actually imported once. This may also solve your autocomplete issues for your editor.
Side-note: Avoid writing code in a specific way to cater for a specific editor. Keep it simple, not everyone who looks at your code will use Pycharm.

would like to have type-checking and autocompletion in my tests
It sounds like you want Something that fits as the type symbol of your test function:
def test_something(my_module_fix: Something):
assert my_module_fix.my_func() = 5
... and from this, [hopefully] your IDE can make some inferences about my_module_fix. I'm not a PyCharm user, so I can't speak to what it can tell you from type signatures, but I can say this isn't something that's readily available.
For some intuition, in this example -- Something is a ModuleType like 3 is an int. Analogously, accessing a nonexistent attribute of Something is like doing something not allowed with 3. Perhaps, like accessing an attribute of it (3.__name__).
But really, I seems like you're thinking about this from the wrong direction. The question a type signature answers is: what contract must this [these] argument[s] satisfy for this function to use it [them]. Using the example above, a type of 3 is the kind of thing that is too specific to make useful functions:
def add_to(i: 3):
return i
Perhaps a better name for your type is:
def test_something(my_module_fix: SomethingThatHasMyFuncMethod):
assert my_module_fix.my_func() = 5
So the type you probably want is something like this:
class SomethingThatHasMyFuncMethod:
def my_func(self) -> int: ...
You'll need to define it (maybe in a .pyi file). See here for info.
Finally, here's some unsolicited advice regarding:
importing the module is very expensive, so we don't want to do it on import-time
You should probably employ some method of making this module do it's thing lazily. Some of the utils in the django framework could serve as a reference point. There's also the descriptor protocol which is a bit harder to grok but may suit your needs.

You want to add type hinting for the arguments of test_something, in particular to my_module_fix. This question is hard because we can't import my_module at the top.
Well, what is the type of my_module? I'm going to assume that if you do import my_module and then type(my_module) you will get <class 'module'>.
If you're okay with not telling users exactly which module it has to be, then you could try this.
from types import ModuleType
#pytest.fixture
def my_module_fix():
import my_module
yield my_module
def test_something(my_module_fix: ModuleType):
assert my_module_fix.my_func() = 5
The downside is that a user might infer that any old module would be suitable for my_module_fix.
You want it to be more specific? There's a cost to that, but we can get around some of it (it's hard to speak on performance in VS Code vs PyCharm vs others).
In that case, let's use TYPE_CHECKING. I think this was introduced in Python 3.6.
from types import ModuleType
from typing import TYPE_CHECKING
if TYPE_CHECKING:
import my_module
#pytest.fixture
def my_module_fix():
import my_module
yield my_module
def test_something(my_module_fix: my_module):
assert my_module_fix.my_func() = 5
You won't pay these costs during normal run time. You will pay the cost of importing my_module when your type checker is doing what it does.
If you're on an earlier verison of Python, you might need to do this instead.
from types import ModuleType
from typing import TYPE_CHECKING
if TYPE_CHECKING:
import my_module
#pytest.fixture
def my_module_fix():
import my_module
yield my_module
def test_something(my_module_fix: "my_module"):
assert my_module_fix.my_func() = 5
The type checker in VS Code is smart enough to know what's going on here. I can't speak for other type checkers.

Should I always use the most pythonic way to import modules?

I am making a tiny framework for games with pygame, on which I wish to implement basic code to quickly start new projects. This will be a module that whoever uses should just create a folder with subfolders for sprite classes, maps, levels, etc.
My question is, how should my framework module load these client modules? I was considering to design it so the developer could just pass to the main object the names of the directories, like:
game = Game()
game.scenarios = 'scenarios'
Then game will append 'scenarios' to sys.path and use __import__(). I've tested and it works.
But then I researched a little more to see if there were already some autoloader in python, so I could avoid to rewrite it, and I found this question Python modules autoloader?
Basically, it is not recommended to use a autoloader in python, since "explicit is better than implicit" and "Readability counts".
That way, I think, I should compel the user of my module to manually import each of his/her modules, and pass these to the game instance, like:
import framework.Game
import scenarios
#many other imports
game = Game()
game.scenarios = scenarios
#so many other game.whatever = whatever
But this doesn't looks good to me, not so confortable. See, I am used to work with php, and I love the way it works with it's autoloader.
So, the first exemple has some problability to crash or be some trouble, or is it just not 'pythonic'?
note: this is NOT an web application

I wouldn't consider letting a library import things from my current path or module good style. Instead I would only expect a library to import from two places:
Absolute imports from the global modules space, like things you have installed using pip. If a library does this, this library must also be found in its install_requires=[] list
Relative imports from inside itself. Nowadays these are explicitly imported from .:
from . import bla
from .bla import blubb
This means that passing an object or module local to my current scope must always happen explicitly:
from . import scenarios
import framework
scenarios.sprites # attribute exists
game = framework.Game(scenarios=scenarios)
This allows you to do things like mock the scenarios module:
import types
import framework
# a SimpleNamespace looks like a module, as they both have attributes
scenarios = types.SimpleNamespace(sprites='a', textures='b')
scenarios.sprites # attribute exists
game = framework.Game(scenarios=scenarios)
Also you can implement a framework.utils.Scenario() class that implements a certain interface to provide sprites, maps etc. The reason being: Sprites and Maps are usually saved in separate files: What you absolutely do not want to do is look at the scenarios's __file__ attribute and start guessing around in its files. Instead implement a method that provides a unified interface to that.
class Scenario():
def __init__(self):
...
def sprites(self):
# optionally load files from some default location
# If no such things as a default location exists, throw a NotImplemented error
...
And your user-specific scenarios will derive from it and optionally overload the loading methods
import framework.utils
class Scenario(framework.utils.Scenario):
def __init__(self):
...
def sprites(self):
# this method *must* load files from location
# accessing __file__ is OK here
...
What you can also do is have framework ship its own framework.contrib.scenarios module that is used in case no scenarios= keyword arg was used (i.e. for a square default map and some colorful default textures)
from . import contrib
class Game()
def __init__(self, ..., scenarios=None, ...):
if scenarios is None:
scenarios = contrib.scenarios
self.scenarios = scenarios

How to mock Python classes when nested several dependencies deep

If I have the following architecture...
Please note the edits below. It occurred to me (after some recent refactoring) that there are actually three classes in three different files. Sorry that the file/class names are getting ridiculous. I assure you those are not the real names. :)
main_class.py
class MainClass(object):
def do_some_stuff(self):
dependent_class = DependentClass()
dependent_class.py
class DependentClass(object):
def __init__():
dependent_dependent_class = DependentDependentClass()
dependent_dependent_class.do_dependent_stuff()
dependent_dependent_class.py
class DependentDependentClass(object):
def do_dependent_stuff(self):
print "I'm gonna do production stuff that I want to mock"
print "Like access a database or interact with a remote server"
class MockDependentDependentClass(object):
def do_dependent_stuff(self):
print "Respond as if the production stuff was all successful."
and I want to call main_class.do_some_stuff during testing but, during its execution, I want instances of DependentDependentClass replaced with MockDependentDependentClass how can I do that pythonically using best practices.
Currently, the best thing I could come up with is to conditionally instantiate one class or the other based on the presence/value of an environment variable. It certainly works but is pretty dirty.
I spent some time reading about the unittest.mock and mock.patch functions and they seem like they might be able to help but each description that I could wrap my head around seemed to be a little different than my actual use case.
The key is that I don't want to define mock return values or attributes but that I want the namespace changed, globally, I guess, such that when my application thinks it is instantiating DependentClass it is actually instantiating MockDependentClass.
The fact that I can't find any examples of anyone doing exactly this means one of two things:
It's because I'm doing it in a very dumb/naive way.
I'm doing something so genius no else has ever encountered it.
... I assume it's number 1...
Full disclosure, unit testing is not something with which I am skilled. It's an effort that my internal tools development team is trying to catch up to step our game up a bit. It's possible that I'm not thinking about testing correctly.
Any thoughts would be most welcome. Thank you, in advance!
SOLUTION!!!
Thanks to #de1 for the help. Given my clever architecture shown above the following accomplishes what I want.
The following code is located in main_class.py
import dependent_class
from dependent_dependent_class import MockDependentDependentClass
with patch.object(dependent_class, "DependentDependentClass", MockDependentDependentClass):
main_class = MainClass()
main_class.do_some_stuff()
The code seems to (and hell if I know how it's doing this) manipulate the namespace within the module dependent_class so that, while inside the with block (that's a context manager for anyone who is hung up on that part) anything referring to the class object DependentDependentClass will actually be referencing MockDependentDependentClass.

The mock module does indeed seem to be a good fit in this case. You can specify the mock (your mock) to use when calling the various patch methods.
If you are importing only the class rather than the module you can patch the imported DependentDependentClass in DependentClass:
import .DependentClass as dependent_class
from .DependentDependentClass import MockDependentDependentClass
with patch.object(dependent_class, 'DependentDependentClass', MockDependentDependentClass):
# do something while class is patched
Alternatively:
with patch('yourmodule.DependentClass.DependentDependentClass', MockDependentDependentClass):
# do something while class is patched
or the following will only work if you are accessing the class via a module or import it after it is being patched:
with patch('yourmodule.DependentDependentClass.DependentDependentClass', MockDependentDependentClass):
# do something while class is patched
Just bare in mind what object is being patched, when.
Note: you might find it less confusing naming your files in lower case, slightly different to the embedded class(es).
Note 2: If you need to mock a dependency of a dependency of the module under test then it might suggest that you are not testing at the right level.

Module name different than directory name?

Let's assume I have a python package called bestpackage.
Convention dictates that bestpacakge would also be a directory on sys.path that contains an __init__.py to make the interpreter assume it can be imported from.
Is there any way I can set a variable for the package name so the directory could be named something different than the directive I import it with? Is there any way to make the namespacing not care about the directory name and honor some other config instead?
My super trendy client-side devs are just so much in love with these sexy something.otherthing.js project names for one of our smaller side projects!
EDIT:
To clarify, the main purpose of my question was to allow my client side guys continue to call the directories in their "projects" (the one we all have added to our paths) folder using their existing convention (some.app.js), even though in some cases they are in fact python packages that will be on the path and sourced to import statements internally. I realize this is in practice a pretty horrible thing and so I ask more out of curiosity. So part of the big problem here is circumventing the fact that the . in the directory name (and thereby the assumed package name) implies attribute access. It doesn't really surprise me that this cannot be worked around, I was just curious if it was possible deeper in the "magic" behind import.
There's some great responses here, but all rely on doing a classical import of some kind where the attribute accessor . will clash with the directory names.

A directory with a __init__.py file is called a package.
And no, the package name is always the same as the directory. That's how Python can discover packages, it matches it against directory names found on the search path, and if there is a __init__.py file in that directory it has found a match and imports the __init__.py file contained.
You can always import something into your local module namespace under a shorter, easier to use name using the from module import something or the import module as alias syntax:
from something.otherthing.js import foo
from something.otherthing import js as bar
import something.otherthing.js as hamspam

There is one solution wich needs one initial import somewhere
>>> import sys
>>> sys.modules['blinot_existing_blubb'] = sys
>>> import blinot_existing_blubb
>>> blinot_existing_blubb
<module 'sys' (built-in)>
Without a change to the import mechanism you can not import from an other name. This is intended, I think, to make Python easier to understand.
However if you want to change the import mechanism I recommend this: Getting the Most Out of Python Imports

Well, first I would say that Python is not Java/Javascript/C/C++/Cobol/YourFavoriteLanguageThatIsntPython. Of course, in the real world, some of us have to answer to bosses who disagree. So if all you want is some indirection, use smoke and mirrors, as long as they don't pay too much attention to what's under the hood. Write your module the Python way, then provide an API on the side in the dot-heavy style that your coworkers want. Ex:
pythonic_module.py
def func_1():
pass
def func_2():
pass
def func_3():
pass
def func_4():
pass
indirection
/dotty_api_1/__init__.py
from pythonic_module import func_1 as foo, func_2 as bar
/dotty_api_2/__init__.py
from pythonic_module import func_3 as foo, func_4 as bar
Now they can dot to their hearts' content, but you can write things the Pythonic way under the hood.

Actually yes!
you could do a canonical import Whatever or newmodulename = __import__("Whatever")
python keeps track of your modules and you can inspect that by doing:
import sys
print sys.modules
See this article more details
But that's maybe not your problem? let's guess: you have a module in a different path, which your current project can't access because it's not in the sys-path?
well the just add:
import sys
sys.path.append('path_to_the_other_package_or_module_directory')
prior to your import statement or see this SO-post for a more permanent solution.

I was looking for this to happen with setup.py at sdist and install time, rather than runtime, and found the directive package_dir:
https://docs.python.org/3.5/distutils/setupscript.html#listing-whole-packages

How can I inject an object into another namespace in python?

I'm writing some unittests for code written by someone else here at the office. Python is not my strongest language. While I've been successful with basic unit tests, mocking in python is throwing me for a loop.
What I need to do is override a call to ConfigObj and inject my own mock config/fixture into any ConfigObj call.
settings.py
from configobj import ConfigObj
config = ConfigObj('/etc/myapp/config')
utils.py
from settings import config
"""lots of stuff methods using various config values."""
What I would like to do is, in my unittests for utils.py, inject myself either for ANY call to ConfigObj or settings.py itself.
Many of the mocking libraries expect me to Mock my own classes but in the case of this app, it doesn't have any explicit classes.
Can it be done or are the python namespace restrictions too strict that I can't intervene in what a module that I'm importing imports itself?
Side note: running 2.7 so I can't do any of the tricks I've read about from 2.5.

If the tests are in a separate file from from settings.py and utils.py you can create a file mock.py
import configobj
class MockConfigObj(object):
#mock whatever you wan
configobj.ConfigObj = MockConfigObj
and then import mock before importing (from) any module that itself imports settings. This will ensure that settings.config is created with MockConfigObj. If you want a uniform global mocking, import it before any file that imports configobj.
This works because python will store configobj in sys.modules and check that before actually reading from a file on subsequent imports. in mock.py, the identifier ConfigObj is just a reference to the entry in sys.modules so that any changes that you make will be globally visible.
This strikes me as a little hacky though but it's the best that I can think of.

Python namespaces are not strict at all within the same scope. Just override the variable name containing your object (or the class itself and provided it) within the same scope you'd be expecting the original and that is good enough.
Now, whether or not what you're replacing it with behaves the same is up to you...

Couldn't you just overwrite the original function with another one?
There are no constants in Python, you can change everything, you could even do True = False.

I faced a similar situation before. Here is how I would go about addressing your problem.
Consider a test case for a function from utils.py.
import utils, unittest
class FooFunctionTests(unittest.TestCase):
def setUp(self):
utils._old_config = utils.config
utils.config = MockClass()
def tearDown(self):
utils.config = utils._old_config
del utils._old_config
def test_foo_function_returns_correct_value(self):
self.assertEqual("success!", utils.foo())

The following page is a good one on mocking and import
http://www.relaxdiego.com/2014/04/mocking-objects-in-python.html
Say you have a file named my_package1.py with the following code:
class A(object):
def init(self):
and you then import that in my_package2.py with the code
from my_package1 import A
class A(object):
def init(self):
The first line of my_package2.py creates a variable under the my_package2 namespace called A. Now you have two variables my_package1.A and my_package2.A that both point to the same class in memory. If you want the code in my_package2.py to use a mocked up class A, then you will need to mock my_package2.A not my_package1.A