I'm currently writing a class that needs os, stat and some others.
What's the best way to import these modules in my class?
I'm thinking about when others will use it, I want the 'dependency' modules to be already
imported when the class is instantiated.
Now I'm importing them in my methods, but maybe there's a better solution.
If your module will always import another module, always put it at the top as PEP 8 and the other answers indicate. Also, as #delnan mentions in a comment, sys, os, etc. are being used anyway, so it doesn't hurt to import them globally.
However, there is nothing wrong with conditional imports, if you really only need a module under certain runtime conditions.
If you only want to import them if the class is defined, like if the class is in an conditional block or another class or method, you can do something like this:
condition = True
if condition:
class C(object):
os = __import__('os')
def __init__(self):
print self.os.listdir
C.os
c = C()
If you only want it to be imported if the class is instantiated, do it in __new__ or __init__.
import sys
from importlib import import_module
class Foo():
def __init__(self):
# Depends on the configuration of the application.
self.condition = True # "True" Or "False"
if self.condition:
self.importedModule = import_module('moduleName')
# ---
if 'moduleName' in sys.modules:
self.importedModule.callFunction(params)
#or
if self.condition:
self.importedModule.callFunction(params)
# ---
PEP 8 on imports:
Imports are always put at the top of the file, just after any module
comments and docstrings, and before module globals and constants.
This makes it easy to see all modules used by the file at hand, and avoids having to replicate the import in several places when a module is used in more than one place. Everything else (e.g. function-/method-level imports) should be an absolute exception and needs to be justified well.
This (search for the section "Imports") official paper states, that imports should normally be put in the top of your source file. I would abide to this rule, apart from special cases.
Related
I know there are ways to perform dynamic import of Python modules themselves, but I would like to know if there's a way to write a module such that it can dynamically create its own module contents on demand. I am imagining a module hook that looks something like:
# In some_module.py:
def __import_name__(name):
return some_object
Such that if I were to write from some_module import foo in a script, Python will call some_module.__import_name__("foo") and let me dynamically create and return the contents.
I haven't found anything that works like this exactly in the documentation, though there are references to an "import protocol" with "finders" and "loaders" and "meta hooks" and "import path hooks" that permit customization of the import logic, and I imagine that such a thing is possible.
I discovered you can modify the behavior of a Module from within itself in arbitrary ways by setting sys.modules[__name__].__class__ to a class that implements whatever your chosen behavior.
import sys
import types
class DynamicModule(types.ModuleType):
# This function is what gets called on `from this_module import whatever`
# or `this_module.whatever` accesses.
def __getattr__(self, name):
# This check ensures we don't intercept special values like __path__
# if they're not set elsewhere.
if name.startswith("__") and name.endswith("__"):
return self.__getattribute__(name)
return make_object(name)
# Helpful to define this here if you need to dynamically construct the
# full set of available attributes.
#property
def __all__(self):
return get_all_objects()
# This ensures the DynamicModule class is used to define the behavior of
# this module.
sys.modules[__name__].__class__ = DynamicModule
Something about this feels like it may not be the intended path to do something like this, though, and that I should be hooking into the importlib machinery.
This question already has answers here:
Short description of the scoping rules?
(9 answers)
Closed 1 year ago.
Context: I'm writing a translator from one Python API to another, both in Python 3.5+. I load the file to be translated with a class named FileLoader, described by Fileloader.py. This file loader allows me to transfer the file's content to other classes doing the translation job.
All of the .py files describing each class are in the same folder
I tried two different ways to import my FileLoader module inside the other modules containing the classes doing the translation job. One seems to work, but the other didn't and I don't understand why.
Here are two code examples illustrating both ways:
The working way
import FileLoader
class Parser:
#
def __init__(self, fileLoader):
if isinstance(fileLoader, FileLoader.FileLoader)
self._fileLoader = fileLoader
else:
# raise a nice exception
The crashing way
class Parser:
import FileLoader
#
def __init__(self, fileLoader):
if isinstance(fileLoader, FileLoader.FileLoader)
self._fileLoader = fileLoader
else:
# raise a nice exception
I thought doing the import inside the class's scope (where it's the only scope FileLoader is used) would be enough, since it would know how to relate to the FileLoader module and its content. I'm obviously wrong since it's the first way which worked.
What am I missing about scopes in Python? Or is it about something different?
2 things : this won't work. And there is no benefit to doing it this way.
First, why not?
class Parser:
#this assigns to the Parser namespace, to refer to it
#within a method you need to use `self.FileLoader` or
#Parser.FileLoader
import FileLoader
#`FileLoader` works fine here, under the Parser indentation
#(in its namespace, but outside of method)
copy_of_FileLoader = FileLoader
#
def __init__(self, fileLoader):
# you need to refer to modules under in Parser namespace
# with that `self`, just like you would with any other
# class or instance variable 👇
if isinstance(fileLoader, self.FileLoader.FileLoader)
self._fileLoader = fileLoader
else:
# raise a nice exception
#works here again, since we are outside of method,
#in `Parser` scope/indent.
copy2_of_FileLoader = FileLoader
Second it's not Pythonic and it doesn't help
Customary for the Python community would be to put import FileLoader at the top of the program. Since it seems to be one of your own modules, it would go after std library imports and after third party module imports. You would not put it under a class declaration.
Unless... you had a good (probably bad actually reason to).
My own code, and this doesn't reflect all that well on me, sometimes has stuff like.
class MainManager(batchhelper.BatchManager):
....
def _load(self, *args, **kwargs):
👉 from pssystem.models import NotificationConfig
So, after stating this wasn't a good thing, why am I doing this?
Well, there are some specific circumstances to my code going here. This is a batch, command-line, script, usable within a Django context and it uses some Django ORM models. In order for those to be used, Django needs to be imported first and then setup. But that often happens too early in the context of these types of batch programs and I get circular import errors, with Django complaining that it hasn't initialized yet.
The solution? Defer execution until the method is called, when all the other modules have been imported and Django has been setup elsewhere.
NotificationConfig is now available, but only within that method as it is a local variable in it. It works, but... it's really not great practice.
Remember: anything in the global scope gets executed at module load time, anything under classes at module load time, anything withing method/function bodies when the method/function is called.
#happens at module load time, you could have circular import errors
import X1
class DoImportsLater:
.
#happens at module load time, you could have circular import errors
import X2
def _load(self, *args, **kwargs):
#only happens when this method is called, if ever
#so you shouldn't be seeing circular imports
import X3
import X1 is std practice, Pythonic.
import X2, what are doing, is not and doesn't help
import X3, what I did, is a hack and is covering up circular import references. But it "fixes" the issue.
I'm writing a simple Python library in which I have several "private" functions starting with underscore:
def _a():
pass
def _b():
pass
def public_interface_call():
_a()
_b()
This way my library users can simply do from MyLib.Module import * and their namespace won't be cluttered with implementation detail.
However I'm also writing unit tests in which I'd love to test these functions separately and simple importing truly all symbols from my module would be very handy. Currently I'm doing from Mylib.Module import _a _b public_interface_call but I wonder if there's any better/quicker/cleaner way to achieve what I want?
I'm not sure if it was a blackout or something when I wrote that question but today I realized (inspired by Underyx's comment) that I can simply do this:
import MyLib.Module
MyLib.Module._a()
MyLib.Module._b()
Or even to shorten things a little (because I'm a lazy bastard):
import MyLib.Module as mm
mm._a()
mm._b()
According to docs,
There is even a variant to import all names that a module defines:
from fibo import *
...
This imports all names except those beginning with an underscore (_).
Not sure why this is the case however.
The best and most common solution for your problem already has been given:
import MyLib.Module as mm
If one still wants to make use of the variant from MyLib.Module import *, there is the possibility to override its default behavior: Add a list __all__ to the module's source file MyLib/Module.py and declare which objects should be exported.
Note that you need to do this for every object you want to be visible. This does not affect the behavior of the other import mechanism above.
Example:
# MyLib/Module.py
__all__ = ["_a", "_b"]
def _a(): pass
def _b(): pass
def c(): pass
# ...
# Import
from MyLib.Module import *
# => this imports _a() and _b(), but not c()
To specify a package index __all__ can make sense to control which submodules should be loaded when importing a package. See here for more information, or this SO thread on a related question.
I am developing a Python package for dealing with some scientific data. There are multiple frequently-used classes and functions from other modules and packages, including numpy, that I need in virtually every function defined in any module of the package.
What would be the Pythonic way to deal with them? I have considered multiple variants, but every has its own drawbacks.
Import the classes at module-level with from foreignmodule import Class1, Class2, function1, function2
Then the imported functions and classes are easily accessible from every function. On the other hand, they pollute the module namespace making dir(package.module) and help(package.module) cluttered with imported functions
Import the classes at function-level with from foreignmodule import Class1, Class2, function1, function2
The functions and classes are easily accessible and do not pollute the module, but imports from up to a dozen modules in every function look as a lot of duplicate code.
Import the modules at module-level with import foreignmodule
Not too much pollution is compensated by the need to prepend the module name to every function or class call.
Use some artificial workaround like using a function body for all these manipulations and returning only the objects to be exported... like this
def _export():
from foreignmodule import Class1, Class2, function1, function2
def myfunc(x):
return function1(x, function2(x))
return myfunc
myfunc = _export()
del _export
This manages to solve both problems, module namespace pollution and ease of use for functions... but it seems to be not Pythonic at all.
So what solution is the most Pythonic? Is there another good solution I overlooked?
Go ahead and do your usual from W import X, Y, Z and then use the __all__ special symbol to define what actual symbols you intend people to import from your module:
__all__ = ('MyClass1', 'MyClass2', 'myvar1', …)
This defines the symbols that will be imported into a user's module if they import * from your module.
In general, Python programmers should not be using dir() to figure out how to use your module, and if they are doing so it might indicate a problem somewhere else. They should be reading your documentation or typing help(yourmodule) to figure out how to use your library. Or they could browse the source code yourself, in which case (a) the difference between things you import and things you define is quite clear, and (b) they will see the __all__ declaration and know which toys they should be playing with.
If you try to support dir() in a situation like this for a task for which it was not designed, you will have to place annoying limitations on your own code, as I hope is clear from the other answers here. My advice: don't do it! Take a look at the Standard Library for guidance: it does from … import … whenever code clarity and conciseness require it, and provides (1) informative docstrings, (2) full documentation, and (3) readable code, so that no one ever has to run dir() on a module and try to tell the imports apart from the stuff actually defined in the module.
One technique I've seen used, including in the standard library, is to use import module as _module or from module import var as _var, i.e. assigning imported modules/variables to names starting with an underscore.
The effect is that other code, following the usual Python convention, treats those members as private. This applies even for code that doesn't look at __all__, such as IPython's autocomplete function.
An example from Python 3.3's random module:
from warnings import warn as _warn
from types import MethodType as _MethodType, BuiltinMethodType as _BuiltinMethodType
from math import log as _log, exp as _exp, pi as _pi, e as _e, ceil as _ceil
from math import sqrt as _sqrt, acos as _acos, cos as _cos, sin as _sin
from os import urandom as _urandom
from collections.abc import Set as _Set, Sequence as _Sequence
from hashlib import sha512 as _sha512
Another technique is to perform imports in function scope, so that they become local variables:
"""Some module"""
# imports conventionally go here
def some_function(arg):
"Do something with arg."
import re # Regular expressions solve everything
...
The main rationale for doing this is that it is effectively lazy, delaying the importing of a module's dependencies until they are actually used. Suppose one function in the module depends on a particular huge library. Importing the library at the top of the file would mean that importing the module would load the entire library. This way, importing the module can be quick, and only client code that actually calls that function incurs the cost of loading the library. Further, if the dependency library is not available, client code that doesn't need the dependent feature can still import the module and call the other functions. The disadvantage is that using function-level imports obscures what your code's dependencies are.
Example from Python 3.3's os.py:
def get_exec_path(env=None):
"""[...]"""
# Use a local import instead of a global import to limit the number of
# modules loaded at startup: the os module is always loaded at startup by
# Python. It may also avoid a bootstrap issue.
import warnings
Import the module as a whole: import foreignmodule. What you claim as a drawback is actually a benefit. Namely, prepending the module name makes your code easier to maintain and makes it more self-documenting.
Six months from now when you look at a line of code like foo = Bar(baz) you may ask yourself which module Bar came from, but with foo = cleverlib.Bar it is much less of a mystery.
Of course, the fewer imports you have, the less of a problem this is. For small programs with few dependencies it really doesn't matter all that much.
When you find yourself asking questions like this, ask yourself what makes the code easier to understand, rather than what makes the code easier to write. You write it once but you read it a lot.
For this situation I would go with an all_imports.py file which had all the
from foreignmodule import .....
from another module import .....
and then in your working modules
import all_imports as fgn # or whatever you want to prepend
...
something = fgn.Class1()
Another thing to be aware of
__all__ = ['func1', 'func2', 'this', 'that']
Now, any functions/classes/variables/etc that are in your module, but not in your modules's __all__ will not show up in help(), and won't be imported by from mymodule import * See Making python imports more structured? for more info.
I would compromise and just pick a short alias for the foreign module:
import foreignmodule as fm
It saves you completely from the pollution (probably the bigger issue) and at least reduces the prepending burden.
I know this is an old question. It may not be 'Pythonic', but the cleanest way I've discovered for exporting only certain module definitions is, really as you've found, to globally wrap the module in a function. But instead of returning them, to export names, you can simply globalize them (global thus in essence becomes a kind of 'export' keyword):
def module():
global MyPublicClass,ExportedModule
import somemodule as ExportedModule
import anothermodule as PrivateModule
class MyPublicClass:
def __init__(self):
pass
class MyPrivateClass:
def __init__(self):
pass
module()
del module
I know it's not much different than your original conclusion, but frankly to me this seems to be the cleanest option. The other advantage is, you can group any number of modules written this way into a single file, and their private terms won't overlap:
def module():
global A
i,j,k = 1,2,3
class A:
pass
module()
del module
def module():
global B
i,j,k = 7,8,9 # doesn't overwrite previous declarations
class B:
pass
module()
del module
Though, keep in mind their public definitions will, of course, overlap.
I'm writing some unittests for code written by someone else here at the office. Python is not my strongest language. While I've been successful with basic unit tests, mocking in python is throwing me for a loop.
What I need to do is override a call to ConfigObj and inject my own mock config/fixture into any ConfigObj call.
settings.py
from configobj import ConfigObj
config = ConfigObj('/etc/myapp/config')
utils.py
from settings import config
"""lots of stuff methods using various config values."""
What I would like to do is, in my unittests for utils.py, inject myself either for ANY call to ConfigObj or settings.py itself.
Many of the mocking libraries expect me to Mock my own classes but in the case of this app, it doesn't have any explicit classes.
Can it be done or are the python namespace restrictions too strict that I can't intervene in what a module that I'm importing imports itself?
Side note: running 2.7 so I can't do any of the tricks I've read about from 2.5.
If the tests are in a separate file from from settings.py and utils.py you can create a file mock.py
import configobj
class MockConfigObj(object):
#mock whatever you wan
configobj.ConfigObj = MockConfigObj
and then import mock before importing (from) any module that itself imports settings. This will ensure that settings.config is created with MockConfigObj. If you want a uniform global mocking, import it before any file that imports configobj.
This works because python will store configobj in sys.modules and check that before actually reading from a file on subsequent imports. in mock.py, the identifier ConfigObj is just a reference to the entry in sys.modules so that any changes that you make will be globally visible.
This strikes me as a little hacky though but it's the best that I can think of.
Python namespaces are not strict at all within the same scope. Just override the variable name containing your object (or the class itself and provided it) within the same scope you'd be expecting the original and that is good enough.
Now, whether or not what you're replacing it with behaves the same is up to you...
Couldn't you just overwrite the original function with another one?
There are no constants in Python, you can change everything, you could even do True = False.
I faced a similar situation before. Here is how I would go about addressing your problem.
Consider a test case for a function from utils.py.
import utils, unittest
class FooFunctionTests(unittest.TestCase):
def setUp(self):
utils._old_config = utils.config
utils.config = MockClass()
def tearDown(self):
utils.config = utils._old_config
del utils._old_config
def test_foo_function_returns_correct_value(self):
self.assertEqual("success!", utils.foo())
The following page is a good one on mocking and import
http://www.relaxdiego.com/2014/04/mocking-objects-in-python.html
Say you have a file named my_package1.py with the following code:
class A(object):
def init(self):
and you then import that in my_package2.py with the code
from my_package1 import A
class A(object):
def init(self):
The first line of my_package2.py creates a variable under the my_package2 namespace called A. Now you have two variables my_package1.A and my_package2.A that both point to the same class in memory. If you want the code in my_package2.py to use a mocked up class A, then you will need to mock my_package2.A not my_package1.A