I have an application that runs on Postgres & Mysql. Each program checks to determine the database and then imports either postgres_db as db_util or mysql_dt as db_util. All works well when code in main references db_util, but if a class is imported, the reference to db_util is not defined.
I created the following to classes and main to test the problem and found another interesting side effect. Classes B & C reference ClassA under different import cases. B & C are identical except B is in main and C is imported.
ClassX.py
class ClassA(object):
def print_a(self):
print "this is class a"
class ClassC(object):
def ref_a(self):
print 'from C ref a ==>',
xa=ClassA()
xa.print_a()
def ref_ca(self):
print 'from C ref ca ==>',
xa=ca()
xa.print_a()
test_scope.py
from classes.ClassX import ClassA
from classes.ClassX import ClassA as ca
from classes.ClassX import ClassC as cb
class ClassB(object):
def ref_a(self):
print 'from B ref a ==>',
xa=ClassA()
xa.print_a()
def ref_ca(self):
print 'from B ref ca ==>',
xa=ca()
xa.print_a()
print 'globals:',dir()
print 'modules','ca:',ca,'cb:',cb,'CA:',ClassA
print ''
print 'from main'
xb=ClassB()
xb.ref_a()
xb.ref_ca()
print ''
print 'from imports'
xbs=cb()
xbs.ref_a()
xbs.ref_ca()
And the results:
globals: ['ClassA', 'ClassB', '__builtins__', '__doc__', '__file__', '__name__', '__package__', 'ca', 'cb']
modules ca: <class 'classes.ClassX.ClassA'> cb: <class 'classes.ClassX.ClassC'> CA: <class 'classes.ClassX.ClassA'>
from main
from B ref a ==> this is class a
from B ref ca ==> this is class a
from imports
from C ref a ==> this is class a
from C ref ca ==>
Traceback (most recent call last):
File "test_scope.py", line 32, in <module>
xbs.ref_ca()
File "R:\python\test_scripts\scope\classes\ClassX.py", line 13, in ref_ca
xa=ca()
NameError: global name 'ca' is not defined
Press any key to continue . . .
From my test, I see that the object ca (imported as) is not available to the ClassC, however, the module ClassA is available (imported without as).
Why the difference between import and import as behavior? I am unclear why mains globals are not available to classes main imports.
What is a good approach to dynamically determine appropriate db_util module to import and have it accessible to other imported classes?
Update:
After reading yet another post on Namespaces: "Visibility of global variables from imported modules", I understand that in my example above the reason classA is available to ClassC is that A & C are in the same imported file, thus the same namespace.
So the remaining question is a design question:
if I have code like this:
if db == 'MySQL':
from mysql_db import db_util
elif db == 'Postgres'
from postgres_db import db_util
What is a good approach to make db_util available to all imported modules?
UPDATE:
from the reponse by Blckknght, I added the code
cb.ca =ca
to the scope_test script. This requires the class call to xa=ca() be changed to xa=self.ca(). I also think that adding objects to a class from outside the class, though Python allows it, is not a good design methodology and will make debugging a nightmare.
However, since I think modules and classes should be standalone or specifically declare their dependencies, I am going to implement the class like this, using the code sample from above.
break out ClassA and ClassC to separate modules and at the top of ClassC, before the class definition, do the imports
from ClassA import ClassA
from ClassA import ClassA as ca
class ClassB(object):
and in my real situation, where I need to import the db_util module into several modules
ci.py #new module to select class for appropriate db
if db == 'MySQL':
from mysql_db import db_util
elif db == 'Postgres'
from postgres_db import db_util
in each module needing the db_util class
import ci
db_util=ci.db_util #add db_util to module globals
class Module(object):
One problem with this is it requires each module using the db_util to import it, but it does make the dependencies known.
I will close this question and want to thank Blckknght and Armin Rigo for their responses which help clarify this issue for me. I would appreciate any design related feedback.
In Python, each module has it's own global namespace. When you do an import, you're only adding the imported modules to the current module's namespace, not to the namespace of any other module. If you want to put it in another namespace, you need to tell Python this explicitly.
main.py:
if db == "mysql": # or whatever your real logic is
import mysql_db as db_util
elif db == "postgres":
import postgres_db as db_util
import some_helper_module
some_helper_module.db_util = db_util # explicitly add to another namespace
#...
Other modules:
import some_helper_module
db = some_helper_module.db_util.connect() # or whatever the real API is
#...
Note that you can't usually use your main module (which is executed as a script) as a shared namespace. That's because Python uses the module's __name__ attribute to determine how to cache the module (so that you always get the same object from multiple imports), but a script is always given a __name__ of "__main__" rather than its real name. If another module imports main, they'll get a separate (duplicate) copy!
You're approaching the problem with the wrong point of view. Every module is a namespace that starts empty and is filled with a name for (typically) each statement it runs:
import foo => defines foo
from foo import bar => defines bar
from foo import bar as baz => defines baz
class kls:
pass => defines kls
def fun():
pass => defines fun
var = 6 * 7 => defines var
Looking at ClassX.py we see that the name ca is not defined in this module, but ClassA and ClassC are. So the execution of the line xa=ca() from ClassX.py fails.
In general, the idea is that every single module imports what it needs. You can also patch a name "into" a module from outside, but this is generally considered very bad style (reserved for very special cases).
Related
I'm trying to figure out the python import system aliases the dependencies of a module imported using the 'from' keyword.
In particular, my use-case is writing unittest.mock.patch statements for mocking attributes of an object that is imported using "from", and that object uses something imported from another module. However, more broadly I just want to understand how the import system works.
My question may make more sense with an example:
Assume we have some class A in some module a, that uses a class from module b
a.py:
from b import B
class A:
def __init__(self):
B.use_some_static_method()
# OR
self.b_instance = B()
Then, for example, in some test code we want to mock B when testing our A object. How would I write a patch statement?
test_a.py:
from a import A
def test_a(mocker):
# Mock with pytest-mock's mocker
mock_b = mocker.patch('<some path>')
a = A()
a.do_something_with_B()
What path would I put in place of ? I realize that I could simply use import a instead, and then mock a.B, but I'm more interested in understanding the import system and how this works and why it works that way.
I'm using python 3.5. Fairly new to python but not new to programming. I have three source files as follows (a much simplified version of what I'm actually doing):
c.py
class C:
def __init__(self, x):
self.x = x
def method(self):
print(self.x)
init.py
import shelve
from c import C
db = shelve.open("DB")
db['key1'] = C("test")
db.close()
test.py
import shelve
db = shelve.open("DB")
obj = db['key1']
obj.method() # this works
C.method(obj) # this doesn't -- 'C' not defined
db.close()
So I run init.py to set up my shelved database. Then I run test.py. It is happy with executing obj.method(), so it seems to know about class C even though I haven't explicitly imported it (Lutz says something about it being stored in the database). But if I try to do C.method(obj) (not that I'd necesarily need to call it this way, but using C as a class (for example to create new objects) has its usefulness) it says 'C' is not defined. But if I add 'from c import C' to test.py then it works. So in one way it seems to know about C's definition, but then again it doesn't. I am curious to know why this is.
When shelve serializes an object (I believe by pickling it), it stores the import path to the class for unpickling later. When it retrieves the object, pickle imports the module (c) and returns an instance of C back to you (that is equivalent to the one that was originally serialized).
So far, this isn't anything new to you based on your observations. However, when it imports c, it doesn't import it into your current namespace. In fact, it imports it into a very localized namespace. So while the c module has been imported (you can find it in sys.modules), you won't find c or C in your current namespace because you haven't imported it there.
Put another way, simply importing something doesn't make it accessible to every module in your program. It only makes it accessible to the module (actually scope) where it was imported. I can import os, but just because os imports sys doesn't mean that I immediately have access to sys. I need to import it too before I can start using it.
>>> import os
>>> sys
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'sys' is not defined
>>> os.sys
<module 'sys' (built-in)>
>>> import sys
>>> sys
<module 'sys' (built-in)>
What you have doesn't work for the reasons stated in mgilson's answer.
A work-around for the problem would be to manually import the class from the module and assign the class to the name C — something along these lines (tested with Python 3.5.1):
import shelve
db = shelve.open("DB")
obj = db['key1']
obj.method() # this works
## Begin added code ##
classname = obj.__class__.__name__
module_name = obj.__module__
module = __import__(module_name, globals(), locals(), [classname], 0)
globals()[classname] = getattr(module, classname)
## Added code end ##
C.method(obj) # this also works now
db.close()
I have an interesting application and I would like to access a global variable from a different module.
I currently have module x
from pkg.mod import *
in order to get the namespace foo.
where my pkg.mod looks like the following
import x
foo = 0
# do some stuff
foo = myClass()
That all works just great, however, I would like NOT to do that, mostly just because its bad.
If I try the following
from pkg import mod
I get a circular dependency issue. ImportError: cannot import name mod which makes sense to me.
What I want to do is
from pkg.mod import foo
but that also gives me an import name error: ImportError: cannot import name foo
Which I am not exactly sure why, except I dont think you can import variables specifically.
Is there anyway to import foo following correct practices?
---edit---
I have tried to remove the shared variable and put it in a third module in such a way
pkg.y
foo = 0
pkg.mod
from pkg.y import foo
foo = myClass()
x.dostuff()
x
from pkg.y import foo
def dostuff():
print(foo)
and now x thinks foo is still 0. (I can guarantee that mod is calling x, and thus happening first.)
When I need a true global variable I find it cleaner to create a module called trueglobals.py; it can remain empty but for proper documentation...
# -*- coding: utf-8 -*-
# Object references that are used extensively across many modules are
# stored here.
# someObject1 - instantiated in main.py
# someObject2 - instantiated in someothermodule.py
In the module instantiating the object...
import trueglobals
trueglobals.someObject1 = myClass(somevar)
In any module requiring the object...
import trueglobals
localVar = trueglobals.someObject1
Say I have two Python modules:
module1.py:
import module2
def myFunct(): print "called from module1"
module2.py:
def myFunct(): print "called from module2"
def someFunct(): print "also called from module2"
If I import module1, is it better etiquette to re-import module2, or just refer to it as module1.module2?
For example (someotherfile.py):
import module1
module1.myFunct() # prints "called from module1"
module1.module2.myFunct() # prints "called from module2"
I can also do this: module2 = module1.module2. Now, I can directly call module2.myFunct().
However, I can change module1.py to:
from module2 import *
def myFunct(): print "called from module1"
Now, in someotherfile.py, I can do this:
import module1
module1.myFunct() # prints "called from module1"; overrides module2
module1.someFunct() # prints "also called from module2"
Also, by importing *, help('module1') shows all of the functions from module2.
On the other hand, (assuming module1.py uses import module2), I can do:
someotherfile.py:
import module1, module2
module1.myFunct() # prints "called from module1"
module2.myFunct() # prints "called from module2"
Again, which is better etiquette and practice? To import module2 again, or to just refer to module1's importation?
Quoting the PEP 8 style guide:
When importing a class from a class-containing module, it's usually okay to spell this:
from myclass import MyClass
from foo.bar.yourclass import YourClass
If this spelling causes local name clashes, then spell them
import myclass
import foo.bar.yourclass
Emphasis mine.
Don't use module1.module2; you are relying on the internal implementation details of module1, which may later change what imports it is using. You can import module2 directly, so do so unless otherwise documented by the module author.
You can use the __all__ convention to limit what is imported from a module with from modulename import *; the help() command honours that list as well. Listing the names you explicitly export in __all__ helps clean up the help() text presentation:
The public names defined by a module are determined by checking the module’s namespace for a variable named __all__; if defined, it must be a sequence of strings which are names defined or imported by that module. The names given in __all__ are all considered public and are required to exist. If __all__ is not defined, the set of public names includes all names found in the module’s namespace which do not begin with an underscore character ('_'). __all__ should contain the entire public API. It is intended to avoid accidentally exporting items that are not part of the API (such as library modules which were imported and used within the module).
Just import module2. Re-importing is relatively costless, since Python caches module objects in sys.modules.
Moreover, chaining dots as in module1.module2.myFunct is a violation of the Law of Demeter. Perhaps some day you will want to replace module1 with some other module module1a which does not import module2. By using import module2, you will avoid having to rewrite all occurrences of module1.module2.myFunct.
from module2 import * is generally a bad practice since it makes it hard to trace where variables come from. And mixing module namespaces can create variable-name conflicts. For example, from numpy import * is a definite no-no, since doing so would override Python's builtin sum, min, max, any, all, abs and round.
I am using the Borg pattern with mutual inclusion of modules. See the example code (not the real code but it shows the problem) below. In this case, I have two different Borgs because the class names (and I guess the class) are seen as different by the interpreter.
Is there a way to use the Borg in that case without reworking the module architecture?
Module borg.py
import borg2
class Borg:
_we_are_one = {}
def __init__(self):
self.__dict__ = Borg._we_are_one
try:
self.name
except AttributeError:
self.name = "?"
print self.__class__, id(self.__dict__)
def fct_ab():
a = Borg()
a.name = "Bjorn"
b = Borg()
print b.name
if __name__ == "__main__":
fct_ab()
borg2.fct_c()
Module borg2.py
import borg
def fct_c():
c = borg.Borg()
print c.name
The result is
__main__.Borg 40106720
__main__.Borg 40106720
Bjorn
borg.Borg 40106288
?
In order to clarify my problem:
Why does Python consider __main__.Borg and borg.Borg as two different classes?
After a long day of struggling with Singletons and Borg, my conclusion is the following:
It seems that a Python module imported multiple times using different 'import paths' is actually imported multiple times. If that module contains a singleton, you get multiple instances.
Example:
myproject/
module_A
some_folder/
module_B
module_C
If module_A imports module_C using from myproject.some_folder import module_C and module_B imports the same module_C using import module_C, the module is actually imported twice (at least according to my observations). Usually, this doesn't matter, but for singletons or borg, you actually get 2 instances of what should be unique. (That's 2 sets of borgs sharing 2 different internal states).
Solution: Give yourself an import statement convention and stick to it: I import all modules starting from a common root folder, even if the module file is located parallel to the one I am working on, so in the example above, both module_A and module_B import module_C using from myproject.some_folder import module_C.
The problem only occurs in your main-function. Move that code
to its own file and everything is as you'd expect. This code
import borg
import borg2
if __name__ == "__main__":
borg.fct_ab()
borg2.fct_c()
delivers this output:
borg.Borg 10438672
borg.Borg 10438672
Bjorn
borg.Borg 10438672
Bjorn
It's not the class names that is the problem. I'm not entirely sure why Python see the Borg class and the borg.Borg class as different, perhaps it's because you run this from __main__, I think python does not realize that __main__ and borg is the same module.
The solution is easy. Change fct_ab to:
def fct_ab():
import borg
a = borg.Borg()
a.name = "Bjorn"
b = borg.Borg()
print b.name
This solves the problem.
I've fixed the issue in my real application by fixing an error in the import.
In fact, I have two different modules using the same 3rd module.
The 1st one was importing mypackage.mymodule while the 2nd one was importing mymodule. mypackage is installed as a python egg and the code I was working on is on my development folder.
So both codes were importing different modules and I guess that it is normal to have two different classes in this case.
Regarding the example code I've used, the problem comes from the current modules to receive the main as name. I've tried to rename by doing __name__ = 'borg'. It works but it breaks the if __name__ == "__main__" condistion. As a conclusion, I would say that mutual inclusion must be avoid and is in most cases not necessary.
Thanks all for your help.
The solution --- as has already been mentioned --- is to avoid a recursive import of the main module, but borg.py is not being "imported twice". The problem is that importing it at all while it is already executing causes you define the Borg class twice, in two different namespaces.
To demonstrate, I added a few lines to the top of both borg.py and borg2.py, and inserted my print_module function before and after most points of interest:
#!/usr/bin/env python2
from __future__ import print_function
def print_module(*args, **kwargs):
print(__name__ + ': ', end='')
print(*args, **kwargs)
return
print_module('Importing module borg2...')
import borg2
print_module('Module borg2 imported.')
print_module('Defining class Borg...')
class Borg:
...
# etc.
The output is:
__main__: Importing module borg2...
borg2: Importing module borg...
borg: Importing module borg2...
borg: Module borg2 imported.
borg: Defining class Borg...
borg: id(_we_are_one) = 17350480
borg: Class Borg defined.
borg: id(Borg) = 139879572980464
borg: End of borg.py.
borg2: Module borg imported.
borg2: End of borg2.py.
__main__: Module borg2 imported.
__main__: Defining class Borg...
__main__: id(_we_are_one) = 17351632
__main__: Class Borg defined.
__main__: id(Borg) = 139879572981136
__main__: Borg 17351632
__main__: Borg 17351632
__main__: Bjorn
borg: Borg 17350480
borg2: ?
__main__: End of borg.py.
The first thing borg.py does (not counting the bits I added) is import borg2 into the __main__ namespace. This happens before the Borg class is defined anywhere.
The first thing borg2 does is import borg, which again attempts to import borg2... and Python refuses to do so. (Note that nothing happens between lines 3 and 4.) borg finally defines the Borg class and the fct_ab function in the borg namespace, and exits.
borg2 then defines fct_c and exits ("borg2: End of borg2.py."). All the import statements are done.
Now, borg.py finally gets to execute "for real". Yes, it already ran once when it was imported, but this is still the "first" time through the borg.py file. The Borg class gets defined again, this time in the __main__ namespace, and both the class and its dictionary have new IDs.
borg.py was not "imported twice". It was executed once from the command line, and it was executed once when it was imported. Since these happened in two different namespaces, the "second" definition of Borg did not replace the first, and the two functions modified two different classes, which just happened to be created from the same code.