In a Django project, I have a directory structure that looks something like this:
project/
├╴module_name/
│ ├╴dbrouters.py
│ ...
...
In dbrouters.py, I define a class that starts out like this:
class CustomDBRouter(object):
current_connection = 'default'
...
The idea is to have a database router that sets the connection to use at the start of each request and then uses that connection for all subsequent database queries similarly to what is described in the Django docs for automatic database routing.
Everything works great, except that when I want to import CustomDBRouter in a script, I have to use the absolute path, or else something weird happens.
Let's say in one part of the application, CustomDBRouter.current_connection is changed:
import project.module_name.dbrouters.CustomDBRouter
...
CustomDBRouter.current_connection = 'alternate'
In another part of the application (assume that it is executed after the above code), I use a relative import instead:
import .dbrouters.CustomDBRouter
...
print CustomDBRouter.current_connection # Outputs 'default', not 'alternate'!
I'm confused as to why this is happening. Is Python creating a new class object for CustomDBRouter because I'm using a different import path?
Bonus points: Is there a better way to implement a 'global' class property?
It depends on how this script is being executed. When you're using relative imports, you have to make sure the name of the script the import is in has a __name__ attribute other than __main__. If it does, import .dbrouters.CustomDBRouter becomes import __main__.dbrouters.CustomDBRouter.
I found this here.
It turns out, the problem was being caused by a few lines in another file:
PROJECT_ROOT = '/path/to/project'
sys.path.insert(0, '%s' % PROJECT_ROOT)
sys.path.insert(1, '%s/module_name' % PROJECT_ROOT)
The files that were referencing .dbrouters were imported using the "shortcut" path (e.g., import views instead of import module_name.views).
Related
Let's say I have a folder structure as below.
project/
-> app/
--> __init__.py (has db = SQLAlchemy(app))
--> model.py
I need to import db in model.py. I can either import it using
from app import db
or
from . import db
Is there a difference between the two? Does one method have any advantages over the other method?
Absolute imports are preferred because they are quite clear and straightforward. It is easy to tell exactly where the imported resource is, just by looking at the statement. In fact, pep8 explicitly recommends absolute imports.
Sometimes, however, absolute imports can get quite verbose, depending on the complexity of the directory structure. Imagine having a statement like this:
from package1.subpackage2.subpackage3.subpackage4.module5 import function6
This looks ridiculous! Right?
So, Relative imports comes into picture. A relative import specifies the resource to be imported relative to the current location—that is, the location where the import statement is.
Above complex import statement becomes:
from ..subpackage4.module5 import function6
Hope this helps!
I am trying to set up a library in python. I have created a setup.py file and in a folder under that I have a library folder, and then I tried to create an sample and test folder (for sample code and tests that I would include)
Folders:
- setup.py
- cvImageUtils # this is the library
- __init__.py # this includs ColorUtils
- ColorUtils.py # this includes a class called ColorUtils
- examples
- color.py # this is shown below
init.py in ColorUtils folder
from . import ColorUtils
ColorUtils.py
class ColorUtils:
def __...
Color.py
from cvImageUtils import ColorUtils
m1 = cv2.imread(os.path.join(image_folder, "bb.jpeg"), 1) # 1 = load color
cv2.imshow('grayscale', ColorUtils.convert_rbg_to_grayscale(m1))
At first, it said, unable to find the module, so I added to the top of the file the following based on another SO solution:
import sys
sys.path.append('../')
Now that seems broken to me already, but it did get me past the no module found, but now it says ColorUtils has no method convert_rbg_to_grayscale. So Then I had to change it to ColorUtils.ColorUtils.convert_rbg_to_grayscale
cv2.imshow('grayscale', ColorUtils.ColorUtils.convert_rbg_to_grayscale(m1))
How can I setup the folder so that it allows me to include the library without sys, and call it without declaring ColorUtils twice.
change your __init__.py:
from cvImageUtils.ColorUtils import ColorUtils
I don't think you'll need to import sys anymore, and you don't have import ColorUtils twice. but just like you have to instantiate an object, you should create a ColorUtils object.
my personal preference would be not creating a Class for Utils.
you might have done this already, but if you want to use a method straight from a class like you did in python, you might want to declare it static.
class ColorUtils:
#staticmethod
def util_method():
pass
then you can just do:
ColorUtils.util_method()
Update:
you can read more about relative/absolute import from here as well.
to fix your actual problem though, you can do:
color.py
remove your import sys and sys call from color.py
change: import cvImageUtils.ColorUtils as ct
to: from cvImageUtils.ColorUtils import *
remove all your ct reference instead just use the actual functions.
cvImageUtils/__init__.py
change: from . import ColorUtils
to __all__=['ColorUtils']
I was able to run color.py to get all the images printed out on screen.
a image.png was also generated locally as well.
Every directory that you want to expose in module search(we usually hide test.py) in python need a init.py file. That should be rule of thumb, by using sys module you can add the module to your "module search path".
After having init.py in your directories, you need to import packages/modules/funcitons you want to use:-
import cvImageUtils.ColorUtils.convert_rbg_to_grayscale
You can execute following code in python to see, what have included in your sys path(used by python to search for modules/packages)
import sys
sys.path
Look into below links for more detailed explations
https://www.programiz.com/python-programming/package
https://www.programiz.com/python-programming/modules#search
Short:
How can I import a module if a different module with the same name has already been imported? I don't need the old module anymore, but I can't import the new one under a different name.
importlib.reload() doesn't have the desired effect and apparently you really shouldn't mess with sys.modules
Context:
I'm automating a testing workflow by writing a script (A). The test script is provided and can not be altered.
The target of the Testing is a class in a script (B) of which there are different versions in subfolders. Unfortunately Script B and the class always have the same name, nothing I can do about that.
main_folder
├──script_a.py
├──test_script.py
│
├──version_1
│ ├──script_b.py
│
├──version_2
│ ├──script_b.py
│
├──version_3
│ ├──script_b.py
The test script imports the object and runs tests on it.
# test_script
from script_b import my_class
# creat instance of my_class
# and test it
script_a iterates over the version folders and runs the test script on script_b.
In each iteration one subfolder is added to the import path so the test script will find the corresponding script_b. The path is removed after the iteration.
If sys.modules already contains a version of test_script,
# script_a
import sys
import os
import importlib
folders = ['version_1', 'version_2', 'version_3']
folders = [os.getcwd() + '/' + folder for folder in folders]
for folder in folders:
sys.path.append(folder)
if 'test_script' in sys.modules:
importlib.reload(sys.modules['test_script'])
else:
importlib.import_module('test_script')
sys.path.remove(folder)
Issue:
It seems reload has no effect on script_b, which is imported by test_script. So although I change the import path to different subfolders test_script always runs on version 1.
How can I make test_script use the different versions of script_b without altering test_script itself?
Follow up:
Although the original question is answered, I was wondering, how is this solution from a design perspective? Is there a better / more elegant way to automate this testing process?
From what I found, it isn't considered good practice to reload modules.
Figured it out while writing the question. Hope it will help someone some day
Even if you can't alter test_script (or script_b) there's a work around.
Because an import statement does nothing if the relevant module is already imported we can reload script_b from the path we want directly in script_a. Since it then is replaced with the new version in sys.modules the import statement in test_script will cause no problem.
Updated code:
# script_a
import sys
import os
import importlib
folders = ['version_1', 'version_2', 'version_3']
folders = [os.getcwd() + '/' + folder for folder in folders]
for folder in folders:
sys.path.append(folder)
if 'test_script' in sys.modules:
importlib.reload(sys.modules['script_b']) # this line added
importlib.reload(sys.modules['test_script'])
else:
importlib.import_module('test_script')
sys.path.remove(folder)
OK guys I can't find a solution anywhere for my problem, and I hope the solution is a simple one. Previously I had a flat file system for my gae project with no folders. I've been refactoring some code and I tried to put some in a folder. I'm a bit new and I've never done something like this before, but nothing on the internet suggests that I shouldn't easily be able to move my files into a folder. I added the __init__.py file to the folder and I import the folder name from my main program. However when I attempt to access a particular function in one of the files, it chokes and says AttributeError: 'module' object has no attribute 'site1_ripper'
here is my file structure:
main.py
SiteCrawlers\
__init__.py
site1_ripper.py
here are important parts of the files:
main.py
import SiteCrawlers
class Updater(webapp.RequestHandler):
def get(self):
SiteCrawlers.site1_ripper.siteCrawler()
site1_ripper.py
def siteCrawler()
#stuff here
I think the problem is that you need to explicitly import site1_ripper unless it's specified in __init__.py. Make your main import be:
import SiteCrawlers.site1_ripper
In your main file try:
from SiteCrawlers.site1_ripper import siteCrawler
class Updater(webapp.RequestHandler):
def get(self):
siteCrawler()
Summary: when a certain python module is imported, I want to be able to intercept this action, and instead of loading the required class, I want to load another class of my choice.
Reason: I am working on some legacy code. I need to write some unit test code before I start some enhancement/refactoring. The code imports a certain module which will fail in a unit test setting, however. (Because of database server dependency)
Pseduo Code:
from LegacyDataLoader import load_me_data
...
def do_something():
data = load_me_data()
So, ideally, when python excutes the import line above in a unit test, an alternative class, says MockDataLoader, is loaded instead.
I am still using 2.4.3. I suppose there is an import hook I can manipulate
Edit
Thanks a lot for the answers so far. They are all very helpful.
One particular type of suggestion is about manipulation of PYTHONPATH. It does not work in my case. So I will elaborate my particular situation here.
The original codebase is organised in this way
./dir1/myapp/database/LegacyDataLoader.py
./dir1/myapp/database/Other.py
./dir1/myapp/database/__init__.py
./dir1/myapp/__init__.py
My goal is to enhance the Other class in the Other module. But since it is legacy code, I do not feel comfortable working on it without strapping a test suite around it first.
Now I introduce this unit test code
./unit_test/test.py
The content is simply:
from myapp.database.Other import Other
def test1():
o = Other()
o.do_something()
if __name__ == "__main__":
test1()
When the CI server runs the above test, the test fails. It is because class Other uses LegacyDataLoader, and LegacydataLoader cannot establish database connection to the db server from the CI box.
Now let's add a fake class as suggested:
./unit_test_fake/myapp/database/LegacyDataLoader.py
./unit_test_fake/myapp/database/__init__.py
./unit_test_fake/myapp/__init__.py
Modify the PYTHONPATH to
export PYTHONPATH=unit_test_fake:dir1:unit_test
Now the test fails for another reason
File "unit_test/test.py", line 1, in <module>
from myapp.database.Other import Other
ImportError: No module named Other
It has something to do with the way python resolves classes/attributes in a module
You can intercept import and from ... import statements by defining your own __import__ function and assigning it to __builtin__.__import__ (make sure to save the previous value, since your override will no doubt want to delegate to it; and you'll need to import __builtin__ to get the builtin-objects module).
For example (Py2.4 specific, since that's what you're asking about), save in aim.py the following:
import __builtin__
realimp = __builtin__.__import__
def my_import(name, globals={}, locals={}, fromlist=[]):
print 'importing', name, fromlist
return realimp(name, globals, locals, fromlist)
__builtin__.__import__ = my_import
from os import path
and now:
$ python2.4 aim.py
importing os ('path',)
So this lets you intercept any specific import request you want, and alter the imported module[s] as you wish before you return them -- see the specs here. This is the kind of "hook" you're looking for, right?
There are cleaner ways to do this, but I'll assume that you can't modify the file containing from LegacyDataLoader import load_me_data.
The simplest thing to do is probably to create a new directory called testing_shims, and create LegacyDataLoader.py file in it. In that file, define whatever fake load_me_data you like. When running the unit tests, put testing_shims into your PYTHONPATH environment variable as the first directory. Alternately, you can modify your test runner to insert testing_shims as the first value in sys.path.
This way, your file will be found when importing LegacyDataLoader, and your code will be loaded instead of the real code.
The import statement just grabs stuff from sys.modules if a matching name is found there, so the simplest thing is to make sure you insert your own module into sys.modules under the target name before anything else tries to import the real thing.
# in test code
import sys
import MockDataLoader
sys.modules['LegacyDataLoader'] = MockDataLoader
import module_under_test
There are a handful of variations on the theme, but that basic approach should work fine to do what you describe in the question. A slightly simpler approach would be this, using just a mock function to replace the one in question:
# in test code
import module_under_test
def mock_load_me_data():
# do mock stuff here
module_under_test.load_me_data = mock_load_me_data
That simply replaces the appropriate name right in the module itself, so when you invoke the code under test, presumably do_something() in your question, it calls your mock routine.
Well, if the import fails by raising an exception, you could put it in a try...except loop:
try:
from LegacyDataLoader import load_me_data
except: # put error that occurs here, so as not to mask actual problems
from MockDataLoader import load_me_data
Is that what you're looking for? If it fails, but doesn't raise an exception, you could have it run the unit test with a special command line tag, like --unittest, like this:
import sys
if "--unittest" in sys.argv:
from MockDataLoader import load_me_data
else:
from LegacyDataLoader import load_me_data