Develop Reference

Python Importing modules in a package

I currently have a module I created that has a number of functions.
It's getting quite large so I figured I should make it into a package and split the functions up to make it more manageable.
I'm just testing out how this all works before I do this for real so apologies if it seems a bit tenuous.
I've created a folder called pack_test and in it I have:
__init__.py
foo.py
bar.py
__init__.py contains:
__all__ = ['foo', 'bar']
from . import *
import subprocess
from os import environ
In the console I can write import pack_test as pt and this is fine, no errors.
pt. and two tabs shows me that I can see pt.bar, pt.environ, pt.foo and pt.subprocess in there.
All good so far.
If I want to reference subprocess or environ in foo.py or bar.py how do I do it in there?
If in bar.py I have a function which just does return subprocess.call('ls') it errors saying NameError: name 'subprocess' is not defined. There must be something I'm missing which enables me to reference subprocess from the level above? Presumably, once I can get the syntax from that I can also just call environ in a similar way?
The alternative as I could see it would be to have import subprocess in both foo.py and bar.py but then this seems a bit odd to me to have it appear across multiple files when I could have it the once at a higher level, particularly if I went on to have a large number of files rather than just 2 in this example.

TL;DR:
__init__.py :
import foo
import bar
__all__ = ["foo", "bar"]
foo.py:
import subprocess
from os import environ
# your code here
bar.py
import subprocess
from os import environ
# your code here
There must be something I'm missing which enables me to reference subprocess from the level above?
Nope, this is the expected behaviour.
import loads a module (if it isn't already), caches it in sys.modules (idem), and bind the imported names in the current namespace. Each Python module has (or "is") it's own namespace (there's no real "global" namespace). IOW, you have to import what you need in each module, ie if foo.py needs subprocess, it must explicitely import it.
This can seem a bit tedious at first but in the long run it really helps wrt/ maintainability - you just have to read the imports at the top of your module (pep 08: always put all imports at the beginning of the module) to know where a name comes from.
Also you should not use star imports (aka wild card imports aka from xxx import *) anywhere else than in your python shell (and even then...) - it's a maintainance time bomb. Not only because you don't know where each name comes from, but also because it's a sure way to rebind an already import name. Imagine that your foo module defines function "func". Somewhere you have "from foo import *; from bar import *", then later in the code a call to func. Now someone edits bar.py and adds a (distinct) "func" function, and suddenly you call fails, because you're not calling the expected "func". Now enjoy debugging this... And real-life examples are usually a bit more complex than this.
So if you fancy your mental sanity, don't be lazy, don't try to be smart either, just do the simple obvious thing: explicitely import the names you're interested in at the top of your modules.
(been here, done that etc)

You could create modules.py containing
import subprocess
import os
Then in foo.py or any of your files just have.
from modules import *
Your import statements in your files are then static and just update modules.py when you want to add an additional module accessible to them all.

How to avoid circular import within django's tests dir?

I'm trying to fix a circular import issue I'm having with Django tests.
My dir structure is:
app/tests:
test_user_level01.py
test_user_level02.py
Within each file I have classes to test get 200s and get 403s (among other classes). Users in level 2 should have all the get 200s that level 1 users have and users in level 1 should have all the get 403s that the level 2 users have. Thus I have a circular import.
Normally I would solve this with an absolute import but I can't figure out what that would be.
I've tried:
"""test_user_level01.py"""
from . import test_user_level02
"""test_user_level02.py"""
from . import test_user_level01
But this seems to give a circular import error (missing classes).
Thank you ahead of time for your help.
PS. The following doesn't work:
import app.tests.test_user_level01 as level01
OR:
import tests.test_user_level01 as level01
OR:
import .test_user_level01 as level01

Move common code, i.e. code which is used in both test_user_level01.py and test_user_level02.py, to a separate file, e.g. app/tests/common.py, and then import from there. Don't forget to make app/tests a package (i.e. create __init__.py file in the dir).
"""test_user_level01.py"""
from app.tests.common import some_common_class_name
"""test_user_level02.py"""
from app.tests.common import some_common_class_name

Python not able to reference module in parent

I am trying to set up a library in python. I have created a setup.py file and in a folder under that I have a library folder, and then I tried to create an sample and test folder (for sample code and tests that I would include)
Folders:
- setup.py
- cvImageUtils # this is the library
- __init__.py # this includs ColorUtils
- ColorUtils.py # this includes a class called ColorUtils
- examples
- color.py # this is shown below
init.py in ColorUtils folder
from . import ColorUtils
ColorUtils.py
class ColorUtils:
def __...
Color.py
from cvImageUtils import ColorUtils
m1 = cv2.imread(os.path.join(image_folder, "bb.jpeg"), 1) # 1 = load color
cv2.imshow('grayscale', ColorUtils.convert_rbg_to_grayscale(m1))
At first, it said, unable to find the module, so I added to the top of the file the following based on another SO solution:
import sys
sys.path.append('../')
Now that seems broken to me already, but it did get me past the no module found, but now it says ColorUtils has no method convert_rbg_to_grayscale. So Then I had to change it to ColorUtils.ColorUtils.convert_rbg_to_grayscale
cv2.imshow('grayscale', ColorUtils.ColorUtils.convert_rbg_to_grayscale(m1))
How can I setup the folder so that it allows me to include the library without sys, and call it without declaring ColorUtils twice.

change your __init__.py:
from cvImageUtils.ColorUtils import ColorUtils
I don't think you'll need to import sys anymore, and you don't have import ColorUtils twice. but just like you have to instantiate an object, you should create a ColorUtils object.
my personal preference would be not creating a Class for Utils.
you might have done this already, but if you want to use a method straight from a class like you did in python, you might want to declare it static.
class ColorUtils:
#staticmethod
def util_method():
pass
then you can just do:
ColorUtils.util_method()
Update:
you can read more about relative/absolute import from here as well.
to fix your actual problem though, you can do:
color.py
remove your import sys and sys call from color.py
change: import cvImageUtils.ColorUtils as ct
to: from cvImageUtils.ColorUtils import *
remove all your ct reference instead just use the actual functions.
cvImageUtils/__init__.py
change: from . import ColorUtils
to __all__=['ColorUtils']
I was able to run color.py to get all the images printed out on screen.
a image.png was also generated locally as well.

Every directory that you want to expose in module search(we usually hide test.py) in python need a init.py file. That should be rule of thumb, by using sys module you can add the module to your "module search path".
After having init.py in your directories, you need to import packages/modules/funcitons you want to use:-
import cvImageUtils.ColorUtils.convert_rbg_to_grayscale
You can execute following code in python to see, what have included in your sys path(used by python to search for modules/packages)
import sys
sys.path
Look into below links for more detailed explations
https://www.programiz.com/python-programming/package
https://www.programiz.com/python-programming/modules#search

Python import one subpackage without others

I have the following package structure
package
__init__.py
sub1
__init__.py
foo.py # Contains class Foo
sub2
__init__.py
bar.py # Contains class Bar
I want to be able to just import package and have package.Foo and package.Bar, i.e. I want to have the subpackages be transparent to users.
The catch is that importing sub2 takes a long time, and many users don't care at all about the stuff in sub2 and only want the stuff in sub1. Thus I want users to be able to say import package.sub1 or from package import sub1 to just import sub1 and skip the import of sub2.
I know I can achieve the first part by having package/__init__.py contain
from .sub1 import *
from .sub2 import *
and having package/sub1/__init__.py be from .foo import Foo and similarly for sub2. However, this will always import sub1 and sub2 even if the user tries to import only package.sub1.
Correspondingly, I can achieve the second part by having package/__init__.py be empty and using the same sub1/__init__.py as above. However, then just saying import package doesn't load sub1 or sub2, so users would have to explicitly load them and then refer to package.sub1.Foo.
Ideally a solution would work both in 2.7.10 and 3.5.0, but I'll accept one or the other if both isn't possible.

The LazyLoader class is provided for exactly this kind of situation: postponing loading of the module when it is actually used, instead of at the point of importing it.
To build a lazy loader you can follow the example in the documentation:
suffixes = importlib.machinery.SOURCE_SUFFIXES
loader = importlib.machinery.SourceFileLoader
lazy_loader = importlib.util.LazyLoader.factory(loader)
finder = importlib.machinery.FileFinder(path, [(lazy_loader, suffixes)])
then you can use finder.find_spec to obtain the spec of a module and pass the result to Loader.create_module to load it.
This is a bit cumbersome to do manually for just one module.
Note that searching for "lazy import python" you'll find quite a few solutions that have different pro and cons, some of which run in python2.x. However the LazyLoader class above is the official way of doing it in python3.5+

you could add your shortcuts to the __init__.py of your module:
package/__init__.py
__all__ = [
... add everything you want to be listed for this module
'Foo',
'Bar',
...
]
from package.sub1.foo import Foo
from package.sub2.bar import Bar
now you should be able to call:
from package import Bar

Python: Unit Testing Module and Relative Imports

Currently have the following file hierarchy:
\package
__init__.py
run_everything.py
\subpackage
__init__.py
work.py
work1.py
work2.py
\test
__init__.py
test_work.py
test_work1.py
My first question is regarding relative imports. Suppose in \subpackage\work.py I have a function called custom_function(), and I would like to test that function in test_work.py. For some reason I can not figure out how to make this import from one module to another. Trying from .. subpackage.work1 import custom_function() does not seem to work, and yields the error Attempted relative import in non-package Is there any way to resolve this?
2)
I would like to run all test files from run_everything.py with one function, would adding a suite() function in each test_work*.py file, which adds each unit_testing class to suite.addTest(unittest.makeSuite(TestClass)), and finally importing them into the top-level run_everything.py be the most conventional way in Python2.7?

Here is a hack*
Insert the path's to "subpackage" and "test" to your python path in run_everything using:
import sys
sys.path.insert(0, '/path/to/package/subpackage')
sys.path.insert(0, '/path/to/package/test')
And then, you can import all your files using vanilla imports in run_everything:
import work, work1, work2
import test_work, test_work1
*This won't permanently affect your PYTHONPATH.