Develop Reference

Namespace corrupted when using setup.py and causes AttributeError: module has no attribute

I wrote a small tool (package) that reuses an existing namespace, pki.server. I named my package as pki.server.healthcheck. The old namespace did not use setuptools to install the package, while my package uses it.
Contents of setup.py
from setuptools import setup
setup(
name='pkihealthcheck',
version='0.1',
packages=[
'pki.server.healthcheck.core',
'pki.server.healthcheck.meta',
],
entry_points={
# creates bin/pki-healthcheck
'console_scripts': [
'pki-healthcheck = pki.server.healthcheck.core.main:main'
]
},
classifiers=[
'Programming Language :: Python :: 3.6',
],
python_requires='!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*',
setup_requires=['pytest-runner',],
tests_require=['pytest',],
)
The installation tree (from scenario 1 below) looks like:
# tree /usr/lib/python3.8/site-packages/pki/
├── __init__.py <---- Has methods and classes
├── cli
│   ├── __init__.py <---- Has methods and classes
│   ├── <some files>
├── server
│   ├── cli
│   │   ├── __init__.py <---- Has methods and classes
│   │   ├── <Some files>
│   ├── deployment
│   │   ├── __init__.py <---- Has methods and classes
│   │   ├── <some files>
│   │   └── scriptlets
│   │   ├── __init__.py <---- Has methods and classes
│   │   ├── <some files>
│   ├── healthcheck
│   │   ├── core
│   │   │   ├── __init__.py <---- EMPTY
│   │   │   └── main.py
│   │   └── pki
│   │   ├── __init__.py <---- EMPTY
│   │   ├── certs.py
│   │   └── plugin.py
│   └── instance.py <---- Has class PKIInstance
└── <snip>
# tree /usr/lib/python3.8/site-packages/pkihealthcheck-0.1-py3.8.egg-info/
├── PKG-INFO
├── SOURCES.txt
├── dependency_links.txt
├── entry_points.txt
└── top_level.txt
I read the official documentation and experimented with all 3 suggested methods. I saw the following results:
Scenario 1: Native namespace packages
At first, everything seemed smooth. But:
# This used to work before my package gets installed
>>> import pki.server
>>> instance = pki.server.instance.PKIInstance("pki-tomcat")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'pki.server' has no attribute 'instance'
Now, only this works
>>> import pki.server.instance
>>> instance = pki.server.instance.PKIInstance("pki-tomcat")
>>> instance
pki-tomcat
Scenario 2: pkgutil-style namespace packages
I am restricted from using this method as my other __init__.py contain classes and functions
Scenario 3: pkg_resources-style namespace packages
Though this method was not-recommended, I went ahead and experimented with it by adding namespace=pki.server.healthcheck to my setup.py. This made all pki.* modules invisible
So I am convinced that Scenario 1 seems to be the closest to what I'm trying to achieve. I was reading an old post to understand more on how import in python works.
My question is: Why does a perfectly working snippet break after I install my package?

Your __init__.py files need to import the files. You have two options--absolute and relative imports:
Relative Imports
pki/__init__.py:
from . import server
pki/server/__init__.py:
from . import instance
Absolute Imports
pki/__init__.py:
import pki.server
pki/server/__init__.py:
import pki.server.instance

When package has same name as .py file inside the package, import X imports the module and not the package itself. How to avoid?

So the package Im constructing looks something like this:
Blur/
├── blur
│   ├── __init__.py
│   ├── blur.py
│   ├── funcs
│   │   ├── __init__.py
│   │   ├── face_funcs.py
│   │   └── funcs.py
│   ├── tests
│   │   ├── __init__.py
│   │   └── test_blur.py
│   └── utils
│   ├── __init__.py
│   └── timer.py
└── setup.py
Doing import blur imports blur.py module, and not the whole package itself. If I change blur.py's name, then do the import, i get the whole package. Any way to get the whole package without changing blur.py's name?

If you specify the file path to the "Blur" folder in the import statement, that should force python to import the package, and not the .py file. I believe by default python looks for files ending in .py when importing.

AFAIK it is not possible as a built-in Python mechanism, so changing the name would be your best bet.
There are, however, workarounds available, but in my opinion such solutions are rather ugly.
Here is an example:
import importlib
import os
import sys
def custom_import(repo_path, directory, package=False):
absolute_path = repo_path if package else os.path.join(repo_path, directory)
if os.path.exists(absolute_path):
sys.path.insert(0, absolute_path)
if package:
imported = importlib.import_module(f'{directory}')
else:
imported = importlib.import_module(f'{directory}.{directory}')
sys.path.pop(0)
return imported
raise ImportError(' No such package/module found')
Example use:
package = custom_import('D:\PycharmProjects\python-tools', 'roomba', package=True)
module = custom_import('D:\PycharmProjects\python-tools', 'roomba')
module.main()

Python: AttributeError module x has no attribute y

I have a project with directory structure like below
.
├── Pipfile
├── Pipfile.lock
├── module
│   ├── __init__.py
│   ├── helpers
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   │   └── __init__.cpython-36.pyc
│   │   ├── dynamo.py
│   │   └── logger.py
│   └── test.py
Relavant code
logger.py
import click
import sys
from tabulate import tabulate
def formatter(string, *rest):
return string.format(*rest)
def info(*rest):
"""Write text in blue color
"""
click.echo(click.style('☵ ' + formatter(*rest), fg='blue'))
test.py
import helpers
helpers.logger.info('Trying')
When I try to run using the command
python3 module/test.py
I get this error
Traceback (most recent call last):
File "module/test.py", line 4, in <module>
helpers.logger.info('Trying')
AttributeError: module 'helpers' has no attribute 'logger'
I have tried restructuring the code. Putting the helpers directory outside, in level with module directory. But still it didn't work, which it should not have, from what I have read. I tried researching a bit about __init__.py and python module system. The more I read, the more confusing it gets. But from whatever I learned, I created another sample project. With the following structure,
.
└── test
├── __init__.py
├── helpers
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-36.pyc
│   │   └── quote.cpython-36.pyc
│   └── quote.py
├── index.py
├── logger
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-36.pyc
│   │   └── info.cpython-36.pyc
│   └── info.py
Code is same as first project.
Here when I do,
python3 test/index.py
It works as expected. The only difference between the two projects:
In the first project, I used pipenv to install deps and create virtual environment.

Using your initial layout (with loggers as a submodule of the helpers package), you'd need to explicitely import loggers in helpers/__init__.py to expose it as an attribute of the helpers package:
# helpers/__init__.py
from . import logger

logger is module and not attribute and helpers.logger evalutes logger as attribute. Actually you should do as follow:
from helpers import logger
print(logger.info('Trying'))

Two implementations of a module with a same name-import problems

I use two libraries built on top of caffe: crf-rnn(https://github.com/torrvision/crfasrnn/tree/master/python-scripts) and hed(https://github.com/s9xie/hed/blob/master/examples/hed/), the former for semantic image segmentation, the latter for contour detection. Finally, I realized how to get them to work together for object tracking, but now I face an embarrassing problem: as both are built on top of caffe, they import the same package, but each with very different content, i.e. crf-rnn uses caffe.Segmenter which hed doesn't have and ed uses caffe.TEST which crf-rnn doesn't have.
Python doesn't allow import of two packages with the same name. I've tried finding a workaround by puting hed in a separate Python file and importing it in the main script, and using as to import caffe as cf for one of the packages, but so far nothing has worked out.
Any suggestions?
EDIT: this is a file called Aux.py
def import_hed_caffe():
import sys,os
caffe_dir = '/home/alex/Downloads/hed/python'
sys.path.insert(0,caffe_dir)
hed_model = 'deploy.prototxt'
hed_pretrained = 'hed_pretrained_bsds.caffemodel'
import caffe as cf
net = cf.Net(hed_model, hed_pretrained, cf.TEST)
return net
This is the main script:
caffe_root = '../caffe-crfrnn/'
sys.path.insert(0, caffe_root + 'python')
import caffe as espresso
import AuxScript
net = espresso.Segmenter(MODEL_FILE, PRETRAINED, gpu=False)
a=AuxScript.import_hed_caffe()
and I get
AttributeError: 'module' object has no attribute 'TEST'
Needless to say, separately everything works fine, so it's just the import
EDIT 2:
./CMakeFiles
./CMakeFiles/pycaffe.dir
./CMakeFiles/pycaffe.dir/caffe
./caffe
./caffe/imagenet
./caffe/proto
./caffe/test
EDIT 3:
├── caffe
│   ├── _caffe.cpp
│   ├── _caffe.so -> /home/alex/Downloads/hed/lib/_caffe.so
│   ├── classifier.py
│   ├── classifier.pyc
│   ├── detector.py
│   ├── detector.pyc
│   ├── draw.py
│   ├── imagenet
│   │   └── ilsvrc_2012_mean.npy
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── io.py
│   ├── io.pyc
│   ├── net_spec.py
│   ├── net_spec.pyc
│   ├── proto
│   │   ├── caffe_pb2.py
│   │   └── __init__.py
│   ├── pycaffe.py
│   ├── pycaffe.pyc
│   └── test
│   ├── test_layer_type_list.py
│   ├── test_net.py
│   ├── test_net_spec.py
│   ├── test_python_layer.py
│   ├── test_python_layer_with_param_str.py
│   └── test_solver.py
├── classify.py
├── CMakeFiles
│   ├── CMakeDirectoryInformation.cmake
│   ├── progress.marks
│   └── pycaffe.dir
│   ├── build.make
│   ├── caffe
│   │   └── _caffe.cpp.o
│   ├── cmake_clean.cmake
│   ├── CXX.includecache
│   ├── DependInfo.cmake
│   ├── depend.internal
│   ├── depend.make
│   ├── flags.make
│   ├── link.txt
│   └── progress.make
├── cmake_install.cmake
├── CMakeLists.txt
├── detect.py
├── draw_net.py
├── Makefile
├── requirements.txt

I have seen your last edit, and I must say that changing/tampering with python sys.path is necessary in your context but not sufficient here: you have to rename one of the caffe packages.
Ex: if the caffe package is a directory called caffe containing a __init__.py file, rename caffe to espresso and in your code simply:
import espresso
(if it's just a caffe.py file, rename to espresso.py although it may be more problematic if there are other modules in the same directory, well worth a try)
BTW: When importing a module, say, xxx, you can know which full filepath it is using by typing:
print(xxx.__file__)
(useful when you have a doubt)

OK, so I found the least sophisticated solution possible: I wrote two scripts, one for crf-rnn producing the blobs that I ran on the full dataset just once and stored the output.
Then I wrote the second script, with hed edge detector that I use every time I detect and track objects.

Django Project and Standalone Python project

i have the following file structure:
ihe/
├── dcmt
│   ├── actions
│   ├── calendar_observer
│   ├── cms
│   ├── consumption
│   ├── data_mining
│   ├── dcmt
│   ├── dcmt_db
│   ├── dcmt_db.bak.bak
│   ├── dcmt_db.sqlite
│   ├── devices
│   ├── d.py
│   ├── gadgets
│   ├── history
│   ├── houses
│   ├── hwc_settings
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── manage.py
│   ├── notifications
│   ├── profitable
│   ├── rules
│   └── schedule
├── hwc
│   ├── configuration
│   ├── daemons
│   ├── database
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── utils
│   └── wrapper
├── __init__.py
├── __init__.pyc
dcmt is a django project. hwc is pure python. however for instance in hwc/daemons there is a runme.py script. in that runme.py script i want to be able to import the models from the django project. Now as i understand it i have to have the correct python path and then somehow set the django settings. My question is how do i best do this so that for the whole hwc modules I only have to do that once?

Your project structure seems a bit confused.
It's probably not a good idea to have a Django project inside another package hierarchy. A lot of the import paths assume your project is in a top-level package and the only reason you're probably not running into issues already is that Python 2.x still supports relative imports (which have been removed in 3.x). This makes references to packages very ambiguous and can cause weird bugs.
From what I can see your settings package is actually called (fully-qualified) ihe.dcmt.hwc_settings. If ihe is in your Python path (check the value of sys.path in the script you're trying to run), that (i.e. the fully-qualified path) is probably what DJANGO_SETTINGS_MODULE should point at.
If you want to hook into Django's functionality in your scripts, you might want to look into the documentation for writing manage.py commands. This would let you write Django-related scripts more consistently and save you the worry about referencing and initialising Django's settings correctly yourself.