I use two libraries built on top of caffe: crf-rnn(https://github.com/torrvision/crfasrnn/tree/master/python-scripts) and hed(https://github.com/s9xie/hed/blob/master/examples/hed/), the former for semantic image segmentation, the latter for contour detection. Finally, I realized how to get them to work together for object tracking, but now I face an embarrassing problem: as both are built on top of caffe, they import the same package, but each with very different content, i.e. crf-rnn uses caffe.Segmenter which hed doesn't have and ed uses caffe.TEST which crf-rnn doesn't have.
Python doesn't allow import of two packages with the same name. I've tried finding a workaround by puting hed in a separate Python file and importing it in the main script, and using as to import caffe as cf for one of the packages, but so far nothing has worked out.
Any suggestions?
EDIT: this is a file called Aux.py
def import_hed_caffe():
import sys,os
caffe_dir = '/home/alex/Downloads/hed/python'
sys.path.insert(0,caffe_dir)
hed_model = 'deploy.prototxt'
hed_pretrained = 'hed_pretrained_bsds.caffemodel'
import caffe as cf
net = cf.Net(hed_model, hed_pretrained, cf.TEST)
return net
This is the main script:
caffe_root = '../caffe-crfrnn/'
sys.path.insert(0, caffe_root + 'python')
import caffe as espresso
import AuxScript
net = espresso.Segmenter(MODEL_FILE, PRETRAINED, gpu=False)
a=AuxScript.import_hed_caffe()
and I get
AttributeError: 'module' object has no attribute 'TEST'
Needless to say, separately everything works fine, so it's just the import
EDIT 2:
./CMakeFiles
./CMakeFiles/pycaffe.dir
./CMakeFiles/pycaffe.dir/caffe
./caffe
./caffe/imagenet
./caffe/proto
./caffe/test
EDIT 3:
├── caffe
│ ├── _caffe.cpp
│ ├── _caffe.so -> /home/alex/Downloads/hed/lib/_caffe.so
│ ├── classifier.py
│ ├── classifier.pyc
│ ├── detector.py
│ ├── detector.pyc
│ ├── draw.py
│ ├── imagenet
│ │ └── ilsvrc_2012_mean.npy
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── io.py
│ ├── io.pyc
│ ├── net_spec.py
│ ├── net_spec.pyc
│ ├── proto
│ │ ├── caffe_pb2.py
│ │ └── __init__.py
│ ├── pycaffe.py
│ ├── pycaffe.pyc
│ └── test
│ ├── test_layer_type_list.py
│ ├── test_net.py
│ ├── test_net_spec.py
│ ├── test_python_layer.py
│ ├── test_python_layer_with_param_str.py
│ └── test_solver.py
├── classify.py
├── CMakeFiles
│ ├── CMakeDirectoryInformation.cmake
│ ├── progress.marks
│ └── pycaffe.dir
│ ├── build.make
│ ├── caffe
│ │ └── _caffe.cpp.o
│ ├── cmake_clean.cmake
│ ├── CXX.includecache
│ ├── DependInfo.cmake
│ ├── depend.internal
│ ├── depend.make
│ ├── flags.make
│ ├── link.txt
│ └── progress.make
├── cmake_install.cmake
├── CMakeLists.txt
├── detect.py
├── draw_net.py
├── Makefile
├── requirements.txt
I have seen your last edit, and I must say that changing/tampering with python sys.path is necessary in your context but not sufficient here: you have to rename one of the caffe packages.
Ex: if the caffe package is a directory called caffe containing a __init__.py file, rename caffe to espresso and in your code simply:
import espresso
(if it's just a caffe.py file, rename to espresso.py although it may be more problematic if there are other modules in the same directory, well worth a try)
BTW: When importing a module, say, xxx, you can know which full filepath it is using by typing:
print(xxx.__file__)
(useful when you have a doubt)
OK, so I found the least sophisticated solution possible: I wrote two scripts, one for crf-rnn producing the blobs that I ran on the full dataset just once and stored the output.
Then I wrote the second script, with hed edge detector that I use every time I detect and track objects.
Related
My tests were working a few hours before. But suddenly tests located in root folder/tests are giving me ModuleNotFoundError: No module named 'tests.test_* I imported the tests.test_a in a top level python file and it worked. Any guesses why cant pytest load them?
I am asking this because this is weird that the test folder located inside root folder/some folder/tests also contains __init__.py and they work fine. I read this question and deleted the __init__.py and it worked. But it is confusing as to why the top level tests would not work.
The folder structure is along the lines of:
.
├── authenticate
│ ├── exceptions.py
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── exceptions.cpython-38.pyc
│ │ ├── __init__.cpython-38.pyc
│ │ ├── repository.cpython-38.pyc
│ │ └── use_case.cpython-38.pyc
│ ├── Readme.md
│ ├── repository.py
│ ├── tests
│ │ ├── __init__.py
│ │ ├── __pycache__
│ │ └── test_authenticate.py
│ └── use_case.py
├── tests
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── test_authenticate.cpython-38-pytest-5.4.2.pyc
│ ├── test_authenticate.py
As weird as it sounds, I had forgotten to add __init__.py in one of my packages. I don't know why it was causing tests in my top level directory to fail
I'm developing a python package foo. My project structure looks like this:
.
├── foo
│ ├── foo
│ │ ├── bar.py
│ │ ├── foo.py
│ │ ├── __init__.py
│ ├── README.md
│ └── setup.py
├── footest
│ ├── test.py
test.py only has 1 line: import foo
In order for test.py to be able to import the package foo I install it with the command pip3 install -e foo.
Now a new folder called foo.egg-info is created under foo/
.
├── foo
│ ├── foo
│ │ ├── bar.py
│ │ ├── foo.py
│ │ ├── __init__.py
│ ├── foo.egg-info
│ │ ├── dependency_links.txt
│ │ ├── PKG-INFO.txt
│ │ ├── requires.txt
│ │ ├── SOURCES.txt
│ │ ├── top_level.txt
│ ├── README.md
│ └── setup.py
├── footest
│ ├── test.py
What's the purpose of this folder? I tried deleting it and test.py still ran properly. Is is just leftover garbage, similar to the .o files when compiling C projects? If so, is there a way to automatically remove it?
The package.egg-info saves meta data about your installed package like version.
It is used when you for example uninstall it, or "pip list" to see what is installed.
you can open it and take a look.
I want to create a pybuilder project with unit tests and packages. As an example, modified simple python app example, with "helloworld" moved to package "hello".
My first instinct was to match the package structure with "main" and "unittest" sources:
+---src
+---main
| \---python
| \---hello
| helloworld.py
| __init__.py
|
\---unittest
\---python
\---hello
helloworld_tests.py
__init__.py
This does not work because of conflicting "hello" package.
BUILD FAILED - 'module' object has no attribute 'helloworld_tests'
I see pybuilder itself just skips the top-level pybuilder package in unittests, but won't do if there are multiple top-level packages.
My second guess would be to create extra top level package for unittests.
\---unittest
\---python
\---tests
| __init__.py
\---hello
helloworld_tests.py
__init__.py
Is there a better solution or established convention how to organize python tests in packages?
Probably nothing really new for the OP, but I just wanted to collect all options that I could come up with in one place:
1) Just append _tests to names of top-level packages
The easiest way to mirror the structure of src/main/python in the src/test/python almost 1:1 would be by simply appending _tests to the names of the top-level packages. For example, if I have only one top-level package rootPkg, then I can add the corresponding rootPkg_tests to the test/ subdirectory:
src
├── main
│ ├── python
│ │ └── rootPkg
│ │ ├── __init__.py
│ │ ├── pkgA
│ │ │ ├── __init__.py
│ │ │ └── modA.py
│ │ └── pkgB
│ │ ├── __init__.py
│ │ └── modB.py
│ └── scripts
│ └── entryPointScript.py
└── test
└── python
└── rootPkg_tests
├── __init__.py
├── pkgA
│ ├── __init__.py
│ └── modA_tests.py
└── pkgB
├── __init__.py
└── modB_tests.py
This seems to work nicely with PyBuilder 0.11.15 and unittest plugin (notice that I've deviated from PyBuilders convention, and put tests in test instead of unittest, you probably shouldn't do this if you intend to use multiple testing frameworks).
2) If there is only one package: do it like PyBuilder
The PyBuilder is itself built with PyBuilder. This is what its source directory looks like (drastically reduced, unnecessary details omitted):
src
├── main
│ ├── python
│ │ └── pybuilder
│ │ ├── __init__.py
│ │ ├── cli.py
│ │ ├── core.py
│ │ └── plugins
│ │ ├── __init__.py
│ │ ├── core_plugin.py
│ │ └── exec_plugin.py
│ └── scripts
│ └── pyb
└── unittest
└── python
├── cli_tests.py
├── core_tests.py
├── plugins
│ ├── exec_plugin_tests.py
│ ├── __init__.py
│ ├── python
│ │ ├── core_plugin_tests.py
│ │ ├── __init__.py
If I understand it correctly, the tree in unittest mirrors the tree in src, but the directory for the top-level package pybuilder is omitted. That's what you have described in your question as first workaround. The drawback is that it doesn't really work if there are multiple top-level packages.
3) Add one additional tests top-level package
That's what you have proposed as a workaround: mirror the tree in main, but wrap everything in an additional tests-package. This works with many top-level packages in /src/main/python and prevents any package name collisions.
I'm not aware of any convention. Upvote one of the comments below if you have an opinion on that matter.
i have the following file structure:
ihe/
├── dcmt
│ ├── actions
│ ├── calendar_observer
│ ├── cms
│ ├── consumption
│ ├── data_mining
│ ├── dcmt
│ ├── dcmt_db
│ ├── dcmt_db.bak.bak
│ ├── dcmt_db.sqlite
│ ├── devices
│ ├── d.py
│ ├── gadgets
│ ├── history
│ ├── houses
│ ├── hwc_settings
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── manage.py
│ ├── notifications
│ ├── profitable
│ ├── rules
│ └── schedule
├── hwc
│ ├── configuration
│ ├── daemons
│ ├── database
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── utils
│ └── wrapper
├── __init__.py
├── __init__.pyc
dcmt is a django project. hwc is pure python. however for instance in hwc/daemons there is a runme.py script. in that runme.py script i want to be able to import the models from the django project. Now as i understand it i have to have the correct python path and then somehow set the django settings. My question is how do i best do this so that for the whole hwc modules I only have to do that once?
Your project structure seems a bit confused.
It's probably not a good idea to have a Django project inside another package hierarchy. A lot of the import paths assume your project is in a top-level package and the only reason you're probably not running into issues already is that Python 2.x still supports relative imports (which have been removed in 3.x). This makes references to packages very ambiguous and can cause weird bugs.
From what I can see your settings package is actually called (fully-qualified) ihe.dcmt.hwc_settings. If ihe is in your Python path (check the value of sys.path in the script you're trying to run), that (i.e. the fully-qualified path) is probably what DJANGO_SETTINGS_MODULE should point at.
If you want to hook into Django's functionality in your scripts, you might want to look into the documentation for writing manage.py commands. This would let you write Django-related scripts more consistently and save you the worry about referencing and initialising Django's settings correctly yourself.
I've the following project setup
....
├── lira
│ ├── __init__.py
│ ├── admin.py
│ ├── ajax.py
│ ├── authentication.py
│ ├── context_processors.py
│ ├── fencoder
│ │ ├── __init__.py
│ │ ├── encoder.py
│ │ ├── ffmpeg_commands.py
│ │ ├── frames.py
│ │ ├── utils.py
│ │ └── video.py
│ ├── models.py
....
And when I try to import from lira.fencoder import encoder I get an error ImportError: cannot import name encoder.
What is wrong with the above project setup though this morning it was the same and it worked?
Sultan
If you want to debug import problems, sometimes a simple launch of
python -v -m path/to/python/module/to_start
will help you, it will show you all the imports done by the python interpreter and help detect cyclic imports.