I can't seem to find a way to create a standalone package for image recognition. I have a project I'm writing in python, and I found a way to do what I need using OpenCV, but I can't find a way to import the library into my project unless it is installed at the system level on Ubuntu. In other words, I can't seem to plop the build folder into my project after building the OpenCV library. And I can't find the equivalent of cv2.matchTemplate() in PIL or Pillow. So really there are two questions here.
1) How can I attach the build folder to my project, in order to avoid installing the OpenCV at the system level.
2) Is there an equivalent of cv2.matchTemplate() in PIL or Pillow that I can't seem to find?
Thanks.
You need to:
Download OpenCV
Use CMake to tell it to compile statically and to tell it to compile the Python module
Compile, and install into a directory you want.
Find in that directory the file under a directory called python, called cv2.so
Distribute that file with your Python code.
Now that I told you how to do it, let me tell you why your approach isn't a very good idea:
If the version of Python changes, you need to recompile (the so file) and redistribute your entire application
If the version of OpenCV changes you will need to recompile (the so file) and redistribute your entire application
You don't control what version of Python your users have
There can be important subtleties in version of libjpg, libtiff, zlib and others that could prevent your application from working, all outside your control.
You are converting a multi-platform application into a platform specific solution.
Related
I've looked around (including StackOverflow) but can't really find something similar to my case so I'll just ask it here. I'm building an application with Python (it is currently a source code that I'm planning to turn into a Windows executable) and is using this library: https://github.com/xiph/rnnoise. It is not a Python package but a third-party library written primarily in C.
Since I have to install it into my computer in order to use the library inside my code, I want to ask if there is a way to include this library along with the executable so that anyone can just download and use it without having to set up the library? This is important since I can only set up the library on a Linux machine and the executable needs to work with the library platform-independently.
Update: Thanks to Vimalan E and Marat, I had a bit more clue on what I need to do. I was managed to locate the .so files of my required library after running make install on it. The question left is I don't know how to link it with an executable that will be made from my Python source code. For now, I want to include the library as a binary file along with my application (though I am not sure how to achieve that, maybe putting putting the .so near the .exe should do the trick).
Thank you.
I installed Opencv in a Docker container so that I can upload the linux binaries for support a git project that isn't mine. I need it to run python 3.6 and opencv 3.x, which all seems to be working fine. However, the directory containing the cv2 folder only has one .so file: cv2.cpython-36m-x86_64-linux-gnu.so . The project I'm trying to contribute to has the build for opencv py2.7, and that folder has dozens of .so files for many relevant opencv packages, so I feel like something is wrong. Can anyone help?
And here's the link to the project I am trying to add support to.
https://github.com/Miserlou/lambda-packages/tree/master/lambda_packages/OpenCV
Assuming you don't want to unpack the tar, here's how the inside looks for the python 2.7 package
There is nothing wrong as long as you can import it in python and utilize it's functionality. I'm using ROS Kinetic for my research, which comes with a built in release of opencv. It also has just one cv2.so file and it's working perfectly fine.
This is probably a question that has a very easy and straightforward answer, however, despite having a few years programming experience, for some reason I still don't quite get the exact concepts of what it means to "build" and then to "install". I know how to use them and have used them a lot, but have no idea about the exact processes which happen in the background...
I have looked across the web, wikipedia, etc... but there is no one simple answer to it, neither can I find one here.
A good example, which I tried to understand, is adding new modules to python:
http://docs.python.org/2/install/index.html#how-installation-works
It says that "the build command is responsible for putting the files to install into a build directory"
And then for the install command: "After the build command runs (whether you run it explicitly, or the install command does it for you), the work of the install command is relatively simple: all it has to do is copy everything under build/lib (or build/lib.plat) to your chosen installation directory."
So essentially what this is saying is:
1. Copy everything to the build directory and then...
2. Copy everything to the installation directory
There must be a process missing somewhere in the explanation...complilation?
Would appreciate some straightforward not too techy answer but in as much detail as possible :)
Hopefully I am not the only one who doesn't know the detailed answer to this...
Thanks!
Aivoric
Building means compiling the source code to binary in a sandbox location where it won't affect your system if something goes wrong, like a build subdirectory inside the source code directory.
Install means copying the built binaries from the build subdirectory to a place in your system path, where they become easily accessible. This is rarely done by a straight copy command, and it's often done by some package manager that can track the files created and easily uninstall them later.
Usually, a build command does all the compiling and linking needed, but Python is an interpreted language, so if there are only pure Python files in the library, there's no compiling step in the build. Indeed, everything is copied to a build directory, and then copied again to a final location. Only if the library depends on code written in other languages that needs to be compiled you'll have a compiling step.
You want a new chair for your living-room and you want to make it yourself. You browse through a catalog and order a pile of parts. When they arrives at your door, you can't immediately use them. You have to build the chair at your workshop. After a bit of elbow-grease, you can sit down in it. Afterwards, you install the chair in your living-room, in a convenient place to sit down.
The chair is a program you want to use. It arrives at your house as source code. You build it by compiling it into a runnable program. You install it by making it easier to use.
The build and install commands you are refering to come from setup.py file right?
Setup.py (http://docs.python.org/2/distutils/setupscript.html)
This file is created by 3rd party applications / extensions of Python. They are not part of:
Python source code (bunch of c files, etc)
Python libraries that come bundled with Python
When a developer makes a library for python that he wants to share to the world he creates a setup.py file so the library can be installed on any computer that has python. Maybe this is the MISSING STEP
Setup.py sdist
This creates a python module (the tar.gz files). What this does is copy all the files used by the python library into a folder. Creates a setup.py file for the module and archives everything so the library can be built somewhere else.
Setup.py build
This builds the python module back into a library (SPECIFICALLY FOR THIS OS).
As you may know, the computer that the python library originally came from will be different from the library that you are installing on.
It might have a different version of python
It might have a different operating system
It might have a different processor / motherboard / etc
For all the reasons listed above the code will not work on another computer. So setup.py sdist creates a module with only the source files needed to rebuild the library on another computer.
What setup.py does exactly is similar to what a makefile would do. It compiles sources / creates libraries all that stuff.
Now we have a copy of all the files we need in the library and they will work on our computer / operating system.
Setup.py install
Great we have all the files needed. But they won't work. Why? Well they have to be added to Python that's why. This is where install comes in. Now that we have a local copy of the library we need to install it into python so you can use it like so:
import mycustomlibrary
In order to do this we need to do several things including:
Copy files to their library folders in our version of python.
Make sure library can be imported using import command
Run any special install instructions for this library. (seting up paths, etc)
This is the most complicated part of the task. What if our library uses BeautifulSoup? This is not a part of Python Library. We'd have to install it in a way such that our library and any others can use BeautifulSoup without interfering with each other.
Also what if python was installed someplace else? What if it was installed on a server with many users?
Install handles all these problems transparently. What is does is make the library that we just built able to run. All you have to do is use the import command, install handles the rest.
I have some small Python programs which depend on several big libraries, such as:
NumPy & SciPy
matplotlib
PyQt
OpenCV
PIL
I'd like to make it easier to install these programs for Windows users. Currently I have two options:
either create huge executable bundles with PyInstaller, py2exe or similar tool,
or write step-by-step manual installation instructions.
Executable bundles are way too big. I always feel like there is some magic happening, which may or may not work the next time I use a different library or a new library version. I dislike wasted space too. Manual installation is too easy to do wrong, there are too many steps: download this particular interpreter version, download numpy, scipy, pyqt, pil binaries, make sure they all are built for the same python version and the same platform, install one after another, download and unpack OpenCV, copy its .pyd file deep inside Python installation, setup environment variables and file asssociations... You see, few users will have the patience and self-confidence to do all this.
What I'd like to do: distribute only a small Python source and, probably, an installation script, which fetches and installs all the missing dependencies (correct versions, correct platform, installs them in the right order). That's a trivial task with any Linux package manager, but I just don't know which tools can accomplish it on Windows.
Are there simple tools which can generate Windows installers from a list of URLs of dependencies1?
1 As you may have noticed, most of the libraries I listed are not installable with pip/easy_install, but require to run their own installers and modify some files and environment variables.
npackd exists http://code.google.com/p/windows-package-manager/ It could be done through here or use distribute (python 3.x) or setuptools (python 2.x) with easy_install, possibly pip (don't know it's windows compatibility). But I would choose npackd because PyQt and it's unusual setup for pip/easy_install (doesn't play with them nicely, using a configure.py instead of setup.py). Though you would have to create your own repo for npackd to use for some of them. I forget what is contributed in total for python libs with it.
AFAIK there is no tool (and I'd assume you googled), so you must make one yourself.
Fetching the proper library versions seems simple enough -- using python's ftplib you can fetch the proper installers for every library. How would you know which version is compatible with the user's python? You can store different lists of download URLs, each for a different python version (this method came off the top of my head and there is probably a better way; not that it matters much if it's simple and it works).
After you figure out how to make each installer run, you can py2exe your installer script, and even use it to fetch the program itself.
EDIT
Some Considerations
There are a couple of things that popped into my mind just as I posted:
First, some pseudocode (how I would approach it, anyway)
#first, we check modules
try:
import numpy
except ImportError:
#flag numpy for installation
#lather, rinse repeat for all dependencies
#next we check version compatibility -- note that if a library version you need
#is not backwards-compatible, you're in DLL hell, and there is little we can do.
<insert version-checking code here>
#once you have your unavailable dependencies, you install them
import ftplib
<all your file-downloading here>
#now you install. sorry I can't help you here.
There are a few things you can do to make your utility reusable --
put all URL lists, minimum version numbers, required library names etc in config files
Write a script which helps you set up an installer
Py2exe the installer-maker-script
Sell it
Even better, release it under GPL so we can all feast upon fruits of your labours.
I have a similar need as you, but in addition I need the packaged application to work on several platforms. I'm currently exploring the currently available solutions, here are a few interesting ones:
Use SnakeBasket, which wraps around Pip and add a recursive dependency resolution plus a heuristic to choose the right version when there are conflicts.
Package all dependencies as an egg, but not your sourcecode which will still be editable: https://stackoverflow.com/a/528064/1121352
Package all dependencies in a zip file and directly import the modules on the fly: Cross-platform alternative to py2exe or http://davidf.sjsoft.com/mirrors/mcmillan-inc/install1.html
Using buildout: http://www.buildout.org/en/latest/install.html
Using virtualenv with virtualenv-tools (instead of "relocate")
If your main problem when freezing your code using PyInstaller or similar is that you end up with a big single file, you can customize the process so that you get several files, one for each dependency, instead of one big executable.
I will update here if I find something that fills my bill.
What's the appropriate way of constructing a Python package via disutils when that Python package relies on a large system library?
I found this similar question, but it refers to an installable Python app, not a generic package.
I've written a package that relies on OpenCV. I'm only concerned with supporting Linux distros, but most distros either don't provide OpenCV or provide a version that's too old to use. Unfortunately, OpenCV is to large and cumbersome (and depends on several other system libraries) to include in the package and compile during the build step.
My current approach is to simply do nothing special in my setup.py and just import its Python modules in a try/except, showing a detailed error message if the import fails. Is there a better way?
You could use zc.buildout: http://www.buildout.org/
You should be able to extend the buildout config for your project from this one: https://github.com/cpsaltis/opencv-buildout