I am trying to build a package using the "conda build fibtestpackage" command.
The "fibtestpackage" is built using Cython, hence in the .sh file I have the command:
python setup.py build_ext --inplace
Even though the Build ends successfully, and the output is as expected:
Fetching packages ...
fibtestpackage-1.0.0 100% |################################| Time: 0:00:00 3.03 MB/s
Extracting packages ... [ COMPLETE ] |##################################################| 100%
Linking packages ...
[ COMPLETE ] |##################################################| 100%
the one test I have in the YAML file fails. The only test I have looks like this:
test:
# Python imports
imports:
- fibtestpackage
#- numpy
#- samplers
Hence, its an import error,
ImportError: No module named fibtestpackage
My first guess for why there is an importerror is because the .SO, .C, and .PYX files generated by the build (and the other files of the library) are NOT being place into the anaconda/lib/python2.7/site-packages/ folder where the other modules are located. Do I need to specify this in the shell file or should this be covered by the "python setup.py build_ext --inplace" command?
My second guess is that there are two different versions of conda (after updating) in the distribution,
1) Python 2.7.8 |Anaconda 2.1.0 (64-bit)| (default, Aug 21 2014, 18:22:21)
2) Python 2.7.8 |Continuum Analytics, Inc.| (default, Aug 21 2014, 18:22:21)
Could this be causing the issue? Perhaps the build environment is installing libraries in the wrong one?
Thank you in advance for your help!
Conda build creates a build conda environment, called _build (generally in ~/anaconda/envs/_build), which is where you should install everything in. For Python, generally doing python setup.py install is good enough if you include Python as a build dependency in the meta.yaml because the python will be the one installed in the _build environment and it will install there.
At the end of the build, conda build wraps up all the new files in the build environment and creates a conda package out of them. If there is a test phase, it deletes the _build environment and creates a test environment, called _test, and installs and runs the package there.
To debug this problem, try running the Python in ~/anaconda/envs/_test.
In your case, the issue is likely that python setup.py build_ext --inplace does not actually install the files, it just builds them. Check at the end of the build to see how many files it says there are. If there are 0, then this is the issue.
Related
I am trying to install nlopt onto macOS 10.15.5. I downloaded the nlopt-2.6.2.tar.gz file from the NLopt docs and ran the following from the nlopt-2.6.2 directory:
mkdir build
cd build
cmake -DNLOPT_OCTAVE=Off -DNLOPT_MATLAB=Off -DNLOPT_GUILE=Off ..
make
sudo make install
I got the following output: cmake.txt.
The header file (nlopt.h) installs correctly to /usr/local/include and the dynamic library (libnlopt.dylib) installs correctly to /usr/local/lib/, but neither the dist-info file nor the nlopt module itself installs.
I have also tried installing via pip, brew, and conda, none of which have worked. I have also tried cloning from this Github, which didn't work either.
I appreciate any help with this, because I am completely lost. I am relatively new to this kind of stuff, and I couldn't find any good answers online.
The official docs are somewhat laconic about the exact steps required for building nlopt with Python bindings. First of all, you'll need SWIG installed:
$ brew install swig
Then, you'll need numpy to be available for the target Python interpreter. It is already preinstalled for the system Python, otherwise install it via Homebrew or pip, depending on your Python installation.
Now run cmake:
$ cmake -DNLOPT_GUILE=OFF -DNLOPT_MATLAB=OFF -DNLOPT_OCTAVE=OFF -DNLOPT_TESTS=OFF
This will build bindings against the default Python 2.7 installation preinstalled on MacOS. If you need to build against custom Python installation (e.g. when you've installed Python 3 via Homebrew or PKG installer from https://www.python.org/downloads), pass it via the PYTHON_EXECUTABLE arg:
$ cmake -DNLOPT_GUILE=OFF -DNLOPT_MATLAB=OFF -DNLOPT_OCTAVE=OFF -DNLOPT_TESTS=OFF -DPYTHON_EXECUTABLE=/usr/local/bin/python3
Inspect the log now - Python, SWIG and numpy headers should be successfully located. Example output snippet (you may have different paths/versions printed):
-- Found PythonInterp: /usr/local/bin/python3.8 (found version "3.8.3")
-- Found PythonLibs: /Library/Frameworks/Python.framework/Versions/3.8/lib/libpython3.8.dylib (found suitable exact version "3.8.3")
-- Found NumPy: /Users/hoefling/Library/Python/3.8/lib/python/site-packages/numpy/core/include (found version "1.19")
-- Found SWIG: /usr/local/bin/swig (found version "4.0.2")
If any of those conditions is not satisfied (e.g. you see Could NOT find NumPy, Could NOT find PythonLibs or Could NOT find SWIG), then stop and make sure the configuration succeeds before proceeding next.
Now compile:
$ make
...
Scanning dependencies of target nlopt_python_swig_compilation
[ 96%] Swig compile nlopt.i for python
[ 96%] Built target nlopt_python_swig_compilation
Scanning dependencies of target nlopt_python
[ 98%] Building CXX object src/swig/CMakeFiles/nlopt_python.dir/CMakeFiles/nlopt_python.dir/nloptPYTHON_wrap.cxx.o
[100%] Linking CXX shared module _nlopt.so
[100%] Built target nlopt_python
Install:
$ make install
...
-- Installing: /usr/local/lib/python3.8/site-packages/nlopt.py
-- Installing: /usr/local/lib/python3.8/site-packages/_nlopt.so
Test the Python bindings are importable:
$ python -c "import nlopt; print(nlopt.__version__)"
2.6.2
I'm trying to use Python build tools to run CI for my project.
However the recomended "Install Python" task fails with Unable to process command '##vso[task.setvariable :
2018-11-27T09:01:45.8077207Z ##[section]Starting: Install Python
2018-11-27T09:01:45.8084595Z ==============================================================================
2018-11-27T09:01:45.8084738Z Task : Install Python on Windows
2018-11-27T09:01:45.8084853Z Description : Installs one or more Python versions into your build environment
2018-11-27T09:01:45.8084937Z Version : 1.0.13
2018-11-27T09:01:45.8085016Z Author : Steve Dower
2018-11-27T09:01:45.8085188Z Help : The *Install Python* task will install Python using a package from [nuget.org](https://nuget.org) and add it to `PATH`. These packages are part of the official release of CPython.
Most other tasks assume you have already run this task to install the desired runtime. The folder where they are installed is customizable, but be aware that you may need to update subsequent task configuration if you change this.
The recommended configuration for using this task with multiple versions of Python is to enable multi-configuration on the phase and specify the version number as a Multiplier. See [the documentation](https://go.microsoft.com/fwlink/?linkid=835763) for information about parallel execution.
(Note that the first-party *Use Python Version* task is preferable on the Hosted queues and works on all platforms, though as of 02 May 2018 is not fully available.)
2018-11-27T09:01:45.8085489Z ==============================================================================
2018-11-27T09:02:00.5624146Z ##[command]"C:\vsts-agent\_work\_tasks\InstallPython_6095fbbe-853d-4f2e-bc15-a88c92a78b4d\1.0.13\nuget.exe" install -OutputDirectory "C:\vsts-agent\_work\_tool\PythonNuget" python
2018-11-27T09:02:02.3259235Z Feeds used:
2018-11-27T09:02:02.3260274Z https://api.nuget.org/v3/index.json
2018-11-27T09:02:02.3265409Z
2018-11-27T09:02:02.3356523Z Installing package 'python' to 'C:\vsts-agent\_work\_tool\PythonNuget'.
2018-11-27T09:02:03.3744998Z GET https://api.nuget.org/v3/registration3-gz-semver2/python/index.json
2018-11-27T09:02:04.2195046Z OK https://api.nuget.org/v3/registration3-gz-semver2/python/index.json 844ms
2018-11-27T09:02:04.6414413Z Package "python.3.7.1" is already installed.
2018-11-27T09:02:04.9698862Z ##[error]Unable to process command '##vso[task.setvariable variable=pythonLocation;issecret=False]C:\vsts-agent\_work\_tool\PythonNuget\python.3.7.1\tools' successfully. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=817296)
2018-11-27T09:02:04.9723742Z ##[error]Value cannot be null.
Parameter name: RefName
2018-11-27T09:02:05.0583055Z ##[section]Finishing: Install Python
The install itself seems to have worked, because I can disable that step and run the package task with the variable manually set.
However this task also fails with a similar error:
2018-11-27T09:19:58.1789219Z ##[section]Starting: Build wheels from setup.py
2018-11-27T09:19:58.1797911Z ==============================================================================
2018-11-27T09:19:58.1798116Z Task : Build Python wheels
2018-11-27T09:19:58.1798220Z Description : Creates a wheel using an installed copy of Python.
2018-11-27T09:19:58.1798313Z Version : 1.0.6
2018-11-27T09:19:58.1798408Z Author : Steve Dower
2018-11-27T09:19:58.1798530Z Help : The *Build Python wheels* task uses a `setup.py` file to compile wheels. It will ensure that `pip`, `wheel`, `setuptools` and `cython` are installed before running.
2018-11-27T09:19:58.1798632Z ==============================================================================
2018-11-27T09:20:00.1574022Z ##[command]"C:\vsts-agent\_work\_tool\PythonNuget\python.3.7.1\tools\python.exe" -m pip install pip setuptools wheel cython
2018-11-27T09:20:09.3277504Z Requirement already satisfied: pip in c:\vsts-agent\_work\_tool\pythonnuget\python.3.7.1\tools\lib\site-packages (10.0.1)
2018-11-27T09:20:09.3647762Z Requirement already satisfied: setuptools in c:\vsts-agent\_work\_tool\pythonnuget\python.3.7.1\tools\lib\site-packages (39.0.1)
2018-11-27T09:20:09.4030940Z Collecting wheel
2018-11-27T09:20:10.1048438Z Downloading https://files.pythonhosted.org/packages/ff/47/1dfa4795e24fd6f93d5d58602dd716c3f101cfd5a77cd9acbe519b44a0a9/wheel-0.32.3-py2.py3-none-any.whl
2018-11-27T09:20:10.2095319Z Collecting cython
2018-11-27T09:20:10.8171412Z Downloading https://files.pythonhosted.org/packages/de/c8/daeef2f83ae402fc3c3eef4295e07fc46b267e5f12c7a8ee05b685de5c59/Cython-0.29.1-cp37-cp37m-win_amd64.whl (1.7MB)
2018-11-27T09:20:11.6943676Z Installing collected packages: wheel, cython
2018-11-27T09:20:11.8283445Z The script wheel.exe is installed in 'C:\vsts-agent\_work\_tool\PythonNuget\python.3.7.1\tools\Scripts' which is not on PATH.
2018-11-27T09:20:11.8283671Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2018-11-27T09:20:15.3438913Z The scripts cygdb.exe, cython.exe and cythonize.exe are installed in 'C:\vsts-agent\_work\_tool\PythonNuget\python.3.7.1\tools\Scripts' which is not on PATH.
2018-11-27T09:20:15.3439571Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2018-11-27T09:20:15.3891222Z Successfully installed cython-0.29.1 wheel-0.32.3
2018-11-27T09:20:15.8153804Z You are using pip version 10.0.1, however version 18.1 is available.
2018-11-27T09:20:15.8154881Z You should consider upgrading via the 'python -m pip install --upgrade pip' command.
2018-11-27T09:20:15.9164164Z ##[command]"C:\vsts-agent\_work\_tool\PythonNuget\python.3.7.1\tools\python.exe" "C:\vsts-agent\_work\1\s\setup.py" build --build-base "C:\vsts-agent\_work\1\b" bdist_wheel -d "C:\vsts-agent\_work\1\a" --universal
2018-11-27T09:20:17.2811564Z running build
2018-11-27T09:20:17.2812503Z running build_py
2018-11-27T09:20:17.7476566Z creating C:\vsts-agent\_work\1\b\lib
2018-11-27T09:20:17.7478546Z creating C:\vsts-agent\_work\1\b\lib\pvsdockerlib
2018-11-27T09:20:17.7487714Z copying pvsdockerlib\functions.py -> C:\vsts-agent\_work\1\b\lib\pvsdockerlib
2018-11-27T09:20:17.7509731Z copying pvsdockerlib\__init__.py -> C:\vsts-agent\_work\1\b\lib\pvsdockerlib
2018-11-27T09:20:17.7528017Z running bdist_wheel
2018-11-27T09:20:17.7948184Z installing to C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel
2018-11-27T09:20:17.7950366Z running install
2018-11-27T09:20:17.7960351Z running install_lib
2018-11-27T09:20:17.8287382Z creating C:\vsts-agent\_work\1\b\bdist.win-amd64
2018-11-27T09:20:17.8290061Z creating C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel
2018-11-27T09:20:17.8300030Z creating C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel\pvsdockerlib
2018-11-27T09:20:17.8310198Z copying C:\vsts-agent\_work\1\b\lib\pvsdockerlib\functions.py -> C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel\.\pvsdockerlib
2018-11-27T09:20:17.8333787Z copying C:\vsts-agent\_work\1\b\lib\pvsdockerlib\__init__.py -> C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel\.\pvsdockerlib
2018-11-27T09:20:17.8348679Z running install_egg_info
2018-11-27T09:20:18.0161526Z running egg_info
2018-11-27T09:20:18.0162158Z creating pvs_dockerlib.egg-info
2018-11-27T09:20:18.0169182Z writing pvs_dockerlib.egg-info\PKG-INFO
2018-11-27T09:20:18.0184157Z writing dependency_links to pvs_dockerlib.egg-info\dependency_links.txt
2018-11-27T09:20:18.0199803Z writing requirements to pvs_dockerlib.egg-info\requires.txt
2018-11-27T09:20:18.0208911Z writing top-level names to pvs_dockerlib.egg-info\top_level.txt
2018-11-27T09:20:18.0227310Z writing manifest file 'pvs_dockerlib.egg-info\SOURCES.txt'
2018-11-27T09:20:18.0262855Z reading manifest file 'pvs_dockerlib.egg-info\SOURCES.txt'
2018-11-27T09:20:18.0307698Z writing manifest file 'pvs_dockerlib.egg-info\SOURCES.txt'
2018-11-27T09:20:18.0320556Z Copying pvs_dockerlib.egg-info to C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel\.\pvs_dockerlib-0.1-py3.7.egg-info
2018-11-27T09:20:18.0449785Z running install_scripts
2018-11-27T09:20:18.2548207Z adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
2018-11-27T09:20:18.2597715Z creating C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel\pvs_dockerlib-0.1.dist-info\WHEEL
2018-11-27T09:20:18.2616849Z creating 'C:\vsts-agent\_work\1\a\pvs_dockerlib-0.1-py2.py3-none-any.whl' and adding 'C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel' to it
2018-11-27T09:20:18.2642603Z adding 'pvsdockerlib/__init__.py'
2018-11-27T09:20:18.2660569Z adding 'pvsdockerlib/functions.py'
2018-11-27T09:20:18.2668323Z adding 'pvs_dockerlib-0.1.dist-info/LICENSE'
2018-11-27T09:20:18.2674321Z adding 'pvs_dockerlib-0.1.dist-info/METADATA'
2018-11-27T09:20:18.2679798Z adding 'pvs_dockerlib-0.1.dist-info/WHEEL'
2018-11-27T09:20:18.2685103Z adding 'pvs_dockerlib-0.1.dist-info/top_level.txt'
2018-11-27T09:20:18.2688000Z adding 'pvs_dockerlib-0.1.dist-info/RECORD'
2018-11-27T09:20:18.2691808Z removing C:\vsts-agent\_work\1\b\bdist.win-amd64\wheel
2018-11-27T09:20:18.2719804Z C:\vsts-agent\_work\_tool\PythonNuget\python.3.7.1\tools\lib\distutils\dist.py:274: UserWarning: Unknown distribution option: 'long_description_content_type'
2018-11-27T09:20:18.2720555Z warnings.warn(msg)
2018-11-27T09:20:18.5079776Z ##[error]Unable to process command '##vso[task.setvariable variable=dist;issecret=False]C:\vsts-agent\_work\1\a' successfully. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=817296)
2018-11-27T09:20:18.5106520Z ##[error]Value cannot be null.
Parameter name: RefName
2018-11-27T09:20:18.5664259Z ##[section]Finishing: Build wheels from setup.py
How is this possible when Python build tools uses the built in powershell command Set-VstsTaskVariable?
Github Python build tools install set pythonLocation var
Github Python build tools package set dist var
Is this a bug in the azure-pipelines-agent? I'm using the latest pre-release v2.141.2, and on-prem TFS 2017 update 3.1
I have made a bug fix in a popular python package fastparquet https://github.com/dask/fastparquet and attempting to try it out on my machine. The package comes with a setup.py and so I have attempted to use that to install:
fastparquet $ python setup.py install
This completed and appears to have done the requisite installation of files to /usr/local/lib/python2.7/site-packages:
..
Adding python-dateutil 2.6.0 to easy-install.pth file
Using /usr/local/lib/python2.7/site-packages
Finished processing dependencies for fastparquet==0.1.0
However when we examine that location - specifically for the api.py file that was updated - it is still the original github version:
11:35:38/fastparquet $ll /usr/local/lib/python2.7/site-packages/fastparquet/api.py
-rw-r--r-- 1 sboesch admin 27294 Jun 19 17:18 /usr/local/lib/python2.7/site-packages/fastparquet/api.py
After having looked a bit into setup.py it seems that were a standard approach used across most python packages: https://docs.python.org/2/distutils/setupscript.html.
So then
am I misunderstanding how the install works /what it does?
Due to lack of support for some libraries I want to use, I moved some Python development from Windows to Linux development. I've spent most of the day messing about getting nowhere with dependencies.
The question
Whenever I pick up Linux, I usually run into some kind of dependency issue, usually with development libraries, whether they're installed via apt-get, easy_install or pip. I can waste days on what should be simple tasks, spending longer on getting libraries to work than writing code. Where can I learn about strategy for dealing with these kind of issues rather than aimlessly googling for someone who's come across the same problem before?
An example
Just one example: I wanted to generate some QR codes. So, I thought I'd use github.com/bitly/pyqrencode which is based on pyqrcode.sourceforge.net but supposedly without the Java dependencies. There are others (pyqrnative, github.com/Arachnid/pyqrencode) but that one seemed like the best bet for my needs.
So, I found the package on pypi and thought using that would make life easier:
(I've perhaps made life more difficult for myself by using virtualenv to keep things neat and tidy.)
(myenv3)mat#ubuntu:~/myenv3$ bin/pip install pyqrencode
Downloading/unpacking pyqrencode
Downloading pyqrencode-0.2.tar.gz
Running setup.py egg_info for package pyqrencode
Installing collected packages: pyqrencode
Running setup.py install for pyqrencode
building 'qrencode' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c qrencode.c -o build/temp.linux-i686-2.7/qrencode.o
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions build/temp.linux-i686-2.7/qrencode.o -lqrencode -o build/lib.linux-i686-2.7/qrencode.so
Successfully installed pyqrencode
Cleaning up...
(I guess I probably sudo apt-get install libqrencode-dev at some point prior to that too.)
So then I tried to run the test script:
(myenv3)mat#ubuntu:~/myenv3$ python test_qr.py
Traceback (most recent call last):
File "test_qr.py", line 1, in <module>
from qrencode import Encoder
File "qrencode.pyx", line 1, in init qrencode (qrencode.c:1520)
ImportError: No module named ImageOps
:(
Well, investigations revealed that ImageOps appears to be part of PIL...
(myenv3)mat#ubuntu:~/myenv3$ pip install pil
Downloading/unpacking pil
Downloading PIL-1.1.7.tar.gz (506Kb): 122Kb downloaded
Operation cancelled by user
Storing complete log in /home/mat/.pip/pip.log
(myenv3)mat#ubuntu:~/myenv3$ bin/pip install pil
Downloading/unpacking pil
Downloading PIL-1.1.7.tar.gz (506Kb): 506Kb downloaded
Running setup.py egg_info for package pil
WARNING: '' not a valid package name; please use only.-separated package names in setup.py
Installing collected packages: pil
Running setup.py install for pil
WARNING: '' not a valid package name; please use only.-separated package names in setup.py
building '_imaging' extension
gcc ...
building '_imagingmath' extension
gcc ...
--------------------------------------------------------------------
PIL 1.1.7 SETUP SUMMARY
--------------------------------------------------------------------
version 1.1.7
platform linux2 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24)
[GCC 4.5.2]
--------------------------------------------------------------------
*** TKINTER support not available
*** JPEG support not available
*** ZLIB (PNG/ZIP) support not available
*** FREETYPE2 support not available
*** LITTLECMS support not available
--------------------------------------------------------------------
To add a missing option, make sure you have the required
library, and set the corresponding ROOT variable in the
setup.py script.
To check the build, run the selftest.py script.
...
Successfully installed pil
Cleaning up...
Hmm, PIL's installed but hasn't picked up the libraries I installed with sudo apt-get install libjpeg62 libjpeg62-dev libpng12-dev zlib1g zlib1g-dev earlier. I'm not sure how to tell pip to feed the library locations to setup.py. Googling suggests a variety of ideas which I've tried, but none of them seem to help much other than to send me round in circles.
Ubuntu 11.04: Installing PIL into a virtualenv with PIP suggests using the pillow package instead, so let's try that:
(myenv3)mat#ubuntu:~/myenv3$ pip install pillow
Downloading/unpacking pillow
Downloading Pillow-1.7.5.zip (637Kb): 637Kb downloaded
Running setup.py egg_info for package pillow
...
Installing collected packages: pillow
Running setup.py install for pillow
building '_imaging' extension
gcc ...
--------------------------------------------------------------------
SETUP SUMMARY (Pillow 1.7.5 / PIL 1.1.7)
--------------------------------------------------------------------
version 1.7.5
platform linux2 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24)
[GCC 4.5.2]
--------------------------------------------------------------------
*** TKINTER support not available
--- JPEG support available
--- ZLIB (PNG/ZIP) support available
--- FREETYPE2 support available
*** LITTLECMS support not available
--------------------------------------------------------------------
To add a missing option, make sure you have the required
library, and set the corresponding ROOT variable in the
setup.py script.
To check the build, run the selftest.py script.
...
Successfully installed pillow
Cleaning up...
Well, we seem to have the JPEG and PNG support this time, yay!
(myenv3)mat#ubuntu:~/myenv3$ python test_qr.py
Traceback (most recent call last):
File "test_qr.py", line 1, in <module>
from qrencode import Encoder
File "qrencode.pyx", line 1, in init qrencode (qrencode.c:1520)
ImportError: No module named ImageOps
Still no ImageOps though. Now I'm stumped, is ImageOps missing from pillow, or is it a different problem that was also there with pil.
I see two separate problems here:
Keeping track of all the python modules you need for your project.
Keeping track of all the dynamic libraries you need for the python modules in your project.
For the first problem, I have found that buildout is good help, althought it takes a litle while to grasp.
In your case, I would start by creating a directory for my new project. I would then go into that directory and download bootstrap.py
wget http://python-distribute.org/bootstrap.py
I would then create a buildout.cfg file:
[buildout]
parts = qrproject
python
eggs = pyqrencode
[qrproject]
recipe = z3c.recipe.scripts
eggs = ${buildout:eggs}
entry-points= qrproject=qrprojectmodule:run
extra-paths = ${buildout:directory}
# This is a simple way of creating an interpreter that will have
# access to all the eggs / modules that this project uses.
[python]
recipe = z3c.recipe.scripts
interpreter = python
eggs = ${buildout:eggs}
extra-paths = ${buildout:directory}
In this buildout.cfg I'm referencing the module qrprojectmodule (in entry-points under [qrproject]. This will create a bin/qrproject that runs the function run in the module qrprojectmodule. So I will also create the file qrprojectmodule.py
import qrencode
def run():
print "Entry point for qrproject. Happily imports qrencode module"
Now it's time to run bootstrap.py with the python binary you want to use:
python bootstrap.py
Then run the generated bin/buildout
bin/buildout
This will create two additional binaries in the bin/ directory - bin/qrproject and bin/python. The former is your project's main binary. It will be created automatically each time you run buildout and will have all the modules and eggs you want loaded. The second is simply a convenient way to get a python prompt where all your modules and eggs are loaded, for easy debugging. The fine thing here is that bin/buildout will automatically install any python eggs that the eggs (in your case pyqrencode) have specified as dependencies.
Actually, you will probably get a compilation error in the step where you run bin/buildout. This is because you need to address problem 2: All dynamic libraries being available on your system. On Linux, it's usually best to get help from your packaging system. I'm going to assume you're using a Debian derivate such as Ubuntu here.
The pyqrencode web site specifies that you need the libqrencode library for pyqrencode to work. So I used my package manager to search for that:
$ apt-cache search libqrencode
libqrencode-dev - QR Code encoding library -- development
libqrencode3 - QR Code encoding library
qrencode - QR Code encoder into PNG image
In this case, I want the -dev package, as that installs linkable libraries and header files required to compile python C-modules. Also, the dependency system in the package manager will make sure that if I install libqrencode-dev, I will also get libqrencode3, as that is required at runtime, i.e. after compilation of the module.
So, I install the package:
sudo apt-get install libqrencode-dev
Once that has completed, rerun bin/buildout and the pyqrencode module will (hopefully) compile and install successfully. Now try to run bin/qrproject
$ bin/qrproject
Entry point for qrproject. Happily imports qrencode module
Success! :-)
So, in summary:
Use buildout to automatically download and install all the python modules/eggs you need for your project.
Use your system's package manager to install any dynamic (C) libraries required by the python modules you use.
Be aware that in many cases there are already packaged versions of your python modules available in the package system. For example, pil is available by installing the python-imaging package on Ubuntu. In this case, you don't need to install it via buildout, and you don't need to worry about libraries being available - the package manager will install all dependencies required for the module to run. Doing it via buildout can however make it easier to distribute your project and make it run on other systems.
Your story reminds me of my early experiences with Linux, and why I love APT.
There is no universal solution to your general problem; the best you can do is to take advantage of the work or others. The Debian packagers do a great job of flagging the dependencies of packages, so apt-get will pull in what you need. So, my strategy is simply to avoid building and installing stuff on my own, and use apt-get wherever possible.
Note that Ubuntu is based on Debian and thus gains the benefit of the work of the Debian packagers. I haven't used Fedora but I hear that the packages are not as well-organized as the ones from Debian.
I have a test setup file, which I made for a simple "hello world" script. I have a package named mytest which has a function hello. Now, I have a very simple setup.py. Everything works fine, if I just run python setup.py install. But if I want to install lib into home folder (python setup.py install --home=/home/blah), the package is not available anymore (running import mytest in python gives me ImportError: No module named mytest).
Should I add pth-file manually into site-packages folder? I tried it (with contents /home/blah/lib/python, where my package is put) and importing mytest worked fine. Shouldn't it be done automatically? Or have I missed something?
EDIT:
output of install:
ago#dellbert:~/py/mytest-0.1$ python setup.py install --home=/home/ago/py/
running install
running build
running build_py
copying src/mytest/mytest.py -> build/lib.linux-x86_64-2.6/mytest
running build_scripts
copying and adjusting src/main.py -> build/scripts-2.6
running install_lib
copying build/lib.linux-x86_64-2.6/mytest/mytest.py -> /home/ago/py//lib/python/mytest
byte-compiling /home/ago/py//lib/python/mytest/mytest.py to mytest.pyc
running install_scripts
copying build/scripts-2.6/main.py -> /home/ago/py//bin
changing mode of /home/ago/py//bin/main.py to 755
running install_egg_info
Removing /home/ago/py//lib/python/mytest-0.1.egg-info
Writing /home/ago/py//lib/python/mytest-0.1.egg-info
and setup.py:
from distutils.core import setup
setup(name='mytest',
description='test',
author='Ago',
author_email='email',
version='0.1',
package_dir={'mytest': 'src/mytest'},
packages=['mytest'],
scripts=['src/main.py']
)
Folder structure:
-src:
-mytest:
__init__.py
mytest.py
main.py
setup.py
main.py is just an executable which imports mytest and calls function to print hello world. But I have tried to just run import mytest in python to see, whether lib is installed.
It seems python unified at least the parameters in Unix- and Windows environments. Looking at the python reference today (https://docs.python.org/2/install/index.html, Dec. 2017), it shows that in both OS you can use the --prefix=<head installation path>. Have a look on the reference, section "Alternate installation: Unix (the prefix scheme)" and "Alternate installation: Windows (the prefix scheme)". I just tested it with Oct2Py (Octave-to-Python converter), which was a trouble to install with easy_install or pip, but did the job this way quite well.
Your python package will be then in (assuming you use Python 2.7) <head installation path>/lib/python2.7/site-packages or <head installation path>/lib/python2.7/dist-packages.
I don't know if this code will help, I didn't tested it.
If your path is /src/lib/:
python setup.py install --install-lib /src/lib/
python setup.py install --user --prefix=
or
import sys
sys.path.append('your-path')