Develop Reference

Python wrapper for C++ .so is only useable when COPY'ing code into docker image and not when mounted by volume?

I have somewhat successfully dockerized a software repository (KPConv) that I plan to work with and extend with the following Dockerfile
FROM tensorflow/tensorflow:1.12.0-devel-gpu-py3
# Install other required python stuff
RUN apt-get update && apt install -y --fix-missing --no-install-recommends\
python3-setuptools python3-pip python3-tk
RUN pip install --upgrade pip
RUN pip3 install numpy scikit-learn psutil matplotlib pyqt5 laspy
# Compile the custom operations and CPP wrappers
# For some reason this must be done within container, cannot access libcuda.so during docker build
# Ref: https://stackoverflow.com/questions/66575232
#COPY . /kpconv
#WORKDIR /kpconv/tf_custom_ops
#RUN sh compile_op.sh
#WORKDIR /kpconv/cpp_wrappers
#RUN sh compile_wrappers.sh
# Set the working directory to kpconv
WORKDIR /kpconv
# Set root user password so we can su/sudo later if need be
RUN echo "root:pass" | chpasswd
# Create a user and group akin to the host within the container
ARG USER_ID
ARG GROUP_ID
RUN addgroup --gid $GROUP_ID user
RUN adduser --disabled-password --gecos '' --uid $USER_ID --gid $GROUP_ID user
USER user
#Build
#sudo docker build -t kpconv-test \
# --build-arg USER_ID=$(id -u) \
# --build-arg GROUP_ID=$(id -g) \
# .
At the end of this Dockerfile I followed a post found here which describes a way to correctly set the permissions of files generated by/within a container so that the host machine/user can access them without having to alter the file permissions.
Also, this software repository makes use of custom tensorflow operations in C++ (KPConv/tf_custom_ops) along with Python wrappers for custom C++ code (KPConv/cpp_wrappers). The author of KPConv, Thomas Hugues, provides a bash script which compiles each to generate various .so files.
If I COPY the repository into the image during the build process (COPY . /kpconv), startup the container, call both of the compile bash scripts, and run the code then Python correctly loads the C++ wrapper (the generated .so grid_subsampling.cpython-35m-x86_64-linux-gnu.so) and begins running the software as expected/intended.
$ sudo docker run -it \
> -v /<myhostpath>/data_sets:/data \
> -v /<myhostpath>/_output:/output \
> --runtime=nvidia kpconv-test /bin/bash
user#eec8553dcb5d:/kpconv$ cd tf_custom_ops
user#eec8553dcb5d:/kpconv/tf_custom_ops$ sh compile_op.sh
user#eec8553dcb5d:/kpconv/tf_custom_ops$ cd ..
user#eec8553dcb5d:/kpconv$ cd cpp_wrappers/
user#eec8553dcb5d:/kpconv/cpp_wrappers$ sh compile_wrappers.sh
running build_ext
building 'grid_subsampling' extension
<Redacted for brevity>
user#eec8553dcb5d:/kpconv/cpp_wrappers$ cd ..
user#eec8553dcb5d:/kpconv$ python training_ModelNet40.py
Dataset Preparation
*******************
Loading training points
1620.2 MB loaded in 0.6s
Loading test points
411.6 MB loaded in 0.2s
<Redacted for brevity>
This works well and allows me run the KPConv software.
Also to note for later the .so file has the hash
user#eec8553dcb5d:/kpconv/cpp_wrappers/cpp_subsampling$ sha1sum grid_subsampling.cpython-35m-x86_64-linux-gnu.so
a17eef453f6d2370a15bc2a0e6714c978390c5c3 grid_subsampling.cpython-35m-x86_64-linux-gnu.so
It also has the permissions
user#eec8553dcb5d:/kpconv/cpp_wrappers/cpp_subsampling$ ls -al grid_subsampling.cpython-35m-x86_64-linux-gnu.so
-rwxr-xr-x 1 user user 561056 Mar 14 02:16 grid_subsampling.cpython-35m-x86_64-linux-gnu.so
Though it produces a difficult workflow for quickly editing and the software for my purposes and quickly running it within the container. Every change to the code requires a new build of the image. Thus, I would much rather mount/volume the KPConv code from the host into the container at runtime and then the edits are "live" within the container as it is running.
Doing this and using the Dockerfile at the top of the post (no COPY . /kpconv) to compile an image, perform the same compilation steps, and run the code
$ sudo docker run -it \
> -v /<myhostpath>/data_sets:/data \
> -v /<myhostpath>/KPConv_Tensorflow:/kpconv \
> -v /<myhostpath>/_output:/output \
> --runtime=nvidia kpconv-test /bin/bash
user#a82e2c1af21a:/kpconv$ cd tf_custom_ops/
user#a82e2c1af21a:/kpconv/tf_custom_ops$ sh compile_op.sh
user#a82e2c1af21a:/kpconv/tf_custom_ops$ cd ..
user#a82e2c1af21a:/kpconv$ cd cpp_wrappers/
user#a82e2c1af21a:/kpconv/cpp_wrappers$ sh compile_wrappers.sh
running build_ext
building 'grid_subsampling' extension
<Redacted for brevity>
user#a82e2c1af21a:/kpconv/cpp_wrappers$ cd ..
user#a82e2c1af21a:/kpconv$ python training_ModelNet40.py
I receive the following Python ImportError
user#a82e2c1af21a:/kpconv$ python training_ModelNet40.py
Traceback (most recent call last):
File "training_ModelNet40.py", line 36, in <module>
from datasets.ModelNet40 import ModelNet40Dataset
File "/kpconv/datasets/ModelNet40.py", line 40, in <module>
from datasets.common import Dataset
File "/kpconv/datasets/common.py", line 29, in <module>
import cpp_wrappers.cpp_subsampling.grid_subsampling as cpp_subsampling
ImportError: /kpconv/cpp_wrappers/cpp_subsampling/grid_subsampling.cpython-35m-x86_64-linux-gnu.so: failed to map segment from shared object
Why is this Python wrapper for C++ only useable when COPY'ing code into the docker image and not when mounted by volume?
This .so file has the same hash and permissions as the first described situation
user#a82e2c1af21a:/kpconv/cpp_wrappers/cpp_subsampling$ sha1sum grid_subsampling.cpython-35m-x86_64-linux-gnu.so
a17eef453f6d2370a15bc2a0e6714c978390c5c3 grid_subsampling.cpython-35m-x86_64-linux-gnu.so
user#a82e2c1af21a:/kpconv/cpp_wrappers/cpp_subsampling$ ls -al grid_subsampling.cpython-35m-x86_64-linux-gnu.so
-rwxr-xr-x 1 user user 561056 Mar 14 02:19 grid_subsampling.cpython-35m-x86_64-linux-gnu.so
On my host machine the file has the following permissions (it's on the host because /kpconv was mounted as a volume) (for some reason the container is in the future too, check the timestamps)
$ ls -al grid_subsampling.cpython-35m-x86_64-linux-gnu.so
-rwxr-xr-x 1 <myusername> <myusername> 561056 Mar 13 21:19 grid_subsampling.cpython-35m-x86_64-linux-gnu.so
After some research on the error message it looks like every result is specific to a situation. Though most seem to mention that the error is the result of some sort of permissions issue.
This Unix&Linux Stack answer I think provides the answer to what the actual problem is. But I am a bit too far from my days of working with C++ as an intern in college to necessarily understand how to use it to fix this issue. But I think the issue lies with the permissions between the container and host and between the users on each (that is, root on the container, user (Dockerfile) on the container, root on host, and <myusername> on host).
I have also attempted to first elevate permissions within the container using the root password created in the Dockerfile, then compiling the code, and running the software. But this results in the same issue. I have also tried compiling the code as user in the container, but running the software as root, again with the same issue.
Thus another clue I have found and provide is that there is seemingly something different with the .so when compiled "only within" the container (no --volume) and when it is compiled within the --volume (thus why I attempted to compare the file hashes). So maybe its not so much permissions but how the .so is loaded within the container by the kernel or how its location within the --volume effects that loading process?
EDIT: As for a SSCCE you should be able to clone the linked repository to your machine and use the same Dockerfile. You do not need to specify the /data or /output volumes or alter the code in any way (It attempts to load the .so before loading the data (which will just error and end execution))
If you do not have a GPU or do not want to install nvidia-runtime you should be able to alter the Dockerfile base image to tensorflow:1.12.0-devel-py3 and run the code on CPU.

Your problem is created by the linker trying to dynamically load the library. There could be several root-causes for this:
Permissions. The user should have permission to load the library, so when mounting file systems in docker, the owner id and the group id that are in the host are not necessary the same id in the container although they might be the same name.
Wrong binary format. The host OS is compiling the binary in wrong format. This can happen if you run the compile on (by example) macOS and use it in a linux container.
Wrong mounting. The mounting, by example, with noexec will also prevent the library to be loaded.
Difference in libraries from both environments. Due to the differences of the environment where the library was compiled, you might be missing some libraries, so use ldd grid_subsampling.cpython-35m-x86_64-linux-gnu.so and ldd -r -d -v grid_subsampling.cpython-35m-x86_64-linux-gnu.so check all the libraries that are linked.

How to create any AWS Lambda Python Layer? (Usage example with XGBoost)

I am having trouble creating a lambda layer for the xgboost library. Im running:
Im grabbing a zip of xgboost and it's dependencies from here (https://github.com/alexeybutyrev/aws_lambda_xgboost) and loading it into a layer. When I try to test my lambda, I get this error:
Unable to import module 'lambda_function': No module named 'xgboost.core'
It looks like __init__.py is trying to reference core.py via from .core import <stuff>
Has anyone encountered this error with AWS Lambda before?

EDIT: As #Marcin has remark, the first answer provided works for packages under 262 MB large.
A. Python Packages within Lambda Layer size limit
You can also do it with AWS sam cli and Docker (see this link to install the SAM cli), to build the packages inside a container. Basically you initialize a default template with Python as runtime and then you specify the packages under the requirements.txt file. I found it more easy than the article you mentioned. I let you steps if you want to consider them for future use.
1. Initialize a default SAM template
Under any folder that you want to keep the project, you can type
sam init
this will prompt a series of questions, for a quick set up we will be choosing the Quick Start Templates as follows
1 - AWS Quick Start Templates
2 - Python 3.8
Project name [sam-app]: your_project_name
1 - Hello World Example
By choosing the Hello World Example it generates a default lambda function with a requirements.txt file. Now, we're going to edit with the name of the package that you want, in this case xgboost
2. Specify packages to install
cd your_project_name
code hello_world/requirements.txt
as I have Visual Studio Code as editor, this will open the file on it. Now, I can specify the xgboost package
your_python_package
Here comes the reason to have Docker installed. Some packages relied on C++. Thus, it is recommended to build inside a container (case on Windows). Now, move to the folder where the template.yaml file is located. Then, type
sam build -u
3. Zip packages
there are some files that you do not want to be included in your lambda layer, because we only want to keep the python libraries. Thus, you could remove the following files
rm .aws-sam/build/HelloWorldFunction/app.py
rm .aws-sam/build/HelloWorldFunction/__init__.py
rm .aws-sam/build/HelloWorldFunction/requirements.txt
and then zip the remaining content of the folder.
cp -r .aws-sam/build/HelloWorldFunction/ python/
zip -r my_layer.zip python/
where we place the layer in the python/ folder according to the docs
On Windows system the zip command should be replaced with
Compress-Archive my_layer/ my_layer.zip.
4. Upload your Layer to AWS
On AWS go to Lambda, then choose Layers and Create Layer. Now, you can upload your .zip file as the image below shows
Notice that for zip files over 50 MB, you should upload the .zip file to an s3 bucket and provide the path, for exampl, https://s3:amazonaws.com//mybucket/my_layer.zip.
B. Python packages that exceeds Lambda Layer limits
The xgboost package on its own is more than 300 MB and will throw the following error
As #Marcin has kindly pointed out, the prior approach with SAM cli would not directly work for Python layers that exceed the limit. There's an open issue on github to specify a custom docker image when running sam build -u and a possible solution retagging the default lambda/lambci image.
So, how could we pass through this?. There are already some useful resources that I would just point to.
First, the Medium article that #Alex took as solution that follow this repo code.
Second, alexeybutyrev approach that works by applying the strip command to reduce the libraries sizes. One can find this approach under a github repo, the instructions are provided.
Edit (December 2020)
This month AWS releases container Image support for AWS Lambda. Following the next tree structure for your project
Project/
|-- app/
| |-- app.py
| |-- requirements.txt
| |-- xgb_trained.bin
|-- Dockerfile
You can deploy an XGBoost model with the following Docker image. Follow this repo instructions for a detailed explanation.
# Dockerfile based on https://docs.aws.amazon.com/lambda/latest/dg/images-create.html
# Define global args
ARG FUNCTION_DIR="/function"
ARG RUNTIME_VERSION="3.6"
# Choose buster image
FROM python:${RUNTIME_VERSION}-buster as base-image
# Install aws-lambda-cpp build dependencies
RUN apt-get update && \
apt-get install -y \
g++ \
make \
cmake \
unzip \
libcurl4-openssl-dev \
git
# Include global arg in this stage of the build
ARG FUNCTION_DIR
# Create function directory
RUN mkdir -p ${FUNCTION_DIR}
# Copy function code
COPY app/* ${FUNCTION_DIR}/
# Install python dependencies and runtime interface client
RUN python${RUNTIME_VERSION} -m pip install \
--target ${FUNCTION_DIR} \
--no-cache-dir \
awslambdaric \
-r ${FUNCTION_DIR}/requirements.txt
# Install xgboost from source
RUN git clone --recursive https://github.com/dmlc/xgboost
RUN cd xgboost; make -j4; cd python-package; python${RUNTIME_VERSION} setup.py install; cd;
# Multi-stage build: grab a fresh copy of the base image
FROM base-image
# Include global arg in this stage of the build
ARG FUNCTION_DIR
# Set working directory to function root directory
WORKDIR ${FUNCTION_DIR}
# Copy in the build image dependencies
COPY --from=base-image ${FUNCTION_DIR} ${FUNCTION_DIR}
ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]
CMD [ "app.handler" ]

So I was never able to figure out why it failed in this way. The solution I found that worked was to create an EC2 instance running amazon linux, install and zip the libraries there, then save to S3. See here for detailed instructions:
https://medium.com/#lucashenriquessilva/how-to-create-a-aws-lambda-python-layer-db2830e08b12

sphinx-build command not found in gitlab ci pipeline / python 3 alpine image

I want to specify a GitLab job that creates a sphinx html documentation.
I am using a Python 3 alpine image (cannot specify which exactly).
the build stage within my .gitlab-ci.yml looks like this:
pages:
stage: build
tags:
- buildtag
script:
- pip install -U sphinx
- sphinx-build -b html docs/ public/
only:
- master
however, the pipeline fails: sphinx-build: command not found. (same error for make html)
According to This Tutorial, my .gitlab-ci.yml should be more or less correct.
What am I doing wrong? Is this issue related to the alpine image I am using?

As #Yasen correctly noted, the path to sphinx-build was not contained in $PATH. However, adding command in before sphinx-build did not solve the problem for me.
Anyway I found the solution in the the runner logs: The output of pip install -U sphinx produced the following warning:
WARNING: The scripts sphinx-apidoc, sphinx-autogen, sphinx-build and sphinx-quickstart are installed in 'some/path' which is not on PATH.
so I added export PATH="some/path" to the script-step in the .gitlab-ci.yml:
script:
- pip install -U sphinx
- export PATH="some/path"
- sphinx-build -b html docs/ public/

Did the command pip install -U sphinx succeed? (You should be able to tell that from the CI job log.)
If so, you may need to specify the full path to sphinx-build, as Yasen said.
If it did not succeed, you should troubleshoot the installation of Sphinx.

Most likely the reason is that $PATH doesn't contain path to sphinx-build
TL;DR try to use command
Try this:
pages:
stage: build
tags:
- buildtag
script:
- pip install -U sphinx
- command sphinx-build -b html docs/ public/
only:
- master
Explanation
GitLab runners run different way
Since GitLab CI uses runners, runner's shell profile may differ from commonly used.
So, your runner may be configured without declared $PATH to the directory that contains sphinx-build
Zsh/Bash startup files loading order (.bashrc, .zshrc etc.)
See this explanation:
The issue is that Bash sources from a different file based on what kind of shell it thinks it is in. For an “interactive non-login shell”, it reads .bashrc, but for an “interactive login shell” it reads from the first of .bash_profile, .bash_login and .profile (only). There is no sane reason why this should be so; it’s just historical.
What command does mean?
Since we don't know the path where sphinx-build installed, you may use commands like: which, type, etc.
As per this great answer(shell - Why not use "which"? What to use then? - Unix & Linux Stack Exchange, author recommends to use command <name>, or $(command -v <name>)

How do I add more python modules to my yocto/openembedded project?

I wish to add more python modules to my yocto/openembedded project but I am unsure how to? I wish to add flask and its dependencies.

some python packages having corresponding recipes in the meta folders, like Enum class for example:
meta-openembedded/meta-python/recipes-devtools/python/python-enum34_1.1.6.bb
unfortunately lot's of useful classes aren't available, but some might be needed for the python application. get used of installing missing packages using pip already on booted platform? but what if the target product is not IP network connected? the solution is to implement a new recipe and add to the platform meta layer (at least). Example is a recipe for the module keyboard useful for intercepting keys/buttons touch events:
use PyPi web site to identify if the package is available:
https://pypi.org/project/keyboard/
download archive available on the package description page:
https://github.com/boppreh/keyboard/archive/master.zip
collect some useful information required to fill-out a new recipe:
SUMMARY - could be obtained from the package description page
HOMEPAGE - the project URL on github or bitbucket or sourceforge, etc
LICENSE - verify license type
LIC_FILES_CHKSUM by executing md5sum on existing LICENSE or README or PKG-INFO file located in the root of the package (preferrably)
SRC_URI[md5sum] - is md5sum of the archive itself. it will be used to discover and download archive on pypi server automatically with the help of supporting script inherit pypi
PYPI_PACKAGE_EXT - if the package is not tar.gz require to supply the correct extension
create missing python-keyboard_0.13.1.bb recipe:
`
SUMMARY = "Hook and simulate keyboard events on Windows and Linux"
HOMEPAGE = "https://github.com/boppreh/keyboard"
LICENSE = "BSD-3-Clause"
LIC_FILES_CHKSUM = "file://PKG-INFO;md5=9bc8ba91101e2f378a65d36f675c88b7"
SRC_URI[md5sum] = "d4b90e53bbde888e7b7a5a95fe580a30"
SRC_URI += "file://add_missing_CHANGES_md.patch"
PYPI_PACKAGE = "keyboard"
PYPI_PACKAGE_EXT = "zip"
inherit pypi
inherit setuptools
BBCLASSEXTEND = "native nativesdk"
`
the package has been patched by adding
SRC_URI += "file://add_missing_CHANGES_md.patch"
directive to the recipe due to missing CHANGES.md file used by setup.py script to identify package version (this step is optional). the patch itself has to be placed inside the folder next to the recipe matching recipe name but without version:
python-keyboard

This question is old, but currently in 2020 there is a python package called pipoe.
pipoe can generate .bb classes corresponding to python packages for you!
Usage:
$ pip3 install pipoe
$ pipoe -p requests
OR
$ pipoe -p requests --python python3
Now copy the generated .bb files to your layer and use them.
https://pypi.org/project/pipoe/

The OE layer index at layers.openembedded.org lists all known layers and the recipes they contain, so searching that should bring up the meta-python layer that you can add to your build and use recipes from.

In your Image recipe you can add a Python module by adding it to the IMAGE_INSTALL variable:
IMAGE_INSTALL += "python-numpy"
You can find possible modules for example by searching for them with wildcards:
find -name *python*numpy*bb
in the Yocto Folder brings:
./poky/meta/recipes-devtools/python/python-numpy_1.7.0.bb

The pipoe did not work for me either, I ended up making this bash script. Someone else might find it useful.
You will need to change this in my script below:
local my_layers_dir="my/layers/directory"
To run this script:
./pypi.sh <modulename>
#example:
./pypi.sh humanfriendly #this should generate the bb file for the humanfriendly python module
pypi.sh:
#!/bin/bash
set -ex
function argstovars()
{
for change in $#; do
set -- `echo $change | tr '=' ' '`
eval $1=$2
done
}
function main(){
local module=""
argstovars $#
local my_layers_dir="my/layers/directory"
local url_files="https://pypi.org/project/$module/#files"
mkdir -p /tmp/pypi
rm -fr /tmp/pypi/*
pushd /tmp/pypi
wget $url_files
local targz_url=$(cat index.html | grep https://files | grep tar.gz | sed -r "s/<a href=\"(.*)\">/\1/g")
wget $targz_url
local targz_file=$(ls | grep tar.gz)
local md5=$(md5sum $targz_file)
md5=${md5%% *}
local sha256=$(sha256sum $targz_file)
sha256=${sha256%% *}
tar -xf $targz_file
local module_with_version=$(echo "$targz_file" | sed -r "s/(.*)\.tar\.gz/\1/g")
pushd $module_with_version
local license_file=$(find . -name LICENSE*)
local md5lic=$(md5sum $license_file)
md5lic=${md5lic%% *}
popd
popd
module_with_version="${module_with_version//-/_}"; echo $foo
mkdir -p "$my_layers_dir/$module"
pushd "$my_layers_dir/$module"
echo "SUMMARY = \"This is a python module for $module\"
HOMEPAGE = \"https://pypi.org/project/$module/\"
LICENSE = \"MIT\"
LIC_FILES_CHKSUM = \"file://$license_file;md5=$md5lic\"
SRC_URI[md5sum] = \"$md5\"
SRC_URI[sha256sum] = \"$sha256\"
PYPI_PACKAGE = \"$module\"
inherit pypi setuptools3
RDEPENDS_${PN} += \" \
python3-psutil \
\"
" > "${module_with_version}.bb"
popd
}
time main module=$#

Python executable not finding libpython shared library

I am installing Python 2.7 on CentOS 5. I built and installed Python as follows
./configure --enable-shared --prefix=/usr/local
make
make install
When I try to run /usr/local/bin/python, I get this error message
/usr/local/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory
When I run ldd on /usr/local/bin/python, I get
ldd /usr/local/bin/python
libpython2.7.so.1.0 => not found
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000030e9a00000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000030e9200000)
libutil.so.1 => /lib64/libutil.so.1 (0x00000030fa200000)
libm.so.6 => /lib64/libm.so.6 (0x00000030e9600000)
libc.so.6 => /lib64/libc.so.6 (0x00000030e8e00000)
/lib64/ld-linux-x86-64.so.2 (0x00000030e8a00000)
How do I tell Python where to find libpython?

Try the following:
LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/python
Replace /usr/local/lib with the folder where you have installed libpython2.7.so.1.0 if it is not in /usr/local/lib.
If this works and you want to make the changes permanent, you have two options:
Add export LD_LIBRARY_PATH=/usr/local/lib to your .profile in your home directory (this works only if you are using a shell which loads this file when a new shell instance is started). This setting will affect your user only.
Add /usr/local/lib to /etc/ld.so.conf and run ldconfig. This is a system-wide setting of course.

Putting on my gravedigger hat...
The best way I've found to address this is at compile time. Since you're the one setting prefix anyway might as well tell the executable explicitly where to find its shared libraries. Unlike OpenSSL and other software packages, Python doesn't give you nice configure directives to handle alternate library paths (not everyone is root you know...) In the simplest case all you need is the following:
./configure --enable-shared \
--prefix=/usr/local \
LDFLAGS="-Wl,--rpath=/usr/local/lib"
Or if you prefer the non-linux version:
./configure --enable-shared \
--prefix=/usr/local \
LDFLAGS="-R/usr/local/lib"
The "rpath" flag tells python it has runtime libraries it needs in that particular path. You can take this idea further to handle dependencies installed to a different location than the standard system locations. For example, on my systems since I don't have root access and need to make almost completely self-contained Python installs, my configure line looks like this:
./configure --enable-shared \
--with-system-ffi \
--with-system-expat \
--enable-unicode=ucs4 \
--prefix=/apps/python-${PYTHON_VERSION} \
LDFLAGS="-L/apps/python-${PYTHON_VERSION}/extlib/lib -Wl,--rpath=/apps/python-${PYTHON_VERSION}/lib -Wl,--rpath=/apps/python-${PYTHON_VERSION}/extlib/lib" \
CPPFLAGS="-I/apps/python-${PYTHON_VERSION}/extlib/include"
In this case I am compiling the libraries that python uses (like ffi, readline, etc) into an extlib directory within the python directory tree itself. This way I can tar the python-${PYTHON_VERSION} directory and land it anywhere and it will "work" (provided you don't run into libc or libm conflicts). This also helps when trying to run multiple versions of Python on the same box, as you don't need to keep changing your LD_LIBRARY_PATH or worry about picking up the wrong version of the Python library.
Edit: Forgot to mention, the compile will complain if you don't set the PYTHONPATH environment variable to what you use as your prefix and fail to compile some modules, e.g., to extend the above example, set the PYTHONPATH to the prefix used in the above example with export PYTHONPATH=/apps/python-${PYTHON_VERSION}...

I had the same problem and I solved it this way:
If you know where libpython resides at, I supposed it would be /usr/local/lib/libpython2.7.so.1.0 in your case, you can just create a symbolic link to it:
sudo ln -s /usr/local/lib/libpython2.7.so.1.0 /usr/lib/libpython2.7.so.1.0
Then try running ldd again and see if it worked.

I installed Python 3.5 by Software Collections on CentOS 7 minimal. It all worked fine on its own, but I saw the shared library error mentioned in this question when I tried running a simple CGI script:
tail /var/log/httpd/error_log
AH01215: /opt/rh/rh-python35/root/usr/bin/python: error while loading shared libraries: libpython3.5m.so.rh-python35-1.0: cannot open shared object file: No such file or directory
I wanted a systemwide permanent solution that works for all users, so that excluded adding export statements to .profile or .bashrc files. There is a one-line solution, based on the Red Hat solutions page. Thanks for the comment that points it out:
echo 'source scl_source enable rh-python35' | sudo tee --append /etc/profile.d/python35.sh
After a restart, it's all good on the shell, but sometimes my web server still complains. There's another approach that always worked for both the shell and the server, and is more generic. I saw the solution here and then realized it's actually mentioned in one of the answers here as well! Anyway, on CentOS 7, these are the steps:
vim /etc/ld.so.conf
Which on my machine just had:
include ld.so.conf.d/*.conf
So I created a new file:
vim /etc/ld.so.conf.d/rh-python35.conf
And added:
/opt/rh/rh-python35/root/usr/lib64/
And to manually rebuild the cache:
sudo ldconfig
That's it, scripts work fine!
This was a temporary solution, which didn't work across reboots:
sudo ldconfig /opt/rh/rh-python35/root/usr/lib64/ -v
The -v (verbose) option was just to see what was going on. I saw that it did:
/opt/rh/rh-python35/root/usr/lib64:
libpython3.so.rh-python35 -> libpython3.so.rh-python35
libpython3.5m.so.rh-python35-1.0 -> libpython3.5m.so.rh-python35-1.0
This particular error went away. Incidentally, I had to chown the user to apache to get rid of a permission error after that.
Note that I used find to locate the directory for the library. You could also do:
sudo yum install mlocate
sudo updatedb
locate libpython3.5m.so.rh-python35-1.0
Which on my VM returns:
/opt/rh/rh-python35/root/usr/lib64/libpython3.5m.so.rh-python35-1.0
Which is the path I need to give to ldconfig, as shown above.

This worked for me...
$ sudo apt-get install python2.7-dev

On Solaris 11
Use LD_LIBRARY_PATH_64 to resolve symlink to python libs.
In my case for python3.6 LD_LIBRARY_PATH didn't work but LD_LIBRARY_PATH_64 did.
Hope this helps.
Regards

This answer would be helpful to those who have limited auth access on the server.
I had a similar problem for python3.5 in HostGator's shared hosting. Python3.5 had to be enabled every single damn time after login. Here are my 10 steps for resolution:
Enable the python through scl script python_enable_3.5 or scl enable rh-python35 bash.
Verify that it's enabled by executing python3.5 --version. This should give you your python version.
Execute which python3.5 to get its path. In my case, it was /opt/rh/rh-python35/root/usr/bin/python3.5. You can use this path get the version again (just to verify that this path is working for you.)
Awesome, now please exit out of current shell by scl.
Now, lets get the version again through this complete python3.5 path /opt/rh/rh-python35/root/usr/bin/python3.5 --version.
It won't give you the version but an error. In my case, it was
/opt/rh/rh-python35/root/usr/bin/python3.5: error while loading shared libraries: libpython3.5m.so.rh-python35-1.0: cannot open shared object file: No such file or directory
As mentioned in Tamas' answer, we gotta find that so file. locate doesn't work in shared hosting and you can't install that too.
Use the following command to find where that file is located:
find /opt/rh/rh-python35 -name "libpython3.5m.so.rh-python35-1.0"
Above command would print the complete path (second line) of the file once located. In my case, output was
find: `/opt/rh/rh-python35/root/root': Permission denied
/opt/rh/rh-python35/root/usr/lib64/libpython3.5m.so.rh-python35-1.0
Here is the complete command for the python3.5 to work in such shared hosting which would give the version,
LD_LIBRARY_PATH=/opt/rh/rh-python35/root/usr/lib64 /opt/rh/rh-python35/root/usr/bin/python3.5 --version
Finally, for shorthand, append the following alias in your ~/.bashrc
alias python351='LD_LIBRARY_PATH=/opt/rh/rh-python35/root/usr/lib64 /opt/rh/rh-python35/root/usr/bin/python3.5'
For verification, reload the .bashrc by source ~/.bashrc and execute python351 --version.
Well, there you go, now whenever you login again, you have got python351 to welcome you.
This is not just limited to python3.5, but can be helpful in case of other scl installed softwares.

I installed using the command:
./configure --prefix=/usr \
--enable-shared \
--with-system-expat \
--with-system-ffi \
--enable-unicode=ucs4 &&
make
Now, as the root user:
make install &&
chmod -v 755 /usr/lib/libpython2.7.so.1.0
Then I tried to execute python and got the error:
/usr/local/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory
Then, I logged out from root user and again tried to execute the Python and it worked successfully.

All it needs is the installation of libpython [3 or 2] dev files installation.

just install python-lib. (python27-lib). It will install libpython2.7.so1.0. We don't require to manually set anything.