Develop Reference

how to install thirdparty so file along with pybind11 package

I'm working on a python package which is based on pybind11 and build by cmake.
In this project a prebuild thirdparty library along with header files will be downloaded as a dependancy to pybind11 module.
cmake file is like this:
# unpackage.sh will download a tar file contains libvega.so and some so files libvega.so depends, and they will be unpacked to build dir
add_custom_target(
build-vega
COMMAND bash ${CMAKE_BINARY_DIR}/unpack.sh)
list(APPEND VEGA_LIBRARIES ${VEGA_INSTALL_DIR}/lib/libvega.so)
pybind11_add_module(${CMAKE_PROJECT_NAME})
add_dependencies(${CMAKE_PROJECT_NAME} build-vega)
list(APPEND SRC_FILES vegapy.cpp)
target_sources(${CMAKE_PROJECT_NAME} PRIVATE ${SRC_FILES})
target_link_libraries(${CMAKE_PROJECT_NAME} PRIVATE ${VEGA_LIBRARIES})
the setup file is based on cmake-example, only changes some names.
when install package by pip install ., vegapy.cpython-38-x86_64-linux-gnu.so will be installed to python dist-packages, but it depends on the so file inside project build dir.
root#ubuntu:/usr/local/lib/python3.8/dist-packages# ldd vegapy.cpython-38-x86_64-linux-gnu.so
libvega.so => /data/jgq/pv/build/temp.linux-x86_64-3.8/vegapy/vega/lib/libvega.so (0x00007f65a00cc000)
libexpreval.so => not found # depended by libvega.so
libmanage.so => not found # depended by libvega.so
in this way, when vegapy is imported in python, it will work, but if /data/jgq/pv/build/temp.linux-x86_64-3.8/vegapy/vega/lib is deleted, import will fail for libvega.so can not be found even I copy those so files into dist-packages.
I expect that:
libvega.so and its dependancies will be installed along with vegapy.cpython-38-x86_64-linux-gnu.so into a same dir.
vegapy should link to installed so files in dist-packages when python import vegapy package

ta-lib replit python install problem, ERROR: No matching distribution found for talib-binary

I use it on my windows machine by downloading its binary. I also use it in Heroku from its herokus build pack. I don't know what operating system replit use. But I try every possible commed like.
!pip install ta-lib
!pip install talib-binary
It's not working with replit. I thought it work like google co-lab but its not the same.
can anyone use TA-LIB with replit. if so. How you install it?

Getting TA-Lib work on Replit
(by installing it from sources)
Create a new replit with Nix toolset with a Python template.
In main.py write:
import talib
print (talib.__ta_version__)
This will be our test case. If ta-lib is installed the python main.py (executed in Shell) will return something like:
$ python main.py
b'0.6.0-dev (Jan 1 1980 00:00:00)'
We need to prepare a tools for building TA-Lib sources. There is a replit.nix file in your project's root folder (in my case it was ~/BrownDutifulLinux). Every time you execute a command like cmake the Nix reports that:
cmake: command not installed. Multiple versions of this command were found in Nix.
Select one to run (or press Ctrl-C to cancel):
cmake.out
cmakeCurses.out
cmakeWithGui.out
cmakeMinimal.out
cmake_2_8.out
If you select cmake.out it will add a record about it into the replit.nix file. And next time you call cmake, it will know which cmake version to launch. Perhaps you may manually edit replit.nix file... But if you're going to add such commands in a my way, note that you must execute them in Shell in your project root folder as replit.nix file is located in it. Otherwise Nix won't remember your choice.
After all my replit.nix file (you may see its content with cat replit.nix) content was:
{ pkgs }: {
deps = [
pkgs.libtool
pkgs.automake
pkgs.autoconf
pkgs.cmake
pkgs.python38Full
];
env = {
PYTHON_LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath [
# Needed for pandas / numpy
pkgs.stdenv.cc.cc.lib
pkgs.zlib
# Needed for pygame
pkgs.glib
# Needed for matplotlib
pkgs.xorg.libX11
];
PYTHONBIN = "${pkgs.python38Full}/bin/python3.8";
LANG = "en_US.UTF-8";
};
}
Which means I executed libtool, autoconf, automake and cmake in Shell. I always choose a generic suggestion from Nix, without a specific version. Note: some commands may report errors as we executing them in a wrong way just to add to a replit.nix.
3.
Once build tools are set up we need to get and build TA-Lib C library sources. To do that execute in Shell:
git clone https://github.com/TA-Lib/ta-lib.git
then
cd ta-lib/
libtoolize
autoreconf --install
./configure
If configure script is completed without any problems, build the library with:
make -j4
It will end up with some compilation errors, but they are related to some additional tools which are used to add new TA-Lib indicators and build at the end, but not the library itself. The library will be successfully compiled and you should be able to see it with:
$ ls ./src/.libs/
libta_lib.a libta_lib.lai libta_lib.so.0
libta_lib.la libta_lib.so libta_lib.so.0.0.0
Now we have our C library built, but we can't install it to a system default folders. So we have to use the library as is from the folders where it was build. All we need is just one more additional preparation:
mkdir ./include/ta-lib
cp ./include/*.h ./include/ta-lib/
This will copy a library headers to a subfolder, as they are designed to be used from a such subfolder (which they don't have due to impossibility to perform the installation step).
4.
Now we have TA-Lib C library built and prepared to be used locally from its build folders. All we need after that - is to compile the Python wrapper for it. But Python wrapper will look for a library only in system default folders, so we need to instruct it where our library is.
To do this, execute pwd and remember the absolute path to your project's root folder. In my case it was:
/home/runner/FormalPleasedOffice
Then adjust the paths (there are two) in a following command to lead to your project path:
TA_INCLUDE_PATH=/home/runner/FormalPleasedOffice/ta-lib/include/ TA_LIBRARY_PATH=/home/runner/FormalPleasedOffice/ta-lib/src/.libs/ pip install ta-lib
This is one line command, not a two commands.If the paths would be shorter it would look like:
TA_INCLUDE_PATH=/path1/ TA_LIBRARY_PATH=/path2/ pip install ta-lib.
After execution of this command the wrapper will be installed with two additional paths where it will look for a library and its header files.
That's actually all.
An alternative way would be to clone the wrapper sources, edit its setup.py and install wrapper manually. Just for the record this would be:
cd ~/Your_project
git clone https://github.com/mrjbq7/ta-lib.git ta-lib-wrapper
cd ta-lib-wrapper
Here edit the setup.py. Find the lines include_dirs = [ and library_dirs = [ and append your paths to these lists. Then you just need to:
python setup.py build
pip install .
Note the dot at the end.
5.
Go to the project's folder and try our python script:
$python main.py
b'0.6.0-dev (Jan 1 1980 00:00:00)'
Bingo!

The #truf answer is correct.
after you add the
pkgs.libtool
pkgs.automake
pkgs.autoconf
pkgs.cmake
in the replit.nix dippendancies.
git clone https://github.com/TA-Lib/ta-lib.git
cd ta-lib/
libtoolize
autoreconf --install
./configure
make -j4
mkdir ./include/ta-lib
cp ./include/*.h ./include/ta-lib/
TA_INCLUDE_PATH=/home/runner/FormalPleasedOffice/ta-lib/include/ TA_LIBRARY_PATH=/home/runner/FormalPleasedOffice/ta-lib/src/.libs/ pip install ta-lib
Note : FormalPleasedOffice should be your project name
Done.
Here is the youtube video :
https://www.youtube.com/watch?v=u20y-nUMo5I

How can I generate deb installable file for my ROS package(python code in the script folder)?

How can I generate deb installable file from the python files in my package(in the script folder), so I can deliver the deb file to the client who can run the python executables in the package?
For example, I just completed the ROS tutorial and created the beginner_tutorials package in my catkin workspace. I also completed the Simple Publisher and Subscriber tutorials which mean the python talker.py and listener.py in the script folder in the beginner_tutorials package.Then I run the following command in the package folder,~/ROS/catkin_workspace/src/beginner_tutorials:
$ bloom-generate rosdebian --os-name ubuntu --ros-distro kinetic
$ fakeroot debian/rules binary
So it generated the deb file. After I installed the deb to a new machine and I try to run the command:
$ rosrun beginner_tutorials talker
$ rosrun beginner_tutorials talker.py
$ rosrun beginner_tutorials listener
$ rosrun beginner_tutorials listener.py
After the installation on the new machine. I can't run the talker and listener. There is no python file in the /opt/ros/kinetic/lib. but look inside of the deb file, there are python files in the /opt/ros/kinetic/lib folder. I don't know where the python file went after install the deb file. I was wondering how can I export the package with python script to a installable deb file and after I install the deb file on a new machine, I can run the python executable file on the new machine?
If someone can give me some hints or guidance, I would really appreciate it. Thank you in advance!
Jue Wang(Patrick)

You may just use CPack with following options:
(Package-wide CMakeLists.txt file)
#############
## Install ##
#############
set(CPACK_GENERATOR "DEB")
#Creating UNIX structured package
set(CPACK_CMAKE_GENERATOR "Unix Makefiles")
set(CPACK_PACKAGE_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/../..")
set(CPACK_PACKAGE_VERSION 0.2.9)
#Set directory to /../ to avoid /share part of ROS_PACKAGE_PATH
set(CPACK_PACKAGING_INSTALL_PREFIX "$ENV{ROS_PACKAGE_PATH}/../")
set(CPACK_INSTALL_CMAKE_PROJECTS "${CMAKE_CURRENT_BINARY_DIR};${PROJECT_NAME};ALL;/./")
set(CPACK_PACKAGE_DESCRIPTION_SUMMARY
"${PROJECT_NAME} - ${PROJECT_DESCRIPTION}")
set(CPACK_PACKAGE_NAME "${PROJECT_NAME}")
set(CPACK_PACKAGE_VENDOR "RCS")
#Now we can simply create the dependencies using almost the same package names as they declared in ROS repository
set(CPACK_DEBIAN_PACKAGE_DEPENDS "ros-kinetic-roscpp, ros-kinetic-std-msgs, ros-kinetic-message-runtime, ros-kinetic-sensor-msgs")
#IMPORTANT: set this to keep on auto-introspection for system dependencies like libc++6
set (CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON)
#Keep Debian package structure untouched
set(CPACK_DEBIAN_PACKAGE_CONTROL_STRICT_PERMISSION TRUE)
#Standard naming schema for Debian-based distros
set(CPACK_PACKAGE_FILE_NAME "${CPACK_PACKAGE_NAME}-${CPACK_PACKAGE_VERSION}~${CMAKE_SYSTEM_NAME}_${CMAKE_SYSTEM_PROCESSOR}")
set(CPACK_PACKAGE_VENDOR "Technical University")
set(CPACK_PACKAGE_CONTACT "<your#email.com>")
#Turn the process on
include(CPack)
In case of simple catkin package structure you can simply uncomment and edit install sections of your CMakeLists.txt according to files your package required to install. Then run
catkin_make package
After that you will have the .deb package archive in the root directory of your ROS package. Now you can use dpkg-deb -c command to ensure the paths defined within the package are correct. To obtain the example download and examine any ROS package from the official binaries repository.

PyInstaller UAC not working in onefile mode

Hello everybody.
I have a small python project and want to make it to single executable file. I am using...
Windows 7
Python 3.4
PyInstaller 3.2.1
My PyInstaller command is,
PyInstaller -y -w -F -n output_file_name --uac-admin --clean source_file.py
this command works properly. But single output file does not ask for admin rights when executed. And there's no shield mark on executable file icon.
When remove the -F option (equivalent to --onefile), output executable file has shield mark on its icon and ask me admin rights. But this is not what I want. I want a single executable file.
I found a manifest file (output_file_name.exe.manifest) in dist\output_file_name folder. So I did...
PyInstaller -y -w -F -n output_file_name --manifest output_file_name.exe.manifest --uac-admin --clean source_file.py
But this command doesn't work. Its single executable file still does not ask for admin rights.
I have removed PyInstaller and installed recent development version.
pip install git+https://github.com/pyinstaller/pyinstaller.git#develop
But the result is same. Its output doesn't have a shield mark on icon and does not ask admin rights.
Do you have any idea?
Please help me!

I found what's wrong!
The key point is...
Install PyInstaller 3.0
Manifest file must be located in dist folder where single excutable file located
The name of manifest file must be same as output file.
If manifest file is located in dist folder, no need to specify --manifest option. --uac-admin is enough.
You can find manifest file at build folder.
Thank you.

Adding up passion053's answer, in my case I used -F instead of --onefile and it works fine for me but yeah you need to add manifest file in the same directory as your single executable.
Note: I used pyinstaller version 3.5. and it works fine for my
Happy Coding!

Adding -r prog.exe.manifest,1 to pyinstaller command line worked for me, after that no need for putting manifest file near exe, pure single exe file.

Add a Python module to a Google App Engine project

I'm trying to use the feed-parser module for this project im working on. When I upload the files to App Engine and I run the script it comes back with the error that the there is no module named feed-parser.
So I'm wondering if and how can I install this module on App Engine, so that I can fix this error and use RSS.
Error:
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/base/data/home/apps/s~vlis-mannipulus-bot/1.391465315184045822/main.py", line 7, in <module>
import feedparser
ImportError: No module named feed parser
Development 1:
So I've tried installing the module in the lib directory i created(in this fail example i forgot the /lib at the --prefix=..). And i get PYTHONERROR as is shown in the shell. Ive done some research on python paths and the solutions i tried didn't work for me.
kevins-MacBook-Pro-2:~ KevinH$ cd /Users/KevinH/Downloads/feedparser -5.2.1
kevins-MacBook-Pro-2:feedparser-5.2.1 KevinH$ sudo python setup.py install --prefix=/Users/KevinH/Documents/Thalia\ VMbot/Thalia-VMbot/
Password:
running install
Checking .pth file support in /Users/KevinH/Documents/Thalia VMbot/Thalia-VMbot//lib/python2.7/site-packages/
/usr/bin/python -E -c pass
TEST FAILED: /Users/KevinH/Documents/Thalia VMbot/Thalia- VMbot//lib/python2.7/site-packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from. The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/Users/KevinH/Documents/Thalia VMbot/Thalia-VMbot//lib/python2.7/site- packages/
and your PYTHONPATH environment variable currently contains:
''
Here are some of your options for correcting the problem:
* You can choose a different installation directory, i.e., one that is
on PYTHONPATH or supports .pth files
* You can add the installation directory to the PYTHONPATH environment
variable. (It must then also be on PYTHONPATH whenever you run
Python and want to use the package(s) you are installing.)
* You can set up the installation directory to support ".pth" files by
using one of the approaches described here:
https://pythonhosted.org/setuptools/easy_install.html#custom- installation-locations
Please make the appropriate changes for your system and try again.
Then i tried with the "pip" command but then i get this:
can't open file 'pip': [Errno 2] No such file or directory
According to what I have read "pip" should be a default program installed with python 2.7 and up. So to be sure i did install python3.5 and ran it with that and still get the same error. I typed this with both pythons:
kevins-MacBook-Pro-2:feedparser KevinH$ python3 pip -m install feedparse
--
Not sure if this would work, but via terminal i went to the default directory where feed parser has been installed on my system and copied it to the lib directory i made. Then I've created the config file with the following:
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
Deployed it and im still getting the same error as above no module named feeedparser.
Apologies if im doing something stupidly wrong, im still in the learning process.

The App Engine documentation that explains how to add third party modules is here
In summary, you will need to add a folder, usually named 'lib', to the top level off your app, and then install feedparser into that folder using the commands described in the documentation. You will also need to create an appengine_config.py file as descibed in the documentation.
Note that not all third party packages can be uploaded to App Engine - those with C extensions are forbidden. Feedparser looks OK to me though.
EDIT: further comments based on edit "development1" to the question.
Your appengine_config.py looks good.
You "lib" folder should be your application folder, that is the same folder as your app.yaml and appengine_config.py files.
You need to install the feedparser package into the lib folder. The Google docs recommend that you do this by running the command
pip install -t lib feedparser
This command will install the feedparser package into your lib folder.
You need to install and run a version of pip that works with Python2.x - the Python3 version will create a version of feedparser that only runs under Python3.
If you can't install a pip for Python2 this question might provide the right solution, otherwise I'd suggest you ask a separate question about how to install feedparser into a custom install directory on a Mac.I don't have a Mac so I can't help with this.
Once you have feedparser installed in your lib folder, verify that your app works locally; in particular verify that it's using your lib/feedparser installation: try logging feedparser.__file__ after importing feedparser to check the location of the file being imported.
Once everything is working locally you should be able to upload to GAE.
So in summary, your appengine_config.py looks good, but you need to make sure that the right version of feedparser is installed into the lib folder in your app folder before uploading to App Engine.