Develop Reference

Including a C library inside a Python application

I've looked around (including StackOverflow) but can't really find something similar to my case so I'll just ask it here. I'm building an application with Python (it is currently a source code that I'm planning to turn into a Windows executable) and is using this library: https://github.com/xiph/rnnoise. It is not a Python package but a third-party library written primarily in C.
Since I have to install it into my computer in order to use the library inside my code, I want to ask if there is a way to include this library along with the executable so that anyone can just download and use it without having to set up the library? This is important since I can only set up the library on a Linux machine and the executable needs to work with the library platform-independently.
Update: Thanks to Vimalan E and Marat, I had a bit more clue on what I need to do. I was managed to locate the .so files of my required library after running make install on it. The question left is I don't know how to link it with an executable that will be made from my Python source code. For now, I want to include the library as a binary file along with my application (though I am not sure how to achieve that, maybe putting putting the .so near the .exe should do the trick).
Thank you.

How to use c++ and python together and distribute them with systems not having python installed?

I was working around combining c++ and python together and found that there is something called <include Python.h> which can be very helpful and calling functions from a python file or simply communicating with any python script.
I wondered how we will be able to distribute these binding i.e. a program of c++ communicating with a python script to systems not having Python installed or having unmatched versions of python installed.
Also I found that somehow Sublime Text packages Python runtime, libraries and python scripts to distribute to machines where python may or may not be installed. I wanted to know how is this done and what are the reference links which might be useful in knowing more on the topic.
Sublime Text Root Directory Image containing distributable python runtime and python scripts
Similar kind of implementation was also seen with Google Backup and Sync Software.

Does compiling and linking Python in a C++ program mean that target users won't need python installed?

I have a C++ application that uses machine learning from Python and my current approach is making a single file executable with pyInstaller and then just running it from C++. This have obvious drawbacks, notably interapplication communication. At the moment I'm using an intermediate JSON file to talk to each other but this is massively suboptimal for my future requirements. What's beautiful about this, is that is working on all major platforms without too much hassle.
Section 1.6. from Python's manual reads "Compiling and Linking under Unix-like systems"
Does this mean that Python interpreter will be inside my application binary and target system doesn't need to have Python installed as the program will always use embedded interpreter ? If so, whats with python libraries ? Can I embed a whole conda enviroment ?
Also, whats with:
"(...) under Unix-like systems"
Does this means this approach is not multiplatform ?
Thanks in advance.

Embedding the Python interpreter is possible on all platforms. However it will only be the interpreter. Embedding any libraries will be a lot harder or even impossible.
But since you seem to deploy the Python libs already, you can use them just fine from the embedded interpreter. And then you could bridge C++ and Python without IPC, since they are both running in the same process.
pybind11 is very nice for embedding and generating C++ <-> Python interfaces.
A possible alternative, depending on the libraries in use, may be to export the model and use a C++ library to load and use it (for instance Tensorflow -> ONNX -> ONNX runtime).

It means that cpython (Python interpreter) will be inside your application. You will be able to run Python code and observe and manipulate virtual machine state directly from C++ code (good entry point C API reference is here). Your application might have some additional dynamic library dependencies (which ones depends on compilation options of embedded Python). Also interpreter isn't completely self contained and depends on some external .py modules normally shipped with Python distribution (as standard library). If you plan to import external modules that expect standard library, you will have to ship it with your application. There are ways how to build modules into binary too (freeze) but you might run into issues specially with modules that rely on filesystem.
As far as I tried, this procedure works on UNIX like systems and Windows (where easiest way is to link against DLL which you then ship with your application). On Windows you also need to make sure that you compile with same compiler as was used to compile DLL (or you compile Python DLL from source). Here is additional information about embedding on Windows: https://docs.python.org/3/faq/windows.html#how-can-i-embed-python-into-a-windows-application
Just note that embedding Python and shipping 3rd party modules with your application might have some licensing consequences.

Embedding Python on Windows: why does it have to be a DLL?

I'm trying to write a software plug-in that embeds Python. On Windows the plug-in is technically a DLL (this may be relevant). The Python Windows FAQ says:
1.Do not build Python into your .exe file directly. On Windows, Python must be a DLL to handle importing modules that are themselves DLL’s. (This is the first key undocumented fact.) Instead, link to pythonNN.dll; it is typically installed in C:\Windows\System. NN is the Python version, a number such as “23” for Python 2.3.
My question is why exactly Python must be a DLL? If, as in my case, the host application is not an .exe, but also a DLL, could I build Python into it? Or, perhaps, this note means that third-party C extensions rely on pythonN.N.dll to be present and other DLL won't do? Assuming that I'd really want to have a single DLL, what should I do?
I see there's the dynload_win.c file, which appears to be the module to import C extensions on Windows and, as far as I can see, it scans the extension file to find which pythonX.X.dll it imports; but I'm not experienced with Windows and I don't quite understand all the code there.

You need to link to pythonXY.dll as a DLL, instead of linking the relevant code directly into your executable, because otherwise the Python runtime can't load other DLLs (the extension modules it relies on.) If you make your own DLL you could theoretically link all the Python code in that DLL directly, since it doesn't end up in the executable but still in a DLL. You'll have to take care to do the linking correctly, however, as pretty much none of the standard tools (like distutils) will do this for you.
However, regardless of how you embed Python, you can't make do with just the DLL, nor can you make do with just any DLL. The ABI changes between Python versions, so if you compiled your code against Python 2.6, you need python26.dll; you can't use python25.dll or python27.dll. And Python isn't just a DLL; it also needs its standard library, which includes extension modules (which are DLLs themselves, although they have the .pyd extension.) The code in dynload_win.c you ran into is for loading those DLLs, and are not related to loading of pythonXY.dll.
In short, in order to embed Python in your plugin, you need to either ship Python with the plugin, or require that the right Python version is already installed.

(Sorry, I did a stupid thing, I first wrote the question, and then registered, and now I cannot alter it or comment on the replies, because StackOverflow's engine doesn't think I'm the author. I cannot even properly thank those who replied :( So this is actually an update to the question and comments.)
Thanks for all the advice, it's very valuable. As far as I understand with some effort I can link Python statically into a custom DLL, provided that I compile other dynamically loaded extensions myself and link them against the same DLL. (I know I need to ship the standard library too; my plan was to append a zipped archive to the DLL file. As far as I understand, I will even be able to import pure Python modules from it.)
I also found an interesting place in dynload_win.c. (I understand it loads dynamic extensions that use Python C API, e.g. _ctypes.) As far as I can see it not only looks for init_ctypes symbol or whatever the extension name is, but also scans the .pyd file's import table looking for (regex) python\d+\. and then compares the found symbol with known pythonNN. string to make sure the extension was compiled for this version of Python. If the import table doesn't have such a symbol or it refers to another version, it raises an error.
For me it means that:
If I link an extension against pythonNN.dll and try to load it from my custom DLL that includes a statically linked Python, it will pass the check, but — well, here I'm not sure: will it fail because there's no pythonNN.dll (i.e. even before getting to the check) or it will happily load the symbols?
And if I link it against my custom DLL, it will find symbols, but won't pass the check :)
I guess I could rewrite this piece to suit my needs... Are there any other such places, I wonder.

Python needs to be a dll (with a standard name) such that your application, and the plugin, can use the same instance of python.
Plugin dlls are already going to expect to be loading (and using python from) a python26.dll (or whichever version) - if your python is statically embedded in your exe, then two different instances of the python library would be managing the same data structures.
If the python libraries use no static variables at all, and the compile settings are exactly the same this should not be a problem. However, generally its far safer to simply ensure that only one instance of the python interpreter is being used.

On *nix, all shared objects in a process, including the executable, contribute their exported names into a common pool; any of the shared objects can then pull any of the names from the pool and use them as they like. This allows e.g. cStringIO.so to pull the relevant Python library functions from the main executable when the Python library is statically-linked.
On Windows, each shared object has its own independent pool of names it can use. This means that it must read the relevant different shared objects it needs functions from. Since it is a lot of work to get all the names from the main executable, the Python functions are separated out into their own DLL.

How does Dropbox use Python on Windows and OS X?

In Windows the Dropbox client uses python25.dll and the MS C runtime libraries (msvcp71.dll, etc). On OS X the Python code is compiled bytecode (pyc).
My guess is they are using a common library they have written then just have to use different hooks for the different platforms.
What method of development is this? It clearly isn't IronPython or PyObjC. This paradigm is so appealing to me, but my CS foo and Google foo are failing me.

Dropbox uses a combination of wxPython and PyObjC on the Mac (less wxPython in the 0.8 series). It looks like they've built a bit of a UI abstraction layer but nothing overwhelming—i.e., they're doing their cross-platform app the right way.
They include their own Python mainly because the versions of Python included on the Mac vary by OS version (and Dropbox supports back to 10.4 IIRC); also, they've customized the Python interpreter a bit to improve threading and I/O behavior.
(I do not work for Dropbox or have any inside knowledge; all I did was read their forums and examine the filenames in site-packages.zip in the Dropbox app bundle.)

For WINDOWS, Dropbox have employed a module similar to py2exe to package all their .py scripts, required libraries, resources etc into the distribution that you have mentioned above (.exe, library.zip, MS C runtime library and python25.dll) so that they can be run without requiring Python installation. Here's a sample code of how you can achieve this with py2exe.
from distutils.core import setup
import py2exe
options = {'py2exe': {
'compressed':1,
'bundle_files': 2,
'dll_excludes': ['w9xpopen.exe']
}}
setup(console=['myapp.py'],options=options)
Please see the tutorial here for more explanation.
PS: the number of files in the distribution can be controlled using the options parameter as shown in the above example.

These guys reverse engineered Dropbox client code
http://www.openwall.com/presentations/WOOT13-Security-Analysis-of-Dropbox/
https://github.com/kholia/dedrop

Indeed they do bundle their own Python 2.5.4 interpreter found at /Applications/Dropbox.app/Contents/MacOS/python. Poking around in /Applications/Dropbox.app/Contents/Resources/lib/python2.5/lib-dynload it looks to be bundled by PyObjC.
I'm no authority on this, but it seems it is exactly as you suggest in the OP:
My guess is they are using a common
library they have written then just
have to use different hooks for the
different platforms

Python25.dll is probably not their application code, it is a dll containing a copy of the python interpreter which can be called from within a windows application. Those pyc files are probably there in some form on windows, but they might be in an archive or obfuscated.
Python is included in OS/X, so it would be possible for them to execute those pyc file without shipping a python, but would not be surprised if they have there own python version lurking in the app bundle.
I don't know how dropbox builds there distributions, but there are several tools to bundle python apps into executable packages. Take a look at py2exe, py2app, and or cx_freeze.

Recently I published an article on reversing the dropbox client on windows. It is available on slideshare.
In short,
On Windows dropbox uses py2exe. py2exe embeds the python dll as a resource within the executable. The compiled python source files aka pyc files are stored as a zip archive appended to the end of the executable (which is called an overlay).
Extracting the zip archive will give you the pyc files, but that is not the end of the story.
The pyc files are encrypted and not decompilable. They are decrypted only when they are loaded by the embedded python interpreter.
However there is a way to not bother too much about the encryption algorithm used. We can
directly grab decrypted code objects from memory letting dropbox do the decryption for us.