Develop Reference

Python sys.path vs import

What I wish to understand is what is good/bad practice, and why, when it comes to imports. What I want to understand is the agreed upon view by the community on the matter, if there's any one such in some PEP document or similar.
What I see normally is people have a python environment, use conda/pip to install packages and all that's needed to do in the code is to use "import X" (and variants). In my current understanding this is the right way to do things.
Whenever python interacts with C++ at my firm, though, it always ends up with the need to use sys.path and absolute imports (but we have some standardized paths to use as "base" and usually define relative paths based on those).
There are 2 major cases:
Python wrapper for C++ library (pybind/ctype/etc) - in this case the user python code must use sys.path to specify where the C++ library to import is.
A project that establish communications between python and C++ (say C++ server, python clients, TCP connections and flatbuffer serialization between the two) - Here the python code lives together with the C++ code, and if it some python files end up using sys.path to import python modules from the same project but that live in a different directory - Essentially we deploy the python together with the C++ through our C++ deployment procedure.
I am not fully sure if we can do something better for case #1, but case #2 seems quite unnecessary, and basically forced just by the choice to not deploy the python code through a python package manager. Choice ends up forcing us to use sys.path on both the library and user code.
This seems very bad to me as basically this way of doing things doesn't allow us to fully manage our python environments (since we have some libraries that we import even thought they are not technically installed in the environment), and that is probably why I have a negative view of using sys.path for imports. But I need to find if I'm right, and if so I need some official (or almost) documents to support my case if I'm to propose fixes to our procedures.

For your scenario 2, my understanding is you have some C++ and accompanying python in one place, and a separate python project wants to import that python.
Could you structure the imported python as a package and install it to your environment with pip install path/to/package? If it's a package that you'll continue to edit, you can add the -e flag to pip install so that when the package changes your imports get the latest code.

Python Extension Dll Installation

I have a large program written in C++ that I wish to make usable via Python. I've written a python extension to expose an interface through which python code can call the C++ functions. The issue I'm having with this is that installing seems to be nontrivial.
All documentation I can find seems to indicate that I should create a setup.py which creates a distutils.core.Extension. In every example I've found, the Extension object being created is given a list of source files, which it compiles. If my code was one or two files, this would be fine. Unfortunately, it's dozens of files, and I use a number of relatively complicated visual studio build settings. As a result, building by listing .c files seems to be challenging to say the least.
I've currently configured my Python extension to build as a .dll and link against python39.lib. I tried changing the extension to .pyd and including the file in a manifest.in. After I created a setup.py and ran it, it created a .egg file that I verified did include the .pyd I created. However, after installing it, when I imported the module into python, the module was completely empty (and I verified that the PyInit_[module] function was not called). Python dll Extension Import says that I can import the dll if I change the extension to .pyd and place the file in the Dlls directory of python's installation. I've encountered two problems with this.
The first is that it seems to me that it's not very distributable like this. I'd like to package this into a python wheel, and I'm not sure how a wheel could do this. The second is even more problematic - it doesn't exactly work. It calls the initialization function of my extension, and I've verified in WinDbg that it's returning a python module. However, this is what I always get from the console.
>>> import bluespawn
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
SystemError: initialization of bluespawn did not return an extension module
The Python documentation has a section on publishing binary extensions, but for the past four years, it has been left as a placeholder. The github issue linked here isn't that helpful either; it boils down to either use distutils to build or use enscons to build. But since my build is a fairly complicated procedure, completely rewriting it to use enscons is less than desirable, to say the least.
It seems to me like placing the file in the DLLs directory is the wrong way of going about this. Given that I have a DLL and making setuptools compile everything itself seems infeasible, how should I go about installing my extension?
For reference, here's my initialization function, in case that's incorrect.
PyModuleDef bsModule{ PyModuleDef_HEAD_INIT, "bluespawn", "Bluespawn python bindings", -1, methods };
PyMODINIT_FUNC PyInit_bluespawn() {
PyObject* m;
Py_Initialize();
PyEval_InitThreads();
PyGILState_STATE state = PyGILState_Ensure(); // Crashes without this. Call to PyEval_InitThreads() required for this.
m = PyModule_Create(&bsModule);
PyGILState_Release(state);
Py_Finalize();
return m;
}
The python interface is available here: https://github.com/ION28/BLUESPAWN/blob/client-add-pylib/BLUESPAWN-win-client/src/user/python/PythonInterface.cpp
EDIT: I have a working solution that I am sure is not best practice. I created a very small C file that simply passes all calls it receives onto the large DLL I've already created. The C file is responsible for initializing the module, but everything else is handled inside the DLL. It works, but it seems like a very bad way of doing things. What I'm looking for is a better way of doing this.

Let me try and divide your post into two separate questions:
How to package a C++ library with a non-trivial compilation process using setuptools
Is it possible to distribute a python package with a precompiled library
1. How to package a C++ library with a non-trivial compilation process using setuptools
It is possible. I was quite surprised to see that setuptools offers many ways to override the compilation process, see the documentation here. For example, you can use the keyword argument extra_compile_args to pass extra arguments to the compiler.
In addition, as setup.py is a python file, you could relatively easily write some code to automatically collect all files needed for compilation. I'd done this myself in a project (github), and it worked quite well for me.
Here's some code from the setup.py:
libinjector = Extension('pyinjector.libinjector',
sources=[str(c.relative_to(PROJECT_ROOT))
for c in [LIBINJECTOR_WRAPPER, *LIBINJECTOR_SRC.iterdir()]
if c.suffix == '.c'],
include_dirs=[str(LIBINJECTOR_DIR.relative_to(PROJECT_ROOT) / 'include')],
export_symbols=['injector_attach', 'injector_inject', 'injector_detach'],
define_macros=[('EM_AARCH64', '183')])
2. Is it possible to distribute a python package with a precompiled library
I understand from your edit that you've managed to get it to work, but I'll say a few words anyway. Releasing precompiled binaries with your source distribution is possible, and it is possible to release your manually-compiled binaries in a wheel file as well, but it is not recommended.
The main reason is compatibility with the target architecture. First, you'll have to include two DLLs in your distribution, one for x64 and one for x86. Second, you might lose some nice optimizations, because you'll have to instruct the compiler to ignore optimizations available for the specific CPU type (note that this applies to normal wheel distributions as well). If you're compiling against windows SDK, you'll probably want to use the user's version too. In addition, including two DLLs in your release might grow it to an awkward size for a source distribution.

Can you cimport an .so file?

I have an .so file called tissue-classifier.cpython-37m-x86_64-linux-gnu.so from an external library that I want to import so that I can extend it in one of my local classes. Since I am extending a class, I need to import it as an extension type using cimport and I am wondering if this is even possible. If I use a normal import statement then I will be left with a Python compiled version which cannot be used to extend a cdef class in my current file.
When I try to cimport the tissue_classifier file, it gives me the error that tissue_classifier.pxd file could not be found which makes sense since it is in .so format. Sorry if this is a dumb question, I just haven't been able to figure this out for a while.

No, a *.so-file cannot be cimported.
If one is having C/CPP-backgrounds, then pyx/pxd/so-business is propably easiest to understand using the following model:
the resulting extension (*.so-file) corresponds to the final artefact in C/CPP-world which could be an executable, a shared-object (*.so), or a library/object-file collection. If you just runs the resulting program it is all you need. For example you can use (and probably do) a CPython-interpreter without having built it or having its source code. In analogy, if you have a binary extension (*.so) you can import and use it whitout having to build it (or even having corresponding pyx-files or a compiler on your machine) - that is what is provided by a wheel.
*.pyx corresponds to c/cpp-files, which have the definitions of the functionality. These files are needed if you want to build the resulting artifact from the source. In C/CPP-world this build process would be triggered by using make or similar. pyx-files are needed if you install the package via python setup.py install - which corresponds to calling make.
*.pxd corresponds to headers (h/hpp-files): it decribes the functionality in the resulting so-files, so it can be reused. For example just having CPython-interpreter isn't enough to build extensions - one has to install the dev-version so also the includes Python.h&Co. are present at the machine.
So what can be done?
First possibility:
If authors of the package consider *.pxd-files being part of the public API, then they can put the corresponding pxd-files next to *.so-files into the installation, so the c-interface of the module can be used/extended.
If they don't put the pxd-file into the installation, so chances are high that this c-interface is an implementation detail and you should not be using it, as it may change without notice in the future.
However, it is possible to take the risk and to copy the necessary pxd-files to the installation per hand, but making first sure that it is the right pxd-version (i.e. the same with which so-files in the installation were built).
Second possibility:
The easiest way to ensure, that the right pxd-version is used is to build package from the source, i.e.
dowloading the the right versioin from github (master or last release)
calling python setup.py install or what README tells you to do
Now, instead of copying pdx-files to the installation, one could add include_path to the downloaded package-source via include_path for cythonize-function or by adding it to sys.path.
Alternatively, as #BeforeFlight has pointed out in the comments, one could use python setup.py develop (or pip install -e the same folder so it can be deinstalled), and because it creates a link instead of copying data, the pxd-files will be found.
The above solutions will help to build the module, distributing it is a completely different story however.

using a C extension in python, without installing it as a module

I am writing C extensions for python. I am just experimenting for the time being and I have written a hello world extension that looks like this :
#include <Python2.7/Python.h>
static PyObject* helloworld(PyObject* self)
{
return Py_BuildValue("s", "Hello, Python extensions!!");
}
static char helloworld_docs[] = "helloworld( ): Any message you want to put here!!\n";
static PyMethodDef helloworld_funcs[] = {
{"helloworld", (PyCFunction)helloworld, METH_NOARGS, helloworld_docs},
{NULL,NULL,0,NULL}
};
void inithelloworld(void)
{
Py_InitModule3("helloworld", helloworld_funcs,"Extension module example!");
}
the code works perfectly fine, after installing it from a setup.py file I wrote, and installing it from command line
python setup.py install
What I want is the following :
I want to use the C file as a python extension module, without installing it, that is I want to use it as just another python file in my project, and not a file that I need to install before my python modules get to use its functionality. Is there some way of doing this ?

You can simply compile the extension without installing (usually something like python setup.py build). Then you have to make sure the interpreter can find the compiled module (for example by copying it next to a script that imports it, or setting PYTHONPATH).

You can create your "own interpreter" by not extending python, but embedding it into your application. In that way, your objects will be always available for the users who are running your program. This is a pretty common thing to do in certain cases, for example look at the Blender project where all the bpy, bmesh and bge modules are already included.
The downside is, your users can't use the python command directly, they have to use your hello_world_python instead. (But of course you can provide your extension as a module as well.) And that also means, you have to compile and distribute your application for all platforms you want to support -- in case you want to distribute it as a binary, to make your users lives a bit easier.
For further information on embedding python into your program, read the propriate sections of the documentation:
Embedding Python in Another Application
Personal suggestion: Use Python 3.5 whenever you can, and stop supporting the old 2.x versions. For more information, read this article: Should I use Python 2 or Python 3 for my development activity?

Manually adding libraries

Given that every library is a python code itself, I think its logical that instead of using the import command, we can actually copy the whole code of that library and paste it to the top of our main.py.
I'm working on a remote pc, I cannot install libraries, can I use a library by just doing this?
Forgive me if this a very silly question.
Thanks

In some cases yes, you can. But there are (a lot of) libraries that have some of their functionality written in C and compiled to binary (e.g. the famous numpy). You can't just paste those.
Another thing that the pasting might introduce are naming colisions. If you use
import module
than any name in the module module can be safely used in the importing module using module.name even if the name name is already defined somewhere in the importing module. If you just paste the code this won't work.

While pasting the entire library at the top of your main file can work, I don't think it's the best way to go about solving your problem.
One option is to move the library and put it in to the same folder as your main.py file. I believe the import statement will check the current working directory for the library before looking elsewhere.
Another option is to use a virtual environment(virtualenv) and then install all the required libraries within this virtual environment. I'm not sure that this will work for you since you said you cannot install on this libraries and virtualenv requires pep. If you are interested in learning more about python virtual environments, take a look here.

Most modules are actually written in C, like Pygame for example. Python itself is based on C. You can't jump to conclusions, but if the library is pure Python, I'd suggest copying the package into your project directory and importing, rather than copying and pasting code snippets.