python module dlls - python

Is there a way to make a python module load a dll in my application directory rather than the version that came with the python installation, without making changes to the python installation (which would then require I made an installer, and be careful I didn't break other apps for people by overwrting python modules and changing dll versions globaly...)?
Specifically I would like python to use my version of the sqlite3.dll, rather than the version that came with python (which is older and doesn't appear to have the fts3 module).


If you're talking about Python module DLLs, then simply modifying sys.path should be fine. However, if you're talking about DLLs linked against those DLLs; i.e. a libfoo.dll which a foo.pyd depends on, then you need to modify your PATH environment variable. I wrote about doing this for PyGTK a while ago, but in your case I think it should be as simple as:
import os
os.environ['PATH'] = 'my-app-dir' + os.pathsep + os.environ['PATH']
That will insert my-app-dir at the head of your Windows path, which I believe also controls the load-order for DLLs.
Keep in mind that you will need to do this before loading the DLL in question, i.e., before importing anything interesting.
sqlite3 may be a bit of a special case, though, since it is distributed with Python; it's obviously kind of tricky to test this quickly, so I haven't checked sqlite3.dll specifically.


The answer with modifying os.environ['PATH'] is right but it didn't work for me because I use python 3.9.
Still I was getting an error:
ImportError: DLL load failed while importing module X: The specified
module could not be found.
Turned out since version python 3.8 they added a mechanism to do this more securely.
Read documentation on os.add_dll_directory https://docs.python.org/3/library/os.html#os.add_dll_directory
Specifically see python 3.8 what's new:
DLL dependencies for extension modules and DLLs loaded with ctypes on Windows are now resolved more securely. Only the system paths, the directory containing the DLL or PYD file, and directories added with add_dll_directory() are searched for load-time dependencies. Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() and if it exists, use it to add your DLLs directory while loading your library.
So now this is the new way to make it work in python 3.8 and later:
import os
os.add_dll_directory('my-app-dir')
Again, the old way is still correct and you will have to use it in python 3.7 and older:
import os
os.environ['PATH'] = 'my-app-dir' + os.pathsep + os.environ['PATH']
After that my module with a dll dependency has been successfully loaded.


Ok it turns out python always loads the dll in the same directory as the pyd file, regardless of what the python and os paths are set to.
So I needed to copy the _sqlite3.pyd from python/v2.5/DLLS to my apps directory where the new sqlite3.dll is, making it load my new dll, rather than the one that comes with python (since the pyd files seem to follow the PYTHONPATH, even though the actual dlls themselves don't).


If your version of sqlite is in sys.path before the systems version it will use that. So you can either put it in the current directory or change the PYTHONPATH environment variable to do that.


I had the same issue as administrative rights to the default python library is blocked in a corporate environment and its extremely troublesome to perform installations.
What works for me:
Duplicate the sqlite3 library in a new location
Put in the latest sqlite3.dll (version you want from sqlite3 web) and the old _sqlite3.pyd into the new location or the new sqlite3 library. The old _sqlite3.pyd can be found in the default python library lib/DLLs folder.
Go to the new sqlite3 library and amend the dbapi2.py as follows:
Change "from _sqlite3 import *" to "from sqlite3._sqlite3 import *"
Make sure python loads this new sqlite3 library first. Add the path to this library if you must.


I encountered the same problem for my .pyd that depends on cuda/cudnn/tensorrt.
I came with a little function I call before importing my module.
def add_cuda_to_path():
if os.name != "nt":
return
path = os.getenv("PATH")
if not path:
return
path_split = path.split(";")
for folder in path_split:
if "cuda" in folder.lower() or "tensorrt" in folder.lower():
os.add_dll_directory(folder)
It is the easiest workaround I found, so I don't need to hardcode any path.
I guess little improvement would to actually check for .dll file, but this snippet serves well my needs.
Then you can simply:
add_cuda_to_path()
import my_module_that_depends_on_cuda

Related

Fatal Python error: initfsencoding: unable to load the file system codec ModuleNotFoundError: No module named 'encodings' [duplicate]

I am attempting to put together a simple c++ test project that uses an embedded python 3.2 interpreter. The project builds fine but Py_Initialize raises a fatal error:
Fatal Python error: Py_Initialize: unable to load the file system codec
LookupError: no codec search functions registered: can't find encoding
Minimal code:
#include <Python.h>
int main (int, char**)
{
Py_Initialize ();
Py_Finalize ();
return 0;
}
The OS is 32bit Vista.
The python version used is a python 3.2 debug build, built from sources using VC++ 10.
The python_d.exe file from the same build runs without any problems.
Could someone explain the problem and how to fix it? My own google-fu fails me.
EDIT 1
After going through the python source code I've found that, as the error says, no codec search functions have been registered. Both codec_register and PyCodec_Register are as they should be. It's just that nowhere in the code are any of these functions called.
I don't really know what this means as I still have no idea when and from where these functions should have been called. The code that raises the error is entirely missing from the source of my other python build (3.1.3).
EDIT 2
Answered my own question below.

Check the PYTHONPATH and PYTHONHOME environment variables and make sure they don't point to Python 2.x.
http://bugs.python.org/issue11288

Parts of this have been mentioned before, but in a nutshell this is what worked for my environment where I have multiple Python installs and my global OS environment set-up to point to a different install than the one I attempt to work with when encountering the problem.
Make sure your (local or global) environment is fully set-up to point to the install you aim to work with, e.g. you have two (or more) installs of, let's say a python27 and python33 (sorry these are windows paths but the following should be valid for equivalent UNIX-style paths just as well, please let me know about anything I'm missing here (probably the DLLs path might differ)):
C:\python27_x86
C:\python33_x64
Now, if you intend to work with your python33 install but your global environment is pointing to python27, make sure you update your environment as such (while PATH and PYTHONHOME may be optional (e.g. if you temporarily work in a local shell)):
PATH="C:\python33_x64;%PATH%"
PYTHONPATH="C:\python33_x64\DLLs;C:\python33_x64\Lib;C:\python33_x64\Lib\site-packages"
PYTHONHOME=C:\python33_x64
Note, that you might need/want to append any other library paths to your PYTHONPATH if required by your development environment, but having your DLLs, Lib and site-packages properly set-up is of prime importance.
Hope this helps.

The core reason is quite simple: Python does not find its modules directory, so it can of course not load encodings, too
Python doc on embedding says "Py_Initialize() calculates the module search path based upon its best guess" ... "In particular, it looks for a directory named lib/pythonX.Y"
Yet, if the modules are installed in (just) lib - relative to the python binary - above guess is wrong.
Although docs says that PYTHONHOME and PYTHONPATH are regarded, we observed that this was not the case; their actual presence or content was completely irrelevant.
The only thing that had an effect was a call to Py_SetPath() with e.g. [path-to]\lib as argument before Py_Initialize().
Sure this is only an option for an embedding scenario where one has direct access and control over the code; with a ready-made solution, special steps may be necessary to solve the issue.

Ran into the same thing trying to install brew's python3 under Mac OS! The issue here is that in Mac OS, homebrew puts the "real" python a whole layer deeper than you think. You would think from the homebrew output that
$ echo $PYTHONHOME
/usr/local/Cellar/python3/3.6.2/
$ echo $PYTHONPATH
/usr/local/Cellar/python3/3.6.2/bin
would be correct, but invoking $PYTHONPATH/python3 immediately crashes with the abort 6 "can't find encodings." This is because although that $PYTHONHOME looks like a complete installation, having a bin, lib etc, it is NOT the actual Python, which is in a Mac OS "Framework". Do this:
PYTHONHOME=/usr/local/Cellar/python3/3.x.y/Frameworks/Python.framework/Versions/3.x
PYTHONPATH=$PYTHONHOME/bin
(substituting version numbers as appropriate) and it will work fine.

From python3k, the startup need the encodings module, which can be found in PYTHONHOME\Lib directory.
In fact, the API Py_Initialize () do the init and import the encodings module.
Make sure PYTHONHOME\Lib is in sys.path and check the encodings module is there.

I had this issue with python 3.5, anaconda 3, windows 7 32 bit. I solved it by moving my pythonX.lib and pythonX.dll files into my working directory and calling
Py_SetPythonHome(L"C:\\Path\\To\\My\\Python\\Installation");
before initialize so that it could find the headers that it needed, where my path was to "...\Anaconda3\". The extra step of calling Py_SetPythonHome was required for me or else I'd end up getting other strange errors where python import files.

I just ran into the exact same problem (same Python version, OS, code, etc).
You just have to copy Python's Lib/ directory in your program's working directory ( on VC it's the directory where the .vcproj is )

There appears to be something going wrong with the release build either failing to include the appropriate codecs or else misidentifying the codec to use for system APIs. Since the python_d executable is working, what does that return for os.getfsencoding()? (Use the C API to call that between your Initialize/Finalize calls)

I had the same issue and found this question. However from the answers here I was not able to solve my problem. I started debugging the cpython code and thought that I might be discovered a bug. Therefore I opened a issue on the python issue tracker.
My mistake was that I did not understand that Py_SetPath clears all inferred paths.
So one needs to set all paths when calling this function.
Link to the issue conversation
For completion I also copied the most important part of the conversation below.
My original issue text
I compiled the source of CPython 3.7.3 myself on Windows with Visual Studio 2017 together with some packages like e.g numpy. When I start the Python Interpreter I am able to import and use numpy. However when I am running the same script via the C-API I get an ModuleNotFoundError.
So the first thing I did, was to check if numpy is in my site-packages directory and indeed there is a folder named numpy-1.16.2-py3.7-win-amd64.egg. (Makes sense because the python interpreter can find numpy)
The next thing I did was to get some information about the sys.path variable created when running the script via the C-API.
#### sys.path content ####
C:\Work\build\product\python37.zip
C:\Work\build\product\DLLs
C:\Work\build\product\lib
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\2017\PROFESSIONAL\COMMON7\IDE\EXTENSIONS\TESTPLATFORM
C:\Users\rvq\AppData\Roaming\Python\Python37\site-packages
Examining the content of sys.path I noticed two things.
C:\Work\build\product\python37.zip has the correct path 'C:\Work\build\product\'. There was just no zip file. All my files and directory were unpacked. So I zipped the files to an archive named python37.zip and this resolved the import error.
C:\Users\rvq\AppData\Roaming\Python\Python37\site-packages is wrong it should be C:\Work\build\product\Lib\site-packages but I dont know how this wrong path is created.
The next thing I tried was to use Py_SetPath(L"C:/Work/build/product/Lib/site-packages") before calling Py_Initialize(). This led to
Fatal Python Error 'unable to load the file system encoding'
ModuleNotFoundError: No module named 'encodings'
I created a minimal c++ project with exact these two calls and started to debug Cpython.
int main()
{
Py_SetPath(L"C:/Work/build/product/Lib/site-packages");
Py_Initialize();
}
I tracked the call of Py_Initialize() down to the call of
static int
zipimport_zipimporter___init___impl(ZipImporter *self, PyObject *path)
inside of zipimport.c
The comment above this function states the following:
Create a new zipimporter instance. 'archivepath' must be a path-like
object to a zipfile, or to a specific path inside a zipfile. For
example, it can be '/tmp/myimport.zip', or
'/tmp/myimport.zip/mydirectory', if mydirectory is a valid directory
inside the archive. 'ZipImportError' is raised if 'archivepath'
doesn't point to a valid Zip archive. The 'archive' attribute of the
zipimporter object contains the name of the zipfile targeted.
So for me it seems that the C-API expects the path set with Py_SetPath to be a path to a zipfile. Is this expected behaviour or is it a bug?
If it is not a bug is there a way to changes this so that it can also detect directories?
PS: The ModuleNotFoundError did not occur for me when using Python 3.5.2+, which was the version I used in my project before. I also checked if I had set any PYTHONHOME or PYTHONPATH environment variables but I did not see one of them on my system.
Answer
This is probably a documentation failure more than anything else. We're in the middle of redesigning initialization though, so it's good timing to contribute this feedback.
The short answer is that you need to make sure Python can find the Lib/encodings directory, typically by putting the standard library in sys.path. Py_SetPath clears all inferred paths, so you need to specify all the places Python should look. (The rules for where Python looks automatically are complicated and vary by platform, which is something I'm keen to fix.)
Paths that don't exist are okay, and that's the zip file. You can choose to put the stdlib into a zip, and it will be found automatically if you name it the default path, but you can also leave it unzipped and reference the directory.
A full walk through on embedding is more than I'm prepared to type on my phone. Hopefully that's enough to get you going for now.

I had the problem and was tinkering with different solutions mentioned here. Since I was running my project from Visual Studio, apparently, I needed to set the environment path inside Visual Studio and not the system path.
Adding a simple PYTHONHOME=PATH\TO\PYTHON\DIR in the project solution\properties\environment solved the problem.

For me this happened when I updated Python 64 bit from 3.6.4 to 3.6.5. It threw some error like "unable to extract python.dll. Do you have permissions."
Pycharm also failed to load interpreter, even though I reloaded it in settings. Running python command gave same error, with and without administrator mode.
Reason
There was error in installation of Python, include folder in python installation directory C:\Users\USERNAME\AppData\Local\Programs\Python\Python36 was missing
Reinstalling Python also dint solve the issue.(Not removal and install)
Solution
Uninstall Python and Install of Python again.
Because running installer was just extracting same files excluding include folder

In my cases, for windows, if you have multiple python versions installed, if PYTHONPATH is pointing to one version the other ones didn't work. I found that if you just remove PYTHONPATH, they all work fine

For those working in Visual Studio simply add the include, Lib and libs directories to the Include Directories and Library Directories under
Projects Properties -> Configuration Properties > VC++ Directories :
For example I have Anaconda3 on my system and working with Visual Studio 2015 This is how the settings looks like (note the Include and Library directories) :
Edit:
As also pointed out by bossi setting PYTHONPATH in your user Environment Variables section seems necessary.
a sample input can be like this (in my case):
C:\Users\Master\Anaconda3\Lib;C:\Users\Master\Anaconda3\libs;C:\Users\Master\Anaconda3\Lib\site-packages;C:\Users\Master\Anaconda3\DLLs
is necessary it seems.
Also, you need to restart Visual Studio after you set up the PYTHONPATH in your user Environment Variables for the changes to take effect.
Also note that :
Make sure the PYTHONHOME environment variable is set to the Python
interpreter you want to use. The C++ projects in Visual Studio rely on
this variable to locate files such as python.h, which are used when
creating a Python extension.

So, for some reason the python dll fails to locate the encodings module. The python.exe executable apparently finds it because it has the expected relative path. Modifying the search path works.
The reason for all of this? Don't know but at least it works. I highly suspect a typo on my part somewhere, that's usually the reason for odd bugs it seems.

Relative file paths in package not working

I'm having a little issue with my package directories. The structure is as follows:
package folder with modules
Databases
In the package folder I have a lot of .py files with functions that I use from everywhere (so on another drive as well). Some functions like "guess_countries" use databases located in a subfolder. I did that because I want to export my code to github (private repo).
Here is the issue:
My module Geo_guesser needs to look for this path (so a subfolder): "Databases/Geo/Countries/Countries (ZIP+Dump).sqlite3"
However upon importing from another folder the current directory gets appended and it becomes "Z:/Other_folder/Databases/Geo/Countries/Countries (ZIP+Dump).sqlite3" instead of "A:/My_package/Databases/Geo/Countries/Countries (ZIP+Dump).sqlite3" where the databases are.
I don't want to use absolute paths because everything is contained in the package folder and in the future I'd like to make it pip-installable or maybe share it with others and so the absolute path won't be the same obviously.
Other infos:
In the module Geo_guesser I've tried using: os.path.realpath, __file__ and sys.argv without success (I looked up many topics before posting this).
I used conda develop to be able to import my package's modules from anywhere
Tools:
Anaconda, Python 3.6 and Jupyter
Thanks in advance for the help :)!

Well nevermind I finally found a code that works for me sorry :( :
import os, sys, inspect
# realpath() will make your script run, even if you symlink it :)
cmd_folder = os.path.realpath(os.path.abspath(os.path.split(inspect.getfile( inspect.currentframe() ))[0]))
if cmd_folder not in sys.path:
sys.path.insert(0, cmd_folder)
Source: Import a module from a relative path

added path in PYTHONPATH changes sys.path entires

I have numerous versions of packages installed. My base, system python is ~/Library/Enthought/Canopy_64bit/User/lib/python2.7/
but I also have an application ('yt') which installed its own python to
~/Applications/yt/yt-x86_64/lib/python2.7/
I've added the yt path so that I can import a module which it includes, when I run my system python. The problem is, when I add the yt-path to PYTHONPATH it adds a ton of other directories, to higher entries in my sys.path so that when I try to import numpy (for example), I end up getting the yt-version, instead of my system version.
Is there a way to keep my sys.path from being modified?

PYTHONPATH values are always inserted in front of the standard python library paths in sys.path
One potential approach around this problem is to add the yt path to sys.path yourself.
So try
# append to the *end* of the system path.
sys.path.append('~/Applications/yt/yt-x86_64/lib/python2.7/path/to/libs')
this will put the yt specific modules at the end of the list, and your system's numpy will be found/imported first.

Python: import _io

I'm trying to determine which files in the Python library are strictly necessary for my script to run. Right now I'm trying to determine where _io.py is located. In io.py (no underscore), the _io.py module (with underscore) is imported on line 60.

Some modules are compiled directly into the interpreter -- there are no files corresponding to them. You can retrieve a list of these modules from sys.builtin_module_names. In my Pyton 3.1 installation, _io is included in this list.
You might want to have a look at snakefood to determine the dependencies of your script.

Not all Python modules are written in Python. Try looking for _io.so or _io.pyd.

Try the DLLs folder under your base python install directory if you are on windows. It contains .pyd modules Ignacio mentions. I had a similar problem with a portable install. Including the DLLs folder contents to my install fixed it. I am using python 2.5.

From python-list email archive: is "_io.py" missing from 2.7.4 ?, the situation for Python 2 and 3 is different:
In Python 2.7:
To find where the _io module lives, at the interactive interpreter run
this:
import _io
_io.__file__
Under Linux, you should get something like this:
'/usr/local/lib/python2.7/lib-dynload/_io.so'
and the equivalent under Windows.
In Python 3:
Note that in Python 3.3, the _io module is now built-in into the
compiler, so _io.__file__ no longer exists.

Installing python with python win32 extensions on a network drive

I need to keep a large number of Windows XP machines running the same version of python, with an assortment of modules, one of which is python-win32. I thought about installing python on a network drive that is mounted by all the client machines, and just adjust the path on the clients. Python starts up fine from the network, but when importing win32com I get a pop-up error saying:
The procedure entry point ?PyWinObject_AsHANDLE##YAHPAU_object##PAPAXH#Z could not be located in the dynamic link library pywintypes24.dll
after dismissing the message dialog I get in the console:
ImportError: DLL load failed: The specified procedure could not be found.
I searched the python directory for the pywintypes24.dll and it is present in "Lib\site-packages\pywin32_system32" .
What am I missing and is there another way in which I can install Python + Python-Win32 + additional module once and have them running on many machines? I don't have access to the Microsoft systems management tools, so I need to be a bit more low-tech than that.

On every machine you have to basically run following pywin32_postinstall.py -install once. Assuming your python installation on the network is N:\Python26, run following command on every client:
N:\Python26\python.exe N:\Python26\Scripts\pywin32_postinstall.py -install
Another important thing is Good Luck!. The reason is that you might need to do this as admin. In my case such setup worked for all but one computer. I still did not figure out why.

Python (or precisely, the OS) searches the DLLs using os.environ["PATH"] and not by searching sys.path.
So you could start Python using a simple .cmd file instead which adds \server\share\python26 to the path (given the installer (or you) copied the DLLs from \server\share\python26\lib\site-packages\pywin32-system32 to \server\share\python26).
Or, you can add the following code to your scripts before they try to import win32api etc:
# Add Python installation directory to the path,
# because on Windows 7 the pywin32 installer fails to copy
# the required DLLs to the %WINDIR%\System32 directory and
# copies them to the Python installation directory instead.
# Fortunately, in Python it is possible to modify the PATH
# before loading the DLLs.
os.environ["PATH"] = sys.prefix + ";" + os.environ.get("PATH")
import win32gui
import win32con

You could use batch files running at boot to
Mount the network share (net use \\server\share)
Copy the Python and packages installers from the network share to a local folder
Check version of the msi installer against the installed version
If different, uninstall Python and all version dependent packages
Reinstall all packages
This would be pretty much a roll your own central management system for that software.

Categories

Resources