Develop Reference

Fatal Python error: initfsencoding: unable to load the file system codec ModuleNotFoundError: No module named 'encodings' [duplicate]

I am attempting to put together a simple c++ test project that uses an embedded python 3.2 interpreter. The project builds fine but Py_Initialize raises a fatal error:
Fatal Python error: Py_Initialize: unable to load the file system codec
LookupError: no codec search functions registered: can't find encoding
Minimal code:
#include <Python.h>
int main (int, char**)
{
Py_Initialize ();
Py_Finalize ();
return 0;
}
The OS is 32bit Vista.
The python version used is a python 3.2 debug build, built from sources using VC++ 10.
The python_d.exe file from the same build runs without any problems.
Could someone explain the problem and how to fix it? My own google-fu fails me.
EDIT 1
After going through the python source code I've found that, as the error says, no codec search functions have been registered. Both codec_register and PyCodec_Register are as they should be. It's just that nowhere in the code are any of these functions called.
I don't really know what this means as I still have no idea when and from where these functions should have been called. The code that raises the error is entirely missing from the source of my other python build (3.1.3).
EDIT 2
Answered my own question below.

Check the PYTHONPATH and PYTHONHOME environment variables and make sure they don't point to Python 2.x.
http://bugs.python.org/issue11288

Parts of this have been mentioned before, but in a nutshell this is what worked for my environment where I have multiple Python installs and my global OS environment set-up to point to a different install than the one I attempt to work with when encountering the problem.
Make sure your (local or global) environment is fully set-up to point to the install you aim to work with, e.g. you have two (or more) installs of, let's say a python27 and python33 (sorry these are windows paths but the following should be valid for equivalent UNIX-style paths just as well, please let me know about anything I'm missing here (probably the DLLs path might differ)):
C:\python27_x86
C:\python33_x64
Now, if you intend to work with your python33 install but your global environment is pointing to python27, make sure you update your environment as such (while PATH and PYTHONHOME may be optional (e.g. if you temporarily work in a local shell)):
PATH="C:\python33_x64;%PATH%"
PYTHONPATH="C:\python33_x64\DLLs;C:\python33_x64\Lib;C:\python33_x64\Lib\site-packages"
PYTHONHOME=C:\python33_x64
Note, that you might need/want to append any other library paths to your PYTHONPATH if required by your development environment, but having your DLLs, Lib and site-packages properly set-up is of prime importance.
Hope this helps.

The core reason is quite simple: Python does not find its modules directory, so it can of course not load encodings, too
Python doc on embedding says "Py_Initialize() calculates the module search path based upon its best guess" ... "In particular, it looks for a directory named lib/pythonX.Y"
Yet, if the modules are installed in (just) lib - relative to the python binary - above guess is wrong.
Although docs says that PYTHONHOME and PYTHONPATH are regarded, we observed that this was not the case; their actual presence or content was completely irrelevant.
The only thing that had an effect was a call to Py_SetPath() with e.g. [path-to]\lib as argument before Py_Initialize().
Sure this is only an option for an embedding scenario where one has direct access and control over the code; with a ready-made solution, special steps may be necessary to solve the issue.

Ran into the same thing trying to install brew's python3 under Mac OS! The issue here is that in Mac OS, homebrew puts the "real" python a whole layer deeper than you think. You would think from the homebrew output that
$ echo $PYTHONHOME
/usr/local/Cellar/python3/3.6.2/
$ echo $PYTHONPATH
/usr/local/Cellar/python3/3.6.2/bin
would be correct, but invoking $PYTHONPATH/python3 immediately crashes with the abort 6 "can't find encodings." This is because although that $PYTHONHOME looks like a complete installation, having a bin, lib etc, it is NOT the actual Python, which is in a Mac OS "Framework". Do this:
PYTHONHOME=/usr/local/Cellar/python3/3.x.y/Frameworks/Python.framework/Versions/3.x
PYTHONPATH=$PYTHONHOME/bin
(substituting version numbers as appropriate) and it will work fine.

From python3k, the startup need the encodings module, which can be found in PYTHONHOME\Lib directory.
In fact, the API Py_Initialize () do the init and import the encodings module.
Make sure PYTHONHOME\Lib is in sys.path and check the encodings module is there.

I had this issue with python 3.5, anaconda 3, windows 7 32 bit. I solved it by moving my pythonX.lib and pythonX.dll files into my working directory and calling
Py_SetPythonHome(L"C:\\Path\\To\\My\\Python\\Installation");
before initialize so that it could find the headers that it needed, where my path was to "...\Anaconda3\". The extra step of calling Py_SetPythonHome was required for me or else I'd end up getting other strange errors where python import files.

I just ran into the exact same problem (same Python version, OS, code, etc).
You just have to copy Python's Lib/ directory in your program's working directory ( on VC it's the directory where the .vcproj is )

There appears to be something going wrong with the release build either failing to include the appropriate codecs or else misidentifying the codec to use for system APIs. Since the python_d executable is working, what does that return for os.getfsencoding()? (Use the C API to call that between your Initialize/Finalize calls)

I had the same issue and found this question. However from the answers here I was not able to solve my problem. I started debugging the cpython code and thought that I might be discovered a bug. Therefore I opened a issue on the python issue tracker.
My mistake was that I did not understand that Py_SetPath clears all inferred paths.
So one needs to set all paths when calling this function.
Link to the issue conversation
For completion I also copied the most important part of the conversation below.
My original issue text
I compiled the source of CPython 3.7.3 myself on Windows with Visual Studio 2017 together with some packages like e.g numpy. When I start the Python Interpreter I am able to import and use numpy. However when I am running the same script via the C-API I get an ModuleNotFoundError.
So the first thing I did, was to check if numpy is in my site-packages directory and indeed there is a folder named numpy-1.16.2-py3.7-win-amd64.egg. (Makes sense because the python interpreter can find numpy)
The next thing I did was to get some information about the sys.path variable created when running the script via the C-API.
#### sys.path content ####
C:\Work\build\product\python37.zip
C:\Work\build\product\DLLs
C:\Work\build\product\lib
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\2017\PROFESSIONAL\COMMON7\IDE\EXTENSIONS\TESTPLATFORM
C:\Users\rvq\AppData\Roaming\Python\Python37\site-packages
Examining the content of sys.path I noticed two things.
C:\Work\build\product\python37.zip has the correct path 'C:\Work\build\product\'. There was just no zip file. All my files and directory were unpacked. So I zipped the files to an archive named python37.zip and this resolved the import error.
C:\Users\rvq\AppData\Roaming\Python\Python37\site-packages is wrong it should be C:\Work\build\product\Lib\site-packages but I dont know how this wrong path is created.
The next thing I tried was to use Py_SetPath(L"C:/Work/build/product/Lib/site-packages") before calling Py_Initialize(). This led to
Fatal Python Error 'unable to load the file system encoding'
ModuleNotFoundError: No module named 'encodings'
I created a minimal c++ project with exact these two calls and started to debug Cpython.
int main()
{
Py_SetPath(L"C:/Work/build/product/Lib/site-packages");
Py_Initialize();
}
I tracked the call of Py_Initialize() down to the call of
static int
zipimport_zipimporter___init___impl(ZipImporter *self, PyObject *path)
inside of zipimport.c
The comment above this function states the following:
Create a new zipimporter instance. 'archivepath' must be a path-like
object to a zipfile, or to a specific path inside a zipfile. For
example, it can be '/tmp/myimport.zip', or
'/tmp/myimport.zip/mydirectory', if mydirectory is a valid directory
inside the archive. 'ZipImportError' is raised if 'archivepath'
doesn't point to a valid Zip archive. The 'archive' attribute of the
zipimporter object contains the name of the zipfile targeted.
So for me it seems that the C-API expects the path set with Py_SetPath to be a path to a zipfile. Is this expected behaviour or is it a bug?
If it is not a bug is there a way to changes this so that it can also detect directories?
PS: The ModuleNotFoundError did not occur for me when using Python 3.5.2+, which was the version I used in my project before. I also checked if I had set any PYTHONHOME or PYTHONPATH environment variables but I did not see one of them on my system.
Answer
This is probably a documentation failure more than anything else. We're in the middle of redesigning initialization though, so it's good timing to contribute this feedback.
The short answer is that you need to make sure Python can find the Lib/encodings directory, typically by putting the standard library in sys.path. Py_SetPath clears all inferred paths, so you need to specify all the places Python should look. (The rules for where Python looks automatically are complicated and vary by platform, which is something I'm keen to fix.)
Paths that don't exist are okay, and that's the zip file. You can choose to put the stdlib into a zip, and it will be found automatically if you name it the default path, but you can also leave it unzipped and reference the directory.
A full walk through on embedding is more than I'm prepared to type on my phone. Hopefully that's enough to get you going for now.

I had the problem and was tinkering with different solutions mentioned here. Since I was running my project from Visual Studio, apparently, I needed to set the environment path inside Visual Studio and not the system path.
Adding a simple PYTHONHOME=PATH\TO\PYTHON\DIR in the project solution\properties\environment solved the problem.

For me this happened when I updated Python 64 bit from 3.6.4 to 3.6.5. It threw some error like "unable to extract python.dll. Do you have permissions."
Pycharm also failed to load interpreter, even though I reloaded it in settings. Running python command gave same error, with and without administrator mode.
Reason
There was error in installation of Python, include folder in python installation directory C:\Users\USERNAME\AppData\Local\Programs\Python\Python36 was missing
Reinstalling Python also dint solve the issue.(Not removal and install)
Solution
Uninstall Python and Install of Python again.
Because running installer was just extracting same files excluding include folder

In my cases, for windows, if you have multiple python versions installed, if PYTHONPATH is pointing to one version the other ones didn't work. I found that if you just remove PYTHONPATH, they all work fine

For those working in Visual Studio simply add the include, Lib and libs directories to the Include Directories and Library Directories under
Projects Properties -> Configuration Properties > VC++ Directories :
For example I have Anaconda3 on my system and working with Visual Studio 2015 This is how the settings looks like (note the Include and Library directories) :
Edit:
As also pointed out by bossi setting PYTHONPATH in your user Environment Variables section seems necessary.
a sample input can be like this (in my case):
C:\Users\Master\Anaconda3\Lib;C:\Users\Master\Anaconda3\libs;C:\Users\Master\Anaconda3\Lib\site-packages;C:\Users\Master\Anaconda3\DLLs
is necessary it seems.
Also, you need to restart Visual Studio after you set up the PYTHONPATH in your user Environment Variables for the changes to take effect.
Also note that :
Make sure the PYTHONHOME environment variable is set to the Python
interpreter you want to use. The C++ projects in Visual Studio rely on
this variable to locate files such as python.h, which are used when
creating a Python extension.

So, for some reason the python dll fails to locate the encodings module. The python.exe executable apparently finds it because it has the expected relative path. Modifying the search path works.
The reason for all of this? Don't know but at least it works. I highly suspect a typo on my part somewhere, that's usually the reason for odd bugs it seems.

Error in launching Stanford corenlp server

I want to use Stanford parser in order to get typed dependency of a text. I was trying to follow the instructions provided in https://bitbucket.org/torotoki/corenlp-python, however, I got an error, both while launching the server and using the python library:
from corenlp import *
corenlp = StanfordCoreNLP("./stanford-corenlp-full-2014-08-27/")
This is the error:
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/
pexpect/__init__.py", line 1554, in expect_loopraise EOF(str(err) + '\n' + str(self))
pexpect.EOF: End Of File (EOF). Empty string style platform.
It seems the problem is about pexpect package. I already installed it as it is explained in the insturctuion. I saw similiar problem here(EOF when using pexpect and pxssh), but it is different from my case. I am using Mac and Python 2.7.
Could you please help me!

This is not the solution you wanted.
Are you sure that it is the same error if you use corenlp using the package implementation ?
pexpect is required only when there is interaction with the interactive shell of corenlp, which is present in the server implementation. In the package implementation, a file containing a list of input files is given to the parser. (using batch_parse)
In the server implementation, there is a JSON-RPC server created by the wrapper. If you want to use the server implementation, you have to remotely call one of the procedures and you don't need the package on the client end. I guess, you have mistaken in putting the right code here.
PS:
I, myself is one of the corenlp users but i used a different wrapper.

I hit the same issue while on mac. For posterity, just follow verbatim the instructions at https://bitbucket.org/torotoki/corenlp-python to use python wrapper around Stanford-corenlp. This package gets around the "End Of File (EOF). Empty string style platform." error that throws up on the some mac (or other OS, didnt check though but I know someone who also got it on windows) versions.
Also don't forget to download the stanford-corenlp-full-2014-08-27.zip instead of the latest version.
The solution of using expect(pexpect.EOF) didn't work for me (https://github.com/pexpect/pexpect/blob/master/doc/overview.rst) as the Stanford-corenlp jar was not loading properly in this case.
I also tried https://github.com/dasmith/stanford-corenlp-python as well as https://github.com/Wordseer/stanford-corenlp-python but neither of them worked. Both threw up the EOF error

HTTP Error 502.2 - Bad Gateway Configuring IIS 7 with Python 2.7

I am a newbie to Stack Overflow (first post), but really see the use of this website.
I'm stumped. We are trying to setup IIS 7.0 to run with WinPython 2.7 on a Windows 7 machine.
I am an IIS newb, but veteran Python user. IIS 7 can NOT find a library, which python finds, and executes, perfectly when ran on it's own. When executed via IIS, the script fails with a traceback, and IIS returns the 502.2.
I found this thread http://forums.iis.net/p/1209465/2073173.aspx?HTTP+Error+502+2+Bad+Gateway+Frustrations but the advised solution is simply another troubleshooting suggestion.
I found IIS's description (http://support.microsoft.com/kb/942057) of the error helpful, but futile.
I found Python's start-up options/parameters helpful (http://docs.python.org/2/using/cmdline.html), but futile.
I found IIS's advice for configuring Python helpful (http://support.microsoft.com/kb/276494, but (questionably?) incomplete.
This thread on manually defining an alternate bin folder (http://forums.asp.net/t/1303052.aspx?Tell+IIS+to+load+dll+from+another+directory+not+Bin+web+config+) might be where my solution lies, but I don't think it is because of the fact that this all worked on 2.6 without doing that to IIS.
IIS seems to allow python to import any module that is just a python script. As soon as it gets to a *.pyd (basically just python's version of a dll file) file, it screams. I'm no pro when it comes to DLLs and windows environments, but wouldn't IIS have to have paths to a bin folder of some kind? Do I have to manually edit them, as discussed in the last link above?
ACTUAL ERROR Details below for DLL failed Load:
The Error :
" HTTP Error 502.2 - Bad Gateway The specified CGI application
misbehaved by not returning a complete set of HTTP headers. The
headers it did return are "Traceback (most recent call last): File
"\estorage.equitable.int\riskmgmt\Quants\web\LinksPage.py", line 2,
in import pyweb File
"\estorage.equitable.int\riskmgmt\Quants\Common2014\Python\pyweb__init__.py",
line 5, in from core import * File
"\estorage.equitable.int\riskmgmt\Quants\Common2014\Python\pyweb\core.py",
line 2, in from pylib import pgdb File
"\estorage.equitable.int\riskmgmt\Quants\Common2014\Python\pylib\pgdb.py",
line 8, in from scikits import timeseries as ts File
"C:\WinPython-32bit-2.7.6.2-20140401\python-2.7.6\lib\site-packages\scikits.timeseries-0.91.3-py2.7-win32.egg\scikits\timeseries__init__.py",
line 13, in import const File
"C:\WinPython-32bit-2.7.6.2-20140401\python-2.7.6\lib\site-packages\scikits.timeseries-0.91.3-py2.7-win32.egg\scikits\timeseries\const.py",
line 79, in from cseries import freq_constants ImportError:
DLL load failed: The specified module could not be found. ".
I'm confident that the python environment is configured properly, as the script runs from the same executable (python.exe) via a command line. I'm thinking that I don't have IIS configured properly, for the new Python 2.7 install. The same script worked yesterday, on IIS and python 2.6. But during our upgrade from 2.6 to 2.7, a bunch of PATH and PYTHONPATH parameters all changed, plus we went from ActivePython to WinPython. WinPython is "registered" on the machine.
What I've tried
confirming python's sys.path is as expected at run-time in both IIS and command line - it is.
using the module from python command line.
recompiling the failing module using two different compilers (ming32 and VS2008).
putting duplicates of my new 2.7 modules in the old python26 folder.
pulling out lots of hair and other hacky stuff.
My next step, is to post this same message on a python forum. If anybody can advise on a good one for python-IIS related challenges, that would be appreciated.
Please help! Thanks in advance.

I got this 502.2 error when doing a clean installation of PHP 5.5 in Windows Server 2012 R2 with IIS 8.5.
It turns out PHP is a Visual C++ application which needs the library MSVCR110.dll in order to run properly. My computer does not have Visual Studio 2012 installed and thus it does not have this file. I got my problem solved by installing the Visual C++ Redistributable Packages https://www.microsoft.com/en-us/download/details.aspx?id=30679#

(Note: jc77 is my associate, and I'm actually the OP, as this was an x-post from IIS forums.)
We solved the problem.
tl,dr; portable python + sloppy/rookie compiling = strange behaviour + frustrations.
Bottom line, compile properly. For scikits.timeseries, using ming32 everything will walk, talk, and sound like it works in Spyder.exe, but not in python.exe. You have to use VS2008, if you want it to work in both.
More Info:
Winpython (as well as others) presents itself as identical to any other python installations, if you "register" the installation. It works great, 99% of the time. We learned the hard way, that "Winpython Interpreter.exe" and "python.exe" provided in the install are in fact different. Can't explain why, but the two executables gave different behavior. We were doing all our testing in Spyder, which must use "winpython interpreter.exe". The module which IIS couldn't find, would import and run no problem in Spyder. Then, in IIS, using python.exe, the module wouldn't import. We were operating on the assumption that the IDE would use python.exe, and that the stack was identical. As, 99% of the time, they appear to be. The way we were compiling scikits worked in winpython interpreter.exe. We were making a rookie mistake when compiling scikits, but it went un-noticed because it was working fine in our IDE (Spyder).
I'm adding these keywords for others : Anybody else who receives errors like this is likely using a portable python installation AND not compiling something properly. Winpython, Portable Python, eGenix, [and possibly?] Active State and Enthought Canopy.

While trying to configure CGI to run Perl in Windows 8.1, I had HTTP Error 502.2, but then I read loste's post and solved the problem. I had previously installed both Perl64 and Strawberry Perl. Although the IIS EventHandler pointed to only the Perl64 directory, both directories appeared in my Windows PATH variable. I prefer Strawberry Perl, so I changed the EventHandler to point to the Strawberry Perl directory and deleted the paths to Perl64 from the Windows PATH variable to solve the error.

Try this
print("Content-Type: text/html\n")
print("Hello Python World!")
You must specify the type of document

syntax error during py2app build

New to programming (so please bear with me), but loving it so far. I coded a game using pygame and am having trouble compiling it as a stand-alone application using py2app. I'm using Macports Python 2.7, though I tried switching back to the default Mac installation (2.7) as well as the default 2.6 and still get this error during the py2app build:
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sympy/mpmath/libmp/exec_py3.py", line 1
exec_ = exec
^
SyntaxError: invalid syntax
Exec_py3.py consists of all of one line:
exec_ = exec
Just for laughs I commented it out and the py2app proceeded further along in the build but then choked here:
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/macholib/MachO.py", line 304, in synchronize_size
raise ValueError("New Mach-O header is too large to relocate")
ValueError: New Mach-O header is too large to relocate
I don't know if the issues are related. Py2app is working fine for a couple little test scripts I wrote that don't use pygame. Any suggestions for what I can try next?
Thanks!
edit - I found a couple of links that may be related, but can't really understand what's going on in the conversation. Can anyone translate for a relative newby?
http://code.google.com/p/mpmath/issues/detail?id=204
https://bitbucket.org/ronaldoussoren/py2app/issue/93/mach-o-header-may-be-too-large-to-relocate

Well, I found a workaround in case anybody else runs into this. I uninstalled MacPorts and just used the Mac system Python (2.7). Everything then compiled fine. It was not enough just to use the port select command to switch to the system Python; I had to deinstall the whole thing.

Installing Boost Python on Windows 8.1, Setting up the toolchain correctly

I am trying to install Boost Python, however http://www.boost.org/doc/libs/1_55_0/libs/python/doc/tutorial/doc/html/python/hello.html is too compressed for me. I am switchig to Windows from Linux, setting up all the technical stuff drives me crazy :)
So when I follow the instructions http://www.boost.org/doc/libs/1_55_0/more/getting_started/windows.html#prepare-to-use-a-boost-library-binary and try to execute bootstrap.bat it complains about a missing VCVARS32.BAT and it does not recognice the "cl" command - I have however both MS VS Express 2012 and 2013 installed, both boost versions (32, 64) are installed in C:/local/, documentations says that both should work. So why does it not find the compiler?
Update: So http://msdn.microsoft.com/en-us/library/9s7c9wdw%28v=vs.110%29.aspx says that you cannot use the compiler from the command line... how do I build my bjam then? This must be something totally trivial and stupid... :/
Update: Okay, one step further. vcvars32.bat actually sets all environment variables to go, so i can invoke the command line compiler now! Also there are actually prebuild binaries of bjam from the official boost site http://sourceforge.net/projects/boost/files/, I got it and copied it to by boost_xxx/tools folder and added the site to the path, so i can invoke bjam from command line as well.
Next I put a user-config.jam in my home folder, it only says:
# MSVC configuration
using msvc : 11.0 ;
# Python configuration
using python : : C:/Users/me/Anaconda ;
In the Jamroot file provided with C:\local\boost_1_55_0\libs\python\example\tutorial, I only edited
use-project boost
: C:/local/boost_1_55_0 ;
then I call bjam from the very same directory which gives my some cryptic output
C:\local\boost_1_55_0\libs\python\example\tutorial>bjam
warning: mismatched versions of Boost.Build engine and core
warning: Boost.Build engine (bjam) is 03.1.18
warning: Boost.Build core (at C:/local/boost_1_55_0/tools/build/v2) is 2011.12-s
vn
link.jam: No such file or directory
C:/local/boost_1_55_0/tools/build/v2/util\path.jam:458: in path.makedirs
rule MAKEDIR unknown in module path.
C:/local/boost_1_55_0/tools/build/v2/build\configure.jam:233: in configure.set-l
og-file
C:/local/boost_1_55_0/tools/build/v2\build-system.jam:695: in load
C:\local\boost_1_55_0\libs\python\example\..\..\..\tools\build\v2/kernel\modules
.jam:289: in import
C:\local\boost_1_55_0\libs\python\example\..\..\..\tools\build\v2\kernel\bootstr
ap.jam:139: in boost-build
C:\local\boost_1_55_0\libs\python\example\boost-build.jam:7: in module scope
C:\local\boost_1_55_0\libs\python\example\tutorial>
Now, can you give me any hints as to what went wrong and where to go from here? I really want a stable, robust working copy of boost.python - it doesn't make sense to me if I need to hack something together. I thought it was quite standard/reliable, isn't there something like "boost.python for guys who did not study computer science"? Can I ignore the warnings?

Perhaps you can use the binary from http://www.lfd.uci.edu/~gohlke/pythonlibs/#boost.python
Regarding your compiler problems: there is a bunch of questions on this on SO already. Basically, the compiler you need depends on your version of python. You will probably need to install some Windows SDK and setup some stuff manually. It has always been a pain in the ass for me and in the end you might be better of with using some python distribution like Canopy (there are a few others out there).