Develop Reference

PyWin32 (226) and virtual environments

[PyPI]: pywin32 226 has been released on 20191110. It works on most of the Python installations (e.g. works on the official versions downloaded from Python), but not on virtual environments (e.g. created with VirtualEnv (v16.7.7), and (based on further research) Python's venv). I used Python v3.8.0 and v3.7.3 as lab rats. Here's the output for the former:
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q058805040]> "e:\Work\Dev\VEnvs\py_064_03.08.00_test0\Scripts\python.exe" -c "import win32api"
Fatal Python error: init_import_size: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
File "e:\Work\Dev\VEnvs\py_064_03.08.00_test0\lib\site.py", line 769, in <module>
main()
File "e:\Work\Dev\VEnvs\py_064_03.08.00_test0\lib\site.py", line 746, in main
paths_in_sys = addsitepackages(paths_in_sys)
File "e:\Work\Dev\VEnvs\py_064_03.08.00_test0\lib\site.py", line 279, in addsitepackages
addsitedir(sitedir, known_paths)
File "e:\Work\Dev\VEnvs\py_064_03.08.00_test0\lib\site.py", line 202, in addsitedir
addpackage(sitedir, name, known_paths)
File "e:\Work\Dev\VEnvs\py_064_03.08.00_test0\lib\site.py", line 170, in addpackage
exec(line)
File "<string>", line 1, in <module>
File "e:\Work\Dev\VEnvs\py_064_03.08.00_test0\lib\site-packages\win32\lib\pywin32_bootstrap.py", line 14, in <module>
for maybe in site.getsitepackages():
AttributeError: partially initialized module 'site' has no attribute 'getsitepackages' (most likely due to a circular import)

Note: I branched this answer (and also the question) from [SO]: PyWin32 and Python 3.8.0 (#CristiFati's answer), as it's a different issue. You might want to check that one before going further.
After some digging, it turns out it's a VirtualEnv bug (or at least, that's how I see things, because VirtualEnv's site.py doesn't contain getsitepackages - although it was present in Python's site.py since v2.7). There are several issues (that were) open revolving this absence (e.g. [GitHub]: pypa/virtualenv - site.getsitepackages() missing), but they appear to be closed without a fix (many of them due to inactivity). Also, [GitHub]: [WIP] The next-gen virtualenv (rewrite) which is a big refactor, doesn't seem to address it.
Recap: this error (in VirtualEnv context):
Also applies to other Python versions (I can confirm for v3.7.3 (32bit))
Happens every time when the interpreter starts
In conclusion, do not install PyWin32 226 in (VirtualEnv) virtual environments, as they will end up in a broken state!
PyWin32 (official .whls) "compatibility table" (didn't check Anaconda (or other such tools)):
"Normal" (official) Python installations:
v226 is OK
VirtualEnvs (and also Python's standard venvs):
v225 is OK
For Python 3.8.0, [GitHub]: CristiFati/Prebuilt-Binaries - (master) Prebuilt-Binaries/PyWin32/v225 should be used
Submitted [GitHub]: mhammond/pywin32 - Workaround for virtual environments (VirtualEnv) (merged on 20191114). Applying the changes locally (check the referenced question for details on how to do it), fixed it:
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q058805040]> sopr.bat
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###
[prompt]> ".\venv_py_064_030800\Scripts\python.exe" -m pip list
Package Version
---------- -------
pip 19.3.1
pywin32 226
setuptools 41.6.0
wheel 0.33.6
[prompt]> ".\venv_py_064_030800\Scripts\python.exe" -c "import win32api"
[prompt]>
Update #0
[PyPI]: pywin32 227 (which addresses this issue), was published on 20191114!

Shipping PyGObject/GTK+ app on Windows with MingW

I'm using PyGObject/GTK+ with Python3.7 on Windows and is worried about how to package my app. With official Python ecosystem, it's easy to use freezers and venvs. However, the PyGObject and GTK+ on Windows requires the MingW environment to run.
Does this mean that I have to package the entire MingW, along with GTK+?
How should I freeze the app?

Since no one cares soon enough, I'll report my findings here. The short answer, as of the date I write this post, is: The PyGObject eco-system is not commercial-ready on Windows, although nothing stops you from shipping a functional app.
The deal-breaker is the lack of functional freezers on Windows for PyGOBject. So you can't really ship a closed-source commercial product with it (although one could argue that the entire Python eco-system is not made for that). The freezers that I tried:
The officially recommended PyInstaller right now does not install with the officially recommended msys/MingGW universe; Even after building PyInstaller from source, the frozen .exe crashes with complaints about missing modules. Below is the crash report from a hello-world app.
$ app.exe
[8340] mod is NULL - structTraceback (most recent call last):
File "C:/Apps/msys64/mingw64/lib/python3.7/struct.py", line 13, in <module>
from _struct import *
ModuleNotFoundError: No module named '_struct'
[8340] mod is NULL - pyimod02_archiveTraceback (most recent call last):
File "C:/Apps/msys64/mingw64/lib/python3.7/site-packages/pyinstaller-3.5-py3.7.egg/PyInstaller/loader/pyimod02_archive.py", line 28, in <module>
import struct
ModuleNotFoundError: No module named 'struct'
[8340] mod is NULL - pyimod03_importersTraceback (most recent call last):
File "C:/Apps/msys64/mingw64/lib/python3.7/site-packages/pyinstaller-3.5-py3.7.egg/PyInstaller/loader/pyimod03_importers.py", line 24, in <module>
from pyimod02_archive import ArchiveReadError, ZlibArchiveReader
ModuleNotFoundError: No module named 'pyimod02_archive'
Traceback (most recent call last):
File "site-packages/pyinstaller-3.5-py3.7.egg/PyInstaller/loader/pyiboot01_bootstrap.py", line 15, in <module>
ModuleNotFoundError: No module named 'pyimod03_importers'
[8340] Failed to execute script pyiboot01_bootstrap
cx_Freeze, another prominent alternative, does not build/install on MingW, either. You end up with gcc woes, no matter how many gcc variants you install.
py2exe is now abandonware since Python 3.4.
Since a PyGObject app still runs with MinGW python, it's not impossible to ship such an app for Windows. But that means you'll go with the big baggage and figure out a way to pack everything yourself. Let alone maintain the whole mess. Mind you that the procedure will be different on Mac if your app is cross-platform. PyInstaller works on Mac according to my test.
QuodLibet provides an example of how to ship an open-source software based on PyGObject. Unfortunately, as of today, the upstream's packaging procedure fails at the build.sh run on my Windows 10:
copying data/quodlibet.zsh -> E:\_dev\quodlibet\win_installer\_build_root\mingw64\share\zsh\vendor-completions\_quodlibet
Traceback (most recent call last):
File "E:/_dev/quodlibet/win_installer/misc/depcheck.py", line 141, in <module>
main(sys.argv)
File "E:/_dev/quodlibet/win_installer/misc/depcheck.py", line 130, in main
libs = get_things_to_delete(sys.prefix)
File "E:/_dev/quodlibet/win_installer/misc/depcheck.py", line 108, in get_things_to_delete
for lib in get_dependencies(path):
File "E:/_dev/quodlibet/win_installer/misc/depcheck.py", line 66, in get_dependencies
data = data.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 72: invalid start byte
This turns out to be an easy fix: locale issue on non-UTF8 Windows.
A trivial patch would be replacing "utf-8" with locale.getdefaultlocale(). After that, the build succeeds.
However, the global distribution issues remain challenging:
Bloated pipeline: The officially recommended msys2 build system basically drags the entire GNOME universe and much of the Python universe to your machine, while you may already have a Python environment on your native Windows. It proves hard to cherry-pick the necessary dependencies among this whole pile of stuff.
Extra C pipeline required to build the executable. To build a Windows executable without a freezer, you must create your app as a Python package (big hassle), install it to your msys2-Python site-packages (another setup.py hassle), and then build a wrapper C application using the Python C API (opaque hassle). The process is not transparent, especially when Python packaging system is sensitive to working directory and path differences. Launching the executable can easily go wrong without much debug info.
Manual stripping. As mentioned above, you must manually cherry-pick dependencies or your app would be shipped with basically the entire msys2/MinGW universe that's worth 100MB+ for a hello-world app.

How to install ortools library on Google Cloud

I am using the command pip install -t lib/ ortools , the library ortools is installed to the lib/ folder. But when I deploy my Flask project that contains that library on the Google App Engine, I got the following error:
(/base/alloc/tmpfs/dynamic_runtimes/python27g/931d17f05408b3ef/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py:263)
Traceback (most recent call last):
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/931d17f05408b3ef/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/931d17f05408b3ef/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/931d17f05408b3ef/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/base/data/home/apps/b~cidy-1539116366694/20190316t002011.416796594015344313/main.py", line 5, in <module>
from ortools.constraint_solver import pywrapcp
File "/base/data/home/apps/b~cidy-1539116366694/20190316t002011.416796594015344313/lib/ortools/constraint_solver/pywrapcp.py", line 17, in <module>
_pywrapcp = swig_import_helper()
File "/base/data/home/apps/b~cidy-1539116366694/20190316t002011.416796594015344313/lib/ortools/constraint_solver/pywrapcp.py", line 16, in swig_import_helper
return importlib.import_module('_pywrapcp')
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/931d17f05408b3ef/python27/python27_dist/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named _pywrapcp

The 1st generation (Python 2.7) standard environment sandbox has very strict requirements. Particularly relevant in this context is the Pure Python one:
All code for the Python runtime environment must be pure Python, and
not include any C extensions or other code that must be compiled.
The OR-tools fail to meet this requirement since they require (platform-specific) compilation. From their installation page:
Note: you can build OR-Tools suite from source for any supported platform only from that platform. OR-Tools Makefile does not support
cross-compiling for any supported platform.
You might still be able to use them on GAE:
in the 2nd generation standard environment (Python 3.7, more relaxed restrictions) - but I'm not certain if pip-driven package builds are supported and if all the tools required for it are provided, YMMV
in the flexible environment, most likely using a custom-built runtime which allows you to add even non-python dependencies - the extra libraries and tools you might need for building ortools.

Do you run pip on macOS or Linux ? If build on macOS, please see Using Homebrew Python on macOS?
According to the official web of OR-Tools Python, I think ortools depends on the platform.
When you use platform dependency python lib, it's better use pip install -t lib/ ortools.
on linux environment.

python - pyinstaller "RuntimeWarning: Parent module 'PyInstaller.hooks.hook-PIL' not found while handling absolute import" and "tcl" related errors

I get a warning message while trying to create exectable file using pyinstaller. This warning appeared after installing Pillow. Previously i nevre got any warnings and was able to make it through.
the warning i get by pyinstaller is:
7314 INFO: Analyzing main.py
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/hooks/hook-PIL.Image.py:14: RuntimeWarning: Parent module 'PyInstaller.hooks.hook-PIL' not found while handling absolute import
from PyInstaller.hooks.shared_PIL_Image import *
Also when i tried to run the executable's exe/consol version of my code that lies inside the dist folder created by the pyinstaller (dist/main/main), these are displayed..
Traceback (most recent call last):
File "<string>", line 26, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/loader/pyi_importers.py", line 276, in load_module
exec(bytecode, module.__dict__)
File "/Users/..../build/main/out00-PYZ.pyz/PIL.PngImagePlugin", line 40, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/loader/pyi_importers.py", line 276, in load_module
exec(bytecode, module.__dict__)
File "/Users/..../build/main/out00-PYZ.pyz/PIL.Image", line 53, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/loader/pyi_importers.py", line 276, in load_module
exec(bytecode, module.__dict__)
File "/Users/..../build/main/out00-PYZ.pyz/FixTk", line 74, in <module>
OSError: [Errno 20] Not a directory: '/Users/.../dist/main/tcl'
logout
[Process completed]
so, i tried by uninstalling pillow, installing tk tcl dev version. And then installed pillow. Even that didnt helped.
I also tried reinstalling pyinstaller,. didnt help too
Update 1:
It seems Pyinstaller.hooks.hook-PIL.py file was missing in the Pyinstaller/hooks directory. And it was missing on all platforms(Mac, windows and linux). This is the warning/error message that i get on windows, which is the same i got on mac and on linux.
Later i found a link which said, its just to need Python import machinery happy. so i created as said so. Then i dont get the same error on all platforms, But on mac i still get the PILImagePlugin,Image and FixTk errors

Solution for tcl:
I found what was going wrong,.. Every problem that i faced on OSX was the OS itself(exactly the macport). Python by default comes with the mac OS. And this version of python may be useful for just learning basic python, but is not suitable for Development purpose.
Installing brew's python helped. I followed this SO link. After doing these i was still getting errors. Later i had to change the paths on /etc/paths. Basically rearranging them should work. But still then i wasn't getting it right.
Then i had to change the .bash_profile, which worked for most users, But still i was getting mac's version of python and pip, not the brews version of python.
Finally i had to restart the machine for a couple of times and do the /etc/paths and .bash_profile steps repeatedly to get the system wide effect to accept brews version of python and pip
Solution for PIL:
just adding a file called hook-PIL.py with an empty content would serve the purpose. I found a link which was having the hook files content of pyinstaller.
The location to create
for mac : /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyInstaller-2.1.1dev_-py2.7.egg/PyInstaller/hooks/ Actually for mac this step wouldn’t be required. When we install python through brew and change the path, everything that you try to install later either through pip install or from source packages tend to choose a different path. And everything will be taken care of.
for windows:C:\Python27\lib\site-packages\PyInstaller-2.1.1.dev0-py2.7.egg\PyInstaller\hooks
**Please check if this is a valid path on your machine before creating the file and then create the file. And im not sure or i don't know if just adding an empty file is the right way. But it worked for me

easy_install : ImportError: Entry point ('console_scripts', 'easy_install') not found

I used easy_install to install pip, pip to install django, virtualenv, and virtualenvwrapper.
I have just returned to it a few weeks later and django does not seem to work anymore, but more concerning is I can't start the process over again as easy_install is returning the following error:
Traceback (most recent call last):
File "/usr/bin/easy_install-2.7", line 10, in <module>
load_entry_point('setuptools==0.6c12dev-r88846', 'console_scripts', 'easy_install')()
File "/Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 318, in load_entry_point
File "/Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 2220, in load_entry_point
ImportError: Entry point ('console_scripts', 'easy_install') not found
After a good evening of hunting I am stumped as to how to resolve this.

You seem to have a version conflict; note the setuptools-0.6c11-py2.7.egg path, but the /usr/bin/easy_install-2.7 script wants to load 0.6c12dev-r88846 instead.
The latter is a development version; it has the revision number of a subversion repository embedded in the version (dev-r88846).
I suspect you have two python installations; one is the system version (in /System/Library and the other is installed with the python installer into /Library/, and the stub script in /usr/bin/ may be installed with the system python.
If so, there'll be another copy of the stub at /Library/Python/2.7/bin/easy_install-2.7, which should work correctly.

I had the same issue, eventually ended up running the 2.7.3 easy_install by
/opt/python2.7.3/bin/easy_install which worked fine