Develop Reference

Getting "bad escape" when using nltk in py3

NLTK version 3.4.5. Python 3.7.4. OSX version 10.14.5.
Upgrading the codebase from 2.7, started running into this issue just now. I've done a fresh no-cache reinstall of all packages and extensions, in a fresh virtualenv. Pretty mystified as to how this could be happening to only me and I can't find anyone else having the same error online.
(venv3) gmoss$ python
Python 3.7.4 (default, Sep 7 2019, 18:27:02)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/gmoss/Documents/constructor/autocomplete/venv3/lib/python3.7/site-packages/nltk/__init__.py", line 150, in <module>
from nltk.translate import *
File "/Users/gmoss/Documents/constructor/autocomplete/venv3/lib/python3.7/site-packages/nltk/translate/__init__.py", line 23, in <module>
from nltk.translate.meteor_score import meteor_score as meteor
File "/Users/gmoss/Documents/constructor/autocomplete/venv3/lib/python3.7/site-packages/nltk/translate/meteor_score.py", line 10, in <module>
from nltk.stem.porter import PorterStemmer
File "/Users/gmoss/Documents/constructor/autocomplete/venv3/lib/python3.7/site-packages/nltk/stem/__init__.py", line 29, in <module>
from nltk.stem.snowball import SnowballStemmer
File "/Users/gmoss/Documents/constructor/autocomplete/venv3/lib/python3.7/site-packages/nltk/stem/snowball.py", line 314, in <module>
class ArabicStemmer(_StandardStemmer):
File "/Users/gmoss/Documents/constructor/autocomplete/venv3/lib/python3.7/site-packages/nltk/stem/snowball.py", line 326, in ArabicStemmer
r'[\u064b-\u064c-\u064d-\u064e-\u064f-\u0650-\u0651-\u0652]'
File "/Users/gmoss/Documents/constructor/autocomplete/venv3/lib/python3.7/re.py", line 234, in compile
return _compile(pattern, flags)
File "/Users/gmoss/Documents/constructor/autocomplete/venv3/lib/python3.7/re.py", line 286, in _compile
p = sre_compile.compile(pattern, flags)
File "/Users/gmoss/Documents/constructor/autocomplete/venv3/lib/python3.7/sre_compile.py", line 764, in compile
p = sre_parse.parse(p, flags)
File "/Users/gmoss/Documents/constructor/autocomplete/venv3/lib/python3.7/sre_parse.py", line 930, in parse
p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, 0)
File "/Users/gmoss/Documents/constructor/autocomplete/venv3/lib/python3.7/sre_parse.py", line 426, in _parse_sub
not nested and not items))
File "/Users/gmoss/Documents/constructor/autocomplete/venv3/lib/python3.7/sre_parse.py", line 536, in _parse
code1 = _class_escape(source, this)
File "/Users/gmoss/Documents/constructor/autocomplete/venv3/lib/python3.7/sre_parse.py", line 337, in _class_escape
raise source.error('bad escape %s' % escape, len(escape))
re.error: bad escape \u at position 1

The Python regular expressions dont support the \u escape, as the error message says.
It's strange though that the error comes from the nltk package. The authors of that package know for sure how to write regular expressions. Did you accidentally pick up the Python 2.7 version of the nltk package, even though it Kaminstaller in your 3.7 directory?
I expect that the nltk package has unit tests for all its code. I'd file a bug report against that package.

In case anyone else runs in to this, downgrading to 3.4.2 fixes the issue, as this is before the introduction of ArabicStemmer into the relevant file. I’ve opened an issue with nltk and hopefully it gets resolved.

To follow-up, this was a false alarm: an errant cleanup script was deleting NLTK's shared object file inside my virtual environment and I guess it was falling back to some other version.

How to compile the resources.qrc file with pyrcc5

I am trying to make a plugin for QGIS 3(I am using Windows 10) but before it is created one step before it shows this message :
"The resource compiler pyrcc5 was not found in your path. You'll have to manually compile the resources .qrc file with pyrcc5 before installing your plugin".
This is a setback because after setting a plugin path in pyqgis it deploys but the plugin says it cannot find the class module.
I have managed to make a plugin but it says error when calling its classFactory() method thus i think this is because I fail to compile the pyrcc5.
It brings this error upon after selecting it couldn't load plugin 'remove_feature' due to an error when calling its classFactory() method :
ModuleNotFoundError: No module named 'remove_feature.resources'
Traceback (most recent call last):
File "C:/PROGRA~1/QGIS3~1.4/apps/qgis-ltr/./python\qgis\utils.py", line 335, in startPlugin
plugins[packageName] = package.classFactory(iface)
File "C:/Users/Arnold Kilaini M/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\remove_feature\__init__.py", line 35, in classFactory
from .Remove_feature import Remove_feature
File "C:/PROGRA~1/QGIS3~1.4/apps/qgis-ltr/./python\qgis\utils.py", line 672, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "C:/Users/Arnold Kilaini M/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\remove_feature\Remove_feature.py", line 29, in
from .resources import *
File "C:/PROGRA~1/QGIS3~1.4/apps/qgis-ltr/./python\qgis\utils.py", line 672, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'remove_feature.resources'
Python version: 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)]
QGIS version: 3.4.5-Madeira Madeira, 89ee6f6e23
Python Path:
C:/PROGRA~1/QGIS3~1.4/apps/qgis-ltr/./python
C:/Users/Arnold Kilaini M/AppData/Roaming/QGIS/QGIS3\profiles\default/python
C:/Users/Arnold Kilaini M/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins
C:/PROGRA~1/QGIS3~1.4/apps/qgis-ltr/./python/plugins
C:\Program Files\QGIS 3.4\bin\python37.zip
C:\PROGRA~1\QGIS3~1.4\apps\Python37\DLLs
C:\PROGRA~1\QGIS3~1.4\apps\Python37\lib
C:\Program Files\QGIS 3.4\bin
C:\PROGRA~1\QGIS3~1.4\apps\Python37
C:\PROGRA~1\QGIS3~1.4\apps\Python37\lib\site-packages
C:\PROGRA~1\QGIS3~1.4\apps\Python37\lib\site-packages\win32
C:\PROGRA~1\QGIS3~1.4\apps\Python37\lib\site-packages\win32\lib
C:\PROGRA~1\QGIS3~1.4\apps\Python37\lib\site-packages\Pythonwin
C:/Users/Arnold Kilaini M/AppData/Roaming/QGIS/QGIS3\profiles\default/python
C:\Users\Arnold Kilaini M\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins\mmqgis/forms

From the OSGeo4W Shell, run qt5_env.bat and py3_env.bat.
Then, change the directory to
C:\Users\Arnold Kilaini M\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins\mmqgis".
Run pyrcc5 -o resources.py resources.qrc
Also, I found the link to a gis.stackexchange question related to yours:
Hope this helps!

With this .bat I don't have any problem aymore.
#echo off
SET QGIS_ROOT=C:\GIS\QGIS
call "%QGIS_ROOT%"\bin\o4w_env.bat
call "%QGIS_ROOT%"\apps\grass\grass78\etc\env.bat
path %PATH%;%QGIS_ROOT%\apps\qgis\bin
path %PATH%;%QGIS_ROOT%\apps\grass\grass78\lib
path %PATH%;C:\GIS\QGIS\apps\Qt5\bin
path %PATH%;C:\GIS\QGIS\apps\Python37\Scripts
set PYTHONPATH=%PYTHONPATH%;%QGIS_ROOT%\apps\qgis\python
set PYTHONPATH=%PYTHONPATH%;%QGIS_ROOT%\apps\Python37\lib\site-packages
set PYTHONHOME=%QGIS_ROOT%\apps\Python37
set QT_PLUGIN_PATH=%QGIS_ROOT%\apps\qgis\qtplugins;%QGIS_ROOT%\apps\qt5\plugins
set PATH=C:\Program Files\Git\bin;%PATH%
cmd.exe

for Ubuntu users, i tried this and it didn't work out:
pb_tool compile
also tied to install python-qt5 and Ubuntu 20.04 couldn't find it, then used
sudo apt install pyqt5-dev-tool
then pb_tool compile and it worked.

Can't install pygame_sdl2

I'm creating a tower defense game using python and pygame. I would like to add this one on the Google Play Store, but I need to create an apk file.
I'm following a tutorial on GitHub:
https://github.com/renpytom/rapt-pygame-example.
I follow all the instructions but can't install pygame_sdl2.
I use the command prompt, and use $ python setup.py install in the appropriate directory.
But it gave me this error :
C:\Users\gomes\Downloads\pygame_sdl2-master>python setup.py install
Traceback (most recent call last):
File "setup.py", line 59, in <module>
parse_cflags([ "sh", "-c", "sdl2-config --cflags" ])
File "C:\Users\gomes\Downloads\pygame_sdl2-master\setuplib.py", line 93, in parse_cflags
output = subprocess.check_output(command, universal_newlines=True)
File "C:\Python27\lib\subprocess.py", line 212, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "C:\Python27\lib\subprocess.py", line 390, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 640, in _execute_child
startupinfo)
WindowsError: [Error 2] Le fichier spÚcifiÚ est introuvable
I tried to reinstall pygame_sdl2-master, but gave me the same result.

You need the appropriate Windows dependencies and cython. I did some researching and testing and found this solution:
Install cython, type this in a terminal:
$ pip install cython
Go to your pygame_sdl2 directory and download the Windows dependencies by typing:
$ git clone https://github.com/renpy/pygame_sdl2_windeps
If it's not in the pygame_sdl2 directory, just move pygame_sdl2_windeps there manually.
Start building the library:
$ python setup.py install

I directly download pygame_sdl2_windeps on github. I have copied this one one the pygame_sdl2 directory and use the "python setup.py install" command in the command prompt, and IT WORKS !!!
C:\PGS4A\pygame_sdl2>python
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygame_sdl2
>>>
Thank you for your help !!!

How can I use virtualenv to use 32-bit and 64-bit Python in Windows?

I have 64-bit Python (2.7.5) installed at C:\Python27 and 32-bit Python at C:\Python27_32.
I would like to use virtualenv to set up a 32-bit virtual environment that I can switch into when I need to use 32-bit Python. Once that environment is set up, I plan to edit the bin\activate file to change all the necessary paths to point to the 32-bit directories.
However, when I try to create the virtual environment, I get the following error:
> virtualenv --python=C:\Python27_32\python.exe foo
Running virtualenv with interpreter C:\Python27_32\python.exe
PYTHONHOME is set. You *must* activate the virtualenv before using it
New python executable in foo\Scripts\python.exe
Installing setuptools...............
Complete output from command C:\Users\<user>\Drop...o\Scripts\python.exe -c "#!python
\"\"\"Bootstra...sys.argv[1:])
" C:\Python27\lib\site...ols-0.6c11-py2.7.egg:
Traceback (most recent call last):
File "<string>", line 278, in <module>
File "<string>", line 238, in main
File "build/bdist.linux-i686/egg/setuptools/command/easy_install.py", line 21, in <module>
File "build/bdist.linux-i686/egg/setuptools/package_index.py", line 2, in <module>
File "C:\Python27\Lib\urllib2.py", line 94, in <module>
import httplib
File "C:\Python27\Lib\httplib.py", line 71, in <module>
import socket
File "C:\Python27\Lib\socket.py", line 47, in <module>
import _socket
ImportError: DLL load failed: %1 is not a valid Win32 application.
----------------------------------------
...Installing setuptools...done.
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\virtualenv.py", line 2577, in <module>
main()
File "C:\Python27\lib\site-packages\virtualenv.py", line 979, in main
no_pip=options.no_pip)
File "C:\Python27\lib\site-packages\virtualenv.py", line 1091, in create_environment
search_dirs=search_dirs, never_download=never_download)
File "C:\Python27\lib\site-packages\virtualenv.py", line 611, in install_setuptools
search_dirs=search_dirs, never_download=never_download)
File "C:\Python27\lib\site-packages\virtualenv.py", line 583, in _install_req
cwd=cwd)
File "C:\Python27\lib\site-packages\virtualenv.py", line 1057, in call_subprocess
% (cmd_desc, proc.returncode))
OSError: Command C:\Users\<user>\Drop...o\Scripts\python.exe -c "#!python
\"\"\"Bootstra...sys.argv[1:])
" C:\Python27\lib\site...ols-0.6c11-py2.7.egg failed with error code 1
It seems to be doing imports in the 64-bit folder instead of in the 32-bit folder. I'm not sure if it's because of the way my environment variables are set up, or because I installed virtualenv under 64-bit Python in the first place.
These are my user environment variables:
Path: %PYTHONHOME%;C:\Python27\Scripts
PYTHONHOME: C:\Python27
PYTHONPATH: C:\Python27\Lib;C:\Python27\Lib\lib-tk;C:\Python27\DLLs;
But if I change every C:\Python27 to C:\Python27_32 in my environment variables, then I can't virtualenv to run (ImportError: No module named pkg_resources).
This is my first time messing with virtualenv, so I'm sure I'm missing something basic. How can I create a virtual environment that uses my 32-bit Python installation?

For your virtual env to run after you have changed your paths you will need to install virtualenv into the 32 bit python - there is nothing stopping you having a copy of virtualenv in each python.
Assuming you have python 2.7.c 64-bit as your default python and you have also installed python 2.7.x 32-bit you would need both anyway - also assuming that you are on windows your two pythons will be installed somewhere like:
C:\Python27 and C:\Python27_64
With the latter on your path.
Also assuming that you have pip installed in both, you will need it for virtualenv anyway - to install virtualenv to the 32 bit python you can either run:
Path\To\32Bit\pip install virtualenv
or
set path=C:\Python27;C:\Python27\Scripts;%path%
rem The above should set your 32 bit to be found before your 64 bit
pip install virtualenv

If you installed the 32Bit version first, and install the 64Bit version second (And you added python to path), Then you can use the updated python launcher (py) to create the 64Bit version of your virtualenv
py -m venv my-env-name

in command prompt Use:
set CONDA_FORCE_32BIT=1
conda create -n virtualenv_name python=x.x anaconda
The above can be found # How to install win-32 package on a 64-bit system with conda install I personally tried it and it worked successfully (32-bit python x.x installed). Using Anaconda is not necessary but it will install all of the anaconda packages like pandas.
Disclaimer: The comment below is a caution and not an attack on the answer submitted by anyone else.
As my friend told me "changing the path manually is discouraged, mostly because you'd have to inform other applications that use Python (and that you do not necessarily know of) to point to a different folder, and changes are not certain to be consistent across your system. In a way, virtualenvs do the same but in a cleaner and (sort of) predictable fashion".

Python help('modules') command throws WindowsError in IDLE (idle.pyw) but not command line prompt (python.exe)

If I type the following at the python.exe command prompt...
>>>help('modules')
...I get a list of available modules as expected.
However when I type the same thing in the IDLE environment (idle.pyw), it throws an error:
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
help('modules')
File "C:\Python27\ArcGIS10.1\lib\site.py", line 467, in __call__
return pydoc.help(*args, **kwds)
File "C:\Python27\ArcGIS10.1\lib\pydoc.py", line 1727, in __call__
self.help(request)
File "C:\Python27\ArcGIS10.1\lib\pydoc.py", line 1766, in help
elif request == 'modules': self.listmodules()
File "C:\Python27\ArcGIS10.1\lib\pydoc.py", line 1887, in listmodules
ModuleScanner().run(callback, onerror=onerror)
File "C:\Python27\ArcGIS10.1\lib\pydoc.py", line 1938, in run
for importer, modname, ispkg in pkgutil.walk_packages(onerror=onerror):
File "C:\Python27\ArcGIS10.1\lib\pkgutil.py", line 105, in walk_packages
for importer, name, ispkg in iter_modules(path, prefix):
File "C:\Python27\ArcGIS10.1\lib\pkgutil.py", line 147, in iter_modules
for name, ispkg in iter_importer_modules(i, prefix):
File "C:\Python27\ArcGIS10.1\lib\pkgutil.py", line 211, in iter_modules
for fn in os.listdir(path):
WindowsError: [Error 5] Access is denied: 'C:\\WINDOWS\\system32\\Msdtc/*.*'
Why does Windows say "Access is denied" to IDLE but not the command line? I have tried using "Run as Administrator" on IDLE without success. Do I need to reconfigure IDLE in some way?
For information:
My installation of Python 2.7.2 (32-bit version running on a Windows 7 64-bit machine) came bundled with ArcGIS 10.1, which uses Python as a scripting language and certain numerical operations (via Numpy). Could this have something to do with the differing behaviour in the prompt and IDLE?
The root Python directory is: C:\Python27\ArcGIS10.1\.
IDLE is installed at C:\Python27\ArcGIS10.1\Lib\idlelib\idle.pyw.
The Python command prompt is installed at: C:\Python27\ArcGIS10.1\python.exe.

I just successfully ran help('modules') in both 2.7.8 and 3.4.2. The response depends on sys.path. It appears than ArcGIS adds C:/Windows/System32/Msdtc to the path, but you could check before running help. In any case, accessing that folder cannot be done through Windows Explorer, even when running as an admin. It requires special user action through a 'security tab'. Since MS does not want me poking around there, I did not pursue getting access. I don't know why the difference betweeon console and Idle.