Develop Reference

ModuleNotFoundError: No module named 'pandas._libs.interval'

Suddenly, I can't import pandas in python. I am using anaconda as package manager, but it seems that no matter how many times I uninstall and install pandas, I still get the same error:
(base) C:\>conda install pandas
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: C:\Users\au207178\Anaconda3
added / updated specs:
- pandas
The following NEW packages will be INSTALLED:
blas pkgs/main/win-64::blas-1.0-mkl
bottleneck pkgs/main/win-64::bottleneck-1.3.2-py38h2a96729_1
intel-openmp pkgs/main/win-64::intel-openmp-2021.4.0-haa95532_3556
mkl pkgs/main/win-64::mkl-2021.4.0-haa95532_640
mkl-service pkgs/main/win-64::mkl-service-2.4.0-py38h2bbff1b_0
mkl_fft pkgs/main/win-64::mkl_fft-1.3.1-py38h277e83a_0
mkl_random pkgs/main/win-64::mkl_random-1.2.2-py38hf11a4ad_0
numexpr pkgs/main/win-64::numexpr-2.8.1-py38hb80d3ca_0
numpy pkgs/main/win-64::numpy-1.21.2-py38hfca59bb_0
numpy-base pkgs/main/win-64::numpy-base-1.21.2-py38h0829f74_0
pandas pkgs/main/win-64::pandas-1.3.5-py38h6214cd6_0
Proceed ([y]/n)? y
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(base) C:\>python
Python 3.8.12 (default, Oct 12 2021, 03:01:40) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\au207178\AppData\Roaming\Python\Python310\site-packages\pandas\__init__.py", line 22, in <module>
from pandas.compat import (
File "C:\Users\au207178\AppData\Roaming\Python\Python310\site-packages\pandas\compat\__init__.py", line 15, in <module>
from pandas.compat.numpy import (
File "C:\Users\au207178\AppData\Roaming\Python\Python310\site-packages\pandas\compat\numpy\__init__.py", line 7, in <module>
from pandas.util.version import Version
File "C:\Users\au207178\AppData\Roaming\Python\Python310\site-packages\pandas\util\__init__.py", line 1, in <module>
from pandas.util._decorators import ( # noqa
File "C:\Users\au207178\AppData\Roaming\Python\Python310\site-packages\pandas\util\_decorators.py", line 14, in <module>
from pandas._libs.properties import cache_readonly # noqa
File "C:\Users\au207178\AppData\Roaming\Python\Python310\site-packages\pandas\_libs\__init__.py", line 13, in <module>
from pandas._libs.interval import Interval
ModuleNotFoundError: No module named 'pandas._libs.interval'
I have also tried running 'conda update conda'.
What's the next thing to do?
Update:
I checked, and the file is definitely there, in "C:\Users\au207178\AppData\Roaming\Python\Python310\site-packages\pandas_libs\interval.pyx".
However, I notice that part of the path to this library says 'python310'. However, my python version is 3.8.8. Is it possible that python is somehow looking in the wrong place when trying to import pandas._libs.interval? I have tried both python 3.10 (couldn't even install pandas) and python 3.9.7 (no effect).
I have also tried conda install --revision 0, which also had no effect...

Yes, it appears to be loading pandas from a user-level installation. User-level installs can leak into Conda environments and lead to unpredictable behavior, such as what you are seeing.
There are two routes of action of which I know. You may want to try the second one first, which would confirm the cause. However, the first option is likely more manageable going forward, since once it's done, the issue should be resolved.
Option 1: Remove External Python(s)
If you would like Conda to simply work as expected, then uninstall the user-level Python. Note that the one detected (Python 3.10) may not be the only one, so you may have to track down multiple copies. I'm not on Windows, so I can't suggest concrete steps for uninstallation.
However, you may be already using this user-level Python for other projects, so this option may not be practicable.
Option 2: Launch Python with Isolation Flags
There are some pertinent flags that Python provides that has it ignore the various sources that can lead to leaking in other site-packages. Here are the three important ones:
$ python --help
usage: python [option] ... [-c cmd | -m mod | file | -] [arg] ...
Options and arguments (and corresponding environment variables):
...
-E : ignore PYTHON* environment variables (such as PYTHONPATH)
-s : don't add user site directory to sys.path; also PYTHONNOUSERSITE
-S : don't imply 'import site' on initialization
...
Try them individually, or all together, and see if this fixes the issue. Individually, you may be able to narrow down whether it is due to PYTHONPATH (-E alone would fix) or generic user-level searching (-s alone would fix).
Additional Thoughts
I find it odd that a Python 3.10 installation is leaking into a Python 3.8 interpreter run. Usually, the site module only loads within matching major+minor versions (e.g., 3.8 should only pick up another 3.8). This might be indicative that something else is going on. Perhaps the PYTHONPATH environment is set? (It really never should be.)
Generally, I don't recommend using the base environment for work. This is never documented in Anaconda documentation, but it is the painful truth that many long-term Conda users come to, usually after their base either breaks or get's so overloaded with packages that it takes unreasonably long for them to upgrade the conda package. Instead, create a new environment and have that activate by default instead of base.

RKeras "unknown url type: https" error six.urlretrieve (Python Code From R)

TL;DR :-) fetching https urls is working in both python and R, but not when python code is run from R.
While running some code in documentation of package RKeras , I have repeatedly run into the fatal error : "unknown url type: https" The problem originates in Keras which is implemented in Python.
I dug into the problem and found that this was happening when Keras was calling the function urlretrieve in the module six to retrieve data from https url.
I then tested the code in iPython and found it to be working perfectly.
<!-- language: lang-python -->
from six.moves.urllib.request import urlretrieve
urlretrieve(url="https://www.google.com")
Then I try doing the same in R and it fails
<!-- language: lang-r -->
library(reticulate)
py_run_string('from six.moves.urllib.request import urlretrieve')
py_run_string("urlretrieve(url='https://www.google.com')")
However, same thing works in R with plain http
<!-- language:lang-r -->
py_run_string("urlretrieve(url='http://www.google.com')")
For the record, https works fine within my R with packages like httr.
I am totally out of my depth here. What could be happening?
Here is some output about my env
R:
sessionInfo()
# R version 3.4.4 (2018-03-15)
# Platform: x86_64-pc-linux-gnu (64-bit)
# Running under: Ubuntu 16.04.4 LTS
# other attached packages:
# [1] reticulate_1.6 kerasR_0.8.0
And Python (as seen from R)
py_config()
version: 3.5.2 |Anaconda custom (64-bit)|
(default, Jul 2 2016, 17:53:06) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
Will much appreciate your time and effort.
Edit: Some more debug info in R:
py_run_string("from OpenSSL import SSL")
# ImportError: /xxx/python3.5/lib-dynload/_ssl.so: undefined symbol: SSLv2_method

For the time being, I am posting a partial solution to my particular problem. This is not going to solve the general problem with Reticulate and SSL, but is a decent temporary workaround to help anyone using KerasR or Keras and finding it is not able to download models and databases
Keras / KerasR uses a cache to avoid future downloads of the same object. In my case, it is ~/.keras/
So copy the URL of failed download from the console, download the object with the browser and save to keras' cache directory.
cache for models : ~/.keras/models/
cache for datasets : ~/.keras/datasets/
(where ~ is my home directory)
Of course, this is a workaround and I am still looking forward to someone posting a proper solution that works system-wide.

Using Boost.Python to build a shared lib and import it in Blender through Python

What I currently try to achieve is building a python mapping of my C++ classes through Boost.Python.
After this I want to use the resulting shared library in a blender add-on to be able to take advantage of already existing functionality coming from the mapped C++ classes.
I can already build my shared library and write sample scripts in python, which are using my library as well.
Everything fine here but the problem is that as soon as I try to use it in an add-on, Blender 2.74 keeps crashing all the time as soon as I add an import statement with this little hint in the crash report:
6 libboost_python.dylib 0x000000010aa7cc3e boost::python::detail::init_module(PyModuleDef&, void (*)()) + 30 (module.cpp:44)
In module.cpp inside of boost line 41-46:
BOOST_PYTHON_DECL PyObject* init_module(PyModuleDef& moduledef, void(*init_function)())
{
return init_module_in_scope(
PyModule_Create(&moduledef),
init_function);
}
My boost 1_58 is compiled using Python 3.4.2:
otool -L /usr/local/lib/libboost_python.dylib
/usr/local/lib/libboost_python.dylib:
libboost_python.dylib (compatibility version 0.0.0, current version 0.0.0)
/Library/Frameworks/Python.framework/Versions/3.4/Python (compatibility version 3.4.0, current version 3.4.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
Running the python3 bin from that directory gives me:
python3
Python 3.4.2 (v3.4.2:ab2c023a9432, Oct 5 2014, 20:42:22)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
The Blender Python console gives me:
PYTHON INTERACTIVE CONSOLE 3.4.2 (default, Nov 25 2014, 12:01:44) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.54)]
Command History: Up/Down Arrow
Cursor: Left/Right Home/End
Remove: Backspace/Delete
Execute: Enter
Autocomplete: Ctrl-Space
Zoom: Ctrl +/-, Ctrl-Wheel
Builtin Modules: bpy, bpy.data, bpy.ops, bpy.props, bpy.types, bpy.context, bpy.utils, bgl, blf, mathutils
Convenience Imports: from mathutils import *; from math import *
Convenience Variables: C = bpy.context, D = bpy.data
>>>
My own project uses this python version as well.
Also Blender uses this Python version.
I really don't know what to try next here since in standalone mode EVERYTHING works as expected.
Also the fact that the crash even occurs as soon as I run a new script inside Blenders Text Editor having the import statement.
Anybody having experience with Boost.Python and Blender?
Thanks in advance
UPDATE:
After building and running blender in debug mode, I got the following state:
This doesn't look right and describes the crash in the first place.
Still searching for the cause...

Blender itself (as well as oiio and osl libs) are built using boost, ensure that the version you use to build boost-python matches the one used by blender. As blender has already loaded boost libs into ram, when you import boost-python it will use the already loaded boost libs not the ones you built boost-python with, if each is a different version it can cause problems, if each is built using different options it can also cause issues.
If you are building blender yourself also build OpenImageIO and OpenShadingLanguage so they use the same boost version. By building them yourself you can ensure they all load the version that you want to use. Check your cmake or sconscript settings point to the same boost lib files.
% ldd blender | grep boost
libboost_filesystem.so.1.55.0 => /usr/local/lib/libboost_filesystem.so.1.55.0 (0x806009000)
libboost_regex.so.1.55.0 => /usr/local/lib/libboost_regex.so.1.55.0 (0x80621e000)
libboost_system.so.1.55.0 => /usr/local/lib/libboost_system.so.1.55.0 (0x806525000)
libboost_thread.so.1.55.0 => /usr/local/lib/libboost_thread.so.1.55.0 (0x806728000)
libboost_date_time.so.1.55.0 => /usr/local/lib/libboost_date_time.so.1.55.0 (0x806942000)
libboost_wave.so.1.55.0 => /usr/local/lib/libboost_wave.so.1.55.0 (0x806b4f000)
libboost_locale.so.1.55.0 => /usr/local/lib/libboost_locale.so.1.55.0 (0x806e94000)
libboost_chrono.so.1.55.0 => /usr/local/lib/libboost_chrono.so.1.55.0 (0x814ee3000)

After a lot of trial and error I got it working!
I'll try to give a step by step tutorial on how to do that for OSX.
In the next couple of weeks I'll also try to get it working on windows and will update this answer as I succeed!
Step 1
Install python 3.4.2 from https://www.python.org/downloads/release/python-342/
Step 2
Checkout the blender git repo and the svn repo from blender (found at http://wiki.blender.org/index.php/Dev:Doc/Building_Blender/Mac):
mkdir ~/blender-build
cd ~/blender-build
git clone http://git.blender.org/blender.git
cd blender
git submodule update --init --recursive
git submodule foreach git checkout master
git submodule foreach git pull --rebase origin master
Download external libs:
cd ~/blender-build
mkdir lib
cd lib
svn checkout https://svn.blender.org/svnroot/bf-blender/trunk/lib/darwin-9.x.universal
Step 3 Use CMake to create the XCode project for building blender
Make sure you check the following settings for python:
This will automatically configure to build blender using the installed python framework on your mac.
Step 4 Download and build boost with the Python 3.4.2
Here you first have to check which version blender uses.
In the external libraries svn repository boost is contained as a pre build library at:
blender-build/lib/darwin-9.x.universal/README
For me this is boost 1.51: http://www.boost.org/users/history/version_1_51_0.html
Step 5 Build your shared library with Boost 1.51 and Python 3.14
Step 6 Build blender and make sure it is using the installed Python 3.4.2
Step 7 Copy your shared library into the addons folder of the blender.app
Here a tiny small little hint: In order to let python recognize your shared library it has to have the extension .so rather than .dylib (for windows this is .pyd rather than .dll)!
Step 8 Recheck
Check that your boost is compiled agains Python 3.4.2:
otool -L /usr/local/lib/libboost_python.dylib
/usr/local/lib/libboost_python3.dylib:
libboost_python3.dylib (compatibility version 0.0.0, current version 0.0.0)
/Library/Frameworks/Python.framework/Versions/3.4/Python (compatibility version 3.4.0, current version 3.4.0) <- IMPORTANT
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
You can also check if your library can be used in a simple python script like test.py by executing the following script in a folder were your .so is contained:
import mylib
print('hello world!')
Run:
python3 test.py
Basically now you should also be able to execute this simple script inside of blender, which is the case for me.

Cant Find HashLib Module After Server Update

I run Python Scripts on our Dreamhost Server. Our Python scripts use Python 2.7 - we made a custom installation because Dreamhost uses Python 2.6. Everything worked fine for 1 year.
Dreamhost did a server update yesturday and now our scripts fail to find a specific module - MD5. The scripts output the error below when we go to import hashlib.
What do I need to do to rectify this?
Should I reinstall Python 2.7?
Should I reinstall Pip and Easy_Install?
Should I reinstall VirtualEnv?
Is there something else you recommend I do?
Error from all Python scripts:
/home/user/script.py in ()
import hashlib
hashlib undefined
/home/user/python/lib/python2.7/hashlib.py in ()
# version not supporting that algorithm.
try:
globals()[__func_name] = __get_hash(__func_name)
except ValueError:
import logging builtin globals = <built-in function globals, __func_name = 'md5', __get_hash = <function __get_builtin_constructor /home/user/python/lib/python2.7/hashlib.py in __get_builtin_constructor(name='md5')
return _sha.new
elif name in ('MD5', 'md5'):
import _md5
return _md5.new
elif name in ('SHA256', 'sha256', 'SHA224', 'sha224'):
_md5 undefined
<type 'exceptions.ImportError': No module named _md5
args = ('No module named _md5',)
message = 'No module named _md5'

I was having the exact same issue. I run Python 2.7 in my own virtualenv.
I am trying to avoid reinstalling python and run a Django 1.7 application.
The following approach works for me.
STEP 1. (This step might not be necessary)
I uninstalled pythonbrew since it says here: http://wiki.dreamhost.com/Python
that pythonbrew has been deprecated.
If you were doing this from scratch pyenv is way to go, but you don't need to reinstall
virtualenv, etc. Just get rid of pythonbrew to start with.
$ rm -Rf ~/.pythonbrew
Removed references in .bashrc to pythonbrew
STEP 2.
There is no need to re-install virtualenv. Just create a new virtual env
$~/env> virtualenv myNewEnvironment
$~/env/myNewEnvironment/bin> source activate
$ pip freeze
You have a clean slate now, start rebuilding dependencies from scratch.
At least is solves the "import hashlib" issue. This gives you a clean
version of python properly linked to the new Ubuntu OS.
(myNewEnvironment):~> which python
~/env/myNewEnvironment/bin/python
(myNewEnvironment):~> python
Python 2.7.3 (default, Feb 27 2014, 19:58:35)
[GCC 4.6.3] on linux2
installed on Ubuntu 12.04 (which is the new OS)
Verify:
import hashlib should not throw error
STEP 3.
pip install Django
pip install MySQL-python
Its also probably safer to complete/recheck the remaining steps listed out in
http://wiki.dreamhost.com/Django (or appropriate wiki page for your framework)
For now this allows me to get my site up and running, (but) there is a warning
that I am ignoring for now until I figure out more:
You have unapplied migrations; your app may not work properly until they are applied.
Run 'python manage.py migrate' to apply them.
Good luck!

Python does not find custom PyQt5

As the offical pyqt5-installation in the ubuntu repositories seem to lack support for QtQuick, I tried to install pyqt5 from source. The installation itself seems to work correctly, but when running a python script that uses PyQt5, python complains that it cannot find that PyQt.
After building sip 4.15.5, I downloaded PyQt5.2. It should be compatible to my version of Qt (output of qmake --version):
QMake version 3.0
Using Qt version 5.2.0 in /opt/qt5.1.1/5.2.0/gcc_64/lib
I ran
The output of configure.py of pyqt can be found here: https://gist.github.com/Mitmischer/8677889 .
The installation output of pyqt can be found here: https://gist.github.com/Mitmischer/8677780 .
After sudo make install, I can see a folder PyQt5 in /usr/lib/python3.3/site-packages which is quite nice. However, if I run cat PyQt5/__init__.py, there is no actual code inside:
# Copyright (c) 2014 Riverbank Computing Limited <info#riverbankcomputing.com>
#
# This file is part of PyQt5.
#
# This file may be used under the terms of the GNU General Public License
# version 3.0 as published by the Free Software Foundation and appearing in
# the file LICENSE included in the packaging of this file. Please review the
# following information to ensure the GNU General Public License version 3.0
# requirements will be met: http://www.gnu.org/copyleft/gpl.html.
#
# If you do not wish to use this file under the terms of the GPL version 3.0
# then you may purchase a commercial license. For more information contact
# info#riverbankcomputing.com.
#
# This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE
# WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
Yep, that's all what is inside that file. I don't know whether it's supposed to be that way, but it looks strange to me. Furthermore (ls PyQt5):
__init__.py QtCore.so QtGui.so QtMultimediaWidgets.so QtPositioning.so QtQuick.so Qt.so QtTest.so QtX11Extras.so
_QOpenGLFunctions_2_0.so QtDBus.so QtHelp.so QtNetwork.so QtPrintSupport.so QtSensors.so QtSql.so QtWebKit.so QtXmlPatterns.so
QtBluetooth.so QtDesigner.so QtMultimedia.so QtOpenGL.so QtQml.so QtSerialPort.so QtSvg.so QtWidgets.so uic/
Doesn't look that pythonic.
As suggested elsewhere, I (hopefully) set my pythonpath appropriately:
> echo $PYTHONPATH
/usr/lib/python3.3/site-packages/
Now, if I start an interactive python3.3-session (or a script), PyQt5 cannot be found:
Python 3.3.2+ (default, Oct 9 2013, 14:50:09)
[GCC 4.8.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from PyQt5 import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'PyQt5'
>>>
Has anyone else tried to install PyQt5 from source? What can I do to make PyQt work?

You probrably messed PYTHONPATH in someway.
I have successfully built, installed and used PyQT using virtual environment. So here is how to install it using virtualenv. There are tons of tutorals, so please read about it.
So install python-virtualenv, virtualenvwrapper (at least that's what they are called on Debian).
mkvirtualenv -p /path/to/python3.3 name
workon name
cd PyQtSource
configure
make
make install
To use this enviorment do:
workon name
python