I'm trying to understand the cause of the following error. First if I type the following into python
>>> import scipy.sparse
>>> import torch
it runs without error. However, when I type in in
>>> import torch
>>> import scipy.sparse
I get the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/global/software/sl-7.x86_64/modules/langs/python/3.6/lib/python3.6/site-packages/scipy/sparse/__init__.py", line 229, in <module>
from .csr import *
File "/global/software/sl-7.x86_64/modules/langs/python/3.6/lib/python3.6/site-packages/scipy/sparse/csr.py", line 15, in <module>
from ._sparsetools import csr_tocsc, csr_tobsr, csr_count_blocks, \
ImportError: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /global/software/sl-7.x86_64/modules/langs/python/3.6/lib/python3.6/site-packages/scipy/sparse/_sparsetools.cpython-36m-x86_64-linux-gnu.so)
I can even go the directory "/global/software/sl-7.x86_64/modules/langs/python/3.6/lib/python3.6/site-packages/scipy/sparse/" and import the binary "_sparsetools.cpython-36m-x86_64-linux-gnu.so" followed by torch without issue. But if I try it the other way around I again get the above error.
Does anyone have any idea why changing the order of these imports should have a different effect?
The simple search strategy for shared objects assumes that a single version of each exists—or at least that directories containing newer versions are put first on the search path. The path includes $LD_LIBRARY_PATH (which should be avoided), DT_RPATH and its newer variant DT_RUNPATH (which crucially depend on the client being loaded), and system directories like /lib. This works well for systems following the FHS with global package management, but packages that are installed, with copies of their dependencies, in a single per-package directory (as is common on Windows and with some “normal user” package managers) can easily produce multiple versions of a shared object with the same soname.
The expectation is that sharing that name is harmless because one can be used in place of the other (and therefore be put first on the path). The reality is that there is no single directory that is newest for all libraries, and there’s not even a single path to configure given the DT_ tags.
The result is that whichever one is loaded first wins: the dynamic loader can’t load both of them (since they provide many of the same symbols), so the second request has only the effect of checking the library version tags, which here are found to be inadequate. In this case, one client (torch) is relying on the system’s C++ standard library, while the other (_sparsetools) has its own, newer version. It may or may not even need its newer version: since it was built against it, it is conservatively marked as needing it by default.
This is a hard problem: not even tools like virtual environments or environment modules can handle it, since the problem lies in the incompatible compilation of extant packages. Systems that rebuild everything from source (like Nix or Spack) can, but only at their usual cost of controlling all relevant builds. It may be that simply controlling the import order is, unfortunately, the best choice available.
#DavisHerring's answer gives you an explanation what is going on, here is a possible workaround to ensure that the right version is loaded - the LD_PRELOAD-trick:
1. Step:
Find the right libc++.so-version via console:
$ ldd _sparsetools.cpython-36m-x86_64-linux-gnu.so
libstdc++.so.6 => <path to right version>/libstdc++.so.6(...)
2. Step:
While starting python, preload the right version, so the loader picks the right version:
$ LD_PRELOAD=<path to right version>/libstdc++.so python
However the best solution would to rebuild pytorch with the right libc++-dependency.
Related
RDKit could be a nice package if it wasn't so complicated to install.
Here on SO, there are several questions having problems with the installation of RDKit.
However, on different operating systems or different environments.
My configuration is:
Win10, Python 3.7.4, pip is installed, PATH is set, PYTHONPATH is set.
The installation of other modules is working fine via python -m pip install <package>.
I'm aware that the site recommends the fastest installation with Anaconda.
However, I don't have and don't want Anaconda.
On the webpage it says:
"Get the appropriate windows binary build from: https://github.com/rdkit/rdkit/releases".
However, there are no binaries of the latest versions.
This means, I would have to build it from source. I'm hesitating because the process seems to be pretty complicated, many extra installations with new problems and unknowns, and furthermore, the instructions seem to be outdated and incomplete for somebody who would build binaries from the source for the first time.
So, then I tried some unofficial binaries of RDKit.
If I unpack them and set the paths according to instructions, I get this error message:
>>> from rdkit import Chem
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\xyz\Programs\RDKit\rdkit\__init__.py", line 2, in <module>
from .rdBase import rdkitVersion as __version__
ImportError: DLL load failed: The specified module could not be found.
So, finally my questions:
How to properly install RDKit with the above mentioned configuration?
What is the specified DLL which is missing?
Where is it expecting it and searching it?
Are these RDKit 3.6 binaries maybe incompatible with Python 3.7.4?
I'm pretty sure it is probably a "small" thing (a path here or a check there), but I'm stuck. Thank you for any hints.
Update:
Apparently, it is not just a "small" thing. Chances to get this to work are most likely very low.
In the meantime I found this:
https://github.com/rdkit/rdkit/issues/1812
https://github.com/rdkit/rdkit/issues/2389
If the author of rdkit writes (April 2019):
I would be happy to be able to do pip distributions of the RDKit, but
to the best of my knowledge no one has managed to figure out how to
make it actually work.
I'd be happy to accept a PR from someone who has figured this out, but
I am not likely to have the time to do this myself anytime in the near
future.
So, if anybody feels capable achieving this, please feel free.
I will invest time in something else or will have to switch to Anaconda if I want to use RDKit.
On the webpage you linked there is a section about missing DLLs:
"In Win7 systems, you may run into trouble due to missing DLLs, see one thread from the mailing list: http://www.mail-archive.com/rdkit-discuss#lists.sourceforge.net/msg01632.html You can download the missing DLLs from here: http://www.microsoft.com/en-us/download/details.aspx?id=5555"
Not sure if this helps
This is the first time I have attempted to use anything other than what's provided by python.
I have recently gotten into pythons provided Tkinter, though due to some issues I decided to use another GUI, and heard that PyQt was highly recommended, so I downloaded that and looked into various tutorials. In these tutorials, I cannot seem to execute any of the import statements in said tutorials that relate to PyQt, primarily PyQt5 (I have checked I have the correct version number by the way).
So for instance:
import PyQt5
raises the error:
Traceback (most recent call last):
File "/Users/MEBO/PycharmProjects/Music/testing.py", line 1, in <module>
import Qt
ImportError: No module named 'Qt'
[Finished in 0.1s with exit code 1]
I have a lot of research into this. I've heard people talk of using pip to install modules, and I have done this be safe (as well as downloading it from the internet), I've tried changing the project interpreter to versions Python3/ 2.7/ 2.6, appending the path name to the sys.path directory, (which I really know nothing about to be honest, I was hoping I'd get lucky), though nothing seems to work.
Are you supposed to be able to just import a module off the bat, or do you have to set some things up first?
For windows download the package and extract it to (path where python installed)\Python27\Lib and then try to import.
Specific to PyQt
This package cannot just be downloaded and imported, it must be built because it is not pure python, it uses Qt (C++) and requires dependancies. Read this tutorial on installation.
There is also a very complete python package distribution, Anaconda, that includes pyqt and much more. Almost all the packages I ever looked at are in there.
In general to pure python code
In other cases, if you place modules/code that has been download into the directory that your python script is run from, you can import off the bat, or you can append/insert any folder to the sys.path.
# importer will search here last
sys.path.append('/path/to/code/')
# importer will search here second, right after script's directory
# this can be useful to override a module temporarily...
sys.path.insert(1,'/path/to/code/')
I'm trying to use a multi-buildpack setup on Heroku with these two buildpacks:
https://github.com/virtualstaticvoid/heroku-buildpack-r.git
https://github.com/dbrgn/heroku-buildpack-python-sklearn/
I am using rpy2 to call R from python. I detailed the full process I used to get the slug to compile here.
It works fine for numpy, scipy, and scikit-learn with rpy2. However, I'm also trying to get matplotlib to work with this setup, and I'm getting an error.
I used matplotlib==1.1.0, as suggested by this StackOverflow post.
However, when I have my LD_LIBRARY_PATH set so that rpy2 will work, like this:
LD_LIBRARY_PATH=/app/vendor/R/lib64/R/modules:/app/vendor/R/lib64/R/lib:/app/vendor/gcc-4.3/lib64
I get this error:
>>> from matplotlib import ft2font
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /app/vendor/R/lib64/R/lib/libstdc++.so.6: version `GLIBCXX_3.4.11' not found (required by /app/.heroku/python/lib/python2.7/site-packages/matplotlib/ft2font.so)
If I remove the LD_LIBRARY_PATH settings, then matplotlib works, but rpy2 can't find the R library located in /app/vendor/R/lib64/R/lib. Changing the order of the directories in LD_LIBRARY_PATH doesn't seem to have an effect, for some reason.
So I can get either matplotlib or rpy2 to work, but not at the same time.
I have libraries in these locations:
~ $ find . -name "*libstd*"
/app/vendor/gcc-4.3/gcc-4.3/lib64/libstdc++.so.6.0.10
/app/vendor/gcc-4.3/gcc-4.3/lib64/libstdc++.so.6
/app/vendor/gcc-4.3/gcc-4.3/lib64/libstdc++.so
/app/vendor/gcc-4.3/gcc-4.3/lib64/libstdc++.a
/app/vendor/gcc-4.3/gcc-4.3/lib64/libstdc++.la
/app/vendor/gcc-4.3/gcc-4.3/lib/libstdc++.so.6.0.10
/app/vendor/gcc-4.3/gcc-4.3/lib/libstdc++.so.6
/app/vendor/gcc-4.3/gcc-4.3/lib/libstdc++.so
/app/vendor/gcc-4.3/gcc-4.3/lib/libstdc++.a
/app/vendor/gcc-4.3/gcc-4.3/lib/libstdc++.la
/app/vendor/R/lib64/R/lib/libstdc++.so.6.0.10
/app/vendor/R/lib64/R/lib/libstdc++.so.6
/app/vendor/R/lib64/R/lib/libstdc++.so
/app/vendor/R/lib64/R/lib/libstdc++.a
/app/vendor/R/lib64/R/lib/libstdc++.la
I suspect that matplotlib should be using /app/vendor/gcc-4.3/gcc-4.3/lib64/libstdc++.so.6 (how do I tell?), but I can't seem to get it to use that one.
Any suggestions? I'm totally stuck. I must say the multi buildpack process is pretty messed up for this kind of thing.
Ok, I figured it out. It turns out that the correct library wasn't in /app/vendor/gcc-4.3/gcc-4.3/lib64/ after all, but just in /usr/lib. That explains why changing the order of the directories in LD_LIBRARY_PATH had no effect, since it wasn't in any of them.
There must be some logic that looks in the directories of LD_LIBRARY_PATH, and it doesn't find a match there, then looks in /usr/lib. That is why it works some of the time. The file in /app/vendor/R/lib64/R/lib/ must have been close enough to be considered a match (so it didn't look in /usr/lib), but in the wrong format so it caused a later error.
The fix is just to include /usr/lib in LD_LIBRARY_PATH. I added /usr/local/lib as well, for good measure. You should now use:
heroku config:set LD_LIBRARY_PATH=/usr/lib:/usr/local/lib:/app/vendor/R/lib64/R/modules:/app/vendor/R/lib64/R/lib:/app/vendor/gcc-4.3/lib64
I'm trying to run a Python program, which uses the pygame modules, from MATLAB. I know I can use either
system('python program.py')
or just
! python program.py
However, I keep getting the error:
Traceback (most recent call last):
File "program.py", line 1, in <module>
import pygame
ImportError: No module named pygame
What is strange is that if I run the program from the command line, it works just fine. Does anyone know why if run from within MATLAB, Python can't find pygame?
The problem may be that MATLAB is not seeing your PYTHONPATH, which normally stores Python libraries and modules. For custom modules, PYTHONPATH should also include the path to your custom folders.
You can try setting the value of PYTHONPATH from within a MATLAB running session:
PATH_PYTHON = '<python_lib_folder>'
setenv('PYTHONPATH', PATH_PYTHON); % set env path (PYTHONPATH) for this session
system('python program.py');
See also the possibly relevant SO answer here: How can I call a Qtproject from matlab?
As I haven't used matlab too often and don't have the program available now I cannot say for sure, but matlab may be creating a custom environment with custom paths (this happens a lot, so the user has a very consistent experience in their software). When matlab installs it may not export paths to its own modules to your default environment. So when calling for pygame.py outside of matlab, python cannot find pygame.py under its usual lookup paths.
Solutions could be:
find the pygame.py, and map the path to it directly in your code, though this could cause you headaches later on during deployment
Try just copying the pygame.py file to your working directory, could have dependences that need to addressed.
Install pygame directly from its developer at http://www.pygame.org. Version differences could be a problem but pygame gets put under the usual lookup paths for python. (This would be my preferred solution personally.)
Or just export the location of path to pygame in matlab's library to your default enivronment. This could be a problem during deployment too.
For posterity, first try everything that Stewie noted here ("Undefined variable "py" or class" when trying to load Python from MATLAB R2014b?). If it doesnt work then it's possible that you have multiple pythons. You can try and check which python works (with all the related installed modules) on your bash/terminal. And then use
pyversion PYTHONPATH
to let matlab know the right path.
Also use py.importlib.import_module('yourmodule') to import the module after that.
That should get you started.
Ok, so I finally got poker-eval and pypoker to install correctly. If I go to the pypoker folder in cygwin, and type python test.py, it runs the program fine. However, if I copy the test.py file over to a netbeans project main file, I get the error ImportError: No module named _pokereval_2_7. The problem is, I am using wxpython, and it won't let me execute my program from cygwin. I am using NetBeans, so my entire project is developed there. If I try to include pypoker-eval, I get this error.
> Traceback (most recent call last):
> File
> "C:\Users\JeremyLaptop\Documents\NetBeansProjects\testing\src\testing.py",
> line 36, in <module>
> from pokereval import PokerEval File
> "C:\Users\JeremyLaptop\Desktop\pypoker-eval-137.0\pokereval.py",
> line 29, in <module>
> _pokereval = __import__('_pokereval_' + sys.version[0] + '_' + sys.version[2])
> ImportError: No module named _pokereval_2_7.
I have been trying to figure out how to get this to work for hours, hopefully one of you guys can help.
FYI: Windows 7, pypoker and pokereval folders are on my desktop. Projects are saved in C:/users/myname/my documents/netbeansprojects.
A quick look at the source code makes it seem that Python 2.7 is not supported by that library. Try using Python 2.6 instead.
An example of a ready made Texas Hold'em 7- and 5-card evaluator can be found here with documentation and further explained here. It goes by the name of "SpecialKEval". All feedback welcome at the e-mail address found therein.
You can typically (~97% of the time) get away with just 6 additions and a couple of bit shifts. The algo uses a generated look up table which occupies about 9MB of RAM and is generated in a near-instant. Cheap. All of this is done inside of 32-bits, and "inlining" the 7-card evaluator is good for evaluating about 50m randomly generated hands per second on my laptop.
This might be a useful alternative to you in your project.
I had a similar problem. It did not install by default into the default location for my python installation. If something similar is going on for you, then it is probably just not in your PYTONPATH. You can get around that by providing one (or adjusting the path in netbeans, not 100% sure how to do this, but it should be a standard operation) or copying the contents of the site package into your project.