I'm trying to get the R package reticulate working on a CentOS 7.8 system using RStudio Server v1.2.5042 with a custom environment created with conda. When I initiate a Python job with reticulate, I get an error that some system libraries are not the correct versions, specifically, libstdc++.so.6 and libz.so.1.
First off, I realize CentOS 7.8 is a bit old and some of the problem might be solved by upgrading the OS, but that's not an option in this case.
The conda environment does work and I can run the target Python script in a terminal window without any errors. In RStudio using reticulate, the code is extremely simple at this point:
library(reticulate)
use_condaenv('test')
py_run_file('test_script.py')
When the script is run, I get the following error:
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/<user>/.conda/envs/test/lib/python3.8/site-packages/scipy/_lib/_uarray/_uarray.cpython-38-x86_64-linux-gnu.so)
When I look into the /usr/lib64 directory, I find libstdc++.so, but running strings libstdc++.so | grep ^GLIBC | sort shows me that it indeed does not support version GLIBCXX 3.4.21. No surprises. If I navigate to the /home/<user>/.conda/envs/test/lib directory, I find another copy of libstdc++.so.6 and this one does support version GLIBCXX 3.4.21. So, the correct version of the library is present in the correct conda environment directory, but for some reason RStudio and reticulate are not finding it.
I tried changing LD_LIBRARY_PATH to have the conda environment directory listed first, but that does not work. I found a lengthy discussion here, which points out that LD_LIBRARY_PATH isn't really a good fix unless it is set prior to the RStudio process initialization. (And then goes on a tangent about which version of Python gets used.) For my situation, there may be multiple conda environments to support and it isn't possible to know which will be active for any given session, and any given user my use different/multiple environments. I would rather not try to harmonize all conda environments into one, big, uber-environment.
I've also verified that the Python version and other libraries are correctly set:
python: /home/<user>/.conda/envs/test/bin/python
libpython: /home/<user>/.conda/envs/test/lib/libpython3.8.so
pythonhome: /home/<user>/.conda/envs/test:/home/<user>/.conda/envs/test
version: 3.8.5 (default, Aug 5 2020, 08:36:46) [GCC 7.3.0]
numpy: /home/<user>/.conda/envs/test/lib/python3.8/site-packages/numpy
numpy_version: 1.19.1
NOTE: Python version was forced by use_python function
I have been able to get it running by resetting the links in /usr/lib64 to point to the copy in the conda directory. While this gets it working for this instance, I'm not sure I want to push a fix like this to production. My guess is that if I link to the most inclusive version of the library across all conda environments, and that version fully supports all versions that the system level library supports, everything will be fine, but this feels like a hack, at best.
If anyone has found a good solution to this, I would appreciate to know the details.
After struggling with a similar issue for several days, and trying many of the suggested solutions on the web (mostly based on symlinks, LD_LIBRARY_PATH variable, or installing/upgrading/downgrading packages like libgcc), I finally found something that is only mentioned once here : https://github.com/rstudio/reticulate/issues/338#issuecomment-415472406
The problem possibly seems to be a conflict when R is installed on the OS (with apt-get r-base for example) rather than with (Ana/Mini)conda.
In that case, it will try to load system libraries rather than conda environment ones.
So here is a possible solution to people still having this kind of issue, that worked for me :
(maybe optional) uninstall R on the local system : apt-get remove r-base r-base-dev
Activate conda environment : conda activate my_R_env
Install R within this environment : conda install r r-essentials --channel conda-forge
Install R packages like reticulate inside the same env : conda install r-reticulate --channel conda-forge
You can check that your Python packages installed within your conda environment are now properly loaded with reticulate :
R
> library(reticulate)
> repl_python()
>>> import pandas or any package that was causing the initial issue
Note : I still had some issues due to R loading base conda environment instead of activated one, but deactivating/reactivating environments or calling use_condaenv('my_R_env') seems to solve the problem.
Related
so I've been trying to use Keras with Miniconda on my machine for a few weeks now (in Rstudio) and I've still been unable to make significant progress.
Most recently, here are the steps I followed:
create an environment in Miniconda, using Python 3.6
Download R and RStudio (an older version, Version 3.6.0) into said environment and run them
Install Keras into the environment (after Rcpp, Devtools etc)
Attempt to install Tensorflow
But then I get the error message that my Python installation isn't found and the bindings aren't loaded.
I've put the environment on the PATH, as well as the location where Python is and some folders within the environment.
I followed the advice given in a comment in my last post, which helped me to create the environment, but now Python isn't loading in Miniconda and I'm so so frustrated that I haven't been able to make this work. Please help.
When I look for the Python version I get this:
C:\Users\NK>python --version [1] Python 3.8.4rc1
Here is the beginning of the code I'm running:
library(keras)
library(tensorflow)
library(reticulate)
use_python("C:\\Users\\NK\\AppData\\Local\\Microsoft\\WindowsApps", required=T)
Sys.setenv(RETICULATE_MINICONDA_PATH = "C:/mini/envs/r_3.6.0.1")
mnist <- dataset_mnist()
and I get this error:
Error: Installation of Python not found, Python bindings not loaded. Use reticulate::install_miniconda() if you'd like to install a Miniconda Python environment. even though I'm doing this entirely in a Miniconda environment. I'm even opening RStudio straight from the Miniconda command prompt.
Some basic info: Dell XPS 15 running Windows 10, using RStudio
Side question: Is there an easier way to do all this? It's been fatiguing.
I finally got it to work! Here were my steps for anyone who stumbles across this:
Reinstall Anaconda
Create Anaconda Environment with Python 2.7 and R
Install Keras and Tensorflow packages and dependencies one by one into the environment through Anaconda
Install RStudio in the environment through Anaconda
use use_python("[env]", required=T) and Sys.setenv(RETICULATE_MINICONDA_PATH = "[env]") in any scripts you run using Keras
I am new at this and I wonder why I cannot install or import geonamescache library in anaconda.
Apparently, I am the only one among my friends with this problem. Could you help me?
Thanks in advance.
Screenshot of error
If you're using conda you should make sure you jupyter notebook is either installed in the same conda environment containing geonamescache or has its kernel pointed to that environment. Check the output conda list in your activated environment for the said package. If you don't find it (I'm expecting you won't) it means pip3 installed geonamescache as a system wide package. You'll need to install it your activated environment using pip install geonamescache. Try not to use the pip3 command in conda environments.
Not sure what OS and version of Anaconda you are using, so I can't be very specific.
According to the official list of packages for the distribution of Anaconda 3.7 for Mac OS(I am currently using it), there are currently 657 packages supported by anaconda(For Windows that number stands at around 620 at the moment). Here is the full list for up-to-date distributions.
I have looked into geonamescache, and it looks like it might not be supported by Anaconda at all. I suggest you looking into installing it via Pip(keep in mind that I would do that only if necessary as it is better to avoid using Pip with Conda that much). Look at this answer if interested.
I want to upgrade my base environment to Python 3.8. According to the official documentation, I should just run conda install -c anaconda python. This, however, only upgrades Python 3.7, so, in the end:
(base) C:\Users\bob>conda install -c anaconda python
Collecting package metadata (current_repodata.json): done
Solving environment: done
# All requested packages already installed.
(base) C:\Users\bob>python --version
Python 3.7.6
conda update -c anaconda python only updated a number of packages, but not Python itself.
And if I specify a version, like explained here, after two hours analyzing dependencies, the upgrade fails because of conflicts.
I managed to set up a 3.8 environment, but I'd rather upgrade my base environment as it's the one my system path for use outside of Anaconda.
Any suggestion?
It doesn't appear to be possible to update the base version of anaconda to 3.8 yet because of the conflicts you mention. I assume once all the standard packages are updated to support 3.8+ (and updated versions of all dependencies), then the anaconda release will include 3.8.x as the default. This might take a while.
The problem here is version conflicts in the packages that anaconda installs by default. The reason you can install 3.8 in a clean environment is because no other packages are installed in that environment, so no conflicts.
This does now present a solution, although it requires reinstalling everything from scratch...
disclaimer: I don't use windows, so adapt my instructions accordingly to your OS.
Remove your existing anaconda install from the path and delete
anything in your bashrc (or windows equivalent) that points to it.
You can just delete/uninstall anaconda, but it's good to have a backup just incase, so leaving it there and removing links to it is a good option.
Download and install miniconda, and put it in your path as you had before.
Install python 3.8 in the base environment conda install python=3.8
Install all the packages that you need to do whatever you do.
Depending on what you install, at some point you may run into the same conflicts that prevent a standard anaconda install from updating at the moment, but that hasn't happened for me.
The latest stable version of anaconda is version 3.7
Please see here:
https://www.anaconda.com/distribution/
There isn't an option to get python 3.8, this is due to anaconda sometimes running a version behind the release of python.
I had the same issue and found there is a 2020-07 release of Anaconda with Python 3.8 https://www.anaconda.com/products/individual
I want to downgrade tornado to a previous version because the new one causes an error according to the answers here: Jupyter notebook kernel not connecting. I am on Ubuntu, in a virtual environment.
To check the current version of it, I used pip freeze and got this: tornado==6.0.1. When I use apt-cache policy tornado the output is: "Unable to locate package tornado". When I type apt-cache policy python-tornado the output is "python-tornado: Installed: 4.5.3-1".
How do I proceed from here? My ultimate goal is to make the jupyter notebook run, and I need to figure out this tornado module for that. What is the difference between tornado and python-tornado? Which one I should care about?
One of those names is the actual package name under which it is published to the Python Package Index (PyPI), which is the namespace that pip deals in.
The other is the name as set by your Ubuntu operating system, and given the version string, I am guessing that you are using Ubuntu 18.04 Bionic Beaver. Ubuntu uses a strict naming convention, where all Python packages must start with a python- prefix. These packages are managed and installed by your OS package manager.
How to proceed depends on your Jupyter setup. If it is installed and running from a virtualenv, then you can use the pip command when the virtualenv is active to alter versions there. Take into account that using pip should already ensure you are getting compatible versions installed; you could try to upgrade jupyter if tornado was upgraded independently.
If you are using the Ubuntu-managed jupyter package then there too the package manager should take care of matching versions.
However, if you you are using a virtualenv that still has access to the OS-mananged jupyter system while locally only tornado is installed, then you want to add jupyter to your virtualenv to mask the system version, which is too old.
I'm installing vowpal wabbit python3 interface using OSX 10.13.2, python 3.5.1, Anaconda3 virtual env for Anaconda.
I've followed this step.
# edit Makefile
# change BOOST_INCLUDE to use anaconda env dir: /anaconda/envs/vw/include
# change BOOST_LIBRARY to use anaconda lib dir: /anaconda/envs/vw/lib
I've followed the step above.
https://gist.github.com/suisuiwudi/86f9657d785b3e4fdc02e11e3e35899b
I got the error, please check the details in the gist above.
ld: library not found for -lpython3.5m
I also followed their methods, but these didn't work.
https://github.com/JohnLangford/vowpal_wabbit/issues/1267
https://github.com/ContinuumIO/anaconda-issues/issues/2871
Any idea about how to install these on MAC?
Your configuration of the system might be a problem here it is a very known issue in VowpalWabbit's installation that it doesn't work with MacOSX because a dependency BoostC++(in which all the reductions are implemented i.e in C++) requires a file named libstdC++ and this file was discontinued by Apple 2 years ago. Also I wouldn't recommend using Anaconda along with it although users having installation problems with Anaconda was resolved in an issue but just to be on the safe side. This is due to the dynamic linking nature of the libraries used in it namely Cmake and boost-python.
Really there are two solutions to your problem
1) Find an older version of Xcode and copy that file into its required path
2) Using a virtual machine with Windows or linux installed and install VowpalWabbit there
Apple does sometimes annoy even its most loyal customers.