Develop Reference

Issue installing python 3.6.8 on Mac M1

I'm trying to run an old app that requires python =3.6.8 on Mac M1 machine with Monterey OS X 12.6.1.
I've installed pyenv-virtualenv and pyenv and successfully installed python 3.6.15. However, when I try to install 3.6.8, I'm getting error -
~ % pyenv86 install --patch 3.6.8 <<(curl -sSL https://raw.githubusercontent.com/pyenv/pyenv/master/plugins/python-build/share/python-build/patches/3.6.8/Python-3.6.8/0008-bpo-45405-Prevent-internal-configure-error-when-runn.patch\?full_index\=1)
Error: No available formula with the name "openssl#1.0". Did you mean openssl#1.1 or openssl#3?
Downloading openssl-1.1.0j.tar.gz...
-> https://www.openssl.org/source/old/1.1.0/openssl-1.1.0j.tar.gz
Installing openssl-1.1.0j...
Installed openssl-1.1.0j to /Users/lokesh.raizada/.pyenv/versions/3.6.8
python-build: use readline from homebrew
Downloading Python-3.6.8.tar.xz...
-> https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tar.xz
Installing Python-3.6.8...
patch unexpectedly ends in middle of line
patch: **** Only garbage was found in the patch input.
BUILD FAILED (OS X 12.6.1 using python-build 20180424)
Inspect or clean up the working tree at /var/folders/41/4c6czpkx6c18sfcfj0lzjkd00000gq/T/python-build.20230127152902.5735
Results logged to /var/folders/41/4c6czpkx6c18sfcfj0lzjkd00000gq/T/python-build.20230127152902.5735.log
Last 10 log lines:
/Users/lokesh.raizada/.pyenv/versions/3.6.8/openssl/share/doc/openssl/html/man3/X509_REVOKED_get_ext.html -> /Users/lokesh.raizada/.pyenv/versions/3.6.8/openssl/share/doc/openssl/html/man3/X509v3_get_ext_by_NID.html
/Users/lokesh.raizada/.pyenv/versions/3.6.8/openssl/share/doc/openssl/html/man3/X509_REVOKED_get_ext_by_NID.html -> /Users/lokesh.raizada/.pyenv/versions/3.6.8/openssl/share/doc/openssl/html/man3/X509v3_get_ext_by_NID.html
/Users/lokesh.raizada/.pyenv/versions/3.6.8/openssl/share/doc/openssl/html/man3/X509_REVOKED_get_ext_by_OBJ.html -> /Users/lokesh.raizada/.pyenv/versions/3.6.8/openssl/share/doc/openssl/html/man3/X509v3_get_ext_by_NID.html
/Users/lokesh.raizada/.pyenv/versions/3.6.8/openssl/share/doc/openssl/html/man3/X509_REVOKED_get_ext_by_critical.html -> /Users/lokesh.raizada/.pyenv/versions/3.6.8/openssl/share/doc/openssl/html/man3/X509v3_get_ext_by_NID.html
/Users/lokesh.raizada/.pyenv/versions/3.6.8/openssl/share/doc/openssl/html/man3/X509_REVOKED_delete_ext.html -> /Users/lokesh.raizada/.pyenv/versions/3.6.8/openssl/share/doc/openssl/html/man3/X509v3_get_ext_by_NID.html
/Users/lokesh.raizada/.pyenv/versions/3.6.8/openssl/share/doc/openssl/html/man3/X509_REVOKED_add_ext.html -> /Users/lokesh.raizada/.pyenv/versions/3.6.8/openssl/share/doc/openssl/html/man3/X509v3_get_ext_by_NID.html
/var/folders/41/4c6czpkx6c18sfcfj0lzjkd00000gq/T/python-build.20230127152902.5735 ~
~
/var/folders/41/4c6czpkx6c18sfcfj0lzjkd00000gq/T/python-build.20230127152902.5735 ~
/var/folders/41/4c6czpkx6c18sfcfj0lzjkd00000gq/T/python-build.20230127152902.5735/Python-3.6.8 /var/folders/41/4c6czpkx6c18sfcfj0lzjkd00000gq/T/python-build.20230127152902.5735
I have found this link which installs 3.6.15 with a patch. Is there any patch associated with 3.6.8 as well ?
Install python 3.6.* on Mac M1
I tried the same for 3.6.8 but no luck -
~ % pyenv86 install --patch 3.6.8 <<(curl -sSL https://raw.githubusercontent.com/pyenv/pyenv/master/plugins/python-build/share/python-build/patches/3.6.8/Python-3.6.8/0008-bpo-45405-Prevent-internal-configure-error-when-runn.patch\?full_index\=1)

Try uninstalling all python modules, python version, pyenv, XCode commandtools as well.
Do not use pyenv, as pyenv support is not available below 3.6.15 -
https://github.com/pyenv/pyenv/issues/1737#issuecomment-1423110250
Then proceed to Install 3.6.8 from - https://www.python.org/downloads/release/python-368/
https://www.python.org/ftp/python/3.6.8/python-3.6.8-macosx10.9.pkg

How should I install keras-bert to use properly on R side?

I am trying to install keras-bert as explained here. Although it is successfully installed on the environment as, I cannot see keras-bert inside the R side.
(bert_env) C:\Users\xxxxx\Dropbox\Rcode\ProjectBERT>pip list
Package Version
-------------------------------- ---------
certifi 2022.6.15
keras 2.6.0
keras-bert 0.89.0
keras-embed-sim 0.10.0
keras-layer-normalization 0.16.0
keras-multi-head 0.29.0
keras-pos-embd 0.13.0
keras-position-wise-feed-forward 0.8.0
keras-self-attention 0.51.0
keras-transformer 0.40.0
mkl-fft 1.3.1
mkl-random 1.2.2
mkl-service 2.4.0
numpy 1.23.1
pip 22.1.2
setuptools 61.2.0
six 1.16.0
wheel 0.37.1
wincertstore 0.2
Another proof which shows keras-bert is successfully install on the current environment (bert_env):
(bert_env) C:\Users\xxxxx\Dropbox\Rcode\ProjectBERT>pip install keras-bert
Requirement already satisfied: keras-bert in c:\anaconda3\envs\bert_env\lib\site-packages (0.89.0)
Requirement already satisfied: keras-transformer==0.40.0 in c:\anaconda3\envs\bert_env\lib\site-packages (from keras-bert) (0.40.0)
However, I cannot properly call the package on the R side which says:
> reticulate::conda_list()
name python
1 base C:\\Anaconda3/python.exe
2 bert_env C:\\Anaconda3\\envs\\bert_env/python.exe
3 py27 C:\\Anaconda3\\envs\\py27/python.exe
> reticulate::use_condaenv("bert_env", required=TRUE)
> reticulate::py_config()
C:\ANACON~2\envs\bert_env\lib\site-packages\numpy\__init__.py:138: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service
from . import _distributor_init
python: C:/Anaconda3/envs/bert_env/python.exe
libpython: C:/Anaconda3/envs/bert_env/python310.dll
pythonhome: C:/Anaconda3/envs/bert_env
version: 3.10.4 | packaged by conda-forge | (main, Mar 30 2022, 08:38:02) [MSC v.1916 64 bit (AMD64)]
Architecture: 64bit
numpy: C:/Anaconda3/envs/bert_env/Lib/site-packages/numpy
numpy_version: 1.23.1
I have almost tried everything which offered as solution such as:
conda update conda
conda update --all
Then I have created a fully clean new environment in order to reinstall keras-bert on there. Also looked scipy and numpy package conflicts which is mentioned on their github side. No success at all!
Of course, once I check availability, it returns false!
reticulate::py_module_available('keras_bert')
[1] FALSE
I am aware of this post and applied exactly same patterns with YAML. No improvements as well. Any suggestions to properly install keras-bert or how should I approach the issue are greatly appreciated.

After a couple of days search, I found a way to set the reticulate path properly. This issue solves the main issue as well. .Renviron file is a user-controllable file that can be used to create environment variables. Hence, the .Renviron file can be used to store the information which can be accessed from all R projects in your computer.
First of all, you need to find the location of your .Renviron file. In order to find the path of your .Renviron file, you can type this on RStudio console:
usethis::edit_r_environ()
In my case, it returns:
* Modify 'C:/Users/xxxxx/Documents/.Renviron'
* Restart R for changes to take effect
Then, you might either navigate the location of .Renviron file manually. (Note: It was empty once I opened it in my scenario.) Then paste the full path of python.exe inside to the file as:
RETICULATE_PYTHON="YourEnvironmentPath/python.exe"
Tip: If you are not sure about your full python path, you can get it as:
reticulate::conda_python("keras_bert")
Please do not forget to type your environment name instead of my environment nam > keras_bert).
Then I got:
[1] "C:\\Anaconda3\\envs\\keras_bert/python.exe"
In my case, I pasted:
RETICULATE_PYTHON="C:\\Anaconda3\\envs\\keras_bert/python.exe"
Or (on Windows), you can click on start and open Windows Powershell. Then copy this code into powershell to handle same thing:
Add-Content C:\Users\xxxxx\Documents/.Renviron 'RETICULATE_PYTHON="C:\\Anaconda3\\envs\\keras_bert/python.exe"'
In order to see updates, you must close and reopen RStudio to have a clean RStudio session. Now, run the following line to check whether everything is fine or not:
Sys.getenv('RETICULATE_PYTHON')
or:
reticulate::py_config()
In my case, it returns:
[1] "C:/Anaconda3/envs/keras_bert/python.exe"
or:
python: C:/Anaconda3/envs/keras_bert/python.exe
libpython: C:/Anaconda3/envs/keras_bert/python38.dll
pythonhome: C:/Anaconda3/envs/keras_bert
version: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:59:45) [MSC v.1929 64 bit (AMD64)]
Architecture: 64bit
numpy: C:/Anaconda3/envs/keras_bert/Lib/site-packages/numpy
numpy_version: 1.23.2
NOTE: Python version was forced by RETICULATE_PYTHON
Briefly, with this solution, you do not need to care about whether you declare the environment at the start of the R session or any other messy scenarios.

Python setup.py: Could not find suitable distribution for Requirement.parse('tensorflow')

I have tensorflow listed as a requirement in the install_requires section of the setup.py of my project.
When I attempt to install my project into a new Anaconda environment I get the following error:
$ python setup.py install
...
Searching for tensorflow
Reading https://pypi.org/simple/tensorflow/
No local packages or working download links found for tensorflow
error: Could not find suitable distribution for Requirement.parse('tensorflow')
I can get past this by installing tensorflow "manually" via conda:
$ conda install tensorflow
Once I do this the install via setup.py works without a hitch.
Am I mistaken in assuming that something is amiss with my environment? If not then what is going on and how can I avoid this issue? (My concern is that users of my package will not be able to install from source using setup.py)
Another oddity that I assume is related or may provide a clue is that the version of TensorFlow listed in my Anaconda environment is 2.0 but if I import it when running Python it appears to instead be using version 1.15. For example:
$ conda list tensorflow
# packages in environment at /home/james/miniconda3/envs/cvdata_test:
#
# Name Version Build Channel
tensorflow 2.0.0 mkl_py37h66b46cc_0
tensorflow-base 2.0.0 mkl_py37h9204916_0
tensorflow-estimator 2.0.0 pyh2649769_0
$ python
Python 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 22:33:48)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.__version__
'1.15.0-rc2'
This is on a Dell laptop running Ubuntu 18.04 without a GPU, so perhaps the version shown in the interpreter is akin to tensorflow-cpu? If I run pip freeze I see tensorflow==2.0.0 and tensorflow-cpu==1.15.0rc2, which is a bit confusing...

This appears to have been caused by using Python version 3.8, which is currently an unsupported version of Python. Once I created a new Anaconda environment with Python version 3.7 this issue went away.
The only remaining issue is this error that I see when I run pip install -e . for my project which includes tensorflow:
ERROR: tensorflow-cpu 1.15.0rc2 has requirement tensorboard<1.16.0,>=1.15.0, but you'll have tensorboard 2.1.0 which is incompatible.
ERROR: tensorflow-cpu 1.15.0rc2 has requirement tensorflow-estimator==1.15.1, but you'll have tensorflow-estimator 2.1.0 which is incompatible.
So the issue of tensorflow-cpu version 1.15.0rc2 actually being installed when version 2.1.0 shows as being the installed version is still a mystery. To wit:
$ conda list tensorflow
# packages in environment at /home/james/miniconda3/envs/cvd:
#
# Name Version Build Channel
tensorflow 2.1.0 pypi_0 pypi
tensorflow-estimator 2.1.0 pypi_0 pypi
$ python -c "import tensorflow as tf; print(tf.__version__)"
1.15.0-rc2

How to install cvxopt without downgrading mkl and python

Environment:
OS = Windows 10, Python ver = 3.7.0, mkl ver = 2019.0-118, conda ver = 4.5.12. All up to date as of now (2019/1/13).
Issue:
When attempting conda install cvxopt, it prompts at the end:
The following packages will be DOWNGRADED:
mkl: 2019.0-118 --> 2018.0.3-1
python: 3.7.0-hea74fb7_0 --> 3.6.8-h9f7ef89_0
Proceed ([y]/n)? n
I don't want to downgrade either. Actually, when I visit the cvxopt's installation page it states clearly that
Wheels for Windows:
are available for Python 27, 3.4, 3.5, 3.6, and 3.7 (64 bit only)
The suggested installation procedure is a bit complicated, though.
I'm just wondering if there's a simple way (that is, for dummies) such as a one-line command that can install cvxopt without compromising python or mkl.
Thanks.

It's best not to mix conda and pip packages. It looks like conda-forge has the latest version of cvxopt, so simply do:
conda install -c conda-forge cvxopt

You could just try pip3 install cvxopt
I've just tried it and it works on my Windows machine

Managing packages: PyCharm vs conda vs pip

I'm new to Python and recently installed PyCharm 2016.3 on Windows 10. I'm also using Anaconda 3.
I don't know much about package management and would like to understand it better. Normally I just use conda update --all but I noticed (by checking the package list of my local PyCharm Interpreter) that this doesn't upgrade all packages to the latest version.
One such package is Pillow of which there's a version 4.0.0 but conda (4.3.11) won't update it past 3.4.2. I tried conda install pillow: 4.0.0 and got:
UnsatisfiableError: The following specifications were found to be in conflict:
- pillow 4.0.0*
- python 3.5*
- spyder-app
Use "conda info <package>" to see the dependencies for each package.
Later I found out that Pillow is also available on conda-forge so I tried conda install -c conda-forge pillow=4.0.0 and got:
The following NEW packages will be INSTALLED:
libiconv: 1.14-vc14_4 conda-forge [vc14]
libxml2: 2.9.3-vc14_9 conda-forge [vc14]
olefile: 0.44-py35_0 conda-forge
vc: 14-0 conda-forge
The following packages will be UPDATED:
freetype: 2.5.5-vc14_2 [vc14] --> 2.7-vc14_0 conda-forge [vc14]
jpeg: 8d-vc14_2 [vc14] --> 9b-vc14_0 conda-forge [vc14]
libtiff: 4.0.6-vc14_2 [vc14] --> 4.0.6-vc14_7 conda-forge [vc14]
pillow: 3.4.2-py35_0 --> 4.0.0-py35_2 conda-forge
The following packages will be SUPERCEDED by a higher-priority channel:
conda: 4.3.11-py35_0 --> 4.2.13-py35_0 conda-forge
conda-env: 2.6.0-0 --> 2.6.0-0 conda-forge
qt: 4.8.7-vc14_9 [vc14] --> 4.8.7-vc14_6 conda-forge [vc14]
I decided not to proceed and instead tried pip install pillow. Since this command doesn't ask for confirmation the package was simply installed. Now when I type conda list I get:
Pillow 4.0.0 <pip>
pillow 3.4.2 py35_0
The package list of the PyCharm Interpreter now shows Pillow as being version 4.0.0 but conda update pillow still returns:
# All requested packages already installed.
pillow 3.4.2 py35_0
My questions are:
1) What should I rely on to keep all my packages up to date, without compatibility issues?
2) Why did conda install pillow: 4.0.0 return an error but conda install -c conda-forge pillow=4.0.0 didn't?
3) What do the * next to pillow 4.0.0 and python 3.5 in the list of dependencies mean?
4) Since now I have both Pillow 3.4.2 (in /anaconda3/pkgs) and Pillow 4.0.0 (in /anaconda3/lib/site-packages) which one would be used if I imported Pillow?
5) Does the superseding conda: 4.3.11-py35_0 --> 4.2.13-py35_0 conda-forge mean conda is getting downgraded?
6) What is the difference between the tags pip, py35_0, py35_4, np111py35_2, etc?
7) PyCharm tells me there's a version 2.9.5 of package Jinja2 but both normal conda and conda-forge only find 2.9.4. From which channel is PyCharm getting this information?

Ok, I can't answer all of your questions but here goes:
1) Conda defers to the "pain up front" approach for handling dependency/conflict resolution. You'll have to get all of your packages to play nicely together in the repo's/channels that you have available to even make a package or keep them in an environment together. You can try running it with --force or --no-deps to try getting it in but ..... that can cause issues for you in the future (I don't know if that would even work with the later versions of conda, it changes a lot). Simply keeping packages up to date, and up to latest, I would just use pip. Its come a long way in the last few years (https://glyph.twistedmatrix.com/2016/08/python-packaging.html)
2) I am not completely sure, I believe it would have something to do with providing an explicit non-url channel for conda to look at. Typically you pass it the URL to the conda-forge repo (I think, again we don't use conda-forge internally).
3) The * means you are ignoring the patch/build 4.0.0 == Major.Minor.Build. Likewise, 3.5* == any version of 3.5
4) I would import pillow in a terminal, and then print out the module to see where its getting pulled from, why guess?
5) pass (although I think so)
6)
pip : means you installed that package via pip. It will not be picked up if you do conda list --explicit
py35_0 : has a requirement / only available to envs / packages that use python 3.5
py35_4 : not sure (always forget that one)
np111py35_2 : requires python3.5 and also numpy 1.11 (I think *)
7) I tend to steer clear of pycharm, I believe that you can inspect the python interpreter that pycharm is pointing at to see what environment its in. Based on the root environment, you can do a conda info and get a list of all of the channels you are pointing to.
Note: if you are going to use conda, you may just want to add conda-forge to your channels list instead of passing the -c (but seeing how the other channels are organized should help you see how you should pass the -c flag)