Develop Reference

Installing Tensorflow in M1 Mac

I followed this tutorial, https://www.youtube.com/watch?v=ykCY_tJbhNw to install tensorflow in a mac (M1 processor). However when I import tensrflow I get the following error. Any ideas what it might be?
The steps include,
installing brew (was alreadyy installed)
install conda (was already installed)
installing miniforge
download tensorflow-macos from github page
conda activate mlEnv
/Users/imantha/Software/tensorflow_macos/install_venv.sh -p
When prompted to input the path : /Users/imantha/Software/anaconda3/envs/mlEnv/
and select yes to install tensorflow
tensorflow was successfully installed and when typed conda list, the package is listed.
However when I start python session in the shell and type import tensorflow as tf, I get the following error
Traceback (most recent call last):
File "/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-packages/tensorflow/python/pywrap_tensorflow.py", line 64, in <module>
from tensorflow.python._pywrap_tensorflow_internal import *
ImportError: dlopen(/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so, 6): no suitable image found. Did find:
/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: mach-o, but wrong architecture
/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: mach-o, but wrong architecture
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-
packages/tensorflow/__init__.py", line 41, in <module>
from tensorflow.python.tools import module_util as _module_util
File "/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-
packages/tensorflow/python/__init__.py", line 39, in <module>
from tensorflow.python import pywrap_tensorflow as _pywrap_tensorflow
File "/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-
packages/tensorflow/python/pywrap_tensorflow.py", line 83, in <module>
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-
packages/tensorflow/python/pywrap_tensorflow.py", line 64, in <module>
from tensorflow.python._pywrap_tensorflow_internal import *
ImportError:
dlopen(/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-
packages/tensorflow/python/_pywrap_tensorflow_internal.so, 6): no suitable image
found. Did find:
/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-
packages/tensorflow/python/_pywrap_tensorflow_internal.so: mach-o, but wrong
architecture
/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-
packages/tensorflow/python/_pywrap_tensorflow_internal.so: mach-o, but wrong
architecture
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.

There are quite a few things to do in order to install TensorFlow on the Macbook M1 chip:
First, install Python 3.8.9 (I have tried on this version, not sure about any other versions)
Go to your project dir
install hdf5 by running brew install hdf5 if you do not have brew, you can download it here: https://brew.sh/
Above hdf5 install will spit out its location: use it and run:
export HDF5_DIR=<hdf5_location> for version 1.13.0 (my case) the following:
export HDF5_DIR=/opt/homebrew/Cellar/hdf5/1.13.0
create a virtual environment using virtualenv venv (choose whatever method for creating virtual environment or do not create virtual environment, it's fine. To install virtualenv, run pip3 install virtualenv)
(Skip this if not using virtual environment) Activate virtual environment by running: source venv/bin/activate
Now, we have everything require to install Tensorflow on mac with M1 chip including GPU version
To install Tensorflow and related libs, run following:
(If using virtual env, run following)
venv/bin/pip install --no-binary=h5py h5py
venv/bin/pip install tensorflow-macos
venv/bin/pip install tensorflow-metal
Not using virtual env, run following:
pip3 install --no-binary=h5py h5py
pip3 install tensorflow-macos
pip3 install tensorflow-metal
Now, your TensorFlow installation has been completed!
Try it out:
import tensorflow as tf
print("TF version:", tf.__version__)
print("Number GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))
print("Number CPUs Available: ", len(tf.config.experimental.list_physical_devices('CPU')))

Build Tensorflow from source by following the instructions mentioned here.
See this comment.
OR,
Try the workaround mentioned here.

The below way is the simplest and efficient way from which we can install tensorflow in Mac M1
NOTE: Mac M1 has ARM64 arch not X86_64 so we need to be very careful while downloading packages. Anaconda is available for Mac M1 users : https://repo.anaconda.com/archive/Anaconda3-2022.05-MacOSX-arm64.sh
Following are the steps to victory:
Download Anaconda from above link
create virtual env ( Optional )
pip install tensorflow-macos
Thats it!
Reference video link which may be useful : https://www.youtube.com/watch?v=5DgWvU0p2bk

If you want to use an Anaconda distribution, you might want to consider Miniconda3. Currently, I'm not sure whether it's possible to install TensorFlow using Anaconda.
Download Miniconda3 using this link. Both the .bash and the .pkg files should do - choose one of them. However, for M1, only the .bash file seem to have worked, while for M2 both worked.
If you used the .bash file above, then do the following to start the installation:
chmod +x ~/Downloads/Miniforge3-MacOSX-arm64.sh
sh ~/Downloads/Miniforge3-MacOSX-arm64.sh
Create a conda environment with Python 3.9: conda create -n tf_env python=3.9 (latest Python version didn't work as I'm writing this).
Install TensorFlow and dependencies:
conda install -c apple tensorflow-deps
python -m pip install tensorflow-macos
python -m pip install tensorflow-metal
At this point, you should make sure that the Numpy version is up to date to not run into errors when importing TensorFlow. I used version 1.23.3. python -m pip install numpy --upgrade.
Install jupyter notebook using conda install notebook.
If everything has been installed without errors, running the following code in a jupyter notebook should correctly import TensorFlow and print out the version as well as verifying the presence of a GPU.
import tensorflow as tf
gpu = len(tf.config.list_physical_devices('GPU')) > 0
print(f"TensorFlow Version: {tf.__version__}")
print("GPU is", "available" if gpu else "NOT available")

missing attribute in lda2vec module in notebook

A jupyter notebook cannot import dirichlet_likelihood.py from lda2vec.
This py file exists in github for the current lda2vec.
I installed the module and opened the workbook, then attempted to run it. I suspect there is a very simple reason for my problem.
The notebook is
https://github.com/cemoody/lda2vec/blob/master/examples/twenty_newsgroups/lda2vec/lda2vec.ipynb
When I try the below at the python command-line (in the current environment) it does not give the below error and instead it wanted keras, which I installed. On the command line it says it cannot import preprocess.
uname -a
Linux ubuntu 4.18.0-15-generic #16~18.04.1-Ubuntu SMP Thu Feb 7 14:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
sudo apt-get install python3-venv
python3.6 -m venv .env
source .env/bin/activate
pip install --upgrade pip
pip install jupyter
pip install lda2vec
from lda2vec import preprocess, Corpus
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-4-2b87256bea6b> in <module>
----> 1 from lda2vec import preprocess, Corpus
2 import matplotlib.pyplot as plt
3 import numpy as np
4 get_ipython().run_line_magic('matplotlib', 'inline')
5
~/.env/lib/python3.6/site-packages/lda2vec/__init__.py in <module>
----> 1 import lda2vec.dirichlet_likelihood as dirichlet_likelihood
2 import lda2vec.embedding_mixture as embedding_mixture
3 from lda2vec.Lda2vec import Lda2vec as model
4 import lda2vec.word_embedding as word_embedding
5 import lda2vec.nlppipe as nlppipe
AttributeError: module 'lda2vec' has no attribute 'dirichlet_likelihood'
python
from lda2vec import preprocess, Corpus
Using TensorFlow backend.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name 'preprocess'
EDIT
I got it work by doing these things
- Get ubuntu-18.04.2-desktop-amd64.iso
- Have BIOS virtualisation settings changed for hyper-V
- Make a VM in VMWare
- Increase memory to 3G
- Give it 40G of disk
Then in a terminal
sudo apt install python2.7
sudo apt install python-pip
pip install virtualenv
Mkdir 2.7env
Cd 2.7env
python2.7 -m venv .env
python2.7 -m virtualenv .env
source .env/bin/activate
pip install --upgrade pip
pip install jupyter
pip install -U spacy
python -m spacy download en
pip install wheel nltk gensim pyLDAvis lda2vec
sudo apt install git
git clone https://github.com/cemoody/lda2vec.git
cp ~/lda2vec/build/lib.linux-x86_64-2.7/lda2vec/corpus.py ~/2.7env/.env/lib/python2.7/site-packages/lda2vec/Corpus.py
cp ~/lda2vec/build/lib.linux-x86_64-2.7/lda2vec/preprocess.py ~/2.7env/.env/lib/python2.7/site-packages/lda2vec/preprocess.py
python -m pip install ipykernel
python -m ipykernel install --user
python lda2vec/examples/twenty_newsgroups/lda2vec/lda2vec_run.py
cd lda2vec/examples/twenty_newsgroups/lda2vec/
jupyter notebook
change the kernel to 2
In firefox open lda2vec.ipynb
As above, I am now stuck trying to get it to re-create the twenty_newsgroups npz file so I can eventually provide my own content. In case anyone out there understands this better, I suspect trying to run this script on VM with little RAM might be the problem but the error is reported as
(.env) craig#ubuntu:~/whcjimmy/lda2vec/examples/twenty_newsgroups/data$ python preprocess.py
Traceback (most recent call last):
File "preprocess.py", line 31, in <module>
n_threads=4)
File "/home/craig/whcjimmy/.env/lib/python3.6/site-packages/lda2vec-0.1-py3.6.egg/lda2vec/preprocess.py", line 104, in tokenize
vocab = {v: nlp.vocab[v].lower_ for v in uniques if v != skip}
File "/home/craig/whcjimmy/.env/lib/python3.6/site-packages/lda2vec-0.1-py3.6.egg/lda2vec/preprocess.py", line 104, in <dictcomp>
vocab = {v: nlp.vocab[v].lower_ for v in uniques if v != skip}
File "vocab.pyx", line 242, in spacy.vocab.Vocab.__getitem__
File "lexeme.pyx", line 44, in spacy.lexeme.Lexeme.__init__
File "vocab.pyx", line 157, in spacy.vocab.Vocab.get_by_orth
File "strings.pyx", line 138, in spacy.strings.StringStore.__getitem__
KeyError: "[E018] Can't retrieve string for hash '9243420536193520'. This usually refers to an issue with the `Vocab` or `StringStore`."

Alright I got this to work. The problems were
pick the right python to run a 4 year old git project. Python 2.7.
check the installed module has the code from the git repo
run through the problem in a python terminal
copy and edit python files go to 3
One problem above was to do with a changed API for a dependency.
ImportError: No module named 'spacy.en'
The original problem was likely due to something about git or python I am not familiar with. Still the git project self-tests all fail and its build fails.
But I have my jupyter notebook running and producing convincing outputs.

Error in creating LMDB database file in Python for Caffe

I'm trying to create an LMDB data base file in Python to be used with Caffe according to this tutorial. The commands import numpy as np and import caffe run perfectly fine. However, when I try to run import lmdb and import deepdish as dd, I'm getting the following errors:
>>> import lmdb
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named lmdb
>>> import deepdish as dd
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named deepdish
I'm running Python 2.7.9 through Anaconda 2.2.0 (64-bit) on Ubuntu 14.04. While installing the dependencies for Caffe according to this page, I've already installed the lmdb package through sudo apt-get install liblmdb-dev.
Any ideas why this error might be occuring?

Well, the apt-get install liblmdb-dev might work with bash (in the terminal) but apparently it doesn't work with Anaconda Python. I figured Anaconda Python might require it's own module for lmdb and I followed this link. The Python installation for lmdb module can be performed by running the command pip install lmdb in the terminal. And then import lmdb in Python works like a charm!
The above installation commands may require sudo.

If you're using Anaconda, then this can solve your problem (it worked for me):
conda install -c https://conda.binstar.org/dougal lmdb

For Anaconda users, installing python-lmdb package from conda-forge should fix the lmdb import error:
conda install -c conda-forge python-lmdb
This was tested on conda 4.5.11 on an lxc-containerized system running Ubuntu 18.04.
Note that there is a conda package named lmdb (without python-), installable via:
conda install -c conda-forge lmdb
that does not fix the import error.

Installation Error while installing portia

Followed instructions on the github page. Of course, had to make minor changes since I was working with a windows 7 system. I got to the point post creating the virtual environment for portia to run. And I was trying to install the required packages using pip.
pip install -r requirements.txt
It failed with a log.
Now in the shell I try to run twistd, it gives error saying command not found. I even tried as follows:
deostroll#DEOTOP /c/Portia/portia/slyd (master)
$ python ../../portia_env/Scripts/twistd.py -n slyd
Traceback (most recent call last):
File "../../portia_env/Scripts/twistd.py", line 13, in <module>
from twisted.scripts.twistd import run
File "c:\Portia\portia_env\lib\site-packages\twisted\__init__.py", line 53, in
<module>
_checkRequirements()
File "c:\Portia\portia_env\lib\site-packages\twisted\__init__.py", line 37, in
_checkRequirements
raise ImportError(required + ": no module named zope.interface.")
ImportError: Twisted requires zope.interface 3.6.0 or later: no module named zop
e.interface.
(portia_env)
deostroll#DEOTOP /c/Portia/portia/slyd (master)
$
Is there an alternate procedure to follow in order to make this work on windows?

It looks like zope.interface didn't install when you ran:
pip install -r requirements.txt
Could you try running the following and see if it works?
pip install zope.interface

I'm having troubles getting pybrain working through anaconda

Here is my problem:
After I managed to install anaconda (having python 3.4), I apparently managed to install pybrain too. But when i use 'import pybrain' from anaconda or from the terminal too I get this error:
>>> import pybrain
Traceback (most recent call last):
File "<ipython-input-2-0fb7233d2a8c>", line 1, in <module>
import pybrain
File "//anaconda/lib/python3.4/site-packages/PyBrain-0.3-py3.4.egg/pybrain/__init__.py", line 1, in <module>
from structure.__init__ import *
ImportError: No module named 'structure'

Simply running sudo pip3 install git+https://github.com/pybrain/pybrain.git worked for me after having the same issue.
The version up on PyPi isn't Python 3 compatible.
Installing the latest commit directly using pip3 should take care of your old package (from PyPi) as well.