Develop Reference

Installing Tensorflow in M1 Mac

I followed this tutorial, https://www.youtube.com/watch?v=ykCY_tJbhNw to install tensorflow in a mac (M1 processor). However when I import tensrflow I get the following error. Any ideas what it might be?
The steps include,
installing brew (was alreadyy installed)
install conda (was already installed)
installing miniforge
download tensorflow-macos from github page
conda activate mlEnv
/Users/imantha/Software/tensorflow_macos/install_venv.sh -p
When prompted to input the path : /Users/imantha/Software/anaconda3/envs/mlEnv/
and select yes to install tensorflow
tensorflow was successfully installed and when typed conda list, the package is listed.
However when I start python session in the shell and type import tensorflow as tf, I get the following error
Traceback (most recent call last):
File "/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-packages/tensorflow/python/pywrap_tensorflow.py", line 64, in <module>
from tensorflow.python._pywrap_tensorflow_internal import *
ImportError: dlopen(/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so, 6): no suitable image found. Did find:
/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: mach-o, but wrong architecture
/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: mach-o, but wrong architecture
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-
packages/tensorflow/__init__.py", line 41, in <module>
from tensorflow.python.tools import module_util as _module_util
File "/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-
packages/tensorflow/python/__init__.py", line 39, in <module>
from tensorflow.python import pywrap_tensorflow as _pywrap_tensorflow
File "/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-
packages/tensorflow/python/pywrap_tensorflow.py", line 83, in <module>
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-
packages/tensorflow/python/pywrap_tensorflow.py", line 64, in <module>
from tensorflow.python._pywrap_tensorflow_internal import *
ImportError:
dlopen(/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-
packages/tensorflow/python/_pywrap_tensorflow_internal.so, 6): no suitable image
found. Did find:
/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-
packages/tensorflow/python/_pywrap_tensorflow_internal.so: mach-o, but wrong
architecture
/Users/imantha/Software/anaconda3/envs/mlEnv/lib/python3.8/site-
packages/tensorflow/python/_pywrap_tensorflow_internal.so: mach-o, but wrong
architecture
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.

There are quite a few things to do in order to install TensorFlow on the Macbook M1 chip:
First, install Python 3.8.9 (I have tried on this version, not sure about any other versions)
Go to your project dir
install hdf5 by running brew install hdf5 if you do not have brew, you can download it here: https://brew.sh/
Above hdf5 install will spit out its location: use it and run:
export HDF5_DIR=<hdf5_location> for version 1.13.0 (my case) the following:
export HDF5_DIR=/opt/homebrew/Cellar/hdf5/1.13.0
create a virtual environment using virtualenv venv (choose whatever method for creating virtual environment or do not create virtual environment, it's fine. To install virtualenv, run pip3 install virtualenv)
(Skip this if not using virtual environment) Activate virtual environment by running: source venv/bin/activate
Now, we have everything require to install Tensorflow on mac with M1 chip including GPU version
To install Tensorflow and related libs, run following:
(If using virtual env, run following)
venv/bin/pip install --no-binary=h5py h5py
venv/bin/pip install tensorflow-macos
venv/bin/pip install tensorflow-metal
Not using virtual env, run following:
pip3 install --no-binary=h5py h5py
pip3 install tensorflow-macos
pip3 install tensorflow-metal
Now, your TensorFlow installation has been completed!
Try it out:
import tensorflow as tf
print("TF version:", tf.__version__)
print("Number GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))
print("Number CPUs Available: ", len(tf.config.experimental.list_physical_devices('CPU')))

Build Tensorflow from source by following the instructions mentioned here.
See this comment.
OR,
Try the workaround mentioned here.

The below way is the simplest and efficient way from which we can install tensorflow in Mac M1
NOTE: Mac M1 has ARM64 arch not X86_64 so we need to be very careful while downloading packages. Anaconda is available for Mac M1 users : https://repo.anaconda.com/archive/Anaconda3-2022.05-MacOSX-arm64.sh
Following are the steps to victory:
Download Anaconda from above link
create virtual env ( Optional )
pip install tensorflow-macos
Thats it!
Reference video link which may be useful : https://www.youtube.com/watch?v=5DgWvU0p2bk

If you want to use an Anaconda distribution, you might want to consider Miniconda3. Currently, I'm not sure whether it's possible to install TensorFlow using Anaconda.
Download Miniconda3 using this link. Both the .bash and the .pkg files should do - choose one of them. However, for M1, only the .bash file seem to have worked, while for M2 both worked.
If you used the .bash file above, then do the following to start the installation:
chmod +x ~/Downloads/Miniforge3-MacOSX-arm64.sh
sh ~/Downloads/Miniforge3-MacOSX-arm64.sh
Create a conda environment with Python 3.9: conda create -n tf_env python=3.9 (latest Python version didn't work as I'm writing this).
Install TensorFlow and dependencies:
conda install -c apple tensorflow-deps
python -m pip install tensorflow-macos
python -m pip install tensorflow-metal
At this point, you should make sure that the Numpy version is up to date to not run into errors when importing TensorFlow. I used version 1.23.3. python -m pip install numpy --upgrade.
Install jupyter notebook using conda install notebook.
If everything has been installed without errors, running the following code in a jupyter notebook should correctly import TensorFlow and print out the version as well as verifying the presence of a GPU.
import tensorflow as tf
gpu = len(tf.config.list_physical_devices('GPU')) > 0
print(f"TensorFlow Version: {tf.__version__}")
print("GPU is", "available" if gpu else "NOT available")

Installing tensorflow with virtualenv pip in Anaconda

I'm trying to install tensorflow with pip in virtualenv in Anaconda for Windows 10. After activating my virtualenv, when I run pip install tensorflow, I get an error:
ERROR: Could not find a version that satisfies the requirement tensorflow (from versions: none)
ERROR: No matching distribution found for tensorflow
This might be because tensorflow doesn't support Python3.8 which I am using. The update of the post says that it should now work, but it doesn't for me. I tried what the answers suggest with no better luck.
So, I tried the following to downgrade to Python3.7:
conda install python=3.7.1
setting a virtualenv with Python3.7 version
creating a conda environment with Python3.7
installing an older version of Anaconda that contains Python3.7
But any of these attempts incredibly see Python switch back to version 3.8 when I activate the virtualenv. Even if I destroy the environment and create it again in the older Anaconda.
I also tried py -3.7 -m pip install tensorflow in the Anaconda with Python3.7 (it doesn't find the correct version of Python when using the more recent Anaconda). It lead to the error:
Cannot uninstall 'wrapt'. It is a distutils installed project and thus
we cannot accurately determine which files belong to it which would
lead to only a partial uninstall.
I fixed it running conda update wrapt and launched the tensorflow installation command again. After many package installations and removals, it displayed the following error:
ERROR: After October 2020 you may experience errors when installing or
updating packages. This is because pip will change the way that it
resolves dependency conflicts.
We recommend you use --use-feature=2020-resolver to test your packages
with the new resolver before it becomes the default.
pytest-doctestplus 0.2.0 requires pytest>=2.8.0, but you'll have
pytest 0.0.0 which is incompatible. pytest-astropy 0.5.0 requires
pytest>=3.1.0, but you'll have pytest 0.0.0 which is incompatible.
And in effect, I can see that the installation didn't work:
pip freeze > requirements.txt creates an empty requirements.txt
in the Python console, when I try to import keras, it resolves to:
Traceback (most recent call last):
File "C:\Users\my_private_path\venv\lib\site-packages\keras\__init__.py", line 3, in <module>
from tensorflow.keras.layers.experimental.preprocessing import RandomRotation
ModuleNotFoundError: No module named 'tensorflow'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\my_private_path\venv\lib\site-packages\keras\__init__.py", line 5, in <module>
raise ImportError(
ImportError: Keras requires TensorFlow 2.2 or higher. Install TensorFlow via `pip install tensorflow`
It seems that no matter how many paths I try, tensorflow doesn't like virtualenv in Anaconda. Would someone have a path that works?

I use virtual environments with tensorflow all the time and it works fine. First use Anaconda to create a new environment. From the Anaconda home page click on Environments.
When the Environment page appears click on create. A new window will pop up. Give the environment a name (for example call it tf), check the python checkbox. From the dropdown select python 3.7. A new environment called tf is created with python 3.7 installed.
Now go to the Anaconda prompt which is conda. type in conda activate tf. Now use conda to install tensorflow (do not use pip). Conda automatically installs cudnn 7.6.5 and CUDA Toolkit 10.1.243 and tensorflow 2.1.0. If you want tensorflow 2.2 first install 2.1.0 as described. Then use pip to install 2.2 with pip install tensorflow ==2.2.0. 2.2 is compatible with the toolkit and cudnn versions installed with 2.1. Conda can only install tensorflow up to version 2.1.0. Install whatever other packages you need using either pip or conda.

ModuleNotFoundError: No module named 'torch'

I try to use pytorch module by conda but I get an error
Traceback (most recent call last):
File "train.py", line 8, in <module>
import torch
ModuleNotFoundError: No module named 'torch'
when I write conda list | findstr torch I see that torch is installed:
What is the problem?
I tried:
conda update conda -n root
conda install mkl=2018
but get:
Collecting package metadata (current_repodata.json): done
Solving environment: done
# All requested packages already installed.
Could Not Find C:\WINDOWS\TEMP\conda-23721-26470.tmp
Could Not Find C:\WINDOWS\TEMP\tmpry_dlvar.bat
and the same error occurrence

Try the below mentioned one, surely it will work.
conda install -c pytorch pytorch

This worked for me
conda install pytorch torchvision -c pytorch

Try to run the following on the local environment as given in the pytorch website during installation. Open Jupyter Notebook locally and run the following
from __future__ import print_function
import torch
x = torch.rand(5, 3)
print(x)
if this works then most likely the environment variable is not set properly.
I hope this helps.

Install and import Caffe with Anaconda (on Windows 10)

I'm struggling to just install and import caffe on my windows 10's Anaconda (conda 4.7.11). I really don't know why such a well-known deep-learning package is that hard to just install!
First, I tried to install it. With huge difficulties, I did the following and at last, apparently installation was successful:
conda create -n caffe python=3.7
// activate the environment REM Depending on your conda version you may have to use activate caffe
conda activate caffe
// add the anaconda channel to resolve all dependencies
conda config --add channels anaconda
// Install caffe with cuda support or install caffe-cpu if you do not want CUDA support
conda install caffe-cpu -c willyd
And now, just trying to import caffe, I get this error:
>>> import caffe
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\caffe\lib\site-packages\caffe\__init__.py", line 1, in <module>
from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver, NCCL, Timer
File "C:\Users\user\AppData\Local\Continuum\anaconda3\envs\caffe\lib\site-packages\caffe\pycaffe.py", line 13, in <module>
from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
ImportError: DLL load failed: The specified module could not be found.
What should I do? How to fix it?
BTW, is Caffe2 different from Caffe?

Caffe2 is built on caffe but yes it is different. This link : http://gianni.rosagallina.com/en/posts/2018/10/04/caffe2-gpu-windows-1.html#step-4-build provides a good overview of what steps need to be taken to install caffe2. Incase you hit and error you cannot resolve, you can report the problems on the pytorch git and the team will help you out.

"import torch" giving error "from torch._C import *, DLL load failed: The specified module could not be found"

I am currently using Python 3.5.5 on Anaconda and I am unable to import torch. It is giving me the following error in Spyder:
Python 3.5.5 |Anaconda, Inc.| (default, Mar 12 2018, 17:44:09) [MSC v.1900
64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.
IPython 6.2.1 -- An enhanced Interactive Python.
import torch
Traceback (most recent call last):
File "<ipython-input-1-eb42ca6e4af3>", line 1, in <module>
import torch
File "C:\Users\trish\Anaconda3\envs\virtual_platform\lib\site-
packages\torch\__init__.py", line 76, in <module>
from torch._C import *
ImportError: DLL load failed: The specified module could not be found.
Many suggestions on the internet say that the working directory should not be the same directory that the torch package is in, however I've manually set my working directory to C:/Users/trish/Downloads, and I am getting the same error.
Also I've already tried the following: reinstalling Anaconda and all packages from scratch, and I've ensured there is no duplicate "torch" folder in my directory.
Pls help! Thank you!

I had this similar problem in windows 10...
Solution:
Download win-64/intel-openmp-2018.0.0-8.tar.bz2 from https://anaconda.org/anaconda/intel-openmp/files
Extract it and put the dll files in Library\bin into C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin
Make sure your cuda directory is added to your %PATH% environment variable

I had the same problem. In my case I didn't want the GPU version of pytorch.
I uninstalled it. The version was pytorch: 0.3.1-py36_cuda80_cudnn6he774522_2 peterjc123.
The problem is that cuda and cudnn . then installed with the following command and now it works!
conda install -c peterjc123 pytorch-cpu

I also encountered the same problem when I used a conda environment with python 3.6.8 and pytorch installed by conda from channel -c pytorch.
Here is what worked for me:
1:) conda create -n envName python=3.6 anaconda
2:) conda update -n envName conda
3:) conda activate envName
4:) conda install pytorch torchvision cudatoolkit=9.0 -c pytorch
and then tested torch with the given code:
5:) python -c "import torch; print(torch.cuda.get_device_name(0))"
Note: 5th step will return your gpu name if you have a cuda compatible gpu
Summary: I just created a conda environment containing whole anaconda and then to tackle the issue of unmatched conda version I updated conda of new environment from the base environment and then installed pytorch in that environment and tested pytorch.
For CPU version, here is the link for my another answer: https://gist.github.com/peterjc123/6b804651288e76db7b5fabe5348e1f03#gistcomment-2842825
https://gist.github.com/peterjc123/6b804651288e76db7b5fabe5348e1f03#gistcomment-2842837

Had the same problem and fixed it by re-installing numpy with mkl (Intel's math kernel library)
https://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy
Download the right .whl for your machine. For me it was numpy‑1.14.5+mkl‑cp36‑cp36m‑win_amd64.whl (python 3.6, windows, 64-bit)
and then install using pip.
pip install numpy‑1.14.5+mkl‑cp36‑cp36m‑win_amd64.whl

I am using a Windows 10 computer with an NVIDIA GeForce graphics card. NVIDIA showed I had CUDA 10.1, but I was getting this error when running import torch in Jupyter Lab and suspected it had something to do with CUDA support.
I fixed this problem by downloading and installing the CUDA Toolkit directly from NVIDIA. It installed all required Visual Studio components. When I returned to Jupyter Lab, import torch ran without error.

Make sure you installed the right version of pytorch for your enviroment. I had the same problem I was using pytorch on windows but I had the default package installed which was meant for cuda 8. So I reinstalled the pytorch package for cpu which was what I needed.

I had the same issue with running torch installed with pure pip and solved it by switching to conda.
Following steps:
uninstall python 3.6 from python.org (if exists)
install miniconda
install torch in conda ("conda install pytorch -c pytorch")
Issue with pip installation:
import torch
File "C:\Program Files\Python35\lib\site-packages\torch\__init__.py", line 78, in <module>
from torch._C import *
ImportError: DLL load failed: The specified module could not be found.
After switching to conda it works fine. I believe the issue was resolved by conda through installing the vs_redist 2017
vs2017_runtime 15.4.27004.2010 peterjc123
But I have tried it w/o conda and it did not help. Could not find how to check (and tweak) Python's vs_redist.

Windows10 Solution(This worked for my system):
I was having the same issue in my system. Previously I was using Python 3.5 and I created a virtual environment named pytorch_test using the virtualenv module because I didn't want to mess up my tensorflow installation(which took me a lot of time). I followed every instruction but it didn't seem to work. I installed python 3.6.7 added it to the path. Then I created the virtual environment using:
virtualenv --python=3.6 pytorch_test
Then go to the destination folder
cd D:\pytorch_test
and activate the virtual environment entering the command in cmd:
.\Scripts\activate
After you do this the command prompt will show:
(pytorch_test) D:\pytorch_test>
Update pip if you have not done it before using:
(pytorch_test) D:\pytorch_test>python -m pip install --upgrade pip
Then go for installing numpy+mkl from the site:
https://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy
Choose the correct version from the list if you have python 3.6.7 go with the wheel file:
numpy‑1.15.4+mkl‑cp36‑cp36m‑win_amd64.whl (For 64 bit)
(Note if the whole thing doesnot work just go with simple numpy installation and mkl installation separately)
Then go for installing openmp using:
(pytorch_test) D:\pytorch_test>pip install intel-openmp
Now you are done with the prerequisites. To install pytorch go to the previous versions site:
https://pytorch.org/get-started/previous-versions/
Here select the suitable version from the list of Windows Binaries. For example I am having CUDA 9.0 installed in my system with python 3.6.7 so I went with the gpu version:
cu90/torch-1.0.0-cp36-cp36m-win_amd64.whl
(There are two available versions 0.4.0 and 1.0.0 for pytorch, I went with 1.0.0)
After downloading the file install it using pip(assuming the whl file is in D:).You have to do this from the virtual environment pytorch_test itself:
(pytorch_test) D:\pytorch_test>pip install D:\torch-1.0.0-cp36-cp36m-win_amd64.whl
Prerequisites like six, pillow will be installed automatically.
Then once everything is done, install the models using torchvision.
Simply type :
(pytorch_test) D:\pytorch_test>pip install torchvision
To check everything is working fine try the following script:
import torch
test = torch.rand(4, 7)
print(test)
If everything was good then it wont be an issue. Whenever there is an issue like this it is related to version mismatch of one or more dependencies. This also occurred during tensorflow installation.
Deactivate the following virtual environment using the command deactivate in the cmd:
(pytorch_test) D:\pytorch_test>deactivate
This is the output of pip list in my system:
Package Version
------------ -----------
intel-openmp 2019.0
mkl 2019.0
numpy 1.16.2
Pillow 6.0.0
pip 19.0.3
setuptools 41.0.0
six 1.12.0
torch 1.0.0
torchvision 0.2.2.post3
wheel 0.33.1
Hope this helps. This is my first answer in this community, hope you all find it helpful. I setup pytorch today in the afternoon after trying all sorts of combinations. The same import problem occurred to me while installing CNTK and tensorflow. Anyway I kept them separate in different virtual environments so that I can use them anytime.