Because of the incompatibility between the Apple's new M1 chip and TensorFlow in Python, I followed some online guides to create an environment under Miniforge3 (so that we can run the python in the arm64 instead of the x86_64 one? I don't fully understand the tricks here.).
Following the tutorial (at https://medium.com/codex/installing-tensorflow-on-m1-macs-958767a7a4b3), I created an environment named tf and successfully installed tensorflow. I checked that the modules are installed inside the miniforge3 folder:
(tf) pokfung#pn-204-145 ~ % which python
/Users/pokfung/miniforge3/envs/tf/bin/python
(tf) pokfung#pn-204-145 ~ % python
Python 3.8.10 | packaged by conda-forge | (default, May 11 2021, 06:27:18)
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy, tensorflow
>>> print(numpy.__file__)
/Users/pokfung/miniforge3/envs/tf/lib/python3.8/site-packages/numpy/__init__.py
>>> print(tensorflow.__file__)
/Users/pokfung/miniforge3/envs/tf/lib/python3.8/site-packages/tensorflow/__init__.py
However, when I initiate the Jupyter notebook by (tf) pokfung#pn-204-145 ~ % jupyter notebook, I found that the notebook was not loading modules from the above directories:
import numpy
print(numpy.__file__)
# Output of the print statement is:
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/__init__.py
I wonder how I shall configure the Jupyter notebook so that it can correctly run in the tf conda environment.
Jupyter runs a kernel in the background that in your case supports Python, but it chooses the kernel from your main installation not the virtual environment you just installed. You can tell Jupyter which kernel to use - Have you tried that?
Related
I am learning to develop with PyTorch as well as LibTorch. I have the following line in my ~/.bashrc for dynamic linking of libtorch libraries:
# libtorch linking path
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/user/.dev_libraries/libtorch/lib/
However, when this path is in LD_LIBRARY_PATH, importing torch in Python reports segmentation fault:
user#host:~$ $LD_LIBRARY_PATH
bash: /home/user/packages/embree-2.16.0.x86_64.linux/lib:/home/user/packages/embree-2.16.0.x86_64.linux/lib::/usr/local/lib/:/usr/local/cuda-11.1/lib64:/usr/local/lib/:/usr/local/cuda-11.1/lib64:/home/user/.dev_libraries/libtorch-cpu/libtorch/lib/: No such file or directory
user#host:~$ python
Python 3.8.10 (default, Jun 2 2021, 10:49:15)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Segmentation fault (core dumped)
user#host:~$
As soon as I remove that path from the environment variable LD_LIBRARY_PATH, torch can be correctly imported in Python.
I am guessing the cause is that some shared libraries of PyTorch having the same names as the ones in LibTorch. Does this mean PyTorch and LibTorch cannot be installed simultaneously, or is my environment setting incorrect? I'd prefer not to reset LD_LIBRARY_PATH every time I switch between the two.
System specs:
Ubuntu 20.04 + CUDA 11.1 + python 3.8.10 + GCC 9.3.0
pytorch 1.9.0+cu111
libtorch is downloaded from here: https://download.pytorch.org/libtorch/nightly/cpu/libtorch-shared-with-deps-latest.zip
I faced the same problem too.
You can type
import torch
print(torch.__version__)
to see the version of torch, and use the same version of libtorch, that would solve the problem probably.
I have a python script that was converted from an .ipynb notebook from Google Colab that I'm trying to run natively on my Mac running Big Sur.
When I try to run the script using python scriptname.py it gives me the following error:
NTEL MKL ERROR: dlopen(/Users/MyUser/opt/anaconda3/lib/libmkl_core.dylib, 9): image not found.
Intel MKL FATAL ERROR: Cannot load libmkl_core.dylib.
It occurs immediately after I try to import pandas.
I tried conda update numpy as suggested here but the problem persists.
I wondered if I didn't have pandas installed but I did pip install pandas and I got Requirement already satisfied for all 6 packages.
Any help would be appreciated.
Try following, to make sure it's not something with your conda stuff.
> python3 -m venv venv-38
> source venv-38/bin/activate
(venv-38) > pip3 install pandas
...
...
(venv-38) > python
Python 3.8.1 (v3.8.1:1b293b6006, Dec 18 2019, 14:08:53)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> quit()
** Update **
You can easily reuse this environment in all future terminal sessions. Let's say you have this venv-38 inside $HOME. All you have to do is to open terminal session, and then
> source $HOME/venv-38/bin/activate
since now, your Python environment will be used as if it was installed inside venv-38 directory.
I finally found a solution for this. numpy also gave me the same problem so I uninstalled it pip uninstall numpy and reinstalled it pip install numpy.
I suppose I'll have to do this with all packages as I use them, but perhaps pyenv will provide a useful solution to my problem if. ever get it working.
I'm running an Anaconda virenv that has successfully installed OpenCV ver.3.4.1. From the python prompt I can import cv2 and can confirm it's version per the command line below:
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:07:29)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'3.4.1'
However, in VSCode I run the following simple program and get an error:
import numpy as numpy
import argparse
import cv2
pass
The error is:
Traceback (most recent call last):
File "detect_faces.py", line 3, in <module>
import cv2
ModuleNotFoundError: No module named 'cv2'
I don't understand why the command line import works, but the program from VSCode fails to import the same package.
There are multiple SO Q&A on this topic but they are all incredibly time/date sensitive and platform specific -- most deal with the situation where the package itself isn't recognized anywhere. The closest question is this one which is from 2015 and deals with the Android platform. Any thoughts?
Update
The version installed should be 3.5.5, but I noticed that in the command prompt launched from anaconda it's shown as Python 3.5.5 :: Anaconda, Inc. while the version shown within VSCode is Python 3.6.5 :: Anaconda, Inc.. When I look at the available env in VSCode it shows some 2.7s and several 3.6 versions - including the one I'm using which is 3.6.5.
I've just done a new install of a fresh Python 3.6 env and noticed that the command prompt reports Python 3.6.6 while the output from a sys check reports Python 3.6.5 -- again, both have the same executable path.
Although I was unable to replicate this same error, I did determine that it had to do with the installation of numpy. For whatever reason, it originally installed and downgraded the Python version within the venv. Then later when upgrading python it created a mismatch. The fix was to recreate the venv with a pip install of all three packages in the same command line. When installing all three at once the packages all reconcile - - I don't know if there was another variable - I couldn't recreate the original error.
I use sudo pip install tensorflow-1.7.0rc0-cp27-cp27mu-manylinux1_x86_64.whl in centos7,python version is 2.7.14
I get successfully installed tensorflow,but when I run import tensorflow as tf,I get a error:No module named tensorflow
what should I do? thanks!
As mentioned in the comments you best method for installing pyhton packages is to use a virtual environment. For a simple way to do this you can install Minicaonda for linux x86 using the following command:
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash ./Miniconda3-latest-Linux-x86_64.sh
Then it is worth reading the instructions on using conda, to set up a virtual environment called tensorflow. But the basic command should be:
conda create -n tensorflow python=3.6 tensorflow
provided you want to use tensorflow with python3.6. Then activate your tensorflow conda environment:
source activate tensorflow
and you should be able to import tensorflow without issue.
(tensorflow) user#machine:~$ python
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>>
It seems that openCV has been activated, but there is Error message when using iPython Jupyter Notebook to import cv2 : no module named cv2. However, using terminal is good.
Macbook:// Macbook$ source activate opencv
(opencv) Macbook:// Macbook$ python
Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:43:17)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import cv2
>>>
Maybe you forgot to export it to environment. Try:
export PYTHONPATH
I tried this and it worked on one of my Macbooks, but not the other... :
Anyway, hope this could be helpful , though my problem is not solved yet:
” conda install -c https://conda.binstar.org/menpo opencv3 ”
in the Anaconda Prompt window, but this time not from within the opencv environment.
, from a comment within https://rivercitylabs.org/up-and-running-with-opencv3-and-python-3-anaconda-edition/
That means,
step1, source deactivate opencv
step2, conda install -c https://conda.binstar.org/menpo opencv3
Note however the link give different steps. Do not follow it. It tells us to do
conda create -n opencv numpy scipy scikit-learn matplotlib python=3
source activate opencv
conda install -c https://conda.binstar.org/menpo opencv3
install python and opencv in virtual environment using packages virtualenv and virtualenvwrapper, detailed explanation can be found here
And then use jupyter notebook in that virtual environment, then import cv2 should work.