For python 3.8.8 and using the new mac air (with the m1 chip), in jupyter notebooks and in python terminal, import jax raises this error
Python 3.8.8 (default, Apr 13 2021, 12:59:45)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import jax
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/steve/Documents/code/jax/jax/__init__.py", line 37, in <module>
from . import config as _config_module
File "/Users/steve/Documents/code/jax/jax/config.py", line 18, in <module>
from jax._src.config import config
File "/Users/steve/Documents/code/jax/jax/_src/config.py", line 26, in <module>
from jax import lib
File "/Users/steve/Documents/code/jax/jax/lib/__init__.py", line 63, in <module>
cpu_feature_guard.check_cpu_features()
RuntimeError: This version of jaxlib was built using AVX instructions, which your CPU and/or operating system do not support. You may be able work around this issue by building jaxlib from source.
I suspect it occurs because of the m1 chip.
I tried using jax with pip install jax, then I built it from source as suggested by the comment, by cloning their repository and following the instructions given here, but the same error message shows.
Thanks #jakevdp I looked at the issue you linked and found a workaround :
Thanks to Noah who mentioned in issue #5501 that you could just use a previous version of jax and jaxlib, for my purposes jaxlib==0.1.60 and jax==0.2.10 work just fine!
JAX does not yet provide pre-built jaxlib wheels that are compatible with M1 chips. The best source of information I know on building jaxlib on M1 is probably this github issue: https://github.com/google/jax/issues/5501, which also tracks improving this support.
Hopefully M1 support will be improved in the near future, but it's taking a while for the scientific computing infrastructure up and down the stack to catch up with the requirements of the new chips.
I had a similar problem. Since I already had Anaconda installed and didn't want to clutter up my space with Anaconda + miniconda + homebrew versions of python and package management and whatever, I hunted around for a simple solution. What ended up working for me was first uninstalling jax and jaxlib and then installing jax and jaxlib via conda-forge directly:
pip uninstall jax jaxlib
conda install -c conda-forge jaxlib
conda install -c conda-forge jax
As of now (January 2022), jax is available for M1 Macs. Make sure to uninstall jax and jaxlib and then install the new packages via pip:
pip install --upgrade jax jaxlib
Afterwards, you can use jax without problems.
--Edit--
I am running on a machine with the following specs:
ProductName: macOS
ProductVersion: 12.1
BuildVersion: 21C52
and with Python 3.9.6 within a conda environment.
Related
I recently bought a MacBook Air with the Apple M1 chip, and I'm trying to install keras for Python 3.9.10 (installed using homebrew). Using the command
pip3 install keras
in the terminal, I get the following output:
Collecting keras
Using cached keras-2.8.0-py2.py3-none-any.whl (1.4 MB)
Installing collected packages: keras
Successfully installed keras-2.8.0
Next, I entered the following series of commands (find output included as well):
~ % python3
Python 3.9.10 (main, Jan 15 2022, 11:40:53)
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import keras
After running the last line (import keras), I got the following output in the terminal:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/homebrew/lib/python3.9/site-packages/keras/__init__.py", line 21, in <module>
from tensorflow.python import tf2
ModuleNotFoundError: No module named 'tensorflow'
My understanding was that keras is built on top of TensorFlow, however this seems to indicate that I need TensorFlow installed on my computer as well. Is this correct? Or is there something else I'm missing?
I've heard that TensorFlow can be very difficult to install on Apple M1 computers, so I've been trying to avoid doing so, but if I do need to, how should I go about it?
Thank you ahead of time.
UPDATE:
I decided to try and install TensorFlow2 in the terminal using the command
pip3 install tf2
and got the following output:
ERROR: Could not find a version that satisfies the requirement tf2 (from versions: none)
ERROR: No matching distribution found for tf2
I guess this is where the difficulty starts... is there any straightforward way to install TensorFlow2 on an Apple M1 computer? I'd really prefer not to switch compilers or anything as I don't have a great deal of experience with macOS and don't want to make any irreversible mistakes.
Thank you again.
UPDATE 2:
I just tried the command
pip3 install tensorflow
and I got the following output:
ERROR: Could not find a version that satisfies the requirement tensorflow (from versions: none)
ERROR: No matching distribution found for tensorflow
Thank you to Dr. Snoopy for pointing out that the package was called tensorflow and not tf2. I've read the problem might have something to do with a missing multi-threading file or something, but still can't figure out how to fix it.
A tested method using conda:
Ensure you upgrade Mac OS X > 12.3.
Install XCode CLI tools
xcode-select --install
Install Miniforge. I prefer installing using brew
brew install miniforge
Create a conda environment and activate it
conda create --name tensyflow python=3.8
conda activate tensyflow
Install Tensorflow-MacOS
conda install -c apple tensorflow-deps
pip install tensorflow-macos # or pip3
I am trying to install Numba on a cluster so I can run my Jitted python codes on there. However I keep running into an error with "libllvmlite.so" when I try to import Numba. This is done on the Cedar cluster using virtualenv. Numba is installed using pip (and also tried pip3).
I start off my activating my enviroment using (... is the directory to my enviroment folder) :
source ~/.../ENV/bin/activate.
Then I use pip to install Numba (I've also tried used pip uninstall to remove and reinstall Numba).
However when I load up Python 3.7.0 on my enviroment, I obtain an error whenever I try to import Numba.
I use pip show to check versions: Python 3.7.0, Numba 0.53.1, llvmlite 0.36.0.
Here is the output of my terminal:
(ENV) [04:12:46] patrick7#cedar1 > [~/projects/def-mann/patrick7/diffusioncluster]python
Python 3.7.0 (default, Sep 25 2018, 18:19:16)
[GCC 5.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/patrick7/projects/def-mann/patrick7/diffusioncluster/ENV/lib/python3.7/site-packages/numba/__init__.py", line 19, in <module>
from numba.core import config
File "/home/patrick7/projects/def-mann/patrick7/diffusioncluster/ENV/lib/python3.7/site-packages/numba/core/config.py", line 16, in <module>
import llvmlite.binding as ll
File "/home/patrick7/projects/def-mann/patrick7/diffusioncluster/ENV/lib/python3.7/site-packages/llvmlite/binding/__init__.py", line 4, in <module>
from .dylib import *
File "/home/patrick7/projects/def-mann/patrick7/diffusioncluster/ENV/lib/python3.7/site-packages/llvmlite/binding/dylib.py", line 3, in <module>
from llvmlite.binding import ffi
File "/home/patrick7/projects/def-mann/patrick7/diffusioncluster/ENV/lib/python3.7/site-packages/llvmlite/binding/ffi.py", line 191, in <module>
raise OSError("Could not load shared object file: {}".format(_lib_name))
OSError: Could not load shared object file: libllvmlite.so
I have tried other solutions found online (reinstall, try different versions...), so far none has worked. Any help would be greatly appreciated! Thanks!
I figured out the solution to my problem.
What I've tried and didn't work:
Install llvmlite using pip manually
Downgrade to Numba version 0.43 since its the oldest version compatible with Python 3.7
Uninstall (llvmlite numpy and numba) and reinstalling Numba (which automatically installs it's dependancies) after upgrading Pip to the latest version
What worked:
Upgrading Python from 3.7.0 to 3.7.10
It seems like for Python version >= 3.7.3, the default command pip install numba works flawlessly!
Thanks for reading!
Do you know how I could fix this? I am trying to use https://github.com/benjiebob/SMALViewer/issues/3 repo however I get error on neural_renderer port:
$ python smal_viewer.py
Traceback (most recent call last):
File "smal_viewer.py", line 2, in <module>
import pyqt_viewer
File "/home/mona/research/3danimals/SMALViewer/pyqt_viewer.py", line 13, in <module>
from smal.smal3d_renderer import SMAL3DRenderer
File "/home/mona/research/3danimals/SMALViewer/smal/smal3d_renderer.py", line 6, in <module>
import neural_renderer as nr
File "/home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/__init__.py", line 3, in <module>
from .load_obj import load_obj
File "/home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/load_obj.py", line 8, in <module>
import neural_renderer.cuda.load_textures as load_textures_cuda
ImportError: /home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/cuda/load_textures.cpython-37m-x86_64-linux-gnu.so: undefined symbol: THPVariableClass
Here is some details:
$ python
Python 3.7.6 (default, Jan 8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.6.0'
>>> torch.version.cuda
'10.1'
>>> torch.cuda.is_available()
True
$ gcc --version
gcc (Ubuntu 9.3.0-10ubuntu2) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$ lsb_release -a
LSB Version: core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal
Here is the neural renderer git repo: https://github.com/daniilidis-group/neural_renderer
I installed the neural renderer using pip install neural_renderer_pytorch
It looks like you didn't build the neural_renderer_pytorch yourself, but used a wheel. However, this wheel was built with an older pytorch version and doesn't work with the current pytorch version on your machine.
Build neural_renderer from the source (after deinstalling neural_renderer you have now) using your current pytorch-version i.e.
$ pip uninstall neural-renderer-pytorch
$ pip install https://github.com/daniilidis-group/neural_renderer/zipball/master
and it should work.
Until pytorch 1.5, it used a someway brittle way of building extensions on Linux: despite depending on torch, extensions didn't link explicitly against libtorch.so. The missing symbols were provided only because import torch loaded libtorch.so with RTLD_GLOBAL, thus making its symbols globally visibile/accessible - this is the reason why prior to loading those extensions (e.g. neural_renderer_pytorch like here) one had to import torch.
One could enforce the old behavior setting RTLD_GLOBAL prior to importing torch for the very first time it is happening:
import sys; import ctypes;
sys.setdlopenflags(sys.getdlopenflags() | ctypes.RTLD_GLOBAL)
import torch # now all symbols of torch
# have global visibility and can be used in
# other extensions
However, using RTLD_GLOBAL is quite dangerous as it could possibly interpose symbols that are unrelated and lead to subtle bugs or even crashes.
Thus, since 1.5 pytorch no longer uses RTLD_GLOBAL, but links explicitly against libpytorch.so (see this commit) and extensions built with older pytorch versions will not work.
I was able to fix this by first finding all the AT_CHECK and converting them to AT_ASSERT. Checking out the at_assert_fix branch didn't solve the problem for me. I am using PyTorch 1.6 and CUDA 10.1
https://github.com/daniilidis-group/neural_renderer/search?q=AT_CHECK
(base) mona#mona:~/neural_renderer/neural_renderer/cuda$ vi create_texture_image_cuda.cpp
(base) mona#mona:~/neural_renderer/neural_renderer/cuda$ vi load_textures_cuda.cpp
(base) mona#mona:~/neural_renderer/neural_renderer/cuda$ vi rasterize_cuda.cpp
(base) mona#mona:~/neural_renderer$ pip install .
Processing /home/mona/neural_renderer
Building wheels for collected packages: neural-renderer-pytorch
Building wheel for neural-renderer-pytorch (setup.py) ... done
Created wheel for neural-renderer-pytorch: filename=neural_renderer_pytorch-1.1.3-cp37-cp37m-linux_x86_64.whl size=6215781 sha256=fbd9e8ef7340fa71b9fcd69da8ee06a1079ebbe5b4c2c3ce92e8124bc8cea7c5
Stored in directory: /tmp/pip-ephem-wheel-cache-tddilaof/wheels/c6/9b/9b/d2cda4f9ac2127278c21ea5c5e23c3354fe0e63365b7af7842
Successfully built neural-renderer-pytorch
Installing collected packages: neural-renderer-pytorch
Successfully installed neural-renderer-pytorch-1.1.3
(base) mona#mona:~/neural_renderer$ python examples/example1.py
Drawing: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 90/90 [00:04<00:00, 19.37it/s]
https://github.com/daniilidis-group/neural_renderer/issues/94
I have already installed pytorch for MacOS 10.14 with pip3, but I can not import it in the python script. What should I do?
System: MacOS 10.14
Python3: v3.7
➜ ~ pip3 list
Package Version
----------- -----------
numpy 1.16.2
Pillow 5.4.1
pip 18.1
pycairo 1.17.1
pygobject 3.28.3
setuptools 40.5.0
six 1.12.0
torch 1.0.1.post2
torchvision 0.2.2.post3
virtualenv 16.1.0
wheel 0.32.2
➜ ~ python3
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
>>>
To expand upon my comment:
There's no strict guarantee that a pip3 wrapper script somewhere on your system is related to the pip package manager/module for your python3 binary. That wrapper may be created by a different installation of Python – maybe your system's own, maybe something else. (You can see where the script is located with which pip3 and see which interpreter it uses with less $(which pip3) and looking at the shebang line at the top.)
Each version of Python you have installed has its own site-packages directory, which contains the globally (as far as that version is concerned) installed packages. Fortunately, pip can be run exactly equally as the wrapper script would with the -m switch, so to be sure Torch and Torchvision get installed into your python3 (which appears to be Python 3.7.0 at this time),
python3 -m pip install torch torchvision
should do the trick.
However, globally (well, interpreter-globally, as discussed above) installed packages should be avoided, since you can easily get into hairy conflicts when you're working on multiple projects. You should instead use virtualenvs to separate your library installations from each other – the venv module is included in Python these days, and the official documentation has a guide on it. (Other options are pipenv and poetry, but it's worth knowing the lower-level tooling.)
I have installed (and uninstalled) and installed GDAL 2.2.2 in Windows 10 using Christoph wheel. However, when I try to import it I get:
Installing collected packages: GDAL
Successfully installed GDAL-2.2.2
(e4) C:\Users\R1702898\Documents\Empirical_v4>python
Python 3.4.4 |Continuum Analytics, Inc.| (default, Jun 15 2016, 15:25:08)
[MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from osgeo import ogr
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\R1702898\Documents\e4\lib\site-packages\osgeo\__init__.py", line 35, in <module>
_gdal = swig_import_helper()
File "C:\Users\R1702898\Documents\e4\lib\site-packages\osgeo\__init__.py", line 31, in swig_import_helper
_mod = imp.load_module('_gdal', fp, pathname, description)
File "C:\Users\R1702898\Documents\e4\lib\imp.py", line 243, in load_module
return load_dynamic(name, filename, file)
ImportError: DLL load failed with error code -1073741795
The only way it is possible to install in Windows is using the wheels. All else fails. The only available wheel is for version 2.2.2
All other dependencies: Fiona, six, numpy, pandas are installed. Although geopandas won't import either (because it needs ogr). I have also set the path to the bin.
Any help. Please. Thanks.
One of the things i would avoid is to mix and match different ways of installing. If you are using Conda already, stick to it. I only use pip install as a last resort. Because GDAL uses compiled code, it can get messy if you install packages relying on GDAL which are build with different compilers, something you probably are encountering. That's exactly what Conda solves by carefully looking at requirements and dependencies.
In your case, part of the difficulty comes from using Fiona, although its a great package, they have some very strict version requirements making it not the easiest dependency to have. See the yaml from Conda-Forge:
https://github.com/conda-forge/fiona-feedstock/blob/master/recipe/meta.yaml
On my machine (Win 10 64bit) its still possible to get all packages you list above by only using Conda,
My approach would be:
1) Update your Conda to a recent version (conda update conda -n root)
2) Create a new environment:
conda create -n gdaltest python=3.6 gdal fiona geopandas numpy -c conda-forge
Because of the restrictions in some of the recipes, it solves (for me) to using for example Numpy 1.11, instead of the latest 1.13.
3) Activate the new environment (activate gdaltest) and check if the activation script has worked. That should set the environment variables correctly, this is an important step when it comes to conflicts.
In cmd run SET and check if GDAL_DATA is set to the environment you just activated. Also check if your activate environment comes first in the PATH variable, it should have added several folders from your environment (like ..\Library\bin, ..\Scripts etc).
That should be sufficient. You can check it by starting Python (make sure the correct environment is active) and start importing gdal, fiona etc.