I try to use pycocotools from python:
$ ipython
Python 3.6.5 | packaged by conda-forge | (default, Apr 6 2018, 13:39:56)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from pycocotools.coco import COCO
Segmentation Error (core dumped)
Can this be related to my CPU missing AVX instructions (xeon E5520) ?
In case pycocotools has issues with the compiled part, please run
pip uninstall -y pycocotools
pip install --no-binary :all: pycocotools
This should compile the C code shipped with pycoctools on your machine.
If your program tried to execute AVX instructions on a CPU that doesn't support them, you'd get SIGILL (Illegal Instruction) on OSes like Linux.
"Segmentation Error" seems to be some kind of custom error message, because the normal string is always "Segmentation Fault". It's possible that there's a bug or stricter alignment requirement in the non-AVX version of code that's selected at runtime; runtime dispatch would be a plausible mechanism for code working on an AVX CPU and segfaulting without AVX.
Related
When compiling my program using Caffe2 I get this warnings:
[E init_intrinsics_check.cc:43] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
[E init_intrinsics_check.cc:43] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
[E init_intrinsics_check.cc:43] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
Since I do want to get multi-threading support for Caffe2, I've searched what to do. I've found that Caffe2 has to be re-compiled setting some arguments while creating the cmake or in the CMakeLists.
Since I already had installed pytorch in a conda env, I have first uninstalled Caffe2 with:
pip uninstall -y caffe2
Then I've followed the instructions from the Caffe2 docs, to build it from sources.
I first installed the dependencies as indicated. Then I downloaded pytorch inside my conda env with:
git clone https://github.com/pytorch/pytorch.git && cd pytorch
git submodule update --init --recursive
At this time I think is the moment to change the pytorch\caffe2\CMakeLists file just downloaded. I have read that in order to enable the multi-threading support is sufficient to enable the option USE_NATIVE_ARCH inside this CMakeLists, however I'm not able to find such option where I'm looking. Maybe I'm doing something wrong. Any thoughts? Thanks.
Here some details about my platform:
I'm on macOS Big Sur
My python version is 3.8.5
UPDATE:
To answer Nega this is what I've got:
python3 -c 'import torch; print(torch.__config__.parallel_info())'
ATen/Parallel:
at::get_num_threads() : 1
at::get_num_interop_threads() : 4
OpenMP not found
Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
mkl_get_max_threads() : 4
Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
std::thread::hardware_concurrency() : 8
Environment variables:
OMP_NUM_THREADS : [not set]
MKL_NUM_THREADS : [not set]
ATen parallel backend: OpenMP
UPDATE 2:
It turned out that the Clang that comes with XCode doesn't support OpenMP. The gcc that I was using was just a symlink to Clang. In fact after running gcc --version I got:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.0 (clang-1200.0.32.29)
Target: x86_64-apple-darwin20.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
I installed from Homebrew gcc-10 and set the alias like this alias gcc='gcc-10'. In fact now with gcc --version this is what I get:
gcc-10 (Homebrew GCC 10.2.0_4) 10.2.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
I've also tried a simple Hello World for OpenMP using 8 threads and everything seems to be working. However after re-running the command:
python3 -c 'import torch; print(torch.__config__.parallel_info())'
I get the same outcome. Any thoughts?
AVX, AVX2, and FMA are CPU instruction sets and are not related to multi-threading. If the pip package for pytorch/caffe2 used these instructions on a CPU that didn't support them, the software wouldnt work. Pytorch, installed via pip comes with multi-threading enabled though. You can confirm this with torch.__config__.parallel_info()
❯ python3 -c 'import torch; print(torch.__config__.parallel_info())'
ATen/Parallel:
at::get_num_threads() : 6
at::get_num_interop_threads() : 6
OpenMP 201107 (a.k.a. OpenMP 3.1)
omp_get_max_threads() : 6
Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
mkl_get_max_threads() : 6
Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
std::thread::hardware_concurrency() : 12
Environment variables:
OMP_NUM_THREADS : [not set]
MKL_NUM_THREADS : [not set]
ATen parallel backend: OpenMP
That being said, if you still want to continue building pytorch and caffe2 from source, the flag your looking for, USE_NATIVE is in pytorch/CMakeLists.txt, one level up from caffe2. Edit that file and change USE_NATIVE to ON. Then continue building pytorch with python3 setup.py build. Note that the flag USE_NATIVE doesn't do what you think it does. It only allows the building of MKL-DNN with CPU native optimization flags. It does not trickle down to caffe2 (except where caffe2 use MKL-DNN obviously.)
New python3 environment created with
conda create -n py3CVCaffe python=3.7
caffe installed with
conda install caffe-gpu=1.0=py37hdc87d0a_5
among the many packages installed are these two
libopencv-3.4.2 | hb342d67_1 40.4 MB
py-opencv-3.4.2 | py37hb342d67_1 1.2 MB
both packages are installed and seem to be functional
$ python
Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>> import cv2
>>>
However certain video functions are absent as the openCV which is provided has not been compiled
cv2.error: OpenCV(3.4.2) /tmp/build/80754af9/opencv-suite_1535558553474/work/modules/highgui/src/window.cpp:632: error: (-2:Unspecified error) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Carbon support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script in function 'cvShowImage'
This is a well known problem as discussed here
So I need to install an openCV that has been compiled with these libraries. Before I do that I have to remove the openCV that was pulled in with caffe.
Unfortunately, conda does not want to-it also wants to remove caffe!
$conda uninstall py-opencv
Collecting package metadata: done
Solving environment: done
## Package Plan ##
environment location: /home/stefan/miniconda3/envs/p37cu9CfeG
removed specs:
- py-opencv
The following packages will be REMOVED:
caffe-gpu-1.0-py37hdc87d0a_5
py-opencv-3.4.2-py37hb342d67_1
Proceed ([y]/n)?
If you only want to remove py-opencv and not any of its reverse dependencies, then use the --force flag. From the conda remove --help:
--force-remove, --force
Forces removal of a package without removing packages
that depend on it. Using this option will usually
leave your environment in a broken and inconsistent
state.
which should be read as "do so at your own risk".
I have installed tensorflow following the https://www.tensorflow.org/install/pip
steps. I installed it through Anaconda.
I also tried using a virtual environment without using anaconda as this page provided (please check image). It also gave the same error.
Following images shows the versions I have installed and its given error.
I am using,
Ubuntu 18.04.1 LTS
x86_64
Processor: Intel(R) Core(TM) 2 Duo CPU T5870 # 2.00 GHz 2.00 GHz
Installed memory (RAM): 2.00 GB
Version:
Error:
I tried to overcome this but, I couldn't find a solution. I am new to tensorflow and trying to install and learn it. Please help me to solve this.
Thanks for the earlier answers.
I was facing the exact problem for SSE4.2 instruction.
I installed tensorflow 1.5 instead of 1.12 and it is working.
Error message on my system
The TensorFlow library was compiled to use SSE4.2 instructions, but these aren't available on your machine.
Aborted (core dumped)
Solution
Here is what I did:
Please run these inside virtual environment:
pip uninstall tensorflow
pip install tensorflow==1.5
You can replace 1.5 with any version that suits your machine.
Try importing it again it should run now
Your CPU is too old for running TensorFlow as it lacks the required Instruction set. Starting with TensorFlow 1.6, binaries use AVX instructions which may not run on older CPUs. - According to Tensorflow.
You have 2 possible solutions:
Using cloud/web based solution, e.g. Google Colab
Upgrade your Machine to a modern build
I have the same error on the laptop HP ProBook 4720s:
python3
Python 3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2019-04-16 17:50:25.829023: F tensorflow/core/platform/cpu_feature_guard.cc:37] The TensorFlow library was compiled to use AVX instructions, but these aren't available on your machine.
Aborted (core dumped)
But I installed TensorFlow via Anaconda conda installation:
conda install tensorflow # install TensorFlow for CPU
conda activate DL3 # activate virtual environment DL3
python
Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> print(tf.__version__)
1.13.1
>>> exit()
I'm trying to get TensorFlow to work on my Win7x64 following this guide:
https://www.tensorflow.org/install/pip
Everything has gone well until 3. Install the TensorFlow pip package
When i do "python" to see the version i get the following:
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit (AMD64)] on win32
When i do "pip install tensorflow" i get:
Could not find a version that satisfies the requirement of tensorflow (from versions: )
No matching distribution found for tensorflow
If i try "python -m pip install --upgrade https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow-1.2.1-cp35-cp35m-win_amd64.whl" i get this:
tensorflow-1.2.1-cp55-cp35m-win_amd64.whl is not a supported wheel on this platform.
Please help me understand what am i doing wrong or why is it not working? How to get it to work? Thank you very much!
You have an unsupported version of Python. Tensorflow currently only supports Python 3.6 and below, not 3.7 and even if you do manage to install it you will run into issues as Python 3.7 made async a keyword and many libraries, like Tensorflow, use this for variable names. In order to make this work on your system you need to downgrade to 3.6, there is actually a thread with troubleshooting steps on this issue on GitHub: https://github.com/tensorflow/tensorflow/issues/17022
I am new to pythran and right now encountering the problem of File format not recognized as follow:
D:\DevWorkSpace\cython>pythran dprod.py
WARNING Pythran support disabled for module: omp
Looking for python27.dll
objdump.exe: D:\Program Files\Anaconda2\python27.dll: File format not recognized
CRITICAL Chair to keyboard interface error
E: Symbol table not found
Mine OS is Win 10 64bit. The version of Pythran is 0.7.4.post1. The python I use is anaconda 2:
Python 2.7.11 |Anaconda 2.3.0 (64-bit)| (default, Feb 16 2016, 09:58:36) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
Because I first tried Cython, I installed Visual Studio 2008 SP1 and Windows SKD for VC 2008 first. Then I installed MinGW for the gcc complier(gcc version 4.93) and follow this answer to use gcc compiler. This was the first time I got this problem File format not recognized.
Then I uninstall pythran and reinstall it again using:
pip install --global-option build_ext --global-option --compiler=mingw32 pythran
and I got this: failed to create process.
I also tried to install pythran from source code from pypi and installed it using:
python setup.py install. It didn't work.
In addition, I can't install pythran from conda, which gave me the following message:
C:\Users\JHW>conda install -c serge-sans-paille pythran
Fetching package metadata: ......
Solving package specifications: .
Error: Package missing in current win-64 channels:
- pythran
Did you mean one of these?
python, ipython
You can search for this package on anaconda.org with
anaconda search -t conda pythran
Finally, I also tried to replace it with another python27.dll, which is copied from \MySQL\MySQL Workbench 6.3 CE. This File format not recognized disappeared, but still pythran didn't work. By "pythran didn't work", I mean
D:\DevWorkSpace\cython>pythran dprod.py
WARNING Pythran support disabled for module: omp
CRITICAL Chair to keyboard interface error
E: Unknown MS Compiler version 1800
Thanks to #cdarke's comment, it seems that I should turn to the issue of compiler though. I have to admit that I omit this error when I posted this question. The reason I left out this question is that when I replaced the python27.dll in anaconda2, the iPython notebook in anaconda can't work (when I started it, a window of cmd pops out and disappears too soon for me to catch errors)
So my questions are:
where to find the suitable python27.dll to solve the File format not recognized problem without damaging the iPython notebook function. (or how to keep the errors at start of iPython notebook)
hints to solve the Unkown MS Compiler version 1800. (I am trying to figure out how to solve the compiler problem right now, but I am new to MS compiler. So any hints would be greatly appreciated!)
Thank you very much in advance!
According to the documentation, Pythran only supports Windows through WinPython.
The reason behind this choice is the lack of good support for all C++11 features support in Visual Studio C++ Compiler…