Develop Reference

Conda environments in VScode

I have two Anaconda environments I created, say A and B, plus the base env.
I have created environments A and B specific to what I do with them (different libraries).
Now I installed VScode. When I open VScode and set first the correct interpreter (A or B) in the lower part and run some python programs specific to A and B, the program does not run and error appears.
D:\Anaconda\envs\practica-mldl\lib\site-packages\numpy\__init__.py:138: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service
from . import _distributor_init
Traceback (most recent call last):
File "D:\Anaconda\envs\practica-mldl\lib\site-packages\numpy\core\__init__.py", line 22, in <module>
from . import multiarray
File "D:\Anaconda\envs\practica-mldl\lib\site-packages\numpy\core\multiarray.py", line 12, in <module>
from . import overrides
File "D:\Anaconda\envs\practica-mldl\lib\site-packages\numpy\core\overrides.py", line 7, in <module>
from numpy.core._multiarray_umath import (
ImportError: DLL load failed while importing _multiarray_umath: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "d:\Do\Python\ML practice\iris\own-knn.py", line 15, in <module>
import numpy as np
File "D:\Anaconda\envs\practica-mldl\lib\site-packages\numpy\__init__.py", line 140, in <module>
from . import core
File "D:\Anaconda\envs\practica-mldl\lib\site-packages\numpy\core\__init__.py", line 48, in <module>
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.8 from "D:\Anaconda\envs\practica-mldl\python.exe"
* The NumPy version is: "1.19.2"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: DLL load failed while importing _multiarray_umath: The specified module could not be found.
PS D:\Do\Python\ML practice\iris> conda activate practica-mldl
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
If using 'conda activate' from a batch script, change your
invocation to 'CALL conda.bat activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- cmd.exe
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
But when I open VScode through anaconda prompt by first activating the environment and opening VScode by typing code and running some programs, the program runs.
So from what I understand, simply changing the interpreter in VScode is not the same as opening VScode by activating first the environment. I thought by just changing the interpreter in VScode will work.
Is there a setup in VScode where I do not have to open VScode by activating the environments first so I can run python programs specific to that environment?
Thanks a lot!
Edit 1. Added error

Python line_profiler not finding module

I recognize that this is an installation failure on my part, and I'm sorry to lay this uninteresting and inconsequential question at your feet, but for the life of me I can't manage to figure out what is going wrong and I've run out of ideas. I'm hoping someone will be able to quickly point out the obvious.
I am trying to profile a python script (using Kern's line_profiler), and the script needs to load the netCDF4 module. I have installed both line_profiler and netCDF4 with pip. Both are reported as present and updated when I queue pip for the list of installed packages.
Without using the profiler, my script runs without problems, meaning that the netCDF4 module is loaded properly. However, if I run 'kernprof -l -v myscript.py' from the "myscript" directory, I get the following error:
Traceback (most recent call last):
File "/usr/local/bin/kernprof", line 9, in <module>
load_entry_point('line-profiler==1.0', 'console_scripts', 'kernprof')()
File "Library/Python/2.7/site-packages/kernprof.py", line 221, in main
execfile(script_file, ns, ns)
File "myscript.py", line 5, in <module>
from netCDF4 import Dataset
ImportError: No module named netCDF4
I am running Python from an installation at /opt/local/bin/python, which is listed first in my PATH.
So, in any case, if the default Python version that I have set is the same as that which appears first in my PATH, and that default version is able to access the netCDF4 module, why isn't line_profiler?

kernprof has a shebang that redirects to the default python install which doesn't have all the required modules.
You can force the use of your "complete" python install by doing:
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/‌​Resources/Python.app‌​/Contents/MacOS/Pyth‌​on /usr/local/bin/kernprof -l -v myscript.py
So shebang is ignored, and you run the profiler with the version of python containing all the required packages.

Error with TensorFlow MNIST [duplicate]

Since updating from Homebrew Python 2.7.11 (from 2.7.10) I'm suddenly unable to test register my package on PyPi from the PyCharm IDE console.
Running (as an "External Tool")
python -B setup.py register -r pypitest
I now get
Traceback (most recent call last):
File "setup.py", line 22, in <module>
from setuptools import setup
File "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 12, in <module>
from setuptools.extension import Extension
File "/usr/local/lib/python2.7/site-packages/setuptools/extension.py", line 8, in <module>
from .dist import _get_unpatched
File "/usr/local/lib/python2.7/site-packages/setuptools/dist.py", line 16, in <module>
from setuptools.depends import Require
File "/usr/local/lib/python2.7/site-packages/setuptools/depends.py", line 6, in <module>
from setuptools import compat
File "/usr/local/lib/python2.7/site-packages/setuptools/compat.py", line 17, in <module>
import httplib
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 80, in <module>
import mimetools
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/mimetools.py", line 6, in <module>
import tempfile
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tempfile.py", line 32, in <module>
import io as _io
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/io.py", line 51, in <module>
import _io
ImportError: dlopen(/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyCodecInfo_GetIncrementalDecoder
Referenced from: /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
Expected in: flat namespace
in /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
Process finished with exit code 1
I'm not sure how to proceed. I only get this issue if I execute from within my IDE's console. If I do it directly at the system command line (Terminal on OS X) I have no problems.
OS X 10.11.3; Homebrew Python 2.7.11; PyCharm 5.0.3

tl;dr: Fix this issue by doing one of the following:
type hash -r python, OR
log out and log in.
EDIT: An answer to my related question makes it clear what's happening here. When you install a new version of python, you may need to run hash -r python to tell bash to reset the "cached" location to the python executable.
In my case, I was typing python, which was on my $PATH at /usr/local/bin/python. But bash was still using the old cache location /usr/bin/python. So, the old executable was called, but the new path was provided to python in sys.argv[0]. This means that the old executable was running, but the new sys.executable value caused all the wrong modules to get loaded (including the io module).
I'm having the same problem. I installed python 2.7.11 via an installer from Python.org. Strangely, the issue seems to be related to some subtle difference between how OSX launches python when I invoke it from the shell using the full path vs. using just the word python.
So, for me, this works (invoking python via the full path /usr/local/bin/python):
$ which python
/usr/local/bin/python
$ /usr/local/bin/python -c "import io"
$
... but this doesn't:
$ python -c "import io"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/io.py", line 51, in <module>
import _io
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyCodecInfo_GetIncrementalDecoder
Referenced from: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
Expected in: flat namespace
in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
So, as a workaround, you can try doing the same thing.
Elsewhere, I've posted a separate question about this puzzling behavior. Maybe somehow merely calling python invokes some strange mix of the 2.7.11 executable with the 2.7.10 dylibs??

According to https://github.com/klen/python-mode/issues/634:
I had the same issue, but successfully fixed. In my case I compiled
python and vim with homebrew, when PYTHON_PATH has been specified and
set to one of my dev environments, where I also had some libraries,
including io. Workaround was simple: open new terminal, make sure that
you do not have custom PYTHON_PATH, uninstall python, uninstall vim.
Reinstall both of them.
and
Problem solved.
Culprit is the update from python 2.7.10 to 2.7.11.
If you are using conda package control, simply run "conda install
python=2.7.10" will solve this problem.
This doesn't give the root cause though. Since this happens with _io, this looks like a bug in python 2.7.11 (unlikely, there would be a world-scale outcry and a prompt fix if it was) or some packaging bug or version mismatch specifically with the homebrew version (and maybe some related ones, too).
Try to import _io in the console and if it succeeds, check if it was loaded from the same path.

Reinstall python.
brew unlink python && brew reinstall python
Secure the path
export PYTHONPATH=$PYTHONPATH:/usr/local/bin/
BACKUP and Change the order of "paths" file.
sudo nano /etc/paths
it seems, the order of paths, it is decisive to run python properly. In my case, the result was:
#sudo nano /etc/paths
/usr/bin
/usr/local/bin
/bin
/usr/sbin
/sbin
On my mac, path is like this.
$ which python
/usr/local/bin/python
Now I can run both:
$ /usr/local/bin/python -c "import io"
$ python -c "import io"

I had the same issue, it is successfully fixed by just replacing the _io.so file.
sudo find / -name _io.so
copy the path of the _io.so file which DOES NOT belong to python-2.7.11. For example, copy the path of _io.so which is under python-2.7.5:
/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
Replace the /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so file with the _io.so that you just found.

This happened to me as well in MacVim. I solved it by making sure :python print(sys.path) is using system Python (e.g. /Library/Python/2.7/...)
Since I installed MacVim via Homebrew, I just did that by:
Spawn a new shell that had which python -> /usr/bin/python. For my case I needed to remove the pyenv line from my .bash_profile. If you installed Python via Homebrew you may want to brew unlink python first
brew reinstall macvim

If your problem is caused by anaconda, it is unnecessary to remove //anaconda directory.
Just open your ~/.bash_profile, find the line
export PATH="//anaconda/bin:$PATH
and comment it out, then restart your terminal session.

Another quick workaround if you don't mind sticking with Python 2.7.10 is to specify the path of the Python interpreter executable that will be used for the virtualenv. On OSX that path is usually /usr/bin/python:
virtualenv venv --python=/usr/bin/python

Can't add comment (?) so this just to share my exp., downgrade to 2.7.10 works fr me.

I got this error after a failed NLTK download, I needed to uninstall anaconda:
sudo rm -rf ~/anaconda
update PATH variable

This happened when I already had tried to create a venv in a folder, and mistakenly was trying to initialize a second one! So I just removed venv directory and re-ran the command. Very likely this is not the answer to this solution, but searching my error brought me here, so it may help some others who are stuck.

I solved this issue by removing the symbolic link that was in /usr/local/bin and copying the actual python binary, that was pointed to by said link, there.

I had the same issue when I tried to use PyCharm. Solved by setting "python interpreter" in project configuration to point to the python virtual env I wanted to use, which was an Anaconda env. Somehow the interpreter path was missing the "anaconda" portion of ~/.../anaconda/.../_io.so. No need to uninstall anaconda.

Error Installation Google Cloud SDK

While following the installation steps for "Mac OS X" Installation and Quick Start
I used the below command:-
curl https://sdk.cloud.google.com | bash
and directory extract under /Users/xxxxx/google-cloud-sdk/
but I got this error:-
Traceback (most recent call last):
File "/Users/xxxxx/google-cloud-sdk/bin/bootstrapping/install.py", line 8, in <module>
import bootstrapping
File "/Users/xxxxx/google-cloud-sdk/bin/bootstrapping/bootstrapping.py", line 9, in <module>
import setup
File "/Users/xxxxx/google-cloud-sdk/bin/bootstrapping/setup.py", line 44, in <module>
from googlecloudsdk.core.util import platforms
File "/Users/xxxxx/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/util/platforms.py", line 6, in <module>
import platform
ImportError: No module named platform
Environment:-
Mac OS X Yosemite 10.10
Python 2.7.6
$ whereis python /usr/bin/python
Can someone help?

I have found different locations of standard modules, for example:-
>>> import platform >>> platform.__file__ => '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/platform‌​.pyc'
>>> import os >>> os.__file__ => somewhere inside my profile /Users/xxxxx/Library/Frameworks/Python.framework/... ,
So, I have solved this issue by following steps:-
removed /Users/xxxxx/Library/Frameworks/Python.framework/.
reinstalled python through brew to keep Apple python away.
then, Google Cloud SDK was installed smoothly and is working properly.
and the new location of standard modules is /usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/

I can't comment since i don't have enough rep, so posting as an "answer".
As others have commented, platform is a standard module. It should be there barring a bad python installation.
Since the python in PATH seems to import platform correctly, maybe Cloud SDK is using a different python.
Check if the environment variable CLOUDSDK_PYTHON is set. That usually tells gcloud to use a different python.
You can also change /Users/xxxxx/google-cloud-sdk/bin/gcloud and right before the last line, echo the bash variables $CLOUDSDK_PYTHON, $CLOUDSDK_PYTHON_ARGS, $CLOUDSDK_ROOT_DIR, and $PYTHONPATH. This should tell us if something is configured badly with the python that gcloud uses.

on linux i fixed this by running install.py directly .you can apply the same on mac os i guess,
go to the folder where you have extracted google cloud sdk then,
cd /bin/bootstrapping
there you will find install.py ,run it using
./install.py
then installation begins and proceed with the steps .After that open a new terminal gcloud works!

Had a similar issue with a different system package (contextlib), solved it by setting the environment parameter: CLOUDSDK_PYTHON_SITEPACKAGES=1 before installing.

Import caffe error

i compiled caffe successfully in my ubuntu machine but cannot import in python.
Caffe is installed /home/pbu/Desktop/caffe
i tried adding the /home/pbu/caffe/python path to sys.path.append, still not working
i am trying to import caffe
root#pbu-OptiPlex-740-Enhanced:/home/pbu/Desktop# python ./caffe/output.py
Traceback (most recent call last):
File "./caffe/output.py", line 13, in <module>
import caffe
File "/home/pbu/Desktop/caffe/python/caffe/__init__.py", line 1, in <module>
from .pycaffe import Net, SGDSolver
File "/home/pbu/Desktop/caffe/python/caffe/pycaffe.py", line 10, in <module>
from ._caffe import Net, SGDSolver
ImportError: No module named _caffe

This happens when you have not run make for the python files separately.
Run make pycaffe soon after running make in the Caffe directory.
You may have to set the path to the python library correctly in Makefile.config

Adding to the above best answer. After you run make for python files by running make pycaffe where you ran your previous makes. Then you have to export that python path by running export PYTHONPATH=<path-to-caffe>/python. You can choose to run this everytime before running a python code which utilizes caffe or add it to your ~/.bashrc.

Well, I use the cmake-gui for making Caffe. There you need to set the Python paths to the Anaconda-python:
PYTHON_EXECUTABLE <path_to_anaconda_home>/bin/python2.7
PYTHON_INCLUDE_DIRECTORY <path_to_anaconda_home>/include/PYTHON2.7
PYTHON_LIBRARY <path_to_anaconda_home>/lib/libpython2.7.so

You should build caffe and pycaffe using the command:
cd $FRCN_ROOT/caffe-fast-rcnn
make -j8 && make pycaffe
and before the compilation, you should create a Makefile.config file and set the corresponding library path, such as python.
More details are presented on the web: bgirshick/py-faster-rcnn.
What's more, when I run the "Beyond the demo" section, it seams that if I Create a symlink of the folder "VOCdevkit" as "VOCdevkit2007" which turns out to be "can't find the dataset". So, I change the folder name as "VOCdevkit2007", and it runs well.

I posted my Caffe install notes (my architecture: Arch Linux x86_64 | Intel i7 CPU ...) in an Anaconda Python 2.7 virtual environment here:
Caffe Installation Notes
https://gist.github.com/victoriastuart/fb2cb22209ccb2771963a25c06221213
I also encountered the (downstream) "Import caffe error," for which I needed to resolve my $PYTHONPATH to complete the make compilation and get Caffe finally installed, and also to be able to import it (in Python).