Develop Reference

Install tensorflow GPU

i'm trying install tensorflow gpu on win10.
my steps:
install Python 3.5.2
install Nvidia CUDA(v9)+cudnn
copy cudnn files on prog.files directory NVIDIA GPU Computing Toolkit
install tensorflow gpu (pip3 install --upgrade tensorflow-gpu)
C:\Python>python
Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
import tensorflow as tf
Traceback (most recent call last):
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in swig_import_helper
return importlib.import_module(mname)
File "C:\Python\lib\importlib__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 986, in _gcd_import
File "", line 969, in _find_and_load
File "", line 958, in _find_and_load_unlocked
File "", line 666, in _load_unlocked
File "", line 577, in module_from_spec
File "", line 906, in create_module
File "", line 222, in _call_with_frames_removed
ImportError: DLL load failed: Не найден указанный модуль.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 41, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 21, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 20, in swig_import_helper
return importlib.import_module('_pywrap_tensorflow_internal')
File "C:\Python\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: No module named '_pywrap_tensorflow_internal'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python\lib\site-packages\tensorflow\__init__.py", line 24, in <module>
from tensorflow.python import *
File "C:\Python\lib\site-packages\tensorflow\python\__init__.py", line 49, in <module>
from tensorflow.python import pywrap_tensorflow
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 52, in <module>
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in swig_import_helper
return importlib.import_module(mname)
File "C:\Python\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
File "<frozen importlib._bootstrap>", line 577, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 906, in create_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
ImportError: DLL load failed: Не найден указанный модуль.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 41, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 21, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Python\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 20, in swig_import_helper
return importlib.import_module('_pywrap_tensorflow_internal')
File "C:\Python\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: No module named '_pywrap_tensorflow_internal'
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/install_sources#common_installation_problems
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.
If I install tensorflow without GPU- all OK.

It is really important to follow the TensorFlow install instructions, thereby choosing the correct versions:
CUDA® Toolkit 8.0. For details, see NVIDIA's documentation Ensure that you append the relevant Cuda pathnames to the %PATH% environment variable as described in the NVIDIA documentation.
The NVIDIA drivers associated with CUDA Toolkit 8.0.
cuDNN v6 or v6.1. For details, see NVIDIA's documentation. Note that cuDNN is typically installed in a different location from the other CUDA DLLs. Ensure that you add the directory where you installed the cuDNN DLL to your %PATH% environment variable.
GPU card with CUDA Compute Capability 3.0 or higher. See NVIDIA documentation for a list of supported GPU cards.
Currently, the newest version for the CUDA Toolkit is 9.0, but for Tensorflow you need version 8, which is available on the CUDA toolkit archive:
https://developer.nvidia.com/cuda-80-ga2-download-archive
And for the cuDNN (Deep Neural Network libraries), you need to select cuDNN v6, and not the newest version v7: https://developer.nvidia.com/rdp/cudnn-download
Finally, be sure to put the Windows %PATH% variable to the CUDA bin directory (where the dll libraries are).
Then, you can install tensorflow:
pip3 install --upgrade tensorflow-gpu
If it does not work yet, you might want to restart first.

This is how i resolved it,
If you are using pip or pip3 you have less chance succeeding in first time.
Uninstall Python,CUDA, CuNN, Tensorflow-gpu (if you already installed)
Install ANACONDA or MINICONDA
Open ANACONDA prompt and run following command:
conda create --name tf_gpu tensorflow-gpu
This will create an environment tf_gpu whcih will install all compatible versions of Python, CUDA, CuNN and Tensorflow
once all the packages installed open the ANACONDA prompt and type the following command
conda activate tf_gpu
This will change the conda '(base)' to '(tf_gpu)' environment
Then type python as usual to activate the python prompt
If you want to deactivate the 'tf_gpu' environment
conda deactivate

Currently, these versions are supported. Do not try to install more recent versions.
Visual Studio 2015.
CUDA 9.0.
cuDNN 7.0 (make sure to copy the dlls to the right directory).
Python 3.5.
Then you can install tf

I have tried so many blogs myself. I always got one or the other errors.
However, the following worked perfectly for me.
conda create --name tf_gpu tensorflow-gpu
You simply need to have Nvidia drivers installed on your system. You don't need to install CUDA and CudNN, and worry about the version compatibility.
This command will install the CUDA and CudNN alongwith the compatible version of Tensorflow for you.

Finally this documentation worked for me: Anaconda | TensorFlow
The document is self-explanatory. If you already have an environment, just run:
conda install tensorflow-gpu
Important notes:
GPU TensorFlow uses CUDA.
On Windows and Linux only CUDA 10.0 is supported for the TensorFlow 2.0 release. Previous versions of TensorFlow support other version of CUDA.

Ok, so I have referred to many blogs and StackOverflow answers, but none of them worked for me. However, I have at last found a procedure that worked perfectly fine for me. So, I have an NVIDIA GTX 1650 GPU, and below are the steps that I followed to get GPU enabled TensorFlow up and running:
1) Install the NVIDIA Drivers (Studio Driver) and restart your PC.
2) Download the visual studio 2019 Community version (In it, install all the c++, python/AI/DL dependencies).
3) Install Cuda 10.1 (update 2,released in august,2019).
4) Install CuDNN 7.6.x (choose x accordingly, so as per Cuda 10.1 (august,2019), the appropriate value for x is 3). So I basically installed CuDNN 7.6.3 for Cuda 10.1
5) Then, we have to add the bin, include, lib\x64 folders of Cuda to the System Path. (Note that include, lib\x64 will be added by default to the path after installing Cuda. So we just have to add the bin folder of Cuda to the path).
6) Next, we have to add the bin folder of CuDNN to the System Path.
7) Finally, type pip install TensorFlow in CMD, and restart your PC.
Note that if you have a different GPU, you can check the official websites of Cuda and CuDNN to check which versions of Cuda and CuDNN are compatible with respect to your GPU.

If you are using anaconda then run this command in your anaconda prompt
conda install -c anaconda tensorflow-gpu
This will also install the cuda toolkit and cudnn for you and you are good to go
At website it mentions that this command is for linux so I am not sure whether it will work on window, but you can try

"import cntk" works from command line but not in PyCharm

I'm using Anaconda3, python 3.6.1, and PyCharm 2017.1.5.
Using conda, I can perform the following just fine:
(C:\Anaconda3) C:\Users\tim>activate cntk
(note: cntk is a conda virtual environment)
(cntk) C:\Users\tim>python
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:25:24)
[MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import cntk
Here it waits for a while... but works.
>>> print(cntk.__version__)
2.0
So, working fine.
On to PyCharm:
I have a file, tutorial101.py, that just contains import cntk.
Running this in PyCharm:
C:\Anaconda3\envs\cntk\python.exe C:/coding/python/cntk-sandbox/tutorial101.py
Traceback (most recent call last):
File "C:\Anaconda3\envs\cntk\lib\site-packages\cntk\cntk_py.py", line 18, in swig_import_helper
return importlib.import_module(mname)
File "C:\Anaconda3\envs\cntk\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 978, in _gcd_import
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 948, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'cntk._cntk_py'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/coding/python/cntk-sandbox/tutorial101.py", line 1, in <module>
import cntk
File "C:\Anaconda3\envs\cntk\lib\site-packages\cntk\__init__.py", line 10, in <module>
from . import cntk_py
File "C:\Anaconda3\envs\cntk\lib\site-packages\cntk\cntk_py.py", line 21, in <module>
_cntk_py = swig_import_helper()
File "C:\Anaconda3\envs\cntk\lib\site-packages\cntk\cntk_py.py", line 20, in swig_import_helper
return importlib.import_module('_cntk_py')
File "C:\Anaconda3\envs\cntk\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: DLL load failed: The specified module could not be found.
Process finished with exit code 1
Why does this work from the command line, but not in PyCharm? What am I doing wrong? I can't find a satisfactory answer anywhere.

The issue is that the system environment that is setup when you are using Pycharm does not include the C:\Anaconda3 directory as part of the Windows file system search path. It is necessary for this directory to appear in the search path because the CNTK DLLs will have been installed (assuming a standard conda install) to that directory.
You will be able to fix this issue by adding C:\ANACONDA3 (or the path where CNTK was installed) to the PATH environment variable for your system.
You can access the environment variables in Windows under System->Advanced System Settings->Environment Variables, then add a new row to the system PATH variable by selecting it and clicking Edit.
Ian

In case anyone else comes across this, I had the same problem using VS2015. Adding the path to the virtual environment being used (e.g. C:\Users\<your user name>\AppData\Local\Continuum\Miniconda2\envs\<environment with cntk>) in the Environment variable PATH then restarting visual studio resolved the issue

I ended up adding D:\Docs\Anaconda3\Lib\site-packages\cntk to my path
since i have anaconda installed in D:\Docs\Anaconda3

When switching/activating conda environment in Anaconda Console, the script will set/change bunch of environment variables. It seems PyCharm didn't simulate such operation. I hope PyCharm could provide better solution, I worked around this issue by adding "C:\Users\\AppData\Local\Continuum\anaconda3\envs\" into environment variable.
Here is how I deal with it:
At the right corner of PyCharm Navigation bar, there is drop-down menu with your current model name on it. choose "Edit Configurations..."
On the right window pane, you should be able to see "Configuration" tab, and in "Environment" session, there is "Environment variables:" input box
Click "..." to edit Environment variables
Click "+" and add "PATH" variable, the value is set to "C:\Users\\AppData\Local\Continuum\anaconda3\envs\"
Save your change

Can't install pygame_sdl2

I'm creating a tower defense game using python and pygame. I would like to add this one on the Google Play Store, but I need to create an apk file.
I'm following a tutorial on GitHub:
https://github.com/renpytom/rapt-pygame-example.
I follow all the instructions but can't install pygame_sdl2.
I use the command prompt, and use $ python setup.py install in the appropriate directory.
But it gave me this error :
C:\Users\gomes\Downloads\pygame_sdl2-master>python setup.py install
Traceback (most recent call last):
File "setup.py", line 59, in <module>
parse_cflags([ "sh", "-c", "sdl2-config --cflags" ])
File "C:\Users\gomes\Downloads\pygame_sdl2-master\setuplib.py", line 93, in parse_cflags
output = subprocess.check_output(command, universal_newlines=True)
File "C:\Python27\lib\subprocess.py", line 212, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "C:\Python27\lib\subprocess.py", line 390, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 640, in _execute_child
startupinfo)
WindowsError: [Error 2] Le fichier spÚcifiÚ est introuvable
I tried to reinstall pygame_sdl2-master, but gave me the same result.

You need the appropriate Windows dependencies and cython. I did some researching and testing and found this solution:
Install cython, type this in a terminal:
$ pip install cython
Go to your pygame_sdl2 directory and download the Windows dependencies by typing:
$ git clone https://github.com/renpy/pygame_sdl2_windeps
If it's not in the pygame_sdl2 directory, just move pygame_sdl2_windeps there manually.
Start building the library:
$ python setup.py install

I directly download pygame_sdl2_windeps on github. I have copied this one one the pygame_sdl2 directory and use the "python setup.py install" command in the command prompt, and IT WORKS !!!
C:\PGS4A\pygame_sdl2>python
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygame_sdl2
>>>
Thank you for your help !!!

cloudera 5 python API cannot create cluster

I have installed the new cloudera 5 manager by manual installation
wget http://archive.cloudera.com/cdh5/one-click-install/redhat/6/x86_64/cloudera-cdh-5-0.x86_64.rpm
rpm -ivh cloudera-cdh-5-0.x86_64.rpm
cd /etc/yum.repos.d/
wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo
cd /usr/src
wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/RPMS/x86_64/oracle-j2sdk1.7-1.7.0+update45-1.x86_64.rpm
rpm -ivh oracle-j2sdk1.7-1.7.0+update45-1.x86_64.rpm
yum -y install cloudera-manager-server cloudera-manager-server-db-2
/etc/init.d/cloudera-scm-server-db stop
/etc/init.d/cloudera-scm-server-db start
/etc/init.d/cloudera-scm-server stop
/etc/init.d/cloudera-scm-server start
I am using the python API to create the cluster...
http://cloudera.github.io/cm_api/docs/python-client/
however I am getting this error
[kshk#localhost ~]$ python
Python 2.7.5 (default, Nov 12 2013, 16:19:08)
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from cm_api.api_client import ApiResource
>>> import socket
>>> cm_host = "192.168.174.190"
>>> api = ApiResource(cm_host, username="admin", password="admin")
>>> cluster = api.create_cluster("clusterD", "CDH5")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/site-packages/cm_api-6.0.1-py2.7.egg/cm_api/api_client.py", line 108, in create_cluster
return clusters.create_cluster(self, name, version)
File "/usr/lib/python2.7/site-packages/cm_api-6.0.1-py2.7.egg/cm_api/endpoints/clusters.py", line 34, in create_cluster
data=[apicluster])[0]
File "/usr/lib/python2.7/site-packages/cm_api-6.0.1-py2.7.egg/cm_api/endpoints/types.py", line 141, in call
return ApiList.from_json_dict(ret, method.im_self, ret_type)
File "/usr/lib/python2.7/site-packages/cm_api-6.0.1-py2.7.egg/cm_api/endpoints/types.py", line 419, in from_json_dict
items = [ attr.from_json(resource_root, x) for x in dic[ApiList.LIST_KEY] ]
File "/usr/lib/python2.7/site-packages/cm_api-6.0.1-py2.7.egg/cm_api/endpoints/types.py", line 97, in from_json
return self._atype.from_json_dict(data, resource_root)
File "/usr/lib/python2.7/site-packages/cm_api-6.0.1-py2.7.egg/cm_api/endpoints/types.py", line 286, in from_json_dict
obj._set_attrs(dic, allow_ro=True)
File "/usr/lib/python2.7/site-packages/cm_api-6.0.1-py2.7.egg/cm_api/endpoints/types.py", line 221, in _set_attrs
attr = self._check_attr(k, allow_ro)
File "/usr/lib/python2.7/site-packages/cm_api-6.0.1-py2.7.egg/cm_api/endpoints/types.py", line 234, in _check_attr
(name, self.__class__.__name__))
AttributeError: Invalid property fullVersion for class ApiCluster.
any ideas???

there was an issue with the python cloudera API, after four hours of this post; the API was updated and it works fine now...

what's wrong, why python manage.py runserver failed?

failed to run django for lack of sqlite? in fact sqlite is already install in my system.
(py1)[mirror#home ci]$ python manage.py runserver
Validating models...
Unhandled exception in thread started by <bound method Command.inner_run of <django.contrib.staticfiles.management.commands.runserver.Command object at 0x9ea3d8c>>
Traceback (most recent call last):
File "/home/mirror/build/py1/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 91, in inner_run
self.validate(display_num_errors=True)
File "/home/mirror/build/py1/lib/python2.7/site-packages/django/core/management/base.py", line 266, in validate
num_errors = get_validation_errors(s, app)
File "/home/mirror/build/py1/lib/python2.7/site-packages/django/core/management/validation.py", line 23, in get_validation_errors
from django.db import models, connection
File "/home/mirror/build/py1/lib/python2.7/site-packages/django/db/__init__.py", line 40, in <module>
backend = load_backend(connection.settings_dict['ENGINE'])
File "/home/mirror/build/py1/lib/python2.7/site-packages/django/db/__init__.py", line 34, in __getattr__
return getattr(connections[DEFAULT_DB_ALIAS], item)
File "/home/mirror/build/py1/lib/python2.7/site-packages/django/db/utils.py", line 92, in __getitem__
backend = load_backend(db['ENGINE'])
File "/home/mirror/build/py1/lib/python2.7/site-packages/django/db/utils.py", line 24, in load_backend
return import_module('.base', backend_name)
File "/home/mirror/build/py1/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
File "/home/mirror/build/py1/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 31, in <module>
raise ImproperlyConfigured("Error loading either pysqlite2 or sqlite3 modules (tried in that order): %s" % exc)
django.core.exceptions.ImproperlyConfigured: **Error loading either pysqlite2 or sqlite3 modules** (tried in that order): No module named _sqlite3
the focus is that : Error loading either pysqlite2 or sqlite3 modules
evidence to show that sqlite is already installed:
(py1)[mirror#home ci]$ rpm -qa | grep sqlite
python-sqlite-1.1.7-1.2.1
sqlite-devel-3.3.6-5
sqlite-3.3.6-5
(py1)[mirror#home ci]$ python -V
Python 2.7.3
(py1)[mirror#home ci]$ python manage.py runserver
so, why python manage.py runserver failed?
Edit 1:
now i know that i haven't build sqlite modue:
(py1)[mirror#home Python-2.7.3]$ python
Python 2.7.3 (default, Sep 23 2012, 21:42:51)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/mirror/insp/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
from dbapi2 import *
File "/home/mirror/insp/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: No module named _sqlite3
i find sqlite code is included by python source code, but how to build it?
[mirror#home Python-2.7.3]$ ls Modules/_s
_sqlite/ _sre.c _sre.o _ssl.c _struct.c
(py1)[mirror#home Python-2.7.3]$ ls Modules/_sqlite/
cache.c connection.h microprotocols.c module.h row.c statement.c util.h
cache.h cursor.c microprotocols.h prepare_protocol.c row.h statement.h
connection.c cursor.h module.c prepare_protocol.h sqlitecompat.h util.c
./configure --help | grep sqlite returns nothing....
(py1)[mirror#home Python-2.7.3]$ ./configure --help | grep sqlite
(py1)[mirror#home Python-2.7.3]$

You need to apt-get install libsqlite3-dev (on debian - sqlite-devel possibly elsewhere) and recompile python.