Related
Problem
I cannot load VTK into python on Windows with support for CUDA, MPI, and TBB. Volume rendering is unreasonably slow for production with the default PyPI distribution (hours or more) and I need to leverage my hardware to speed up computations.
I have successfully built VTK 9.3.0 for Python 3.8.10-x64 with CMake and Visual Studio and have bin, lib, include, and share folders and have added these to PATH, but Python does not see vtk:
Python 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import vtk
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'vtk'
>>>
I attempted to place the generated Lib/site-packages, bin, lib, and include folders in appropriate places in C:\Program Files\Python38, but I still get a DLL load error:
> py -3.8
Python 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import vtk
Traceback (most recent call last):
File "C:\Program Files\Python38\lib\site-packages\vtkmodules\__init__.py", line 13, in <module>
from . import vtkCommonCore
ImportError: DLL load failed while importing vtkCommonCore: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files\Python38\lib\site-packages\vtk.py", line 30, in <module>
all_m = importlib.import_module('vtkmodules.all')
File "C:\Program Files\Python38\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "C:\Program Files\Python38\lib\site-packages\vtkmodules\__init__.py", line 15, in <module>
import _vtkmodules_static
ModuleNotFoundError: No module named '_vtkmodules_static'
>>>
Setup
OS: Windows 10 Enterprise, x64-bit, Build 1909
CPU: 2x Intel(R) Xeon(R) Gold 6248R
Disk: 2TB NVMe M.2 SSD
RAM: 192 GB DDR4
Compute GPUs: 2x NVIDIA Quadro RTX8000 in TCC mode with NVLink
Display GPU: 1x NVIDIA Quadro RTX4000
Visual Studio: Visual Studio 16 2019 Community
Windows SDK: 10.0.19041.0, targeting 10.0.18363
Python: 3.8.10 x64-bit
CMake: 3.21.1 (I use CMake-GUI)
vcpkg: 2021-08-12
VTK Source: vtk-9.0.3-cp38-cp38-win_amd64 (download from VTK website; not GitHub master)
VTKPythonPackage: GitHub Link (...not sure if I need this...)
CUDA Toolkit: 11.4
TBB: 2021.3.0 (via Intel(R) oneAPI Base Toolkit)
MPI: 2021.3.0 (via Intel(R) oneAPI HPC Toolkit)
NOTE: If desired, Embree, OSPRay, and OpenVKL are available in the Intel(R) oneAPI Rendering Toolkit
Preparation
NOTE: There are A LOT of programs to be downloaded, installed, and configured, and they are important for proper building. In an attempt to shorten this post to an MRE (Minimal Reproducable Example), please go straight to the Steps to Reproduce section.
NOTE: When setting Environment Variables, you may get an error that a PATH length greater than 2047 is not permitted. To overcome this hurdle, it is simplest to edit the path directly in the Windows Registry. Open regedit as an Admin, navigate to
Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Environment
and set the Path variable there.
TIP: To edit PATH, I copy and paste from the Registry into Word, then selected and replace ;'s with paragraph endings ^p by selecting Ctrl+H and then using the Replace tool as follows:
When finished, I simply reverse the changes by replacing ^p's with ;s. This is simpler for me than struggling with regular expressions.
1. Install Visual Studio
Download the Visual Studio Installer
Run the installer, choose Visual Studio Community 2019, and select Desktop development with C++. Ensure the following are selected in Individual Components
Windows 10 SDK
C++ ATL for latest...
C++ MFC for latest...
C++ CMake tools for Windows
NOTE: (For me on Windows 10 with Visual Studio 2019, these are v142)
Ensure the path to the corresponding bin, lib, and include folders are in the PATH of System Environment Variables. These can be either:
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\lib
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include
and/or
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64\
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\lib\x64\
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include
NOTE: (I recommend the latter at a minimum. )
Ensure the following additional System Environment Variables have been set by the installer:
VS140COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\
VS160COMCOMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\Tools\
VS2019INSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community
VSSDK140INSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VSSDK\
NOTE: Even though the installer downloads these to Program Files (x86), they are in fact the x64-bit programs (I know...Windows is weird...)
2. Install CUDA
Download and run cuda_11.4.1_471.41_win10.exe (link)
TIP: The local installer will take longer to download, but your installs will run faster. I recommend saving the installer for the future anyway in case your CUDA Toolkit becomes corrupted or you need to install on another machine.
Ensure the following are added to PATH in System Environment Variables
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\bin
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\libnvvp
ASIDE: You MUST use version 11.4 for VTK 9. However, as an aside, when using 10.2 in CMake, if CMake cannot find CUDA, this is probably because the CUDA Toolkit 10.2 installer for some reason does not copy the Visual Studio MSBuild Extensions over to your Visual Studio folder. Copy the files at
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\extras\visual_studio_integration\MSBuildExtensions\
to
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\VC\VCTargets\BuildCustomizations
NOTE: If you want to run your GPU in TCC mode (WARNING: This disables graphics output), and/or if you have two GPUs connected via NVLink and want to activate NVLink, run the following command in command prompt:
> nvidia-smi -i 0 -dm 1
> nvidia-smi -i 1 -dm 1
Assuming you have two GPUs with ID's 0 and 1 you want to run in TCC mode. (To check, simply run nvidia-smi and see which GPUs you want to activate in TCC mode with the -dm switch). The PugetSystems Instructions have change with CUDA 11. NVIDIA's newer drivers, which ship with the NVIDIA Control Panel, no longer have the Configure SLI, Surround, PhysX under 3D Settings, particularly if using a Quadro card rather than a GeForce card. Instead, NVLink will be activated once TCC mode is set, which disables graphics output. The program they have provided only works for CUDA 10, so you will get an error with it when using CUDA 11 even though it is enabled.
NOTE: If you get an error "Intel Graphics Driver for Windows not found", ignore it. You only need the drivers for the CPU.
NOTE: Don't add more than the /lib/ and /libnvvp/ folders to path. CMake might not be able to find CUDA if you do. If you have to add other folders (e.g. CUPTI), I recommend the lib/ and libnvvp/ folder appear FIRST in PATH.
3. Install CMake
Download and install CMake (...that was easy)
4. Install vcpkg
Download and configure vcpkg:
cd C:\
git clone https://github.com/microsoft/vcpkg
cd vcpkg
.\bootstrap-vcpkg.bat
vcpkg integrate install
Set the default triplet in the System Environment Variables to x64-windows:
VCPKG_DEFAULT_TRIPLET=x64-windows
Install ports using vcpkg install. Press Configure in CMake and every time it cannot find a dependency you want installed, search for the dependency with vcpkg search and install it with vcpkg install (to remove a package, use vcpkg remove)._
NOTE: BE AWARE that depending on what you install, these dependencies take A LOT of space and take A LONG TIME to download.
5. Install Intel oneAPI Toolkits
NOTE: These include TBB, MPI, OSPRay, and the Intel Fortran compiler ifort.exe, which you need.
Intel(R) oneAPI Base Toolkit
Intel(R) oneAPI HPC Toolkit
Intel(R) oneAPI Rendering Toolkit (for Embree, OSPRay, and OpenVKL, optionally)
and add the corresponding lib, bin, and include folders to PATH.
6. Install Intel SPMD Compiler
Intel(R) Implicit SPMD Program Compiler
7. Additional Dependencies with Installers
NOTE: You may not need or want need these. I am adding them here for completeness. If CMake complains about not finding these, I recommend installing them.
Nullsoft Scriptable Install System
WiX Toolset
MySQL Community (GPL) Installer for Windows
MySQL ODBC Connector
Strawberry Perl
8. Dependencies that are Unsuccessful on Windows
NOTE: I have tried installing binaries for these and adding them to PATH, but they just don't seem to work on Windows..
NOTE: For each of these, you can also try downloading corresponding python 3.8 zip source file (on GitHub, under Releases, and on conda forge, under Files) to a folder on C:` and then add the subfolders bin, lib, include (and share if available) to the PATH System Environment Variable.
OpenTurns 1.17
GDAL 3.3.1
Steps to Reproduce
I set my PATH first. VTK will make bin, include, lib, and share folders, so I make sure these are at the front of PATH. Here is what I have (HINT: Refer to setting PATH in regedit to avoid the Windows error of PATH length greater than 2047 characters in the Preparation section above)
Open an x64 Native Tools Command Prompt for VS 2019 terminal as an Administrator and run
> cmake-gui
This ensures vcvarsall.bat is run and the corresponding Environment Variables it sets are passed on to CMake-GUI.
Set the Source folder to C:\VTK\src and Build folder to C:\VTK
UPDATE 27-AUG-2021:
Step 3 was the problem. The build folder was previously C:\VTK\build. It needed to be C:\VTK. I corrected this.
Press Configure and select Visual Studio 16 2019 as the generator with x64 as the optional platform:
NOTE: I do not know if libraries for VTK and CUDA should be built for 32-bit. I am using 64-bit as my best guess.
After Configuration completes, you will see many variables, which is understandably overwhelming and it is unclear what the differences are between DEFAULT, YES, NO, WANT, and DON'T WANT. Unfortunately, the answer is hidden tucked away in VTK's CMake Module API Doxygen docs
VTK CMake Module API
For now, I leave everything default and edit the following CMake variables:
TIP: Check the Grouped and Advanced check boxes next to the Add Entry button to see all variables grouped by corresponding project/module:
CMAKE_CXX_MP_FLAG Checked
CMAKE_INSTALL_PREFIX C:/VTK
VTK_GROUP_ENABLE_Imaging WANT
VTK_GROUP_ENABLE_MPI WANT
VTK_GROUP_ENABLE_Qt WANT
VTK_PYTHON_VERSION 3
VTK_SMP_IMPLEMENTATION_TYPE TBB
VTK_USE_CUDA Checked
VTK_USE_MPI Checked
VTK_WRAP_PYTHON Checked
VTK_ENABLE_KITS Checked
and delete
Python2_EXECUTABLE
under Ungrouped Entries
Rerun Configure
Change the following CMake Variables:
Python3_EXECUTABLE C:/Program Files/Python38/python.exe
Python3_INCLUDE_DIR C:/Program Files/Python38/include
Python3_LIBRARY C:/Program Files/Python38/libs/python38.lib
Rerun Configure
Output from 3rd Configure:
Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.18363.
Found Python3: C:/Program Files/Python38/python.exe (found suitable version "3.8.10", minimum required is "3.2") found components: Interpreter Development.Module Development.Embed
Configuring done
My CMake variables now look like this:
ASIDE: Would be nice if the CMake GUI had an option to output variables to a text file, especially for debugging...
ASIDE: Would be nice if CMake GUI could tell you what had actually been activiated/used since everything shows up with DEFAULT value and we don't want to change that...
ASIDE: Would be nice if there was a high level overview somewhere of what all of these modules entail so users can understand what they want/need...
Click Generate
Output from Generate
Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.18363.
Configuring done
Generating done
Open Visual Studio 2019 as an Administrator
Open the Solution file VTK.sln in C:\VTK\build
Change the build type from Debug to Release for speed:
Right-click on ALL_BUILD and select Build. In the end, the output is
========== Build: 398 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========
Right-click on INSTALL and select Build. In the end, the output is:
========== Build: 116 succeeded, 0 failed, 283 up-to-date, 0 skipped ==========
Issue
Please see problem at top of post.
Update: 26-AUG-2021
Per #JensMunk suggestion, I copied the VTK master branch source code to overwrite vtkmodules/__init__.py with support for os.add_dll_directory(), but to no avail:
r"""
Currently, this package is experimental and may change in the future.
"""
from __future__ import absolute_import
import sys
def _windows_dll_path():
import os
_vtk_python_path = 'Lib/site-packages/vtkmodules'
_vtk_dll_path = 'bin'
# Compute the DLL path based on the location of the file and traversing up
# the installation prefix to append the DLL path.
_vtk_dll_directory = os.path.dirname(os.path.abspath(__file__))
# Loop while we have components to remove.
print("Folder Search:")
print("==============")
while _vtk_python_path not in ("", ".", "/"):
# Strip a directory away.
_vtk_python_path = os.path.dirname(_vtk_python_path)
_vtk_dll_directory = os.path.dirname(_vtk_dll_directory)
if _vtk_python_path != "C:\\":
print(_vtk_python_path)
print("==============")
print()
_vtk_dll_directory = os.path.join(_vtk_dll_directory, _vtk_dll_path)
print(f"VTK DLL Directory: {_vtk_dll_directory}")
print()
print("Adding DLLs...")
if os.path.exists(_vtk_dll_directory):
# We never remove this path; it is required for VTK to work and there's
# no scope where we can easily remove the directory again.
print(f"DLL: {_vtk_dll_directory}")
_ = os.add_dll_directory(_vtk_dll_directory)
print()
print("Build tree...")
# Build tree support.
try:
from . import _build_paths
# Add any paths needed for the build tree.
for path in _build_paths.paths:
if os.path.exists(path):
print(f"Path: {path}")
_ = os.add_dll_directory(path)
except ImportError:
# Relocatable install tree (or non-Windows).
print("Passing: Relocatable install tree")
pass
print()
print("Imported!")
# CPython 3.8 added behaviors which modified the DLL search path on Windows to
# only search "blessed" paths. When importing SMTK, ensure that SMTK's DLLs are
# in this set of "blessed" paths.
if sys.version_info >= (3, 8) and sys.platform == "win32":
_windows_dll_path()
# ------------------------------------------------------------------------------
# this little trick is for static builds of VTK. In such builds, if
# the user imports this Python package in a non-statically linked Python
# interpreter i.e. not of the of the VTK-python executables, then we import the
# static components importer module.
try:
from . import vtkCommonCore
except ImportError:
import _vtkmodules_static
# ------------------------------------------------------------------------------
# get the version
__version__ = "9.0.3"
Attempting to import again, I get the same error, so DLLs are still missing:
Python 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import vtk
Folder Search:
==============
Lib/site-packages
Lib
==============
VTK DLL Directory: C:\Program Files\Python38\bin
Adding DLLs...
DLL: C:\Program Files\Python38\bin
Build tree...
Passing: Relocatable install tree
Imported!
Traceback (most recent call last):
File "C:\Program Files\Python38\lib\site-packages\vtkmodules\__init__.py", line 69, in <module>
from . import vtkCommonCore
ImportError: DLL load failed while importing vtkCommonCore: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files\Python38\lib\site-packages\vtk.py", line 30, in <module>
all_m = importlib.import_module('vtkmodules.all')
File "C:\Program Files\Python38\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "C:\Program Files\Python38\lib\site-packages\vtkmodules\__init__.py", line 71, in <module>
import _vtkmodules_static
ModuleNotFoundError: No module named '_vtkmodules_static'
Update: 26-AUG-2021 (Continued...)
NOTE: This is NOT an answer.
Pretty much doing the exact same thing, but in vtk.py instead, plus including all other system dependencies, using os.add_dll_directory() does not work:
"""This is the vtk module."""
import sys
if sys.version_info < (3, 5):
# imp is deprecated in 3.4
import imp
import importlib
# import vtkmodules package.
vtkmodules_m = importlib.import_module("vtkmodules")
# import vtkmodules.all
all_m = importlib.import_module("vtkmodules.all")
# create a clone of the `vtkmodules.all` module.
vtk_m = imp.new_module(__name__)
for key in dir(all_m):
if not hasattr(vtk_m, key):
setattr(vtk_m, key, getattr(all_m, key))
# make the clone of `vtkmodules.all` act as a package at the same location
# as vtkmodules. This ensures that importing modules from within the vtkmodules package
# continues to work.
vtk_m.__path__ = vtkmodules_m.__path__
# replace old `vtk` module with this new package.
sys.modules[__name__] = vtk_m
else:
if sys.version_info >= (3, 8):
import os
if os.name == "nt":
WIN_DLLS = set([r"C:\WINDOWS\System32\downlevel"])
PY_ROOT = sys.exec_prefix
PY_DLLS = set(
[
os.path.join(PY_ROOT, "DLLs"),
]
)
try:
VTK_DIR = os.environ["VTK_DIR"]
except KeyError:
VTK_DIR = None
VTK_DLLS = set([])
if VTK_DIR is not None:
# Assumes default folder "bin" used in CMake configuration
for root, dirs, files in os.walk(os.path.join(VTK_DIR, "bin")):
for file_ in files:
if file_.lower().endswith(".dll") or file_.lower().endswith(
".pyd"
):
if root not in VTK_DLLS:
VTK_DLLS.add(root)
break
try:
VULKAN_PATH = os.environ["VK_SDK_PATH"]
except KeyError:
VULKAN_PATH = None
VULKAN_DLLS = set([])
if VULKAN_PATH is not None:
for root, dirs, files in os.walk(VULKAN_PATH):
for file_ in files:
if file_.lower().endswith(".dll"):
if root not in VULKAN_DLLS:
VULKAN_DLLS.add(root)
break
try:
CUDA_PATH = os.environ["CUDA_PATH"]
except KeyError:
CUDA_PATH = None
CUDA_DLLS = set([])
if CUDA_PATH is not None:
for root, dirs, files in os.walk(CUDA_PATH):
for file_ in files:
if file_.lower().endswith(".dll"):
if root not in CUDA_DLLS:
CUDA_DLLS.add(root)
break
try:
ONEAPI_ROOT = os.environ["ONEAPI_ROOT"]
except KeyError:
ONEAPI_ROOT = None
ONEAPI_DLLS = set([])
if ONEAPI_ROOT is not None:
for root, dirs, files in os.walk(CUDA_PATH):
for file_ in files:
if file_.lower().endswith(".dll"):
if root not in ONEAPI_DLLS:
ONEAPI_DLLS.add(root)
break
try:
ISPC_EXECUTABLE = os.environ["ISPC_EXECUTABLE"]
except KeyError:
ISPC_EXECUTABLE = None
ISPC_DLLS = set([])
if ISPC_EXECUTABLE is not None:
ISPC_DLLS.add(os.path.dirname(ISPC_EXECUTABLE))
# INCLUDE = os.path.join(PY_ROOT, "include")
# LIB = os.path.join(PY_ROOT, "Lib")
# SITEPACKAGES = os.path.join(PY_ROOT, "Lib", "site-packages")
# LIBS = os.path.join(PY_ROOT, "libs")
# SCRIPTS = os.path.join(PY_ROOT, "Scripts")
dll_directories = [
WIN_DLLS,
PY_DLLS,
VTK_DLLS,
VULKAN_DLLS,
CUDA_DLLS,
ONEAPI_DLLS,
ISPC_DLLS,
]
print(f"WIN_DLLS: {WIN_DLLS}")
print(f"PY_DLLS: {PY_DLLS}")
print(f"VTK_DLLS: {VTK_DLLS}")
print(f"VULKAN_DLLS: {VULKAN_DLLS}")
print(f"CUDA_DLLS: {CUDA_DLLS}")
print(f"ONEAPI_DLLS: {ONEAPI_DLLS}")
print(f"ISPC_DLLS: {ISPC_DLLS}")
for dll in dll_directories:
if dll is not None:
for dll_ in dll:
os.add_dll_directory(dll_)
import importlib
# import vtkmodules.all
all_m = importlib.import_module("vtkmodules.all")
# import vtkmodules
vtkmodules_m = importlib.import_module("vtkmodules")
# make vtkmodules.all act as the vtkmodules package to support importing
# other modules from vtkmodules package via `vtk`.
all_m.__path__ = vtkmodules_m.__path__
# replace old `vtk` module with the `all` package.
sys.modules[__name__] = all_m
gives output
❯ py
Python 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import vtk
WIN_DLLS: {'C:\\WINDOWS\\System32\\downlevel'}
PY_DLLS: {'C:\\Program Files\\Python38\\DLLs'}
VTK_DLLS: {'C:\\VTK\\bin\\Lib\\site-packages\\mpi4py', 'C:\\VTK\\bin', 'C:\\VTK\\bin\\Lib\\site-packages\\vtkmodules'}
VULKAN_DLLS: {'C:\\VulkanSDK\\1.2.182.0\\Tools', 'C:\\VulkanSDK\\1.2.182.0\\Tools32\\styles', 'C:\\VulkanSDK\\1.2.182.0\\Third-Party\\Bin32', 'C:\\VulkanSDK\\1.2.182.0\\Tools\\styles', 'C:\\VulkanSDK\\1.2.182.0\\Tools32\\iconengines', 'C:\\VulkanSDK\\1.2.182.0\\Tools\\imageformats', 'C:\\VulkanSDK\\1.2.182.0\\Tools\\platforms', 'C:\\VulkanSDK\\1.2.182.0\\Tools32\\bearer', 'C:\\VulkanSDK\\1.2.182.0\\Bin32', 'C:\\VulkanSDK\\1.2.182.0\\Tools\\bearer', 'C:\\VulkanSDK\\1.2.182.0\\Bin', 'C:\\VulkanSDK\\1.2.182.0\\Third-Party\\Bin', 'C:\\VulkanSDK\\1.2.182.0\\Tools32', 'C:\\VulkanSDK\\1.2.182.0\\Tools32\\imageformats', 'C:\\VulkanSDK\\1.2.182.0\\Tools\\iconengines', 'C:\\VulkanSDK\\1.2.182.0\\Tools32\\platforms'}
CUDA_DLLS: {'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.4\\extras\\visual_studio_integration\\MSBuildExtensions', 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.4\\extras\\demo_suite', 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.4\\extras\\CUPTI\\lib64', 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.4\\nvvm\\bin', 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.4\\compute-sanitizer', 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.4\\libnvvp\\plugins\\org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.200.v20140603-1326', 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.4\\bin'}
ONEAPI_DLLS: {'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.4\\extras\\visual_studio_integration\\MSBuildExtensions', 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.4\\extras\\demo_suite', 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.4\\extras\\CUPTI\\lib64', 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.4\\nvvm\\bin', 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.4\\compute-sanitizer', 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.4\\libnvvp\\plugins\\org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.200.v20140603-1326', 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.4\\bin'}
ISPC_DLLS: {'C:\\Program Files\\ISPC\\ispc-v1.16.1-windows\\bin'}
Traceback (most recent call last):
File "C:\Program Files\Python38\lib\site-packages\vtkmodules\__init__.py", line 13, in <module>
from . import vtkCommonCore
ImportError: DLL load failed while importing vtkCommonCore: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files\Python38\lib\site-packages\vtk.py", line 143, in <module>
all_m = importlib.import_module("vtkmodules.all")
File "C:\Program Files\Python38\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "C:\Program Files\Python38\lib\site-packages\vtkmodules\__init__.py", line 15, in <module>
import _vtkmodules_static
ModuleNotFoundError: No module named '_vtkmodules_static'
>>>
This is also not an answer - but a short recipe for what worked for me.
I use Python 3.8.8 from python.org - should not make any difference.
I use VTK-9.0.1 - before they attempted to fix the os.add_dll_directory issue.
I use Qt5.15.2
I am not using vcpkg
Make sure the right DLLs are found
In [1]: import sys
In [2]: sys.path.insert(0, "c:/{VTK_BUILD_DIR}/bin/Lib/site-packages/")
Make sure that they are loaded and no errors occur (some are silent)
In [5]: import os
In [6]: os.add_dll_directory("c:/{VTK_BUILD_DIR}/bin/Lib/site-packages/vtkmodules/")
In [7]: os.add_dll_directory("c:/{VTK_BUILD_DIR}/bin/Release/")
In [8]: os.add_dll_directory("c:/QT/Qt5.15/5.15.2/msvc2019_64/bin/")
This is made on top of an empty virtual environment, where only ipython is added.
I was putting my generated CMake files into C:\VTK\build\ and then compiling and installing in C:\VTK, which is why Python couldn't find the DLLs. I have corrected the corresponding step in my post so the install instructions are complete and correct.
See section Steps to Reproduce, step 3
I am trying to install Lapack on my Windows 7 64bit to use SciPy. I have run into many different answers about this topic but I have not seen anywhere anything about this.
I am following the guide from this website: https://icl.cs.utk.edu/lapack-for-windows/lapack/ from the section "Easy Windows Build". As the site states, I need to install CMAKE, Visual Studio and download lapack. I have installed "cmake-3.5.0-rc3-win32-x86", "Visual Studio 13" and minGW with all the packages that were available (just in case).
Finally, when I open the "cmake-gui" (as administrator) to perform the steps it says to build "Lapack" and I specify my source code folder (the lapack decompressed folder "lapack-3.6.0") and my output folder (a folder in Documents) and I click the Configure button I get the following error message:
Error in configuration process, project files may be invalid
I checked the log files of cmake and this is what I get:
CMakeError.log
Compiling the Fortran compiler identification source file "CMakeFortranCompilerId.F" failed.
Compiler: C:/MinGW/bin/gfortran.exe
Build flags:
Id flags:
The output was:
The system cannot find the file specified
Checking whether the Fortran compiler is Compaq using "-what" did not match "Compaq Visual Fortran":
gfortran.exe: error: unrecognized command line option '-what'
gfortran.exe: fatal error: no input files
compilation terminated.
Checking whether the Fortran compiler is NAG using "-V" did not match "NAG Fortran Compiler":
gfortran.exe: error: unrecognized command line option '-V'
gfortran.exe: fatal error: no input files
compilation terminated.
Determining if the Fortran compiler works failed with the following output:
Change Dir: C:/Program Files (x86)/LAPACK/CMakeFiles/CMakeTmp
Run Build Command:"devenv.com" "CMAKE_TRY_COMPILE.sln" "/build" "Debug" "/project" "cmTC_249e8"
Generator: execution of make failed. Make command was: "devenv.com" "CMAKE_TRY_COMPILE.sln" "/build" "Debug" "/project" "cmTC_249e8"
CMakeOutput.log
The system is: Windows - 6.1.7601 - AMD64
Determining if the Fortran compiler is GNU succeeded with the following output:
# 1 "C:/Program Files (x86)/CMake/share/cmake-3.5/Modules/CMakeTestGNU.c"
# 1 "<command-line>"
# 1 "C:/Program Files (x86)/CMake/share/cmake-3.5/Modules/CMakeTestGNU.c"
void THIS_IS_GNU();
void THIS_IS_MINGW();
I have installed MinGW and I checked it is under the directory:
C:/MinGW
since I have also specified native compilers for CMAKE from inside the bin folder
gfortran.exe
gcc.exe
g++.exe
for Fortran, C and C++ respectively.
I checked online about the error in configuration process and from what I found out is that I need to have installed Visual Studio but I already have that.
Finally, I added the directory
C:\MinGW\bin
to the PATH variable.
Any suggestions as to why I cannot get to install LAPACK would be helpful.
Thank you
When I try to import Theano in Python 2.7, I get the following error message:
'C:Program' is not recognized as an internal or external command,
operable program or batch file.
ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: ('nvcc return status', 1, 'for cmd', 'nvcc -shared -O3 -LC:\\Anaconda\\libs -use_fast_math -LC:\\Anaconda\\libs --compiler-bindir C:Program Files (x86)Microsoft Visual Studio 12.0VCbinamd64 -Xlinker /DEBUG -D HAVE_ROUND -m64 -Xcompiler -DCUDA_NDARRAY_CUH=18715462c72ed6afcd7ca5d52813ce90,-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION,/Zi,/MD -IC:\\Anaconda\\lib\\site-packages\\theano\\sandbox\\cuda -IC:\\Anaconda\\lib\\site-packages\\numpy\\core\\include -IC:\\Anaconda\\include -IC:\\Anaconda\\lib\\site-packages\\theano\\gof -o C:\\Users\\JACOB #1\\AppData\\Local\\Theano\\compiledir_Windows-10-10.0.10240-Intel64_Family_6_Model_58_Stepping_9_GenuineIntel-2.7.11-64\\cuda_ndarray\\cuda_ndarray.pyd mod.cu -LC:\\Anaconda\\libs -LC:\\Anaconda -lpython27 -lcublas -lcudart
I noticed that in the --compiler-bindir section, the path has no slashes (C:Program Files (x86)Microsoft Visual Studio 12.0VCbinamd64). I have checked my PATH, THEANO_FLAGS, and .theanorc.txt file and none have this path without the slashes. Is there a way to resolve this?
When I try to paste this command, inserting the slashes, into the command prompt, I get the following results:
nvcc fatal : Cannot find compiler 'cl.exe' in PATH
This makes no sense to me, as the first entry in my PATH is:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64
My .theanorc.txt is as follows:
[global]
device = gpu
floatX = float32
[cuda]
root = -LJ:\NVIDIA GPU Computing Toolkit\CUDA\v7.0
[nvcc]
flags = -LC:\Anaconda\libs
fastmath = True
compiler-bindir = -LC:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64
Any help would be greatly appreciated!
I could solve this problem by setting windows environment variable, instead of directly mentioning the path in compiler-bindir.
You can set windows environment variable by right click on
"start->computer->properties->advanced system settings"
Just check if cl.exe actually exists in the directory:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64
If it doesn't, then install it(along with all the other MS runtime files for C++, which doesn't come by default) by running the VS setup, select Modify option, then expand and select:
Programming Language -> C++
and install.
Hopefully, now it might work. Please be patient as that download might take quite a while...
PS: Iff possible, migrate your Theano workbase to UNIX based OS like (Ubuntu/Kali Linux/Manjaro) + Anaconda for SciPy and NumPy, as it poses much less of a problem than in Windows
I understand that the Theano support for Windows 8.1 is at experimental stage only but I wonder if anyone had any luck with resolving my issues. Depending on my config, I get three distinct types of errors. I assume that the resolution of any of my errors would solve my problem.
I have installed Python using WinPython 32-bit system, using MinGW as described here. The contents of my .theanorc file are as follows:
[global]
openmp=False
device = gpu
[nvcc]
flags=-LC:\TheanoPython\python-2.7.6\libs
compiler_bindir=C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\
[blas]
ldflags =
When I run import theano the error is as follows:
nvcc fatal : nvcc cannot find a supported version of Microsoft Visual Studio.
Only the versions 2010, 2012, and 2013 are supported
['nvcc', '-shared', '-g', '-O3', '--compiler-bindir', 'C:\\Program Files (x86)\\
Microsoft Visual Studio 10.0\\VC\\bin# flags=-m32 # we have this hard coded for
now', '-Xlinker', '/DEBUG', '-m32', '-Xcompiler', '-DCUDA_NDARRAY_CUH=d67f7c8a21
306c67152a70a88a837011,/Zi,/MD', '-IC:\\TheanoPython\\python-2.7.6\\lib\\site-pa
ckages\\theano\\sandbox\\cuda', '-IC:\\TheanoPython\\python-2.7.6\\lib\\site-pac
kages\\numpy\\core\\include', '-IC:\\TheanoPython\\python-2.7.6\\include', '-o',
'C:\\Users\\Matej\\AppData\\Local\\Theano\\compiledir_Windows-8-6.2.9200-Intel6
4_Family_6_Model_60_Stepping_3_GenuineIntel-2.7.6-32\\cuda_ndarray\\cuda_ndarray
.pyd', 'mod.cu', '-LC:\\TheanoPython\\python-2.7.6\\libs', '-LNone\\lib', '-LNon
e\\lib64', '-LC:\\TheanoPython\\python-2.7.6', '-lpython27', '-lcublas', '-lcuda
rt']
ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: ('nvcc return st
atus', 1, 'for cmd', 'nvcc -shared -g -O3 --compiler-bindir C:\\Program Files (x
86)\\Microsoft Visual Studio 10.0\\VC\\bin# flags=-m32 # we have this hard coded
for now -Xlinker /DEBUG -m32 -Xcompiler -DCUDA_NDARRAY_CUH=d67f7c8a21306c67152a
70a88a837011,/Zi,/MD -IC:\\TheanoPython\\python-2.7.6\\lib\\site-packages\\thean
o\\sandbox\\cuda -IC:\\TheanoPython\\python-2.7.6\\lib\\site-packages\\numpy\\co
re\\include -IC:\\TheanoPython\\python-2.7.6\\include -o C:\\Users\\Matej\\AppDa
ta\\Local\\Theano\\compiledir_Windows-8-6.2.9200-Intel64_Family_6_Model_60_Stepp
ing_3_GenuineIntel-2.7.6-32\\cuda_ndarray\\cuda_ndarray.pyd mod.cu -LC:\\TheanoP
ython\\python-2.7.6\\libs -LNone\\lib -LNone\\lib64 -LC:\\TheanoPython\\python-2
.7.6 -lpython27 -lcublas -lcudart')
WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu is not availabl
e
I have also tested it using Visual Studio 12.0 which is installed on my system with the following error:
mod.cu
nvlink fatal : Could not open input file 'C:/Users/Matej/AppData/Local/Temp/tm
pxft_00001b70_00000000-28_mod.obj'
['nvcc', '-shared', '-g', '-O3', '--compiler-bindir', 'C:\\Program Files (x86)\\
Microsoft Visual Studio 12.0\\VC\\bin\\', '-Xlinker', '/DEBUG', '-m32', '-Xcompi
ler', '-LC:\\TheanoPython\\python-2.7.6\\libs,-DCUDA_NDARRAY_CUH=d67f7c8a21306c6
7152a70a88a837011,/Zi,/MD', '-IC:\\TheanoPython\\python-2.7.6\\lib\\site-package
s\\theano\\sandbox\\cuda', '-IC:\\TheanoPython\\python-2.7.6\\lib\\site-packages
\\numpy\\core\\include', '-IC:\\TheanoPython\\python-2.7.6\\include', '-o', 'C:\
\Users\\Matej\\AppData\\Local\\Theano\\compiledir_Windows-8-6.2.9200-Intel64_Fam
ily_6_Model_60_Stepping_3_GenuineIntel-2.7.6-32\\cuda_ndarray\\cuda_ndarray.pyd'
, 'mod.cu', '-LC:\\TheanoPython\\python-2.7.6\\libs', '-LNone\\lib', '-LNone\\li
b64', '-LC:\\TheanoPython\\python-2.7.6', '-lpython27', '-lcublas', '-lcudart']
ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: ('nvcc return st
atus', 1, 'for cmd', 'nvcc -shared -g -O3 --compiler-bindir C:\\Program Files (x
86)\\Microsoft Visual Studio 12.0\\VC\\bin\\ -Xlinker /DEBUG -m32 -Xcompiler -LC
:\\TheanoPython\\python-2.7.6\\libs,-DCUDA_NDARRAY_CUH=d67f7c8a21306c67152a70a88
a837011,/Zi,/MD -IC:\\TheanoPython\\python-2.7.6\\lib\\site-packages\\theano\\sa
ndbox\\cuda -IC:\\TheanoPython\\python-2.7.6\\lib\\site-packages\\numpy\\core\\i
nclude -IC:\\TheanoPython\\python-2.7.6\\include -o C:\\Users\\Matej\\AppData\\L
ocal\\Theano\\compiledir_Windows-8-6.2.9200-Intel64_Family_6_Model_60_Stepping_3
_GenuineIntel-2.7.6-32\\cuda_ndarray\\cuda_ndarray.pyd mod.cu -LC:\\TheanoPython
\\python-2.7.6\\libs -LNone\\lib -LNone\\lib64 -LC:\\TheanoPython\\python-2.7.6
-lpython27 -lcublas -lcudart')
WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu is not availabl
e
In the latter error, several pop-up windows ask me how would I like to open (.res) file before error is thrown.
cl.exe is present in both folders (i.e. VS 2010 and VS 2013).
Finally, if I set VS 2013 in the environment path and set .theanorc contents as follows:
[global]
base_compiledir=C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin
openmp=False
floatX = float32
device = gpu
[nvcc]
flags=-LC:\TheanoPython\python-2.7.6\libs
compiler_bindir=C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\
[blas]
ldflags =
I get the following error:
c:\theanopython\python-2.7.6\include\pymath.h(22): warning: dllexport/dllimport conflict with "round"
c:\program files\nvidia gpu computing toolkit\cuda\v6.5\include\math_functions.h(2455): here; dllimport/dllexport dropped
mod.cu(954): warning: statement is unreachable
mod.cu(1114): error: namespace "std" has no member "min"
mod.cu(1145): error: namespace "std" has no member "min"
mod.cu(1173): error: namespace "std" has no member "min"
mod.cu(1174): error: namespace "std" has no member "min"
mod.cu(1317): error: namespace "std" has no member "min"
mod.cu(1318): error: namespace "std" has no member "min"
mod.cu(1442): error: namespace "std" has no member "min"
mod.cu(1443): error: namespace "std" has no member "min"
mod.cu(1742): error: namespace "std" has no member "min"
mod.cu(1777): error: namespace "std" has no member "min"
mod.cu(1781): error: namespace "std" has no member "min"
mod.cu(1814): error: namespace "std" has no member "min"
mod.cu(1821): error: namespace "std" has no member "min"
mod.cu(1853): error: namespace "std" has no member "min"
mod.cu(1861): error: namespace "std" has no member "min"
mod.cu(1898): error: namespace "std" has no member "min"
mod.cu(1905): error: namespace "std" has no member "min"
mod.cu(1946): error: namespace "std" has no member "min"
mod.cu(1960): error: namespace "std" has no member "min"
mod.cu(3750): error: namespace "std" has no member "min"
mod.cu(3752): error: namespace "std" has no member "min"
mod.cu(3784): error: namespace "std" has no member "min"
mod.cu(3786): error: namespace "std" has no member "min"
mod.cu(3789): error: namespace "std" has no member "min"
mod.cu(3791): error: namespace "std" has no member "min"
mod.cu(3794): error: namespace "std" has no member "min"
mod.cu(3795): error: namespace "std" has no member "min"
mod.cu(3836): error: namespace "std" has no member "min"
mod.cu(3838): error: namespace "std" has no member "min"
mod.cu(4602): error: namespace "std" has no member "min"
mod.cu(4604): error: namespace "std" has no member "min"
31 errors detected in the compilation of "C:/Users/Matej/AppData/Local/Temp/tmpxft_00001d84_00000000-10_mod.cpp1.ii".
ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: ('nvcc return status', 2, 'for cmd', 'nvcc -shared -g -O3 -Xlinker /DEBUG -m32 -Xcompiler -DCUDA_NDARRAY_CUH=d67f7c8a21306c67152a70a88a837011,/Zi,/MD -IC:\\TheanoPython\\python-2.7.6\\lib\\site-packages\\theano\\sandbox\\cuda -IC:\\TheanoPython\\python-2.7.6\\lib\\site-packages\\numpy\\core\\include -IC:\\TheanoPython\\python-2.7.6\\include -o C:\\Users\\Matej\\AppData\\Local\\Theano\\compiledir_Windows-8-6.2.9200-Intel64_Family_6_Model_60_Stepping_3_GenuineIntel-2.7.6-32\\cuda_ndarray\\cuda_ndarray.pyd mod.cu -LC:\\TheanoPython\\python-2.7.6\\libs -LNone\\lib -LNone\\lib64 -LC:\\TheanoPython\\python-2.7.6 -lpython27 -lcublas -lcudart')
ERROR:theano.sandbox.cuda:Failed to compile cuda_ndarray.cu: ('nvcc return status', 2, 'for cmd', 'nvcc -shared -g -O3 -Xlinker /DEBUG -m32 -Xcompiler -DCUDA_NDARRAY_CUH=d67f7c8a21306c67152a70a88a837011,/Zi,/MD -IC:\\TheanoPython\\python-2.7.6\\lib\\site-packages\\theano\\sandbox\\cuda -IC:\\TheanoPython\\python-2.7.6\\lib\\site-packages\\numpy\\core\\include -IC:\\TheanoPython\\python-2.7.6\\include -o C:\\Users\\Matej\\AppData\\Local\\Theano\\compiledir_Windows-8-6.2.9200-Intel64_Family_6_Model_60_Stepping_3_GenuineIntel-2.7.6-32\\cuda_ndarray\\cuda_ndarray.pyd mod.cu -LC:\\TheanoPython\\python-2.7.6\\libs -LNone\\lib -LNone\\lib64 -LC:\\TheanoPython\\python-2.7.6 -lpython27 -lcublas -lcudart')
mod.cu
['nvcc', '-shared', '-g', '-O3', '-Xlinker', '/DEBUG', '-m32', '-Xcompiler', '-DCUDA_NDARRAY_CUH=d67f7c8a21306c67152a70a88a837011,/Zi,/MD', '-IC:\\TheanoPython\\python-2.7.6\\lib\\site-packages\\theano\\sandbox\\cuda', '-IC:\\TheanoPython\\python-2.7.6\\lib\\site-packages\\numpy\\core\\include', '-IC:\\TheanoPython\\python-2.7.6\\include', '-o', 'C:\\Users\\Matej\\AppData\\Local\\Theano\\compiledir_Windows-8-6.2.9200-Intel64_Family_6_Model_60_Stepping_3_GenuineIntel-2.7.6-32\\cuda_ndarray\\cuda_ndarray.pyd', 'mod.cu', '-LC:\\TheanoPython\\python-2.7.6\\libs', '-LNone\\lib', '-LNone\\lib64', '-LC:\\TheanoPython\\python-2.7.6', '-lpython27', '-lcublas', '-lcudart']
If I run import theano without the GPU option on, it runs without a problem. Also CUDA samples run without a problem.
Theano is a great tool for machine learning applications, yet I found that its installation on Windows is not trivial especially for beginners (like myself) in programming. In my case, I see 5-6x speedups of my scripts when run on a GPU so it was definitely worth the hassle.
I wrote this guide based on my installation procedure and is meant to be verbose and hopefully complete even for people with no prior understanding of building programs under Windows environment. Most of this guide is based on these instructions but I had to change some of the steps in order for it to work on my system. If there is anything that I do that may not be optimal or that doesn't work on your machine, please, let me know and I will try to modify this guide accordingly.
These are the steps (in order) I followed when installing Theano with GPU enabled on my Windows 8.1 machine:
CUDA Installation
CUDA can be downloaded from here. In my case, I chose 64-bit Notebook version for my NVIDIA Optimus laptop with Geforce 750m.
Verify that your installation was successful by launching deviceQuery from command line. In my case this was located in the following folder: C:\ProgramData\NVIDIA Corporation\CUDA Samples\v6.5\bin\win64\Release . If successful, you should see PASS at the end of the test.
Visual Studio 2010 Installation
I installed this via dreamspark. If you are a student you are entitled for a free version. If not, you can still install the Express version which should work just as well. After install is complete you should be able to call Visual Studio Command Prompt 2010 from the start menu.
Python Installation
At the time of writing, Theano on GPU only allows working with 32-bit floats and is primarily built for 2.7 version of Python. Theano requires most of the basic scientific Python libraries such as scipy and numpy. I found that the easiest way to install these was via WinPython. It installs all the dependencies in a self-contained folder which allows easy reinstall if something goes wrong in the installation process and you get some useful IDE tools such as ipython notebook and Spyder installed for free as well. For ease of use you might want to add the path to your python.exe and path to your Scripts folder in the environment variables.
Git installation
Found here.
MinGW Installation
Setup file is here. I checked all the base installation files during the installation process. This is required if you run into g++ error described below.
Cygwin installation
You can find it here. I basically used this utility only to extract PyCUDA tar file which is already provided in the base install (so the install should be straightforward).
Python distutils fix
Open msvc9compiler.py located in your /lib/distutils/ directory of your Python installation. Line 641 in my case reads: ld_args.append ('/IMPLIB:' + implib_file). Add the following after this line (same indentation):
ld_args.append('/MANIFEST')
PyCUDA installation
Source for PyCUDA is here.
Steps:
Open cygwin and navigate to the PyCUDA folder (i.e. /cygdrive/c/etc/etc) and execute tar -xzf pycuda-2012.1.tar.gz.
Open Visual Studio Command Prompt 2010 and navigate to the directory where tarball was extracted and execute python configure.py
Open the ./siteconf.py and change the values so that it reads (for CUDA 6.5 for instance):
BOOST_INC_DIR = []
BOOST_LIB_DIR = []
BOOST_COMPILER = 'gcc43'
USE_SHIPPED_BOOST = True
BOOST_PYTHON_LIBNAME = ['boost_python']
BOOST_THREAD_LIBNAME = ['boost_thread']
CUDA_TRACE = False
CUDA_ROOT = 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v6.5'
CUDA_ENABLE_GL = False
CUDA_ENABLE_CURAND = True
CUDADRV_LIB_DIR = ['${CUDA_ROOT}/lib/Win32']
CUDADRV_LIBNAME = ['cuda']
CUDART_LIB_DIR = ['${CUDA_ROOT}/lib/Win32']
CUDART_LIBNAME = ['cudart']
CURAND_LIB_DIR = ['${CUDA_ROOT}/lib/Win32']
CURAND_LIBNAME = ['curand']
CXXFLAGS = ['/EHsc']
LDFLAGS = ['/FORCE']
Execute the following commands at the VS2010 command prompt:
set VS90COMNTOOLS=%VS100COMNTOOLS%
python setup.py build
python setup.py install
Create this python file and verify that you get a result:
# from: http://documen.tician.de/pycuda/tutorial.html
import pycuda.gpuarray as gpuarray
import pycuda.driver as cuda
import pycuda.autoinit
import numpy
a_gpu = gpuarray.to_gpu(numpy.random.randn(4,4).astype(numpy.float32))
a_doubled = (2*a_gpu).get()
print a_doubled
print a_gpu
Install Theano
Open git bash shell and choose a folder in which you want to place Theano installation files and execute:
git clone git://github.com/Theano/Theano.git
python setup.py install
Try opening python in VS2010 command prompt and run import theano
If you get a g++ related error, open MinGW msys.bat in my case installed here: C:\MinGW\msys\1.0 and try importing theano in MinGW shell. Then retry importing theano from VS2010 Command Prompt and it should be working now.
Create a file in WordPad (NOT Notepad!), name it .theanorc.txt and put it in C:\Users\Your_Name\ or wherever your users folder is located:
#!sh
[global]
device = gpu
floatX = float32
[nvcc]
compiler_bindir=C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin
# flags=-m32 # we have this hard coded for now
[blas]
ldflags =
# ldflags = -lopenblas # placeholder for openblas support
Create a test python script and run it:
from theano import function, config, shared, sandbox
import theano.tensor as T
import numpy
import time
vlen = 10 * 30 * 768 # 10 x #cores x # threads per core
iters = 1000
rng = numpy.random.RandomState(22)
x = shared(numpy.asarray(rng.rand(vlen), config.floatX))
f = function([], T.exp(x))
print f.maker.fgraph.toposort()
t0 = time.time()
for i in xrange(iters):
r = f()
t1 = time.time()
print 'Looping %d times took' % iters, t1 - t0, 'seconds'
print 'Result is', r
if numpy.any([isinstance(x.op, T.Elemwise) for x in f.maker.fgraph.toposort()]):
print 'Used the cpu'
else:
print 'Used the gpu'
Verify you got Used the gpu at the end and you're done!
Here are my simple steps for installing theano on a
64-bit windows 10 machine. It's tested on the code listed here
(All installation are with default installation path)
install anaconda python 3.x distribution (it already includes numpy,
scipy, matlibplot, etc.)
run 'conda install mingw libpython' in command-line
install theano by downloading it from the official website and do `python setup.py install'
install lastest CUDA toolkit for 64-bit windows 10 (now is 7.5)
install visual studio 2013 (free for windows 10)
create .theanorc.txt file under %USERPROFILE% path and here are
the content in the .theanorc.txt file to run theano with GPU
[global]
floatX = float32
device = gpu
[nvcc]
fastmath = True
compiler_bindir=C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\cl.exe
[cuda]
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v7.5
Here's a guide to installing theano with CUDA on 64-bit Windows.
It seems straightforward, but I have not actually tested it to ensure that it works.
http://pavel.surmenok.com/2014/05/31/installing-theano-with-gpu-on-windows-64-bit/
Following the tutorial by Matt, I ran into issues with nvcc.
I needed to add the path to VS2010 executables in nvcc.profile (you can find it in the cuda bin folder):
"compiler-bindir = C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64"
In case you want to upgrade to MS Visual Studio 2012 and CUDA 7 on Windows 8.1 x64, check out this tutorial here:
http://machinelearning.berlin/?p=383
It should work as long as you stick to it exactly.
All the best
Christian
I could compile the cu files by adding the required dependencies in the nvcc profile located in “C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v7.5\bin\nvcc.profile”
I modified the include and the lib path and it started working.
INCLUDES += “-I$(TOP)/include” $(SPACE) “-IC:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/include” $(SPACE) “-IC:\Program Files (x86)\Microsoft SDKs\Windows\v7.1A\Include” $(SPACE)
LIBRARIES =+ $(SPACE) “/LIBPATH:$(TOP)/lib/$(_WIN_PLATFORM_)” $(SPACE) “/LIBPATH:C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/lib/amd64” $(SPACE) “/LIBPATH:C:\Program Files (x86)\Microsoft SDKs\Windows\v7.1A\Lib\x64” $(SPACE)
I have made a full documentation of the install, hope it helps https://planetanacreon.wordpress.com/2015/10/09/install-theano-on-windows-8-1-with-visual-studio-2013-cuda-7-5/
I used this guide, and it was quite helpful.
What many of Windows Theano guides only mention in passing (or not at all) is that you will need to compile theano from mingw shell, not from your IDE.
I ran mingw-w64.bat, and from there "python" and "import theano". Only after that importing it from pycharm works.
Additionally, official instructions on deeplearning.net are bad because they tell you to use CUDA 5.5, but it won't work with newer video cards.
The comments are also quite helpful. If it complains about missing crtdefs.h or basetsd.h, do what Sunando's answer says. If AFTER THAT it still complains that identifier "Iunknown" is undefined in objbase.h, stick the following in
C:\Program Files (x86)\Microsoft SDKs\Windows\v7.1A\Include\objbase.h file, on line 236:
#include <wtypes.h>
#include <unknwn.h>
I had to do this last part to make it work with bleeding edge install (required for parts of Keras).
I also wrote a list of things that worked for me, here:
http://acoupleofrobots.com/everything/?p=2238
This is for 64 bit version.
I have been trying to install Cython for Python 2.7 on my Window 7 system. In particular, I prefer everything in 64 bits. (In case you wonder, I need Cython because Cython is one of the components I need for another package for some specialized numerical analysis. And x64 is potentially an advantage for storage of large data sets.)
So I downloaded the x64 Python 2.7 from the official website. Got Cython from Christoph Gohlke. The amd64 version for Python 2.7 of course. Before I installed, I added Python 2.7 into the registry with the .reg file found in Joe DF's answer here. Afterwards, I installed the Visual C++ compiler from here (I don't think it works) and here (but this one should). At this point, I have in my C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin directory all of vcvars32.bat, vcvars64.bat, vcvars86_amd64.bat, vcvarsx86_ia64.bat. I ran vcvars64.bat and there was no error message.
Subsequently, I followed this page to create a "hello world" test file.
Then, I got this error.
...\Tests>python setup.py build_ext --inplace
running build_ext
building 'hello' extension
error: Unable to find vcvarsall.bat
So I thought to myself: Well, Cython can't find that file because it does not exist. (I searched my whole harddrive for it.) So I copied vcvars64.bat from ...\VC\bin to \VC and changed the name to vcvarsall.bat. Now vcvarsall error is gone. And ... naturally, a new error emerges.
...\Tests>python setup.py build_ext --inplace
running build_ext
building 'hello' extension
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c /nolog
o /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python27\include -IC:\Python27\PC /Tchello.c /F
obuild\temp.win-amd64-2.7\Release\hello.obj
hello.c
C:\Python27\include\pyconfig.h(227) : fatal error C1083: Cannot open include fil
e: 'basetsd.h': No such file or directory
error: command '"C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64
\cl.exe"' failed with exit status 2
Now I have no idea how to proceed. What should I do? Your help is much appreciated.
(For clarity, I did try setting the compiler to Mingw32. But there are other errors. I am focusing on getting VC to work for now.)
EDIT: I tried using the SDK build environment CMD following the instructions on this page. I received the same error.
In case anyone is currently (2017) facing same error with visual C++ 2015 tools, launch setup again and also select windows 8.1 / 10 SDK depending upon your OS. This will fix basestd.h error.
If it is still not working, try launching build tools from: C:\Program Files (x86)\Microsoft Visual C++ Build Tools.
Another alternative would be, just install anaconda 2 or 3 (64 bit if also you don't want memory errors). It contains all the important packages prebuilt: sklearn, matplotlib, scipy, numpy, pandas and even web development frameworks such as flask.
I encountered this problem while trying to install pandas in 'develop' mode. I'm up & running now. My environment:
Windows XP Pro x64 SP2
WinPython 64bit 2.7.5.3 (Python 2.7.5.amd64) <-- "registered" as system's python distro using WinPython Control Panel application
Problems encountered when running python setup.py develop:
Unable to find vcvarsall.bat
fatal error C1083: Cannot open include file: 'basetsd.h': No such file or directory ... exit code 2
Solution:
Download and install Microsoft Visual C++ 2008 Express.
Download and install Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1. You will need only:
Developer Tools > Windows Headers and Libraries <-- this gives you basetsd.h
Developer Tools > Visual C++ Compilers <-- this gives you the 64-bit compilers
Copy C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars64.bat
to C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\vcvarsamd64.bat. Observe the change in file name.
Add C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin to PATH. This may not be required (I did so before re-installing the SDK w/ Headers & Libs; therefore, the necessity is unknown).
At this point, running python setup.py develop completed successfully and python -c "import pandas; print pandas.__version__" resulted in the git tag I expected.
[EDIT] Further reading:
error: Unable to find vcvarsall.bat
answer if you just want binaries to install
answer if you want to compile on x64 machine
answer about the file location/name change
distutils issue 7511
For those with Windows 10, download the SDK from here to fix the header file basestd.h.
I downloaded the .exe, installed and worked great. SDK was ~2.3GB.
Note: For those like me with 64-bit systems who got an additional error:
`LINK : fatal error LNK1158: cannot run 'rc.exe'
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\link.exe' failed with exit status 1158
There is a great answer here which solves the problem by placing the rc.exe file in the correct folder for your system.
Add these to the visual studio installation directories to your Environment variables
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin
C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools
hope it solves the issue.
For suggestion, you can try it for build cython 64 bit with setup.py
I'm using Anaconda for python distribution, Microsoft's Windopws SDK 7, and MSVC 12.0.
Firstly, I make library for my cuda function,
Second, I run python setup.py build_ext -i for make temp folder (you wil get error),
then compile Makefile below,
Mostly, I get this from setup.py when it is compile 32 bit app, then change it for compile 64 bit.
Makefile
CL_EXE = "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64\cl.exe"
LINK_EXE = "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64\link.exe"
CL_FLAGS = /c /nologo /Ox /MD /W3 /GS- /DNDEBUG /O2 /fp:fast
CL_INC = -I. -I"C:\Program Files\Microsoft SDKs\Windows\v7.1\Include" \
-I"C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include" -IC:\Users\name\Anaconda2\envs\py27\Lib\site-packages\numpy\core\include -IC:\Users\name\Anaconda2\envs\py27\include -IC:\Users\name\Anaconda2\envs\py27\PC
LINK_FLAGS = /DLL /nologo /INCREMENTAL:NO /MACHINE:X64
LINK_LIBPATH = "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v7.5\lib\x64" \
/LIBPATH:C:\Users\name\Anaconda2\envs\py27\libs \
/LIBPATH:C:\Users\name\Anaconda2\envs\py27\PCbuild\amd64 \
/LIBPATH:C:\Users\name\Anaconda2\envs\py27\PC\VS9.0\amd64 \
"/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\lib\amd64" \
"/LIBPATH:C:\Program Files\Microsoft SDKs\Windows\v7.1\Lib\x64"
LINK_LIB = cuda_multiply4D.lib cudart.lib MSVCRT.lib kernel32.lib
LINK_EXPORT = /EXPORT:initmy_cuda
LINK_OBJ = build\temp.win-amd64-2.7\Release\my_cuda.obj
LINK_OUT = "/OUT:E:\my_cuda.pyd"
LINK_OTHER_CONF = /IMPLIB:build\temp.win-amd64-2.7\Release\my_cuda.lib /MANIFESTFILE:build\temp.win-amd64-2.7\Release\my_cuda.pyd.manifest
all:
$(CL_EXE) $(CL_FLAGS) $(CL_INC) /Tcmy_cuda.c /Fobuild\temp.win-amd64-2.7\Release\my_cuda.obj
$(LINK_EXE) $(LINK_FLAGS) $(LINK_LIBPATH) $(LINK_LIB) $(LINK_EXPORT) $(LINK_OBJ) $(LINK_OUT) $(LINK_OTHER_CONF)
Note:
MSVCRT.lib kernel32.lib are optional,
/EXPORT:initmy_cuda is mandatory. Its format init%yourfilename_WO_Ext%
"/OUT:E:\my_cuda.pyd" for determine your output location, its filename, and its extension.
I could solve this problem by downloading and installing the latest Windows 10 SDK from this link:
https://software-download.microsoft.com/download/pr/19041.685.201201-2105.vb_release_svc_prod1_WindowsSDK.iso
but then another problem appeared with (cpython):
error C2039: 'exc_type': is not a member of '_ts'
which I could resolve by declaring additional pointers in
(...\python38\include\cpython\pystate.h)
I found these at line 77:
/* The exception currently being raised */
PyObject *curexc_type;
PyObject *curexc_value;
PyObject *curexc_traceback;
and I add the following just after them (without the first three letters "cur"):
/* The exception currently being raised */
PyObject *exc_type;
PyObject *exc_value;
PyObject *exc_traceback;