Visual Studio 2017 now ships with clang, and I was wondering if it is possible to set clang to be the default compiler used when trying to install a python package with Cython dependencies via pip install.
As far as I can tell the issue seems to be that pip doesn't know where to look for clang; it defaults to the path containing the MSVC compiler to compile the .c files specified in the package setup.py. Is there any way I can either 1) set the path to the compiler to be used, or 2) specify that clang should be used explicitly?
I'm working on Windows 10, using the Anaconda distribution and Python 3.6. Any help would be much appreciated!
Do not do it.
It can be done on the command line by setting compiler:
python setup.py build_ext -c <compiler>
python setup.py build_ext --help-compiler to see available options.
However, the same compiler used to build the python distribution must be used for all extensions as the extension modules will be loaded by the interpreter.
If they are built by a different compiler, they cannot be safely loaded by the python interpreter.
In the case of Anaconda, its python distribution is built with MSVC, so the same compiler used by the anaconda environment should be used for extensions. In that case, activating the anaconda environment should suffice, though compiler can be set explicitly as above.
See Windows Compilers python wiki entry.
Related
I have the "Enthought Canopy" v1.5 distribution of Python 2.7 installed with cython 0.22 (installed via the Canopy package manager) on a Windows 64-bit computer. Trying to compile a cython file leads to the following error:
building 'my_cython_extension' extension
error: Could not find Visual Studio 2008 in your path.
If you do not have Visual Studio 2008 installed, you can use
the MinGW compiler instead. To install mingw, do:
enpkg mingw
To use the MinGW compiler to build an extension module, use
the '-c' flag, e.g.:
python setup.py build_ext -c mingw64
Note that building Python extensions with MinGW is not officially
supported, although it is known to work in many cases.
Visual Studio 2008 is no longer available, and Visual Studio 2008 Express doesn't include a 64-bit compiler. I've read that it's necessary to compile cython with the same compiler that Python was compiled with, which rules out using a newer version of Visual Studio. Additionally, the message clearly states that MinGW is not officially supported.
I've tried installing "Microsoft Visual C++ Compiler for Python 2.7" as per the Cython Wiki, but that has no effect on making the above error message disappear. I've read that updating setuptools may fix this error (as it can find the compiler in different locations than the old version expects to find it), but that didn't work for me, and additionally Enthought have stated on their website that:
Setuptools 0.6 is already supported, and higher versions are not yet compatible with Enthought installers.
... so updating setuptools may break something in Canopy.
Finally, I've tried installing "MS Windows SDK for Windows 7 and .NET Framework 3.5 SP1" as written later on in the cython Wiki, but this also had no effect on making the message disappear, and additionally it's a huge unwieldy package which I'd rather not install just for the compiler if possible.
So, is there a way to get "Microsoft Visual C++ Compiler for Python 2.7" working with cython and Enthought Canopy? If not, what is the official way to install and configure the cython compiler?
It took me the greater part of a day to figure out how to get this working, and I sincerely hope there's a better way, which I would love to hear, but this is what got the job done for me:
1) Install Microsoft Visual C++ 2008 Express edition -- you should find a direct download link by Googling for it.
2) Download the "Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1 (ISO)" image (be sure to choose the 64-bit version GRMSDKX_EN_DVD.iso) from here
3) Extract the iso with 7-zip, or mount / burn it with your preferred software
4) Run setup.exe and install. You can probably safely deselect everything except for "Visual C++ Compilers" and "Windows Headers and Libraries"
5) Copy the file C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars64.bat to the folder C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\ and rename it to vcvarsamd64.bat
6) Not sure if this is necessary, but you may need to run the following commands in the Visual Studio command prompt, as written in the cython wiki
set DISTUTILS_USE_SDK=1
setenv /x64 /release
Note 1: Step 1 appears to be necessary because the 64-bit Windows SDK for .net 3.5 doesn't seem to come with a vcvarsall.bat file.
Note 2: Step 5 is necessary because the vcvarsall.bat that comes with Visual C++ 2008 Express points to the wrong location for the 64-bit compiler.
(Enthought Support Here)
For the most up to date info, see our Knowledge Base that summarizes issues and provides useful links to some of the needed components for building extensions on Windows (as well other platforms)
Regarding the "Microsoft Visual C++ Compiler for Python 2.7", we are working on updating Canopy's Python version to 2.7.9 in our next release, which will include needed changes to support Microsoft's new compiler. We expect Canopy 1.5.3 to be released sometime next week.
For using the SDK now, the steps we recommend are:
Download the Windows SDK for Python 2.7 as either a web installer or a DVD ISO.
Install the SDK.
If you use the DVD .iso, the SDK is provided as a DVD .iso image. To install it, you'll need software to mount this as a virtual DVD. Windows 8 provides this, but on Windows 7 you will need something like the following: http://www.microsoft.com/en-us/download/details.aspx?id=38780
Once the SDK is installed, you need to start the SDK Command Prompt and set two environment variables before compiling your extensions:
set DISTUTILS_USE_SDK=1
setenv /x64 /release
Assuming you didn't change the default SDK install path, this should look like:
C:\Program Files\Microsoft SDKs\Windows\v7.0>set DISTUTILS_USE_SDK=1
C:\Program Files\Microsoft SDKs\Windows\v7.0>setenv /x64 /release
(Note: you may need to swap /x64 for /x86 if on a 32-bit system.)
From what I understand, using the SDK Command Prompt to set these variables will help avoid all of the extra configuration/surgery surrounding vcvarsall.bat.
I have tried everything given and the tutorials all point in the same direction about using mingw as a compiler in python instead of visual c++.
I do have visual c++ and mingw both. Problem started coming when I wanted to install using pip. It always gave Unable to find vcvarsall.bat
So as per suggestions I created a file distutils.cfg under the following path c:/python27/Lib/distutils/ and added the following two lines:
[build]
compiler=mingw32
However, this file has no effect whatsoever. The same error still exists while using pip. I am trying to install numpy by the way currently although the error came earlier also.
Cost me several hours to figure this out but now works like a charm.
I am on Windows 64bit and using a scientific Python distribution Enthought Canopy.
Ensure you have installed MinGW and the Compilers you need (C, C++, Fortran..)
Add C:\MingW\bin to your PATH in Environment variables
In C:\Users\MyName\AppData\Local\Enthought\Canopy32\App\appdata\canopy-1.6.2.3262.win-x86\Lib\distutils, create with notepad++ or similiar a new file distutils.cfg
Write and save:
[build]
compiler = mingw32
[build_ext]
compiler = mingw32
Difficult Solution (Building using the C/C++ compiler)
Are you on Windows 32bit or 64bit? Are you specifying the build when you're compiling? For example, do the following in your shell or bash:
python setup.py build --compiler==mingw32
python setup.py install
Easier Solution (Use pre-built binaries)
If you don't have to compile the program, you can use the unofficial Windows binaries from here
Do you happen to use virtualenv as well?
Anyway, use Microsoft Visual C++ Compiler for Python 2.7
and a newer setuptools. The one bundled with virtualenv (3.6) does not support this compiler.
pip install "setuptools>=6.0"
Credits: http://blog.ionelmc.ro/2014/12/21/compiling-python-extensions-on-windows/
I am having a very difficult time installing pymc on windows. I do
easy_install pymc
But it gives the error
error: Setup script exited with error: extension 'pymc.flib' has Fortran sources but no Fortran compiler found
However, I installed the gFortran compiler within MinGW. Do I have to change the environment variables or something?
thanks,
First, ensure that you do in fact have the Fortran compiler installed with MinGW. If you open up MinGW Installation Manager (in C:\MinGW\libexec\mingw-get\guimain.exe) you will see a list of packages. If mingw32-gcc-fortran is unchecked, check it and click Installation > Apply Changes.
Secondly, check your PATH to make sure that C:\MinGW\bin is there and that there are not other versions of MinGW in the path (like a 64 bit version).
Lastly, if you are using 32-bit Python then your MinGW should be 32-bit, if 64-bit Python then use 64-bit MinGW.
That sounds like the issue. You should probably add C:\MinGW\bin to your PATH.
I installed distribute and pip using the links I have just given. I also installed the Microsoft Visual C++ 2008 redistributable package. However when I try to use pip.exe I get
error: Unable to find vcvarsall.bat
How can I fix this?
Installing the Microsoft Visual C++ 2008 Redistributable Package is not sufficient to compile packages. You need to install a compiler, not just the support files.
There are three ways to do this:
Install Visual C++.
Use mingw's port of gcc instead of Visual C++.
Use cygwin's port of gcc instead of either, and a cygwin build on Python instead of the native one.
If you want to go with option 1, you need to install Visual C++ itself. The free version should work just as well as the paid version, as long as you're not going to build binary packages to redistribute to others. Unfortunately, I'm not sure where to find the 2008 version anymore. As of May 2013, the download page only has 2010 and 2012.
When you install this, it will create a batch file called vcvarsall.bat (not vcvarshall.bat!), and give you the option of putting that batch file in your PATH. Running that batch file sets up a DOS prompt for building with that version of Visual C++. (This is handy if you have multiple versions of Visual C++, or other compilers, around.) If you skip that option, you will have to do it manually.
This question shows how to use a newer Visual Studio with older Python, and also shows how to point distutils at a vcvarsall.bat that's not on your PATH, and has links to a whole lot of other relevant questions and blog posts.
Many people find option 2 simpler. Install mingw, modify your PATH in the environment to include C:\MinGW\bin (or wherever you choose to install it), and pass -c mingw32 whenever you run a setup.py script.
The problem is that it's not as clearly documented how to tell easy_install and pip to use mingw instead of VC++. To do that, you need to find or create a distutils.cfg file, find or create a [build] section within it, and add compiler=mingw32. Not too hard. This blog post looks like it explains things pretty well, or see this answer.
Option 3 is by far the simplest. Install cygwin, tell it to install the Python and gcc packages, and you're done.
The problem is that you don't have native Windows Python, you have a Unix Python running in a fake Unix environment on top of Windows. If you like Cygwin, you'll love this; otherwise, you won't.
You'll receive such error only for packages (or one of package's dependencies) that has CPython extensions. Pip internally:
downloads the source
runs distutils python setup install
install prepares setup files and tries to build CPython extensions in windows environment
windows environment calls MS Visual Studio vcvarsall.bat script which setups DOS environment variables to enable MS Visual Studio's C compiler in the shell
if vcvarsall.bat is not found - you'll get this message
Usual solution
For python libraries which have CPython extensions that are portable on windows, it is usual to have windows binary package, which are downloadable from pypi or library web site.
In such cases it is more suitable (and painless) to install library by downloading and running windows binary package.
There is a feature request for pip to Add support for installation of binary distutils packages on Windows.
New way to do it - wheels
Thanks to comment from #warren-p: That feature request has been superseeded by Wheels support in PIP.
Official description: A wheel is a ZIP-format archive with a specially formatted filename and the .whl extension.
As I have understood, if there is windows binary package with extension .whl then start by installing wheel first:
# Make sure you have the latest pip that supports wheel
pip install --upgrade pip
pip install wheel
and then install .whl like this:
pip install full-path-or-url-to-your-library.whl
References:
pythonwheels.com
https://pypi.python.org/pypi/wheel
http://wheel.readthedocs.org/en/latest/
You can download Visual Studio 2008 Express SP1 from
http://visual-studio-2008.en.malavida.com/
You can deselect the two add to browser options it offers.
I found these links on microsoft.com that still work to Install Visual C++.
http://download.microsoft.com/download/8/B/5/8B5804AD-4990-40D0-A6AA-CE894CBBB3DC/VS2008ExpressENUX1397868.iso
2008 SP1 here
http://download.microsoft.com/download/E/8/E/E8EEB394-7F42-4963-A2D8-29559B738298/VS2008ExpressWithSP1ENUX1504728.iso
I'm starting out some projects in words processing and I needed NumPy and NLTK.
That was the first time I got to know easy_install and how to compile new module of python into the system.
I have Python 2.7 x64 plus VS 11 and VS 12. Also Cygwin (the latest one I guess).
I could see in the file that compiles using VS that it looks for VS env with the same version as the one that compiled the python code, why?
When I hardcoded 11.0 which is my version, numpy failed to build on several strange errors regarding vcvarsall (it found vcvarsall, probably misused it).
Can't I build python binaries on Windows?
If not, can I cross compile on Linux for Windows? (using the same method as Google for the Android SDK)
Update: This answer is now over 5 years old! Python-2.7 is about to be deprecated, so what you really need is a Microsoft Visual C compiler for Python-3. Take a look at this Python wiki on MS Windows Compilers. MS Build Tools 2015 with VC-14.0 is what you need to build extensions for Python-3.5 and 3.6. You can either install the older MS build tools 2015 which includes VC-14.0 or the newer MS build tools for 2017 - click the link, then scroll down until you find Build Tools for Visual Studio 2017 - which also includes VC-14.0.
Also if your versions of pip and setuptools are up to date, then you can ignore all of that silly old school MSSDK nonsense. Especially if you are using the VC for Python 2.7 or MS build tools 2015. Since setuptools-24, it just knows where to look for these compilers, hopefully.
Update: As Zooba mentions below, free x86 and AMD64 (x86-64) VC90 c-compilers for Python-2.7 are now available from Microsoft.
Update: Patch vcvarsall.bat to use x64 compilers from SDK v7.0 directly with pip in any shell instead of using SDK shell and setting DISTUTILS_USE_SDK and MSSdk environmental variables as in directions below. See Fix vcvarsall.bat to install Python-2.7 x64 extensions with v90 instead of sdk7.
tl;dr: Use Windows SDK v7.0 compilers, open the SDK shell and call
C:\> setenv /release /x64
C:\> set DISTUTILS_USE_SDK=1
C:\> MSSdk=1
to build Python 2.7.x extensions using distutils, pip or easy_install. See Python x64 Package Extensions with pip, MSVC 2008 Express & SDK 7.
Note: You can install Numpy optimized using Intel MKL from Christoph Gohlke. For virtualenv's, try converting the binary distribution windows installer to a wheel file which can be installed with pip. Building NumPy from source is not trivial. It has become a possibility with the recent introduction and availability of OpenBLAS, a fork of GotoBLAS. There are optimized binaries available for Windows x86 and x86-64 as well as source which is relatively simpler to compile than GotoBLAS or ATLAS.
Note: In the past I hadn't recommended compiling NumPy from source because you had to optimize BLAS for your platform which was very time-intensive, and the reference BLAS (FORTRAN) or CBLAS (C/C++) implementations were relatively low-performing. Also NumPy depends on LAPACK which also depends on BLAS an additional hurdle to building NumPy. However OpenBLAS binaries are already compiled with LAPACK, so this obstacle has already been removed. See Carl Kleffner's static MinGW-w64 toolchains to build NumPy with OpenBLAS on Windows - he has resolved some of the ABI incompatibilities and issues linking to the correct Microsoft Visual C Common Runtime library (msvcr90.dll for Python-2.7).
So onward to the main question about installing Python extensions on Windows 7 x64.
The recommended way to compile extensions is to use the same compiler used to compile the Python shared library [1-3]. The official Python 2.7.x for Windows was compiled using Microsoft Visual C++ Compilers version 9.0 [4] (aka MSVC90 or simply VC90) from Microsoft Visual Studio 2008 Professional, which you might be able to get for free from Microsoft DreamSpark. Evidentally the x64 compilers are not installed by default, so make sure they are installed along with the x86 compilers. Note: MS Visual Studio 2008 may no longer be available, but MS Visual Studio 2010 will let you use the MSVC90 toolchain if installed, which can be installed from Windows SDK7.
You can tell what version your Python was built with by looking at the header when you execute the python interpreter from a command line. e.g.: Python 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] on win32 means that it was built with VS 2008 C++. You can build Python yourself from source, but this is also an arduous task. In general it is not recommended to mix compiler runtimes, although in practice you may find that they still work.
If you have your heart set on using GNU GCC, then you would have to use mingw-w64 because MinGW is only for native x86 applications, not AMD64/x86-64. You will need to
create a distutils config file to specify the mingw compiler,
remove the -mno-cygwin bug from distutils.cygwinccompiler.Mingw32ccompiler and
export a library definitions file using either pexports.exe from msys or gendef.exe from msys2 and make a static library libpython27.a using dlltool.exe (msys/msys2) (but the newest releases of Official Python for Windows already have this file in the Python27\lib folder, thanks!).
However in the end you will still need to link against the runtime that python was built with, msvcr90.dll, so you will need Visual C++ 2008 redistributable. Note: Windows GCC from mingw-w64 uses msvcrt.dll which is not the same as msvcr90.dll or later.
If you want to do it for free you can use Microsoft Visual C++ 2008 Express SP1 but you will need to add the Windows SDK 7 for .NET Frameworks 3.5 SP1 because the express version does not have the x64 compilers. The procedure for installing x64 C++ extensions with VS2008 Express & SDK 7 are very similar to the those on the cython site for windows x64 extensions. Note: MS Visual Studio 2008 Express is no longer available or supported.
FYI: You do not necessarily need Visual Studio to build using Microsoft compilers. They are available free with the appropriate SDK package. CL.EXE is the compiler executable, but you will have to set whatever platform options that are normally configured by Autotools or some other tool such as CMAKE. CMAKE plays well with MSVC, but Autotools AFAIK doesn't work with MSVC and would require some POSIX environment and utilities which on Windows are available in MSYS.
For many Python packages that use distutils or setuptools, they can compile extensions using Windows SDK 7 by following the directions that are posted in various places through the reference documentation and wikis:
From the Start Menu select All Programs then Microsoft Windows SDK v7.0 and start CMD Shell to open a command window optimized for Windows SDK.
Step #1 is the equivalent of typing the following in the Run box from the Start Menu or from a Command Prompt (aka C:\Windows\System32\cmd.exe):
%COMSPEC% /E:ON /V:ON /K "%PROGRAMFILES%\Microsoft SDKs\Windows\v7.0\Bin\SetEnv.cmd"
NOTE: The flags /E:ON, which enables command extensions, and /V:ON, which enables delayed variable expansion, are both necessary to for SetEnv.cmd to function, or you will get an message that the x64 compilers are not installed, &c.
Then type setenv /Release /x64 which will set the SDK environment variables specifically for Windows 7 x64 release (vs debug or x86 which are the default).
Type set DISTUTILS_USE_SDK=1 hit return and then type set MSSdk=1 and return to tell distutils.msvccompiler that you are using the SDK, and that the SDK will determine all of the environment variables.
Now use your installation method of choice:
pip install pyyaml ntlk which is the recommended way, see ntlk, but you must have setuptools and pip installed.
python setup.py install for each downloaded, extracted tarball
easy_install pyyaml ntlk which is the old way and the only way to install eggs.
[1] Building C and C++ Extensions on Windows
[2] distutils.msvccompiler — Microsoft Compiler
[3] Python Dev Guide: Getting Started: Windows
[4] What version of visual studio is this python compiled with
Since the other answers, Microsoft has released a compiler package specifically for building extensions for Python 2.7 (and any version of Python that used VS 2008/VC9). Here's how to use it:
Go to http://aka.ms/vcpython27, download and install the package (it's fairly small and does not require administrator rights)
Update setuptools to 6.0 or later (pip install -U setuptools)
Install your package (pip install <whatever>)
The combination of the latest setuptools version and this compiler pack should make it really easy to do this.
there are several ways to do it apparently.
build using mingw
build python 2.7 using VS 2008 express.
i'm not sure regarding which version is good to build 3.2 but it could be VS 2010.
you can compile python x64 from source using your desired VS, but u'll have competability issues with other pre built packages.
This worked best for me:
Install Visual Studio Express 2008
Install Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1 from http://www.microsoft.com/en-us/download/details.aspx?id=3138
Go to C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin and open the Visual Studio 2008 x64 Win64 Command Prompt
In that shell, run pip install <package>
You should not have to install any redistributable packages on client systems when done this way.
For Windows 10, Python 2.7.11, Cygwin 2.5.2 and x86_64-w64-mingw32-gcc 4.9.2
or 5.4.0 (but see note under 1 below), pulling together comments from elsewhere, here's what I had to do for my Python extension to compile, link, and beat a race condition between msvcrt.dll and msvcr90.dll:
Edit distutils/cygwinccompiler, and in class Mingw32CCompiler
Replace the old -mno-cygwin fix with
if self.gcc_version < '4' or is_cygwingcc():
no_cygwin = ' -nostdlib -lgcc'
for gcc 4.9.2 -- note for 5.4.0, no 'if', just
no_cygwin = ''
Get the right compiler in place:
self.set_executables(compiler='x86_64-w64-mingw32-gcc%s -O -Wall' % no_cygwin,
compiler_so='x86_64-w64-mingw32-gcc%s -mdll -O -Wall' % no_cygwin,
compiler_cxx='x86_64-w64-mingw32-g++%s -O -Wall' % no_cygwin,
linker_exe='x86_64-w64-mingw32-gcc%s' % no_cygwin,
linker_so='x86_64-w64-mingw32-%s%s %s %s'
note for x86_64-w64-mingw32-gcc 5.4.0, it seems to work better without msvcr90, so make the call to get_msvcr conditional:
if self.gcc_version < '5':
self.dll_libraries = get_msvcr()
Use a setup.py with the following
libdirs=['C:\\Python27\\libs',...]
macros=[('MS_WIN64','1'),...]
platform="WIN32"
Patch one of my source file headers to include
#ifdef MS_WIN64
#define strdup _strdup
#endif
Finally,
/c/Python27/python [editted setup].py -v build --compiler=mingw32
/c/Python27/python [editted setup].py -v install
Some of the above may not be in quite the optimal place, but it got my extension working...