I'm using Python 3.6.3 on Windows 7 Enterprise and when I tried to pip install the Python package "bitarray", the output indicated the need for Microsoft Visual C++ Build Tools. I downloaded and installed the build tools and installed bitarray with no problems.
Here's where the problem comes in: I now need to distribute bitarray to other employees within the company who don't have Microsoft Visual C++ Build Tools installed, but do have Python installed (and can use pip).
Can I just simply "re-package" the bitarray folder in "C:\Python363\Lib\site-packages\bitarray" (which contains the already compiled .pyd file) and just make it a local package? This way I can use pip with "file:///" to pull down a local copy of the package without the need for the build tools step?
Also, do I need to incorporate the information in the folder "C:\Python363\Lib\site-packages\bitarray-0.8.1.dist-info" to re-package?
Thanks in advance for any help!!!!
Scott
Instead of trying to work around the already installed package, why not building a distribution from source yourself? After all, you've already done the hardest part setting up the C compiler, the rest is just a sequence of commands you have to type. This is what you can do:
Clone bitarray's repository:
$ git clone https://github.com/ilanschnell/bitarray
Navigate into the cloned repository:
$ cd bitarray
Checkout the version tag you want to build (the latest one is 0.8.1):
$ git checkout 0.8.1
Ensure you have wheel installed to be able to build a static wheel:
$ pip install wheel
Build the static wheel:
$ python setup.py bdist_wheel
A new directory dist was created in the current one, check what's inside:
$ ls dist
bitarray-0.8.1-cp36-cp36m-macosx_10_6_intel.whl
(Note: This is what I would enter on my system, list the directory with dir on Windows, also your file should be either bitarray-0.8.1-cp36-cp36m-win_amd64.whl if you are building on a 64 bit system, or bitarray-0.8.1-cp36-cp36m-win32.whl on a 32 bit one).
Now you have built a static wheel that contains the C extensions compiled for Python 3.6 on Windows. It can be installed on Windows without needing to setup the C compiler on the target machine. Just enter
$ pip install bitarray-0.8.1-cp36-cp36m-win_amd64.whl
Note, however, that this wheel file can be installed only on Windows and only with Python 3.6. Should you need to provide a wheel for another setup (like Python 3.5 on Windows 32 bit), you would need to build another wheel file using the correct Python version on a correct target system, but the steps would be just the same.
Building without Git
If you don't have Git installed and you can't/don't want to install it, just download the zipped repository from Github, unzip it, navigate to the extracted directory and perform steps 4-6.
Related
I am trying to install a library in a virtualenv instance with pip. The library version I want (wxPython 3.0.2)
is not available on PyPi; it is only available for download from SourceForge. Thus, I have the source tarball downloaded on my machine and I am trying to install it in such a way that it will play nicely with virtualenv.
(I am on a Windows computer, running Python 2.7.)
I have tried the following:
doing a direct install: pip install wxPython-src-3.0.2.0.tar.bz2
extracting the files from the tarball to wxPython-src-3.0.2.0, then installing from the extracted directory: pip install wxPython-src-3.0.2.0
extracting the files from the tarball, then navigating into the extracted folder to the nested wxPython directory, which holds the setup.py file, and then installing from there: pip install wxPython
The last attempt seems the most promising, but I get the following traceback:
Processing \wxpython-src-3.0.2.0\wxpython
Complete output from command python setup.py egg_info:
Setuptools must be installed to build an egg
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in c:\users\__MY_USERNAME__\appdata\local\temp\pip-req-build-q0pxlt\
This is also strange, because it suggests I don't have setuptools even though I can run pip list and see version 40.6.3 installed.
Any help appreciated.
Why not install a precompiled version? There are a lot of .exe files at SF. You probably need wxPython3.0-win64-3.0.2.0-py27.exe.
Also take a look at Christoph Gohlke's collection.
If you still insist on installing from sources please bear in mind that wxPython 3 is so old it predates pip. Forget about pip.
First, you need to install wxWidgets as wxPython is just a Python wrapper for wxWidgets C++ library. Extract wxPython-src-3.0.2.0.tar.bz2 and follow instructions in wxPython-src-3.0.2.0/docs/msw/install.txt.
After compiling and installing wxWidgets compile wxPython. See wxPython-src-3.0.2.0/wxPython/docs/BUILD.txt.
My eventual solution was the easy way out: installing my package (wxPython) locally as #phd suggested, and opting for local package access via either virtualenv --system-site-packages env or deleting the "no-global-site-packages.txt" file in an existing environment folder.
Not what I expected to do, but it works so no complaints.
From the documentation:
The embedded distribution is a ZIP file containing a minimal Python
environment.
Sounds great! The 64-bit Windows embedded v3.6.5 of Python is only 13MB. As an alternative to compiling, I would like to zip some python scripts together with the minimum needed to run them on a Win10 machine that doesn't have Python installed.
Now, I almost always need to import additional packages to provide functionality. But I can't determine how I should do this if I want to send out a python script together with this embedded version of Python. For example, if my script uses numpy, how can I include that library in this "embed?" I.e., so that on any Win10 machine I can unzip the single deployment file and immediately execute my scripts?
(A regular pip install numpy appears to create a Lib subdirectory that's over 50MB! But for an "embedded" deployment I don't need any support for debugging or whatever else is encompassed in that mass of files.)
There is a way to extend Python embedded installation.
I managed to create Flask-ready package, that I can just unzip on target machine and run code.
The trick is to install EXACT same python version (normal full blown python) as your target embedded small python. Not only version but x86, x64 has to match as well.
Then install modules from pip on normal python, go to NormalPython\Lib\site-packages and copy all new files that appear after installing the package to EmbeddedPython\Lib
finally add Lib and Lib\site-packages to pythonXX._pth inside Embedded python folder.
It's extremely important to fully test your application in case you miss some package.
Also this would not work for packages that also add .exe to Scripts folder.
You could still probably copy the exe's to Script folder and add it to path which could do the trick.
There is a similar trick.
Install pip in the embedded version:
curl -sSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
Edit pythonXX._pth by adding
Lib
Lib\site-packages
Use pip to install other packages
There is a way to extend Python embedded installation. The trick is to install the same python version( I will call it NormalPython) as your target embedded python(I will call it EmbeddedPython). The version as well as architecture has to match exactly.
You then install modules from pip on NormalPython. You can find pip on NormalPython\Scripts.
Go to NormalPython\Lib\site-packages and copy all files that appear after installing whatever you want to install through pip to EmbeddedPython\Lib\
Then add Lib to pythonXX._pth inside Embedded python folder.
This worked for me on windows 10 after downloading the latest python embed and python install through https://www.python.org/downloads/windows/ .
I used Michal Rawluk's answer to do it, but it was somewhat hard to understand so I am trying to explain it a bit differently.
At least with the latest Python versions (tested on 3.8 and 3.11) this appears to work fine:
Download the Windows embeddable package you need from the official site, and extract it.
PS> Invoke-WebRequest -Uri https://www.python.org/ftp/python/3.11.1/python-3.11.1-embed-amd64.zip -OutFile python-3.11.1-embed-amd64.zip
PS> Expand-Archive .\python-3.11.1-embed-amd64.zip
PS> cd .\python-3.11.1-embed-amd64
Open the python3xx._pth file corresponding to your version (e.g. python311._pth for Python 3.11), and make sure the following import line is uncommented. This will automatically add site directories used by pip to the Python path:
# Uncomment to run site.main() automatically
import site
In Powershell this can be automated by running:
PS> Add-Content -Path .\python311._pth -Value 'import site'
Download the official pip bootstrap script, e.g. with Powershell:
PS> Invoke-WebRequest -Uri https://bootstrap.pypa.io/get-pip.py -OutFile get-pip.py
Run the downloaded script (make sure you're using the correct Python executable in the current directory .\):
PS> .\python.exe get-pip.py
Collecting pip
Using cached pip-22.3.1-py3-none-any.whl (2.1 MB)
Collecting setuptools
Downloading setuptools-66.0.0-py3-none-any.whl (1.3 MB)
---------------------------------------- 1.3/1.3 MB 16.0 MB/s eta 0:00:00
Collecting wheel
Using cached wheel-0.38.4-py3-none-any.whl (36 kB)
Installing collected packages: wheel, setuptools, pip
WARNING: The script wheel.exe is installed in 'C:\Users\xxxx\Downloads\python-3.11.1-embed-amd64\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts pip.exe, pip3.11.exe and pip3.exe are installed in 'C:\Users\xxxx\Downloads\python-3.11.1-embed-amd64\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-22.3.1 setuptools-66.0.0 wheel-0.38.4
Run pip from the .\Scripts directory:
PS> .\Scripts\pip.exe list
Package Version
---------- -------
pip 22.3.1
setuptools 66.0.0
wheel 0.38.4
PS> .\Scripts\pip.exe install numpy
Collecting numpy
Downloading numpy-1.24.1-cp311-cp311-win_amd64.whl (14.8 MB)
---------------------------------------- 14.8/14.8 MB 11.9 MB/s eta 0:00:00
Installing collected packages: numpy
WARNING: The script f2py.exe is installed in 'C:\Users\xxxx\Downloads\python-3.11.1-embed-amd64\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed numpy-1.24.1
PS> .\python.exe -c 'import numpy; print(numpy.__version__)'
1.24.1
So I have published a conda package (link).
This package contains .c extensions (coming from cython code), which need to be compiled when the package is installed. My problem is that none of the extensions are compiled when running the install command
conda install -c nicolashug scikit-surprise
Compiling the extensions can be done by simply running
python setup.py install
which is exactly what pip does. The package is on PyPI and works fine.
As far as I understand, this setup.py command is only called when I build the conda package using conda build: the meta.yaml file (created with conda skeleton) contains
build:
script: python setup.py install --single-version-externally-managed--record=record.txt
But I need this to be done when the package is installed, not built.
Reading the conda docs, it looks like the install process is merely a matter of copying files:
Installing the files of a conda package into an environment can be thought of as changing the directory to an environment, and then downloading and extracting the .zip file and its dependencies
That would mean I would have to build the package for all platforms and architectures, and then upload them to conda... Which is impossible to me.
So, is there a way to build the package when it is installed, just like pip does?
As far as I know, there is no way to have the compilation happen on the user's machine when installing a conda package. Indeed, the whole idea of a conda package is that you do the compiling so that I don't have to on my machine, and all that's distributed is the compiled library. On Windows in particular, setting up compilers so they work properly (with Python) is a big big PITA, which is one of the biggest reasons for conda (and also wheels installed by pip).
If you don't have access to a particular OS directly, you can use Continuous Integration (CI) services, such as Appveyor (Windows), Travis CI (Linux/macOS), or CircleCI (Linux/macOS) to build packages and upload them to Anaconda cloud (or to PyPI for that matter). These services integrate directly with GitHub and other code-sharing services, and are generally free for FOSS projects. That way, you can build packages on each commit, on each tag, or some other variation that you desire.
In the end, you may save more time by setting up these services, because you won't have to provide compiler support for users who can't install a source package from PyPI.
A little background: I am working on some python modules that other developers on our team will use. A common theme of each module is that one or more messages will be published to Kafka. We intend at this time to use the Confluent Kafka client. We are pretty new to python development in our organization -- we have traditionally been a .NET shop.
The complication: while the code that we create will run on Linux (rhel 7), most of the developers will do their work on Windows.
So we need the librdkafka C library compiled on each developer machine (which has dependencies of its own, one of which is OpenSSL). Then a pip install of confluent-kafka should just work, which means a pip install of our package will work. Theoretically.
To start I did the install on my Linux laptop (Arch). I knew I already had OpenSSL and the other zip lib dependencies available, so this process was painless:
git clone librdkafka repo
configure, make and install per the README
pip install confluent-kafka
done
The install of librdkafka went into /usr/local:
/usr/local/lib/librdkafka.a
/usr/local/lib/librdkafka++.a
/usr/local/lib/librdkafka.so -> librdkafka.so.l
/usr/local/lib/librdkafka++.so -> librdkafka++.so.l
/usr/local/lib/librdkafka.so.l
/usr/local/lib/librdkafka++.so.l
/usr/local/lib/pkgconfig/rdkafka.pc
/usr/local/lib/pkgconfig/rdkafka++.pc
/usr/local/include/librdkafka/rdkafkacpp.h
/usr/local/include/librdkafka/rdkafka.h
Now the painful part, making it work on Windows:
install precompiled OpenSSL
git clone librdkafka repo
open in VS2015
install libz via NuGet
build solution
install to where???
This is where I'm stuck. What would a standard install on a Windows 7/8/10 machine look like?
I have the following from the build output, but no idea what should go where in order to make the pip install confluent-kafka "just work":
/librdkafka/win32/Release/librdkafka.dll
/librdkafka/win32/Release/librdkafka.exp
/librdkafka/win32/Release/librdkafka.lib
/librdkafka/win32/Release/librdkafkacpp.dll
/librdkafka/win32/Release/librdkafkacpp.exp
/librdkafka/win32/Release/librdkafkacpp.lib
/librdkafka/win32/Release/zlib.dll
<and the .h files back in the src>
Any recommendations on an install location?
I'm not sure where the ideal place to install on Windows would be, but I ran the following test with some success.
I copied my output and headers to C:\test\lib and C:\test\include, then ran a pip install with the following options:
pip install --global-option=build_ext --global-option="-LC:\test\lib" --global-option="-IC:\test\include" confluent-kafka
Unfortunately, this doesn't quite work because the confluent-kafka setup does not support Windows at this time: https://github.com/confluentinc/confluent-kafka-python/issues/52#issuecomment-252098462
It's an old question but seems still no easy answer yet. Also Confluent seems too busy to work on Windows supporting...
I had the same headache couple weeks ago and after some research, I managed to make it work for me on Windows. I logged my finding and uploaded a pre-compiled library to my Git, please check and see if it helps. :D
https://github.com/MichaelZhangCA/confluent-kafka-python
My environment is Python 3.6 64 bit version but ideally, it should also work for 32 bit if you follow the same approach.
I assume you have successfully followed instructions from MichaelZhangCA (https://github.com/MichaelZhangCA/confluent-kafka-python/) from previous post.
If you did so, these probably were the last two commands executed:
::Install confluent-kafka
cd C:\confluent-kafka-python\confluent-kafka-python-0.11.4
python setup.py install
If that is correct, those DLLs were created under C:\confluent-kafka-python\librdkafka-reference\release\.
All you have to do is to copy them to a diretory already in Widnows' PATH.
For example, I use Anaconda 5.2 For Windows, with python 3.6. My Anaconda Prompt has an empty directory in PATH, so I copied there those DLL's:
::Anaconda Prompt - copy DLLs to a directory already in PATH
mkdir %CONDA_PREFIX%\Library\usr\bin
copy C:\confluent-kafka-python\librdkafka-reference\release %CONDA_PREFIX%\Library\usr\bin
If you don't use Anaconda, just copy those DLL's to any other directory in Windows' PATH. You may also leave them in C:\confluent-kafka-python\librdkafka-reference\release, and add this directory to PATH.
I have a windows 7 machine with python 2.7 and I am trying to install pyzmq following these steps. I built libzmq got the binaries and copied them from libzmq\bin\Win32\Debug\v140\dynamic\ to libzmq\lib\ so the next step will work(compiler will have access to /lib and /includes from the same parent folder). But, on this step:
$ python setup.py configure --zmq=../libzmq
I installed pyzmq and libzmq on the same parent folder, as in the installation description related to pyzmq, libzmq is here: ../libzmq
But when I need to configure the pyzmq, I get this error:
I have VS Community 2015 installed and everything seems fine.
ZMQ usually provides sln files that will build from Microsoft Visual Studio. You will have to dig around a little to find it. You are better off trying to get that to work then going directly from the python setup.py that you are currently attempting. Note that you also need libsodium to be built and installed. Thankfully, they also provide Microsoft sln files.
In any case, you are probably better using one of the Python wheels like those from Christoph http://www.lfd.uci.edu/~gohlke/pythonlibs/
Just do
pip install <wheel file>
and you should be good.
Good luck!