Adding packages to Python "embedded" installation for Windows - python

From the documentation:
The embedded distribution is a ZIP file containing a minimal Python
environment.
Sounds great! The 64-bit Windows embedded v3.6.5 of Python is only 13MB. As an alternative to compiling, I would like to zip some python scripts together with the minimum needed to run them on a Win10 machine that doesn't have Python installed.
Now, I almost always need to import additional packages to provide functionality. But I can't determine how I should do this if I want to send out a python script together with this embedded version of Python. For example, if my script uses numpy, how can I include that library in this "embed?" I.e., so that on any Win10 machine I can unzip the single deployment file and immediately execute my scripts?
(A regular pip install numpy appears to create a Lib subdirectory that's over 50MB! But for an "embedded" deployment I don't need any support for debugging or whatever else is encompassed in that mass of files.)

There is a way to extend Python embedded installation.
I managed to create Flask-ready package, that I can just unzip on target machine and run code.
The trick is to install EXACT same python version (normal full blown python) as your target embedded small python. Not only version but x86, x64 has to match as well.
Then install modules from pip on normal python, go to NormalPython\Lib\site-packages and copy all new files that appear after installing the package to EmbeddedPython\Lib
finally add Lib and Lib\site-packages to pythonXX._pth inside Embedded python folder.
It's extremely important to fully test your application in case you miss some package.
Also this would not work for packages that also add .exe to Scripts folder.
You could still probably copy the exe's to Script folder and add it to path which could do the trick.

There is a similar trick.
Install pip in the embedded version:
curl -sSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
Edit pythonXX._pth by adding
Lib
Lib\site-packages
Use pip to install other packages

There is a way to extend Python embedded installation. The trick is to install the same python version( I will call it NormalPython) as your target embedded python(I will call it EmbeddedPython). The version as well as architecture has to match exactly.
You then install modules from pip on NormalPython. You can find pip on NormalPython\Scripts.
Go to NormalPython\Lib\site-packages and copy all files that appear after installing whatever you want to install through pip to EmbeddedPython\Lib\
Then add Lib to pythonXX._pth inside Embedded python folder.
This worked for me on windows 10 after downloading the latest python embed and python install through https://www.python.org/downloads/windows/ .
I used Michal Rawluk's answer to do it, but it was somewhat hard to understand so I am trying to explain it a bit differently.

At least with the latest Python versions (tested on 3.8 and 3.11) this appears to work fine:
Download the Windows embeddable package you need from the official site, and extract it.
PS> Invoke-WebRequest -Uri https://www.python.org/ftp/python/3.11.1/python-3.11.1-embed-amd64.zip -OutFile python-3.11.1-embed-amd64.zip
PS> Expand-Archive .\python-3.11.1-embed-amd64.zip
PS> cd .\python-3.11.1-embed-amd64
Open the python3xx._pth file corresponding to your version (e.g. python311._pth for Python 3.11), and make sure the following import line is uncommented. This will automatically add site directories used by pip to the Python path:
# Uncomment to run site.main() automatically
import site
In Powershell this can be automated by running:
PS> Add-Content -Path .\python311._pth -Value 'import site'
Download the official pip bootstrap script, e.g. with Powershell:
PS> Invoke-WebRequest -Uri https://bootstrap.pypa.io/get-pip.py -OutFile get-pip.py
Run the downloaded script (make sure you're using the correct Python executable in the current directory .\):
PS> .\python.exe get-pip.py
Collecting pip
Using cached pip-22.3.1-py3-none-any.whl (2.1 MB)
Collecting setuptools
Downloading setuptools-66.0.0-py3-none-any.whl (1.3 MB)
---------------------------------------- 1.3/1.3 MB 16.0 MB/s eta 0:00:00
Collecting wheel
Using cached wheel-0.38.4-py3-none-any.whl (36 kB)
Installing collected packages: wheel, setuptools, pip
WARNING: The script wheel.exe is installed in 'C:\Users\xxxx\Downloads\python-3.11.1-embed-amd64\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts pip.exe, pip3.11.exe and pip3.exe are installed in 'C:\Users\xxxx\Downloads\python-3.11.1-embed-amd64\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-22.3.1 setuptools-66.0.0 wheel-0.38.4
Run pip from the .\Scripts directory:
PS> .\Scripts\pip.exe list
Package Version
---------- -------
pip 22.3.1
setuptools 66.0.0
wheel 0.38.4
PS> .\Scripts\pip.exe install numpy
Collecting numpy
Downloading numpy-1.24.1-cp311-cp311-win_amd64.whl (14.8 MB)
---------------------------------------- 14.8/14.8 MB 11.9 MB/s eta 0:00:00
Installing collected packages: numpy
WARNING: The script f2py.exe is installed in 'C:\Users\xxxx\Downloads\python-3.11.1-embed-amd64\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed numpy-1.24.1
PS> .\python.exe -c 'import numpy; print(numpy.__version__)'
1.24.1

Related

python modules ffmpeg and ffprobe are installed but youtube-dl not able to find

I'm using python 3.8 on MacOS Big Sur
I installed python module package for youtube-dl with
pip3 install -upgrade youtube-dl
I wanted to do post processing on the content downloaded so I installed python packages ffprobe and ffmpeg.
pip3 install ffprobe
Collecting ffprobe
Downloading https://files.pythonhosted.org/packages/95/9c/adf90d21108d41f611aa921defd2f2e56d3f92724e4b5aa41fae7a9972aa/ffprobe-0.5.zip
Installing collected packages: ffprobe
Running setup.py install for ffprobe ... done
Successfully installed ffprobe-0.5
pip3 install ffmpeg
Collecting ffmpeg
Downloading https://files.pythonhosted.org/packages/f0/cc/3b7408b8ecf7c1d20ad480c3eaed7619857bf1054b690226e906fdf14258/ffmpeg-1.4.tar.gz
Installing collected packages: ffmpeg
Running setup.py install for ffmpeg ... done
Successfully installed ffmpeg-1.4
pip3 list
Package Version
---------- ----------
ffmpeg 1.4
ffprobe 0.5
youtube-dl 2021.12.17
Still it is complaining can't find ffprobe and ffmpeg:
youtube_dl.utils.DownloadError: ERROR: ffprobe/avprobe and ffmpeg/avconv not found. Please install one.
I see both packages in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/
Trying to figure out what am I doing wrong. Many other answers to similar questions on SFO are suggesting installing these packages in the OS using brew etc. but in my case "I want to do everything through python". So shouldn't just installing python modules be enough?
I faced the same issue, but it seems that i fixed it with uploading ffmpeg.exe file directly to Scripts (C:\Users\Sergey Khoteev\PycharmProjects\pythonProject\venv\Scripts).
Roadmap is following:
Download original ffmpeg.exe file from https://www.gyan.dev/ffmpeg/builds/. I've dowloaded "ffmpeg-git-full.7z"
Put to together with youtube-dl.exe it's in eg python37\Scripts folder.
Instaling ffmpeg from PyCharm package manager doesn't help, and idk why(.
Check the Path environment variable. It should contain an entry for the folder that ffprobe and ffmpeg are in.
EDIT: My bad, check this Reddit thread. https://www.reddit.com/r/learnpython/comments/gqhj14/comment/frsq2u3/?utm_source=share&utm_medium=web2x&context=3
It seems you might have to install the command-line tool.
Fix: We need to install and keep binary of ffmpeg on our path.
For Mac the binary could be downloaded from: ffmpeg.org and added to the path by updating PATH env variable.
Why??
When we install packages using pip/pip3 python just downloads and keeps these as available libraries to use for the python programs. That means with /
pip3 install ffprobe
pip3 install ffmpeg
We will have ffmpeg and ffprobe available as python modules to use in our python code. Library Documentation: https://kkroening.github.io/ffmpeg-python/
What youtube_dl expects is a binary installed on the OS and available on the path.
Also it is important to notice that we use youtube_dl python package and that library expects ffmpeg binary on the path.

Python: Warning intalling pip on windows 10

here is the message i get when i try to install pip (to install python packages) on windows 10 on my command prompt:
Collecting pip
Using cached pip-20.2.2-py2.py3-none-any.whl (1.5 MB)
Collecting setuptools
Downloading setuptools-49.6.0-py3-none-any.whl (803 kB)
|████████████████████████████████| 803 kB 1.3 MB/s
Installing collected packages: pip, setuptools
WARNING: The scripts pip.exe, pip3.8.exe and pip3.exe are installed in 'C:\Users\GMU\AppData\Local\Programs\Python\Python38-32\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts easy_install-3.8.exe and easy_install.exe are installed in 'C:\Users\GMU\AppData\Local\Programs\Python\Python38-32\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-20.2.2 setuptools-49.6.0
What does it mean ?
It says it installed but when i try the
pip --version
command or `
pip
i have message saying that pip is not a recognize command.
Thank's for your help.
I found the issue an the solution too.
First in not English native so at the first read in didn't get it.
The path where I've installed python wasn't good or something like that so pip couldn't retrieved it properly. The message is more or less clear about that fact.
Here is what I did:
uninstall pip
uninstall python 3.8.0
install python 3.8.5 with personalized config (not the recommended) because the install exe wanted to install python to the same folder than the last version. So I choose "C:\Program" and install Pip too with the prompt installer in order to get it to same folder.
Now it works fine for me.
Morality => don't install python to this path C:\Users\GMU\AppData\Local\Programs\Python\..
Windows is recommending you to add python to your PATH.
When you add python to your PATH, you will be able to execute pip in your command prompt.
I would recommend having a look at this https://geek-university.com/python/add-python-to-the-windows-path/.

in virtualenv, pip installing locally from source fails

I am trying to install a library in a virtualenv instance with pip. The library version I want (wxPython 3.0.2)
is not available on PyPi; it is only available for download from SourceForge. Thus, I have the source tarball downloaded on my machine and I am trying to install it in such a way that it will play nicely with virtualenv.
(I am on a Windows computer, running Python 2.7.)
I have tried the following:
doing a direct install: pip install wxPython-src-3.0.2.0.tar.bz2
extracting the files from the tarball to wxPython-src-3.0.2.0, then installing from the extracted directory: pip install wxPython-src-3.0.2.0
extracting the files from the tarball, then navigating into the extracted folder to the nested wxPython directory, which holds the setup.py file, and then installing from there: pip install wxPython
The last attempt seems the most promising, but I get the following traceback:
Processing \wxpython-src-3.0.2.0\wxpython
Complete output from command python setup.py egg_info:
Setuptools must be installed to build an egg
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in c:\users\__MY_USERNAME__\appdata\local\temp\pip-req-build-q0pxlt\
This is also strange, because it suggests I don't have setuptools even though I can run pip list and see version 40.6.3 installed.
Any help appreciated.
Why not install a precompiled version? There are a lot of .exe files at SF. You probably need wxPython3.0-win64-3.0.2.0-py27.exe.
Also take a look at Christoph Gohlke's collection.
If you still insist on installing from sources please bear in mind that wxPython 3 is so old it predates pip. Forget about pip.
First, you need to install wxWidgets as wxPython is just a Python wrapper for wxWidgets C++ library. Extract wxPython-src-3.0.2.0.tar.bz2 and follow instructions in wxPython-src-3.0.2.0/docs/msw/install.txt.
After compiling and installing wxWidgets compile wxPython. See wxPython-src-3.0.2.0/wxPython/docs/BUILD.txt.
My eventual solution was the easy way out: installing my package (wxPython) locally as #phd suggested, and opting for local package access via either virtualenv --system-site-packages env or deleting the "no-global-site-packages.txt" file in an existing environment folder.
Not what I expected to do, but it works so no complaints.

Python Re-packaging an Existing Package for Distribution

I'm using Python 3.6.3 on Windows 7 Enterprise and when I tried to pip install the Python package "bitarray", the output indicated the need for Microsoft Visual C++ Build Tools. I downloaded and installed the build tools and installed bitarray with no problems.
Here's where the problem comes in: I now need to distribute bitarray to other employees within the company who don't have Microsoft Visual C++ Build Tools installed, but do have Python installed (and can use pip).
Can I just simply "re-package" the bitarray folder in "C:\Python363\Lib\site-packages\bitarray" (which contains the already compiled .pyd file) and just make it a local package? This way I can use pip with "file:///" to pull down a local copy of the package without the need for the build tools step?
Also, do I need to incorporate the information in the folder "C:\Python363\Lib\site-packages\bitarray-0.8.1.dist-info" to re-package?
Thanks in advance for any help!!!!
Scott
Instead of trying to work around the already installed package, why not building a distribution from source yourself? After all, you've already done the hardest part setting up the C compiler, the rest is just a sequence of commands you have to type. This is what you can do:
Clone bitarray's repository:
$ git clone https://github.com/ilanschnell/bitarray
Navigate into the cloned repository:
$ cd bitarray
Checkout the version tag you want to build (the latest one is 0.8.1):
$ git checkout 0.8.1
Ensure you have wheel installed to be able to build a static wheel:
$ pip install wheel
Build the static wheel:
$ python setup.py bdist_wheel
A new directory dist was created in the current one, check what's inside:
$ ls dist
bitarray-0.8.1-cp36-cp36m-macosx_10_6_intel.whl
(Note: This is what I would enter on my system, list the directory with dir on Windows, also your file should be either bitarray-0.8.1-cp36-cp36m-win_amd64.whl if you are building on a 64 bit system, or bitarray-0.8.1-cp36-cp36m-win32.whl on a 32 bit one).
Now you have built a static wheel that contains the C extensions compiled for Python 3.6 on Windows. It can be installed on Windows without needing to setup the C compiler on the target machine. Just enter
$ pip install bitarray-0.8.1-cp36-cp36m-win_amd64.whl
Note, however, that this wheel file can be installed only on Windows and only with Python 3.6. Should you need to provide a wheel for another setup (like Python 3.5 on Windows 32 bit), you would need to build another wheel file using the correct Python version on a correct target system, but the steps would be just the same.
Building without Git
If you don't have Git installed and you can't/don't want to install it, just download the zipped repository from Github, unzip it, navigate to the extracted directory and perform steps 4-6.

How do you uninstall a python package that was installed using distutils?

Can you simply delete the directory from your python installation, or are there any lingering files that you must delete?
It varies based on the options that you pass to install and the contents of the distutils configuration files on the system/in the package. I don't believe that any files are modified outside of directories specified in these ways.
Notably, distutils does not have an uninstall command at this time.
It's also noteworthy that deleting a package/egg can cause dependency issues – utilities like easy_install attempt to alleviate such problems.
The three things that get installed that you will need to delete are:
Packages/modules
Scripts
Data files
Now on my linux system these live in:
/usr/lib/python2.5/site-packages
/usr/bin
/usr/share
But on a windows system they are more likely to be entirely within the Python distribution directory. I have no idea about OSX except it is more likey to follow the linux pattern.
Another time stamp based hack:
Create an anchor: touch /tmp/ts
Reinstall the package to be removed: python setup.py install --prefix=<PREFIX>
Remove files what are more recent than the anchor file: find <PREFIX> -cnewer /tmp/ts | xargs rm -r
Yes, it is safe to simply delete anything that distutils installed. That goes for installed folders or .egg files. Naturally anything that depends on that code will no longer work.
If you want to make it work again, simply re-install.
By the way, if you are using distutils also consider using the multi-version feature. It allows you to have multiple versions of any single package installed. That means you do not need to delete an old version of a package if you simply want to install a newer version.
In ubuntu 12.04, I have found that the only place you need to look by default is under
/usr/local/lib/python2.7/
And simply remove the associated folder and file, if there is one!
If this is for testing and/or development purposes, setuptools has a develop command that updates every time you make a change (so you don't have to uninstall and reinstall every time you make a change). And you can uninstall the package using this command as well.
If you do use this, anything that you declare as a script will be left behind as a lingering file.
install --record + xargs rm
sudo python setup.py install --record files.txt
xargs sudo rm -rf < files.txt
removes all files and but leaves empty directories behind.
That is not ideal, it should be enough to avoid package conflicts.
And then you can finish the job manually if you want by reading files.txt, or be braver and automate empty directory removal as well.
A safe helper would be:
python-setup-uninstall() (
sudo rm -f files.txt
sudo python setup.py install --record files.txt && \
xargs rm -rf < files.txt
sudo rm -f files.txt
)
Tested in Python 2.7.6, Ubuntu 14.04.
for Python in Windows:
python -m pip uninstall "package_keyword"
uninstall **** (y/n)?
For Windows 7,
Control Panel --> Programs --> Uninstall
, then
choose the python package to remove.
I just uninstalled a python package, and even though I'm not certain I did so perfectly, I'm reasonably confident.
I started by getting a list of all python-related files, ordered by date, on the assumption that all of the files in my package will have more or less the same timestamp, and no other files will.
Luckily, I've got python installed under /opt/Python-2.6.1; if I had been using the Python that comes with my Linux distro, I'd have had to scour all of /usr, which would have taken a long time.
Then I just examined that list, and noted with relief that all the stuff that I wanted to nuke consisted of one directory, /opt/Python-2.6.1/lib/python2.6/site-packages/module-name/, and one file, /opt/Python-2.6.1/lib/python2.6/site-packages/module-x.x.x_blah-py2.6.egg-info.
So I just deleted those.
Here's how I got the date-sorted list of files:
find "$#" -printf '%T# ' -ls | sort -n | cut -d\ -f 2-
(I think that's got to be GNU "find", by the way; the flavor you get on OS X doesn't know about "-printf '%T#'")
I use that all the time.
On Mac OSX, manually delete these 2 directories under your pathToPython/site-packages/ will work:
{packageName}
{packageName}-{version}-info
for example, to remove pyasn1, which is a distutils installed project:
rm -rf lib/python2.7/site-packages/pyasn1
rm -rf lib/python2.7/site-packages/pyasn1-0.1.9-py2.7.egg-info
To find out where is your site-packages:
python -m site
What it worked for me on Windows 10 and using Python for Windows version 3.96 was to just erase the folder containing the module inside the site-package folder. In my case this was located at: C:\Python396\Lib\site-packages.
After this a pip list does not show the module that I wanted to delete anymore.
ERROR: flake8 3.7.9 has requirement pycodestyle<2.6.0,>=2.5.0, but you'll have pycodestyle 2.3.1 which is incompatible.
ERROR: nuscenes-devkit 1.0.8 has requirement motmetrics<=1.1.3, but you'll have motmetrics 1.2.0 which is incompatible.
Installing collected packages: descartes, future, torch, cachetools, torchvision, flake8-import-order, xmltodict, entrypoints, flake8, motmetrics, nuscenes-devkit
Attempting uninstall: torch
Found existing installation: torch 1.0.0
Uninstalling torch-1.0.0:
Successfully uninstalled torch-1.0.0
Attempting uninstall: torchvision
Found existing installation: torchvision 0.2.1
Uninstalling torchvision-0.2.1:
Successfully uninstalled torchvision-0.2.1
Attempting uninstall: entrypoints
Found existing installation: entrypoints 0.2.3
ERROR: Cannot uninstall 'entrypoints'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
Then I type:
conda uninstall entrypoints
pip install --upgrade pycodestyle
pip install nuscenes-devkit
Done!

Categories

Resources