Develop Reference

How to install Python packages from python3-apt in PyCharm on Windows?

I'm on Windows and want to use the Python package apt_pkg in PyCharm.
On Linux I get the package by doing sudo apt-get install python3-apt but how to install apt_pkg on Windows?
There is no such package on PyPI.

There is no way to run apt-get in Windows; the package format and the supporting infrastructure is very explicitly Debian-specific.

Not quite what you're looking for, but it's possible to use apt-offline on Windows to download the packages. However, you still need a Linux box to generate the sig file.
For example:
python apt-offline set vim-offline.sig --install-packages vim
Will not work:
ERROR: This argument is supported only on Unix like systems with apt installed
However, if you run that command on Linux first, the following command should work on Windows:
python apt-offline get vim-offline.sig -d vim
apt-offline source is available here:
https://github.com/rickysarraf/apt-offline
To build it, simply run:
python setup.py build
python setup.py install
I got it to run with Python 3.8.2 on Windows 10.
Note: in the end of the day, you're just downloading a .deb package and it is simply an ar file containing a tarball and can be extracted with tools like 7-zip. However, if it contains a Linux binary (vim example), there isn't much you can do with it on Windows.

One can use chocolatey the equivalent for windows.
https://chocolatey.org/install
add it to the windows PATH environment
C:\ProgramData\chocolatey\bin
Restart python or anaconda. And is ready to use.
To install packages inside a .py script or a Jupiter notebook, use the syntax below
!choco install [package name]

How to move all modules to new version of Python (from 3.6 to 3.7)

I just upgraded to python 3.7 and I realized that all my modules stuck with the previous version. Even Django is not recognised anymore. How can I do to transfer everything to the new version? I am a little lost right now, don't even know where the new version has been installed.
Edit:
When I do $ which python3.6 the terminal tells me it doesn't exist, but I have a python3.6 directory in /usr/local/lib/, where all modules are installed.
In the same directory /usr/local/lib/ I also have a python3.7 directory with some modules installed but many are missing. However when I search for the file python3.7 in my finder it doesn't appear. when I do $ which python3.7 the path is /usr/local/bin so not the same path as the directory.
Anyone sees what happened and knows how I can transfer all modules to python3.7?

Even if the old python version has been removed, it is possible to use the pip of the current python version with the --path option to list all the modules installed in the previous version.
For example, migrating all my user installed python modules from 3.7 to 3.8
pip freeze --path ~/.local/lib/python3.7/site-packages > requirements.txt
pip install --user -r requirements.txt
Incidentally, I always use pip install with --user and leave the system wide installations to the package manager of my linux distro.

It is safer to re-install all packages due to possible compatibility issues:
pip3.6 list | awk '{print $1}' | xargs -I{} pip3.7 install {}

in older version of Python --run the command
pip freeze > requirements.txt
download and install newer version on python.. change the PATH variable to the new version
and run the command
pip install -r requirements.txt

I'm not sure about all modules...but if you want to install a module specifically in python3.7, try this:
python3.7 -m pip install *module_name*

In some cases, we don't have the opportunity to pip freeze in old version--because I've already updated and old version have been purged! There are some measures I've taken to recover some of the packages but I'm NOT sure every package would work with this fix.(e.g. the packages built with wheels)
mv /your/path/to/python3.{6,7}/site-packages/
If the case is packages installed outside venv (in /usr/local/ or ~/.local), reinstall pip with get-pip.py, just to be safe.
If you are recovering a virtualenv. Activate your virtualenv and use my script
Most of your packages should work by now. If anything malfunctions, pip reinstall would works. If you still want it 100% works, pip freeze now.😉

I have an alternative
(Not sure if works outside Windows 10)
I'm currently migrating from 3.7 to 3.8 and the way I found to re-install my previous libraries was by using a script I had that updates all packages via pip install. (Assuming you installed your new Python version as your main version) This checks for all the packages I had and updates/install them in the new Python version.
Note: I prefer to run the script from the command line
Use the file explorer to go to the folder where you have the script;
Click on the path box, write "cmd" and press enter to open a command line from the folder where you are;
Write "python name_of_your_script.py" and press enter to run the command.
The script (adapted from this solution):
import pkg_resources
from subprocess import call
packages = [dist.project_name for dist in pkg_resources.working_set]
[call("pip install " + name + " --upgrade") for name in packages]

I faced a similar problem, now that I upgraded from python 3.7 to python 3.8 (new)
I installed Python 3.8, but the system kept the python37 subfolder with the already installed packages(...\Python37-32\Lib\site-packages) even with the Pyhton38 subfolder created, with the new python.exe.
Usually, it's possible to keep on using the old libraries on your new Python version, because the existent libraries installation folder are already registered in your local computer system path (*).
Even though, I've had problems to use some libraries (some worked in Jupyter Notebook but not in Spyder). I tried the alternatives others proposed to migrate libraries but it did not worked (maybe I did not
So I used brutal force solution.. Not elegant at all, but it worked:
remove the OLD python version folders from the system path or even remove the folder itself for good..
Folders: C:\Users\USERNAME\AppData\Roaming\Python\Python37
C:\Users\USERNAME\AppData\Local\Programs\Python\Python37
Reinstall the packages you need, preferably via Anaconda prompt.
python -mpip install library_name
OR
pip install --user --force-reinstall package_name
OR
pip install --user --force-reinstall package_name == specify_package_version
The libraries will be installed at c:\users\USERNAME\anaconda3\lib\site-packages and recognized by your new python version.
(*) to add the folder to the PATH: System properties --> environment variables --> click "Path"--> edit --> add folder name)

Debian 8: Installed Python 2.7.13 But No Pip; Packages Install to Wrong Path

I am running Debain 8 Jessie and have installed python 2.7.13 into /usr/local/bin.
I do not have a new pip installed, I am still using the pip located at /usr/bin/. Any time I install a package using pip, they are installed to the dist-packages location of my new python install:
/usr/local/lib/python2.7/dist-packages
Question
How do I get my pip installs to permanently install in /usr/lib/python2.7/dist-packages? What is even weirder, is that although these packages are installing to a location seemingly setup by my new install, I cannot import whatever I download when I am using python2.7.13 via /usr/local/bin/python.
PATH
/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/sbin:/sbin:/bin:
Better Yet
How do I just get my python2.7.13 install to see the installed packages from pip?

Run this:
import sys
for p in sys.path:
print p # or print(p) if you prefer.
Look for /usr/lib/python2.7
if not you'll want to
export PYTHONPATH=/usr/local/lib/python2.7:/usr/lib/python2.7
in .bashrc or in a special (host-wide) script in /etc/profile.d/
If /usr/lib/python2.7/
Depending on the difference between your "stock" version and 2.7.13, you might not want to put stuff in /usr/lib/python2.7. On my system I'd really want to avoid any cross contamination, but that's something I'm a bit picky about.

How to properly install Python on OSX for use with OpenCV?

I spent the past couple of days trying to get opencv to work with my Python 2.7 install. I kept getting an error saying that opencv module was not found whenever I try "import cv".
I then decided to try installing opencv using Macports, but that didn't work.
Next, I tried Homebrew, but that didn't work either.
Eventually, I discovered I should modify the PYTHONPATH as such:
export PYTHONPATH="/usr/local/lib/python2.6/site-packages/:$PYTHONPATH"
My problem is that I didn't find /usr/local/lib/python2.*...etc
The folder simply doesn't exist
So my question is this:
How do I properly install Python on OS X Snow Leopard for it to work with opencv?
Thanks a lot,

I spent a couple days on this myself. For me, the problem was that that OpenCV installer was not finding the right python installation. It was defaulting to the MacOS-installed version despite the fact that I had upgraded python with homebrew and was using a virtualenv for python. I have collected most of my setup in a gist here:
https://gist.github.com/4150916
Use homebrew to get all the dependencies, but then download the OpenCV tarball and compile yourself being sure to specify all the python related configuration options.
Assuming a virtualenv named 'opencv'...
cd OpenCV-2.4.3/
mkdir release
cd release
cmake -D PYTHON_EXECUTABLE=$WORKON_HOME/opencv/bin/python \
-D PYTHON_PACKAGES_PATH=$WORKON_HOME/opencv/lib/python2.7/site-packages \
-D INSTALL_PYTHON_EXAMPLES=ON\
-D PYTHON_INCLUDE_DIR=/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Headers\
-D PYTHON_LIBRARY=/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib\
..
make -j8
make install

You need to install the module using your python2.7 installation. Pointing your PYTHONPATH at stuff installed under 2.6 to run under 2.7 is a Bad Idea.
Depending on how you want to install it, do something like python2.7 setup.py or easy_install-2.7 opencv to install.
fwiw, on OS X the modules are usually installed under /System/Library/Frameworks/Python.framework/ but you should almost never need to know where anything installed in your site packages is physically located; if Python can't find them without help you've installed them wrong.

Installing OpenCV with Homebrew
brew tap homebrew/homebrew-science
brew install opencv
Setting up Python
Depending on your install location - OS X Default
cd /Library/Python/2.7/site-packages/
or - Homebrew Python
cd /usr/local/lib/python2.7
Then create the symbolic link
ln -s /usr/local/Cellar/opencv/2.4.9/lib/python2.7/site-packages/cv.py cv.py
ln -s /usr/local/Cellar/opencv/2.4.9/lib/python2.7/site-packages/cv2.so cv2.so
The above method sourced from a blog post.

I searched and tried installing opencv3 with python3 for 3 days. Some links suggest for Brew and some virtual env, some say install xcode but all failed in my case.
Dont use linux steps to instal opencv-python on Mac. Problem with Mac is Python 2.7 is already installed by Mac. On top of that installing and linking all site-packages is little problematic and we end up with errors.
I'll share what I did: easy steps to install complete package opencv3, numpy, matplotlib, notebook, spyder etc.. on Mac.
Install anaconda, it creates a directory and install everything inside that
use this link -> https://www.continuum.io/downloads
download command-line-install
After download, goto terminal and download location of anaconda.
$ bash Anaconda3-4.3.0-MacOSX-x86_64.sh
Installation will ask you to append path to .bash_profile >> say yes
Goto home directory, run .bash_profile
$ source .bash_profile
check python, should be pointing to
$ which python
$ /.../anaconda/bin/python
Last step
$ pip install opencv-pyhton
$ python
$ import cv2
if no errors, we are good to go.

Upgrade Python in Windows [duplicate]

I have a Python 2.7.11 installed on one of my LAB stations. I would like to upgrade Python to at least 3.5.
How should I do that ?
Should I prefer to completely uninstall 2.7.11 and than install the new one ?
Is there a way to update it ? Is an update a good idea ?

Every minor version of Python, that is any 3.x and 2.x version, will install side-by-side with other versions on your computer. Only patch versions will upgrade existing installations.
So if you want to keep your installed Python 2.7 around, then just let it and install a new version using the installer. If you want to get rid of Python 2.7, you can uninstall it before or after installing a newer version—there is no difference to this.
Current Python 3 installations come with the py.exe launcher, which by default is installed into the system directory. This makes it available from the PATH, so you can automatically run it from any shell just by using py instead of python as the command. This avoids you having to put the current Python installation into PATH yourself. That way, you can easily have multiple Python installations side-by-side without them interfering with each other. When running, just use py script.py instead of python script.py to use the launcher. You can also specify a version using for example py -3 or py -3.6 to launch a specific version, otherwise the launcher will use the current default (which will usually be the latest 3.x).
Using the launcher, you can also run Python 2 scripts (which are often syntax incompatible to Python 3), if you decide to keep your Python 2.7 installation. Just use py -2 script.py to launch a script.
As for PyPI packages, every Python installation comes with its own folder where modules are installed into. So if you install a new version and you want to use modules you installed for a previous version, you will have to install them first for the new version. Current versions of the installer also offer you to install pip; it’s enabled by default, so you already have pip for every installation. Unless you explicitly add a Python installation to the PATH, you cannot just use pip though. Luckily, you can also simply use the py.exe launcher for this: py -m pip runs pip. So for example to install Beautiful Soup for Python 3.6, you could run py -3.6 -m pip install beautifulsoup4.

If you are upgrading any 3.x.y to 3.x.z (patch) Python version, just go to Python downloads page get the latest version and start the installation. Since you already have Python installed on your machine installer will prompt you for "Upgrade Now". Click on that button and it will replace the existing version with a new one. You also will have to restart a computer after installation.
If you are upgrading from 3.x to 3.y (minor) then you will be prompted with "Install Now". In this case, you are not upgrading, but you are installing a new version of Python. You can have more than one version installed on your machine. They will be located in different directories. When you have more than one Python version on your machine you will need to use py lanucher to launch a specific version of Python.
For instance:
py -3.7
or
py -3.8
Make sure you have py launcher installed on your machine. It will be installed automatically if you are using default settings of windows installer. You can always check it if you click on "Customize installation" link on the installation window.
If you have several Python versions installed on your machine and you have a project that is using the previous version of Python using virtual environment e.g. (venv) you can upgrade Python just in that venv using:
python -m venv --upgrade "your virtual environment path"
For instance, I have Python 3.7 in my ./venv virtual environment and I would like upgrade venv to Python 3.8, I would do following
python -m venv --upgrade ./venv

Installing/Upgrading Python Using the Chocolatey Windows Package Manager
Let's say you have Python 2.7.16:
C:\Windows\system32>python --version
python2 2.7.16
...and you want to upgrade to the (now current) 3.x.y version. There is a simple way to install a parallel installation of Python 3.x.y using a Windows package management tool.
Now that modern Windows has package management, just like Debian Linux distributions have apt-get, and RedHat has dnf: we can put it to work for us! It's called Chocolatey.
What's Chocolatey?
Chocolatey is a scriptable, command line tool that is based on .NET 4.0 and the nuget package manager baked into Visual Studio.
If you want to learn about Chocolatey and why to use it, which some here reading this might find particularly useful, go to https://chocolatey.org/docs/why
Installing Chocolatey
To get the Chocolatey Package Manager, you follow a process that is described at https://chocolatey.org/docs/installation#installing-chocolatey,
I'll summarize it for you here. There are basically two options: using the cmd prompt, or using the PowerShell prompt.
CMD Prompt Chocolatey Installation
Launch an administrative command prompt. On Windows 10, to do this:
Windows + R
Type cmd
Press Ctrl + ⇧ Shift + ↵ Return
If you don't have administrator rights on the system, go to the Chocolatey website. You may not be completely out of luck and can perform a limited local install, but I won't cover that here.
Copy the string below into your command prompt and type Enter:
#"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"
Chocolatey will be downloaded and installed for you as below:
Getting latest version of the Chocolatey package for download.
Getting Chocolatey from https://chocolatey.org/api/v2/package/chocolatey/0.10.11.
Downloading 7-Zip commandline tool prior to extraction.
Extracting C:\Users\blahblahblah\AppData\Local\Temp\chocolatey\chocInstall\chocolatey.zip to C:\Users\blahblahblah\AppData\Local\Temp\chocolatey\chocInstall...
Installing chocolatey on this machine
Creating ChocolateyInstall as an environment variable (targeting 'Machine')
Setting ChocolateyInstall to 'C:\ProgramData\chocolatey'
WARNING: It's very likely you will need to close and reopen your shell
before you can use choco.
Restricting write permissions to Administrators
We are setting up the Chocolatey package repository.
The packages themselves go to 'C:\ProgramData\chocolatey\lib'
(i.e. C:\ProgramData\chocolatey\lib\yourPackageName).
A shim file for the command line goes to 'C:\ProgramData\chocolatey\bin'
and points to an executable in 'C:\ProgramData\chocolatey\lib\yourPackageName'.
Creating Chocolatey folders if they do not already exist.
WARNING: You can safely ignore errors related to missing log files when
upgrading from a version of Chocolatey less than 0.9.9.
'Batch file could not be found' is also safe to ignore.
'The system cannot find the file specified' - also safe.
chocolatey.nupkg file not installed in lib.
Attempting to locate it from bootstrapper.
PATH environment variable does not have C:\ProgramData\chocolatey\bin in it. Adding...
WARNING: Not setting tab completion: Profile file does not exist at 'C:\Users\blahblahblah\Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1'.
Chocolatey (choco.exe) is now ready.
You can call choco from anywhere, command line or powershell by typing choco.
Run choco /? for a list of functions.
You may need to shut down and restart powershell and/or consoles
first prior to using choco.
Ensuring chocolatey commands are on the path
Ensuring chocolatey.nupkg is in the lib folder
Either Exit the CMD prompt or type the following command to reload the environment variables:
refreshenv
PowerShell Chocolatey Installation
If you prefer PowerShell to the cmd prompt, you can do this directly from there, however you will have to tell PowerShell to run with a proper script execution policy to get it to work. On Windows 10, the simplest way I have found to do this is to type the following into the Cortana search bar next to the Windows button:
PowerShell.exe
Next, right click on the 'Best Match' choice in the menu that pops up and select 'Run as Administrator'
Now that you're in PowerShell, hopefully running with Administrator privileges, execute the following to install Chocolatey:
Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
PowerShell will download Chocolatey for you and launch the installation. It only takes a few moments. It looks exactly like the CMD installation, save perhaps some fancy colored text.
Either Exit PowerShell or type the following command to reload the environment variables:
refreshenv
Upgrading Python
The choco command is the same whether you use PowerShell or the cmd prompt. Launch your favorite using the instructions as above. I'll use the administrator cmd prompt:
C:\WINDOWS\system32>choco upgrade python -y
Essentially, chocolatey will tell you "Hey, Python isn't installed" since you're coming from 2.7.x and it treats the 2.7 version as completely separate. It is only going to give you the most current version, 3.x.y (as of this writing, 3.7.2, but that will change in a few months):
Chocolatey v0.10.11
Upgrading the following packages:
python
By upgrading you accept licenses for the packages.
python is not installed. Installing...
python3 v3.x.y [Approved]
python3 package files upgrade completed. Performing other installation steps.
Installing 64-bit python3...
python3 has been installed.
Installed to: 'C:\Python37'
python3 can be automatically uninstalled.
Environment Vars (like PATH) have changed. Close/reopen your shell to
see the changes (or in powershell/cmd.exe just type `refreshenv`).
The upgrade of python3 was successful.
Software installed as 'exe', install location is likely default.
python v3.x.y [Approved]
python package files upgrade completed. Performing other installation steps.
The upgrade of python was successful.
Software install location not explicitly set, could be in package or
default install location if installer.
Chocolatey upgraded 2/2 packages.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
Either exit out of the cmd/Powershell prompt and re-enter it, or use refreshenv then type py --version
C:\Windows\System32>refreshenv
Refreshing environment variables from registry for cmd.exe. Please wait...Finished..
C:\Windows\system32>py --version
Python 3.7.2
Note that the most recent Python install will now take over when you type Python at the command line. You can run either version by using the following commands:
py -2
Python 2.7.16 (v2.7.16:413a49145e, Mar 4 2019, 01:37:19) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
C:\>py -3
Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>exit()
C:\>
From here I suggest you use the Python pip utility to install whatever packages you need. For example, let's say you wanted to install Flask. The commands below first upgrade pip, then install Flask
C:\>py -3 -m pip install --upgrade pip
Collecting pip
Downloading https://files.pythonhosted.org/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl (1.4MB)
100% |████████████████████████████████| 1.4MB 1.6MB/s
Installing collected packages: pip
Found existing installation: pip 18.1
Uninstalling pip-18.1:
Successfully uninstalled pip-18.1
Successfully installed pip-19.0.3
c:\>py -3 -m pip install Flask
...will do the trick. Happy Pythoning!

A quick and painless way for me was to do the following:
Do a pip freeze > requirements.txt on my affected environments (or whatever method you want for backing up your requirements)
Remove the Old version of Python (in my case it was 3.8)
Remove the associated environments
Install the new version (3.9.5 in my case)
Recreate my environments python -m venv venv or however you wish
Reinstall my plug-ins/apps pip install -r requirements.txt or however you wish

Python 2.x and Python 3.x are different. If you would like to download a newer version of Python 2, you could just download and install the newer version.
If you want to install Python 3, you could install Python 3 separately then change the path for Python 2.x to Python 3.x in Control Panel > All Control Panel Items > System > Advanced System Settings > Environment Variables.

In 2019, you can install using chocolatey. Open your cmd or powershell, type:
choco install python

Nowadays you can just install several Python versions from Microsoft Store. Those are published there by Python Software Foundation.
https://apps.microsoft.com/store/search?hl=en-en&gl=EN&publisher=Python%20Software%20Foundation

Just install python newest version's installer it will automatically detect your python version and will say upgrade python and starts upgrading

I was able to execute PowerShell with the following command and python upgraded with no issue. python -m pip install --upgrade pip please see image

You can use pyenv. It let you easily switch between all versions of python.