Develop Reference

PyPy and Python version

I have installed latest version of PyPy using sudo apt install pypy pypy-dev on my Xubuntu. When I try python --version I see that I have Python 3.8.10 installed. When I try pypy --version I got
Python 2.7.13 (7.3.1+dfsg-2, Apr 21 2020, 05:05:41)
[PyPy 7.3.1 with GCC 9.3.0]
Thats a problem because my script is written in Python 3 and I want to use pandas library in this script. How can I upgrade PyPy to Python 3? Thanks a lot.

It appears PyPy is not fully compatible with Python 3.8 just yet. Latest build is up to Python 3.7.9 with 3.8 support still in beta.
Also, there are different version it seems for different Python main versions, i.e PyPy for Python 2.x and PyPy3 for Python 3.x. Make sure you're either installing the combined package or the appropriate version PyPy3.8.
PyPy download page shows Python 3.8 as a standalone version of PyPy.

Python official newest version, Homebrew version and installed version differ on macOS

According to python.org, 3.8.2 is currently the latest version of
Python.
When I try to upgrade python3 using Homebrew (freshly
upgraded), it tells me version 3.7.6_1 is already installed.
When I ask python3 for it's current version, it tells me 3.7.3.
See terminal runs:
Nannas-Computer:some_folder username$ brew update
Already up-to-date.
Nannas-Computer:some_folder username$ brew upgrade
Nannas-Computer:some_folder username$ brew upgrade python3
Warning: python3 3.7.6_1 already installed
Nannas-Computer:some_folder username$ python3 --version
Python 3.7.3
Why are all these versions different, and how can I install the 3.8.2 version on my mac (macOS Mojave, version 10.14.6)?

You can install Python 3.8.x using command:
brew install python#3.8
As it's not main Python distribution on Homebrew, by default it's not accessible by simple typing python3. Binary is located in /usr/local/Cellar/python#3.8/3.8.2/bin/python3.8 and you can create a symlink to it or use directly by typing full path.
It's not that cumbersome as one may think because usually the only thing you need to do is to create a virtual environment and activate it. Since then it'll be your default Python (for this virtualenv):
$ /usr/local/Cellar/python#3.8/3.8.1/bin/python3.8 -m venv my_venv
$ source my_venv/bin/activate
(my_venv)$ python
Python 3.8.1 (default, Dec 27 2019, 18:06:00)
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

After some research I found out the following reasons:
It is possible to upgrade to Python 3.8.2 via Homebrew, but it isn't listed as supplying python due to some complications. See status here. Until this issue is done, 3.7.6_1 is the newest version you can upgrade to (as Homebrew kept telling me).
When I ran python3 --version in terminal, it returned Python 3.7.3 because I had installed this version myself at some point from the official website, without using Homebrew. Upgrading python to 3.7.6 via Homebrew had no effect on this dominating installation of python. I got rid of the manually installed version using this useful answer, after which I reinstalled python properly using Homebrew just to be on the safe side and now I finally have:
Nannas-Computer:some_folder username $ python3 --version
Python 3.7.6

No module named yum

I tried the solution #yum---no module named yum and "No module named yum" with Python 2.7 but didn't help,it sounds like the yum module is not a stock Python module and need to build yum against your Python 2.7 install,can anyone provide guidance on how to do this?
machine details:
[usernames#machine]$ cat /etc/*elease
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Oracle Linux Server release 6.6
Red Hat Enterprise Linux Server release 6.6 (Santiago)
Oracle Linux Server release 6.6
Error:-
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:
No module named yum
Please install a package which provides this module, or
verify that the module is installed correctly.
It's possible that the above module doesn't match the
current version of Python, which is:
2.7.12 (default, Aug 11 2016, 12:02:22)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-11)]
If you cannot solve this problem yourself, please go to
the yum faq at:
http://yum.baseurl.org/wiki/Faq

It appears what got clobbered was the system Python in /usr/bin, that knows where to find the python modules and shared objects installed by RHEL. So you need to reinstall RHEL python rpms, not just yum.
RHEL uses /usr/bin/python for many of it's utilities. yum is a Python (wrapper) script.
It looks like you are using RHEL 6. So /usr/bin/python must be 2.6. That's what RHEL 6 shipped with. (For RHEL 7 it needs to be 2.7.5)
So, to fix things you need to reinstall the system Python from the RHEL rpms. For the system to work correctly, /usr/bin/python can't be an arbitrary version of Python.
The yum rpm will just reinstall the yum files to the original path but you need the RHEL /usr/bin/python which is loading the modules from the correct directories. The rpm will not install the module in the Python sense. Also, there are many other modules with shared library dependencies for things like DBus interfaces.
So obtain the RHEL python-* rpms for your release. You'll need to download them, or use the ISO/DVD. rpm by itself doesn't know how to access software repos over the network that's yum's job so you have to do it manually.
Alternatively if you have another system running the same version of RHEL you could copy files over. You can use rpm -Va to get a list of all files that belong to rpms that have been modified. There are a number of files that get modified during normal operation.
(Moral of the story, don't overwrite /usr/bin/python, /usr/bin/python2. RHEL (and other platforms, Fedora, etc.) depend on the system python to work correctly.)

Well, assuming rpm is not broken - you could do something like this:
First:
rpm -q yum
for retrieving installed ( but broken ) yum package. For example on my Centos7 i get next result: yum-3.4.3-154.el7.centos.1.noarch.rpm
But that's not all. You should remove part between .el7 and noarch. Then, you can proceed to second step
Second:
rpm -qpR <yum package name>
or:
rpm -qR yum
for getting yum dependecies list.
And the last one:
rpm -Uvh <packagenames>

Apparently you replaced the system version of Python.
Don't do that. Use virtualenv, pip, and/or other tools to run custom versions of Python when you need them.
Try to put back the official distro Python installation.

OS X - Deciding between anaconda and homebrew Python environments

I use Python extensively on my Mac OS X, for both numerical applications and web development (roughly equally). I checked the number of Python installations I had on my laptop recently, and was shocked to find four:
Came with Mac OS X:
/usr/bin/python
Python 2.7.6 (default, Sep 9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Installed via Homebrew
/usr/local/bin/python
Python 2.7.10 (default, Jul 13 2015, 12:05:58)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Installed via Anaconda/Miniconda
~/anaconda/bin/python
Python 2.7.10 |Anaconda 2.3.0 (x86_64)| (default, Oct 19 2015, 18:31:17)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
Came with the downloaded .pkg from python.org
/System/Library/Frameworks/Python.framework/Versions/Current/bin/python
Python 2.7.6 (default, Sep 9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
I decided to unify all of this, and use conda. I removed the Homebrew version and the Python.org download (kept the main system one). Conda is great for numerical computing, because I can install Jupyter/Numpy/Pandas in the root environment, and not have to bother install virtualenvs for every project.
But now my entire web development workflow is messed up. None of my virtualenvs work, since apparently one's not supposed to use conda and virtualenv together. I tried to create conda environments from the requirements.txt file. One package I was using with django was "markdown_deux", which is not available in the Conda repo. I looked at ways of building it, but creating a recipe takes a lot of effort (create YAML file, etc..)
Has anyone found a good compromise for this? I'm thinking of going back to the homebrew version for general use, and writing an alias for changing the path back to the conda version as necessary. Though this will also require tracking which one I'm using now..

I use Homebrew Python for all my projects (data science, some web dev).
Conda is nothing fancy, you can have the same packages by hand with a combination of pip and Homebrew science. Actually, it is even better because you have more control on what you install.
You can use your virtualenvs only when you do web development. For the numerical applications you will probably want to have the latest versions of your packages at all times.
If you want to update all your packages at once with pip, you can use this command:
sudo -H pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 sudo -H pip install -U
EDIT: This answer is old, if you want a more up-to-date comparison, I found this nice blog article which compares the two approaches:
https://towardsdatascience.com/pipenv-vs-conda-for-data-scientists-b9a372faf9d9
I still use Homebrew Python, and pip / pipenv over conda.

Workflow that I've found the best:
Use conda for virtual environment management. Never use / install into system python.
Use pip to install into the active virtual environment, just like normal.
Use conda packages only for hard to install software, such as Qt.
Automation / extras
Use autoenv or direnv and automatically activate virtual environments when you enter a directory by putting the conda command inside the .env or .envsrc file.

Can python.org releases play nice with Apple framework builds?

(Advanced apologies for the lack or proper links; the system won't allow me to add more than two.)
Unfortunately, I've learnt the hard way that you shouldn't mess with the default Python installations in Mac OS X (specifically, 10.6.8).
After using the python.org installers for 2.6.6 (http://www.python.org/getit/releases/2.6.6/) and 2.5.4 (http://www.python.org/getit/releases/2.5.4/), I have Python versions which are more mature than those provided by Apple (which is great for development), but have broken core system functionality (which is bad for just about everything else.) The most visible breaks so far have been when trying to run namebench (https://code.google.com/p/namebench/), Blink (http://icanblink.com/) and Mercurial (https://www.mercurial-scm.org/). From what I can gather, it's down to the paths.
In a default installation, the paths should resemble something those outlined in this question. Instead, mine look like this:
$ /usr/bin/python2.6 -V
Python 2.6.6
$ which python
/usr/bin/python
$ python
Python 2.6.6 (r266:84374, Aug 31 2010, 11:00:51)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for i in sys.path:
... print i
...
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pip-1.0.2-py2.6.egg
/Library/Frameworks/Python.framework/Versions/2.6/lib/python26.zip
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-old
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages
(It's a similar story for 2.5 but, for simplicity, I'll stick with 2.6.)
The issue seems to be the non-inclusion of:
/Library/Python/2.6/site-packages/*.egg
/Library/Python/2.6/site-packages
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python
This explains why Blink and namebench can't find PyObjC (Is PyObjC pre-installed on OSX SL?) and Mercurial can't find its modules (in /Library/Python/2.6)
Issue #4865 in the Python bug tracker partially addresses this particular problem, which has since been fixed for versions 2.7 and greater. But because it's "a feature request and not a bugfix", has not been back-ported to 2.5 and 2.6. Even then, looking at the committed fix (http://hg.python.org/lookup/r70778), I'm not sure it addresses the lack of references to the "Extras" directories.
I understand that I can manually add paths to Python by manually altering site.py. I could also make use of the PYTHONPATH environment variable. I'd rather not cause any further damage by altering site.py, and changes PYTHONPATH would only be valid for scripts/applications run from the shell using my user account.
Can I get these more recent versions of Python to reference the same paths as the default framework installations? If so, what is the best way to go about it? If not, is there an accepted method of rolling back to the system defaults?

You misunderstand how Python installations work on OS X. Each Python instance has its own site-packages directory. The standard location for framework installers is within the framework at ./lib/pythonx.y/site-packages. So for the python.org installers which install into /Library/Frameworks/Python.framework, you will find its 2.6 site-packages here:
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/
Apple makes some modifications to the versions of Python that it ships with OS X releases. From OS X 10.5 on, the system Pythons are installed at:
/System/Library/Frameworks/Python.framework
and Apple chooses to include some extra 3rd-party packages in the non-standard ./Extras directory within each version. It also uses a non-standard location for each version's site-packages directory. They are created in /Library/Python/, presumably so that user-installed packages do not modify anything in /System/Library. So for the Apple-supplied Python 2.6, its site-packages directory is:
/Library/Python/2.6/site-packages
and can be thought of as having been extended by the packages in ./Extras.
Each Python instance has a separate site-packages directory. It is not intended and often not possible to share packages across site-packages of different instances even of the same minor version of Python, i.e. 2.6. The most obvious problem is that there are often differences in the C compiler version, the OS X ABI (MACOSX_DEPLOYMENT_TARGET), the SDK version, and/or the CPU architectures used to build the Python interpreters and which are subsequently used by Distutils to build C extension modules included in 3rd-party packages.
In Mac OS X 10.6, the Apple-supplied Python is built using gcc-4.2 and targeted just for OSX 10.6 and includes 3 CPU architectures (i386, x86_64, and ppc). The python.org installers for Python 2.6 are built to run on older systems as well, so have a target of 10.3 and later, use gcc-4.0, and are 32-bit only (i386 and ppc). So, in general, you cannot run C extension modules built for the one Python with the other.
This means that, in general, you need to install separate copies of 3rd-party packages you need with and for each Python, if they are not already included with that Python. This includes basic items like easy_install (from setuptools or Distribute). The system Pythons in 10.5+ include easy_install version in /usr/bin for them. If you install a python.org Python, you'll need to install a separate version for it; by default, the easy_install command will be installed in that Python's ./bin directory in its framework; that is the Distutils default location. That's why it is recommended that you add this directory to your shell PATH (and the python.org installer for Python 2 do that automatically by default).
The change introduced by Issue4865 is not really a good solution and may fail with C extension modules. I would not depend on it remaining in Python in future versions.
Also installing a python.org Python in no way breaks anything in the system Python because they are completely independent installations using different file system locations. The only thing that may change is which instance of Python is invoked when you type a particular name. That is controlled primarily by the search order of your shell PATH environment variable. As noted, the python.org installer by default changes that order but the system Python is still readily available by using its absolute path /usr/bin/python2.6. Or you can revert the changes to shell profile, for instance, .bash_profile.
$ echo $PATH
/Library/Frameworks/Python.framework/Versions/2.6/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
$ which python python2.6
/Library/Frameworks/Python.framework/Versions/2.6/bin/python
/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6
$ python -V
Python 2.6.6
$ python2.6 -V
Python 2.6.6
$ /usr/bin/python -V
Python 2.6.1
$ /usr/bin/python2.6 -V
Python 2.6.1
#
# remove python.org Python 2.6 from PATH
#
$ export PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
$ which python python2.6
/usr/bin/python
/usr/bin/python2.6
$ python -V
Python 2.6.1
$ python2.6 -V
Python 2.6.1