Develop Reference

Installing packages in python and setting up the working environment

I've been coding with R for quite a while but I want to start learning and using python more for its machine learning applications. However, I'm quite confused as to how to properly install packages and set up the whole working environment. Unlike R where I suppose most people just use RStudio and directly install packages with install.packages(), there seems to be a variety of ways this can be done in python, including pip install conda install and there is also the issue of doing it in the command prompt or one of the IDEs. I've downloaded python 3.8.5 and anaconda3 and some of my most burning questions right now are:
When to use which command for installing packages? (and also should I always do it in the command prompt aka cmd on windows instead of inside jupyter notebook)
How to navigate the cmd syntax/coding (for example the python documentation for installing packages has this piece of code: py -m pip install "SomeProject" but I am completely unfamiliar with this syntax and how to use it - so in the long run do I also have to learn what goes on in the command prompt or does most of the operations occur in the IDE and I mostly don't have to touch the cmd?)
How to set up a working directory of sorts (like setwd() in R) such that my .ipynb files can be saved to other directories or even better if I can just directly start my IDE from another file destination?
I've tried looking at some online resources but they mostly deal with coding basics and the python language instead of these technical aspects of the set up, so I would greatly appreciate some advice on how to navigate and set up the python working environment in general. Thanks a lot!

Python uses a different way of installing packages. Python has a thing named venv which stands for Virtual Environment. You install all of your packages in venv. Usually for each new project you make a new venv.
By using Anaconda on windows you install everything within the anaconda environment that you have specified.
python -m pip install "modulename" is a command that will install modulename to your default venv. You will be able to use this module when no other venv is specified. Here is the docs page. And here is a tutorial on how to use venv
By default python uses the same directory you have your code in. e.g. C:/Users/me/home/mypythonfile.py will run in C:/Users/me/home/ and will be able to access files in this directory. However you can use ../ to navigate directories or you can specify an absolute path to file you want to open e.g. with open("C:/system32/somesystemfile.sys") as file

Going over the technical differences of conda and pip:
So Conda is a packaging tool and installer that aims to do more than what pip does; handle library dependencies outside of the Python packages as well as the Python packages themselves. Both have many similar functionalities as well, you can install packages or create virtual environments with both.
It is generally advisable to generally have both conda and pip installed since there are some packages which might not be available with conda but with pip and vice versa.
The commands to install in both the ways is easy enough, but one thing to keep in mind is that
conda stores packages in the anaconda/pkgs directory
pip stores it in directory under /usr/local/bin/ for a Unix-based system, or \Program Files\ for Windows
You can use both pip or conda inside the jupyter notebook, it will work just fine, but it may be possible that you get multiple versions of the same package.
Most of the times, you will use cmd only to install a module used in your code, or to create environments, py -m pip install "SomeProject" here basically means that the module "SomeProject" will be downloaded in base env.

You could think of conda as python with a variety of additional functionalities, such as certain pre-installed packages and tools, such as spyder and jupyter. Hence, you must be precise when you say:
I've downloaded python 3.8.5 and anaconda3
Does it mean you installed python in your computer and then also anaconda?
In general, or at least in my opinion, using anaconda has advantages for development, but typically you'll just use a simple python installation in production (if that applies to you).
Anaconda has it's own package registry/repository . When you call conda install <package>, it will search for the package there and install it if available. You would better search it first, for instance matplotlib.
pip is a package manager for the Python Package Index. pip also ships with anaconda. Hence, in an anaconda environment you may install packages from either sources (either using pip install or conda install). For instance, pandas from PyPI and pandas from conda. There is no guarantee that packages exist in both sources. You must either search it first or simply try it.
In your first steps, I would suggest you to stick to only one dev env (either simple python or anaconda, recommend the second). Because that simplifies the question: "which python and which pip is executed in the cmd line?". That said, those commands should work as expected in any terminal, it be a simple cmd or an embedded one like in PyCharm or VS Code.
You could inspect that by running (on windows and linux at least):
which python, which pip.
Honestly, this is a question/answer that falls outside the scope of SO and for more info you would better check official websites, such as for anaconda or search for python vs anaconda blogs.

Is there a way to create a virtual environment with an earlier version of python?

I have python 3.8 installed on my pc, but i need 3.7 for a specific task. When i try to set up a virtual environment via
virtualenv -p "my/path/to/python37.exe"
it calls the installer, so i am to install py37 manually, but then it ends up with an error:
Error 0x80070666: Cannot install a product when a newer version is installed
Is there a proper way to implement such a thing?
Thanks!

Have you tried anaconda or miniconda (the lighter version of anaconda)? Having more python versions in different environments and switching between them is quite easy.

I haven't installed Anaconda. If you want to have multiple different environments with pure Python you can do it like this:
1) Install the python versions that you want with the exe installer, ie Python 2.x.x, Python 3.7.x, Python 3.8.x etc, maybe inside a common folder like C:\Python
2) Then edit the System Variables path and pinpoint to the folder for the version you want to create a virtual environment. You need two entries here, one to python folder (for the python.exe) and one to the Scripts folder (for the pip.exe)
3) Open command prompt and hit python. You ll see that it's showing the apporpriate version. Install the virtual environment with "python -m venv name_of_env_you_want"
4) If you want to create a virtual environment with a different version change the paths

how should i be using python 3.6 and python 3.7 side by side on msys2 and cmder?

I'm using msys2 for my dev environment on Windows 10. It's great, and this is the first roadblock I've come up against.
Specifically I'm trying to install some packages that won't allow me to via pip because my platform is incorrect. They require a 3.6 platform and msys2 comes with 3.7
I tried pyenv-win, but that wouldn't seem to work within cmder and it also installed some full windows installers of python 3.6.
Is there a recommended way to get another version of python installed using the msys ecosystem of command lines?

I know you want to use msys2, but you should reconsider, the majority of implementation don’t use msys2. multiple versions and multiple environments can get complicated. If you choose to go forward in a more standard way, you could use what I have written below
You can’t create a virtual env with a version of python that isn’t install in your system.
Downloaded and install the version of python you want to use, from https://www.python.org/
Create a project folder
Create a venv calling the newly installed version of python
venv is part of the standard library
c:\>python -3.x -m venv c:\path\to\myenv
it created a copy of the python executable in the newly created venv
https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/
Activate the new env
c:\> c:\path\to\myenv\Scripts\activate.bat
(venv) path\to\myenv>
Once activated you can pip install
(venv) path\to\myenv> pip install [package.name]
(venv) path\to\myenv> pip list
Any script you run from that venv will used the python exe installed in that virtual environment and use the packages you just installed there

Python: Install VirtualEnv for both Python 2 and Python 3

After moving over to Fedora (from Windows), I realized that it came with both installations of Python 2.7.5 and Python 3.6.6.
As I familiarized myself with using Python, I learned of the great utilities of virtual environments and how organized they keep everything.
However, my current dilemma is for which Python version should I do pip(2 or 3) install virtualenv virtualenvwrapper.
From my research, I understand that the virtualenvwrapper provides the ability to create a virtual environment using a specified version of Python: mkvirtualenv -p /usr/bin/python(2 or 3) {name}.
Therefore, should I only install virtualenv and virtualenvwrapper on one of the Python versions and use the aforementioned feature? Or should I install virtualenv and virtualenvwrapper on both versions of Python.
Would there be any conflicts?
Edit
More importantly, assuming that I have virtualenv and virtualenvwrapper installed for both Python 2.7.5 and Python 3.6.6, which version's command is called when I run any of the following: workon, mkvirtualenv, rmvirtualenv, etc.?

Would there be any conflicts?
Not until you mistakenly run the default system python command with a script that's using the opposite version as compared to the more specific python2 or python3 commands.
The virtualenvs do not conflict, and must be activated to be used. You can also of course have as many virtualenv's as you wish.
To avoid any problems setting up an environment, its suggested to run python2 -m virtualenv for example, rather than simply virtualenv command itself
For the commands listed at the bottom of the question, it depends on how your PATH is configured. Personally, I use pyenv rather than virtualenv directly, which injects itself into the OS PATH variable

What is the difference between pyenv, virtualenv, anaconda?

I am a ruby programmer trying to learn python. I am pretty familiar with pyenv since it is like a copy and paste from rbenv. Pyenv helps allow to have more than one version of python in a system and also to isolate the python without touching sensitive parts of system.
I suppose every python installation comes with pip package. What I still don't understand is, there are many good python libs out there that suggest to use this virtualenv and anaconda. I can even find a virtualenv plugin for pyenv.
Now I am getting confused with the purpose of these two pyenv and virtualenv.
worse inside pyenv there is a virtualenv plugin.
My questions are:
what is the difference between pyenv and virtualenv?
Is there any difference in using pip command inside both pyenv and virtualenv?
what does this pyenv virutalenv do?
Your explanation with example will be highly appreciated.

Edit: It's worth mentioning pip here as well, as conda and pip have similarities and differences that are relevant to this topic.
pip: the Python Package Manager.
You might think of pip as the python equivalent of the ruby gem command
pip is not included with python by default.
You may install Python using homebrew, which will install pip automatically: brew install python
The final version of OSX did not include pip by default. To add pip to your mac system's version of python, you can sudo easy_install pip
You can find and publish python packages using PyPI: The Python Package Index
The requirements.txt file is comparable to the ruby gemfile
To create a requirements text file, pip freeze > requirements.txt
Note, at this point, we have python installed on our system, and we have created a requirements.txt file that outlines all of the python packages that have been installed on your system.
pyenv: Python Version Manager
From the docs: pyenv lets you easily switch between multiple versions of Python. It's simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well. This project was forked from rbenv and ruby-build, and modified for Python.
Many folks hesitate to use python3.
If you need to use different versions of python, pyenv lets you manage this easily.
virtualenv: Python Environment Manager.
From the docs: The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python2.7/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.
To create a virtualenv, simply invoke virtualenv ENV, where ENV is is a directory to place the new virtual environment.
To initialize the virtualenv, you need to source ENV/bin/activate. To stop using, simply call deactivate.
Once you activate the virtualenv, you might install all of a workspace's package requirements by running pip install -r against the project's requirements.txt file.
Anaconda: Package Manager + Environment Manager + Additional Scientific Libraries.
**Anaconda is a commercial distribution of Python with the most popular python libraries, you are not permitted to use Anaconda in an organisation with more than 200 employees.
From the docs: Anaconda 4.2.0 includes an easy installation of Python (2.7.12, 3.4.5, and/or 3.5.2) and updates of over 100 pre-built and tested scientific and analytic Python packages that include NumPy, Pandas, SciPy, Matplotlib, and IPython, with over 620 more packages available via a simple conda install <packagename>
As a web developer, I haven't used Anaconda. It's ~3GB including all the packages.
There is a slimmed down miniconda version, which seems like it could be a more simple option than using pip + virtualenv, although I don't have experience using it personally.
While conda allows you to install packages, these packages are separate than PyPI packages, so you may still need to use pip additionally depending on the types of packages you need to install.
See also:
conda vs pip vs virtualenv (section in documentation from anaconda)
the difference between pip and conda (stackoverflow)
the relationship between virtualenv and pyenv (stackoverflow)

Simple analogy:
pyenv ~ rbenv
pip ~ bundler
virtual env ~ gemset in rvm. This can be managed by bundler directly without gemset.
Since I use python3 I prefer the python3 built-in virtual environment named venv. venv is simple and easy to use. I would recommend you to read its official docs. The doc is short and concise.
In ruby, we don't really need a virtual environment because the bundler takes care of it. Both virtual env and bundler are great, however, they have different solutions to solve the same problem.

Simple explanation: https://docs.conda.io/projects/conda/en/latest/commands.html#conda-vs-pip-vs-virtualenv-commands
If you have used pip and virtualenv in the past, you can use conda to perform all of the same operations.
Pip is a package manager
virtualenv is an environment manager
Conda is both