Develop Reference

Installing packages in python and setting up the working environment

I've been coding with R for quite a while but I want to start learning and using python more for its machine learning applications. However, I'm quite confused as to how to properly install packages and set up the whole working environment. Unlike R where I suppose most people just use RStudio and directly install packages with install.packages(), there seems to be a variety of ways this can be done in python, including pip install conda install and there is also the issue of doing it in the command prompt or one of the IDEs. I've downloaded python 3.8.5 and anaconda3 and some of my most burning questions right now are:
When to use which command for installing packages? (and also should I always do it in the command prompt aka cmd on windows instead of inside jupyter notebook)
How to navigate the cmd syntax/coding (for example the python documentation for installing packages has this piece of code: py -m pip install "SomeProject" but I am completely unfamiliar with this syntax and how to use it - so in the long run do I also have to learn what goes on in the command prompt or does most of the operations occur in the IDE and I mostly don't have to touch the cmd?)
How to set up a working directory of sorts (like setwd() in R) such that my .ipynb files can be saved to other directories or even better if I can just directly start my IDE from another file destination?
I've tried looking at some online resources but they mostly deal with coding basics and the python language instead of these technical aspects of the set up, so I would greatly appreciate some advice on how to navigate and set up the python working environment in general. Thanks a lot!

Python uses a different way of installing packages. Python has a thing named venv which stands for Virtual Environment. You install all of your packages in venv. Usually for each new project you make a new venv.
By using Anaconda on windows you install everything within the anaconda environment that you have specified.
python -m pip install "modulename" is a command that will install modulename to your default venv. You will be able to use this module when no other venv is specified. Here is the docs page. And here is a tutorial on how to use venv
By default python uses the same directory you have your code in. e.g. C:/Users/me/home/mypythonfile.py will run in C:/Users/me/home/ and will be able to access files in this directory. However you can use ../ to navigate directories or you can specify an absolute path to file you want to open e.g. with open("C:/system32/somesystemfile.sys") as file

Going over the technical differences of conda and pip:
So Conda is a packaging tool and installer that aims to do more than what pip does; handle library dependencies outside of the Python packages as well as the Python packages themselves. Both have many similar functionalities as well, you can install packages or create virtual environments with both.
It is generally advisable to generally have both conda and pip installed since there are some packages which might not be available with conda but with pip and vice versa.
The commands to install in both the ways is easy enough, but one thing to keep in mind is that
conda stores packages in the anaconda/pkgs directory
pip stores it in directory under /usr/local/bin/ for a Unix-based system, or \Program Files\ for Windows
You can use both pip or conda inside the jupyter notebook, it will work just fine, but it may be possible that you get multiple versions of the same package.
Most of the times, you will use cmd only to install a module used in your code, or to create environments, py -m pip install "SomeProject" here basically means that the module "SomeProject" will be downloaded in base env.

You could think of conda as python with a variety of additional functionalities, such as certain pre-installed packages and tools, such as spyder and jupyter. Hence, you must be precise when you say:
I've downloaded python 3.8.5 and anaconda3
Does it mean you installed python in your computer and then also anaconda?
In general, or at least in my opinion, using anaconda has advantages for development, but typically you'll just use a simple python installation in production (if that applies to you).
Anaconda has it's own package registry/repository . When you call conda install <package>, it will search for the package there and install it if available. You would better search it first, for instance matplotlib.
pip is a package manager for the Python Package Index. pip also ships with anaconda. Hence, in an anaconda environment you may install packages from either sources (either using pip install or conda install). For instance, pandas from PyPI and pandas from conda. There is no guarantee that packages exist in both sources. You must either search it first or simply try it.
In your first steps, I would suggest you to stick to only one dev env (either simple python or anaconda, recommend the second). Because that simplifies the question: "which python and which pip is executed in the cmd line?". That said, those commands should work as expected in any terminal, it be a simple cmd or an embedded one like in PyCharm or VS Code.
You could inspect that by running (on windows and linux at least):
which python, which pip.
Honestly, this is a question/answer that falls outside the scope of SO and for more info you would better check official websites, such as for anaconda or search for python vs anaconda blogs.

corrupted python virtual environment after update

this winter I've been working on a Flask Application following this tutorial.
Today, hence after 3 months, I decided to work again on it, but all my set up seems corrupted.
The application started simply activating the virtualenv and calling flask run.
As for today, the command returns:
No module named 'flask'
and so for pip, pip3 etc. even if all these modules are in venv/bin.
My only guess is that since then, I updated python systemwide to Python 3.8.3rc1, which somehow became the default python in the venv as well, even if I was working in python 3.7. If that is the case, I wouldn't know how to fix this problem.
Do you have any suggestion?
thanks

When you created your virtual environment (let's call it v), two things happened:
v/bin/python was a link to your system Python 3.7
v/lib/python3.7 was created for packages installed to the virtual environment.
When you updated, the v link remained the same, but now it points to Python 3.8, which will look for a library directory named python3.8. Hence, your old Flask installation is invisible to the new Python.
Ideally, you wouldn't use your system Python for anything; install your own Python (under /usr/local/ or something), so that instead of upgrading to a new version of PYthon, you can simply install a new version along side it. Then your virtual environment can continue using whatever version of Python it was created with.
That said, you should probably just recreate your virtual environment from scratch.

Try to update the Pip
->> pip install --upgrade pip
and then install the falsk again
->>pip install flask
and lemme know if it works

Is there a way to create a virtual environment with an earlier version of python?

I have python 3.8 installed on my pc, but i need 3.7 for a specific task. When i try to set up a virtual environment via
virtualenv -p "my/path/to/python37.exe"
it calls the installer, so i am to install py37 manually, but then it ends up with an error:
Error 0x80070666: Cannot install a product when a newer version is installed
Is there a proper way to implement such a thing?
Thanks!

Have you tried anaconda or miniconda (the lighter version of anaconda)? Having more python versions in different environments and switching between them is quite easy.

I haven't installed Anaconda. If you want to have multiple different environments with pure Python you can do it like this:
1) Install the python versions that you want with the exe installer, ie Python 2.x.x, Python 3.7.x, Python 3.8.x etc, maybe inside a common folder like C:\Python
2) Then edit the System Variables path and pinpoint to the folder for the version you want to create a virtual environment. You need two entries here, one to python folder (for the python.exe) and one to the Scripts folder (for the pip.exe)
3) Open command prompt and hit python. You ll see that it's showing the apporpriate version. Install the virtual environment with "python -m venv name_of_env_you_want"
4) If you want to create a virtual environment with a different version change the paths

Am I using virtualenv wrong or is this a limitation of it?

So I used virtualenv to define environments for a number of projects I am working on. I defined the virtualenv python as being version 3.4. Eventually, my global python was upgraded from 3.4.0 to 3.4.3. This proved to be a problem because the virtualenv was dependent on the global binaries (the contents of /lib/python3.4 in my virtualenv is actually just links to the global binaries), and these aren't defined up to their minor versions. In other words, when the upgrade was done, the contents of the binary folder /usr/lib/python3.4 was replaced. This is because python doesn't install things separately in 3.4.0 and 3.4.3 but only into a single folder named /usr/lib/python3.4. Since the python executable in my virtualenv was 3.4.0, there were obviously compatibility issues with the 3.4.3 binaries (it would fail to load ctypes which prevented just about anything python dependent to run). The only fix to this I've found is to downgrade my global python installation, but this feels "dirty". What if I had one project running 3.4.0 and another running 3.4.3 ? Is there no way to make them work in parallel on the same machine given that only one binary folder can exist for any 3.4.x installation ?
I'm trying to understand if I'm missing something obvious here or if this is a common problem with virtualenv, given that I've heard quite a few people complain about issues with binares when using virtualenv.
In the future, is there anyway of telling virtualenvwrapper to copy the binaries rather than link to them ?

Virtualenvs were not desiged to be portable, both across machines or across Python versions.
This means upgrading Python versions sometimes breaks virtualenvs. You need to recreate them and reinstall everything inside of it (run this in your virtualenv root):
# Save a list of what you had installed
pip freeze > freeze.txt
# Trash the entire virtualenv
deactivate
rm -rf lib/ bin/ share/ man/ include/ .Python pip-selfcheck.json
# Create it anew
virtualenv .
# Install all libraries you had before
pip install -r freeze.txt

Python Anaconda package built and installed, but cannot be imported

I built a python3 package called gender_univ using the Anaconda conda build command. I uploaded the package to the Anaconda cloud and then installed it into my conda environment. Though the package shows up in the list of installed packages when I type conda list, whenever I try to access the package using import gender_univ I get the error no module named gender_univ.
I want to understand why I can't seem to import a package that is apparently installed in my conda virtual environment? Any suggestions.

First, I would check that you are using the correct python (i.e. which python and confirm that it is the python in your conda environment). Next, you can check if your package is in the site-packages directory of that same python.
The most likely thing, I'd bet, is that the conda package doesn't include everything correctly. Are you sure that you have a build.sh (or bld.bat if you are on windows) and a setup.py? Did you try expanding your built conda package and looking for your python classes in there?
If you expand your built conda package, probably something like gender_univ-VERSION-py35_0.tar.bz2, you should see a lib/python3.5/site-packages/gender_univ directory (i.e. python package). Do you? If not, then the failure is with your building.