Develop Reference

Installing packages in python and setting up the working environment

I've been coding with R for quite a while but I want to start learning and using python more for its machine learning applications. However, I'm quite confused as to how to properly install packages and set up the whole working environment. Unlike R where I suppose most people just use RStudio and directly install packages with install.packages(), there seems to be a variety of ways this can be done in python, including pip install conda install and there is also the issue of doing it in the command prompt or one of the IDEs. I've downloaded python 3.8.5 and anaconda3 and some of my most burning questions right now are:
When to use which command for installing packages? (and also should I always do it in the command prompt aka cmd on windows instead of inside jupyter notebook)
How to navigate the cmd syntax/coding (for example the python documentation for installing packages has this piece of code: py -m pip install "SomeProject" but I am completely unfamiliar with this syntax and how to use it - so in the long run do I also have to learn what goes on in the command prompt or does most of the operations occur in the IDE and I mostly don't have to touch the cmd?)
How to set up a working directory of sorts (like setwd() in R) such that my .ipynb files can be saved to other directories or even better if I can just directly start my IDE from another file destination?
I've tried looking at some online resources but they mostly deal with coding basics and the python language instead of these technical aspects of the set up, so I would greatly appreciate some advice on how to navigate and set up the python working environment in general. Thanks a lot!

Python uses a different way of installing packages. Python has a thing named venv which stands for Virtual Environment. You install all of your packages in venv. Usually for each new project you make a new venv.
By using Anaconda on windows you install everything within the anaconda environment that you have specified.
python -m pip install "modulename" is a command that will install modulename to your default venv. You will be able to use this module when no other venv is specified. Here is the docs page. And here is a tutorial on how to use venv
By default python uses the same directory you have your code in. e.g. C:/Users/me/home/mypythonfile.py will run in C:/Users/me/home/ and will be able to access files in this directory. However you can use ../ to navigate directories or you can specify an absolute path to file you want to open e.g. with open("C:/system32/somesystemfile.sys") as file

Going over the technical differences of conda and pip:
So Conda is a packaging tool and installer that aims to do more than what pip does; handle library dependencies outside of the Python packages as well as the Python packages themselves. Both have many similar functionalities as well, you can install packages or create virtual environments with both.
It is generally advisable to generally have both conda and pip installed since there are some packages which might not be available with conda but with pip and vice versa.
The commands to install in both the ways is easy enough, but one thing to keep in mind is that
conda stores packages in the anaconda/pkgs directory
pip stores it in directory under /usr/local/bin/ for a Unix-based system, or \Program Files\ for Windows
You can use both pip or conda inside the jupyter notebook, it will work just fine, but it may be possible that you get multiple versions of the same package.
Most of the times, you will use cmd only to install a module used in your code, or to create environments, py -m pip install "SomeProject" here basically means that the module "SomeProject" will be downloaded in base env.

You could think of conda as python with a variety of additional functionalities, such as certain pre-installed packages and tools, such as spyder and jupyter. Hence, you must be precise when you say:
I've downloaded python 3.8.5 and anaconda3
Does it mean you installed python in your computer and then also anaconda?
In general, or at least in my opinion, using anaconda has advantages for development, but typically you'll just use a simple python installation in production (if that applies to you).
Anaconda has it's own package registry/repository . When you call conda install <package>, it will search for the package there and install it if available. You would better search it first, for instance matplotlib.
pip is a package manager for the Python Package Index. pip also ships with anaconda. Hence, in an anaconda environment you may install packages from either sources (either using pip install or conda install). For instance, pandas from PyPI and pandas from conda. There is no guarantee that packages exist in both sources. You must either search it first or simply try it.
In your first steps, I would suggest you to stick to only one dev env (either simple python or anaconda, recommend the second). Because that simplifies the question: "which python and which pip is executed in the cmd line?". That said, those commands should work as expected in any terminal, it be a simple cmd or an embedded one like in PyCharm or VS Code.
You could inspect that by running (on windows and linux at least):
which python, which pip.
Honestly, this is a question/answer that falls outside the scope of SO and for more info you would better check official websites, such as for anaconda or search for python vs anaconda blogs.

Issue adding site-packages directory to PYTHONPATH in Spyder

I am using Spyder and trying to add /usr/local/lib/python3.7/site-packages to the PYTHONPATH Manager. However, I receive an error informing me "This directory cannot be added to PATH. If you want to set a different Python interpreter, please go to Preferences > Main Interpreter".
However, I have already changed my interpreter to point to /usr/bin/python3
At the moment, I am using the rather annoying work around of putting the following at the top of all my code.
import sys
sys.path.append("/usr/local/lib/python3.7/site-packages")
Typing the following gives me the below. Is there a way which I can even ensure after running pip3 install XXX in the terminal, that the packages are downloaded somewhere such as the below?
for p in sys.path: print(p)
/Users/user
/usr/local/lib/python3.7
/Users/user/opt/anaconda3/lib/python37.zip
/Users/user/opt/anaconda3/lib/python3.7
/Users/user/opt/anaconda3/lib/python3.7/lib-dynload
/Users/user/opt/anaconda3/lib/python3.7/site-packages
/Users/user/opt/anaconda3/lib/python3.7/site-packages/aeosa
/Users/user/opt/anaconda3/lib/python3.7/site-packages/IPython/extensions
/Users/user/.ipython
Alternatively, and preferably, advice on how to add the above site-packages directory to my PATH? I feel I am missing something obvious.

(Spyder maintainer here) We forbid adding site-packages directories through our PYTHONPATH manager because it allows people to mix two different Python versions (which is what you're trying to do by adding your system site-packages to your Anaconda's Python).
And we do that because it usually generates odd errors and segfaults for binary packages such as Numpy, Pandas and Matplotlib, given that binary packages for one Python version are incompatible with packages for another one.
Finally, even though you found a workaround for that (by using sys.path), we strongly suggest you to stop doing that because it'll give you nothing by headaches in the future.

Doing what you are asking isn't the recommended path forward but you can solve the underlying problem in either of the following ways (A or B).
To "ensure pip installs packages to another location which Spyder can see" as the asker guessed in a comment on the accepted answer which got no answer (Method B below) is usually not a good idea. Keeping a clean environment for Spyder will ensure that you can determine requirements (including package version) for each of your projects reliably. Therefore, do the reverse of what you guessed: Ensure Spyder uses the Python interpreter in the environment where pip installed your project's required packages.
A. Change the Python interpreter
Go to Tools, Preferences, and set Python interpreter to the python executable that was used to install the package (If using a virtual environment, it would be your_other_env/bin/python).
Close and reopen Spyder (Spyder says to restart the IPython console, but it may not work in this case and show the error where Spyder cannot restart a kernel it didn't start).
Open Spyder again and run any py file. You will get an error that says to install the spyder-kernels package (for some reason pip 22.0.4 will only install spyder_kernels: This issue is at "spyder-kernels should be spyder_kernels" :edit: but the issue is invalid, so upgrade pip first such as via pip install --upgrade pip in your virtual environment). Take note of the version in the error, since that is the version you need.
If you are using conda or are on Windows the instructions will differ, so see Common Illnesses in the Spyder documentation instead of continuing this step.
source your_other_env/bin/activate
pip install --upgrade pip setuptools
pip install spyder-kernels=...
deactivate
but change ... to the version shown in your Spyder error from step 3. If you installed Spyder with conda as recommended, use the commands from the URL above instead.
I don't recommend Method B, as I've explained. However, it may be useful if you are manually installing Spyder plugins or test suites that apply to all projects but aren't in the requirements.txt or setup.py requirements for your project(s) (and therefore don't affect determining requirements for your users).
B. To "ensure pip installs packages to another location which Spyder can see" you would run "spyder_env/bin/python -m pip install ..." to install the package, where spyder_env is the virtualenv where Spyder is installed (but if Spyder is installed in the system using an installer or linux distro package, you may need to use your system's python such as via python3 -m pip install --user ... where ... is the package name. Always use --user instead of sudo or root to avoid mismatched files caused by mashing together the distro-packaged modules and your manually installed modules).

I can't use pip (Windows)

I'm wanting to learn machine learning, but it requires that I use 'pip.' I'm still very new to a lot of this stuff. I've installed Python 3.4 64-bit and get-py.py.
When I run pip install sklearn I get 'pip' is not recognized as an internal or external command, operable program or batch file.
Here's a screenshot of my environmental variable PATH
https://gyazo.com/c7ec926401878845d5c4f9a556cf00ee
Any help would be greatly appreciated. I have checked out other posts where people had similar issues, but I tried various things and nothing worked.

Easiest solution is to just download a platform like Anaconda for example. It is Python with many libraries (sklearn, pandas, numpy, ...) and also with pip, all in a user-friendly Windows installer package, that takes care of everything for you.
https://www.continuum.io/downloads
If you really want to do it yourself, then
download get-pip.py (https://bootstrap.pypa.io/get-pip.py)
make sure you have Python installed and in your PATH, for example by running python --version in command prompt
Navigate to the directory, where you downloaded get-pip.py (in command prompt using the command cd).
Run the command python get-pip.py from a command prompt with admin privileges (possibly not needed, but just to make sure).
This should be all. Close your command prompt, open it again and run pip --version to test it. If it doesn't work, then search your computer for a file named pip.exe (most likely in your Python directory) and if you find it, add its path to your PATH variable (given the screenshot, I assume you know how to do this).

I downloaded Python 3.6 and the pip was installed on my system.
I found pip at below location:
C:\Users\Admin\AppData\Local\Programs\Python\Python36\Scripts
Also, before I found pip I used below command on my command prompt to install pyperclip:
python -m pip install pyperclip

For anyone that might have installed Python using the MSI installer, and ticked the box to include pip, but still gets the error 'pip' is not recognized as an internal or external command, operable program or batch file:
It looks like that the pip executable is installed in the \Scripts subdirectory of the Python installation, but that directory does not get automatically added to PATH.
For me, since I had installed Python 2.7, the directory was C:\Python27\Scripts. After adding this path to the PATH environment variable, and open a new CMD window, pip was available as a command.

I've made a workaround that will help you, copy the batch script below and modify it as specified:
#"PATH to python.exe" "PATH to pip.exe" %*
For example like this:
#"C:\programming\bin\python374\python.exe" "C:\programming\bin\python374\Scripts\pip.exe" %*
Save this file as pip.bat on Desktop and then move it to C:\Windows\system32.

For Python 3.7 or higher at least (as today is October 4th 2019) in the installation window you need to check the box "add to PATH" when you first install Python.

I'm late now, but just add python to your environment variables.
It´s explained in this short video, how to set up python to your environment variables:
https://www.youtube.com/watch?v=Y2q_b4ugPWk
after that pip should work on your cmd prompt, without cd to
"C:\Users\Admin\AppData\Local\Programs\Python\Python36\Scripts"

This is late but in case anyone runs across this issue, pip installed for me as "pip3". Once I used pip3 as my command and not pip, it worked fine. Threw me for quite a loop since I've been using Python for years.

PIP Install and Upgrade
Ensure you are accessing the right location while accessing pip as shown in my link above. Starting Python 3.4, it is included by default with the Python binary installers.
Environment Variable
It is also a good idea to update the environment variable if the path is not accessible by default, once the installation is complete.

If it is specifically pip you want you could install python again making sure to tick the install pip box which it sounds like you failed to upon initial installation.

Anaconda3 2.4 with python 3.5 installation error (procedure entry not found; Windows 10)

I have just made up my mind to change from python 2.7 to python 3.5 and therefore tried to reinstall Anaconda (64 bit) with the 3.5 environment. When I try to install the package I get several errors in the form of (translation from German, so maybe not exact):
The procedure entry "__telemetry_main_return_trigger" could not be found in the DLL "C:\Anaconda3\pythonw.exe".
and
The procedure entry "__telemetry_main_invoke_trigger" could not be found in the DLL "C:\Anaconda3\python35.dll".
The title of the second error message box still points to pythonw.exe. Both errors appear several times - every time an extraction was completed. The installation progress box reads
[...]
extraction complete.
Execute: "C:\Anaconda3\pythonw.exe" "C:\Anaconda3\Lib_nsis.py" postpkg
After torturing myself through the installation I get the warning
Failed to create Anaconda menus
If I ignore it once gives me my lovely error messages and tells me that
Failed to initialize Anaconda directories
then
Failed to add Anaconda to the system PATH
Of course nothing works, if I dare to use this mess it installs. What might go wrong? On other computers with Windows 10 it works well.
P.S.: An installation of Anaconda2 2.4 with python 2.7 works without any error message, but still is not able to be used (other errors).

Finally I have found the reason. So, if anybody else has this problem:
Here the entry points are an issue as well and Michael Sarahan gives the solution. Install the Visual C++ Redistributable for Visual Studio 2015, which is used by the new version of python, first. After that install the Anaconda-package and it should work like a charm.

You can also use your standard Anaconda installation and just create an environment based on 2.7 / 3.4 etc... Anaconda will download and install all compatible packages from the repository so you have a complete installation.
conda create -n py27 python=2.7 anaconda
conda create -n py34 python=3.4 anaconda
To use the new environment (in Windows), you can point your IDE to the required environment folder:
C:\Anaconda3\envs\py34\python.exe
On the Windows command line, you just need to type:
activate py34
Documentation:
http://conda.pydata.org/docs/py2or3.html
My recommendation would be to reinstall your Anaconda version with Python 2.7 that you previously had working, then just add a new environment for Python 3.x and use that as you need.

I had the same problem, then I Installed "for all users": solved.
Much easier than the links provided for a beginner as I am.

For the problem "Failed to create Anaconda menus", it may because:
This might happen if PATH is too long already. How long is your PATH environment variable? Windows limits this to 1024 characters. There are some workarounds go shorten it here: How do you avoid over-populating the PATH Environment Variable in Windows?
This is other people's question and solution
https://groups.google.com/a/continuum.io/forum/#!topic/anaconda/mjrbE6rKyi8
When I shorted the environment path, it works!

If you are getting errors like:
Failed to create Anaconda menus
Failed to initialize Anaconda directories
Failed to add Anaconda to the system PATH
just ignore them while installation and when installation is done look for the directory "anaconda3" is installed and correct the path accordingly in environment variables path.
In my system, path was set "C:\Anaconda3" but actually it was installed at "C:\ProgramData\Anaconda3". You have to change all 3 path entries for anaconda3 and then try to run "jupyter notebook" in CMD.

If you are using windows, launch the command prompt as administrator and execute the following commands
"C:\ProgramData\Anaconda3\pythonw.exe" -E -s "C:\ProgramData\Anaconda3\Lib_nsis.py" addpath
"C:\ProgramData\Anaconda3\pythonw.exe" -E -s "C:\ProgramData\Anaconda3\Lib_nsis.py" mkdirs
"C:\ProgramData\Anaconda3\pythonw.exe" -E -s "C:\ProgramData\Anaconda3\Lib_nsis.py" mkmenus
Don't forget to change the path to the path in your system. Before running this commands there will not be any Anaconda Navigator app in your start menu. After executing this commands make sure Anaconda Navigator app is available in the start menu.

Change the directory. It works
C:\miniconda3

Matplotlab .exe file for Python 3.4 cannot find path to Python34

I am running Windows and am a beginner python user trying to install a few modules to run a python script. I have Python 2.7.9 and 3.4.2 both installed to the C:\ directory. I downloaded matplotlib-1.4.3.win-amd64-py3.4.exe and the corresponding .exe for python 2.7 from the Matplotlib website, but when I run the py3.4 exe the program cannot find Python 3.4 to update (Cannot install: Python version 3.4 is required, which is not found in the registry).
Python 2.7 installer works perfectly. Is there a misset PATH variable in Windows I can modify so the .exe can function properly? In CMD 'Python --version' returns Python 3.4, so unsure how to fix the issue. I installed these months ago, and may have put them in Downloads before transferring both to C:\ for clarity, which may be the problem but am unsure how to fix it.
Also, if your answer involves pip in any way please clarify how exactly to use pip in Windows. A lot of websites say to run eg. 'pip setup.py install' in the 'terminal' but do not specify if they mean Windows CMD terminal, IDLE GUI, or Python.exe command-line interface. Thanks a lot!

Not the answer to your actual question, but some clarification on your last point:
but do not specify if they mean Windows CMD terminal, IDLE GUI, or
Python.exe command-line interface.
Yes, this requires to know some context that a beginner may not have. The command pip is always used in the CMD terminal. So open CMD, and enter
pip3 install matplotlib
Notes:
Use pip3 when installing for Python 3. Then you're certain you're not accidentally installing libraries for Python 2.
pip setup.py install does not exist. You're mixing up two mechanisms to install Python packages/libraries:
One uses pip, with aforementioned pip3 install <something>. Pip goes looking online, finds a corresponding package name in a database, retrieves the URL for that package, downloads the package and installs the package. All in one command.
python3 setup.py install (again explicitly use python3 or python2 to be sure) requires you to find the package, download it, unzip it, and then in the CMD terminal, inside the unzipped folder, run the python3 setup.py install command.
This second method is usually for the latest-greatest version of a package that is not yet in pip's database, or for packages that never were in pip's database in the first place.
Generally, as a beginner, you want to stick with pip. If you ever run into the issue with the package not being available via pip, you may still be able to use pip for downloading and installing, like for example so:
pip install https://github.com/matplotlib/matplotlib/archive/master.zip
which would install the most recent matplotlib (which won't have even a version number yet, so bugs could be around).
All of these commands happen in the CMD terminal: downloading/installing packages generally all go through the terminal.
Also, when people mention "terminal", they will mean (for Windows) something like the CMD terminal. When it has to be done inside Python, it is generally called the "Python prompt". (IDLE is yet a different beast, that I'm not familiar with. I'm guessing that it has several parts, including a text editor section and a Python prompt section.)