Develop Reference

Using command line in Jupyter Notebook not executing commands in virtual environment

I need to use the command line in Jupyter Notebook in a virtual environment, but the commands don't execute in the environment itself. I'm using Anaconda Navigator to open Jupyter Notebook. For background, I use a Mac.
Whenever I do something like ! pip install xyz, it installs the package in the Anaconda directory for libraries (see the image). I made sure that the kernel is in the environment, so that's not the problem.
What's interesting is that if I try to import a package, I can only import packages from the environment itself, not the Anaconda libraries. In other words, installing packages doesn't occur in the environment, but importing packages does.
Image that shows the pip install
This wouldn't matter that much because I could download packages in the Terminal itself, but I also need to do things like in the image below in my virtual environment, which is not possible to do in the Terminal (or is it?).
Example of what I need to do
Any ideas on how to fix this?
EDIT: That error is fixed. However, I'm getting an error if I try to use a Python command: image of error. The first line just defines the file path for the script (the last item in the list is a .py file). I tried doing "%%python" followed by the command, but it doesn't work. Any ideas?

Installing packages in python and setting up the working environment

I've been coding with R for quite a while but I want to start learning and using python more for its machine learning applications. However, I'm quite confused as to how to properly install packages and set up the whole working environment. Unlike R where I suppose most people just use RStudio and directly install packages with install.packages(), there seems to be a variety of ways this can be done in python, including pip install conda install and there is also the issue of doing it in the command prompt or one of the IDEs. I've downloaded python 3.8.5 and anaconda3 and some of my most burning questions right now are:
When to use which command for installing packages? (and also should I always do it in the command prompt aka cmd on windows instead of inside jupyter notebook)
How to navigate the cmd syntax/coding (for example the python documentation for installing packages has this piece of code: py -m pip install "SomeProject" but I am completely unfamiliar with this syntax and how to use it - so in the long run do I also have to learn what goes on in the command prompt or does most of the operations occur in the IDE and I mostly don't have to touch the cmd?)
How to set up a working directory of sorts (like setwd() in R) such that my .ipynb files can be saved to other directories or even better if I can just directly start my IDE from another file destination?
I've tried looking at some online resources but they mostly deal with coding basics and the python language instead of these technical aspects of the set up, so I would greatly appreciate some advice on how to navigate and set up the python working environment in general. Thanks a lot!

Python uses a different way of installing packages. Python has a thing named venv which stands for Virtual Environment. You install all of your packages in venv. Usually for each new project you make a new venv.
By using Anaconda on windows you install everything within the anaconda environment that you have specified.
python -m pip install "modulename" is a command that will install modulename to your default venv. You will be able to use this module when no other venv is specified. Here is the docs page. And here is a tutorial on how to use venv
By default python uses the same directory you have your code in. e.g. C:/Users/me/home/mypythonfile.py will run in C:/Users/me/home/ and will be able to access files in this directory. However you can use ../ to navigate directories or you can specify an absolute path to file you want to open e.g. with open("C:/system32/somesystemfile.sys") as file

Going over the technical differences of conda and pip:
So Conda is a packaging tool and installer that aims to do more than what pip does; handle library dependencies outside of the Python packages as well as the Python packages themselves. Both have many similar functionalities as well, you can install packages or create virtual environments with both.
It is generally advisable to generally have both conda and pip installed since there are some packages which might not be available with conda but with pip and vice versa.
The commands to install in both the ways is easy enough, but one thing to keep in mind is that
conda stores packages in the anaconda/pkgs directory
pip stores it in directory under /usr/local/bin/ for a Unix-based system, or \Program Files\ for Windows
You can use both pip or conda inside the jupyter notebook, it will work just fine, but it may be possible that you get multiple versions of the same package.
Most of the times, you will use cmd only to install a module used in your code, or to create environments, py -m pip install "SomeProject" here basically means that the module "SomeProject" will be downloaded in base env.

You could think of conda as python with a variety of additional functionalities, such as certain pre-installed packages and tools, such as spyder and jupyter. Hence, you must be precise when you say:
I've downloaded python 3.8.5 and anaconda3
Does it mean you installed python in your computer and then also anaconda?
In general, or at least in my opinion, using anaconda has advantages for development, but typically you'll just use a simple python installation in production (if that applies to you).
Anaconda has it's own package registry/repository . When you call conda install <package>, it will search for the package there and install it if available. You would better search it first, for instance matplotlib.
pip is a package manager for the Python Package Index. pip also ships with anaconda. Hence, in an anaconda environment you may install packages from either sources (either using pip install or conda install). For instance, pandas from PyPI and pandas from conda. There is no guarantee that packages exist in both sources. You must either search it first or simply try it.
In your first steps, I would suggest you to stick to only one dev env (either simple python or anaconda, recommend the second). Because that simplifies the question: "which python and which pip is executed in the cmd line?". That said, those commands should work as expected in any terminal, it be a simple cmd or an embedded one like in PyCharm or VS Code.
You could inspect that by running (on windows and linux at least):
which python, which pip.
Honestly, this is a question/answer that falls outside the scope of SO and for more info you would better check official websites, such as for anaconda or search for python vs anaconda blogs.

Embedded python interpreter won't use virtual conda environment

I am trying to build a C++ program that embeds the python interpreter using pybind11. I am using Visual Studio 2019 on Windows with a C++ project (not CMake).
I want to use the packages that are inside a virtual conda environment (Anaconda3/envs/pytorchenv) rather than the root environment. The problem that I can't seem to get past is that the program uses the python installation in the Anaconda root environment no matter what I do.
Even if I "conda activate" the virtual environment in an Anaconda prompt before launching the program - it still uses the root environment. (I'm not sure if "conda activate" is supposed to modify PATH, but when I check my system environment variables, it doesn't change them).
I'm guessing there is something I can set in my Visual Studio project settings to indicate that I want pybind11 to use the virtual environment instead of the conda base path. Any help would be greatly appreciated. Thanks!

How can I use both Anaconda versions (2.7 & 3.5)?

I was using the Anaconda 3.5 distro in a Windows 10 machine. Due to dependencies in libraries that I want to work with, I had to have the 2.7 version installed as well.
The good news is that the libraries I needed can now work with the 2.7 version smoothly and Visual Studio 2015 automagically detected my new Python environment.
The problem comes when using the command line. Upon issuing the command
conda info --envs
I get
root * C:\Users\norah\Anaconda2
i.e. a single environment (to my understanding and search so far, according to this I should see two envs listed). This means I can't use conda to acquire new packages for my Python3.5 installation, at least not at the command line as I used to since conda only refers to Python2.7. The same goes for the GUI version, Anaconda navigator (I'm not very fond of the GUI version but I tried it out).
There's also no way of launching python3 from the command line since
$python
always fires up python2.7 and issuing $python3 or $python3.5 in the command line doesn't seem to work (nor would adding the path of python3 to the system since the actual executable has the same name as python2 i.e. python.exe)
Is my system taken over by Python2.7? Is anyone here using them both smoothly and if so could you please elaborate on how to do that? Was it a "no no" move to install both versions of Anaconda?

You can make Python 3.5 environment with your Anaconda 2.7:
conda create -n py35 python=3.5
Now, activate it:
activate py35
Finally you can install the desired packages:
conda install numpy
or, the whole anaconda:
conda install anaconda
The advantage of this approach is that you can also create Python 3.4 or 3.6 environments. Furthermore, you can create environments with different combinations and versions of libraries.
Actually, it makes sense to create a new environment for each larger project.

I also despise the virtual environment switch that Anaconda tries to force on us. I prefer to have both executables always instantly available from the command line. I'm pretty sure I had this working on a Windows machine once:
Install Anaconda2 and Anaconda3 to the C:\ drive as "C:\Anaconda2\" and "C:\Anaconda3\" respectively.
Edit your "Path" environment variable (Control Panel -> System and Security -> System -> Advanced system settings -> Environment Variables) and make sure that "C:\Anaconda2;C:\Anaconda2\Scripts;C:\Anaconda2\Library\bin" is in front of "C:\Anaconda3;C:\Anaconda3\Scripts;C:\Anaconda3\Library\bin".
Copy and rename the file "C:\Anaconda3\python.exe" to "C:\Anaconda3\python3.exe".
Copy and rename the file "C:\Anaconda3\Scripts\conda.exe" to "C:\Anaconda3\Scripts\conda3.exe"
Copy and rename any other scripts you might use in "C:\Anaconda3\Scripts\", such as "pip.exe" to "pip3.exe", etc.
Now, when you type "python" or "conda" at the command line you will get the python2 version, and when you type "python3" or "conda3", etc. at the command line you will get the python3 version.

The best way to use both Python 2.7x and Python 3.5x together is Jupyter Notebook.
http://jupyter.org/
You will be able to work on your browser with IPython Notebook style interface which is great for working with scripting languages.
I found some of these videos on YouTube very informative:
1) https://www.youtube.com/watch?v=HW29067qVWk
2) https://www.youtube.com/watch?v=e9cSF3eVQv0
Besides Python 2.7 and 3.5, there are a bunch of other languages that you will be able run with your Jupyter Notebook. The various Kernels are available in this link below:
https://github.com/jupyter/jupyter/wiki/Jupyter-kernels
After installation, as you start your Jupyter Notebook, your browser will open up a new page showing your Home Directory. Among the 4 tabs (Files|Running|Clusters|Conda):
1) The Files tab shows all the files in your Home Directory.
2) Running tab shows all the Notebooks that are running.
3) Clusters tab is provided by IPython parallel.
4) Conda tab is where you need to add the Python version 3.5 (and other languages if needed) to your Jupyter Notebook (If Python 2.7 is default).
If you are interested to try C++ with your Jupyter Notebook, there are a couple of Kernels available.
1) JupyRoot - https://github.com/root-mirror/root/tree/master/bindings/pyroot/JupyROOT
2) Cling - https://github.com/root-mirror/cling

Anaconda3 2.4 with python 3.5 installation error (procedure entry not found; Windows 10)

I have just made up my mind to change from python 2.7 to python 3.5 and therefore tried to reinstall Anaconda (64 bit) with the 3.5 environment. When I try to install the package I get several errors in the form of (translation from German, so maybe not exact):
The procedure entry "__telemetry_main_return_trigger" could not be found in the DLL "C:\Anaconda3\pythonw.exe".
and
The procedure entry "__telemetry_main_invoke_trigger" could not be found in the DLL "C:\Anaconda3\python35.dll".
The title of the second error message box still points to pythonw.exe. Both errors appear several times - every time an extraction was completed. The installation progress box reads
[...]
extraction complete.
Execute: "C:\Anaconda3\pythonw.exe" "C:\Anaconda3\Lib_nsis.py" postpkg
After torturing myself through the installation I get the warning
Failed to create Anaconda menus
If I ignore it once gives me my lovely error messages and tells me that
Failed to initialize Anaconda directories
then
Failed to add Anaconda to the system PATH
Of course nothing works, if I dare to use this mess it installs. What might go wrong? On other computers with Windows 10 it works well.
P.S.: An installation of Anaconda2 2.4 with python 2.7 works without any error message, but still is not able to be used (other errors).

Finally I have found the reason. So, if anybody else has this problem:
Here the entry points are an issue as well and Michael Sarahan gives the solution. Install the Visual C++ Redistributable for Visual Studio 2015, which is used by the new version of python, first. After that install the Anaconda-package and it should work like a charm.

You can also use your standard Anaconda installation and just create an environment based on 2.7 / 3.4 etc... Anaconda will download and install all compatible packages from the repository so you have a complete installation.
conda create -n py27 python=2.7 anaconda
conda create -n py34 python=3.4 anaconda
To use the new environment (in Windows), you can point your IDE to the required environment folder:
C:\Anaconda3\envs\py34\python.exe
On the Windows command line, you just need to type:
activate py34
Documentation:
http://conda.pydata.org/docs/py2or3.html
My recommendation would be to reinstall your Anaconda version with Python 2.7 that you previously had working, then just add a new environment for Python 3.x and use that as you need.

I had the same problem, then I Installed "for all users": solved.
Much easier than the links provided for a beginner as I am.

For the problem "Failed to create Anaconda menus", it may because:
This might happen if PATH is too long already. How long is your PATH environment variable? Windows limits this to 1024 characters. There are some workarounds go shorten it here: How do you avoid over-populating the PATH Environment Variable in Windows?
This is other people's question and solution
https://groups.google.com/a/continuum.io/forum/#!topic/anaconda/mjrbE6rKyi8
When I shorted the environment path, it works!

If you are getting errors like:
Failed to create Anaconda menus
Failed to initialize Anaconda directories
Failed to add Anaconda to the system PATH
just ignore them while installation and when installation is done look for the directory "anaconda3" is installed and correct the path accordingly in environment variables path.
In my system, path was set "C:\Anaconda3" but actually it was installed at "C:\ProgramData\Anaconda3". You have to change all 3 path entries for anaconda3 and then try to run "jupyter notebook" in CMD.

If you are using windows, launch the command prompt as administrator and execute the following commands
"C:\ProgramData\Anaconda3\pythonw.exe" -E -s "C:\ProgramData\Anaconda3\Lib_nsis.py" addpath
"C:\ProgramData\Anaconda3\pythonw.exe" -E -s "C:\ProgramData\Anaconda3\Lib_nsis.py" mkdirs
"C:\ProgramData\Anaconda3\pythonw.exe" -E -s "C:\ProgramData\Anaconda3\Lib_nsis.py" mkmenus
Don't forget to change the path to the path in your system. Before running this commands there will not be any Anaconda Navigator app in your start menu. After executing this commands make sure Anaconda Navigator app is available in the start menu.

Change the directory. It works
C:\miniconda3