Develop Reference

Installing packages in python and setting up the working environment

I've been coding with R for quite a while but I want to start learning and using python more for its machine learning applications. However, I'm quite confused as to how to properly install packages and set up the whole working environment. Unlike R where I suppose most people just use RStudio and directly install packages with install.packages(), there seems to be a variety of ways this can be done in python, including pip install conda install and there is also the issue of doing it in the command prompt or one of the IDEs. I've downloaded python 3.8.5 and anaconda3 and some of my most burning questions right now are:
When to use which command for installing packages? (and also should I always do it in the command prompt aka cmd on windows instead of inside jupyter notebook)
How to navigate the cmd syntax/coding (for example the python documentation for installing packages has this piece of code: py -m pip install "SomeProject" but I am completely unfamiliar with this syntax and how to use it - so in the long run do I also have to learn what goes on in the command prompt or does most of the operations occur in the IDE and I mostly don't have to touch the cmd?)
How to set up a working directory of sorts (like setwd() in R) such that my .ipynb files can be saved to other directories or even better if I can just directly start my IDE from another file destination?
I've tried looking at some online resources but they mostly deal with coding basics and the python language instead of these technical aspects of the set up, so I would greatly appreciate some advice on how to navigate and set up the python working environment in general. Thanks a lot!

Python uses a different way of installing packages. Python has a thing named venv which stands for Virtual Environment. You install all of your packages in venv. Usually for each new project you make a new venv.
By using Anaconda on windows you install everything within the anaconda environment that you have specified.
python -m pip install "modulename" is a command that will install modulename to your default venv. You will be able to use this module when no other venv is specified. Here is the docs page. And here is a tutorial on how to use venv
By default python uses the same directory you have your code in. e.g. C:/Users/me/home/mypythonfile.py will run in C:/Users/me/home/ and will be able to access files in this directory. However you can use ../ to navigate directories or you can specify an absolute path to file you want to open e.g. with open("C:/system32/somesystemfile.sys") as file

Going over the technical differences of conda and pip:
So Conda is a packaging tool and installer that aims to do more than what pip does; handle library dependencies outside of the Python packages as well as the Python packages themselves. Both have many similar functionalities as well, you can install packages or create virtual environments with both.
It is generally advisable to generally have both conda and pip installed since there are some packages which might not be available with conda but with pip and vice versa.
The commands to install in both the ways is easy enough, but one thing to keep in mind is that
conda stores packages in the anaconda/pkgs directory
pip stores it in directory under /usr/local/bin/ for a Unix-based system, or \Program Files\ for Windows
You can use both pip or conda inside the jupyter notebook, it will work just fine, but it may be possible that you get multiple versions of the same package.
Most of the times, you will use cmd only to install a module used in your code, or to create environments, py -m pip install "SomeProject" here basically means that the module "SomeProject" will be downloaded in base env.

You could think of conda as python with a variety of additional functionalities, such as certain pre-installed packages and tools, such as spyder and jupyter. Hence, you must be precise when you say:
I've downloaded python 3.8.5 and anaconda3
Does it mean you installed python in your computer and then also anaconda?
In general, or at least in my opinion, using anaconda has advantages for development, but typically you'll just use a simple python installation in production (if that applies to you).
Anaconda has it's own package registry/repository . When you call conda install <package>, it will search for the package there and install it if available. You would better search it first, for instance matplotlib.
pip is a package manager for the Python Package Index. pip also ships with anaconda. Hence, in an anaconda environment you may install packages from either sources (either using pip install or conda install). For instance, pandas from PyPI and pandas from conda. There is no guarantee that packages exist in both sources. You must either search it first or simply try it.
In your first steps, I would suggest you to stick to only one dev env (either simple python or anaconda, recommend the second). Because that simplifies the question: "which python and which pip is executed in the cmd line?". That said, those commands should work as expected in any terminal, it be a simple cmd or an embedded one like in PyCharm or VS Code.
You could inspect that by running (on windows and linux at least):
which python, which pip.
Honestly, this is a question/answer that falls outside the scope of SO and for more info you would better check official websites, such as for anaconda or search for python vs anaconda blogs.

Questions Virtualenv [duplicate]

This question already has answers here:
How can I set up a virtual environment for Python in Visual Studio Code?
(23 answers)
Closed 2 years ago.
just wanted to set up my new MacBookPro M1. As I want to organize my MB this time, i want to start using virtualenv.
So, what I've done so far:
installed brew
installed virtualenv
set up a dir, in there create my first env called sec_env
installed some packages for testing
Now I want to use my virtualenv:
I started it, source sec_env/dir/activate
AND now here we go, I want to code something in this env. So I start up my code-insiders and try to import the package i already installed....does not work ;( (EDIT1: Maybe i failed config it inside vs code?)
Do I missunderstand the use of virtualenv? I thought of it kinda like a virtual machine...So i can install package in need for one project and code it. But if i work on another, i would just switch, start up my vs-code again and keep writing on the other project.
Or is the problem just, that all the project I want to code have to be inside the dir of the virtualenv(sec_env)? At the moment , I have a dir virtualenvs where I store all my environments , start one up and change to desktop to work . And all the projects are on my desktop.
Would be awesome if someone give me any tipps on this, or another way to separate my different projects. I am super new to this topic, since I used different virtual-box images before...now i am forced to use something else...M1 :D !

Your understanding is generally correct as virtualenv are a way to keep projects' dependencies separated from each other, like a VM would.
Your code doesn't need to be in the same directory as where your virtual environment, but many people tend to organize it that way for the sake of convenience. That way you don't need to think about what venv you coded a project with since it's right there in the directory.
With your steps, I think you installed a package before activating the environment. Doing it in that order installs the package in your system site-packages, not your virtual environment packages. Before you install a package, you need to activate your environment. Also, it appears from How to tell Homebrew to install inside virtualenv? that homebrew doesn't support installing a package into a virtual env. So in order to install packages into a virtualenv, I would suggest using pip as your package manager.
So the sequence of commands would be...
source <path to virtualenv>/dir/activate
pip install <modules you want to install>
# Now you can run your code that references those installed modules.

How to set a user-local python interpreter as default on clusters

I got access to the server of my lab yesterday. I was trying to do some scientific computing on it with Numpy or so.
The problem is that the server has a SuSE linux(I never heard of such thing..) installed, and it's very difficult to install anything on it. So the administrator suggested that I should set my python environment locally since I did't have root access.
I have checked the default python interpreter on the server, it has python2.6 and python3.x. But I wish to use python2.7.x.
I followed some instructions to install python2.7 into /home/kevin/.local with something like this:
./configure prefix='/home/kevin/.local'
It works, but not easy, because I have to type
/home/kevin/.local/bin/python
everytime.
So, here is my question: what should I do to set the python2.7 I have installed locally as my default python evironment, so I can use
python test.py
to run my script. And when I use pip, it will install modules locally automatically.
ps: I have went through many of the threads available out there, but they only covers how to install modules locally. So I don't think they meet my purpose. And I also tried virtualenv. As I mentioned before, the default python version is 2.6 in usr/bin/python, so I can only get a python2.6 virtual environment locally, which is not what I want.
Since English is my second language, I'm not sure whether I have got everything clear. If not, please point out. It would be very kind of you to do so. And thanks for help!

If you already have python2.7 and virtualenv installed locally, you already have everything you need.
The piece of information you're missing is that you can specify which version of python you want to use when you create your virtualenv. You can use the -p option for that.
virtualenv -p /home/kevin/.local/bin/python my_virtual_env
Once you activate it, your python command will refer to the proper installation. Also, if you install via pip, your packages will be installed in /home/kevin/.local/
Virtualenv's documentation can help you better understand if you have more issues.

How do you setup virtualenv on linux for both python 2.7 and python 3.2?

I did reference this other SO question to see if it answered my question, and it really didn't. I am referring to this question. Aptana Python stdlib issue with virtualenv . I have some projects that use python 3.x and some that might prefer python 2.7.x. This might be another topic but while I'm here, I was wondering if django and flask require python 2.7.x. If not, then I don't really have any projects requiring python 2.7.x, only python 3.x.y. Anyway, on Ubuntu, when trying to install pip and virtualenv it was using python 2.7.x and I was trying to create a virtual environment which would use python 3.x.y. Strangely, this was easier to accomplish on Windows - so many things are so much harder on Windows than Linux.
The problem that exists on both platforms is that I cannot figure out how to create virtual environments based on the python 2.7 and the python 3.x versions separately. I can setup my path variable on Windows so that python3.3.3 is the first version of python it finds and so that is essentially the default. I would have to go into my path environment variable and put python27 in the path before python33, then when I installed virtualenv I would have a python 2.7 environment. That doesn't seem like the best way to do things, nor would it persist. Similarly on Ubuntu, I'd have to modify the path variable and then source the file to switch between different python installs - 2.7 or 3.x.
Can someone tell me a better way to do this? Perhaps a reference to instructions for how to create two different virtual environments for the different python versions?
Also, confusing about the first reference link above, is how does eclipse/aptana know to activate that virtual env?
Thanks,
Bruce

First you install python-virtualenv. Then create a virtualenv using the virtualenv command. Read the docs for more info .

Best way to install python packages locally for development

Being new to the python games I seem to have missed out on some knowledge on how you can develop on a program but also keep it in your live environment.
Programs like gpodder can be run directly from the source checkout which is really handy however others want to be "installed" to run.
A lot of programs are distributed with a setup.py with instructions to run "python ./setup.py install" as root which will put stuff somewhere in your file-system. There are even install commands like "develop" which seem to hold the promise of what I want. So I tried:
export PYTHONPATH=/home/alex/python
python ./setup.py develop --install-dir=/home/alex/python
Which downloaded a bunch of stuff locally and seems magically ensure the application I'm hacking on is still being run out of the src tree. So I guess my roundabout question is is this the correct way of developing python code? How do things like easy_install and pip factor into this?
So I tried the following:
python /usr/share/pyshared/virtualenv.py /home/alex/src/goobook
cd /home/alex/src/goobook/googbook.git
/home/alex/src/goobook/bin/python ./setup.py develop
And finally linked the program in question to my ~/bin
cd /home/alex/src/goobook
linkbin.pl bin/goobook
However invocation throws up a load of extra chatter which seems to imply it's wrong:
17:17 alex#socrates/i686 [goobook] >goobook --help
/home/alex/bin/goobook:5: UserWarning: Module pkg_resources was already imported from /home/alex/src/goobook/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg/pkg_resources.py, but /home/alex/src/goobook/lib/python2.5/site-packages/distribute-0.6.10-py2.5.egg is being added to sys.path
from pkg_resources import load_entry_point
/home/alex/bin/goobook:5: UserWarning: Module site was already imported from /home/alex/src/goobook/lib/python2.5/site.pyc, but /home/alex/src/goobook/lib/python2.5/site-packages/distribute-0.6.10-py2.5.egg is being added to sys.path
from pkg_resources import load_entry_point

Install:
http://pypi.python.org/pypi/virtualenv
to set up a localized virtual environment for your libraries, and:
http://pypi.python.org/pypi/setuptools
i.e. "easy_install" to install new things.

Virtualenv allows you to work in completely independent and isolated Python environments. It will let you easily create multiple environments which have different Python packages installed or different versions of a same package. Virtualenv also lets you easily switch between your different environments.
As of 2012, the de facto preferred tool for package management in Python is pip rather than setuptools. Pip is able to handle dependencies and to install/uninstall globally or inside a virtual environment. Pip even comes out-of-the-box with virtualenv.
Python 3
Also worth mentioning is the fact that virtual environments are becoming a part of Python itself in release 3.3, with the implementation of PEP 405.

The Python Packaging User Guide, which "aims to be the authoritative resource on how to package, publish and install Python distributions using current tools", recommends using pip to install in "development mode":
pip install -e <path>
Thus in the root directory of your package you can simply
pip install -e .
See installing from a local source tree.

The best way to develop Python apps with dependencies is to:
Download desired version of the python interpreter.
Install and use buildout (http://www.buildout.org/).
Buildout is something like Maven for Java (will fetch all needed packages automatically).
This way your Python interpreter will not be polluted by third party packages (this is important if you will be running developed application on other machines). Additionally you can integrate buildout with virtualenv package (this allows you to create virtual python interpreters for each project).