I have a python installation on my pc (windows 10) which comes from Anaconda. I am a data scientist and using conda as a package manager is very convenient for me.
However, sometimes I want to develop a small app or script to share with my colleagues. In these cases I create a project folder and python -m venv .venv inside it.
This way, I can install only the essential packages I need, and later share the requirements.txt file.
The issue I'm having, is that the python interpreter being used is still the default one, namely, the one that came with Anaconda, even if I activate the virtual environment and deactivate the conda one.
Specifically, if I run python in the terminal, I get this warning message:
Warning:
This Python interpreter is in a conda environment, but the environment has
not been activated. Libraries may fail to load. To activate this environment
please see https://conda.io/activation
This is rather inconvenient. My base python installation is 3.7, but if I wanted to use an earlier version, or 3.8, I can't seem to be able to choose.
I would expect that the python executable being used is the one in the current active environment, but this doesn't seem to be the case.
How can I obtain that?
First you have to install the version of python you want to use in your venv. It must already be available somewhere on your system to create a venv using it.
Then instead of just python -m venv .venv you specify which python with the full path : /path/to/pythonX.Y -m venv .venv
You can't have a venv that shares multiple versions of python, to my knowledge at least.
Related
I've been coding with R for quite a while but I want to start learning and using python more for its machine learning applications. However, I'm quite confused as to how to properly install packages and set up the whole working environment. Unlike R where I suppose most people just use RStudio and directly install packages with install.packages(), there seems to be a variety of ways this can be done in python, including pip install conda install and there is also the issue of doing it in the command prompt or one of the IDEs. I've downloaded python 3.8.5 and anaconda3 and some of my most burning questions right now are:
When to use which command for installing packages? (and also should I always do it in the command prompt aka cmd on windows instead of inside jupyter notebook)
How to navigate the cmd syntax/coding (for example the python documentation for installing packages has this piece of code: py -m pip install "SomeProject" but I am completely unfamiliar with this syntax and how to use it - so in the long run do I also have to learn what goes on in the command prompt or does most of the operations occur in the IDE and I mostly don't have to touch the cmd?)
How to set up a working directory of sorts (like setwd() in R) such that my .ipynb files can be saved to other directories or even better if I can just directly start my IDE from another file destination?
I've tried looking at some online resources but they mostly deal with coding basics and the python language instead of these technical aspects of the set up, so I would greatly appreciate some advice on how to navigate and set up the python working environment in general. Thanks a lot!
Python uses a different way of installing packages. Python has a thing named venv which stands for Virtual Environment. You install all of your packages in venv. Usually for each new project you make a new venv.
By using Anaconda on windows you install everything within the anaconda environment that you have specified.
python -m pip install "modulename" is a command that will install modulename to your default venv. You will be able to use this module when no other venv is specified. Here is the docs page. And here is a tutorial on how to use venv
By default python uses the same directory you have your code in. e.g. C:/Users/me/home/mypythonfile.py will run in C:/Users/me/home/ and will be able to access files in this directory. However you can use ../ to navigate directories or you can specify an absolute path to file you want to open e.g. with open("C:/system32/somesystemfile.sys") as file
Going over the technical differences of conda and pip:
So Conda is a packaging tool and installer that aims to do more than what pip does; handle library dependencies outside of the Python packages as well as the Python packages themselves. Both have many similar functionalities as well, you can install packages or create virtual environments with both.
It is generally advisable to generally have both conda and pip installed since there are some packages which might not be available with conda but with pip and vice versa.
The commands to install in both the ways is easy enough, but one thing to keep in mind is that
conda stores packages in the anaconda/pkgs directory
pip stores it in directory under /usr/local/bin/ for a Unix-based system, or \Program Files\ for Windows
You can use both pip or conda inside the jupyter notebook, it will work just fine, but it may be possible that you get multiple versions of the same package.
Most of the times, you will use cmd only to install a module used in your code, or to create environments, py -m pip install "SomeProject" here basically means that the module "SomeProject" will be downloaded in base env.
You could think of conda as python with a variety of additional functionalities, such as certain pre-installed packages and tools, such as spyder and jupyter. Hence, you must be precise when you say:
I've downloaded python 3.8.5 and anaconda3
Does it mean you installed python in your computer and then also anaconda?
In general, or at least in my opinion, using anaconda has advantages for development, but typically you'll just use a simple python installation in production (if that applies to you).
Anaconda has it's own package registry/repository . When you call conda install <package>, it will search for the package there and install it if available. You would better search it first, for instance matplotlib.
pip is a package manager for the Python Package Index. pip also ships with anaconda. Hence, in an anaconda environment you may install packages from either sources (either using pip install or conda install). For instance, pandas from PyPI and pandas from conda. There is no guarantee that packages exist in both sources. You must either search it first or simply try it.
In your first steps, I would suggest you to stick to only one dev env (either simple python or anaconda, recommend the second). Because that simplifies the question: "which python and which pip is executed in the cmd line?". That said, those commands should work as expected in any terminal, it be a simple cmd or an embedded one like in PyCharm or VS Code.
You could inspect that by running (on windows and linux at least):
which python, which pip.
Honestly, this is a question/answer that falls outside the scope of SO and for more info you would better check official websites, such as for anaconda or search for python vs anaconda blogs.
I need python3.6 for tensorflow installation, so I downloaded python3.6.12.tar. And I found that I should pip install tarfile. However, in this case it is an older version of python. FYI, In my computer(laptop) I installed python3.9.
My question is: can I pip install python.tar inside a virtualenv?
This is not how virtual environments work. I suggest you to do a little bit more research on virtual environments in Python.
Virtual Environments and Packages
Basically you need to install the necessary python version onto your machine. Then go ahead and use that specific python (which is version 3.6 in your case), to create a virtual environment with the command
~ /usr/bin/<path-to-python3.6> -m venv venv
This command will create a folder called venv. Now you need to source the activation script inside this folder to activate your environment.
Handy note: if you are dealing with different versions of python, a more robust way of handling such situations is via using a tool called pyenv.
I am trying to create a new python 3.7 virtual environment on my local computer running Windows 8. I have python versions 3.6, 3.7, and 3.8 installed. Their exe's are named python36, python37, and python, respectively. All three are correctly added to PATH because I can enter each interpreter.
Within my new project's directory I tried to create a virtual environment with python37 -m venv env. It produced an error: Error: [WinError 2] The system cannot find the file specified, but it still created the directory. However the Scripts subfolder is empty except for pythonw.exe.
In this question someone suggests doing python37 -m venv env --without-pip. When I tried this, the activation/deactivation scripts were created, but the virtual environment is using python 3.8.
It is my understanding that venv will create the virtual environment with what ever python exe you use to call it, so I don't understand how this can happen. I've verified that python37 points to the correct place with where python37, and can even enter the 3.7 interactive interpreter.
The problem was that I renamed the python exe's. I don't know exactly what goes wrong, but presumably at some point venv tries to find python.exe and is thrown off by the name.
Changing them back to python.exe and differentiating between the versions with their location fixed the problem.
Edit:
Check out Eryk's comments for more details.
First create folder at any drive then go to that folder and install virtualenv package using pip.
pip install virtualenv
Then create your virtual environment.
mkvirtualenv myvirtualenv
Then use below command to activate virtualenv in windows.
myvirtualenv\Scripts\activate
After this you can install related package in current virtual environment.
The Python Standard Library for Creating Virtual Environment
I have python 3.8 installed on my pc, but i need 3.7 for a specific task. When i try to set up a virtual environment via
virtualenv -p "my/path/to/python37.exe"
it calls the installer, so i am to install py37 manually, but then it ends up with an error:
Error 0x80070666: Cannot install a product when a newer version is installed
Is there a proper way to implement such a thing?
Thanks!
Have you tried anaconda or miniconda (the lighter version of anaconda)? Having more python versions in different environments and switching between them is quite easy.
I haven't installed Anaconda. If you want to have multiple different environments with pure Python you can do it like this:
1) Install the python versions that you want with the exe installer, ie Python 2.x.x, Python 3.7.x, Python 3.8.x etc, maybe inside a common folder like C:\Python
2) Then edit the System Variables path and pinpoint to the folder for the version you want to create a virtual environment. You need two entries here, one to python folder (for the python.exe) and one to the Scripts folder (for the pip.exe)
3) Open command prompt and hit python. You ll see that it's showing the apporpriate version. Install the virtual environment with "python -m venv name_of_env_you_want"
4) If you want to create a virtual environment with a different version change the paths
I recently created a conda environment (in Ubuntu 16.04), which is working great from the command line. I can activate it with no problems, the packages are properly separated, etc.
However, because it is running the same version of Python as the Anaconda installation (Python 3.6.1), it has not created a separate interpreter. Within the /anaconda3/envs folder there are no other folders. This makes it difficult when I want to use editors (such as vscode and Pycharm) because they cannot find the path to some of the packages I have installed in the environment. I would love to start using the great debugging features in Pycharm but without a separate interpreter I have no idea where to even start. Is there a way to initialize the interpreter in the correct environment?
When I run conda info --envs, this is what I get:
# conda environments:
#
chatbot /home/bradley/.conda/envs/chatbot
tensorflow /home/bradley/.conda/envs/tensorflow
tf_testing /home/bradley/.conda/envs/tf_testing
root * /home/bradley/anaconda3
If I try to run the Python in these directories (a small executable script exists), it doesn't include all the packages. Should I maybe add an environment variable? If so, what would I write?