I'm trying to follow this BERT tutorial.
One of the packages is tensorflow-text.
When downloading packages, I usually use the Anaconda Navigator(ver 2.0.4), which I think may prevent conflicts (I'm an amateur, so the prob of being wrong is high).
However, even after updating the index of available python packages,Anaconda does not list the tensorflow-text package as available for download.
Given this situation, how should I proceed? What's a good way to install that package?
You can just use pip in the console of the chosen environment.
I am trying to install OSMNX module in Pycharm (using Python 3.7.2).
I tried installing it using pip install osmnx but got the following error[![error][1]][1]
i have also tried using .whl files from [https://www.lfd.uci.edu/~gohlke/pythonlibs/#fiona][2] but I cannot identify how/what steps to follow. Please provide some clear steps!
Most of the other question are answered w.r.t. conda environment. I have to use Pcharm only.
Input in any form is highly appreciated!
[1]: https://i.stack.imgur.com/RdJDN.png
[2]: https://www.lfd.uci.edu/~gohlke/pythonlibs/#fiona
You said:
I have to use Pcharm only.
Does that mean you cannot use conda + pycharm on your system for some reason? If you can, then:
Install OSMnx with conda
Use Conda environment in pycharm
This is by far the easiest (and recommended) solution.
If you cannot, then you must manually install the dependencies. This is a nontrivial process, especially if you're on Windows. OSMnx itself is pure Python and its installation is simple, but its dependencies have C extensions that require compilation.
You can see OSMnx's dependencies here and you'll have to install them one at a time. All of the tricky dependencies are brought in via geopandas, and you can read more about its installation details and dependencies here.
I'm using a station without admin rights and without pip. I need to use PyCharm (already installed) so as a workaround I installed Anaconda Navigator (doesn't require admin) and am using an environment in Anaconda as my interpreter in PyCharm.
I'm a bit confused regarding the conda install and the packages offered there. Are they all the same as the ones offered by the Python Package Index? Do developers only upload their work once to pypi.org and then it appears on both pip and conda installations or does it not include every single python package out there?
Thanks and I apologize if the question doesn't belong to this section of stack exchange.
First here is a link to an anther great post with a similar question: What is the difference between pip and conda?
But here is a response from my point of view and understanding:
Pip libraries specifically focus on packages related to the python. Conda uses those too, however, it also provides packages not related to python.
The best package example available is HDF5 it was not originally integrated into pip and Conda had their own hdf5 package. Pip has a similar package called h5py.
Also, conda's virtualization environments are what so appealing about it. In a way, Conda is like Docker.
Conda Hdf5: https://anaconda.org/anaconda/hdf5
Pip h5py: https://pypi.org/project/h5py/
Conda Cloud has the ability to read the PyPi libraries index, so it will be aware of newly uploaded packages.
Sorry if my response was not clear enough! English is not my first language, plus I was in the same boat as you a year ago.
In tensorflow installation guide it is said, that I should use "environment" to install tensorflow: https://www.tensorflow.org/install/install_windows#installing_with_anaconda
Why? Can't I just install with pip?
If installed with environment, should I "activate" it each time I use tensorflow?
If I use tensorflow from within other thing like keras and/or PyCharm, then how can I activate environment?
The question is about Windows. I assume you installed python using anaconda. Then you have a default environment, called root. You can create as many environments as you want, think of each as a separate installation of python. Using conda or pip installs stuff at your current installation. Conda stuff is kind of pre-compiled to work with your machine/anaconda environment, while pip stuff is usually compiled on the spot. I assume compiling tensorflow might not be completely trivial...
'Activate' changes from one environment to the other, so unless you have multiple environments you shouldn't need it. You run all these on command prompt.
Bottom line is, unless you have multiple environments (I highly recommend it so you can try different things) I cannot see you using activate. Install tensorflow and keras on the same one and only root environment you have. You should be able to access both (it is also possible just installing keras would install tensorflow, if its a dependancy)
If you see no prompt, it is the default, root environment. You can see all your environments with: conda info --envs But unless you create some environment (using e.g. conda create --name py Python=2) you probably only have root. One of the nice things with environments is you can have one with Python=2 (latest python 2), one with Python=3, another with Python=2.7 etc
On your follow-up, If you have multiple environments, you can switch between them on Pycharm by changing the interpreter. On the image you see me selecting e.g. py2_olv
Professional answer:
Quote from https://machinelearningspace.com/installing-tensorflow-2-0-in-anaconda-environment/:
What is Anaconda and why I recommend it?
...
[dropped intro to Anaconda]
...
For a Python developer or a data science researcher, using Anaconda
has a lot of advantages, such as independently installing/updating
packages without ruining the system. So, we no need to worry about the
system library or anything like that. This can save time and energy
for other things.
Anaconda can be used across different platforms, Windows, macOS, and
Linux. If we want to use a different Python version or package
libraries, just create a different environment and play around without
any risk of crashing the system library.
####
Unprofessional research:
Now in addition my own research. I am not a professional, I have little knowledge of the seemingly chaotic world of different install methods. This refers to some first research at https://superuser.com/questions/1572640/do-i-need-to-install-cuda-separately-after-installing-the-nvidia-display-driver/1572762#1572762. Mind that I am guessing a lot here. Please comment if I am wrong.
We see that at the moment, Pytorch supports version 10.2, Tensorflow supports 10.1, and it is not just the version that differs: mind that "CUDA Toolkit" (standalone) and cudatoolkit (conda binary install) are different! One is a a standalone / executable install, the other is a binary install. And tensorflow needs tensorflow-gpu to reach the standalone cuda install.
Therefore you should consider a separate environment for both Tensorflow and Pytorch, since any update of the conda cudatoolkit to version 11.0 could harm the dependency condition of Pytorch (Though this is not completely right. Pytorch uses a cuda that is installed inside Pytorch. It is still the approach to understand the recommended different envs). For tensorflow, you have to install version CUDA Toolkit 10.1 although 11.0 is already available, so that your whole card must run on a lower version than possible only to support Tensorflow - even if some games would like to have version 11.0.
Unprofessional answer:
If all of the dependencies are so important and so easily wrong when updated separately, like you could do with pip, any install that you do by yourself using pip might crash your sensitive tensorflow install. Therefore it is recommended to keep to a full service approach which Anaconda offers, where all dependencies are kept right, even if you enter conda install --all. That is why you better search for an Anaconda guide, for example https://machinelearningspace.com/installing-tensorflow-2-0-in-anaconda-environment/.
If you would have read through the entire document, it would have stated that the Anaconda installation is community supported, not officially supported. They want you to install TensorFlow using native pip through Python 3.5.x. That being said, from personal experience, I will tell you that if you are looking to run basic level TensorFlow Python scripts, such as training and testing an MNIST model, a Windows installation will be fine, or using a model that has already been trained for some purpose will also be fine. However, if you want to train advanced models such as Inception, which are the state-of-the-art image classifiers with less than 5% error for normal images, Windows is not suitable. You should try using Linux installation for any training purposes. I would recommend using VirtualBox, having used it in the past.
As for activating the environment, as long as, in any script / in the bash, you include the line "import tensorflow as tf", you should be fine, at least for native pip installation.
Good luck!
I recently discovered Conda after I was having trouble installing SciPy, specifically on a Heroku app that I am developing.
With Conda you create environments, very similar to what virtualenv does. My questions are:
If I use Conda will it replace the need for virtualenv? If not, how do I use the two together? Do I install virtualenv in Conda, or Conda in virtualenv?
Do I still need to use pip? If so, will I still be able to install packages with pip in an isolated environment?
Conda replaces virtualenv. In my opinion it is better. It is not limited to Python but can be used for other languages too. In my experience it provides a much smoother experience, especially for scientific packages. The first time I got MayaVi properly installed on Mac was with conda.
You can still use pip. In fact, conda installs pip in each new environment. It knows about pip-installed packages.
For example:
conda list
lists all installed packages in your current environment.
Conda-installed packages show up like this:
sphinx_rtd_theme 0.1.7 py35_0 defaults
and the ones installed via pip have the <pip> marker:
wxpython-common 3.0.0.0 <pip>
Short answer is, you only need conda.
Conda effectively combines the functionality of pip and virtualenv in a single package, so you do not need virtualenv if you are using conda.
You would be surprised how many packages conda supports. If it is not enough, you can use pip under conda.
Here is a link to the conda page comparing conda, pip and virtualenv:
https://docs.conda.io/projects/conda/en/latest/commands.html#conda-vs-pip-vs-virtualenv-commands.
I use both and (as of Jan, 2020) they have some superficial differences that lend themselves to different usages for me. By default Conda prefers to manage a list of environments for you in a central location, whereas virtualenv makes a folder in the current directory. The former (centralized) makes sense if you are e.g. doing machine learning and just have a couple of broad environments that you use across many projects and want to jump into them from anywhere. The latter (per project folder) makes sense if you are doing little one-off projects that have completely different sets of lib requirements that really belong more to the project itself.
The empty environment that Conda creates is about 122MB whereas the virtualenv's is about 12MB, so that's another reason you may prefer not to scatter Conda environments around everywhere.
Finally, another superficial indication that Conda prefers its centralized envs is that (again, by default) if you do create a Conda env in your own project folder and activate it the name prefix that appears in your shell is the (way too long) absolute path to the folder. You can fix that by giving it a name, but virtualenv does the right thing by default.
I expect this info to become stale rapidly as the two package managers vie for dominance, but these are the trade-offs as of today :)
EDIT: I reviewed the situation again in 04/2021 and it is unchanged. It's still awkward to make a local directory install with conda.
Virtual Environments and pip
I will add that creating and removing conda environments is simple with Anaconda.
> conda create --name <envname> python=<version> <optional dependencies>
> conda remove --name <envname> --all
In an activated environment, install packages via conda or pip:
(envname)> conda install <package>
(envname)> pip install <package>
These environments are strongly tied to conda's pip-like package management, so it is simple to create environments and install both Python and non-Python packages.
Jupyter
In addition, installing ipykernel in an environment adds a new listing in the Kernels dropdown menu of Jupyter notebooks, extending reproducible environments to notebooks. As of Anaconda 4.1, nbextensions were added, adding extensions to notebooks more easily.
Reliability
In my experience, conda is faster and more reliable at installing large libraries such as numpy and pandas. Moreover, if you wish to transfer your preserved state of an environment, you can do so by sharing or cloning an env.
Comparisons
A non-exhaustive, quick look at features from each tool:
Feature
virtualenv
conda
Global
n
y
Local
y
n
PyPI
y
y
Channels
n
y
Lock File
n
n
Multi-Python
n
y
Description
virtualenv creates project-specific, local environments usually in a .venv/ folder per project. In contrast, conda's environments are global and saved in one place.
PyPI works with both tools through pip, but conda can add additional channels, which can sometimes install faster.
Sadly neither has an official lock file, so reproducing environments has not been solid with either tool. However, both have a mechanism to create a file of pinned packages.
Python is needed to install and run virtualenv, but conda already ships with Python. virtualenv creates environments using the same Python version it was installed with. conda allows you to create environments with nearly any Python version.
See Also
virtualenvwrapper: global virtualenv
pyenv: manage python versions
mamba: "faster" conda
In my experience, conda fits well in a data science application and serves as a good general env tool. However in software development, dropping in local, ephemeral, lightweight environments with virtualenv might be convenient.
Installing Conda will enable you to create and remove python environments as you wish, therefore providing you with same functionality as virtualenv would.
In case of both distributions you would be able to create an isolated filesystem tree, where you can install and remove python packages (probably, with pip) as you wish. Which might come in handy if you want to have different versions of same library for different use cases or you just want to try some distribution and remove it afterwards conserving your disk space.
Differences:
License agreement. While virtualenv comes under most liberal MIT license, Conda uses 3 clause BSD license.
Conda provides you with their own package control system. This package control system often provides precompiled versions (for most popular systems) of popular non-python software, which can easy ones way getting some machine learning packages working. Namely you don't have to compile optimized C/C++ code for you system. While it is a great relief for most of us, it might affect performance of such libraries.
Unlike virtualenv, Conda duplicating some system libraries at least on Linux system. This libraries can get out of sync leading to inconsistent behavior of your programs.
Verdict:
Conda is great and should be your default choice while starting your way with machine learning. It will save you some time messing with gcc and numerous packages. Yet, Conda does not replace virtualenv. It introduces some additional complexity which might not always be desired. It comes under different license. You might want to avoid using conda on a distributed environments or on HPC hardware.
Another new option and my current preferred method of getting an environment up and running is Pipenv
It is currently the officially recommended Python packaging tool from Python.org
Conda has a better API no doubt. But, I would like to touch upon the negatives of using conda since conda has had its share of glory in the rest of the answers:
Solving environment Issue - One big thorn in the rear end of conda environments. As a remedy, you get advised to not use conda-forge channel. But, since it is the most prevalent channel and some packages (not just trivial ones, even really important ones like pyspark) are exclusively available on conda-forge you get cornered pretty fast.
Packing the environment is an issue
There are other known issues as well. virtualenv is an uphill journey but, rarely a wall on the road. conda on the other hand, IMO, has these occasional hard walls where you just have to take a deep breath and use virtualenv
1.No, if you're using conda, you don't need to use any other tool for managing virtual environments (such as venv, virtualenv, pipenv etc).
Maybe there's some edge case which conda doesn't cover but virtualenv (being more heavyweight) does, but I haven't encountered any so far.
2.Yes, not only can you still use pip, but you will probably have to. The conda package repository contains less than pip's does, so conda install will sometimes not be able to find the package you're looking for, more so if it's not a data-science package.
And, if I remember correctly, conda's repository isn't updated as fast/often as pip's, so if you want to use the latest version of a package, pip might once again be your only option.
Note: if the pip command isn't available within a conda virtual environment, you will have to install it first, by hitting:
conda install pip
Yes, conda is a lot easier to install than virtualenv, and pretty much replaces the latter.
I work in corporate, behind several firewall with machine on which I have no admin acces
In my limited experience with python (2 years) i have come across few libraries (JayDeBeApi,sasl) which when installing via pip threw C++ dependency errors
error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools
these installed fine with conda, hence since those days i started working with conda env.
however it isnt easy to stop conda from installing dependency inside c.programfiles where i dont have write access.