I have just started to use python (within Windows, 64bit) - and I have a basic question on how to install external packages within the anaconda / spyder environment. I understand that for most packages one can simply use “conda install bunnies”. However, certain packages are not in the anaconda repository, and might have be installed externally (e.g. from github). For those packages, in order to have spyder to recognize this package – does one only in addition have to update the PYTHONPATH manager in Spyder to include the directory (e.g. c:\users\bunnies) in which one has downloaded this package? Or should one take additional steps / is there a faster way?
You have several options to use packages that are not (yet) available via conda install:
1.) If the respective package is on PyPi you can build it as described in the manual.
2.) If building from scratch doesn't work and the package is on PyPi you can also try an installation via pip. Not that you have to use the pip in your Anaconda distribution and not the one of your systems Python installation.
3.) If you want to include external packages or local folders that contain Python-scripts you can do the following.
3.1.) Use the sys module and append the required package/folder to the path:
import sys
sys.path.append(r'/path/to/my/package')
3.2) Or put the modules into into site-packages, i.e. the directory $HOME/path/to/anaconda/lib/pythonX.X/site-packages which is always on sys.path. (Source)
3.3) Or add a .pth file to the directory $HOME/path/to/anaconda/lib/pythonX.X/site-packages. This can be named anything (it just must end with .pth). A .pth file is just a newline-separated listing of the full path-names of directories that will be added to your path on Python startup. (Source)
Good luck!
Related
I've been coding with R for quite a while but I want to start learning and using python more for its machine learning applications. However, I'm quite confused as to how to properly install packages and set up the whole working environment. Unlike R where I suppose most people just use RStudio and directly install packages with install.packages(), there seems to be a variety of ways this can be done in python, including pip install conda install and there is also the issue of doing it in the command prompt or one of the IDEs. I've downloaded python 3.8.5 and anaconda3 and some of my most burning questions right now are:
When to use which command for installing packages? (and also should I always do it in the command prompt aka cmd on windows instead of inside jupyter notebook)
How to navigate the cmd syntax/coding (for example the python documentation for installing packages has this piece of code: py -m pip install "SomeProject" but I am completely unfamiliar with this syntax and how to use it - so in the long run do I also have to learn what goes on in the command prompt or does most of the operations occur in the IDE and I mostly don't have to touch the cmd?)
How to set up a working directory of sorts (like setwd() in R) such that my .ipynb files can be saved to other directories or even better if I can just directly start my IDE from another file destination?
I've tried looking at some online resources but they mostly deal with coding basics and the python language instead of these technical aspects of the set up, so I would greatly appreciate some advice on how to navigate and set up the python working environment in general. Thanks a lot!
Python uses a different way of installing packages. Python has a thing named venv which stands for Virtual Environment. You install all of your packages in venv. Usually for each new project you make a new venv.
By using Anaconda on windows you install everything within the anaconda environment that you have specified.
python -m pip install "modulename" is a command that will install modulename to your default venv. You will be able to use this module when no other venv is specified. Here is the docs page. And here is a tutorial on how to use venv
By default python uses the same directory you have your code in. e.g. C:/Users/me/home/mypythonfile.py will run in C:/Users/me/home/ and will be able to access files in this directory. However you can use ../ to navigate directories or you can specify an absolute path to file you want to open e.g. with open("C:/system32/somesystemfile.sys") as file
Going over the technical differences of conda and pip:
So Conda is a packaging tool and installer that aims to do more than what pip does; handle library dependencies outside of the Python packages as well as the Python packages themselves. Both have many similar functionalities as well, you can install packages or create virtual environments with both.
It is generally advisable to generally have both conda and pip installed since there are some packages which might not be available with conda but with pip and vice versa.
The commands to install in both the ways is easy enough, but one thing to keep in mind is that
conda stores packages in the anaconda/pkgs directory
pip stores it in directory under /usr/local/bin/ for a Unix-based system, or \Program Files\ for Windows
You can use both pip or conda inside the jupyter notebook, it will work just fine, but it may be possible that you get multiple versions of the same package.
Most of the times, you will use cmd only to install a module used in your code, or to create environments, py -m pip install "SomeProject" here basically means that the module "SomeProject" will be downloaded in base env.
You could think of conda as python with a variety of additional functionalities, such as certain pre-installed packages and tools, such as spyder and jupyter. Hence, you must be precise when you say:
I've downloaded python 3.8.5 and anaconda3
Does it mean you installed python in your computer and then also anaconda?
In general, or at least in my opinion, using anaconda has advantages for development, but typically you'll just use a simple python installation in production (if that applies to you).
Anaconda has it's own package registry/repository . When you call conda install <package>, it will search for the package there and install it if available. You would better search it first, for instance matplotlib.
pip is a package manager for the Python Package Index. pip also ships with anaconda. Hence, in an anaconda environment you may install packages from either sources (either using pip install or conda install). For instance, pandas from PyPI and pandas from conda. There is no guarantee that packages exist in both sources. You must either search it first or simply try it.
In your first steps, I would suggest you to stick to only one dev env (either simple python or anaconda, recommend the second). Because that simplifies the question: "which python and which pip is executed in the cmd line?". That said, those commands should work as expected in any terminal, it be a simple cmd or an embedded one like in PyCharm or VS Code.
You could inspect that by running (on windows and linux at least):
which python, which pip.
Honestly, this is a question/answer that falls outside the scope of SO and for more info you would better check official websites, such as for anaconda or search for python vs anaconda blogs.
I'm working on a script in python that relies on several different packages and libraries. When this script is transferred to another machine, the packages it needs in order to run are sometimes not present or are older versions that do not have the same functionality and cause the script to fail.
I was considering using a virtual environment, but I can't find a way to have the script use the specific environment I design as it's default, and in order to use the environment a user must manually activate it from the command line.
I've also looked into trying to check the versions of the packages installed on the machine, and if they are not sufficient then updating them from the script as described here:
Installing python module within code
Is there any easier/surefire way to make sure that the needed packages will always be available regardless of where it's run?
The normal approach is to create an installation script and have that manage your dependencies. Then when you move your project to a new environment your installer will check that all dependencies are present.
I recommend you check out setuptools: https://setuptools.readthedocs.io/en/latest/
If you don't want to install dependencies whenever you need to use your script somewhere new, then you could package your script into a Docker container.
If the problem is ensuring the required packages are available in a new environment or virtual environment, you could use pip and generate a requirements.txt and check it in version control or use a tool to do that for you, like pipenv.
If you would prefer to generate a requirements.txt by hand, you should:
Install your depencencies using pip
Type pip freeze > requirements.txt to generate a requirements.txt file
Check requirements.txt in you source management software
When you need to setup a new environment, use pip install -m requirements.txt
The solution that I've been using has been to include a custom library (folder with all of my desired packages) in the folder with my script, and I simply import them from there:
from Customlib import pkg1, pkg2,...
As long as the custom library and script stay together in the same folder, it will always have access to the right packages and the correct versions of those packages.
I'm not sure how robust this solution actually is or what possible bugs may arise from this if it is passed from machine to machine, but for now this seems to work.
Can't seem to get imports to work. I've installed using
pip install pyperclip
I can can confirm that it was successfully installed:
But then when attempt to confirm in in the Shell:
Is there another step to importing that I'm just missing?
Your problem is that pip is installing for the global (all users) version of python, and you're using a version of python installed for only your user c:\Users\bbarker\AppData\Local\Programs\Python\Python36. You'll want to either use the global install instead c:\program files (x86)\python36-32 or change your pip defaults as described here.
You'll notice the folder where pip told you where pyperclip was installed does not show up in sys.path. Therefore python does not know to search there for libraries. Those few file paths you did have in your sys.path are automatically generated defaults that are relative to the install directory of the particular instance of python you're currently using. If you use the instance in your \program files (x86)\ folder, the paths will be relative to that folder instead
tldr;
You have 2 instances of python installed, and you're installing libraries to one and using the other.
I built a python3 package called gender_univ using the Anaconda conda build command. I uploaded the package to the Anaconda cloud and then installed it into my conda environment. Though the package shows up in the list of installed packages when I type conda list, whenever I try to access the package using import gender_univ I get the error no module named gender_univ.
I want to understand why I can't seem to import a package that is apparently installed in my conda virtual environment? Any suggestions.
First, I would check that you are using the correct python (i.e. which python and confirm that it is the python in your conda environment). Next, you can check if your package is in the site-packages directory of that same python.
The most likely thing, I'd bet, is that the conda package doesn't include everything correctly. Are you sure that you have a build.sh (or bld.bat if you are on windows) and a setup.py? Did you try expanding your built conda package and looking for your python classes in there?
If you expand your built conda package, probably something like gender_univ-VERSION-py35_0.tar.bz2, you should see a lib/python3.5/site-packages/gender_univ directory (i.e. python package). Do you? If not, then the failure is with your building.
I have looked everywhere and can't find where the packages are installed.
Further, are the packages from pip, modules, libraries or just packages in python terminology?
Use this command to list global packages with their locations:
pip list -v
# output example (windows 10):
# Package Version Location Installer
# ------------------------- --------- ---------------------------------------------------------------- ---------
# adal 0.4.5 c:\users\test\appdata\roaming\python\python36\site-packages pip
# aiocache 0.10.0 c:\program files\python36\lib\site-packages pip
When using virtual environment the local packages is located in project folder:
<project folder>\venv\Lib\site-packages
for python 3.X default location C:\Users\username \AppData\Local\Programs\Python\Python36\Lib\site-packages
From the docs
Python usually stores its library (and thereby your site-packages folder) in the installation directory. So, if you had installed Python to C:\Python\, the default library would reside in C:\Python\Lib\ and third-party modules should be stored in C:\Python\Lib\site-packages.
It depends on how/where your python was installed.
For example:
if you are using Conda for your python, you will most likely find your packages in the default location of:
c:\users\<username>\miniconda3\lib\site-packages
Whereas, if you installed Python3.8 as a stand-alone, you will find it under:
C:\Users\<username>\AppData\Roaming\Python\Python38\site-packages\
where if you install a simply pip package (that did not use any specific locations), and just having some scripts, it will end up in:
C:\Python38\Scripts\
Noting that if the script depend on libraries, it will probably not work, as they will not be found from this location, if installed on Windows, and not taking this into account.
Or if you used Cygwin, they'll end up in one of:
/usr/lib/python3.8/site-packages
/usr/local/lib/python3.8/site-packages
Again, you best bet is to check with:
pip -V
pip list -v
There is also a pip package called pip-date that contain a utility called pipbyday that will show you the location of each package and the installation time.