Develop Reference

Creating and installing Conda packages into Virtual Envs

I'm working on packaging up a suite of tools that can be installed in different environments, and I've run into many problems with dependencies, which are an issue since this package will be installed in air-gapped environments.
The package will be installed via Anaconda, and I have provided the installation script. In order to create the package, I ran the following command:
conda metapackage toolkit_bundle 0.0.1 --dependencies r-essentials tensorflow gensim spacy r-ggplot2 r-plotly r-dplyr r-rjson r-tm r-reshape2 r-shiny r-sparklyr r-slam r-nlp r-cluster r-ggvis r-plyr r-tidyr r-zoo r-magrittr r-xtable r-htmlwidgets r-formattable r-highcharter --summary "Toolkit Bundle"
This produced a tar.bzip2 file that I held on to and tried to install via the conda command
conda install toolkit_bundle.tar.bz2
The command seemed to run successfully, but I was unsuccessful in importing the modules in Python. I also tried creating a virtual conda environment and importing the package.
conda create -n myenv toolkit_bundle-0.0.1.tar.bz2
There was no error, but none of the modules were able to be imported either.
Am I missing a step in this process, or is my thought process flawed?
Update:
It looks like my thinking was pretty flawed. A quick skim of the conda metapackage command documentation revealed the following:
Tool for building conda metapackages. A metapackage is a package with no files, only metadata. They are typically used to collect several packages together into a single package via dependencies.
So my initial understanding was incorrect, and the package only contains metadata. Are there any other ideas for creating packages with dependencies resolved that can be installed in an air gapped environment?

I think you want to look at the command conda build for making packages, which just requires writing an appropriate meta.yaml file containing the dependencies, along with some other build parameters. There is good documentation for doing so on the conda website: https://conda.io/docs/user-guide/tasks/build-packages and there is a repo of examples.
If you have a working PIP package, you can also auto-generate a conda package recipe using conda skeleton.
Once you have built a set of packages locally, you can use the --use-local option to conda install to install from your local repo, with no need for an internet connection (as long as the packages for all the dependencies are in your local repo).

I was able to download the packages I needed via the pypi website, and after determining the dependencies, I manually downloaded them and wrote a script to install them in the required order.

Why does conda create try to install weird packages?

I am trying to install a new conda environment that will be totally separate from my other environments, so I run:
conda create --name foot35 python=3.5
Anaconda then asks for my approval to install these NEW packages:
asn1crypto: 0.22.0-py35he3634b9_1
ca-certificates: 2017.08.26-h94faf87_0
cachecontrol: 0.12.3-py35h3f82863_0
certifi: 2017.7.27.1-py35hbab57cd_0
cffi: 1.10.0-py35h4132a7f_1
chardet: 3.0.4-py35h177e1b7_1
colorama: 0.3.9-py35h32a752f_0
cryptography: 2.0.3-py35h67a4558_1
distlib: 0.2.5-py35h12c42d7_0
html5lib: 0.999999999-py35h79d4e7f_0
idna: 2.6-py35h8dcb9ae_1
lockfile: 0.12.2-py35h667c6d9_0
msgpack-python: 0.4.8-py35hdef45cb_0
openssl: 1.0.2l-vc14hcac20b0_2 [vc14]
packaging: 16.8-py35h5fb721f_1
pip: 9.0.1-py35h69293b5_3
progress: 1.3-py35ha84af61_0
pycparser: 2.18-py35h15a15da_1
pyopenssl: 17.2.0-py35hea705d1_0
pyparsing: 2.2.0-py35hcabcaab_1
pysocks: 1.6.7-py35hb30ac0d_1
python: 3.5.4-hedc2606_15
requests: 2.18.4-py35h54a615f_1
setuptools: 36.5.0-py35h21a22e4_0
six: 1.10.0-py35h06cf344_1
urllib3: 1.22-py35h8cc84eb_0
vc: 14-h2379b0c_1
vs2015_runtime: 14.0.25123-hd4c4e62_1
webencodings: 0.5.1-py35h5d527fb_1
wheel: 0.29.0-py35hdbcb6e6_1
win_inet_pton: 1.0.1-py35hbef1270_1
wincertstore: 0.2-py35hfebbdb8_0
I don't know why it suggests these specific ones. I looked up lockfile and its website says:
Note: This package is deprecated.
Here is a screenshot of my command prompt as additional information.
I am trying to do a clean install that is unrelated/independent to the root environment.
Why is conda trying to install these things and how do I fix it?

conda create will "Create a new conda environment from a list of specified packages." ( https://conda.io/docs/commands/conda-create.html )
What list??!? The .condarc file is the conda configuration file.
https://conda.io/docs/user-guide/configuration/use-condarc.html#overview
The .condarc file can change many parameters, including:
Where conda looks for packages.
If and how conda uses a proxy server.
Where conda lists known environments.
Whether to update the bash prompt with the current activated environment name.
Whether user-built packages should be uploaded to Anaconda.org.
**Default packages or features to include in new environments.**
Additionally, if you ever typed conda config, even accidentally...
The .condarc file is not included by default, but it is automatically created in your home directory the first time you run the conda config command.
A .condarc file may also be located in the root environment, in which case it overrides any in the home directory.
If you would like a single clean env then Boshika's recommendation of --no-default-packages flag for an instance though, you can check and modify the default packages for all further envs. ( https://conda.io/docs/user-guide/configuration/use-condarc.html#always-add-packages-by-default-create-default-packages )
Always add packages by default (create_default_packages)
When creating new environments, add the specified packages by default. The default packages are installed in every environment you create. You can override this option at the command prompt with the --no-default-packages flag. The default is to not include any packages.
EXAMPLE:
create_default_packages:
- pip
- ipython
- scipy=0.15.0
Lockfile may be there due to legacy requirements across all operating systems. Hopefully, you have the tools to remove it if you choose.

To avoid conda from installing all default packages you can try this
conda create --name foot35 --no-deps python=3.5

please don't loose the hope it's very weird for me also.
What you have to do just follow the steps: -
1.Download the anaconda for you system from it's official site and Install it : https://repo.continuum.io
After the Installation process, you can select your own package from there and please don't need to download anything from anywhere, it's full of packages over the internet.
3.If you want to work on python download Syder IDE its very useful for the Machine learning library.
Don't create other environment instead of root by defaults otherwise you have to duplicate all the file again, if there is any error while installing in root so close the window and again run as administration and after that its works fine.
Cause all the file in your root environment so you don't worry about the path in future and you can install and uninstall the packages : like - numpy , pandas, tensorflow and its gpu , scikit-learn etc from there eaisly.
Thank you

These packages are generally useful if you wish to pip install ... anything. Without many of them doing a pip install requests could result in errors such as these (and more)
No Module named Setuptools
pip: command not found
pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available
The issue that the conda create ... exposes is that the packages it wants to pull down are variable (based on when you create the environment). If you wish to maintain the same environment for you and for those who may collaborate with you, then freezing or pinning conda create's default installed package may be necessary.
One way of doing this is creating your environment with conda env create using a conda environment YAML file such as this example:
dependencies:
- ca-certificates=2018.03.07
- certifi=2018.4.16
- libedit=3.1.20170329
- libffi=3.2.1
- ncurses=6.1
- openssl=1.0.2o
- pip=10.0.1
- python=3.6.6
- readline=7.0
- setuptools=40.0.0
- sqlite=3.24.0
- tk=8.6.7
- wheel=0.31.1
- xz=5.2.4
- zlib=1.2.11
conda env create -n <NAME_OF_ENVIRONMENT> -f <PATH_TO_CONDA_REQUIREMENTS_FILE>
(note it's conda env create not conda create)

Conda PackageNotFoundError - unable to install packages

I'm trying to install a set of libraries used in a data science course, following the instructions from this url, under "Setting up the environment". I have installed Acaconda3 using Python 3.6 on a Windows 10 machine. I have a previously installed version of Python 2.7 which I suspect is the problem, but I´m not sure how to solve this (both versions are now listed in the environment settings).
The problem starts when trying to install the libraries, which gives me the following error when I open a command prompt and enter a "conda install ... " command. The .txt file I´m trying to read is in the project folder. How can I install the packages properly? (anaconda administrator allows a workaround by letting you select the packages and install them from there, but I´d like to know why the command prompt doesn´t work in this case).
(packt-py36) C:\Users\Eric\AnacondaProjects>conda install --file
conda-reqs.txt --yes
Fetching package metadata .............
PackageNotFoundError: Package not found: Conda could not find '

It isn't working because the list of packages was created for OS X. I was able to successfully import these packages on OS X but there are several packages and versions that are not compatible. I have gone through the file and eliminated any non-compatible packages for you. Quite a few packages are missing but you will be able to install from this.
This file may be used to create an environment using:
$ conda create --name <env> --file <this file>
platform: win-64
bleach=1.5.0
cycler=0.10.0=py36_0
decorator=4.0.11=py36_0
entrypoints=0.2.2=py36_1
freetype=2.5.5
html5lib=0.999=py36_0
icu=57.1
ipykernel=4.6.1=py36_0
ipython=6.0.0=py36_0
ipython_genutils=0.2.0=py36_0
ipywidgets=6.0.0=py36_0
jinja2=2.9.6=py36_0
jsonschema=2.6.0=py36_0
jupyter=1.0.0=py36_3
jupyter_client=5.0.1=py36_0
jupyter_console=5.1.0=py36_0
jupyter_core=4.3.0=py36_0
libpng=1.6.27
markupsafe=0.23=py36_2
matplotlib=2.0.0=np112py36_0
mistune=0.7.4=py36_0
mkl=2017.0.1=0
nbconvert=5.1.1=py36_0
nbformat=4.3.0=py36_0
notebook=5.0.0=py36_0
numpy=1.12.1=py36_0
openssl=1.0.2k
pandas=0.19.2=np112py36_1
pandocfilters=1.4.1=py36_0
path.py=10.3.1=py36_0
pickleshare=0.7.4=py36_0
pip=9.0.1=py36_1
prompt_toolkit=1.0.14=py36_0
pygments=2.2.0=py36_0
pyparsing=2.1.4=py36_0
pyqt=5.6.0=py36_2
python=3.6.1=0
python-dateutil=2.6.0=py36_0
pytz=2017.2=py36_0
pyzmq=16.0.2=py36_0
qt=5.6.2
qtconsole=4.3.0=py36_0
scikit-learn=0.18.1=np112py36_1
scipy=0.19.0=np112py36_0
simplegeneric=0.8.1=py36_1
sip=4.18=py36_0
six=1.10.0=py36_0
sqlite=3.13.0
testpath=0.3=py36_0
tk=8.5.18
tornado=4.5.1=py36_0
traitlets=4.3.2=py36_0
wcwidth=0.1.7=py36_0
wheel=0.29.0=py36_0
widgetsnbextension=2.0.0=py36_0
zlib=1.2.8

Conda is corrupted after pip install conda

My conda is corrupted after I run command "pip install conda". Is there any way to recover it ? Thanks
Here's the error I see when running conda command
ERROR: The install method you used for conda--probably either `pip install conda`
or `easy_install conda`--is not compatible with using conda as an application.
If your intention is to install conda as a standalone application, currently
supported install methods include the Anaconda installer and the miniconda
installer. You can download the miniconda installer from
https://conda.io/miniconda.html.

Simply, follow the instructions given in the error:
Download miniconda, then run the script file by typing following command: bash <file_name.sh> e.g.
bash Miniconda3-latest-Linux-x86_64.sh.
Now reopen the terminal for the changes to take effect.
If conda is already installed on your system, you can reinstall it with the -f force option, for example,
bash Miniconda3-latest-Linux-x86_64.sh -f
To test your installation, enter the command conda --version. If installed correctly, you will see the version of conda installed.
miniconda: https://conda.io/en/latest/miniconda.html
conda troubleshooting: https://conda.io/docs/troubleshooting.html

If you are facing this problem in Virtual Machine (VM) then you have to activate the main environment by running below line of code:
source /anaconda_installation_folder_path/bin/activate
Once you are in your main environment you can work with conda.

TL;DR: nothing is corrupted, the message you're seeing is a hardcoded stub and could be fixed.
conda package manager actually can be used with regular python installation.
Update: I've been tinkering with the described method and found that you should use conda install --dry-run ... to see changes that are going to happen. Some conda packages depend on other python version, which would overwrite the installed one. There's might be a solution for this with changing conda channels or using virtualenv. I also found that --dry-run doesn't work when using local package archives.
I'll show you how to run cudatoolkit 9.1 without any Anaconda and python-3.6-amd64. I'm using cuda 9.1 from here.
Since conda is artificially tethered with Anaconda, you have to untie them.
I recommend you to backup up python installation directory you'll be working with (or use virtualenv).
Install menuinst dependency.
At the moment, it's broken from PyPi, so get if from
github. Build it and install python setup.py install
This package is problematic also in Anaconda distribution. It triggers series of requests for admin rights every time, which should be suppressed with conda ... --no-shortcuts option.
pip install pypiwin32, dependency of (1)
pip install conda, requires (1)
Move to python installation directory. ./Scripts/conda.exe should exist.
Move to ./Lib/site-packages/conda
Search directory recursively for pip_warning substring in following TEXT file types: .py, .json, .txt
Replace matching substrings pip_warning with main
Don't forget to abide the syntax of file types you'd be editing.
Now open the ./Scripts/conda.exe executable in any hex-editor and
find pip_warning, carefully overwrite it with main and wipe the
rest with spaces until bytes import main
Check for file size not have changed.
Remove any __pycache__ dirs if found in ./Lib/site-packages/conda
If you only need working conda without cuda, you're done here.
Run conda install mkl, pip install llvmlite numpy
Download packages cudatoolkit-9.1-0.tar.bz2
and numba-0.36.2.tar.bz2
and run
conda install cudatoolkit-9.1-0.tar.bz2
conda install numba-0.36.2-***.tar.bz2
Wait a little while unpacking finished.
Now try these examples, they should work and your gpu monitor show some activity. conda ... commands also do work.
With Linux, I guess instructions are the same, just would be .sh or ELF in place of .exe.

In my case, what worked was:
pip uninstall conda
and then installing miniconda

Download miniconda, then run the script file by typing following command: bash <file_name.sh> e.g. bash Miniconda3-latest-Linux-x86_64.sh -u
'-u' : update tag, used if the original conda bash paths get lost due to certain modifications in the .bashrc file

How can I install XGBoost package in python on Windows

I tried to install XGBoost package in python.
I am using windows os, 64bits . I have gone through following.
The package directory states that xgboost is unstable for windows and is disabled:
pip installation on windows is currently disabled for further invesigation, please install from github.
https://pypi.python.org/pypi/xgboost/
I am not well versed in Visual Studio, facing problem building XGBoost.
I am missing opportunities to utilize xgboost package in data science.
Please guide, so that I can import the XGBoost package in python.
Thanks

If you are using anaconda (or miniconda) you can use the following:
conda install -c anaconda py-xgboost UPDATED 2019-09-20
Docs
Other channels available, such as rapidsai/py-xgboost UPDATED 2021-12-07
Check install by:
Activating the environment (see below)
Running conda list
To activate an environment:
On Windows, in your Anaconda Prompt, run (assumes your environment is named myenv):
activate myenv
On macOS and Linux, in your Terminal Window, run (assumes your environment is named myenv):
source activate myenv
Conda prepends the path name myenv onto your system command.

Build it from here:
download xgboost whl file from here (make sure to match your python version and system architecture, e.g. "xgboost-0.6-cp35-cp35m-win_amd64.whl" for python 3.5 on 64-bit machine)
open command prompt
cd to your Downloads folder (or wherever you saved the whl file)
pip install xgboost-0.6-cp35-cp35m-win_amd64.whl (or whatever your whl file is named)

You first need to build the library through "make", then you can install using anaconda prompt (if you want it on anaconda) or git bash (if you use it in Python only).
First follow the official guide with the following procedure (in Git Bash on Windows):
git clone --recursive https://github.com/dmlc/xgboost
git submodule init
git submodule update
then install TDM-GCC here and do the following in Git Bash:
alias make='mingw32-make'
cp make/mingw64.mk config.mk; make -j4
Last, do the following using anaconda prompt or Git Bash:
cd xgboost\python-package
python setup.py install
Also refer to these great resources:
Official Guide
Installing Xgboost on Windows
Installing XGBoost For Anaconda on Windows

You can pip install catboost. It is a recently open-sourced gradient boosting library, which is in most cases more accurate and faster than XGBoost, and it has categorical features support.
Here is the site of the library:
https://catboost.ai

pip install xgboost works also with python 3.8, whereas other options mentioned above did not work for me

The following command should work but, If you have a problem with this command
conda install -c conda-forge xgboost
First activate your environment . Assume your environment is named
simply write in conda terminal :
activate <MY_ENV>
and then
pip install xgboost

on a macOS the following command worked
conda install -c conda-forge xgboost but before doing this i had read some other article and hence did install gcc using brew

Besides what's already on developers' github, which is building from source(creating a c++ environment, etc.), I have found an easier way to do it, which I explained here with details. Basically, you have to go a website by UC Irvine and download a .whl file, then cd to the folder and install xgboost with pip.

XGBoost is used in Applied Machine Learning and is known for its gradient boost algorithm and it is available as a library in python but has to be compiled using cmake.
Alternatively what you can do is from this link you can download the C pre-compiled library and install it using the pip install < FILE-NAME.whl> command. Ensure you have downloaded the library which is compatible with your python version.
I experienced this problem while I was using the same in Anaconda(Spyder). Then just restart the kernel and your error will go away.