Develop Reference

How to import torchreid in google colab?

I'm trying to install torchreid which is a library for person re-identification in PyTorch. I've followed the steps mentioned on the git repository but is getting this error.
#conda install
!wget -c https://repo.anaconda.com/miniconda/Miniconda3-4.5.4-Linux-x86_64.sh
!chmod +x Miniconda3-4.5.4-Linux-x86_64.sh
!bash ./Miniconda3-4.5.4-Linux-x86_64.sh -b -f -p /usr/local
!conda install -q -y --prefix /usr/local python=3.6 ujson
import sys
sys.path.append('/usr/local/lib/python3.6/site-packages')
!git clone https://github.com/KaiyangZhou/deep-person-reid.git
!cd deep-person-reid/
!conda create --name torchreid python=3.7
!conda activate torchreid
!pip install -r /content/deep-person-reid/requirements.txt
!conda install pytorch torchvision cudatoolkit=9.0 -c pytorch
import torchreid
error
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-1-4faf46a39b5d> in <module>()
----> 1 import torchreid
ModuleNotFoundError: No module named 'torchreid'

Run below command,
# install torchreid (don't need to re-build it if you modify the source code)
python setup.py develop

First deactivate you created conda environment 'torchreid'. Activate it again and try to import.
deactivate torchreid
conda activate torchreid
Note: you are installing torchreid in a specifica conda environment, which in not accessible from outside of that environment.

setup.py install within conda env via Docker

So I'n new-ish to Docker.
What I'm trying to do:
create a docker image (that puts you into an anaconda enviroment and downloads some libraries via conda), within that enviroment run a setup.py install so that the library CPLEX is also in the image
Build the image. Next I run the image, and set a scratch directory that should have all the python programs in it that I want to run.
This is my docker file:
## syntax=docker/dockerfile:1
from continuumio/miniconda3
RUN conda create --name opt python=3.7
ENV PATH /opt/conda/envs/mro_env/bin:$PATH
RUN /bin/bash -c "source activate opt"
RUN conda install -c conda-forge fenics
RUN conda install -c conda-forge scipy
RUN conda install -c conda-forge numpy
RUN conda install -c conda-forge matplotlib
RUN conda install -c conda-forge ipopt
RUN conda install -c conda-forge glpk
RUN conda install -c conda-forge pyomo
RUN conda install -c conda-forge pandas
RUN conda install -c conda-forge scikit-learn
ENTRYPOINT [ "python" ]
COPY CPLEX_Studio129/ /CPLEX_Studio129/
COPY . /codes/
RUN cd /CPLEX_Studio129/ && python cplex/python/3.7/x86-64_osx/setup.py install
WORKDIR /codes/
RUN cd /codes/
Next I build the image, and then run the image :
sudo docker build -t ex:latest .
docker run --rm -it --entrypoint /bin/bash ex
(base) root#bc9beca93d69:/codes# python test_cplex.py
Traceback (most recent call last):
File "/codes/test_cplex.py", line 1, in <module>
import cplex
ModuleNotFoundError: No module named 'cplex'
(base) root#bc9beca93d69:/codes# ls
For what its worth, if I make /CPLEX_Studio129/ the WORKDIR, and install the library, then start a python in that directory, and try to import cplex it works. But my hope was that it would work when I run a program that imports CPLEX in /codes/, which it cannot find.
Any tips, and a bried explation of where I went wrong would be greatly appreciated.

Installing Conda on Deepnote

I am a fairly inexperienced programmer and am struggling with installing Conda into Deepnote. Pip install doesn't work for certain packages. For example, I'm trying to install rdkit, a cheminformatics package, which has rather complex installation instructions or a simple 1 line of code managed through the Anaconda/mini-conda distribution. I really like the Deepnote notebooks and I would really appreciate any help here, please.
So far, I have found this useful code for Conda installation on Google Colab: https://github.com/dataprofessor/code/blob/master/python/google_colab_install_conda.ipynb
! wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh
! chmod +x Miniconda3-py37_4.8.2-Linux-x86_64.sh
! bash ./Miniconda3-py37_4.8.2-Linux-x86_64.sh -b -f -p /usr/local
import sys
sys.path.append('/usr/local/lib/python3.7/site-packages/')
Whilst this successfully works on Google Colab, I'm not sure why it fails as shown below on Deepnote:
--2020-12-04 22:58:34-- https://repo.anaconda.com/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh
Resolving repo.anaconda.com (repo.anaconda.com)... 104.16.130.3, 104.16.131.3, 2606:4700::6810:8203, ...
Connecting to repo.anaconda.com (repo.anaconda.com)|104.16.130.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 85055499 (81M) [application/x-sh]
Saving to: ‘Miniconda3-py37_4.8.2-Linux-x86_64.sh’
Miniconda3-py37_4.8 100%[===================>] 81.12M 133MB/s in 0.6s
2020-12-04 22:58:35 (133 MB/s) - ‘Miniconda3-py37_4.8.2-Linux-x86_64.sh’ saved [85055499/85055499]
PREFIX=/usr/local
./Miniconda3-py37_4.8.2-Linux-x86_64.sh: line 392: /usr/local/conda.exe: Permission denied
chmod: cannot access '/usr/local/conda.exe': No such file or directory
Unpacking payload ...
./Miniconda3-py37_4.8.2-Linux-x86_64.sh: line 404: /usr/local/conda.exe: No such file or directory
./Miniconda3-py37_4.8.2-Linux-x86_64.sh: line 406: /usr/local/conda.exe: No such file or directory
I also want to do conda install -c bioconda gromacs, which I cannot find a work around for, so I am hoping someone can help me resolve this query.
Many thanks in advance!
P.S. I am on a Mac OS

You can check this notebook from Daniel Zvara
Using Conda in Deepnote in 3 simple steps
In sum
# 1. Install Conda and make Conda packages available in current environment
!wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
!chmod +x Miniconda3-latest-Linux-x86_64.sh
!sudo bash ./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local
import sys
sys.path.append('/usr/local/lib/python3.7/site-packages/')
# 2. Package installation
# !sudo conda install -y [EXTRA_OPTIONS] package_name
# So for example Keras from conda-forge channel
!sudo conda install -y -c conda-forge keras
# 3. Package usage
import keras

Also found an alternative solution on the Deepnote community: https://community.deepnote.com/c/custom-environments/custom-environment-for-installing-conda-packages
Can set up a custom environment with the following Dockerfile (just fill in the placeholder for the packages you require).
FROM gcr.io/deepnote-200602/templates/deepnote
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
RUN bash ~/miniconda.sh -b -p $HOME/miniconda
ENV PATH $HOME/miniconda/bin:$PATH
RUN conda install python=3.7 ipykernel -y
RUN conda install <insert packages here> -y
RUN python -m ipykernel install --user --name=conda
ENV DEFAULT_KERNEL_NAME "conda"
And the custom environment in Deepnote is set up through the Environment tab in the sidebar, giving you an option for the Dockerfile. Copy & paste the above, then click build and restart machine - the environment will be set up allowing you to use conda.

You can now choose Anaconda in the built-in environments in Deepnote.

how can I write code of terminal inside python script at windows

How can I send this code below as a command from python script to be run on tirminal:
cd %HOMEPATH%
conda install git
git clone https://github.com/facebookresearch/demucs
cd demucs
conda env update -f environment-cpu.yml
conda activate demucs
python.exe -m demucs.separate -d cpu --dl "PATH_TO_AUDIO_FILE_1" ["PATH_TO_AUDIO_FILE_2" ...]
I applied this code:
import os
os.system("cd %HOMEPATH%")
os.system("conda install git")
os.system("git clone https://github.com/facebookresearch/demucs")
os.system("cd demucs")
os.system("conda env update -f environment-cpu.yml")
os.system("conda activate demucs")
os.system("python.exe -m demucs.separate -d cpu --dl 'C:/Users/Arwa_Analyst/Downloads/CAPSTONE PROJECT/research samples/audioDataset/Eval_S_Wav/-1hDIl9Udkw.wav' ")
and I got this errors:
enter image description here
enter image description here

import os
os.system("cd %HOMEPATH%")
os.system("conda install git")
#...
#..
You can use the import os package and use system function to run terminal commands from python.

ImportError: No module named numpy - Google Cloud Dataproc when using Jupyter Notebook

When starting Jupyter Notebook on Google Dataproc, importing modules fails. I have tried to install the modules using different commands. Some examples:
import os
os.sytem("sudo apt-get install python-numpy")
os.system("sudo pip install numpy") #after having installed pip
os.system("sudo pip install python-numpy") #after having installed pip
import numpy
None of the above examples work and return an import error:
enter image description here
When using command line I am able to install modules, but still the import error remains. I guess I am installing modules in a wrong location.
Any thoughts?

I found a solution.
import sys
sys.path.append('/usr/lib/python2.7/dist-packages')
os.system("sudo apt-get install python-pandas -y")
os.system("sudo apt-get install python-numpy -y")
os.system("sudo apt-get install python-scipy -y")
os.system("sudo apt-get install python-sklearn -y")
import pandas
import numpy
import scipy
import sklearn
If any one has a more elegant solution, please let me know.

Try conda install numpy as Google's jupyter init script is using conda.
I personally prefer to have my own init scripts so I can have more control.

#!/usr/bin/env bash
set -e
ROLE=$(curl -f -s -H Metadata-Flavor:Google http://metadata/computeMetadata/v1/instance/attributes/dataproc-role)
INIT_ACTIONS_REPO=$(curl -f -s -H Metadata-Flavor:Google http://metadata/computeMetadata/v1/instance/attributes/INIT_ACTIONS_REPO || true)
INIT_ACTIONS_REPO="${INIT_ACTIONS_REPO:-https://github.com/GoogleCloudPlatform/dataproc-initialization-actions.git}"
INIT_ACTIONS_BRANCH=$(curl -f -s -H Metadata-Flavor:Google http://metadata/computeMetadata/v1/instance/attributes/INIT_ACTIONS_BRANCH || true)
INIT_ACTIONS_BRANCH="${INIT_ACTIONS_BRANCH:-master}"
DATAPROC_BUCKET=$(curl -f -s -H Metadata-Flavor:Google http://metadata/computeMetadata/v1/instance/attributes/dataproc-bucket)
echo "Cloning fresh dataproc-initialization-actions from repo $INIT_ACTIONS_REPO and branch $INIT_ACTIONS_BRANCH..."
git clone -b "$INIT_ACTIONS_BRANCH" --single-branch $INIT_ACTIONS_REPO
# Ensure we have conda installed.
./dataproc-initialization-actions/conda/bootstrap-conda.sh
#./dataproc-initialization-actions/conda/install-conda-env.sh
source /etc/profile.d/conda_config.sh
if [[ "${ROLE}" == 'Master' ]]; then
conda install jupyter
if gsutil -q stat "gs://$DATAPROC_BUCKET/notebooks/**"; then
echo "Pulling notebooks directory to cluster master node..."
gsutil -m cp -r gs://$DATAPROC_BUCKET/notebooks /root/
fi
./dataproc-initialization-actions/jupyter/internal/setup-jupyter-kernel.sh
./dataproc-initialization-actions/jupyter/internal/launch-jupyter-kernel.sh
fi
if gsutil -q stat "gs://$DATAPROC_BUCKET/scripts/**"; then
echo "Pulling scripts directory to cluster master and worker nodes..."
gsutil -m cp -r gs://$DATAPROC_BUCKET/scripts/* /usr/local/bin/miniconda/lib/python2.7
fi
if gsutil -q stat "gs://$DATAPROC_BUCKET/modules/**"; then
echo "Pulling modules directory to cluster master and worker nodes..."
gsutil -m cp -r gs://$DATAPROC_BUCKET/modules/* /usr/local/bin/miniconda/lib/python2.7
fi
echo "Completed installing Jupyter!"
# Install Jupyter extensions (if desired)
# TODO: document this in readme
if [[ ! -v $INSTALL_JUPYTER_EXT ]]
then
INSTALL_JUPYTER_EXT=false
fi
if [[ "$INSTALL_JUPYTER_EXT" = true ]]
then
echo "Installing Jupyter Notebook extensions..."
./dataproc-initialization-actions/jupyter/internal/bootstrap-jupyter-ext.sh
echo "Jupyter Notebook extensions installed!"
fi

Did you try to execute the following command ?
pip install ipython[numpy]