Develop Reference

Installing Open3d-Ml with Pytorch (on MacOs)

I created a virtualenv with python 3.10 and installed open3d and PyTorch according to the instructions on open3d-ml webpage: Open3d-ML but when I tested it with import open3d.ml.torch I get the error:
Exception: Open3D was not built with PyTorch support!
Steps to reproduce
python3.10 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install open3d
pip install torch torchvision torchaudio
Error
% python -c "import open3d.ml.torch as ml3d"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/xx/.venv/lib/python3.10/site-packages/open3d/ml/torch/__init__.py", line 34, in <module>
raise Exception('Open3D was not built with PyTorch support!')
Exception: Open3D was not built with PyTorch support!
Environment:
% python3 --version
Python 3.10.9
% pip freeze
open3d==0.16.1
torch==1.13.1
torchaudio==0.13.1
torchvision==0.14.1
OS
macOS 12.6
Kernel Version: Darwin 21.6.0
I also checked below similar issues but they don't have answers:
https://github.com/isl-org/Open3D/discussions/5849
https://github.com/isl-org/Open3D-ML/issues/557
Open3D-ML and pytorch
According to this issue 5849 the problem can't be related only to MacOs because, in a docker with Ubuntu20.04, there is a similar error.
Does anyone know how we can tackle this?

You will need to enable PyTorch / Tensorflow when building with cmake - have a look at https://medium.com/#kidargueta/installing-open3d-ml-for-3d-computer-vision-with-pytorch-d640a6862e19 and http://www.open3d.org/docs/release/compilation.html
cmake -DBUILD_CUDA_MODULE=ON -DGLIBCXX_USE_CXX11_ABI=OFF -DBUILD_PYTORCH_OPS=ON -DBUILD_CUDA_MODULE=ON -DBUNDLE_OPEN3D_ML=ON -DOPEN3D_ML_ROOT=Open3D-ML -DBUILD_JUPYTER_EXTENSION:BOOL=ON -DBUILD_WEBRTC=ON -DOPEN3D_ML_ROOT=https://github.com/isl-org/Open3D-ML.git -DPython3_ROOT=/path/to/your/conda-env/bin/python ..

Finally, I decided to build Open3D from the source for Mac M1. I followed almost the official open3d page and thanks to this medium in one of the replies.
Build Open3d-ml with Pytorch on Mac M1
For the OS environment see the main question.
conda create --name open3d-ml-build python==3.10
conda activate open3d-ml-build
# install pytorch from pytorch.org
conda install pytorch torchvision torchaudio -c pytorch
# now clone open3d in a desired location
git clone --branch v0.16.1 git#github.com:isl-org/Open3D.git ./foo/open3d-0.16-build
cd open3d-0.16-build
mkdir build && cd build
git clone git#github.com:isl-org/Open3D-ML.git
Now make sure you are in the activeted conda env.
Build
(takes very long and a lot of memory)
Note on Mac M1 you don't have Cuda but Metal Performance Shaders (MPS) so I made CUDA Flags OFF in the cmake configuration.
which python3
>> /Users/XX/miniconda3/envs/open3d-ml-build/bin/python3
# in the build direcotry
cmake -DBUILD_CUDA_MODULE=OFF -DGLIBCXX_USE_CXX11_ABI=OFF \
-DBUILD_PYTORCH_OPS=ON -DBUILD_CUDA_MODULE=OFF \
-DBUNDLE_OPEN3D_ML=ON -DOPEN3D_ML_ROOT=Open3D-ML \
-DBUILD_JUPYTER_EXTENSION:BOOL=OFF \
-DPython3_ROOT=/Users/XX/miniconda3/envs/open3d-ml-build/bin/python3 ..
make -j$(sysctl -n hw.physicalcpu) [verbose=1]
If it fails, try it again or run it with verbose and look for fatal error.
Install
# Install pip package in the current python environment
make install-pip-package
# if error: Module Not found yapf
pip install yapf
# Create Python package in build/lib
make python-package
# Create pip wheel in build/lib
# This creates a .whl file that you can install manually.
make pip-package
sanity check
Again in the activated conda environment
# if not installed
pip install tensorboard
python3 -c "import open3d; import open3d.ml.torch"
pip freeze | grep torch
torch==1.13.1
torchaudio==0.13.1
torchvision==0.14.1
If you don't get any errors you should be good to go.

How can I use PyMOL API from Python?

I have PyMOL already installed on my Linux machine. I know it is installed because when I write pymol -cp pymol_api_test.py the script executes.
I want to run the following python script using python3:
import pymol
from pymol import cmd
print(cmd.align("/home/bbq_spatial/bbq_input_pdb/pdb1a6j.pdb",
"/home/bbq_spatial/bbq_output_pdb/pdb1a6j.pdb",
cycles=0, transform=0))
However, it doesn't run when I call it using python3:
user_name#server_name:~$ nano pymol_api_test.py
user_name#server_name:~$ python3 pymol_api_test.py
Traceback (most recent call last):
File "pymol_api_test.py", line 1, in <module>
import pymol
ModuleNotFoundError: No module named 'pymol'
user_name#server_name:~$
How can I resolve this?

Install conda (or pip) as a package manger for Python.
And then install pymol with the following commands:
conda install -c conda-forge -c schrodinger pymol-bundle
You can check your installed packages in your env with the following command:
conda list
Use this link to install conda on your Linux machine.
Pay attention,PyMol may require other dependencies on you system, check the official guides for installation.
UPDATE:
Check a specific pkg:
with pip: pip show <pkg>
with conda: conda show <pkg>
List all installed pkgs:
with pip: pip list
with conda: conda list

Make sure that your python interpreter has access to the pymol package. This can be done adding it to the PYTHONPATH environment variable.
$ export PYTHONPATH=/opt/pymol/lib/python3.10/site-packages/:$PYTHONPATH
In linux, the pymol command is not a binary but just a shell script to run the corresponding python package. So you can check it to get the correct path to be added.
$ cat $(which pymol)
#!/bin/sh
exec "/usr/bin/python3" "/opt/pymol/lib/python3.10/site-packages/pymol/__init__.py" "$#"

How can I install python packages in Docker?

I am trying to run a model written in Python and packaged in Docker. The model is available here: https://github.com/pennsignals/chime_sims
The model has a requirements.txt file and a number of python files. Requirements file is supposed to install a number of python packages, including configargparse. The docker file runs, but when it gets to the python code that uses configargparse, I get an error that such a package does not exist.
I added a line that I think should install it:
RUN pip -m install --upgrade configargparse
and then I import it in the python code. But I still get an error stating that no such package exits.
Here is the Docker code:
FROM continuumio/miniconda3
WORKDIR /chime_sims
RUN conda update -yq -n base -c defaults conda
RUN conda create -yq -n chime_sims python=3.7 pip pandas matplotlib scipy numpy seaborn
COPY ./requirements.txt .
FROM python
RUN python -m pip install --upgrade pip
RUN pip -m install --upgrade configargparse
RUN pip freeze > requirements.txt .
RUN py -m pip install -r requirements.txt
And here is the list from requirements.txt:
ConfigArgParse
configargparse
gitpython
seaborn
numpy
pandas
gvar
lsqfit
I suspect I'll have problems installing gvar and lsqfit if I ever get configargparse to work. I had trouble installing those packages outside of docker, but it finally worked.
Any ideas on how I can fix it?
thank you,
i.

how do I install a specific version of the rdkit library for Python2?

I need to install a version of the rdkit library released prior to 2019, when support for Python 2 was removed. This is needed to work with this library: https://github.com/brain-research/deep-molecular-massspec
I have downloaded the library from the git page, eg. https://github.com/rdkit/rdkit/releases/tag/Release_2018_09_1, and tried using pip to install from that.
sudo pip install rdkit-Release_2018_09_1b1.tar.gz
I get the following error:
Processing ./rdkit-Release_2018_09_1b1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
IOError: [Errno 2] No such file or directory: '/tmp/pip-ohIcaj-build/setup.py'
---------------------------------------- Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-ohIcaj-build
I have tried installing the specific version using pip too:
sudo pip install rdkit==2018.09.01
Which gives:
Collecting rdkit==2018.09.01 Could not find a version that satisfies
the requirement rdkit==2018.09.01 (from versions: ) No matching
distribution found for rdkit==2018.09.01
Can someone tell me how to do this?

#paisanco is correct, attempting to install rdkit with pip will not work. The easiest way to install rdkit is by using Anaconda unless you want to build from source.
If you have Anaconda installed you can create a python 2.7 virtual environment:
conda create --name test-env python=2.7
You can then activate it:
conda activate test-env
And then install the rdkit version you require:
conda install -c rdkit rdkit=2018.09.1
Using Python:
import rdkit
print rdkit.__version__
[Out]: '2018.09.1'

Create a new conda environment with python 2.7.15:
conda create -n py27_rdkit python=2.7.15 ipython
Activate environment (python2.7)
conda activate py27_rdkit
Now in the py27_protac environment, install older version of rdkit that won't gripe about python2.7:
conda install -c conda-forge rdkit rdkit=2018.09.1
The conda install command in answer above: 'conda install -c rdkit rdkit=2018.09.1' failed due numerous conflicts.

The problem is what you downloaded, according to that site, is a tar archive containing the source code for that library, not a pip package.
So attempting to install it using pip will not work.
The RDKit project home page gives other options for installing 1) from within an Anaconda conda virtual environment 2) from the source code (what you downloaded) for Windows, Linux, and Mac.
Those instructions are at RDKit installation instructions

conda create -n my_env python=3.7
conda activate my_env
conda install numpy matplotlib
conda install cmake cairo pillow eigen pkg-config
conda install boost-cpp boost py-boost
and download rdkit package https://anaconda.org/rdkit/rdkit/files
# finally
conda install rdkit-2020.09.1b1.0-py37hd50e099_1.tar.bz2

Install gdal using conda?

I used
conda install gdal
to install the GDAL packages. But I had the following error when importing the packages.
>>> from osgeo import gdal
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/danqing0703/anaconda/lib/python2.7/site-packages/osgeo/__init__.py", line 21, in <module>
_gdal = swig_import_helper()
File "/Users/danqing0703/anaconda/lib/python2.7/site-packages/osgeo/__init__.py", line 17, in swig_import_helper
_mod = imp.load_module('_gdal', fp, pathname, description)
ImportError: dlopen(/Users/danqing0703/anaconda/lib/python2.7/site-packages/osgeo/_gdal.so, 2): Library not loaded: libgdal.20.dylib
Referenced from: /Users/danqing0703/anaconda/lib/python2.7/site-packages/osgeo/_gdal.so
Reason: image not found
>>> from osgeo import ogr
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/danqing0703/anaconda/lib/python2.7/site-packages/osgeo/__init__.py", line 21, in <module>
_gdal = swig_import_helper()
File "/Users/danqing0703/anaconda/lib/python2.7/site-packages/osgeo/__init__.py", line 17, in swig_import_helper
_mod = imp.load_module('_gdal', fp, pathname, description)
ImportError: dlopen(/Users/danqing0703/anaconda/lib/python2.7/site-packages/osgeo/_gdal.so, 2): Library not loaded: libgdal.20.dylib
Referenced from: /Users/danqing0703/anaconda/lib/python2.7/site-packages/osgeo/_gdal.so
Reason: image not found
What should I do to have GDAL imported in Python?

For windows users (as of December 2015):
conda install gdal
conda upgrade numpy
Installing gdal will downgrade numpy, so then upgrade it back up.
I recently had occasion to use windows for a change and I was pleasantly surprised that gdal "works" easily now.
Windows+python+gis people worldwide should be celebrating this. (that gdal-python goes in easily on windows...not that windows is one step closer to linux ;) )

I just made the mistake of executing the previously proposed command in the base environment of Conda:
conda install -c conda-forge gdal
This took ages to "solve environment" and, in the end, found numerous conflicts which halted the installation.
Given that, I instead created a separate environment with:
conda create -n gdal python=3.8
And activated it with:
conda activate gdal
And then executed the first command (as well as all others listed in the documentation). This worked fast and without any errors.

you can also use the channel conda-forge
conda install -c conda-forge gdal
as suggested on the anaconda website.

I used
conda install -c conda-forge gdal
On a Fedora 30 machine and it lead me down a path of library conflict hell!
conda install gdal
worked on my first try

The following works reliably for me Ubuntu 20.04:
conda update conda
sudo apt-get install libgdal-dev gdal-bin
export CPLUS_INCLUDE_PATH=/usr/include/gdal
export C_INCLUDE_PATH=/usr/include/gdal
# check gdal version with
gdal-config --version
echo '
name: raster-pipeline
channels:
- defaults
dependencies:
- python=3.7.*
- cython
- numpy
- ipython
- libspatialindex
- libgdal=USE_GDAL_VERSION
- gdal=USE_GDAL_VERSION
- pip
- pip:
- numpy>=1.18.5
- GDAL==USE_GDAL_VERSION
- pyproj>=2.6.1.post1
- rasterio>=1.1.5
' > raster_pipeline.yml
conda env create -f raster_pipeline.yml -v
conda activate raster_pipeline
python -c "from osgeo import gdal"
conda deactivate

This works for me
> CONDA_SUBDIR=osx-64 conda create -n my_env python=3.7
> conda activate my_env
> conda env config vars set CONDA_SUBDIR=osx-64
> conda install gdal
Now use,
> python3
> from osgeo import gdal

I had the same problem, and after several days trying different solutions, I found the problem in conflict between Anaconda version and python version! if you have both Python and Anaconda on your system, then uninstall the python and use anaconda prompt to run this:
pip install gdal

This worked for me . Hope it is useful to someone . I am using Ubuntu 20.04: It install osgeo also with gdal
Step 1 : conda install -c conda-forge gdal
If there is still some error then
Step 2 : conda config --set channel_priority strict”
Step 3 : conda update gdal

For existing conda-env, I did do like below on Ubuntu 20.04:
conda update conda
sudo apt-get install libgdal-dev gdal-bin
$export CPLUS_INCLUDE_PATH=/usr/include/gdal
$export C_INCLUDE_PATH=/usr/include/gdal
$gdal-config --version
gdal3.0.4
$whereis pip
/usr/local/anaconda3/bin/pip # anaconda's pip
$/usr/local/anaconda3/bin/pip install setuptools==57.4.0
$/usr/local/anaconda3/bin/pip install gdal==3.0.4