Skip to content

JorgeIvanJH/torch-points3d

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1,852 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Project Instructions

Here I'll explain all relevant configurations, and execution processes to run the pointcloud experiments for the "Single Object Segmentation In Point Clouds" project by Jorge Ivan Jaramillo Herrera (s2739025). This file overides original information of the TorchPoints3D framework, I'm putting first the configuration and executing processes for my implementation, and at the end the original information that was in this .md file.

Standardised Installation Guide for MinkowskiEngine + TorchPoints3D

The installation process of the TorchPoints3D framework was particularly challenging due to library incompatibility issues and the lack of recent official support from its original authors.

To ensure compatibility, the original requirements.txt had to be revised and cleaned, resulting in a new file named requirements_clean.txt.

The MinkowskiEngine was installed directly from its official repository to guarantee proper alignment with CUDA and PyTorch versions. Additionally, TorchPoints3D was not installed as a standard pip package; instead, it was executed directly from the root of its cloned repository.

This setup preserves the complete functionality of the framework, including support for the MiniMarket dataset developed for this project, as well as the associated experimentation and visualization code.

1. Verify SUDO access and Nvidia GPU

SUDO is required to modify cuda versioning, an NVidia GPU is mandatory to run Minkowski Engine, leveraged in the torchpoints3d for sparce matrix operations.

sudo whoami
nvidia-smi

2. Install Miniconda

cd ~
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
source ~/.bashrc
conda --version

3. Create Conda Environment

conda create -n minkowski python=3.8 -y
conda activate minkowski
which python
python --version

4. CUDA Toolkit (11.8)

Install CUDA 11.8 from NVIDIA’s runfile: https://developer.nvidia.com/cuda-11-8-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=20.04&target_type=runfile_local which as checked on 20/08/2025 was:

wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run
sudo sh cuda_11.8.0_520.61.05_linux.run

Then:

export CUDA_HOME=/usr/local/cuda-11.8
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH

$CUDA_HOME/bin/nvcc --version

5. Install PyTorch with CUDA 11.8

pip install torch==2.1.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

6. Install OpenBLAS (needed for Minkowski)

conda install openblas-devel -c anaconda

7. Build MinkowskiEngine

# Go to home (or wherever you keep projects)
cd ~  

# Clone MinkowskiEngine repository
git clone https://github.com/NVIDIA/MinkowskiEngine.git
cd MinkowskiEngine

# Install with OpenBLAS + CUDA support
python setup.py install \
  --blas_include_dirs=${CONDA_PREFIX}/include \
  --blas=openblas \
  --force_cuda

8. Install TorchPoints3D

# Dependencies
conda install openblas-devel -c anaconda
pip install hydra-core==1.1 omegaconf==2.1.1 pyyaml

# Get source
git clone https://github.com/nicolas-chaulet/torch-points3d.git
cd torch-points3d

# Requirements (cleaned)
pip install -r requirements_clean.txt

9. Fix Compatibility Issues

# Reinstall torch & geometric packages for CUDA 11.8
pip uninstall torch torchvision torchaudio torch-scatter torch-sparse torch-cluster -y
pip install torch==2.1.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# Install PyG extensions
pip install torch-scatter torch-sparse torch-cluster torch-spline-conv \
  -f https://data.pyg.org/whl/torch-2.1.0+cu118.html

# Numba + LLVM fixes
conda install -c conda-forge numba=0.56.4 llvmlite=0.39.1

Dataset

In my project, I ran multiple experiments both with artificial and real life scenes that were processed into HDF5 files, compatible to be read from the TorchPoints3D framework in the added support for the MiniMarket dataset. The support for this dataset, located at "torch-points3d/torch_points3d/datasets/segmentation/minimarket.py", following the configuration in "torch-points3d/conf/data/segmentation/minimarket.yaml", expects the training, validation, and test datasets to be stored in the directory "torch-points3d/data/minimarket/raw" with the following names:

  • "minimarket_train.h5" for the training dataset
  • "minimarket_valid.h5" for the validation dataset
  • "minimarket_test.h5" for the testing dataset

To specify the training, validation, and test set:

Download and place the already preprocessed HDF5 files located in: https://uoe-my.sharepoint.com/:f:/g/personal/s2739025_ed_ac_uk/EoUDFp6VWRhMs-YVGy8GeT4BLGv9cqTyJ_ShE6nAnh9Elg?e=a20Xx4 inside "torch-points3d/data/minimarket/raw", and rename accordingly. These are the processed and ready datasets in HDF5 file format we processed through the pipeline in https://github.com/JorgeIvanJH/Point-Cloud-Segmentation-Benchmarking. They correspond to:

# original Test Scenes with manual ground truth (not compatible):
all_test_scenes.h5 

# 12000 Training Artificial Segmentation scenes
ketchup_heinz_400ml_segmentation_numPoints_2048_minObjects_1_maxObjects_10_numrientations_4.h5
# 1200 Validation Artificial Segmentation scenes
ketchup_heinz_400ml_segmentation_numPoints_2048_maxObjects_10_orientations_1.h5

# 800 Training Real-Life segmentation scenes
augmentedscenes[0, 2, 3, 4, 5, 6, 7, 8]_orientations10_downsamplings10_maxpointsinbox100000.h5
# 100 Validation Real-Life segmentation scenes
augmentedscenes[1]_orientations10_downsamplings10_maxpointsinbox100000.h5
# 100 Testing Real-Life segmentation scenes
augmentedscenes[9]_orientations10_downsamplings10_maxpointsinbox100000.h5

Within "torch-points3d/data/minimarket/raw", change the .h5 file names respectively to: "minimarket_train.h5", "minimarket_valid.h5", or "minimarket_test.h5", according to which of the above is the dataset that you want to use for training, validation, or testing, respectively.

Experiment Execution

The experiment execution is based on the configuration in "torch-points3d/conf/config.yaml", which calls configurations in subfolders for models, dataset, training, tracking, etc.

To run the experiments:

Having activated the conda environment, just run:

poetry run python torch-points3d/train.py

To change the model implemented, modify in torch-points3d/conf/config.yaml:

  - models: segmentation/ppnet # Options: [_/pointnet, _/pointnet2, _/ppnet, _/kpconv, _/rsconv]

with its respective

model_name: PPNet # Options: [PointNet, pointnet2ms, PPNet, KPConvPaper, RSConv_MSN]

which points out the specific version (model_name) of the architecture for pointcloud segmentation in (models) located in "torch-points3d/conf/models/segmentation/_____.yaml"

THE MAIN ADVANTAGE OF THIS PROJECT is the chance of further expanding experimentation on other 5 architectures on top of the 5 presented in the project. These include: pointcnn, pvcnn, randlanet, sparseconv3d, and minkowski_baseline, as found in each of the configuration files at "torch-points3d/conf/models/segmentation".

Executing Experimentations:

When the whole configuration is defined as desired, run:

poetry run python train.py

An api key for WandB (https://wandb.ai/site/) will likely be requested, provide one from your personal account you require online tracking of the training, or set

wandb:
  log: False

in "torch-points3d/conf/training/default.yaml" to ignore online tracking.

The resulting metrics, and model weights will be stored in "torch-points3d/outputs", in folders named as YYYY-MM-DD/HH-MM-SS.

Performance Visualizations:

To use the model resulting from one experimentation in particular, ran and whose results are already found in the "torch-points3d/outputs" folder, to test on scenes individually, you will have to head to the "torch-points3d/notebooks" folder, which contains isolated folder for the data and to store individual inferences.

Similar to how the HDF5 files were added as "minimarket_train.h5", "minimarket_valid.h5", or "minimarket_test.h5, you will have to save 3 copies of the same HDF5 files with the 3 names at "torch-points3d/notebooks/data/minimarket/raw". Then check the script at "torch-points3d/notebooks/inference_tests.py", there, specify the path to the checkpoint folder in the "checkpoint_path" variable, and run:

python torch-points3d/notebooks/inference_tests.py

That will proces the dataset similarly as in the experiments, make inferences and save the visualizations to "torch-points3d/notebooks/outputs" that were presented in the final report.

Below follows the original README for TorchPoints3D:

PyPI version codecov Actions Status Documentation Status slack

This is a framework for running common deep learning models for point cloud analysis tasks against classic benchmark. It heavily relies on Pytorch Geometric and Facebook Hydra.

The framework allows lean and yet complex model to be built with minimum effort and great reproducibility. It also provide a high level API to democratize deep learning on pointclouds. See our paper at 3DV for an overview of the framework capacities and benchmarks of state-of-the-art networks.

Overview

Requirements

  • CUDA 10 or higher (if you want GPU version)
  • Python 3.7 or higher + headers (python-dev)
  • PyTorch 1.8.1 or higher (PyTorch >= 1.9 is recommended)
  • A Sparse convolution backend (optional) see here for installation instructions

For a more seamless setup, it is recommended to use Docker. This approach ensures compatibility and eases the installation process, particularly when working with specific versions of CUDA and PyTorch. You can pull the appropriate Docker image as follows:

docker pull pytorch/pytorch:1.10.0-cuda11.3-cudnn8-devel

After setting up the environment (either natively or through Docker), install the required Python package using pip:

pip install torch-points3d

Project structure

├─ benchmark               # Output from various benchmark runs
├─ conf                    # All configurations for training nad evaluation leave there
├─ notebooks               # A collection of notebooks that allow result exploration and network debugging
├─ docker                  # Docker image that can be used for inference or training
├─ docs                    # All the doc
├─ eval.py                 # Eval script
├─ find_neighbour_dist.py  # Script to find optimal #neighbours within neighbour search operations
├─ forward_scripts         # Script that runs a forward pass on possibly non annotated data
├─ outputs                 # All outputs from your runs sorted by date
├─ scripts                 # Some scripts to help manage the project
├─ torch_points3d
    ├─ core                # Core components
    ├─ datasets            # All code related to datasets
    ├─ metrics             # All metrics and trackers
    ├─ models              # All models
    ├─ modules             # Basic modules that can be used in a modular way
    ├─ utils               # Various utils
    └─ visualization       # Visualization
├─ test
└─ train.py                # Main script to launch a training

As a general philosophy we have split datasets and models by task. For example, datasets has five subfolders:

  • segmentation
  • classification
  • registration
  • object_detection
  • panoptic

where each folder contains the dataset related to each task.

Methods currently implemented

Please refer to our documentation for accessing some of those models directly from the API and see our example notebooks for KPconv and RSConv for more details.

Available Tasks

Tasks

Examples

Classification / Part Segmentation


Segmentation


Object Detection

Panoptic Segmentation

Registration

Available datasets

Segmentation

* S3DIS 1x1
* S3DIS Room
* S3DIS Fused - Sphere | Cylinder

Object detection and panoptic

* S3DIS Fused - Sphere | Cylinder

Registration

Classification

3D Sparse convolution support

We currently support Minkowski Engine > v0.5 and torchsparse >= v1.4.0 as backends for sparse convolutions. Those packages need to be installed independently from Torch Points3d, please follow installation instructions and troubleshooting notes on the respective repositories. At the moment MinkowskiEngine see here (thank you Chris Choy) demonstrates faster training. Please be aware that torchsparse is still in beta and does not support CPU only training.

Once you have setup one of those two sparse convolution framework you can start using are high level to define a unet backbone or simply an encoder:

from torch_points3d.applications.sparseconv3d import SparseConv3d

model = SparseConv3d("unet", input_nc=3, output_nc=5, num_layers=4, backend="torchsparse") # minkowski by default

You can also assemble your own networks by using the modules provided in torch_points3d/modules/SparseConv3d/nn. For example if you wish to use torchsparse backend you can do the following:

import torch_points3d.modules.SparseConv3d as sp3d

sp3d.nn.set_backend("torchsparse")
conv = sp3d.nn.Conv3d(10, 10)
bn = sp3d.nn.BatchNorm(10)

Mixed Precision Training

Mixed precision allows for lower memory on the GPU and slightly faster training times by performing the sparse convolution, pooling, and gradient ops in float16. Mixed precision training is currently supported for CUDA training on SparseConv3d networks with the torchsparse backend. To enable mixed precision, ensure you have the latest version of torchsparse with pip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git. Then, set training.enable_mixed=True in your training configuration files. If all the conditions are met, when you start training you will see a log entry stating:

[torch_points3d.models.base_model][INFO] - Model will use mixed precision

If, however, you try to use mixed precision training with an unsupported backend, you will see:

[torch_points3d.models.base_model][WARNING] - Mixed precision is not supported on this model, using default precision...

Adding your model to the PretrainedRegistry.

The PretrainedRegistry enables anyone to add their own pre-trained models and re-create them with only 2 lines of code for finetunning or production purposes.

  • [You] Launch your model training with Wandb activated (wandb.log=True)
  • [TorchPoints3d] Once the training finished, TorchPoints3d will upload your trained model within our custom checkpoint to your wandb.
  • [You] Within PretainedRegistry class, add a key-value pair within its attribute MODELS. The key should be describe your model, dataset and training hyper-parameters (possibly the best model), the value should be the url referencing the .pt file on your wandb.

Example: Key: pointnet2_largemsg-s3dis-1 and URL value: https://api.wandb.ai/files/loicland/benchmark-torch-points-3d-s3dis/1e1p0csk/pointnet2_largemsg.pt for the pointnet2_largemsg.pt file. The key desribes a pointnet2 largemsg trained on s3dis fold 1.

  • [Anyone] By using the PretainedRegistry class and by providing the key, the associated model weights will be downloaded and the pre-trained model will be ready to use with its transforms.
[In]:
from torch_points3d.applications.pretrained_api import PretainedRegistry

model = PretainedRegistry.from_pretrained("pointnet2_largemsg-s3dis-1")

print(model.wandb)
print(model.print_transforms())

[Out]:
=================================================== WANDB URLS ======================================================
WEIGHT_URL: https://api.wandb.ai/files/loicland/benchmark-torch-points-3d-s3dis/1e1p0csk/pointnet2_largemsg.pt
LOG_URL: https://app.wandb.ai/loicland/benchmark-torch-points-3d-s3dis/runs/1e1p0csk/logs
CHART_URL: https://app.wandb.ai/loicland/benchmark-torch-points-3d-s3dis/runs/1e1p0csk
OVERVIEW_URL: https://app.wandb.ai/loicland/benchmark-torch-points-3d-s3dis/runs/1e1p0csk/overview
HYDRA_CONFIG_URL: https://app.wandb.ai/loicland/benchmark-torch-points-3d-s3dis/runs/1e1p0csk/files/hydra-config.yaml
OVERRIDES_URL: https://app.wandb.ai/loicland/benchmark-torch-points-3d-s3dis/runs/1e1p0csk/files/overrides.yaml
======================================================================================================================

pre_transform = None
test_transform = Compose([
    FixedPoints(20000, replace=True),
    XYZFeature(axis=['z']),
    AddFeatsByKeys(rgb=True, pos_z=True),
    Center(),
    ScalePos(scale=0.5),
])
train_transform = Compose([
    FixedPoints(20000, replace=True),
    RandomNoise(sigma=0.001, clip=0.05),
    RandomRotate((-180, 180), axis=2),
    RandomScaleAnisotropic([0.8, 1.2]),
    RandomAxesSymmetry(x=True, y=False, z=False),
    DropFeature(proba=0.2, feature='rgb'),
    XYZFeature(axis=['z']),
    AddFeatsByKeys(rgb=True, pos_z=True),
    Center(),
    ScalePos(scale=0.5),
])
val_transform = Compose([
    FixedPoints(20000, replace=True),
    XYZFeature(axis=['z']),
    AddFeatsByKeys(rgb=True, pos_z=True),
    Center(),
    ScalePos(scale=0.5),
])
inference_transform = Compose([
    FixedPoints(20000, replace=True),
    XYZFeature(axis=['z']),
    AddFeatsByKeys(rgb=True, pos_z=True),
    Center(),
    ScalePos(scale=0.5),
])
pre_collate_transform = Compose([
    PointCloudFusion(),
    SaveOriginalPosId,
    GridSampling3D(grid_size=0.04, quantize_coords=False, mode=mean),
])

Developer guidelines

Setting repo

We use Poetry for managing our packages. In order to get started, clone this repositories and run the following command from the root of the repo

poetry install --no-root

This will install all required dependencies in a new virtual environment.

Activate the environment

poetry shell

You can check that the install has been successful by running

python -m unittest -v

For pycuda support (only needed for the registration tasks):

pip install pycuda

Getting started: Train pointnet++ on part segmentation task for dataset shapenet

poetry run python train.py task=segmentation models=segmentation/pointnet2 model_name=pointnet2_charlesssg data=segmentation/shapenet-fixed

And you should see something like that

logging

The config for pointnet++ is a good example of how to define a model and is as follow:

# PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space (https://arxiv.org/abs/1706.02413)
# Credit Charles R. Qi: https://github.com/charlesq34/pointnet2/blob/master/models/pointnet2_part_seg_msg_one_hot.py

pointnet2_onehot:
  architecture: pointnet2.PointNet2_D
  conv_type: "DENSE"
  use_category: True
  down_conv:
    module_name: PointNetMSGDown
    npoint: [1024, 256, 64, 16]
    radii: [[0.05, 0.1], [0.1, 0.2], [0.2, 0.4], [0.4, 0.8]]
    nsamples: [[16, 32], [16, 32], [16, 32], [16, 32]]
    down_conv_nn:
      [
        [[FEAT, 16, 16, 32], [FEAT, 32, 32, 64]],
        [[32 + 64, 64, 64, 128], [32 + 64, 64, 96, 128]],
        [[128 + 128, 128, 196, 256], [128 + 128, 128, 196, 256]],
        [[256 + 256, 256, 256, 512], [256 + 256, 256, 384, 512]],
      ]
  up_conv:
    module_name: DenseFPModule
    up_conv_nn:
      [
        [512 + 512 + 256 + 256, 512, 512],
        [512 + 128 + 128, 512, 512],
        [512 + 64 + 32, 256, 256],
        [256 + FEAT, 128, 128],
      ]
    skip: True
  mlp_cls:
    nn: [128, 128]
    dropout: 0.5

Inference

Inference script

We provide a script for running a given pre trained model on custom data that may not be annotated. You will find an example of this for the part segmentation task on Shapenet. Just like for the rest of the codebase most of the customization happens through config files and the provided example can be extended to other datasets. You can also easily create your own from there. Going back to the part segmentation task, say you have a folder full of point clouds that you know are Airplanes, and you have the checkpoint of a model trained on Airplanes and potentially other classes, simply edit the config.yaml and shapenet.yaml and run the following command:

python forward_scripts/forward.py

The result of the forward run will be placed in the specified output_folder and you can use the notebook provided to explore the results. Below is an example of the outcome of using a model trained on caps only to find the parts of airplanes and caps.

resexplore

Containerizing your model with Docker

Finally, for people interested in deploying their models to production environments, we provide a Dockerfile as well as a build script. Say you have trained a network for semantic segmentation that gave the weight <outputfolder/weights.pt>, the following command will build a docker image for you:

cd docker
./build.sh outputfolder/weights.pt

You can then use it to run a forward pass on a all the point clouds in input_path and generate the results in output_path

docker run -v /test_data:/in -v /test_data/out:/out pointnet2_charlesssg:latest python3 forward_scripts/forward.py dataset=shapenet data.forward_category=Cap input_path="/in" output_path="/out"

The -v option mounts a local directory to the container's file system. For example in the command line above, /test_data/out will be mounted at the location /out. As a consequence, all files written in /out will be available in the folder /test_data/out on your machine.

Profiling

We advice to use snakeviz and cProfile

Use cProfile to profile your code

poetry run python -m cProfile -o {your_name}.prof train.py ... debugging.profiling=True

And visualize results using snakeviz.

snakeviz {your_name}.prof

It is also possible to use torch.utils.bottleneck

python -m torch.utils.bottleneck /path/to/source/script.py [args]

Troubleshooting

Cannot compile certain CUDA Kernels or seg faults while running the tests

Ensure that at least PyTorch 1.8.0 is installed and verify that cuda/bin and cuda/include are in your $PATH and $CPATH respectively, e.g.:

$ python -c "import torch; print(torch.__version__)"
>>> 1.8.0

$ echo $PATH
>>> /usr/local/cuda/bin:...

$ echo $CPATH
>>> /usr/local/cuda/include:...

Undefined symbol / Updating Pytorch

When we update the version of Pytorch that is used, the compiled packages need to be reinstalled, otherwise you will run into an error that looks like this:

... scatter_cpu.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN3c1012CUDATensorIdEv

This can happen for the following libraries:

  • torch-points-kernels
  • torch-scatter
  • torch-cluster
  • torch-sparse

An easy way to fix this is to run the following command with the virtual env activated:

pip uninstall torch-scatter torch-sparse torch-cluster torch-points-kernels -y
rm -rf ~/.cache/pip
poetry install

CUDA kernel failed : no kernel image is available for execution on the device

This can happen when trying to run the code on a different GPU than the one used to compile the torch-points-kernels library. Uninstall torch-points-kernels, clear cache, and reinstall after setting the TORCH_CUDA_ARCH_LIST environment variable. For example, for compiling with a Tesla T4 (Turing 7.5) and running the code on a Tesla V100 (Volta 7.0) use:

export TORCH_CUDA_ARCH_LIST="7.0;7.5"

See this useful chart for more architecture compatibility.

Cannot use wandb on Windows

Raises OSError: [WinError 6] The handle is invalid / wandb: ERROR W&B process failed to launch Wandb is currently broken on Windows (see this issue), a workaround is to use the command line argument wandb.log=false

Exploring your experiments

We provide a notebook based pyvista and panel that allows you to explore your past experiments visually. When using jupyter lab you will have to install an extension:

jupyter labextension install @pyviz/jupyterlab_pyviz

Run through the notebook and you should see a dashboard starting that looks like the following:

dashboard

Contributing

Contributions are welcome! The only asks are that you stick to the styling and that you add tests as you add more features!

For styling you can use pre-commit hooks to help you:

pre-commit install

A sequence of checks will be run for you and you may have to add the fixed files again to the stashed files.

When it comes to docstrings we use numpy style docstrings, for those who use Visual Studio Code, there is a great extension that can help with that. Install it and set the format to numpy and you should be good to go!

Finaly, if you want to have a direct chat with us feel free to join our slack, just shoot us an email and we'll add you.

Citing

If you find our work useful, do not hesitate to cite it:

@inproceedings{
  tp3d,
  title={Torch-Points3D: A Modular Multi-Task Frameworkfor Reproducible Deep Learning on 3D Point Clouds},
  author={Chaton, Thomas and Chaulet Nicolas and Horache, Sofiane and Landrieu, Loic},
  booktitle={2020 International Conference on 3D Vision (3DV)},
  year={2020},
  organization={IEEE},
  url = {\url{https://github.com/nicolas-chaulet/torch-points3d}}
}

and please also include a citation to the models or the datasets you have used in your experiments!

About

Pytorch framework for doing deep learning on point clouds.

Resources

License

Stars

Watchers

Forks

Packages

 
 
 

Contributors

Languages

  • Python 81.2%
  • Jupyter Notebook 18.3%
  • Other 0.5%