Pointllm
[ECCV 2024 Best Paper Candidate & TPAMI 2025] PointLLM: Empowering Large Language Models to Understand Point Clouds
Last verified:
What is Pointllm?
PointLLM is a multi-modal large language model capable of understanding colored point clouds of objects. It processes colored object point clouds with human instructions and generates contextually appropriate responses, illustrating its grasp of point clouds and common sense. The model perceives object types, geometric structures, and appearance without concerns for ambiguous depth, occlusion, or viewpoint dependency.
PointLLM leverages a point cloud encoder with a powerful LLM backbone to effectively fuse geometric, appearance, and linguistic information. The point encoder extracts features from the input point cloud and projects them to the latent space of the LLM backbone. The LLM backbone processes sequences of point tokens and text tokens, and generates the predicted tokens as the output.
The tool is designed for researchers and developers working in 3D computer vision, robotics, and multimodal AI. It enables applications such as 3D object captioning, open-vocabulary 3D classification, and conversational interaction with 3D models. PointLLM is particularly useful for those working with Objaverse datasets and needing LLM-powered understanding of 3D point cloud data.
To enable training, the authors collected a novel dataset comprising 660K simple and 70K complex point-text instruction pairs for a two-stage training strategy. The model was accepted by ECCV 2024 as a Best Paper Candidate and demonstrates superior performance over existing 2D baselines, outperforming human annotators in over 50% of human-evaluated object captioning samples.
Pointllm pricing
Pricing model: Freemium
PointLLM is free and open-source under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. The code, checkpoints, datasets, and benchmarks are freely available on GitHub. There is no paid tier. The online Gradio demo was closed in April 2025, but users can run the demo locally on their own machines. Evaluation using GPT-4/ChatGPT costs approximately $1.5 to $2.2 USD for API calls.
Pointllm pros
- Understands colored point clouds directly without 2D projection
- Fuses geometric, appearance, and linguistic information effectively
- Outperforms human annotators in 50%+ of captioning tasks
- Superior performance over existing 2D baselines
- No concerns for ambiguous depth issues
- No concerns for occlusion problems
- No concerns for viewpoint dependency
- Generates contextually appropriate responses to human instructions
- 660K simple instruction pairs for robust training
- 70K complex GPT-4 generated instruction pairs
- Two-stage training strategy for better alignment
- Open-source code and checkpoints available
- ECCV 2024 Best Paper Candidate recognition
- Supports both 7B and 13B model variants
- Works with Objaverse colored point cloud dataset
Pointllm cons
- Online demo closed as of April 2025
- Requires significant GPU memory (14GB minimum for 7B)
- Only supports colored point clouds with 8192 points
- Non-commercial license (CC BY-NC-SA 4.0)
- Training requires downloading 77GB of point cloud data
- No official Windows support (tested on Ubuntu 20.04)
- CUDA 11.7 and specific PyTorch version required
- Traditional metrics like BLEU/ROUGE not recommended for evaluation
Frequently asked questions about Pointllm
What is PointLLM?
PointLLM is a multi-modal large language model capable of understanding colored point clouds of objects. It perceives object types, geometric structures, and appearance, and generates contextually appropriate responses when given human instructions about 3D point clouds.
What datasets were used to train PointLLM?
PointLLM was trained on a novel dataset comprising 660K simple and 70K complex point-text instruction pairs. Both datasets are based on the Objaverse dataset, with complex instructions generated using GPT-4.
What benchmarks does PointLLM use for evaluation?
PointLLM establishes two benchmarks: Generative 3D Object Classification and 3D Object Captioning. These are assessed through three methods: human evaluation, GPT-4/ChatGPT evaluation, and traditional metrics like BLEU, ROUGE, and METEOR.
How do I install PointLLM?
Clone the repository, create a conda environment with Python 3.10, install packages with pip install -e ., and install ninja and flash-attn for training. The code is tested on Ubuntu 20.04 with CUDA 11.7 and PyTorch 2.0.1.
What GPU memory do I need to run PointLLM?
GPU memory requirements vary by model and data type: PointLLM-7B needs 14GB (float16) or 28GB (float32), while PointLLM-13B needs 26GB (float16) or 52GB (float32). Using torch.bfloat16 is recommended if applicable.
Can I use PointLLM for commercial purposes?
No. PointLLM is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, which means it is only for non-commercial use.
What point cloud format does PointLLM accept?
PointLLM accepts point clouds with dimensions (N, 6), where the first three dimensions are xyz coordinates and the last three are RGB color values in range. The model is trained on 8192-point clouds.
Is there an online demo available?
No, the online Gradio demo was closed on April 21, 2025 because the serving machine is needed for other purposes. However, you can run the demo locally using the provided Gradio demo code.
What LLM backbone does PointLLM use?
PointLLM uses Vicuna-7B and Vicuna-13B checkpoints (v1.1 means Vicuna-v1.1). The model does not require downloading original LLaMA weights separately.
How does PointLLM compare to human annotators?
In human-evaluated object captioning tasks, PointLLM outperforms human annotators in over 50% of the samples, demonstrating superior performance in generating accurate 3D object descriptions.