Shap-e
The GitHub repository openai/shap-e is an AI tool that can generate 3D objects based on either text or images. The tool lets users contribu...
Last verified:
What is Shap-e?
Shap-E is OpenAI's conditional generative model for creating 3D assets from text prompts or single images. Unlike previous 3D generative models that produce only a single output representation, Shap-E directly generates the parameters of implicit functions that can be rendered as both textured meshes and neural radiance fields (NeRFs). The model was introduced in the research paper
Shap-e pricing
Pricing model: Free
Shap-E is completely free and open-source under the MIT license. OpenAI released the model weights, inference code, and samples publicly on GitHub at https://github.com/openai/shap-e. Users can use, copy, modify, merge, publish, distribute, sublicense, and sell copies of the software without restriction. There are no paid plans or subscription tiers. The only potential cost is computational resources needed to run the model effectively, including access to a GPU (recommended NVIDIA V100 or similar) for generation. Installation requires pip install -e . and may require additional dependencies like PyTorch, PyTorch3D, and Blender (version 3.3.1 or higher) for the encode_model.ipynb notebook.
Shap-e pros
- Generates 3D assets in approximately 13 seconds on NVIDIA V100 GPU
- Directly generates implicit function parameters for multi-representation output
- Can render outputs as both textured meshes and neural radiance fields
- Works with both text prompts and single image inputs
- Converges faster than Point-E predecessor
- Reaches comparable or better sample quality than Point-E
- Open-source with MIT license for free use and modification
- Model weights and inference code publicly available on GitHub
- Does not require separate text-to-image model for text conditioning
- Generates complex and diverse 3D assets from simple prompts
- Produces softer edges and clearer shadows than predecessor
- Less pixilated output compared to Point-E
- Compact assets under 0.2 MB with low latency around 20 seconds
- Can be used for 3D printing with adequate validation
- Includes example notebooks for text-to-3D, image-to-3D, and model encoding
Shap-e cons
- Generated assets often have rough edges, holes, or blurry surface textures
- Struggles with complex prompts combining multiple objects
- Limited ability to compose concepts together
- Cannot properly bind multiple attributes to different objects
- May fail to produce correct amount of objects when requested
- Lacks fine details and appears rough or like old video game graphics
- Image-conditional model works best only with synthetic renderings
- Requires background removal from input images for best results
- Generation process is computationally intensive
- Models reflect training data style of simple cartoonish 3D assets
- Mesh geometry often has noticeable flaws and deformation
- Texture fidelity and realism remain significant issues
- No commercial infrastructure, API, or updates since 2023 release
- Does not specify exact size or resolution of generated models
- May pose risks for 3D printing tools and parts without safety testing
Frequently asked questions about Shap-e
What is Shap-E and what does it do?
Shap-E is a conditional generative model for 3D assets that directly generates the parameters of implicit functions. Unlike recent 3D generative models producing single output representations, Shap-E can render outputs as both textured meshes and neural radiance fields (NeRFs). It generates complex and diverse 3D assets in seconds when conditioned on text prompts or single images.
How fast is Shap-E compared to Point-E?
Shap-E is significantly faster than Point-E. Each sample takes roughly 13 seconds to generate on a single NVIDIA V100 GPU, while Point-E produced 3D models in one to two minutes on the same hardware. Shap-E also converges faster during training and reaches comparable or better sample quality despite modeling a higher-dimensional, multi-representation output space.
What input formats does Shap-E accept?
Shap-E accepts two input formats: text prompts (text-conditional) and synthetic view images (image-conditional). For text conditioning, users provide descriptive prompts like 'a chair that looks like an avocado.' For image conditioning, users should provide synthetic renderings with backgrounds removed for the best results. The repository includes sample_text_to_3d.ipynb and sample_image_to_3d.ipynb notebooks for each modality.
What output formats can Shap-E generate?
Shap-E generates implicit neural functions that can be rendered from arbitrary viewpoints or exported as triangle meshes. The outputs can be represented as both textured meshes and neural radiance fields (NeRFs). This multi-representation capability allows the generated 3D models to be incorporated into downstream 3D applications and potentially used for 3D printing with adequate validation.
How do I install and run Shap-E?
Install Shap-E with 'pip install -e .' The repository includes example notebooks: sample_text_to_3d.ipynb for text-to-3D generation, sample_image_to_3d.ipynb for image-to-3D generation, and encode_model.ipynb for encoding existing 3D models. For encode_model.ipynb, you need Blender version 3.3.1 or higher and must set the BLENDER_PATH environment variable. Additional dependencies may include PyTorch, PyTorch3D, pyyaml, and ipywidgets.
What are Shap-E's main limitations?
Shap-E has several known limitations: generated assets often have rough edges, holes, or blurry textures; the text-conditional model struggles with complex multi-object prompts; it has limited ability to compose concepts or bind multiple attributes to different objects; it may fail to produce the correct number of objects; outputs lack fine details and resemble old video game graphics; texture fidelity and realism remain issues; and the model reflects the cartoonish style of its training data.
Is Shap-E free to use commercially?
Yes, Shap-E is released under the MIT license, which permits free use, modification, distribution, and commercial use without restriction. The license allows you to use, copy, modify, merge, publish, distribute, sublicense, and sell copies of the software. However, the software is provided 'AS IS' without warranty, and OpenAI notes that using text-to-real objects for 3D printing tools and parts may pose potential risks without adequate validation or safety testing.
What hardware do I need to run Shap-E?
Shap-E is designed to run on NVIDIA GPUs. The paper reports generation times of approximately 13 seconds per sample on a single NVIDIA V100 GPU. While the exact hardware requirements aren't specified beyond needing GPU support for PyTorch, users should have access to a CUDA-compatible NVIDIA GPU for efficient generation. The computational process is intensive, so a reasonably powerful GPU is recommended for practical use.
Can Shap-E models be used for 3D printing?
There is a possibility for Shap-E created models to be used for 3D printing since outputs can be exported as triangle meshes. However, OpenAI explicitly warns that without adequate validation or safety testing, producing tools and parts from text-to-real objects may pose potential risks. Users should perform proper quality checks and safety validation before using generated models for functional 3D printing applications.
Who created Shap-E and when was it released?
Shap-E was created by Heewoo Jun and Alex Nichol from OpenAI. It was introduced in the research paper 'Shap-E: Generating Conditional 3D Implicit Functions' submitted to arXiv on May 3, 2023 (arXiv:2305.02463). The official code and model weights were released publicly on GitHub at https://github.com/openai/shap-e in April 2023, with the paper published in May 2023.