Dualdiffusion
Dual Diffusion is a generative diffusion model for music trained on video game soundtracks.
Last verified:
What is Dualdiffusion?
Dualdiffusion is Dual Diffusion is a generative diffusion model specifically designed for creating video game music. The model is still a work in progress and has not been officially released with public weights yet. It uses a custom architecture based on the EDM2 UNet and a custom 2-stage VAE design, with the most recent U3 model containing 500m parameters in the music diffusion model, 25m in the auto-encoder, and 15m in the diffusion decoder.
The model is conditioned on CLAP audio embeddings and is designed to be used with example audio for generation. It was trained for 460k steps on a single RTX 5090 GPU over 4 weeks. The training data consists of video game music from the mid-90s to present day, comprising 570k tracks. This is substantially larger than the older SNES dataset which had only 20k samples. The model supports various features including inpainting, out-painting to extend or prepend samples, and seamless loop generation through its webUI built with NiceGUI.
This tool is primarily for game music developers, indie game creators, music producers interested in chiptune or video game style music, and AI music enthusiasts who want to experiment with generative music models. The open-source nature means developers and researchers can study the code, modify it, or train their own models on custom datasets. The project was started in August 2023 with goals to familiarize with diffusion models, train a model from scratch on a single consumer GPU, and generate music worth listening to.
Key features include classifier-free guidance for improved quality and control, a custom webUI with spectrogram view and precise time cursor for audio editing, support for multiple GPUs and distributed training through huggingface accelerate, and complete dataset pre-processing code included for training on custom data. The model uses mel-scale spectrogram-based formats and multi-scale spectral loss for better quality.
Dualdiffusion pricing
Pricing model: Freemium
Free and open source under MIT license. The model weights are not yet publicly released as the author is continuing development. Full source code including training code, dataset pre-processing code, and webUI is available on GitHub. No paid plans - designed to run on single consumer GPU (RTX 4090/5090) that users must provide.
Dualdiffusion pros
- Generates video game music from the mid-90s to present day
- Open source with full code available on GitHub under MIT license
- Trained on single consumer GPU (RTX 5090/4090) - accessible hardware
- 570k track dataset - substantially larger than SNES-only version
- CLAP audio embedding conditioning for better audio-based control
- Inpainting support for editing existing audio sections
- Out-painting capability to extend or prepend generated samples
- Seamless loop generation option for game music
- Custom webUI with spectrogram view and precise time cursor
- 2-stage VAE design for improved latent interpretability
- Classifier-free guidance for major quality improvement
- Supports multiple GPUs and distributed training via huggingface accelerate
- Includes complete dataset pre-processing code for custom training
- Works on both Windows and Linux platforms
- EDM2 UNet architecture with multiple performance improvements
- Pre-encoded latents for increased training performance
- SwitchEMA technique improves validation loss with no overhead
Dualdiffusion cons
- Model still work in progress - weights not officially released yet
- Author not satisfied with performance yet, continuing development
- Audio quality varies from phono-realistic to 96kbps WMA quality
- Dataset includes many tracks with poor audio quality/codecs
- Vocals in generated tracks sound like Simlish, not intelligible
- Model not conditioned with lyrics or transcriptions for vocals
- No official releases published on GitHub yet
- Only 50 stars on GitHub - relatively small community
- Performance significantly better on Linux than Windows
- WebUI missing model switching feature and some debug information
Frequently asked questions about Dualdiffusion
What is Dual Diffusion?
Dual Diffusion is a generative diffusion model for video game music. It uses a custom architecture based on the EDM2 UNet and a custom 2-stage VAE design. The model is conditioned on CLAP audio embeddings and is being trained on music from a wide variety of video games from the mid-90s to present day.
Is the model ready to use?
No, the model is still a work in progress. The author is not entirely satisfied with the model's performance and will be continuing development until they have something worth releasing publicly. The model weights are not officially released yet.
Where can I find the code?
Full source code is available on GitHub at github.com/parlance-zz/dualdiffusion. The repository includes everything from web scraping and dataset preparation to training and testing the VAE and diffusion model, under an MIT license.
What hardware do I need to run this?
The model was designed to be trained on a single consumer GPU. The original SNES model used an RTX 4090, and the newer U3 model was trained on a single RTX 5090 GPU over 4 weeks. The code works on both Windows and Linux, though performance is significantly better on Linux.
What datasets were used for training?
The current model uses 570k tracks of video game music from the mid-90s to present day. Previously, the author trained a model exclusively on SNES/SFC music with ~20,000 samples. The new dataset includes tracks with various audio quality levels including poor codecs from Sega Saturn and PS1.
Can I generate music with vocals?
A small minority of tracks in the dataset contain vocals, but the model was not conditioned with lyrics or transcriptions. While you might occasionally hear intelligible words or phrases, most tracks with vocals tend to sound like Simlish (gibberish).
What audio quality can I expect?
Audio quality varies significantly from phono-realistic to 96kbps WMA quality. This variation is due to the training dataset including many tracks with poor audio quality and codecs from older consoles like Sega Saturn and PS1.
Does it support inpainting or editing?
Yes, the model supports inpainting for editing existing audio sections. The webUI includes a custom element with spectrogram view and precise time cursor for audio editing. Out-painting is also available to extend or prepend generated samples.
Can I train my own model on custom data?
Yes, the dataset pre/post-processing code is included in the repository with everything needed to train a new model on your own data. The training code supports multiple GPUs and distributed training through huggingface accelerate.
How do I listen to sample audio?
Demo audio from the model at various stages of training and development is available on the website at g-diffuser.com/dualdiffusion. Samples include both the U3 model (current) and U2 model (previous), plus an older SNES-only model.