"Be horse."
a diffusion language model on an M2 Air
Last verified:
What is "Be horse."?
"Be horse." simple_dlm is a custom PyTorch Diffusion Language Model implementation created by Daniel Boesch (Be Horse) that can be trained on consumer hardware like an M2 MacBook Air in just 2 hours. Unlike traditional autoregressive language models, it uses a diffusion approach where tokens are corrupted by replacing them with a MASK token, and the model learns to predict what was originally there through repeated denoising steps.
"Be horse." pricing
Pricing model: Freemium
Free and open-source under the repository on GitHub. No paid tiers or commercial licensing mentioned. The code is available at github.com/Encrux/simple_dlm for anyone to use, modify, and train on their own hardware.
"Be horse." pros
- Trains in just 2 hours on M2 MacBook Air
- Works on consumer Apple Silicon hardware without GPU cluster
- Non-autoregressive parallel decoding for faster generation
- Simple minimal implementation easy to understand
- Open-source and available on GitHub
- Single network handles all noise levels from 0 to 1
- mask_prob passed as input for flexibility
- Cross-entropy loss only on masked tokens for efficiency
- 128-character chunk training for manageable batch sizes
- Low entropy output through repeated denoising
- Educational value for learning diffusion concepts
- 20 denoising steps with configurable k parameter
- No numeric noise required - uses token masking
- Hot topic cutting-edge ML technology
- Good starting point for custom diffusion LM research
"Be horse." cons
- Limited to 128-character sequence length
- Requires understanding of diffusion concepts
- May produce lower quality than large autoregressive models
- Needs PyTorch knowledge to use effectively
- Only 15 stars on GitHub indicating niche adoption
- New/experimental technology with limited documentation
- Mask-based approach may lose some semantic information
- Trained on personal hardware may lack scale
Frequently asked questions about "Be horse."
What is a Diffusion Language Model?
Diffusion Language Models corrupt data with noise by replacing tokens with a [MASK] token, then train a model to reverse that corruption over many small steps. Unlike image diffusion using numeric noise, text diffusion uses token masking since text is discrete. The model learns to predict what was originally there, and decoding happens in parallel by starting with all [MASK] tokens and iteratively revealing them.
How long does training take?
Training takes approximately 2 hours on an M2 MacBook Air, which makes it accessible for individual researchers and enthusiasts without expensive GPU clusters.
What sequence length does it support?
The model grabs random 128-character chunks from training data for each training step, making it suitable for short text generation tasks.
How many denoising steps are needed for decoding?
The default is k=20 denoising steps. During decoding, all tokens start as [MASK] and the model commits more tokens with each step until the sequence is fully revealed.
What loss function is used for training?
Cross-entropy loss is used, but only on the masked tokens. This focuses the training signal on the tokens the model needs to predict rather than the entire sequence.
How does mask_prob work?
mask_prob is sampled from a uniform distribution U(0, 1) for each training chunk, determining what fraction of tokens get replaced with [MASK]. This value is passed into the model as an input, allowing one network to handle every noise level from barely-masked to fully-masked sequences.
Why use token masking instead of numeric noise?
Text is discrete, not continuous like images, so numeric noise doesn't work well. The fix is to give up on numeric altogether and corrupt the sequence by replacing tokens with a [MASK] token, which the model learns to predict.
Can I run this on my Mac?
Yes, the model was specifically trained on an M2 MacBook Air using PyTorch. Apple Silicon with Metal acceleration makes it feasible for individual developers to train diffusion language models without cloud GPU services.
Where can I get the code?
The code is available on GitHub at github.com/Encrux/simple_dlm. It is open-source and you can clone it to train your own model or study the implementation.
How does parallel decoding work?
With diffusion, you decode the entire sequence in parallel by starting with all [MASK] tokens and reaching a low entropy state (actual text) through repeated denoising over the same sequence. This contrasts with autoregressive models that generate one token at a time sequentially.