Ppo
Watch a neural net learn to play Snake
Last verified:
What is Ppo?
tinyppo-snake is an interactive browser-based demonstration that shows a neural network learning to play the classic Snake game using reinforcement learning. The tool runs entirely in your web browser using WebGPU acceleration and tinygrad, a minimalist deep learning framework. It implements Proximal Policy Optimization (PPO), a state-of-the-art reinforcement learning algorithm, to train the AI agent in real-time.
Key features include real-time training visualization with a 3D renderer showing the snake game, live parameter updates without restarting training, the ability to switch between training mode and watching a trained policy play, configurable grid sizes (1×1, 2×2, 3×3), adjustable simulation delay (40ms default), and support for multiple parallel training runs with different learning rates. The system processes 384 parallel snakes simultaneously with 24 timesteps per rollout, generating 9,216 transitions before performing PPO updates across 3 epochs with 4 mini-batches each.
This tool is designed for machine learning students and educators who want to visualize reinforcement learning in action, researchers interested in browser-based ML demos, developers exploring WebGPU for compute tasks, and anyone curious about how neural networks learn through trial and error. The 10×10 board with a 104-dimensional observation space provides a challenging but tractable environment for demonstrating PPO's capabilities.
The reward structure encourages exploration and survival: -1 for each step, +100 for eating an apple, and +1000 for winning (filling the board). The neural network uses action masking to prevent illegal U-turns, value clipping for stability, and KL early stopping to prevent policy collapse. With proper training, the agent can achieve peak scores around 3959 on the 6×6 board.
Ppo pricing
Pricing model: Freemium
Free - This is an open-source demonstration with no paid tiers. The tool runs entirely in-browser with no server costs or subscription fees.
Ppo pros
- Runs entirely in browser with no installation required
- Real-time visualization of neural network training process
- Uses WebGPU for fast GPU-accelerated computation
- Built on tinygrad, a minimal and educational deep learning framework
- Live parameter updates without restarting training
- Switch between training mode and trained-policy rollouts
- Multiple configurable grid sizes (1×1, 2×2, 3×3)
- Adjustable simulation delay for slower/faster viewing
- Supports multiple parallel training runs simultaneously
- Learning rate sweep preset (1e-3, 3e-3, 1e-2)
- 384 parallel snakes for efficient data collection
- Open-source implementation on GitHub
- Works on mobile devices with WebGPU support
- Educational value for understanding PPO algorithm
- Action masking prevents illegal U-turn moves
- Shows detailed training metrics (avg score, peak, roll/s, episodes)
Ppo cons
- Requires WebGPU support (not available in all browsers)
- Some antivirus software flags it as suspicious activity
- Optimized for speed over training stability
- Linux and Windows supported but NetBSD has issues
- Training temporarily drops when switching to watch mode
- Limited to Snake game demonstration only
- Requires modern browser with WebGPU enabled
- No saved model persistence between sessions
Frequently asked questions about Ppo
What algorithm does this use for training?
The tool uses Proximal Policy Optimization (PPO), a popular reinforcement learning algorithm. Specifically, it uses 3 epochs × 4 mini-batches per rollout, with GAE parameters γ=0.99 and λ=0.95, AdamW optimizer with weight decay 0.01, ratio clip ε=0.1, gradient norm clipping at 0.5, and entropy bonus of 0.00833.
Does this work on mobile phones?
Yes, the creator confirmed it works on phones. The tinygrad WebGPU implementation is designed to be cross-platform and runs on any device with WebGPU support, including mobile browsers that support WebGPU.
What is the neural network architecture?
The network has a 104-dimensional input (flat 10×10 board = 100 cells + 4-dim previous action one-hot encoding). It uses fully connected layers for both policy (fc_pi) and value (fc_v) heads. The policy network is zero-initialized for uniform opening policy, while the value network uses tinygrad's default Kaiming initialization.
How many parallel snakes are running?
The system runs N=384 parallel snakes simultaneously. Each snake takes T=24 timesteps per rollout, generating 9,216 total transitions (384 × 24) before performing PPO updates.
What reward structure is used?
The reward structure is: -1 for each step (discourages stalling), +100 for eating an apple (dot), and +1000 for winning (filling the entire board). The theoretical maximum score on a 6×6 board is approximately 4000 points.
Can I change the learning rate?
Yes, there are preset options including a learning rate sweep with three values (1e-3, 3e-3, 1e-2), a preset with 3 seeds at default learning rate, a 2-seed option, and a custom configuration where you can set your own parameters.
What grid sizes are available?
The tool offers three grid size configurations: 1×1, 2×2, and 3×3. The main demonstration uses a 10×10 board for the observation space, but you can configure multiple smaller grid environments to run in parallel.
Why does my antivirus flag this site?
Some antivirus software (like Bitdefender users report) may show a warning about suspicious activity, but other users confirm Bitdefender shows it as clean. This appears to be a false positive since the site is a legitimate open-source ML demo hosted on gradexp.xyz.
What browsers are supported?
The tool requires WebGPU support, which means you need a browser with WebGPU enabled. Chrome and other browsers with the Dawn WebGPU engine support it. WebGPU is not yet available in all release or late beta builds, so you may need to enable it manually or use a compatible browser version.
What is the best score achieved?
According to the live metrics shown on the Hacker News post, the demo achieved a peak score of 3959.3 (best window) with an average of -4.6 over the last 500 episodes. The agent has trained on over 562,749 episodes at the time of posting, with a rollout speed of 20.68 rolls per second.