Frisson Labs
Show HN: Lessons from building a SIMA2 style agent in Roblox
Last verified:
What is Frisson Labs?
This is an experimental AI agent project from Frisson Labs that attempts to replicate Google DeepMind's SIMA2 (a general embodied AI agent for virtual worlds) on a
Frisson Labs pricing
Pricing model: Freemium
This is a free experimental research prototype from Frisson Labs, not a commercial product. There are no paid plans, subscriptions, or pricing tiers mentioned. The project is a blog post/experiment demonstrating what a budget AI agent stack can achieve, using free/accessible tools like Roblox and Google's Gemini API.
Frisson Labs pros
- Two-loop architecture separates fast action from slower reasoning effectively
- Uses affordable Gemini 3.1 Flash Lite for the fast action loop
- Gemini 2.5 Pro provides strategic goal updates in the reasoning loop
- Recorded human movement skills make avatar movement look player-like
- Named behaviors like explore_open_path and recover_if_stuck are reusable
- Continuous movement sequences feel more natural than single keypresses
- 3.5s average latency is reasonable for the fast loop
- Works in Roblox, an accessible and popular gaming platform
- Includes debug logging overlay showing plan, actions, and key presses
- Human patch loop enables iterative improvement through replay review
- AVOs unsafe clicks and blocks Robux-related actions automatically
- Interruptible action sequences allow flexibility during execution
- Successfully demonstrates screen observation and keyboard/mouse control
- Records sessions with ffmpeg for detailed replay analysis
- Shows glimmers of human-like play including bunny hops and A/D feathering
Frisson Labs cons
- Still plays worse than a 4-year-old at the game
- Agent keeps drifting wrong way and getting pinned against objects
- Model doesn't真正 understand the game despite reading the screen
- Slower reasoning loop takes 13-20 seconds per strategy call, too slow for steering jumps
- Requires hardcoded recovery skills because model repeats low-level mistakes
- No knowledge of traversable terrain, enemies, or affordable gates
- Doesn't know if last recovery actually changed position
- Movement still looks obviously bot-like despite improvements
- Model keeps making full 360 camera turns like an idiot
- No world model to learn what changed or what is blocked
- Learning loop is manual (watching replays and patching) not self-learning like SIMA2
- Can't transfer skills between different games or environments
- Reflexes required hardcoded controller to block bad actions before Gemini votes
- Still fundamentally an LLM-blended behavior tree, not true agent learning
- Doesn't know location relative to spawn, targets, or gates
Frequently asked questions about Frisson Labs
What is SIMA2 from Temu?
It's Frisson Labs' experimental attempt to build a SIMA2-like AI agent on a budget (like shopping on Temu). SIMA2 is Google DeepMind's best public example of game-playing AI agents that learn from human play, act from screen observations, use keyboard/mouse controls, and reason about long-term goals. The Temu version approximates this using Roblox, Gemini APIs, and recorded human movement skills.
What models power the two-loop system?
The fast action loop uses Gemini 3.1 Flash Lite to plan the next few seconds of movement actions. The slower reasoning loop uses Gemini 2.5 Pro to update goals, track failures, and direct the fast loop. Strategy calls with Pro took 13.2s, 20.2s, and 16.6s in the latest run.
What game was used for testing?
Slime RNG on Roblox was chosen because it was trending and had idiot-proof gameplay. The agent clicks the roll button, AFK farms, and unlocks new areas. The Canyon gate cost 216M coins while the agent had only 21.7M coins.
What are the learned movement skills?
These are recorded human keyboard traces turned into reusable movement patterns like human_forward_left_jump_chain_zigzag_00, human_forward_left_jump_chain_zigzag_01, and human_forward_right_jump_chain_zigzag_02. The agent picks from these instead of generating movement from scratch, making it look more player-like.
How does the agent recover when stuck?
The system uses a recover_if_stuck skill with hardcoded camera turns (holding right arrow for a couple seconds) because the model kept getting pinned in trees and walls. Even with the reasoning loop noticing the problem, the model kept making the same low-level mistake, requiring this patch.
What was the evaluation method?
The least scientific evaluation possible: watching replays as a team and arguing about how human the movement felt. There's no clean metric, just vibe evals judging whether it looks like a real player based on timing, intent, hesitation, recovery, and weird keyboard habits.
What are the next incremental improvements?
Add a progress signal to detect if avatar actually moved, blocked-direction memory to avoid repeating failed recovery directions, a tiny affordance map for gates/buttons/enemies/pickups, a learned movement selector based on context instead of RNG, and keep the replay overlay showing planner thoughts and where runs fell apart.
What future approach would enable a real jump?
Stop treating the world model as a note in the prompt and build one directly using JEPA-style/latent-world models that learn what changed, what is blocked, what matters, and what is likely to happen after the next action. This is coming soon in a later post.
What files and resources are available?
The project includes an annotated YouTube recording of the latest run, an events.json file with the latest event log, local keyboard traces from prototype runs for human movement examples, and ffmpeg recordings with debug logs overlaid showing the planner's intent, action batches, keys fired, and walk positions.