Colinator
Colinator is an MCP-exposed REPL that gives LLMs direct programmatic control over live robots, enabling real-time vision and motor control through code generation.
Last verified:
What is Colinator?
Colinator is an MCP-exposed REPL (Read-Eval-Print Loop) system that gives large language models direct, programmatic access to control a real robot in real time. Instead of using Vision-Language-Action (VLA) models trained on massive datasets, Colinator leverages the LLM's existing ability to write code by letting it operate as a programmer in situ on actual hardware.
The system consists of three main components: an MCP server exposing tools like describe() and execute(code), a persistent Python REPL subprocess that maintains state across calls, and a robot proxy injected into the REPL as the 'robot' object. The LLM writes Python code against this live robot object, executes it, inspects results, and revises iteratively—just as a human developer would use a REPL. The robot object bridges to actual hardware via zmq/tcp or could sit atop ROS.
Key features include real-time closed-loop control (LLM can write programs that react to the environment), the ability to install dependencies and define helpers on-the-fly, direct access to raw camera frames and motor position control, and support for computer vision tasks like hand tracking or face following. The system runs in a Docker container for damage control while hardware-side joint limits are still enforced.
Colinator is designed for researchers, developers, and hobbyists working on LLM-robotics integration who want to explore direct programmatic robot control without relying on expensive training datasets. It's particularly useful for those interested in Vision-Language-Code (VLC) approaches, testing LLM capabilities on real hardware, or building custom robot control policies through code generation.
The project has been tested with Codex, Claude, and Gemini, with LLMs successfully writing real-time control loops like 'follow my face' or 'follow my hand' without prebuilt tools, demonstrating impressive ability to query hardware, understand it, and create vision-and-control loops independently.
Colinator pricing
Pricing model: Freemium
Free and open source. The system is available at https://github.com/colinator/Ariel with no mentioned paid tiers or commercial licensing.
Colinator pros
- Direct LLM access to real robot hardware through Python REPL
- No need for expensive training datasets from human teleoperation
- LLM can write and execute code in real time on actual hardware
- Persistent REPL maintains state across multiple calls
- Supports closed-loop programs that react to environment in real time
- LLM can install dependencies on-the-fly (e.g., pip install mediapipe)
- Works with multiple LLMs: Codex, Claude, and Gemini
- Successfully demonstrated hand tracking and face following
- Docker container provides damage control safety layer
- Hardware-side joint limits still enforced for safety
- Can be adapted to ROS or any robotics middleware
- Low-level API gives full control without restrictive abstractions
- Raw camera frames available for custom computer vision
- Direct position control on individual motors
- LLM performs multiple reasoning steps within single prompt request
Colinator cons
- Safety, security, and alignment concerns with direct programmatic control
- MCP handling by harnesses can be painful and problematic
- None of the harnesses deal well with MCP restarts
- Codex sometimes truncates responses over certain length
- Claude often refuses to run ImageContent through image understanding mode
- Claude hallucinates image results frequently
- Currently only tested on trivial pan-tilt robot with USB camera
- Low-level API requires LLM to build high-level abstractions itself
- Gemini works best but still has various faults
Frequently asked questions about Colinator
What is Ariel?
Ariel is an MCP-exposed REPL for direct robot control by LLMs. It gives a large language model direct, programmatic, REPL-enabled access to a robot, allowing the LLM to program a robot in real time by writing Python code against a live robot object.
How does Ariel work?
Ariel consists of three pieces: an MCP server exposing tools like describe() and execute(code), a persistent REPL subprocess behaving like a normal Python session keeping state across calls, and a robot proxy injected into the REPL as 'robot' that bridges to actual hardware via zmq/tcp.
What robots does Ariel support?
The exact physical robot can be easily substituted. The current implementation uses a simple pan-tilt configuration with dynamixel motors and a USB camera, written using roboflex. The same pattern could sit on top of ROS or any other robotics middleware.
Which LLMs work with Ariel?
The system has been tested with Codex, Claude, and Gemini. Gemini appears to handle MCP well despite its faults, while Claude often refuses ImageContent and hallucinates results, and Codex sometimes truncates responses over certain length.
What can the LLM do with Ariel?
The LLM can query cameras or sensors, execute robot/joint commands via for loops, write real-time control loops like 'follow my face' or 'follow my hand', install dependencies, define helpers, build up state, save modules, and generally use the robot like a human developer would use a REPL.
Is Ariel safe to use?
There are obvious safety, security, and alignment concerns since it gives high-level AI direct programmatic control over a robot at a fine level. The system runs in a Docker container for damage control, and joint limits are still enforced on the hardware side.
What is the API like?
The API exposed to the LLM is low-level: direct position control on individual motors and raw camera frames. There are no high-level abstractions like 'look at object'—the model has to build those itself, such as writing its own hand tracking code.
Has Ariel demonstrated real capabilities?
Yes. LLMs have shown impressive ability to query hardware, understand it, pull and understand camera images, point the camera at different things, and write real-time control loops. Gemini successfully wrote a hand-tracker from scratch using mediapipe without any prebuilt track_hand() tool.
What are the current limitations?
The biggest technical limitation is MCP handling by harnesses, which can be painful with none dealing well with MCP restarts. Codex truncates responses, Claude hallucinates image results. The robot used is trivial, and the author awards AI only a 'white belt in robot-fu' so far.
How is Ariel different from VLA models?
Most SOTA robotics use Vision-Language-Action models built on large training datasets from human teleoperation or simulation, often for one specific robot. Ariel proposes Vision-Language-Code (VLC), leveraging that top-tier LLMs are already great at programming, letting the LLM be the programmer operating in-situ on real hardware.