Whissle
Multi-Modal Voice AI in a 500MB Local Docker
Last verified:
What is Whissle?
Whissle Gateway is a self-hosted Docker-based voice AI gateway that delivers the full voice AI stack—ASR (auto speech recognition), TTS (text-to-speech), LLM integration, speaker diarization, emotion detection, intent classification, and metadata extraction—on a single GPU or even CPU. It converts audio, text, or video streams into transcripts enriched with emotion, intent, speaker roles, age/gender estimates, behavior labels, and actionable AI analysis. The system runs entirely locally with no cloud dependency, making it ideal for privacy-sensitive environments like healthcare, legal, finance, and regulated contact centers.
Key features include six API interfaces (batch REST, streaming WebSocket, TTS, video intelligence, voice calling via Pipecat/Twilio, and an intelligent agent), built-in AI coaching modes for sales and collections compliance, real-time intent detection, word-level timestamps, speech pattern analysis (pace, fillers), and support for 23 languages including Hindi-English code-mixed ASR. Models download automatically on first run (~2 GB for en-full variant) and are cached forever in Docker volumes. The gateway supports CPU-only laptops to enterprise GPUs (H100, A100, RTX 6000 Ada) with concurrent call capacity scaling from 1–3 on MacBook to 250–500 on H200.
Whissle Gateway is built for contact centers, sales teams, therapists, interviewers, debt collection agencies, multilingual support teams, and behavioral AI researchers who need private, scalable, real-time voice intelligence. It scores 8 sales best practices, detects buyer/rep roles, flags compliance issues in collections calls, and provides overall scores 0–100 with timestamped highlights.
Whissle pricing
Pricing model: Freemium
Whissle Gateway is self-hosted via Docker with no stated subscription fee—the user downloads and runs the Docker image locally. The website shows an API pricing page with 'Pay only for what you use' and 'Start Free' but the actual pricing table is empty ('Get API Access' button). The self-hosted Docker version appears free to run on your own hardware. Cloud API pricing is not yet published ('Cloud API returning soon'). Every ASR model includes built-in metadata extraction (emotion, intent, entity detection) at no extra cost. Different models are priced by their inference cost for API access, but specific per-minute or per-request rates are not listed on the public page.
Whissle pros
- Fully self-hosted with no cloud dependency for complete data privacy
- Runs on CPU-only laptops (MacBook, Linux, WSL) without GPU requirements
- Single Docker command installs the entire voice AI stack in 2 minutes
- Automatic model downloads with permanent caching in Docker volumes
- Full stack on one GPU: ASR, TTS, LLM, diarization, emotion, intent
- Six API interfaces: batch REST, streaming WebSocket, TTS, video, voice calling, agent
- Built-in sales coaching mode with 8 best practices scored and 0–100 overall score
- Collections compliance mode detects identity verification, reason stated, harassment flags
- Real-time intent detection during speech (pay-back, disputes, hardship, refusal)
- Code-mixed Hindi-English ASR with 20.5% WER vs 34.7% from cloud providers
- 23 languages supported including Mandarin with North/South dialect detection
- Per-segment metadata: emotion, age, gender, behavior, role, intent, dialect
- Word-level timestamps available with word_timestamps parameter
- Speech pattern analysis includes pace, fillers, and fluency metrics
- Kokoro TTS delivers 55 voices with sub-200ms TTFB on CPU
- Custom prompt mode sends any user instruction to LLM with full transcript
- Auto GPU detection with NVIDIA CUDA 12.4 + onnxruntime-gpu support
- PostgreSQL database at port 5432 for conversations and analytics persistence
- Twilio/Voice AI integration for production voice calling at port 8000
- Scalable concurrency: 20–50 calls on RTX 4090, 250–500 on H200
Whissle cons
- Cloud API returning soon—not yet available for cloud deployment
- No GPU detected forces CPU mode which limits concurrent call capacity
- en-full variant requires ~2 GB download on first run (multi-full needs ~4 GB)
- Self-hosting requires Docker expertise and infrastructure management
- Anthropic API key required for LLM analysis (ANTHROPIC_API_KEY environment variable)
- Local token authentication may require manual curl commands for token management
- Limited to pre-built AI analysis modes unless custom prompts are written
- GPU variants require NVIDIA CUDA 12.4 specifically (amd64 architecture only)
- PostgreSQL runs inside container at port 5432 which may conflict with existing DBs
- Multiple port exposures (9000, 8001, 8002, 8003, 8765, 8000) needed for full stack
Frequently asked questions about Whissle
What is Whissle Gateway?
Whissle Gateway is a self-hosted Docker-based voice AI gateway that provides the full voice AI stack—ASR, TTS, LLM, speaker diarization, emotion detection, intent classification, and metadata extraction—on a single GPU or CPU. It converts audio, text, or video streams into transcripts enriched with actionable insights like emotion, intent, speaker roles, age/gender, behavior labels, and AI analysis. It runs entirely locally with no cloud dependency.
How do I install Whissle Gateway?
Run one Docker command: docker run -d --name whissle -p 9000:9000 -p 8001:8001 -p 8003:8003 -v whissle-models:/models -v whissle-data:/data -e VARIANT=en-full -e ANTHROPIC_API_KEY=your-key whissleasr/whissle-gateway:latest. Models download automatically on first run (~2 GB for en-full) and are cached forever. The system is ready in 2 minutes.
What variants are available and which should I choose?
Variants include: hinglish (Hindi-English, ~515 MB, for debt collections), en-lite (English, ~500 MB, for quick testing), en-full (English, ~2 GB, for sales coaching/interviews/therapy), multi-full (23 languages, ~4 GB, multilingual highest quality), multi-zh (23 langs + Mandarin, ~5 GB, with dialect detection), and all (all models, ~6 GB, maximum flexibility). Choose en-full for sales coaching and multi-full for multilingual support.
What hardware does Whissle Gateway support?
It runs on macOS, Linux, and WSL with CPU-only support. GPU support includes Mac Mini M4 Pro (24 GB unified, 3–8 concurrent calls), NVIDIA T4 (16 GB, 5–10 calls), RTX 4090 (24 GB, 20–50 calls), A100 40GB (50–80 calls), RTX 6000 Ada (48 GB, 50–100 calls), H100 (80 GB, 150–300 calls), DGX Spark (128 GB unified, 30–60 calls), and H200 (141 GB, 250–500 calls). Auto-detects GPU and switches to CUDA 12.4 + onnxruntime-gpu.
What API interfaces does Whissle Gateway provide?
Six interfaces: batch REST (POST localhost:8001/transcribe for file uploads), streaming WebSocket for real-time transcription, text-to-speech (Kokoro TTS at port 8003), video intelligence (port 8002), voice calling via Pipecat/Twilio (port 8000), and an intelligent agent (port 8765) that connects to any LLM (cloud or local). PostgreSQL database runs at port 5432 for conversations and analytics.
What AI analysis modes are available?
Four modes: sales_coaching (scores 8 best practices, identifies rep/buyer, provides timestamped highlights, behavior labels per segment, 0–100 overall score), collections (checks identity verification, reason stated, amount mentioned, no harassment; outputs Promise to Pay/Dispute/Hardship outcome and next action), true (general summary with overview, participants, key topics, emotional dynamics, entities, outcome in Markdown), and custom prompt (any user instruction sent to LLM with full transcript and per-segment metadata).
What metadata is extracted per transcript segment?
Every segment includes: emotion (NEUTRAL, HAPPY, SAD, ANGRY, FEAR, SURPRISE), age (0–18, 18–30, 30–45, 45–60, 60+), gender (MALE, FEMALE), plus model-specific tags. en-in-tech-misc adds behavior (26 types like EXPLAIN, QUESTION, ACKNOWLEDGE) and eval (CORRECT, PROBE, PARTIAL, INCORRECT, HINT, SKIP). hinglish-loans adds intent (13 collections intents like pay-back, disputes, hardship) and role (AGENT/Customer). zh adds dialect (NORTH, SOUTH, OTHERS). All models extract emotion, age, gender, and entity detection.
Which languages and models does Whissle Gateway support?
Five ASR models: en-in-tech-misc (485 MB, English, 120M params, 26 behavioral codes for coaching/therapy/interviews, 8 evaluation labels, 6 heads/51 classes), hinglish-loans (479 MB, Hindi-English, 115M params, debt collection intents, agent/customer role detection, 5 heads/26 classes), zh (627 MB, Mandarin with North/South dialect, 160M params, 3 heads/12 classes), whissle-large (2.4 GB, 23 languages, 600M params, 31 intent groups, 18K vocabulary, 5,500+ action tokens), and Kokoro TTS (82 MB, 55 voices, 10 languages, non-autoregressive, sub-200ms TTFB). Plus punctuation (254 MB) and ITN (1.5 MB) for EN + Hinglish.
Is Whissle Gateway private and secure?
Yes, it is fully self-hosted with no cloud dependency. All audio, transcripts, metadata, and analytics stay in your Docker container. Models are cached in whissle-models volume and conversations/analytics in whissle-data volume (PostgreSQL at port 5432). Data persists across container restarts and is only deleted by docker volume rm. This makes it ideal for healthcare, legal, finance, and regulated contact centers that require data privacy.
Can I customize the AI analysis with my own prompts?
Yes, use the custom prompt mode by passing any prompt string with -F 'summarize=your prompt here'. The LLM receives your instructions plus the full diarized transcript with per-segment metadata (emotion, intent, role, age, gender, behavior, eval, dialect). This allows you to define custom analysis criteria, scoring systems, or extraction rules tailored to your specific use case.