AI Metrics
Visually
Last verified:
What is AI Metrics?
AI Metrics, Visually is an interactive, playful educational tool that helps machine learning practitioners and students understand the metrics they encounter when finetuning AI models. The tool organizes metrics by when they're used during the finetuning process: while training (loss & perplexity), judging labels (accuracy, precision, recall, F1), and judging generated text (ROUGE, BLEU, BERTScore).
The core mental model uses a fishing net metaphor where every metric represents an anxious question about what the net caught. The tool features interactive drag-through training epochs visualizations, clickable confusion matrices with 12 items where users toggle model predictions, threshold sliders showing precision-recall trade-offs, ROUGE/BLEU calculators with reference and generated text inputs, and perplexity calculators showing probability distributions.
This tool is designed for ML beginners finetuning language models, data science students learning evaluation metrics, practitioners who need to spot overfitting during training, and anyone who wants to understand why precision and recall pull against each other or why accuracy lies on imbalanced data.
AI Metrics pricing
Pricing model: Freemium
Free and open - no pricing tiers mentioned. The tool appears to be a free educational resource hosted on GitHub Pages (barvhaim.github.io) with no paid plans, subscriptions, or premium features.
AI Metrics pros
- Interactive drag-through training epochs visualization for loss curves
- Clickable confusion matrix with 12 toggleable items (fish vs boots)
- Threshold slider showing precision-recall trade-off in real-time
- Perplexity calculator with probability distribution visualization
- Overfitting detection by comparing train loss vs validation loss
- Fishing net mental model makes abstract metrics concrete and memorable
- ROUGE-1, ROUGE-2, and ROUGE-L calculators with precision/recall/F1
- BLEU vs ROUGE comparison explaining their different anxieties
- BERTScore explanation for meaning-aware text comparison
- LLLM-as-judge section for modern chatbot evaluation
- Explains why accuracy is dangerous on imbalanced data (fraud detector example)
- Cross-entropy loss explained as a 'surprise meter' with intuitive examples
- Harmonic mean vs regular average explained for F1 score
- Good perplexity range given (3-15 for modern LLMs) with baseline comparisons
- Sections tied together by unified fishing net metaphor throughout
AI Metrics cons
- No actual model training or finetuning capability - purely educational
- No API integration for real ML pipelines
- Limited to text-based metrics only (no image/vision metrics)
- No batch processing for multiple models or datasets
- Static examples rather than user-uploadable data
- No export functionality for metrics or visualizations
- No comparison against multiple baseline models simultaneously
- Missing metrics like AUC-ROC, Matthews correlation coefficient
- No interactive ROC curve visualization
- Does not cover reinforcement learning metrics (PPO, SAC specific)
Frequently asked questions about AI Metrics
What is the main mental model for understanding all these metrics?
The tool uses a fishing net metaphor where your AI model is a fishing net and every metric is an anxious question about that net. Precision asks 'Everything I caught - is it actually fish, or did I also pull up boots?' (false positives are boots). Recall asks 'All the fish in the lake - did I actually catch them, or did some swim through?' (false negatives are escaped fish). This unified metaphor ties all metrics together.
How do I spot overfitting during finetuning?
Watch the validation curve, not the training curve. Train loss always keeps dropping because the model can memorize. When validation loss turns upward while train loss continues falling, the model is memorizing而不是 learning. That gap is your signal to stop early. The tool provides interactive training epoch visualization showing train loss, val loss, and perplexity.
What is perplexity and why is lower better?
Perplexity equals e^loss and represents 'how many equally-likely words is the model choosing between at each step?' Perplexity 1 means the model is perfectly certain and correct. Perplexity 50 means it's as unsure as picking from 50 options. Lower perplexity means the model is more confident and correct. A strong modern LLM on general English sits around perplexity 3-15.
Why do precision and recall pull against each other?
Want perfect recall? Cover the whole lake with your net - you'll get every fish but also every boot (precision drops). Want perfect precision? Take only the one fish you're 100% sure of - definitely a fish but you missed 999 others (recall drops). F1 exists to stop both shortcuts by using harmonic mean that dies if either is low.
When should I use precision vs recall vs F1?
Use precision when false alarms are costly, like spam filters where blocking real email is worse than missing some spam. Use recall when misses are costly, like cancer screening where a false alarm beats missing a sick patient. Use F1 when you need balance and don't want a high score from maximizing just one side.
Why is accuracy dangerous on imbalanced data?
Accuracy is 'what fraction did I get right?' - intuitive but dangerous on imbalanced data. A fraud detector where only a few transactions are fraud can have 98% accuracy by the lazy model that predicts 'never fraud' for everything, yet catches 0 fraud cases. Precision, recall, and F1 exist because they ignore the easy majority class and ask 'did you catch the thing that matters?'
What's the difference between ROUGE and BLEU?
ROUGE is recall-leaning asking 'Did I cover everything in the reference?' and is standard for summarization where dropping key points is the failure you fear. BLEU is precision-leaning asking 'Is everything I generated actually correct?' and is standard for translation where inventing words is the failure you fear. Both count word overlap but lead with different anxieties from the fishing net metaphor.
Why do ROUGE-2 and ROUGE-L matter beyond ROUGE-1?
ROUGE-1 counts individual words and can give perfect scores even when meaning is reversed. For example, Reference 'dog bites man' and Generated 'man bites dog' get perfect ROUGE-1 (same 3 words) but ROUGE-2 (word pairs) and ROUGE-L (order) catch what ROUGE-1 misses because the meaning is completely reversed.
What's the limitation of ROUGE and BLEU, and what's the modern fix?
ROUGE and BLEU count surface overlap and don't understand meaning. To them, 'the film was great' and 'the movie was excellent' share almost nothing despite identical meaning. The modern fixes are BERTScore (compares embeddings instead of exact words, so 'film/movie' and 'great/excellent' score as near-matches) and LLM-as-a-judge (ask a strong model like Claude to score answers for helpfulness, correctness, tone).
What metrics should I use for finetuning instruction-tuned chatbots?
For open-ended chat quality, ngram metrics (ROUGE/BLEU) undercount good paraphrases so they're weak judges. The field leans on LLM-as-judge and human preference (win rate / human eval where 'Is answer A better than B?' across many prompts). Pairwise preference is what RLHF and Chatbot-Arena rankings use, and humans remain the gold standard for subjective chat quality.