← All articles

Coding

Claude Skills 2.0: Build, Test, and Optimize Agent Customizations

6 min read

Updated

Learn how to build, test, and optimize Claude Skills 2.0. This guide covers evals, A/B benchmarking, and best practices for Claude.ai and Claude Code.

Claude Skills solve a specific problem: Claude is capable but inconsistent. It can write in any style, follow any process, and use any framework, but it does not naturally remember your preferences between conversations. Skills fix that. They are reusable instruction sets that tell Claude exactly how you want a task performed.

The Skills 2.0 update introduces structured testing, allowing you to run evals, benchmark outcomes against raw Claude models, and tune trigger descriptions to ensure they activate under the right conditions.

This guide covers how to build, test, and optimize skills for both Claude.ai and Claude Code.

What Are Claude Skills?

Skills are folders containing a SKILL.md file and optional supporting resources like templates, examples, or helper scripts. The SKILL.md file consists of two primary parts:

1. Frontmatter

YAML frontmatter tells Claude when to use the skill based on the request:

---
name: code-review
description: Reviews code for security vulnerabilities, performance issues, and maintainability. Use when asked to review code or check for bugs.
---

2. Instructions

Markdown instructions outline the specific steps Claude must follow:

# Code Review Skill

## Process
1. Check for security vulnerabilities (OWASP Top 10)
2. Identify performance bottlenecks
3. Evaluate code readability and maintainability
4. Suggest specific improvements with code examples

## Output Format
- Summary of findings (1-2 sentences)
- Critical issues (must fix)
- Recommendations (should fix)
- Positive patterns (keep doing)

When Claude encounters a user request that matches the skill's description, it loads these instructions and executes them, ensuring consistent and repeatable outputs.

Two Categories of Skills

  • Capability uplift skills: These teach Claude something it cannot do well on its own, such as using a proprietary testing framework, following specific compliance guidelines, or adhering to internal code review checklists.
  • Encoded preference skills: These sequence behaviors Claude already understands into a specific workflow. For example, enforcing a particular structure, tone, and SEO pattern for blog posts.

Understanding this distinction helps guide your testing strategy: capability uplift skills should produce objectively superior results to raw Claude, while encoded preference skills aim for stylistic alignment.

What Changed in Skills 2.0?

The Skills 2.0 update introduces four core features designed to transition agent customizations from experimental to verified implementations:

Evals

Evals function as automated software tests. You define test prompts and describe the expected outputs. The testing system runs the skill against these prompts and reports pass/fail rates. This allows you to pinpoint precisely where a skill falls short.

A/B Benchmarking

Comparator agents run prompts through both the customized skill and raw Claude, evaluating the outputs blindly. This answers whether the skill actually improves performance compared to a standard prompt.

Trigger Tuning

A skill's description determines its activation threshold. If the description is too broad, it runs on unrelated requests; if it is too narrow, it fails to trigger. Internal testing during the 2.0 rollout revealed that many initial skill descriptions required refinement to activate reliably.

Multi-Agent Parallel Testing

Parallel execution spins up independent agents to run evals simultaneously. Each agent runs in a clean context with isolated token and timing metrics, accelerating test speeds and preventing context contamination.

How to Create a Skill

Method 1: The Skill Creator (Recommended)

Within Claude Code or Claude.ai, prompt the model:

Create a skill for [describe what you want]

The system will ask about your workflow, generate the required directory structure, format the SKILL.md file, and bundle any supporting resources.

Method 2: Manual Directory Configuration

Create the following directory layout:

my-skill/
├── SKILL.md          # Required: frontmatter + instructions
├── template.md       # Optional: output templates
├── examples/         # Optional: input/output examples
│   ├── good-output.md
│   └── bad-output.md
└── scripts/          # Optional: helper scripts

Configure SKILL.md with appropriate YAML frontmatter and clear, sequential instructions.

Installation Options

  • Claude Code: Save the skill directory in .claude/skills/ (for project-specific use) or ~/.claude/skills/ (for global availability).
  • Claude.ai: Navigate to Settings → Features → Upload as ZIP. This feature is supported on Pro, Max, Team, and Enterprise tiers.
  • Pre-built Collections: You can source community-made and official skills from public repositories like Anthropic's public skills repository or the community-run awesome-claude-skills list.

Running Your First Eval

Testing is key to ensuring your skill behaves predictably. Follow this step-by-step workflow:

Step 1: Define Test Cases

Write 5 to 10 test prompts representing realistic usage, including edge cases:

Eval test cases:
1. "Review this Python function for security issues" + [sample code with SQL injection]
   Expected: identifies the SQL injection vulnerability
2. "Review this API endpoint" + [sample code with missing auth]
   Expected: flags missing authentication check
3. "Review this React component" + [clean code]
   Expected: reports no critical issues, maybe minor suggestions

Step 2: Execute the Eval Command

Run the eval via Claude Code's skill-creator:

Run evals for my code-review skill

The system runs the test cases and outputs:

  • Pass rate (e.g., 8/10)
  • Detailed failure reports
  • Average response times
  • Token consumption metrics

Step 3: Iterate and Refine

For any failing cases, update the instructions in your SKILL.md:

  • Provide explicit instructions addressing the failed scenario.
  • Add positive or negative examples in the examples/ directory.
  • Clarify any ambiguous terminology.

Re-run the evals until the success rate meets your requirements (aim for 90%+ on critical workflows).

A/B Benchmarking: Measuring Value

A common issue with agent customization is "skill rot"—where base model upgrades make custom instructions redundant or counterproductive. A/B benchmarking validates whether your skill outperforms the standard model.

Running a Benchmark

To start a comparison, run:

Benchmark my code-review skill against raw Claude

The testing suite will run prompts through both versions, employ a blind evaluator model to judge the results, and return a win rate:

  • Win rate > 70%: The skill provides a clear performance uplift.
  • Win rate 50-70%: The skill offers marginal improvement. Consider consolidating the instructions or relying on the base model.
  • Win rate < 50%: The skill degrades performance. Review or deprecate the custom instructions.

It is recommended to run benchmarks quarterly and after major foundation model updates.

Trigger Optimization

The description field in the frontmatter determines when a skill activates. Optimize your triggers using these guidelines:

  • Be explicit about boundaries: Define what the skill handles: "Use when the user asks to review code for security issues, check for vulnerabilities, or audit security."
  • Set negative constraints: Define what the skill should ignore: "Do not use for general code quality reviews, performance optimization, or refactoring suggestions."
  • Verify trigger accuracy: Test the activation behavior using borderline prompts:
Test triggering for my code-review skill with these prompts:
- "Review this code" → should trigger
- "Refactor this function" → should NOT trigger
- "Is this code secure?" → should trigger
- "Write unit tests" → should NOT trigger

Best Practices

  • Keep skills focused: Target narrow, specific tasks (e.g., "SQL Injection Review") rather than broad categories ("Python Quality").
  • Leverage examples: Provide high-quality input/output pairs in the examples/ directory. Few-shot examples are highly effective at guiding output format.
  • Version control: Store your skill folders in Git. Track performance improvements and regressions alongside code changes.
  • Implement layered architectures: For teams, organize skills hierarchically:
    • Personal skills (~/.claude/skills/): Individual workflows and preferences.
    • Project skills (.claude/skills/): Committed to the repository to enforce project-specific coding standards.
    • Organization skills: Shared via team platform settings to enforce company-wide compliance and brand voice.