Agentic Tool-Use Evaluation Harness & Prompt Grading Playbook

Design reproducible evaluation suites for tool-calling AI agents: golden test cases, deterministic rubric graders, latency/token cost tracking, and regress

---
name: anthropic-eval-harness
description: Design reproducible evaluation suites for tool-calling AI agents: golden test cases, deterministic rubric graders, latency/token cost tracking, and regression diffs across prompt edits. Use when working with LLM Evals, Tool-Use Benchmarks, System Prompts, Rubric Grading.
triggers:
  - anthropic eval harness
  - llm evals
  - tool-use benchmarks
---

# Agentic Tool-Use Evaluation Harness & Prompt Grading Playbook

## 1. When to Use This Skill
Load `skill-anthropic-eval-harness` whenever an engineering or operations task involves **LLM Evals, Tool-Use Benchmarks, System Prompts, Rubric Grading**. Specifically:
- Design reproducible evaluation suites for tool-calling AI agents: golden test cases, deterministic rubric graders, latency/token cost tracking, and regression diffs across prompt edits.
- Designing, reviewing, or debugging production configurations without guesswork or deprecated flags.
- Running deterministic pre-flight and post-execution verification checks before reporting completion.

## 2. Prerequisites & Required Environment
- **Target Stack:** LLM Evals, Tool-Use Benchmarks, System Prompts, Rubric Grading
- **Permissions:** Read-only inspection by default; ask for explicit confirmation before executing destructive mutations or production deployments.
- **Reference Repository:** `https://github.com/anthropics/courses`

## 3. Step-by-Step Execution Playbook
1. **Context & Configuration Discovery:**
   Inspect existing project manifests (`package.json`, `wrangler.jsonc`, schema files, or environment declarations) before proposing changes. Never overwrite existing bindings or credentials blindly.
2. **Deterministic Implementation:**
   Apply the canonical patterns for **LLM Evals**:
   - Enforce strict schema/type validation at every external boundary.
   - Include explicit error handling, structured logging, and idempotency keys on all mutating calls.
   - Keep secrets in environment variables or secret stores—never hardcode tokens in source files.
3. **Verification & Regression Gate:**
   Run the verification commands below and inspect both exit codes and output payloads before marking the task complete.

## 4. Reference Commands & CLI Workflow
```bash
npx promptfoo@latest eval
npx promptfoo@latest view
```

## 5. Edge Cases & Anti-Hallucination Guardrails
- **Zero Unverified Claims:** Never claim a deployment, migration, or API call succeeded without inspecting the command output or HTTP status code.
- **Rollback Readiness:** Always record the previous state or migration version so changes can be reverted cleanly if downstream health checks fail.

Synthesized Engineering Blueprint & Setup Guide