An AI model evaluation should answer a business decision, not produce a decorative benchmark. The decision might be whether to approve a model for internal drafting, which model to use for customer support, or whether an agent can safely receive access to a tool.
That decision requires more than a single accuracy score. A useful evaluation tests representative work, identifies unacceptable failure modes, measures human review and cost, and confirms that the surrounding controls can operate in production.
This checklist organizes that work into five stages: Establish, Validate, Assess, Launch and Sustain—the EVALS framework.
Key takeaways
- Start with a bounded use case, named owner and explicit risk level.
- Build the test set from real tasks, edge cases and adversarial conditions—not only ideal prompts.
- Use both a weighted score and hard pass/fail gates; a high average must never cancel a critical safety failure.
- Measure the whole workflow, including retries, reviewer time, latency and cost per accepted result.
- Re-evaluate after model, prompt, data, tool or policy changes.
What an AI model evaluation should decide
Before choosing metrics, write the decision in one sentence:
“We will approve one model for [specific task] used by [specific users] with [allowed data and tools] if it meets [quality threshold], passes [safety gates] and stays within [cost and latency limits].”
This sentence prevents scope drift. It also exposes missing inputs. If the team cannot name the task, users, data, controls and threshold, it is not ready to compare models.
NIST’s AI Risk Management Framework organizes AI risk work around Govern, Map, Measure and Manage. Its companion Playbook supplies suggested actions rather than a one-size-fits-all compliance test. The NIST Generative AI Profile adds considerations such as predeployment testing, content provenance and incident disclosure. Together, these sources support a risk-based evaluation in which technical performance and governance are assessed together.
The evaluation should produce four artifacts:
- A decision record stating what is approved, restricted or rejected
- A scored test report with raw results and reviewer notes
- A risk and control record linked to accountable owners
- A monitoring plan with triggers for re-evaluation or rollback
The EVALS framework
E — Establish the task, owner and risk
Define the task at the level at which a reviewer can judge success. “Summarize documents” is vague. “Create a 150-word summary that captures every decision and due date from an approved meeting transcript without adding unsupported claims” is testable.
Name a business owner and a technical owner. Classify the information involved and the consequences of a wrong answer. Identify laws, contracts or internal policies that may apply, then obtain qualified guidance where required. An evaluation is evidence for governance; it is not a substitute for legal, security or compliance review.
V — Validate with representative cases
Build a fixed dataset that resembles production. Include common tasks, difficult tasks, incomplete inputs, conflicting sources, multilingual or accessibility cases where relevant, and examples that should be refused or escalated.
Keep a hidden holdout set so prompt or workflow designers do not tune only to known examples. Record the source and permission for each test item, remove unnecessary personal data, and prevent confidential evaluation material from entering an unapproved service.
A — Assess quality, safety, usability and economics
Use a documented rubric. Reviewers should know what counts as correct, partially correct and unacceptable. Measure automated indicators where they are valid, but include human review for qualities that require judgment.
Test the surrounding system as well as the model: retrieval, tools, permissions, logging, fallback behavior and approval steps. Many serious failures are system failures rather than pure model failures.
L — Launch a controlled pilot
Give a limited user group access to a bounded workflow. Keep human approval for external messages, record changes, transactions and other consequential actions. Provide a clear way to report a problem and a tested method to disable the feature.
Compare pilot results with the baseline process. A model may score well in a test environment yet create extra review work, confuse users or shift risk to another team.
S — Sustain monitoring and re-evaluation
Monitor accepted-result rate, serious errors, overrides, incidents, latency, cost and user feedback. Set triggers for re-evaluation when the model version, system prompt, retrieval source, tool permissions, user group or applicable policy changes.
Copyable AI model evaluation checklist
Scope and ownership
☐ The use case is one bounded, testable job.
☐ A business owner and technical owner are named.
☐ Intended users and prohibited users are documented.
☐ Allowed and prohibited data classes are defined.
☐ Tool permissions and irreversible actions are listed.
☐ The risk tier and required reviewers are approved.
Test design
☐ The dataset includes routine, difficult and edge cases.
☐ Cases that should be refused or escalated are included.
☐ Prompt-injection and untrusted-content cases are included for retrieval or agent systems.
☐ A hidden holdout set is preserved.
☐ Expected answers or scoring rubrics are documented.
☐ Test data has a lawful and authorized source.
Quality and grounding
☐ Task completion is scored against the rubric.
☐ Material claims are checked against approved sources.
☐ Omissions and contradictions are counted separately from style problems.
☐ Consistency is tested across repeated runs.
☐ The model handles uncertainty and missing information appropriately.
☐ Output format and accessibility requirements are tested.
Safety, privacy and security
☐ Sensitive-data handling follows organizational policy.
☐ The system refuses or escalates prohibited requests correctly.
☐ Prompt injection and data-exfiltration paths are tested.
☐ The model cannot call tools outside its assigned role.
☐ Logs capture the information needed for investigation without over-collecting sensitive data.
☐ A human approval gate protects consequential actions.
Economics and operations
☐ Latency is measured at typical and peak loads.
☐ Token, retrieval, tool and infrastructure costs are recorded.
☐ Retry rate and reviewer minutes are measured.
☐ Cost per accepted result is compared with the baseline.
☐ Failure, fallback and rollback paths have been tested.
☐ Support, incident and model-change owners are named.
Approval and monitoring
☐ Hard failure gates pass before weighted scoring is considered.
☐ The decision and restrictions are recorded.
☐ Pilot users receive training and a reporting route.
☐ Monitoring metrics and thresholds have owners.
☐ Re-evaluation triggers and review dates are documented.
Build a representative test set
A small, thoughtful dataset is often more useful than a large collection of generic prompts. Start with work samples from the actual process, then stratify them by difficulty and risk.
For a support-answer assistant, a 100-case starting set might contain:
- 45 normal questions covered clearly by the knowledge base
- 15 ambiguous questions that require clarification
- 10 questions with conflicting or stale source material
- 10 requests that include sensitive data
- 10 requests outside policy that should be refused or escalated
- 10 prompt-injection or malicious-content cases
This mix is an example, not a universal standard. A high-impact use case may need a much larger set and independent expert review. Preserve the composition when comparing models so one candidate does not receive an easier test.
Where possible, score each case blind so reviewers do not know which model produced the output. Use at least two reviewers for subjective, high-impact criteria and define how disagreements will be resolved.
Choose metrics and hard gates
Metrics should map to the decision. Useful candidates include:
- Task success rate: outputs that meet every essential requirement
- Serious error rate: outputs with a defined material or harmful error
- Grounding rate: material claims supported by approved sources
- Citation precision: cited sources that actually support the associated claim
- Refusal correctness: prohibited requests refused without blocking legitimate work
- Escalation recall: cases requiring human attention that are correctly escalated
- Consistency: equivalent cases receiving materially compatible answers
- Accepted-result rate: outputs approved after the normal review process
- Review time: median human minutes per accepted result
- Latency: median and high-percentile response time
- Cost per accepted result: total operating cost divided by accepted outputs
Do not collapse everything into an average. Define hard gates first. For example: zero unauthorized tool actions; zero exposure of restricted test data; at least 98% correct escalation for a specified high-risk condition; and a serious error rate below a threshold chosen by the accountable owner.
If a hard gate fails, the model does not pass—even if its overall quality score is excellent.
Weighted AI model scorecard
After hard gates pass, a weighted score helps compare candidates transparently.
Dimension | Weight | Example evidence
Task quality and grounding | 35% | Rubric scores, task success, citation checks
Reliability and consistency | 20% | Repeat runs, edge cases, failure recovery
Safety, privacy and security | 20% | Refusals, injection tests, permission tests
Usability and human effort | 10% | Reviewer time, user errors, satisfaction
Cost and latency | 10% | Cost per accepted result, p50 and p95 latency
Operability | 5% | Logs, monitoring, support and rollback
Score every dimension from 1 to 5 and calculate:
Weighted score = Σ(dimension score × dimension weight)
Example: scores of 4.4, 4.0, 4.5, 3.8, 3.6 and 4.2 produce a weighted score of 4.18 out of 5. That number supports comparison; it does not override a hard-gate failure or the accountable owner’s documented risk decision.
Publish the weights before testing. Changing them after seeing the winner invites confirmation bias. Run a sensitivity check if two models are close: would a reasonable change in weights reverse the decision? If so, a short pilot may be more informative than another decimal place.
Run a controlled pilot
The pilot should mirror the intended workflow but limit the blast radius. Start with internal users, low-impact content and read-only tools where possible. Log model version, prompt version, source set, tool calls, reviewer decision and reason for rejection.
Set an exit rule before launch. A pilot can end in approval, restricted approval, redesign or rejection. Examples of restrictions include “internal drafting only,” “no customer personal data,” “no autonomous sending,” or “human approval required for every external output.”
Compare against the current method. Measure time saved only after including review and rework. Ask whether failures are visible to the user and whether the workflow makes it easy to correct them.
Monitor after launch
Production is a changing environment. New users, documents, attack patterns and model updates can change results. A practical monthly or risk-based review can include:
- accepted-result and serious-error trends
- top rejection reasons
- refusal and escalation performance
- prompt-injection alerts
- unusual tool calls or data access
- latency and cost drift
- user-reported issues and near misses
- changes to model, prompt, retrieval data or permissions
Connect significant failures to the AI incident response plan and the AI risk register. Minor quality issues can enter a normal improvement backlog; suspected data exposure, unauthorized actions or systematic unsafe behavior need immediate escalation under the organization’s incident process.
Common evaluation mistakes
Testing only polished prompts
Real users are inconsistent. Include typos, missing context, ambiguous requests and conflicting sources.
Using public benchmarks as the decision
Benchmarks help shortlist models. They rarely measure the organization’s task, data, controls and cost structure.
Ignoring the surrounding system
Retrieval quality, permissions and approval design can dominate risk. Test the complete application.
Optimizing for the average
A strong mean score can hide a small number of catastrophic failures. Use risk-specific hard gates and inspect the distribution.
Measuring token cost but not human cost
Track review time, retries and incident work. The lowest per-token price may not produce the lowest cost per accepted result.
Treating approval as permanent
Material changes require re-evaluation. Record exactly which model, prompt, source set, tools and restrictions the approval covers.
Frequently asked questions
How many test cases are enough for an AI model evaluation?
There is no universal number. Use enough representative cases to cover common tasks, important subgroups, edge conditions and unacceptable failures. Higher-impact decisions require more evidence, stronger statistical design and often independent expert review.
Can automated metrics replace human review?
Not completely for most business use cases. Automated checks are valuable for repeatability and scale, but humans are still needed for context, materiality, tone, policy interpretation and harms that a simple metric may miss.
Should every model use the same scorecard?
Use the same scorecard for models competing for the same job. Change the criteria when the job or risk changes, and document the change before testing.
How often should an AI model be re-evaluated?
Use risk-based intervals and event triggers. Re-evaluate after material changes to the model, prompt, retrieval data, tool access, user population or policy, and after serious incidents or unexpected performance drift.
Is this checklist a compliance audit?
No. It is an operational evaluation template. Organizations should obtain qualified legal, privacy, security and sector-specific advice for their obligations.
