Evaluate advanced models against your hardest real tasks

Define the comparison you need

“Frontier” is a broad label, so translate it into a practical question. Do you need better document analysis, more reliable tool selection, stronger code changes, or fewer unsupported assertions? Each task suggests different examples and a different way to judge the result.

Select a current baseline and a small set of candidates. Use official documentation to confirm the specific endpoint, supported inputs, and access requirements for each. Treat a provider’s model description as a starting point for investigation, not a substitute for your own evidence.

Keep the test conditions comparable

Give candidates equivalent source material and the same task objective. Record differences in prompts or settings when the interfaces require them. Define whether you are measuring a single response, a complete tool-assisted workflow, or the time a person needs to reach an acceptable result.

• Quality: Task completion, evidence use, correctness, and required format.
• Latency: Time to a useful result under realistic input sizes.
• Resource use: Input, output, tool calls, retries, and review effort.
• Failure behavior: Incomplete requests, unsupported inputs, and uncertain answers.

Set acceptance criteria before opening the results. Use blinded review when practical so a familiar model name does not influence the judgment. Preserve failed examples alongside successful ones.

Account for deployment context

NIST’s Generative AI Profile emphasizes identifying and managing risks in the context of an organization’s goals. Apply that perspective to the actual feature: an internal drafting assistant and an automated customer decision require different evidence and oversight.

Estimate operating costs from the provider’s current terms and your measured usage pattern. Include retries and any separate tool activity. Avoid treating a headline token price as the total cost of completing a task.

Write a decision note that says which candidate meets the requirements, where it still needs review, and what would trigger reevaluation. A model change should be a traceable product decision.

Use prompt contracts and evaluation cases to preserve comparable inputs. The API company assessment guide adds provider operations and dependency considerations.