A difference of a few benchmark points may look appealing in a launch announcement, but products pay for every single token, every tool call, and every request that has to be rerun. As scale grows, a cheaper model that is “good enough” often delivers better profit margins than the most powerful one.
Model evaluation should therefore be based on the cost to complete a task successfully. This metric forces teams to examine quality, latency, retry rates, and inference pricing all at once, rather than optimizing for a metric detached from reality.