The internet has been abuzz this past week with chatter about the proposed “pacing” of the frontier by leading US frontier AI labs. While some point to potential doomsday scenarios highlighted by recent Anthropic staff departures and others point to déjà vu, the question appears not to be, “Will we pace the frontier?”, but instead, “What will pace the frontier: regulation or economics?”

First, let’s get our bearings and see what the frontier has actually been doing, based on the Artificial Analysis Intelligence Index¹ benchmark:

Progress marches on, but it comes at a cost on the order of billions of dollars per year.

Now, the astute reader will point out the phenomenon of “benchmaxxing”, in which LLMs max out benchmarks more quickly than anecdotal evidence indicates they are actually improving. In summary, there are two opposing truths:

1. Benchmarks are bad, in general
2. Benchmarks are essentially the only zero-cost way that businesses and consumers can assess model quality

So, even if benchmarks have their problems, businesses will continue to use them as a shared proxy for “quality”. Through that lens, let’s take a look at what has been happening at the frontier of closed-weight (Anthropic, OpenAI, Gemini, etc.) vs open-weight (Qwen, GLM, Mistral, etc.) models:

The relationship between these two lines stands out, as they are apparently keeping lockstep. We can look at this gap over time and see that this is indeed the case, in absolute terms:

On the surface, this looks like a clear win for closed-weight models, but let’s pressure-test this idea: what are we really looking at here? An eight-point gap in performance is not nothing, but we don’t necessarily have much intuition for what that means. First of all, a gap that is consistent in absolute terms is one that is shrinking in relative terms as the scores grow. Moreover, if you take a look at the earlier charts, we’re in fact measuring the gap between two moving targets, where the open-weight models are constantly catching up to the previous closed-weight frontier. Taken together, a more intuitive way of thinking about this data is by measuring the temporal gap between the open-weight and closed-weight frontiers. When a new open-weight model is released, we can ask, “This model is behind the closed frontier, but it is as good as the best-available model from how long ago?”

Plotting this over the past year or so, we see that the “temporal edge” of the closed-weight models has been eroding:

Ever since the release of DeepSeek in January 2025, one can clearly see that open-weight models have been regularly eating away at this time advantage. To summarize, if you used the top open-weight model today, you’d be running something that would have been state-of-the-art just over three months ago.

One elephant in the room worth addressing head-on is that of distillation. Anthropic, OpenAI, and others have publicly claimed that leading Chinese models have unfairly piggy-backed on their success by training their models on the large US models’ behavior. Setting this controversy aside, we can see that even when excluding Chinese models we find a recently closing gap between open and closed-weight models (in particular as of late with some impressive open-weight models from South Korea and Singapore, which are closely followed by models from the UAE and Spain):

This is a problem for OpenAI and Anthropic. Their primary business model is:

1. Incur an enormous up-front capital expenditure in order to produce state-of-the-art AI models
2. Sell those models back to businesses (APIs) and consumers (ChatGPT, etc.)

This only works if those models are perceived as valuable by users. On the surface, it seems like that could be true, since, all things being equal, closed models are giving you a three-to-five month lead time over the open-weight competition. The one snag, however, is that closed-weight models are expensive:

Moreover, they are becoming a worse value over time. If we consider the evolution of the ratio of the closed-weight model prices and open-weight inference prices² compared against the temporal gap’s evolution, the difference is stark:

Bear in mind that these numbers are quarterly medians. The high-frequency data is more volatile, but more stark: the current frontier open-weight model (Xiaomi’s MiMo-V2.6-Pro, released on 21 September) is 16 times less expensive than Claude Opus 5.5, released yesterday.

While these findings have important implications for the heads of frontier AI labs, they are equally relevant for startup founders, tech leads, CTOs, finance, and engineering heads as well. These graphs should make every CTO ask themselves: “Do I want to 16x my AI spend in order to be about three months ahead of my competition?”

There might be some reasons the answer is “yes”:

• Your product has low switching costs
• Customers choose your product because it is “smarter” than the competition
• Mistakes made by your product have a high cost
• You are in a greenfield (or recently greenfield) industry where competition moves rapidly (e.g. coding assistants or AI image generation)
• AI spend is already negligible

For most businesses, however, the answer is likely “no”. In the words of Ramp CEO Eric Glyman in a recent FT article, “You don’t need to rent a Ferrari to go pick up your groceries.” As AI models’ cost-benefit ratios continue to diverge and AI spend continues to be an increasing budget item for companies, having a clear idea of which model is appropriate to use where is becoming more and more mission-critical.

That said, it is not just a simple cost lever. With proper AI governance becoming more essential for organizations relying on AI systems in production, the decision of which foundation model to run needs to be made in the context of use case-specific evaluations. Unfortunately, many teams either don’t have these on-hand or don’t have the bandwidth to test different models themselves, especially as the frontier moves every month.

If you’d like to know whether your AI stack is a Ferrari doing grocery runs, we’d be happy to take a look. With deep expertise in bespoke AI evaluations, we can sanity-check your architecture, run competitive evaluations of open-weight models against your actual tasks, and provide clear advice on what cost-to-quality tradeoff you are making. Contact us to book a review today.