Summary

The canonical paper about Algorithmic Progress is by Ho et al. (2024) who find that, historically, the pre-training compute used to reach a particular level of AI capabilities decreases by about 3× each year. Their data covers 2012-2023 and is focused on pre-training.

In this post I look at AI models from 2023-2025 and find that, based on what I think is the most intuitive analysis, catch-up algorithmic progress (including post-training) over this period is something like 16×–60× each year.

This intuitive analysis involves drawing the best-fit line through models that are on the frontier of training-compute efficiency over time, i.e., those that use the least training compute of any model yet to reach or exceed some capability level. I combine Epoch AI’s estimates of training compute with model capability scores from Artificial Analysis’s Intelligence Index. Each capability level thus yields a slope from its fit line, and these slopes can be aggregated in various ways to determine an overall rate of progress. One way to do this aggregation is to assign subjective weights to each capability level and take a weighted mean of the capability level slopes (in log-compute), yielding an overall estimate of algorithmic progress: 1.76 orders of magnitude per year, or a ~60× improvement in compute efficiency, or a 2 month halving time in the training compute needed to reach a particular capability level. Looking at the median of the slopes yields 16× or a halving time of 2.9 months.

Based on this evidence and existing literature, my overall expectation of catch-up algorithmic progress in the next year is maybe 20× with an 80% confidence interval of [2×–200×], considerably higher than I initially thought.

The body of this post explains catch-up vs. frontier algorithmic progress, discusses the data analysis and results, compares two Qwen models as a sanity check, discusses existing estimates of algorithmic progress, and covers several related topics in the appendices.

‍

For the full text of this post see https://www.lesswrong.com/posts/yXLqrpfFwBW5knpgc/catch-up-algorithmic-progress-might-actually-be-60-per-year.

‍