Why evaluation needs to go beyond accuracy

Large language model evaluation often treats all test items equally and reports static accuracy on a fixed subset. That approach mixes item quality with item difficulty, yields noisy step-to-step measurements, and makes training curves flatten early even while model capability keeps improving. Fluid Benchmarking reframes evaluation: score models in a latent ability space and adaptively select the most informative items for the model’s current ability.

Core idea: ability, not raw accuracy

Fluid Benchmarking replaces raw accuracy with a psychometrics-grounded procedure. A two-parameter logistic (2PL) Item Response Theory model maps binary right/wrong responses to a latent ability θ. For item j with discrimination a_j and difficulty b_j, the correct-response probability is

p(u_ij = 1) = logistic(a_j (θ_i − b_j))

At evaluation, the candidate model’s ability θ^ is estimated (MAP) by maximizing the 2PL likelihood over observed responses on the administered items. Unlike plain accuracy, each item contributes according to its discrimination and difficulty.

/* 



At evaluation, estimate the MAP ability θ^i for the candidate LM by maximizing the 2PL likelihood over its observed right/wrong responses on the administered items. Items are weighted by their discrimination and difficulty, unlike accuracy which weights all equally

How items are chosen: Fisher information drives selection

After estimating the model’s current ability θ^(t), Fluid picks the next item q_j that maximizes Fisher information at that ability:

I(θ_i, a_j, b_j) = a_j^2 · logistic(a_j(θ_i − b_j)) · (1 − logistic(a_j(θ_i − b_j)))

High-information items reduce the variance of the ability estimate. As a model trains, the most informative items naturally shift from easy to hard, so the administered subset evolves with model capability instead of remaining fixed.

What improved evaluation looks like

The authors evaluate four concrete dimensions and report metrics:

• Validity: agreement with the true model ranking (mean rank distance, lower is better).
• Variance: normalized total variation of the training curve across checkpoints (lower is better).
• Saturation: monotonicity of the curve (Spearman correlation with checkpoint index, higher is better).
• Efficiency: quality at small item budgets.

Across six common benchmarks (ARC-C, GSM8K, HellaSwag, MMLU, TruthfulQA, WinoGrande) and multiple checkpoints from several LMs, Fluid substantially improves validity and reduces variance, delays apparent saturation, and performs especially well with small budgets.

Quantitative highlights

• Validity: On very small subsets (10 items), mean rank distance fell roughly from 20.0 to 10.1; on 50-item subsets, from 15.2 to 8.8.
• Variance: Total variation dropped markedly (example AP-10: 28.3 → 10.7; AP-50: 19.1 → 6.5).
• Saturation: Monotonicity improved (AP-10: 0.48 → 0.76; AP-50: 0.62 → 0.86).
• Small-budget efficiency: With only 10 items, Fluid improved mean rank distance by about 9.9 over random; improvements shrink as budget rises (e.g., 0.8 at 500 items).
• Mislabeled items: On MMLU-Redux with 100-item sessions, mislabeled items per session dropped from ~0.75 (random) to ~0.01 (Fluid), approximately a 100× reduction.

Ablation studies show that IRT-based aggregation raises validity, but dynamic Fisher-driven selection is the primary driver of variance reduction and improved monotonicity.

Practical features: dynamic stopping and operational trade-offs

Fluid supports dynamic stopping: terminate when the standard error of the ability estimate falls below a target (for example, the average ability gap between adjacent models on a leaderboard). In practice the required number of items varies with training stage (roughly 20 items early, over 80 mid-run), demonstrating why fixed budgets are suboptimal.

Operational costs include maintaining fresh response matrices, periodically refitting IRT item parameters as models improve, and reliable binary right/wrong labels for open-ended tasks. Fluid does not add new tasks; it re-weights and re-orders existing items to maximize information about latent ability.

Where Fluid fits in the evaluation stack

Fluid is a benchmark-refinement method: it generalizes across pretraining and post-training evaluation and across modalities provided there are enough model responses to fit or update an IRT model. As models get stronger, IRT parameters need refreshing to preserve discrimination among items that become easier over time.

Fluid Benchmarking is a practical default for in-loop evaluation when budget efficiency and stable ranking are priorities. For code, tutorials, and details, consult the authors’ paper and GitHub resources.