Research

Methodology — separate projection and evaluation lanes

The Games page publishes Home Margin projections from EPA-OLS when training coverage permits, or an explicitly labeled SP+ rating-difference fallback. Current SP+ captures and preseason SP+ inputs are distinguished on the card. Their historical edge buckets come from an EPA-OLS replay; those rates are a reference, not SP+ validation or the probability that today's pick covers.

The historical matchup calculator uses its own EPA-OLS fit and logistic scale. That scale cannot convert a current card's margin without a matching fitted projection artifact identity. Missing or incompatible identity means unavailable.

The calibration page evaluates the separate StackedEnsemble probability model. The saved 2025 evaluation trained on 2016–2024 and is retrospective; it does not measure the current SP+ card. Local SRS ratings are another independent lane (ADR-0007). None of these models substitutes for another's validation.

The remainder describes the ensemble probability export and evaluation path.

What the model predicts

One number per game: the probability the home team wins, in [0, 1]. The projected winner is whichever side that probability favors. The confidence tier (high / medium / low) buckets distance from a coin flip and is informational only.

Model

StackedEnsemble: three base models — logistic regression, XGBoost, random forest — produce 5-fold out-of-fold predictions on the training data, and a logistic meta-model combines those into the final probability. Trained on historical FBS games from the Model Training Pack (model_pack/training_data.csv).

A simple baseline is kept for comparison: the market's closing spread. The dashboard reports the model's accuracy next to the market's every week, and specifically how the model does on games where the two disagree.

Features

Every feature comes in a home/away pair. Sources:

• Opponent-adjusted EPA — twelve families (overall, rushing, passing, success rate, standard/passing down success, explosiveness ×3, line yards, second-level yards, open-field yards), each on offense and defense. Fit by ridge regression over play-by-play data, corrected for schedule strength and home field.
• Talent composite — CFBD's recruiting-derived team talent score.
• Pregame Elo — CFBD's per-team rating published before kickoff.
• Conference and week — categorical, one-hot encoded against the exact category set seen in training.

Leakage rules (the important part)

A Week W forecast may use only information available before that week's kickoff. Concretely:

• Play-by-play for the opponent-adjusted ratings comes only from weeks before W (AsOfWeek enforces this; postseason games use the completed regular season, never the training pack's week=1 encoding).
• Elo is the pregame Elo from /games, not a rating that already reflects Week W's result.
• Season-level stats (/stats/season/advanced, talent) are used as published; when used as a raw stand-in for adjusted EPA the slate is explicitly announced as epa_basis="raw_ppa", because the approximation is not the same number the model was trained on.
• Demo validation is a random stratified 80/20 split of the supplied 2016-2024 training pack. It mixes seasons and is not time-aware. Its scores may be optimistic for forecasting later seasons.
• Separate research in analysis/walk_forward.py trains on seasons before each held-out season, with category encoding learned from training rows. analysis/model_comparison.py trains on 2016-2024 and predicts 2025. The ensemble's internal stacking folds remain shuffled in both cases.
• The 2024-2025 results informed feature development, so they are retrospective evidence, not untouched confirmation data. Historical feature availability must also be checked before treating a backtest as a prospective forecast.
• Prospective grading requires predictions frozen before kickoff and graded after results arrive. A retrospective rerun is not prospective evidence.

If a backtest in this repo ever looks excellent, the first hypothesis is that it used future information. That has happened here before; the sign-convention sanity check and the as-of-week cutoff exist because of it.

Missing-data rules

A game is only projected when every feature resolves. Games that cannot be resolved are reported by named reason (team_absent_from_stats, team_stats_null, elo_missing) — never silently imputed, never filled with zero. Export is refused outright if:

• fewer than 3 games resolve (the sign-convention sanity check cannot run), or
• coverage falls below 50% of the FBS slate (a failure that large is systemic, not a few FCS opponents).

When too little play-by-play exists to fit the adjusted ratings (early weeks), the slate falls back to raw season stats and says so on the export.

Live odds checks and rejected data

Live pick generation and settlement validate raw /lines responses against the captured /games identities before using any odds. Team IDs, provider, event ID, UTC kickoff, and neutral-site status must agree. Display names cannot substitute for missing team IDs. When /lines omits neutralSite, the guard uses the locked /games value; an explicit conflict is rejected.

Rejected responses are saved under data/external/odds_quarantine/ with their original bytes, SHA-256 hash, and a machine-readable rejection record. The guard does not change the picks ledger or odds state. Retrying the same rejection reuses its saved evidence, including after an interrupted write. tests/test_betting_controls.py exercises both sides of that write boundary and verifies that recovery preserves the exact rejected bytes.

Versioned predictions

Each week's forecast is frozen to exports/live_<season>_week<week>_predictions.json before kickoff, via uv run python -m cfb_model_lab.export live <season> <week>. One file per week, append-only, never edited after the fact. That file — not a rerun — is what gets graded.

Evaluation

After the week completes, uv run python -m cfb_model_lab.grade <season> <week> resolves the frozen predictions against final scores and closing lines and appends one row to exports/performance.csv plus a per-week exports/calibration_<season>_week<week>.json. Regrading is idempotent — it replaces the week's row rather than duplicating it.

Metrics reported:

• Accuracy — how often the projected winner won.
• Brier score — mean squared error of the probabilities. Rewards honest probabilities, not just right picks.
• Calibration — per-bucket predicted vs. actual win rate for the picked side. A calibrated model's 0.75 picks should win ~75% of the time.
• Market comparison — the closing spread's accuracy on the same games, the model's agreement rate with it, and how the model does specifically on games where the two disagree.

uv run python -m cfb_model_lab.dashboard builds exports/dashboard.html from those files: weekly accuracy vs. the market, aggregate calibration, and the per-week detail table. The dashboard is a view over the CSV/JSON — the files are the source of truth.

Model-comparison report

exports/model_comparison_2025.json separates straight-up winner accuracy from ATS cover rate. Its ATS strategy backs each model's predicted winner at the recorded spread; market_ats backs the market favorite. Pushes are excluded from cover-rate denominators, and the market baseline also excludes pick'em lines. These are not the dashboard's separate model-edge ATS picks.

Before the 2026-09-08 correction, the report mislabeled straight-up model and market-favorite accuracy as ats and market_ats. Those historical values were not spread performance. The corrected report and regression tests in tests/test_model_comparison.py distinguish a winner failing to cover from an ATS win and exclude pushes from the denominator.

What this is not

• Not a betting recommendation. No stakes, no picks-of-the-week, no locks.
• Not a claim of market-beating edge. The market comparison is published every week precisely so underperformance is visible.
• Not finished. Feature additions go through the same leakage and grading gates; the rules above do not change after seeing a week's results.

Interpreting early-season EPA rankings

The /rankings board describes season-to-date opponent-adjusted per-play performance. It is not a forecast that its first-ranked team would beat every other team. The live calculation uses only completed prior regular-season FBS-vs-FBS plays; it does not bring forward preseason strength, Elo, talent, or prior-season ratings. Its implementation is AsOfWeek.rate → oppadj.fit.

For each eligible play, ridge regression fits an intercept plus offense and defense team effects and a home-field term. Published offense is the intercept plus the offense effect; published EPA allowed is the intercept plus the defense effect. Higher offense and lower EPA allowed are better. Net is offense minus EPA allowed, so the common intercept cancels. Regularization shrinks the team effects toward zero. The selected penalty is not a measure of ranking certainty.

The September 14, 2026 source audit reproduces all published values from the immutable Weeks 1–2 capture. Texas ranks 54th at net 0.014208; Mississippi State ranks first at 0.159125. Their raw offensive PPA is 0.303773 and 0.544401, respectively, while raw defensive PPA allowed is 0.146295 and -0.070138. The fitted values, after adjustment and shrinkage, are 0.178818 offense / 0.164611 allowed for Texas and 0.260419 / 0.101295 for Mississippi State. These ranks follow the stated formula; they do not demonstrate a sign error.

The reliability limitation is substantial: the observed FBS opponent graph has 38 disconnected components, and these two teams belong to different components. With two games per team, cross-component ordering depends on the common baseline and regularization assumptions rather than a chain of shared opponents. This is why the page describes early-season efficiency rather than predictive power. Changing that policy requires a separately tested model decision, not an unannounced adjustment of surprising ranks.

The derived [audit receipt](../exports/rankings_audit_20260914.json) records sample sizes, the fitted penalty/intercept, opponents, and component membership. tests/test_rankings_snapshot_audit.py verifies the installed source capture through derived CSV, database, and public rankings; it skips only when the operational capture is not installed. See [ADR-0005](adr/0005-live-slates-run-the-ridge-instead-of-standing-in-raw-stats.md) for the source and temporal policy.

Evaluation inventory and audit

The September 15, 2026 audit distinguishes the following evidence. These reports describe different models and samples; their scores are not a leaderboard.

| Existing producer and evidence | Evaluation contract and limits | | --- | --- | | model_pack/07_stacked_ensemble.ipynb, README historical table | Random stratified holdout across seasons, original eight continuous features. Not chronological forecasting evidence. | | analysis/walk_forward.py, exports/walk_forward_2017_2025.json | Outer training seasons precede test season. Supplied season-final features prevent a pregame availability claim. Includes inverse-normal probability-to-margin assumptions. | | analysis/model_comparison.py, exports/model_comparison_2025.json | Train through 2024, score 2025. Already inspected during feature development; winner accuracy and ATS cover rate are separate. | | analysis/calibration_export.py, exports/calibration_2025_week.json, exports/scoring_metrics_2025.json | Retrospective 2025 reliability, Brier and log loss. The shared grading-file shape and graded_at timestamp do not establish a frozen pregame forecast. | | analysis/walk_forward_production.py, exports/pooled_production_2017_2025.csv | Weekly expanding EPA-OLS replay. Existing reports show ATS edge buckets, including losing buckets. Saved rows contain only projected, lined games. | | analysis/06_power_ratings.ipynb, analysis/output/ benchmark artifacts | Weekly SRS/Elo replay and common-game market comparisons; latest archived odds are not verified closing captures. ADR-0006 defines timing. | | analysis/validate_predictive_srs.py, exports/srs_predictive_validation_20260914/ | Existing probability calibration, reliability, margin metrics, coverage and week-block uncertainty. 2024–2025 is already-inspected confirmation. | | analysis/evaluate_epa_prior.py, exports/epa_prior_20260914/ | Development selection and frozen-mapping confirmation with cutoff and coverage evidence; retrospective forecast generation. | | cfb_model_lab.grade, cfb_model_lab.dashboard, /lab/calibration | Grading and calibration views. Verify the originating forecast's freeze and kickoff metadata before treating any shared-format file as prospective evidence. | | analysis/tracker_review.py, frozen review bundles | Separate original selections and research candidates, common completed games and immutable grading receipts under ADR-0006. |

Existing residual plots in analysis/recruiting_vs_development.py describe Elo relative to talent, and analysis/win_total_model.py describes season wins. Neither diagnoses per-game Home Margin errors. Totals/weather residual studies also target a different outcome. The new report below fills that gap without regenerating calibration or selecting a new forecast mapping.

Feature and stacking cutoff findings

• StackedEnsemble.fit uses KFold(5, shuffle=True, random_state=42). A stacking validation row can receive base predictions trained on later dates inside the outer training set. Outer season exclusion still keeps the test season out, but these are not chronological out-of-fold predictions. Changing stacking requires a separately evaluated proposal.
• analysis/walk_forward.py learns categories on outer training rows, but its season-final input features remain a temporal limitation. A chronological split alone does not repair unavailable features.
• Historical season_frame asks AsOfWeek for prior plays, requests advanced stats through W−1 and Elo at W−1. Talent is fetched by year with no capture timestamp. The supplied training pack does not establish publication-time availability for every feature. Revised historical API values also need capture evidence before a replay can claim prospective equivalence.
• Production EPA-OLS joins ratings by season/team/week. The rating for W uses plays before W; betting.project trains coefficients on games before W, with at least 30 training games. This is a chronological computation audit, not proof of original data publication times. The replay reuses stored 2025 EPA and recorded spreads; its CSV contains no per-row source cutoff receipt.
• Frozen prospective forecasts must be evaluated from their original bundle. A current rerun or a grading timestamp cannot replace a missing pregame capture. Keep those results separate from all retrospective reports above.

Home Margin error diagnostics

Run uv run python -m analysis.residual_diagnostics to write exports/residual_diagnostics_2017_2025/ from the existing production archive. It produces predicted-versus-actual and residual plots, all per-game errors, cohort summaries and common-game coverage by season, week, team and absolute recorded spread bucket. Development through 2023 and already-inspected 2024–2025 confirmation remain separate. No favorable-error filter is applied.

Residual and bias mean predicted Home Margin minus actual Home Margin. Positive values overstate the home side. Team cohorts include home and away appearances but retain this home-side sign. They are not team-perspective strength errors. Each summary reports n, season-week blocks, MAE, RMSE, signed bias, mean predicted/actual Home Margin and error quantiles. All systems use identical completed games within each cohort; missing forecasts remain in coverage. Archive coverage is not full-schedule coverage because the producing walk-forward script already omitted unlined and unavailable projections.

Cohorts with fewer than 30 common games are flagged sparse. Larger cohorts are still descriptive: shared opponents, repeated teams and overlapping cohorts invalidate an independent-game significance interpretation. Error quantiles are not confidence intervals or a calibrated prediction distribution. Do not choose a strategy from the best observed bucket. Existing SRS week-block intervals remain conditional on the seasons inspected and their fixed mappings.

Home Margin alone cannot determine a total: 28–21 and 49–42 both have Home Margin +7 but totals 49 and 91. Win and cover probabilities require a validated score/error distribution or calibration mapping, evaluated chronologically. MAE, residual quantiles, a normal CDF assumption or a team's rank alone do not provide that validation. This audit changes no forecast or probability mapping.

The saved production archive gives the following common-game results. These are point errors, not probabilities or evidence of a profitable betting rule.

| Evidence | Games | EPA-OLS MAE | Recorded market MAE | EPA-OLS bias | Market bias | | --- | ---: | ---: | ---: | ---: | ---: | | Retrospective development, 2017–2023 | 2,826 | 13.213 | 12.244 | +0.136 | +0.137 | | Already-inspected confirmation, 2024–2025 | 1,022 | 12.761 | 11.982 | −0.089 | −0.871 |

The recorded market has lower average absolute error in both groups. A bias near zero does not imply small errors: overpredictions and underpredictions can cancel. Per-team and per-week samples in the generated report make those misses visible without removing the losing cohorts.