# Exports This directory contains generated research and model outputs that are useful for review or downstream consumption. ## Private notebook runs `notebook_runs//` contains executed notebook copies, HTML, execution logs, input inventories, environment and source hashes, and run-local model artifacts. `scripts/run_notebook.py` refuses an existing run ID and checks that source notebooks and input files remain unchanged. Its metadata distinguishes execution failures from successful execution with explicitly unavailable analyses. Every nonempty code cell must execute for the run to pass. These directories are ignored by Git and excluded from the existing Vercel upload and research staging rules. Do not move their HTML or plots to the top level of `exports/`, where the research publisher can discover them. Training and model reload evidence remain local. A successful notebook does not authorize publishing its outputs or promoting its fitted model. Use the [notebook operating workflow](../docs/operations/college-football-refresh.md#notebook-refresh-and-replay) to reproduce a historical or pinned current-season run. Retain failed-run evidence and use a new run ID after repairs. | Artifact | Producer | | --- | --- | | `sample_week_predictions.json` | `uv run python -m cfb_model_lab.export` | | `sp_plus_vs_epa_2026.png` | `uv run python analysis/plot_sp_vs_epa.py` | | `season_previews_2026/` | `./scripts/build_season_previews.sh` | | `epa_prior_20260914/` | `uv run python -m analysis.evaluate_epa_prior` and `uv run python -m analysis.build_epa_prior_shadow` | | `wepa_compatibility/20260915/` | `uv run python -m analysis.wepa_compatibility report`; [capture, findings, and replay](../docs/cfb-data-sources.md#wepa-compatibility-with-training-features-september-15-2026) | | `sp_plus_card_audit_20260908.csv` and `.json` | `python3 analysis/sp_plus_card_audit.py [predictive-table.tsv]` | ## Picks ledger ### Current card audit The reproducible runtime payload is `picks_latest.json`. Audit it without refreshing data or overwriting the payload: ```bash uv run python analysis/audit_picks.py exports/picks_latest.json \ --output exports/picks_audit_20260909 ``` The September 9 payload is a 2026 Week 2 `preseason_sp_plus` run. It contains 49 projected games, 26 picks at the 3.0-point edge threshold, and 23 games below the threshold. The count is therefore a threshold result, not a target of 25. Its 2025 bucket table is a cover backtest, not a win-probability calibration; the 7-14 bucket attached to several picks has a 43.56% historical hit rate. The audit passes the edge arithmetic and preserves the payload hash. The separate `sp_plus_card_audit_20260908` artifact is a 25-game, user-supplied card comparison. It is not the runtime pick payload and cannot establish that the runtime generated 25 picks. ### Branch disposition (September 9, 2026) The exact local and `origin` refs are captured in `research_branch_inventory_20260909.json`; regenerate it with: ```bash uv run python analysis/research_branch_inventory.py \ --output exports/research_branch_inventory_20260909.json ``` `master` includes the direct external forecast research and pick audit from `codex/external-prediction-sources`. `codex/sp-plus-card-research` is already merged into master and its CSV audits remain comparison artifacts. The `codex/current-team-rankings` work is merged into master. The `codex/merge-outstanding-branches` pointer contains the same merged history, including the scoreboard evidence fixture. The remote refresh, publish, and week-one branches contain operational or dashboard updates already represented in the merged history; they do not independently validate the pick model. The `cursor/*`, `hermes/*`, `fix/*`, and `scoreboard/*` branches remain reviewable source history. All local and fetched remote branch tips are ancestors of the merged `master`. Uncommitted worktree edits remain separate, and no branch was deleted. External ratings and SP+ comparisons stay research inputs until a chronological evaluation clears the promotion gate documented in README. `picks_ledger.csv` is generated operational data, kept separate from code changes. The first tracked snapshot preserves an existing local file with 10 records for 2026 Week 1, all carrying the settlement timestamp `2026-09-08T17:09:16.459276+00:00`. It was not regenerated when committed. The exact source captures and command that produced those entries have not been verified; tracking the file does not certify its results. The supported settlement command is `uv run python -m cfb_model_lab.picks settle `. Preserve the original week's picks before running it. See [`docs/operations/college-football-refresh.md`](../docs/operations/college-football-refresh.md). ## September 8 SP+ card comparison The [25-game comparison](sp_plus_card_audit_20260908.csv) uses the user's original rounded spread card and the predictive SP+ table they supplied on September 8. The [metadata](sp_plus_card_audit_20260908.json) records source, input hash, formula, and limitations. With 2.5 points of home field, 14 picks retain positive rating-based edges and 11 reverse direction. These are research comparisons, not official Connelly game picks, cover probabilities, or confirmed professional wagers. The calculation is `home_rating - away_rating + 2.5` for expected Home Margin. For the original picked team, estimated margin plus its quoted spread is the new edge. Positive values favor that original pick. Each CSV row retains the two ratings and original line, so its arithmetic can be checked without the raw ESPN table. Rebuilding from the source requires the locally retained paste or supplying its path explicitly; it needs only Python's standard library. The largest disagreements with the quoted lines favor Buffalo, Jacksonville State, and UTSA at +9.7 points each, followed by UNLV at +9.5. The largest reversals are Auburn -9.1, Clemson -7.0, ECU -5.9, and Ole Miss -5.7. These differences warrant checking source freshness, canonical team mapping, and actual available prices. They are not evidence that those gaps can be bet. The original card's decimal spreads may be median book quotes rounded for display. The engine computes median provider spreads; a median need not be an executable sportsbook price. The original card's exact source snapshot has not been matched, so the audit preserves its displayed numbers. The pick engine's early-season SP+ fallback attaches historical bucket rates from its 2025 in-season EPA margin backtest. Those rates are not calibrated probabilities for this SP+ comparison. Comparing refreshed SP+ ratings with picks already based on SP+ is also not independent model confirmation. The existing [line-drift study](../analysis/output/line_drift_study.md) found 1,669 wins in 3,306 graded bets following drift at the archived final number, or 50.48%. It used retrospective captures and is not a prospective execution test. Its opening-price diagnostic uses future movement and must not be presented as an executable strategy. This audit does not promote any model, change the ledger, refresh odds, or deploy dashboard data. ## September 8 external forecasts and market quotes Run `python3 analysis/tracker_card_comparison.py` to reproduce the [Tracker comparison](tracker_card_comparison_20260908.csv). It retains manually transcribed system average and median home-margin forecasts from [The Prediction Tracker](https://www.thepredictiontracker.com/predncaa.html), whose displayed update was September 8, 2026, 12:11 PM (timezone unspecified). The downloadable CSV returned HTTP 403; this is a bounded transcription of 25 matchups, not an automated feed. Positive home margins mean home favorites. At the original displayed spreads, nine picks have positive edges under SP+, Tracker average, and Tracker median; five have negative edges under all three; eleven are mixed. Positive: Buffalo, Ohio State, Hawaii, Bowling Green, Vanderbilt, Louisiana Tech, Rutgers, Minnesota, UTSA. Negative: Marshall, Fresno State, Auburn, Ole Miss, Missouri. Average and median summarize overlapping systems, so these are not three independent votes or calibrated probabilities. Run `python3 analysis/card_market_audit.py` to reproduce the [50 provider quotes for all 25 games](card_market_audit_20260908.csv) using the [retained CFBD response](../data/external/cfbd_20260908_research/lines_week2.json) and [capture metadata](../data/external/cfbd_20260908_research/metadata.json). The authenticated request returned 86 Week 2 games. Rows retain game IDs, start times, providers, opening and captured spreads, and recalculated forecast edges at each provider's number. These are CFBD-reported quotes; the response has no spread juice or provider quote timestamp to verify execution. `movement_toward_pick = opening_pick_spread - captured_pick_spread`: positive means the picked team became more favored. UNLV moved from -6.5 to -3.5 at DraftKings, against the original pick; Tracker also opposes UNLV while SP+ supports it. Auburn moved from -29.5 to -33.5, toward the original pick, but all three forecasts oppose Auburn at the captured price. Buffalo's quotes were unchanged and all three forecasts support it. Movement and value at the new price answer different questions. Neither establishes who placed bets. ## September 9 Week 2 refresh `picks_refresh_audit_20260909/` preserves the September 9, 20:26 UTC payload and its arithmetic and selection audit: 49 games, 26 picks, and no audit errors. Reproduce with `python -m analysis.audit_picks exports/picks_latest.json --output exports/picks_refresh_audit_20260909`. The projection basis remains `preseason_sp_plus`; historical EPA bucket rates are not calibrated probabilities for this early-season model. The local `external_predictions_refresh_20260909/` and `external_predictions_direct_refresh_20260909/` directories contain fresh Tracker and direct-provider comparisons against that saved pick payload. Tracker supplied 1,916 forecasts across 49 matched games. The direct import retained 2,020 team-rating rows and produced 2,187 combined forecasts across the 86-game provider schedule. Coverage varies by system. Immutable captures are under `data/external/prediction_tracker/20260909T202722841822Z` and `data/external/direct_ratings/20260909T202744022757Z`. Each output directory retains provenance and exact schedule and model inputs for the README's offline replay commands. These ignored local captures remain research inputs; production model weights and historical evaluations are unchanged. ## Week 3 Tracker review `tracker_review_week3_20260914/` replays the immutable September 15, 00:45 UTC Tracker capture against the saved Week 3 card. Reproduce from the repository root: ```bash env -u PYTHONPATH -u PYTHONHOME .venv/bin/python -m analysis.prediction_tracker \ --snapshot data/external/prediction_tracker/20260915T004528613552Z \ --season 2026 --week 3 \ --schedule exports/external_predictions_week3_20260914_exploration/schedule_input.csv \ --picks exports/external_predictions_week3_20260914_exploration/model_picks_input.json \ --output exports/tracker_review_week3_20260914 ``` The original raw capture and exact CSV inputs remain private local artifacts. The full pre-kickoff bundle is preserved privately in `tracker_private/2026-week3.json`. `tracker_snapshots/2026-week3-summary.json` is derived by `analysis.tracker_workflow export-summary`: it retains only Tracker average, median and the updated market quote, the original parsed pick card, Week 3 schedule, matching audit, capture metadata, raw-input hashes and total coverage counts. Its source-snapshot checksum links to the unchanged private freeze. Derivation preserves the original freeze time and evaluation results; it does not establish a new capture time. Individual-system forecasts are not included in the tracked summary bundle. Each checksum covers its own bundle. The root README documents offline replay and post-week grading commands. `tracker_review_selection.json` selects this exact bundle for the dashboard. The retained counts are assertions about this snapshot, not future refreshes. The normalized source is Todd Beck's Prediction Tracker downloadable NCAA CSV, acquired September 15, 2026 at 00:45:28 UTC by `analysis.prediction_tracker`. Its raw SHA-256 is `02897dc4520039cc10cabe9c9115a7249dc8763e9a6913ff48a787bfd0e6a071`. Source forecasts belong to their authors; redistribution rights for the raw feed have not been established. The dashboard exposes the bounded saved-card review, not the raw feed or all individual-system forecasts. `codex/week3-tracker-review` and `codex/sp-plus-week3` both pointed to `42f99d0` when integration began. The isolated integration branch retains that commit once and merges the current `origin/master`. Unrelated uncommitted SP+ work in the shared checkout was left in place. Neither original branch was reset, deleted, or separately cherry-picked. Verified source-name aliases recover all 57 games. Of 22 original selections, 18 have positive edges under both Tracker summaries, three have negative edges under both, and one is mixed. Eleven retain the card's three-point threshold under both. Twelve unselected games qualify as research candidates. All edges use the saved CFBD spread; 29 Tracker updated quotes differ from that spread. The review does not change the published card or use historical bucket hit rates as confidence estimates. The payload's `preseason_sp_plus` basis is preserved; its ratings provenance identifies a current CFBD SP+ capture. ## Season previews `season_previews_2026//` holds a three-artifact set per FBS team: the payload JSON (every figure in the report), the rendered HTML, and a 5-page Letter PDF. `season_previews_2026/manifest.json` is the run record — it lists per-team status, expected wins, warnings, and any failure reason. `season_previews_2026/index.html` plus one page per conference make the run browsable; those pages are built from the manifest and payloads, so they cannot link to a report that does not exist. The pipeline is deterministic and reads only committed snapshots (`data/external/cfbd_2026/`, `data/external/connelly_*_2026/`), so it needs no live API call and no model reasoning. Data assembly (`analysis/season_preview_data.py`) is separated from rendering (`analysis/season_preview_report.py`) so a figure on the page always traces back to the stored payload. Publication uses `season_reports/2026.json.gz`, a deterministic HTML-only bundle of the existing July 29, 2026 run: 138 team reports and 11 navigation pages. It preserves source-page SHA-256 hashes and a sanitized team index, excludes raw JSON, renderer paths, provider captures and PDFs, and redirects PDF links to HTML. Every page states its preseason snapshot and illustrative independent game probability assumptions. The original source tree remains private. The bundle adds no new model calibration or live injury information. To rebuild this reviewed bundle from the preserved local report tree: ```bash uv run python scripts/bundle_season_reports.py exports/season_previews_2026 exports/season_reports/2026.json.gz npm --prefix web run prebuild ``` Prebuild checks every local report link and fails on missing assets or unsafe paths. Vercel's allowlist includes only the 2026 bundle; add future seasons explicitly after review. `verify_research_upload.py` requires this bundle and rejects private report/provider trees. `verify_live_site.py` also compares the built report hub, year inventory, year index and a team page after deployment. No provider fetch is required for publication. PDFs remain local. Regenerate everything with `./scripts/build_season_previews.sh`, a subset with `./scripts/build_season_previews.sh "Michigan State" "Louisville"`, or skip PDF rendering with `FAST=1`. A weekly `no_agent` cron job runs the same script and reports only teams whose projection moved. Keep reproducible outputs here rather than in the repository root. When the generating command is unknown, preserve the artifact in a descriptive subdirectory and add a short provenance note instead of guessing. `public/` is a separate lane: it contains the deploy-ready static dashboard snapshot generated by `uv run python scripts/build_static.py`. ## Home Margin residual diagnostics `uv run python -m analysis.residual_diagnostics` reads the existing `pooled_production_2017_2025.csv` and writes `residual_diagnostics_2017_2025/`. `cohorts.csv` and `coverage.csv` compare EPA-OLS and recorded market spreads on common games by evidence class, season, week, team and absolute spread bucket. `residuals.csv` retains all archived games; `home_margin_errors.png` shows predicted versus actual Home Margin and signed errors. `summary.json` records the input hash, scope and sparse-sample limits. These are retrospective outputs, separate from frozen prospective bundles. See the evaluation audit in [`docs/methodology.md`](../docs/methodology.md) for the inventory and cutoff findings. The generated `index.html` report and its downloads are staged at `/research/exports/residual_diagnostics_2017_2025/index.html` during the web build and linked from `/lab/research`. Vercel includes this explicit bundle. ## CORE rating comparisons `analysis.core_ratings` writes private `core_/` research directories. The September 15, 2026 capture at `data/external/core/20260915T191635640063Z` contains 138 teams through regular-season Week 2, model `core-preseason-v1`. The reviewed output is `core_2026_20260915_week3/`; its six files reproduce byte-for-byte from the saved capture and schedule. Earlier local output `core_2026_20260915/` predates the three reviewed CORE team-name aliases and is retained only as an intermediate artifact. | Reference | Matched teams | Overall rank correlation | Inclusive cutoff | | --- | ---: | ---: | --- | | Opponent-adjusted EPA | 137 / 138 | 0.6673 | Regular Week 2 | | Local predictive SRS | 138 / 138 | 0.8783 | Regular Week 2, plus prior-season carryover | | SP+ | 138 / 138 | 0.8633 | Unknown; captured workbook tab is Week 3 | Northwestern has no EPA reference row. The largest overall rank disagreement against EPA is Georgia Southern, CORE 114 versus EPA 38 on the matched cohort. Against local SRS it is Nevada, CORE 73 versus SRS 128. Against SP+ it is North Texas, CORE 48 versus SP+ 107. Ranks use average ties and each pair's matched cohort, so ranks across these comparisons need not match the full CORE board. These are descriptive disagreements, not evidence that CORE forecasts better. `core.csv` retains native CORE components, play counts, version, boundary and prospective admission status. `comparisons.csv` includes component-specific ranks, source team names, boundaries and cutoff flags. `rank_disagreements.csv` sorts overall gaps by magnitude. `missing_data.csv` audits absent teams and metadata. `summary.json` retains coverage, correlations, source/code hashes and reference metadata. `target_schedule.json` preserves the supplied CFBD schedule. All 138 CORE rows pass the necessary time gate for regular-season Week 3's first FBS-involving kickoff, September 17 at 23:30 UTC. This does not certify SP+ or the other reference boards for prospective evaluation. Acquisition at September 15 19:16:35 UTC is distinct from provider publication time. ## Historical game-flow prototype Issue #55's standalone view is `game_flow_401628439/index.html`, with its replay payload in `game_flow.json`. The same page is published through the existing research library as `game_flow_401628439.html`, served at `/research/exports/game_flow_401628439.html`. Open the HTML directly; it needs no server, API key, external JavaScript, or polling. The slider and previous/next controls show source score, period, clock, possession, and play text. Reproduce from the immutable capture without network access: ```bash uv run python analysis/historical_game_flow.py replay \ data/external/game_flow/401628439_20260915T191715Z \ --publish-html exports/game_flow_401628439.html ``` To capture another completed game, using the ignored `.env` key: ```bash uv run python analysis/historical_game_flow.py capture --game-id 401628439 ``` Each capture creates a new timestamped directory; existing directories are refused. SHA-256 verification precedes replay. The source catalog is [`data/external/game_flow/README.md`](../data/external/game_flow/README.md). ### Evaluation and limits The 2024 Georgia Tech at Georgia capture contains 213 plays, 164 matched probabilities and 49 missing probabilities. All 164 matched score pairs agree between endpoints. There are 37 overtime plays, all labeled period 5 in the raw source, so the view does not claim to identify individual overtime rounds. Drive number and within-drive play number establish order; raw response order, wallclock timestamps and the overtime clock do not. Duplicate order keys and conflicting provider probability order fail replay. The provider's synthetic `playId=0` terminal row resolves to home probability 1 but retains a stale 40–40 score. The verified final score is 44–42 from `/games`. The view displays the terminal record separately, preserving missing late-overtime probabilities and the score discrepancy. Other unmatched provider rows remain in the audit, rather than being attached to a guessed play. Current official [methodology](https://api.collegefootballdata.com/win-probability) defines after-play home probability. Matching scoring-play scores corroborate that convention here; source down, distance and clock remain play context, not a reconstructed after-play state. The documented model eras are through 2024 and 2025 onward, with no backfill across that boundary. The API has no fitted-model version field; its schema version must not be used as one. Browser inspection verified the chart, next-play control and final overtime selection. The prototype is useful for inspecting available historical paths. Defer static game-detail integration until a separate multi-game coverage check, including the 2025 model era, establishes an acceptable missing-data policy. This one-game exercise measures neither calibration nor predictive improvement. Publishing the standalone research page changes no runtime model or live refresh process.