FedCourtsAI

Status

A daily snapshot of the pipeline that makes the forecasts: whether each scheduled job is running, how many forecasts and gradings it has produced, how much of the Supreme Court's docket it follows, and the results of its back-tests. The pipeline writes this snapshot itself every morning; this page shows it as written.

Snapshot Nov 16, 08:20 UTC · ledger commit b1e3401

Is it running?

All 7 scheduled jobs are on time. The committed data passed validation.

JobScheduleLast startedLast resultState
Docket refreshPolls the Supreme Court's docket pages for new filings and orders, and enriches tracked dockets from CourtListener.16 times a dayNov 16, 07:31 UTCSucceeded70 of 70 succeeded this weekOn time
Historical walkerAdds decided petitions from past Terms to the corpus the base rates are read from.daily at 02:31, 08:31, 14:31 and 20:31 UTCNov 16, 02:35 UTCSucceeded17 of 17 succeeded this weekOn time
PredictRuns each model on the events that are owed a forecast.daily at 14:12 and 17:32 UTCNov 16, 01:19 UTCSucceeded17 of 19 succeeded this weekOn time
EvaluateGrades the forecasts on events whose outcome is now known.daily at 15:39 UTCNov 15, 15:43 UTCSucceeded3 of 4 succeeded this weekOn time
Case summariesWrites the plain-language summaries of the predicted cases.daily at 03:43 UTCNov 16, 03:45 UTCSucceeded5 of 5 succeeded this weekOn time
BoardsRebuilds the big-case board daily and the metrics roll-ups weekly.daily at 04:36, 15:33 and 16:09 UTC, plus Mondays at 05:41 and 12:59 UTC, plus Tuesdays at 09:08 and 21:21 UTC, plus Wednesdays at 09:04 and 09:22 UTC, plus Thursdays at 09:14 UTC, plus Fridays at 21:11 UTCNov 16, 05:45 UTCSucceeded23 of 23 succeeded this weekOn time
Status reportChecks the committed data and writes this page's snapshot.daily at 08:00 UTCNov 16, 08:19 UTCSucceeded4 of 4 succeeded this weekOn time

A job is late when it has not started since its most recent scheduled time, allowing two hours because GitHub often starts scheduled jobs late. Only runs on the production branch count. A run that was skipped, or replaced in the queue by a newer one, counts as started but has no result.

What it has produced

Last 7 daysNov 9, 2026 – Nov 16, 2026Last 30 daysOct 17, 2026 – Nov 16, 2026This TermOct 1, 2026 – Nov 16, 2026
Model runs3451,0431,849
Events covered112300520
Gradings, cert120180561
Gradings, interim1869114
Forecasts, cert177632926
Forecasts, interim1269116
Forecasts, merits1893132

Each model run is one model forecasting one event, or one judge grading one event's forecasts. A run is counted once its results are merged into the ledger, so the most recent day can read low.

Coverage

624events forecast251 of them now decided, 2,315 forecasts in all, and 1,773 gradings of forecasts committed before the outcome

864pending petitions distributed for a conferenceas of Nov 16, 2026

175of those with at least one filing storedthe petition, a brief or another document the models can read

185petitions at the next conferenceDec 4, 2026

14,404dockets on the public ledger

2,154,191dockets in the research corpusthe history the base rates are read from

Back-tests

A back-test replays decided cases with the outcome hidden. Every outcome was public before the run, so a back-test measures recall of known history, not foresight. None of these figures is a claim about how well the models forecast; the results are.

Historical replay

metrics/backtest.json · Nov 14, 2026

2 simple baseline predictor(s) replayed over 65,274 resolved events in the corpus. No model runs here.

PredictorDocketEventsAccuracyAlways-deny floorLiftMean Brier
constant-deniedSCOTUS cert petitions45,13095.1%95.1%+0.00%0.0284
constant-deniedSCOTUS applications13,18318.6%18.6%+0.00%0.8112
constant-deniedSCOTUS other dockets6319.0%19.0%+0.00%0.0635
constant-deniedAll courts pooled65,27470.2%70.2%+0.00%0.1873
prior-voteSCOTUS cert petitions45,13082.1%95.1%-13.06%0.1387
prior-voteSCOTUS applications13,18330.5%18.6%+11.93%0.4592
prior-voteSCOTUS other dockets6319.0%19.0%+0.00%0.0635
prior-voteAll courts pooled65,27463.6%70.2%-6.58%0.1925

Retrospective: every outcome was public before the run, so this measures recall of known history, never foresight. It is an iteration instrument, and nothing here is a claimable performance figure. Read each lift only against its own row's floor. The pooled row mixes in courts this pipeline does not predict.

Cert back-test

metrics/cert-backtest.json · Nov 14, 2026

3 model(s) and 2 reference baseline(s) replayed over 10 decided cert petitions drawn at random from paid petitions, with each outcome hidden. Always-deny scores 100.0% on granted or not over this sample.

PredictorKindPetitionsGranted-or-not accuracyLiftMean Brier
constant-deniedbaseline10100.0%+0.00%0.0000
gemini-baselinemodel10100.0%+0.00%0.0003
claude-baselinemodel10100.0%+0.00%0.0003
codex-baselinemodel10100.0%+0.00%0.0018
prior-votebaseline1090.0%-10.00%0.1172

Retrospective: every outcome was public before the run, so this measures recall of known history, never foresight. It is an iteration instrument, and nothing here is a claimable performance figure. The floor is this sample's own denial share, so a different draw is scored against a different floor and is not comparable. No evaluator runs here, so a cell that saw its own outcome stays in and can only flatter its model.

Merits back-test

metrics/merits-backtest.json · Oct 25, 2026

3 model(s) replayed over 15 decided argued case(s) from OT2024, OT2025, at the grant and briefing moments, forecasting whether the judgment below is disturbed.

PredictorMomentCasesAccuracyMajority floorBrier skill
claude-baselinegrant1586.7%73.3%+0.58
claude-baselinebriefed1593.3%73.3%+0.71
codex-baselinegrant1593.3%73.3%+0.53
codex-baselinebriefed15100.0%73.3%+0.71
gemini-baselinegrant1586.7%73.3%+0.48
gemini-baselinebriefed14100.0%71.4%+0.77

Retrospective: every outcome was public before the run, so this measures recall of known history, never foresight. It is an iteration instrument, and nothing here is a claimable performance figure. Read accuracy against the majority floor beside it. Brier skill is against a base rate that could have been known at each case's cutoff; a positive figure beats it. Cases are fenced or set aside in ways that can track their outcome, so every set here is a selected one.

Salience-gate replay

metrics/salience-replay.json · Oct 31, 2026

The gate that picks which petitions are predicted, replayed over OT2017, OT2018, OT2019, OT2020, OT2021, OT2022, OT2023, OT2024 under 3 cutoff policies.

Retrospective: every outcome was public before the run, so this measures recall of known history, never foresight. It is an iteration instrument, and nothing here is a claimable performance figure. It was produced under sal-v4 while sal-v8 is the scorer in force, so it describes that version and is not a current reading of the gate.