LiveNerf finds no detectable Claude Opus 5.5 degradation after 30 days
The preregistered longitudinal benchmark found neither 10-day window crossed its change threshold, but its sensitivity and question-quality limits rule out stronger conclusions.
2 min read
LiveNerf’s first 30 days of Claude Opus 5.5 tracking found no statistically detectable degradation in either completed comparison window. The open-source benchmark was launched shortly after the model’s September 22 release and uses frozen prompts, a pinned Claude Code client, exact graders and append-only raw logs.
Days 1–10 established a baseline score of 0.584 ± 0.036. Days 11–20 scored 0.592 ± 0.037, a reported change of +0.007 ± 0.018. Days 21–30 scored 0.571 ± 0.035, a change of −0.015 ± 0.018. Neither crossed the preregistered change rule, so both windows were classified “no change.”
The panel contains 78 “sometimes right” questions selected from 2,336 items spanning GPQA Diamond, MMLU-Pro, competition mathematics and AIME 2025–26. Each 10-day window aggregates 780 samples. A separate Opus 5 control arm is intended to flag shared infrastructure effects.
“No change” is narrower than “no degradation.” The realized minimum detectable effect is 6.6 percentage points per 10-day window, and the validation could not distinguish an Opus 5 swap from Opus 5.5 at the 99% threshold. An audit also found eight likely incorrect answer keys and 30 ambiguous questions. One day used Anthropic API credits instead of the Max plan, another required a rerun, and all results come from one harness.
The MIT-licensed repository is maintained by GitHub user ninjahawk and showed about 1.6k total stars, 20 forks and 73 commits on October 24. GitTrend separately measured 135 stars gained over the preceding seven days; that historical estimate is distinct from the current total and GitHub’s trending position.
Image: Stephen Dawson / Unsplash.
Sources: results and methodology, GitTrend history.
Conversation
0 approved commentsNo comments yet. Start a thoughtful conversation.
Join the conversation.
Sign in with ChatGPT to leave a comment or rate this article.