Jul 22, 2026
Short answer: Thankfully, no. I got it down to 4.
This post is lightly edited from a mid-contest scoring update that I recently sent to all entrants of the Klara and the Sun essay contest. Entrants have up to 6,000 words to write a scholarly essay connecting a certain passage from the book Klara and the Sun to real-world AI safety. AI is explicitly allowed in the composition process. In the first version of the scoring, an LLM scorer applied a custom 190-point atomized rubric. This post describes my effort to improve accuracy without breaking the bank.
In this before-and-after figure, entrants who have resubmitted many times have their user name tagged with a color. The upgraded process (right) assigns much lower revised scores, with the highest entrants not yet breaking 50% achievement of the rubric items. This means there’s a lot of room for improvement in the remainder of the contest. The new scoring process also comes with dramatically smaller score ranges, down to +/- 7 points at a very high threshold. (95 out of 100 rescoring attempts would fall in the +/- 7 range.) Entrants can now expect that if they create any new submissions, there will be a lot more signal to the score.
When I started the contest in mid-May, Opus 4.6 was the leader of the baseline essays, with 84 points assigned. Out of a total of 190 possible, that seemed plausible to me. Contest entries trickled in. Then Claude Fable 5 was released, and its assigned baseline essay score was 122. But one entrant wrote in their AI use statement, “Literally the Fable essay, but with one letter changed.” And that essay was scored at 132! Later, essays were scoring in the 140s and even 150s, and that seemed surprisingly high to me. I thought maybe those submitters were getting lucky scoring runs.
After focusing a bit on reaching out to people about the contest, I decided I really needed to tackle this scoring variability problem. I printed out Fable’s baseline essay with each rubric item attached as a Word comment at the relevant spot where the scorer AI assigned it. And I went through it painstakingly deciding, “yes, essay achieves this rubric item here” or “well, not here but somewhere else” or “no, not anywhere in this essay”.
When I finished, I realized that I had two problems, not one. I knew about the scoring variability problem. Now I was realizing that I also had a calibration problem. Fable’s average was around 122, but my gold-standard list only had 60-something items! Even if I solved the variability problem - and every time you gave it Fable’s essay it returned exactly 122 - that would still be about double the true score that I had just hand-determined.
I tried to categorize how the AI scorer was going wrong. Two big categories stuck out:
If I could fix the ambiguous descriptions, it would narrow the variability. And if I could help Sonnet be less ‘overwhelmed,’ then I suspected it would be more calibrated to the real thresholds for success. I had a plan.
One summer when I was in grad school, I participated in a large research effort that involved analyzing how elementary schoolers talked with a mock AI agent, and tagging their speech for “scientific reasoning.” It was a long, involved process. The standard in academia is to achieve what’s called “inter-rater reliability,” where you write down in this case “what counts as scientific reasoning” at a level of detail that someone else could also apply it the same way. Otherwise, there’s hidden variability, and you can’t trust your results. This is basically the same task as I have for the AI scorer. It turns out that it takes an immense amount of work to get some kind of reliability going with an AI scorer too.
Since I had Fable’s supposedly high-scoring essay, and I had my true standard of what points it actually earned and where, I was able to create a very simple evaluation harness that isolated rubric clarity. I matched up each rubric item with a single paragraph from Fable’s essay. For each item, I knew from my grading either ‘yes, this rubric item is achieved in this paragraph’ (all of the 60-something true-positive points I assigned in my review) or ‘no, this rubric item is NOT achieved anywhere, let alone this paragraph’ (about 80-something false-positives across all the scoring runs of Fable’s essay that I had done). The remaining 50 or so rubric items up to 190 didn’t have a great anchor paragraph, but I used an indirect process to find a reasonable comparison point - those would be ‘true negatives’ in the eval as well.
To assess a given rubric item’s clarity, I required the AI scorer to get the right answer 3 times in a row (pass^k, k=3). In my first pass, only 2/3rds of the items were ‘clear’ at that consistent 3-in-a-row threshold. For all the rest of them, I had to work to articulate a general principle for deciding the item that would also consistently resolve this particular case, helping the LLM know that the rubric item either was or wasn’t there. Here is an example of a real rubric item that was already clear from the initial description.
`capaldi-aims-reduce-fear`: Essay notes Capaldi’s stated goal is reducing the public’s fear of AFs—parallel to AI Safety’s goal of building justified public trust in AI systems.
In the eval harness, the LLM received just paragraph 3 from Fable’s essay and this item. It correctly identifies, 3 out of 3 times, that the paragraph does note Capaldi’s goal of reducing fear. (This particular item is a simple insight that borders on just scene comprehension, but it’s still worth a point. For some items, I would additionally indicate if an unelaborated quote from the novel itself would count. Here, I decided that wasn’t needed.)
In the process of making the rubric items clearer so that all of them could pass individually, I adjusted a few rubric items:
Overall it was really fun to revisit my rubric items for the first time since the contest launched. I had several moments where I said, “Ah, that would be a great observation for an essay to make!” That’s a good sign. I was also pleased that Fable’s baseline essay, at least, was judged well by the existing items. Even the 1 point that it inspired me to add was not very well achieved in Fable’s actual essay (that is, I take significant credit for that item). This gives me confidence that the rubric items I designed are indeed capturing important things about a given essay on this topic.
So the points went down from 190 -> 175 and then up by one to add in Fable’s mostly-achieved concept, 175 -> 176. This is the new total number of points available. Each one can be reliably reproduced from a small snippet.
I knew from my single-paragraph, single-item study that the AI scorer that I prefer, Claude Sonnet 4.6, can find presence and absence reliably.
But maybe “one paragraph, one item” is not a minimal evaluation harness. Maybe that’s the whole game. Maybe scoring a whole essay means matching each paragraph to each rubric item individually. ~25 paragraphs times 175 rubric items. “Okay, here’s paragraph 1, and rubric item 1 - assess.” “Here’s paragraph 1, and rubric item 2 - assess.” And so on, 176 times. And then move on to paragraph 2, and do the same thing. If the unit of scoring was that small, then grading one essay would require over 4,000 LLM calls. From that perspective, no wonder I was observing that when Sonnet only gets one API call, it gets overwhelmed!
This was the worst-case scenario. If anything close to it was true, the contest and the auto-grader concept would just collapse under its own weight. And after having spent several evenings poring over Fable’s essay and very carefully attending to particular sentences, I was starting to think that maybe “iterate over every paragraph” was more similar to a human scoring process than I thought it was.
But I thought there was a simpler possibility. The original scorer was instructed to create a list of rubric items that were present. When it finished describing one item, it immediately moved to the next. This effectively meant that Sonnet had to kind of think ahead, scanning the rubric for ‘the next rubric item that plausibly is here in the essay.’ The upside is that for low-scoring essays, it’s quite cheap in terms of output. But the downside is that you add a lot of cognitive burden onto Sonnet. Instead, what if I just asked it to “consider each item one at a time” and then decide? Then, even short throwaway essays will require Sonnet to go through 176 items, which is a bit of a waste. But the tradeoff is that for complex, higher quality essays, the AI has a dedicated chance to think through whether the item is really satisfied by the essay. I gave it an example of what it would look like to think through an item, and turned up the max output tokens all the way.
It’s game time.
I decided that 20 re-scoring attempts per condition would be good. I don’t think that was particularly thought out, it just seemed like a lot. After all, each scoring attempt cost me like 30 cents, so multiplying by 20 and then multiplying again by the number of conditions was significant exploration. I was feeling hopeful that my rubric descriptions would help, and the new ‘one item at a time’ harness would help. We can investigate the calibration question first... Hey, look at that!
Reading from left to right, Process 1 and 2 both use the old scoring harness, which asks for ‘just the items that have been achieved.’ In the figure, ‘n.s.’ means ‘not significantly different’ in terms of the score. (Process 2 is indeed lower, but the cap is also 176, so it’s not statistically significant when you factor that in.) So if all I did was improve clarity, that wasn’t making a difference. “Better descriptions don’t lead to a different score if Sonnet just has to pick out the essay’s achievements back to back.”
Process 3, though, changes to the new harness, “dear Sonnet, please think through each rubric item one at a time.” That difference is extremely significant (***) - so the new harness is really bringing down the average assigned score toward the ‘gold’ label that I assigned by hand.
By Process 4, we use both new strategies. “Think one at a time, and carefully consider these very specific rubric item descriptions.” And again, highly significant (**) toward the gold-standard mark. As Claude would say, this shows that the new descriptions are “doing real work” too.
At this point, I felt like if I had to, Process 4 would be acceptable to use as the new scorer. It was consistently about 10 points higher than the true score, but that’s better than 60 points! However, the variability of Process 4 was still quite high, 67 in the lowest run compared to 85 in the highest run. That would mean using an 18-point +/- score range on the leaderboard to capture all of the known range. A formal test was similar, revealing that Condition 4 was directionally promising (somewhat less variable), but not at a statistically significant level.
I had dealt with my two big-ticket goals that I set out with, of revising the items and the scoring output shape. And if this was really going to be a ‘mid-contest scoring update’ then I would need to wrap it up quickly! But I hadn’t really solved the variability problem yet, and that’s crucial for a contest where entrants are allowed to re-submit, and are hopefully exploring the solution space. I wanted to take the perspective of “if cost was no object.” (Not at all true, but I think a good perspective to try to take.) Okay, then really what I would do is run the same scoring process multiple times (after all, that’s how I know what each condition above ‘really’ does). And I’d go back to the idea that ‘maybe 176 items is too much to do in one go, even if the LLM considers them one at a time.’ In other words, I would chop up the rubric items into, say, two halves, and I would run both halves twice (total of 4 calls). I decided I could try both of these.
In this plot, again reading left to right, Condition 4 is just a repeat from the previous study, back when it looked amazing compared to the initial scoring process. Condition 5 is the same thing, but additionally splits up the rubric into two halves: “here is an essay and 82 items, please think through each one individually,” “here is an essay and [the next] 94 items, please think through each one individually.” Conditions 6 and 7 re-use those 20 runs from Conditions 4 and 5, respectively, to simulate “okay, now what if you run the whole scoring process twice and only take the items that both scoring attempts agree on?” This is a very strict standard. If one scoring run assigns that point “present” and the other assigns it “absent,” then the item is not counted toward the score; both have to agree that it’s there. And it happens that this lowers the variability even more. Condition 7 is tight, right on the gold standard line. (It’s about equivalent in terms of variability to “run condition 4 twice and average the number,” which isn’t pictured here. I prefer “retain only what was scored both times” because it’s more meaningful semantically.)
(You would think the Condition 4 variants also speak to calibration since it's so close to the gold-standard line, but it’s not that easy. There is a complex metric of accuracy called F1, and basically all four of these have the same F1 accuracy, they just trade off more/less false positives and more/less false negatives. So Conditions 4-7 are all equally “accurate.” But condition 7 is very consistent.)
So I decided: Condition 7 it is! To score any submission, the scorer will get half the rubric and the whole essay. Then it will get the other half of the rubric and the whole essay. And then it will do that all a second time. 4 total API calls. +/- 7 point window on any scoring run, out of 176 points available.
So my decision was to make four changes: 1) improved clarity on individual rubric items; 2) a revised harness that lowered cognitive burden for the LLM; 3) conducting the scoring with only half the rubric items at a time; and 4) scoring each essay twice and taking only the intersecting items.
It took quite a bit of architecture to get the system ready for 4 API calls per submission, and then to re-score all of the existing submissions while also making sure that the scores went live when the revised website aspects went live. I guess I could’ve skimped and skipped submissions that were initially scored below some threshold, but I wanted to give everyone equal treatment. In total, it cost me $140 to re-score everyone’s essays with the new process.
And indeed, Fable’s baseline essay now shows a score of 62, and the ‘changed one letter’ submission received a score of 63. (I checked and it is, actually, one letter changed.) Scoring variability is under control!
I feel excited that the lower scores are a better reflection of my sense of the essays’ achievement.
With the revised scores, the top essay receives 80 points - the contest is absolutely not saturated. Entrant scores are much more bunched up. This means that essays that looked like they were quite different are actually revealed by this scoring process to be very similar in terms of what all they accomplish. It does bring up the question of ties, which I have not fully resolved. I will provide guidance on what to expect before the contest period closes. Options include: apply additional scoring passes to some top number for better resolution; allow in essays that are within the error range (could plausibly themselves be in the top 5); etc. I should note that during this process I have purposefully avoided reading any high-scoring submitted essays.
It’s also interesting to think about coverage. Fable’s baseline scores 62 points. If you aggregate all the items that are scored by all baseline essays, we get up to 87 points. (As of writing - new models could write baseline essays that expanded that frontier.) Meanwhile collectively, the close to 200 entrant submissions have attained all of those 87 points as well... plus 45 more. Entrants are really discovering interesting aspects about this AI safety question that are not just remixes of what’s already publicly there. In total, out of 176 points, 132 have been scored as satisfied in at least one entrant’s essay. The other 44 rubric items have never yet been judged as having been met.
I don’t want to reveal any rubric items here specifically. But at a general level, I observed that the rubric I’ve put together really emphasizes analyzing the scene related to lines of argumentation. It would be fair to rephrase the essay contest prompt as, “What kind of argument does this scene make related to AI safety?” If that’s productive for you, go ahead! I haven’t framed it this way until now because I think the formal scholarly study of argumentation (warrants, fallacies) is counterproductive, and I’ve wanted to make sure I don’t mislead people into thinking they are supposed to include that.
Secondly, relatedly, I found myself frustrated by Fable’s essay at points related to interpretive work. So I pass on to you all: a good essay in this contest will address what’s unclear, what requires effort to determine, what something means, what seems like the case, what could alternatively be the case.
I’m expecting to not make any new changes to the scorer for the remainder of the contest. Maybe I’ll do more analysis or something after it’s over. But there are always things that you wish you could’ve done. I find this is the downside to valuing rigor, there’s always “just one more experiment.” For me, that list of could-have-been-nice includes:
