Since 2024, Anthropic’s performance optimization team has given job applicants a take-home test to make sure they know their stuff. But as AI coding tools have gotten better, the test has had to change a lot to keep candidates from simply filling in all the answers with Claude.
Team lead Tristan Hume described the history of the challenge in a blog post on Wednesday. “Each new Claude model has forced us to redesign the test,” Hume writes. “When given the same time limit, Claude Opus 4 outperformed most human applicants. That still allowed us to distinguish the strongest candidates — but then, Claude Opus 4.5 matched even those.”
Candidates are allowed to use AI tools on the test, but the situation still presents a serious candidate-assessment problem. If human beings can no longer improve on the model’s output, then the test is only measuring the different models used and it’s no longer useful for finding top performers.
“Under the constraints of the take-home test, we no longer had a way to distinguish between the output of our top candidates and our most capable model,” Hume writes.
The issue of AI use on tests is already wreaking havoc at schools and universities around the world, so it’s ironic that AI labs are having to deal with it too. But Anthropic is also uniquely well-equipped to deal with the problem. In the end, Hume designed a new test that had less to do with optimizing hardware, making it sufficiently novel to stump contemporary AI tools.
But as part of the post, he also shared the original test to see if anyone reading could come up with a better solution.
“If you can best Opus 4.5,” the post reads, “we’d love to hear from you.”
Techcrunch event
San Francisco
|
October 13-15, 2026
Correction: An earlier version of this article misstated Anthropic’s policy towards the use of AI tools on the take-home test. In fact, AI use is expressly permitted. TechCrunch regrets the error.
