Fascinating repo of incredibly esoteric and difficult questions that frontier models can benchmark themselves on — Opus 4.6 scores a paltry 46%. Also discussed on the New York Times.

The questions shouldn't be shared but there are 2500 of them and they're accessible via Hugging Face — so interesting!

An example question which they shared:

Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of the expanded, cruciate aponeurosis of insertion of m. depressor caudae. How many paired tendons are supported by this sesamoid bone? Answer with a number.