From the computer of Peter Clark

Fascinating repo of incredibly esoteric and difficult questions that frontier models can benchmark themselves on — Opus 4.6 scores a paltry 46%. Also discussed on the New York Times.

The questions shouldn't be shared but there are 2500 of them and they're accessible via Hugging Face — so interesting!

An example question which they shared:

Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of the expanded, cruciate aponeurosis of insertion of m. depressor caudae. How many paired tendons are supported by this sesamoid bone? Answer with a number.

I’ve been somewhat facetiously, somewhat seriously, somewhat jokingly, been posing a question to everyone I run into these past few months: Don’t you feel like all meaning is being scrubbed from the world? Like the Langoliers are chomping up purpose, chomping up all the things to which we’ve ascribed purpose these past hundred-thousand years? And that nothing matters?

Really, what I’m asking is: Don’t you think our contemporary education system has long needed an overhaul? That our society has long needed to reconfigure itself? That we need to stop ascribing all our meaning and purpose to being a Web Designer, or Coal Miner, or Airplane Engine Factory Foreman, or Accountant, but instead to being A Good Person, Good Parent, Good Friend, Curious Researcher, Poet, Meditator, Facilitator, or any number of other Ways of Being uncoupled from “work” as we’ve defined it since the industrial revolution? Who is safe from the hunger and capabilities of the models? Yoga instructors?

I teach HS Science in the south. I can only speak for my district, buta few teacher work days in the wave of enthusiasm I'm seeing for AI tools is overwhelming. We're getting district approved ads for AI tools by email, Admin and ICs are pushing it on us, and at least half of the teaching staff seems all in at this point. I was just in a meeting with my team and one of the older teachers brought out a powerpoint for our first lesson and almost everyone agreed to use it after a quick scan- but it was missing important tested material, repetitive, and just totally airy and meaningless. Just slide after slide of the same handful of sentences rephrased with random loosely related stock photos. When I asked him if it was AI generated, he said 'of course', like it was a strange question. Then, I told the team I might go in a different direction because I wanted to avoid using AI in the classroom and the team lead made a face and told me that all of the lessons she brings for our meetings have been AI generated for the last year.