Follow AI Builders — Not Influencers
I've seen a lot of pushback to this, with the following complaints: 1. "The benchmark owners already know about the flaws, and are working on it." 2. "Why are they spending time critiquing and not contributing to the benchmark" OR "critiquing known issues are not helpful". 3. "The flawed verdict is too binary" I find this pushback to be misguided. These benchmarks are used to move millions of dollars of token spend, and many people blindly trust how well these benchmarks generalize to their own products, and don't look any further than the headline numbers. For example, whenever a new model drops, you'll find thousands of youtube videos analyzing the pareto frontier using one of these benchmarks as the primary quality metric without questioning the benchmark. I think we would all be better served by inviting skepticism and encouraging people to develop an independent opinion of how well these benchmarks represent the tasks they care about. Concretely: - For the failing tasks within the benchmark, are they: realistic, have a proper harness, graded properly, sufficiently disambiguated user intent? - Are models and API calls configured properly (ex: inconsistent context windows, reasoning levels). - Are the models equipped with a modern harness and minimal retrieval that is sufficiently realistic? - Does the score even make sense ex: is it just a LLM judge committee that produces a score on a 0-100 scale that isn't aligned with ground truth? It's often the case you'll see many of these issues while looking at traces of a failing benchmark tasks, and it doesn't take long! Often the **very first trace you look at** has a serious issue. I also don't think there is an issue with verified vs. flawed verdict. It's good to have opinions. If you look through enough of what's behind their opinions and you find that you usually agree with their verdict, then they have earned your trust, the same way that a benchmark can lose your trust. I think this eval of benchm...
Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.
agi can prove p = np but not vibecode a better contact form than random gmail address
Announcing three major scientific discoveries, one each in mathematics (harmonic analysis), computer science (complexity theory), and physics (quantum field theory). AI starting to help us improve our understanding of the universe! https://openai.com/index/building-scientific-understanding-with-academic-communities/
View quoted post