Content History
RT Pamela Fox Nice post by @isaac_flath on how he evaluates OCR models, with custom annotation apps and agent citation checkers. Great examples of the many ways OCR/VLMs can get things wrong too - like oversimplified tables, swapped digits, and ignored watermarks. https://isaacflath.com/writing/document-evals
Q: How do I make the case for investing in evaluations to my team? A: Review real user interactions and show your team the problems you find. Fix them, then show what improved. https://hamel.dev/blog/posts/evals-faq/how-do-i-make-the-case-for-investing-in-evaluations-to-my-team.html
Q: Can I use the same model for both the main task and evaluation? A: Usually, yes. Test whether the judge agrees with human labels on a held-out test set. https://hamel.dev/blog/posts/evals-faq/can-i-use-the-same-model-for-both-the-main-task-and-evaluation.html
Q: Should I use "ready-to-use" evaluation metrics? A: No. It's a bad idea in most situations. https://hamel.dev/blog/posts/evals-faq/should-i-use-ready-to-use-evaluation-metrics.html
Saw this job posting for an "anti-slop" professional writer but the post itself written with a slop cannon. This is peak AI psychosis, someone is so enamored with their own writing they can't even see it
Q: Are similarity metrics useful for evaluating LLM outputs? A: Similar wording does not tell you whether an answer works for your application. Check specific failures. Similarity metrics can help with retrieval and output diversity. https://hamel.dev/blog/posts/evals-faq/are-similarity-metrics-bertscore-rouge-etc-useful-for-evaluating-llm-outputs.html
RT Bryan Bischof fka Dr. Donut Huge
We are making Markdown files work natively across Google Drive and Docs. You can now preview .md files in Drive, open them in Docs, edit them, comment on them, and collaborate without converting the file into a Google Doc!
Q: How often should I run my evals? A: Weigh the cost and how saturated the eval is against the business value of catching the error. Read more: https://hamel.dev/blog/posts/evals-faq/how-often-should-i-run-my-evals.html Read all the FAQs here: https://hamel.dev/blog/posts/evals-faq/
You have to respect Schmidhuber’s commitment to the bit 🤣 he was just trolling us all along (which is probably the right way to use this site tbh)
RT Shreya Shankar One of the biggest requests we get is “how do I prepare for an applied AI evals interview?” No better person to give this talk than Hamel. I have heard this talk and it’s fantastic. Free and available to all
Last open lesson of the year! How to crack the AI Evals Interview! AI Evals are now a part of an increasing number of AI job postings, and you're now more likely to encounter them in interviews. I've helped design quite a few evals interviews, so I know what kinds of traps
Last open lesson of the year! How to crack the AI Evals Interview! AI Evals are now a part of an increasing number of AI job postings, and you're now more likely to encounter them in interviews. I've helped design quite a few evals interviews, so I know what kinds of traps exist, along with nuanced things companies tend to look for. I'll discuss this, as well as how to structure a take home or report to help you think through the problem clearly. Tuesday Oct 6, 8:30am PT. This one assumes some basic evals knowledge. Read this first if you're new to the topic to get the most out of the session: https://hamel.dev/evals-faq Sign up here: https://maven.com/p/07e7c7
When I see this it confuses me. ssh into a computer for god sakes, or use codex remote feature w/ mac mini. Once you have been stuck in "i need to keep my laptop open" more than 3 times, its crazy to keep this insanity going.
Q: Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales)? A: Binary labels force clearer thinking and more consistent labeling, and involves significantly lower complexity to operationalize. https://hamel.dev/blog/posts/evals-faq/why-do-you-recommend-binary-passfail-evaluations-instead-of-1-5-ratings-likert-scales.html
RT Gergely Orosz LLM bots are talking in my replies, this site is having a real problem blocking these accounts. Confusingly, a lot of these bots have a blue check Amusing but also depressing
Q: Should I build automated evaluators for every failure mode I find? A: No. Different types of evaluators have different costs (code vs. LLM judge), which you need to weigh before building an eval. https://hamel.dev/blog/posts/evals-faq/should-i-build-automated-evaluators-for-every-failure-mode-i-find.html
People keep asking about this new Auto Eval plugin for Claude this so I wrote a longer post reviewing it. Would love to hear what other people think! If you used it how did it go? Full writeup here: https://hamel.dev/blog/posts/claude-auto-evals/
Claude can now help you build evaluations and hillclimb on them. In this article, we share guidance on eval design & skills that Claude Code can use to improve your applications. https://claude.dev/blog/automating-eval-design-and-hillclimbing/
View quoted post
Q: How can I efficiently sample production traces for review? A: Start with broad exploration, then use signals to find likely failures. Keep some random traces in every batch so you can discover new problems. https://hamel.dev/blog/posts/evals-faq/how-can-i-efficiently-sample-production-traces-for-review.html
Q: How can I do AI evals when traces contain sensitive data? A: Four approaches: negotiate access to a defined scope, work with authorized experts, redaction and synthetic data. https://hamel.dev/blog/posts/evals-faq/how-can-i-do-error-analysis-when-production-traces-contain-sensitive-data.html
Success!!! 🥰
Evals have been coming up more and more in my conversations with podcast guests and PM friends. Nearly half of the 25 awesome PM job openings I shared last week ask for experience writing evals. And leading companies keep sharing what investing in evals bought them: — @tryramp
Q: Can I help with evals if I'm not a domain expert? A: Yes, you can help more than you think! Especially in the beginning, where there's often low hanging fruits that can be uncovered through looking at data. https://hamel.dev/blog/posts/evals-faq/can-i-help-with-evals-if-im-not-a-domain-expert.html
Love this
We instituted an official AI Writing Policy at @Clay. Massive thanks to @sophiebits on our engineering team who wrote this. Originally this was just for eng, but other teams found it so helpful we expanded it company-wide. Here are the four guiding principles: 1. You must
RT Varun Anand We instituted an official AI Writing Policy at @Clay. Massive thanks to @sophiebits on our engineering team who wrote this. Originally this was just for eng, but other teams found it so helpful we expanded it company-wide. Here are the four guiding principles: 1. You must stand behind every idea and sentence It is your responsibility to make sure that the entire document is representative of your own thoughts before you share it. 2. Writing is thinking Spending time on the writing process teaches you more about your topic. If you circumvent this process, you will walk away with a poorer understanding of the subject matter. 3. More time should be spent writing a document than consuming it If you generate a document from a short prompt then ask your readers to go through the longer output, you are disrespecting their time. They can talk to ChatGPT themselves if they want to. 4. Longer is not better AI makes it easy to generate long docs, and it loves padding them with sentences that say nothing. If you're producing docs from a short prompt, consider just sharing the prompt. You can read (and borrow) the full policy below. Hopefully we all communicate a bit more clearly now :)
This is an exciting project ya’ll and it’s just getting started! Some impressive performance numbers already! Can’t wait to see what else is going to come out of this lab https://fsdatalab.github.io/
Excited to share Quail, our new open source AI-SQL engine (a collab with Modal)! By planning queries and LLM inference together, it reaches 1B+ input tokens/min on one H100 for one query 😱🚀 AI-powered data operators create a new, interesting inference workload👇
View quoted postQ: Can I use Jev for evals? A: Jev and an LLM judge are both classifiers that you should validate against trusted labels. Read more: https://hamel.dev/blog/posts/evals-faq/can-i-use-jev-for-evals.html Read all the FAQs here: https://hamel.dev/blog/posts/evals-faq/
RT Charles 🎉 Frye New on the @modal blog: a write-up by me and Shreya on our work to speed up AI-SQL queries - why AI-SQL? why high-throughput inference? - how can we possibly be >10x faster than vLLM? - what does the future of inference engineering look like? Read here: https://modal.com/blog/quail-billion-tpm
Excited to share Quail, our new open source AI-SQL engine (a collab with Modal)! By planning queries and LLM inference together, it reaches 1B+ input tokens/min on one H100 for one query 😱🚀 AI-powered data operators create a new, interesting inference workload👇
View quoted postQ: Should I record problems that aren't the model's fault? A: Yes. Write down anything that makes the product less useful. Decide which problems to fix before investigating root causes. https://hamel.dev/blog/posts/evals-faq/should-i-record-problems-that-arent-the-models-fault.html
RT Shreya Shankar I feel this pain. The corollary here that I’ve noticed: if you send cold emails or messages to me that *aren’t* written or inspired by AI, you are *way* more likely to get a response than in the pre-AI days. In some ways, the bar has never been lower
"Sorry for the delayed response! Middling generative AI has fundamentally altered the strategic balance between offense & defense in the war for our attention, so I have chosen to ignore this messaging channel. Out beyond kino & slop, there is a field. I'll meet you there."
RT Randy Olson I decided to take the research and expert consensus on what makes a data visualization effective and turn it into an open source agent skill that anyone can use. It's called evident-charts and works in Claude Code, Codex, Cursor and more. Really happy with how it turned out!
Q: What is the best approach for generating synthetic data? A: Define the kinds of requests you need to test. Generate combinations of those details first, then turn them into queries and run them through your application. https://hamel.dev/blog/posts/evals-faq/what-is-the-best-approach-for-generating-synthetic-data.html
Q: How can I make AI outputs easier for people to evaluate? A: Start with the product design. Keep the human in the loop by designing workflows that walk through intermediate outputs. https://hamel.dev/blog/posts/evals-faq/what-if-human-reviewers-approve-ai-outputs-without-checking-them-carefully.html
This is the most valuable free resource we've created on AI Evals 🎉 (not exaggerating!) I organized our public materials into this guide. It allows you to find answers to your eval problems w/o searching aimlessly. Humans: pick the row that sounds like you. Agents: point it at the post The material draw on 60+ hours of office hours from our Evals course, where @sh_reya and I have taught 5k engineers and PMs Evals. We add new material often, with 15 FAQs added in the last two weeks. Recent additions are marked with a "New" badge. You can find all of these and more here: https://hamel.dev/blog/posts/evals-faq/#how-to-use-this-faq
Q: What are AI evals? A: Tests that tell you whether an AI system is doing what you want. Model benchmarks and product evals answer different questions. https://hamel.dev/blog/posts/evals-faq/what-are-llm-evals.html
RT Sam Julien Can confirm @HamelHusain and @sh_reya are the eval GOATs. Took their course last year and it fundamentally changed how I build AI products & platforms. I also cite them in my evals chapter in my O’Reilly book. 10/10 would look at my data again
Evals have been coming up more and more in my conversations with podcast guests and PM friends. Nearly half of the 25 awesome PM job openings I shared last week ask for experience writing evals. And leading companies keep sharing what investing in evals bought them: — @tryramp
RT Shreya Shankar Very exciting to see how much this post resonated with the community. The uncomfortable dialectic is that (1) evals are necessary to build good agents, but also (2) agents are extremely helpful (and necessary) for automating tedious parts of the evals lifecycle. Human attention and labeling simply cannot scale up to the high amount of unstructured info that is agents. Unfortunately nobody has cracked the most perfect and efficient workflow to do evals. We share our thoughts on our current iteration in this post!!
Evals have been coming up more and more in my conversations with podcast guests and PM friends. Nearly half of the 25 awesome PM job openings I shared last week ask for experience writing evals. And leading companies keep sharing what investing in evals bought them: — @tryramp
I recommend grabbing these skills that @sh_reya and I created and at the very least doing an /eval-audit of your existing pipeline. We've found that people often find low hanging fruits! https://github.com/ai-evals-course/evals-skills
Pro tip: Install this new evals skill from @HamelHusain and @sh_reya, it'll save you many hours and a lot of mistakes https://github.com/ai-evals-course/evals-skills
RT Lenny Rachitsky Pro tip: Install this new evals skill from @HamelHusain and @sh_reya, it'll save you many hours and a lot of mistakes https://github.com/ai-evals-course/evals-skills
Evals have been coming up more and more in my conversations with podcast guests and PM friends. Nearly half of the 25 awesome PM job openings I shared last week ask for experience writing evals. And leading companies keep sharing what investing in evals bought them: — @tryramp
Q: What should you do when your "gold" eval dataset becomes stale? A: Use regular error analysis to find new problems. Keep updating your evals as your product and users change. https://hamel.dev/blog/posts/evals-faq/what-should-i-do-when-my-gold-eval-dataset-becomes-stale.html
This post was a labor of love! We’ve distilled thousands of hours of work on AI Evals into a 30 min read + skills you can use to quickly uncover errors in your product. https://substack.com/home/post/p-216168140 BTW in addition to high quality content, Lenny effectively **pays you** in AI credits to subscribe https://www.lennysproductpass.com/ 🤯 It's really good!
Evals have been coming up more and more in my conversations with podcast guests and PM friends. Nearly half of the 25 awesome PM job openings I shared last week ask for experience writing evals. And leading companies keep sharing what investing in evals bought them: — @tryramp
RT Lenny Rachitsky Evals have been coming up more and more in my conversations with podcast guests and PM friends. Nearly half of the 25 awesome PM job openings I shared last week ask for experience writing evals. And leading companies keep sharing what investing in evals bought them: — @tryramp took its automatic receipt collection from 35% to 83% accuracy. — @Shopify shipped an AI workflow builder that's 2.2x faster and 68% cheaper than the frontier-model setup it replaced. — @harvey__ai rebuilt its AI contract reviewer, nearly doubling its internal quality score. — @cursor_ai tuned its Auto Balance routing, with much higher user satisfaction at 41% lower cost. So I asked the 🐐s of evals, @HamelHusain and @sh_reya, to write an advanced sequel to their very popular "Building eval systems that improve your AI product." Drawing on their work with 50+ AI companies, they share the key step most teams skip, what you should (and shouldn't) automate, and a free plugin that lets a coding agent do most of the heavy lifting. Read it here: https://www.lennysnewsletter.com/p/advanced-evals-how-to-find-and-fix
What’s wrong with calling it a classifier? Its an established term that describes exactly what this is, along with corresponding literature on how to verify, tune, measure it. Jev is very cool and super useful, because of speed, cost, accuracy and ergonomics. But new terms might hurt more than help. What are your thoughts?
Can we agree to call Jev a "Decision Model"? I feel like they buried this in the explainations and docs. Language models output language. Decision models output decisions.
Can you use Jev for Evals? Yes! Remember that a LLM Judge is also classifier*. Make sure to test your classifiers against human labels and don't overfit. Hope this helps! * https://hamel.dev/blog/posts/evals-faq/why-do-you-recommend-binary-passfail-evaluations-instead-of-1-5-ratings-likert-scales.html
Q: What is a trace? A: The complete record of a user session, from the first query to the final response. It includes the tool calls and intermediate steps. https://hamel.dev/blog/posts/evals-faq/what-is-a-trace.html
If you are a Data Scientist, you have never had more alpha than right now. Data Scientists email me all the time. It is no mistake that I'm focused on evals as a former DS. I talk more about this here: https://www.youtube.com/watch?v=lA4MfpgF91Y
Q: How do you review a trace that is really large? A: Focus on the first upstream failure. Make the relevant evidence easy to inspect, with details reviewers can expand as needed. https://hamel.dev/blog/posts/evals-faq/what-if-the-source-material-is-too-large-for-a-person-to-review.html
