Follow AI Builders — Not Influencers

RT Kritin Vongthongsri a practical way to start evals if you have traces but no test suite: 1. pull 100 recent production traces 2. read them and write down what went wrong in plain english 3. group the notes into failure types 4. count each type 5. write pass/fail checks for the top 3 6. compare those checks against human labels 7. add every new failure to the dataset generic metrics can look healthy while the thing your users care about is broken. the goal is a test suite shaped like your real failures. this approach comes straight from @HamelHusain's evals faq: https://hamel.dev/blog/posts/evals-faq/

Just a normal Thursday.

an agent loop compiled into model weights

Ahrefs 修改了 DR 算法，有两个主要变化: 1.子域名可以查到自己的 DR 了； 2.DR 与 SEO 流量的关系更直接了。 尤其是第二点很重要，我们以前说，DR 不代表流量，DR 高不代表流量就高，现在 Ahrefs 的新版 DR 算法相当于在努力解决不相关性。 不过具体效果如何，得等新版数据开放才知道。

RT Advaith Sridhar Today, we announce five Grand Challenges in Material Science. We’re also committing $1M in prize money to anyone who can solve these challenges. AI can solve Millennium Prize problems in math. Can it do the same in material science? To find out, we reached out to 10 leading professors to identify the hardest open problems in the field.

RT Vasek Mlejnsky We hit an annualized run rate of $120M in October. This year alone, we’re projected to add around $150M in net new revenue. This is an amazing milestone and a testament to the work of the @e2b team. I’m particularly excited about two things: 1/ This showcases how strong the tailwinds of AGL (Agent-Led Growth) can be. If you build a tool for agents and optimize your whole growth engine to serve them, you can see tremendous growth. This isn’t something you can just throw money at and buy. Agents don’t decide based on ads. You need a sophisticated growth engine that fine-tunes things like your AEO. I think this is what all future B2B devtool sales will look like. Increasingly, decisions about tool adoption will be made by agents, not humans. 2/ The use cases we’re serving. The majority of our revenue and usage comes from agents running in production. Not RL. When we started, we had two theses: (i) code is all you need—all agents will use code to complete tasks (which has turned out to be true); and (ii) sandboxes for agentic workloads will be a bigger market than sandboxes for RL workloads, but takeoff will be slower (still playing out). In the past 30 years, every company has become a software company. In the next 10 years, every company will have become an agent company. We’re building an infrastructure layer to power all these agents.

RT Waldek Mastykarz It's tempting to blindly trust public benchmarks but don't get fooled: they're hardly representative of your work, not to mention, by now the results are likely skewed. Do your own evals! https://developer.microsoft.com/blog/what-ai-benchmarks-are-not-telling-you/

Do your own private evals folks. Don’t rely on public benchmarks. 1. Many benchmarks have material flaws 2. Some models have already memorized the answer key.

lol moved my brokerage over from schwab to robinhood but it triggered the same day as the SOXX 3-1 split and now i'm missing the extra 2x soxx shares worse computer glitch of all time holy