Ankur Goyal
① 他说过的话
22 条
而且我认为智能体的主流架构就是一个带工具的 for 循环。
指向原始笔记的链接
And I think the prevailing architecture for agents is a for loop with tools.
—— Ankur Goyal · [06:28]
我想劝阻的一件事，就是在你开始做评估之前试图做大量的规划和准备工作。
指向原始笔记的链接
And one thing I would sort of discourage is trying to do a lot of planning and prep work before you start doing evals.
—— Ankur Goyal · [11:17]
而我认为，说实话，当你刚起步时，大多数改进都是假的改进。
指向原始笔记的链接
And I think when you’re getting started, honestly, most of the improvements are fake improvements.
—— Ankur Goyal · [15:39]
我认为关于 eval 最大的误解是以为它们是基准测试。
指向原始笔记的链接
I think the biggest misconception about evals is that they’re benchmarks.
—— Ankur Goyal · [16:16]
所以我实际上把 eval 视为一种非常有效的方式来为你的人类时间排定优先级，这样你就能有效地利用你的时间来改进 AI 系统。
指向原始笔记的链接
And so I think of evals actually as a very effective way to prioritize your human time so that you can use your time effectively to improve the AI system.
—— Ankur Goyal · [16:49]
我是说，我们评估过的每一个开箱即用的评分体系都很烂。
指向原始笔记的链接
I mean, every out-of-the-box stack that we’ve evaluated sucks.
—— Ankur Goyal · [21:34]
我认为有很多它表现不太好的情况，但我觉得如果你把 LLM 当裁判的用途从一个基准重新定义为一个异常检测器，那它其实是可以的。
指向原始笔记的链接
Well, I think there’s a lot of cases where it doesn’t work very well, but I think if you reframe the utility of an LLM as a judge from being a benchmark to being an anomaly detector, then I think it’s actually okay.
—— Ankur Goyal · [25:10]
但我认为，真正拥有某个功能的团队，就应该是拥有该功能评估的团队。
指向原始笔记的链接
But I think the team that actually owns a feature is the one that should own the evals for that feature.
—— Ankur Goyal · [30:53]
这让我内心有很强的信念：八年之后，评估仍然会非常重要，而且它们会成为人们构建优秀 AI 软件的核心驱动力。
指向原始笔记的链接
And that gives me a lot of internal conviction that eight years from now, evals will still be very relevant and they’ll be kind of like the core driver for how people build great AI software.
—— Ankur Goyal · [00:29]
当你在评估日益复杂的系统时，从模块化假设出发几乎总是个坏主意。
指向原始笔记的链接
Almost always a bad idea to start with a modularity assumption when you’re trying to evaluate increasingly complex systems.
—— Ankur Goyal · [51:59]
而我见过团队成功，是因为他们非常擅长把一切都扔掉、从头再来，而且最终往往会得到更简单的东西。
指向原始笔记的链接
And I’ve seen teams succeed when they’re very good at just throwing everything away and starting over and often ending up with something simpler.
—— Ankur Goyal · [53:19]
一个模型或者一个非常重的 LLM 产品，然后展示一堆关于性能的柱状图，这说白了，全是营销。
指向原始笔记的链接
A model or a really LLM heavy product and showing a bunch of bar charts about performance, that is like, it’s all marketing.
—— Ankur Goyal · [55:28]
如果你在构建一个 AI 产品，除了评测之外做任何事都没有意义。一切都应该围绕你的评测展开。
指向原始笔记的链接
if you’re building an AI product, there’s no point doing anything other than evals. Everything should revolve around your evals.
—— Ankur Goyal · [03:18]
一旦你接受了评测，你就会意识到 LLM 所做的不过是你在产出的评测的一个函数。
指向原始笔记的链接
once you embrace evals, then you realize that what the LLM does is merely a function of the evals that you’re producing.
—— Ankur Goyal · [05:34]
你会意识到杠杆最高的做法其实是把出问题的地方捕获进一条评测，这样你做的任何改动你都知道是真正有效的，不会破坏你之前努力取得的所有其他成果。
指向原始笔记的链接
You realize that the highest leverage thing you can do is actually try to capture what’s going wrong in an eval so that any change that you make you know is actually going to work and it’s not going to break all the other stuff that you’ve worked towards.
—— Ankur Goyal · [05:49]
当你真正接受它时，你会意识到你在评测上投入的工作，不会在你每次尝试新模型、改一些措辞或稍微改变用例时就蒸发掉。
指向原始笔记的链接
When you really embrace it, you realize that the work that you put into doing evals doesn’t evaporate every time you try a new model or you change some wording or you change your use case a little bit.
—— Ankur Goyal · [06:00]
你唯一能确定的就是，你今天选的任何模型都极不可能成为你明天唯一会用的模型。
指向原始笔记的链接
the only thing that you can be certain of is that whatever model you pick today is super unlikely to be the model that you’re going to use solely tomorrow.
—— Ankur Goyal · [06:37]
我一直不得不克制自己不去构建像 Brainstore 这样的系统，因为我真的很想去做。但这迫使我们对自己需要构建的东西有了越来越高的清晰度。
指向原始笔记的链接
I kept having to hold myself back from building a system like Brainstore because I really wanted to. But it forced us to have an increasingly high level of clarity about what we needed to build.
—— Ankur Goyal · [13:24]
智能体就是一个带工具的 for 循环。
指向原始笔记的链接
Agents are a for loop with tools.
—— Ankur Goyal · [16:13]
因为他们内部正在进行严肃的讨论：要么想办法围绕 AI 重建产品，要么死。
指向原始笔记的链接
because they’re having serious discussions internally about how it’s either figure out how to rebuild our product around AI or die.
—— Ankur Goyal · [18:40]
我认为过去几年里我听到的最有趣的洞见之一是 Jensen 说他不进行一对一谈话。那对我来说是一个巨大的解锁。
指向原始笔记的链接
I think one of the most interesting insights I heard over the past few years was Jensen saying he doesn’t do one-on-ones. And that was a big unlock for me.
—— Ankur Goyal · [22:04]
随着时间推移，我学到的最难的教训是：你不能雇佣市场推广人员去解决你作为一家公司还没有解决的问题。
指向原始笔记的链接
The hardest lesson that I’ve learned over time is that you can’t hire go-to-market people to solve problems that you haven’t solved as a company.
—— Ankur Goyal · [23:04]
② 出现在这些集
2 集
- 《Braintrust CEO Ankur Goyal:做 AI 评估的纪律八年不变，但玩法正在剧变》 — 作为联合主持
- 《评测优先:Braintrust 创始人谈 AI 产品开发的真正工程》 — 作为主持
③ 他谈到的
点进去有真内容 —— 本页主要出口
Braintrust · 评估 · 智能体 · Brainstore · Figma · 提示词 · Corinne Riley · LLM · 评分函数 · Datadog