Shreya Shankar
① 他说过的话
4 条
目标不是完美地做评估，而是可操作地改进你的产品。
指向原始笔记的链接
The goal is not to do evals perfectly, it’s to actionably improve your product.
—— Shreya Shankar · [00:18]
当我们试图请求一个 LLM 做这个错误分析时，我们通常发现它只是说追踪看起来很好，因为它没有理解某样东西是否可能是坏的产品味道所需的上下文。
指向原始笔记的链接
What we usually find when we try to ask an LLM to do this error analysis is it just says the trace looks good because it doesn’t have the context needed to understand whether something might be bad product smell or not.
—— Shreya Shankar · [24:09]
人们对好坏的看法随着他们审查更多输出而改变，他们只有在看到 10 个输出后才会想到失败模式，而这些输出是他们最初从未梦想到的。
指向原始笔记的链接
People’s opinions of good and bad change as they review more outputs, they think of failure modes only after seeing 10 outputs they would never have dreamed of in the first place.
—— Shreya Shankar · [64:23]
我认为正在这样做的产品，他们对自己应用程序的性能如何有着非常敏锐的感觉，人们不谈论这个，因为这是他们的护城河。
指向原始笔记的链接
I think the products that are doing this, they have a very sharp sense of how well their application is performing, and people don’t talk about it, because this is their moat.
—— Shreya Shankar · [68:16]
② 出现在这些集
1 集
- 《做 evals 不是写单元测试，是从看数据开始的错误分析》 — 作为联合主持
③ 他谈到的
点进去有真内容 —— 本页主要出口
Lenny · Hamel Husain · 评估 · 错误分析 · 开放式编码 · 轴向编码 · LLM 当裁判 · 智能体 · trace · Anthropic