跨国深谈

基准测试 (benchmarks)

12分钟阅读

跨国深谈← 返回←基准测试 (benchmarks)
基准

基准测试 (benchmarks)

概念
本站收录 2 集 · 14 条金句 · 关联 10 个

集里怎么说它

① 提到它的金句

14 条

我认为关于 eval 最大的误解是以为它们是基准测试。
I think the biggest misconception about evals is that they’re benchmarks.
—— Ankur Goyal · [16:16]

指向原始笔记的链接

你可以有一个在我们基准测试上表现非常好的模型，但它们在某种程度上过拟合了，然后你把它放进一个更通用的助手类产品里，当你超出分布范围时它就有点崩溃了。
you can have a model that does very well on our benchmarks but they’ve overfit to some degree and then you put it in a more generic assistant like product and it kind of falls apart for when you go out of distribution
—— Gabe Pereyra · [22:29]

指向原始笔记的链接

仅仅是格式上的变化,就曾被观察到能让基准成绩波动 76 个百分点;而重排少样本示例的顺序,能把模型在某些基准上从接近随机(约 50%)翻转到接近最先进水平。
Formatting changes alone have been seen to swing benchmark by 76 percentage points, and reordering few-shot examples flips a model from near-random, so near 50%, to near-state-of-the-art on some benchmarks.
—— Jared Joselowitz · [11:37]

指向原始笔记的链接

以某种不会在基准测试中显现的方式搞砸后训练，真的太容易了。
So it’s like really easy to screw up post-training in some way that doesn’t show up in benchmarks.
—— 嘉宾 · [27:39]

指向原始笔记的链接

看 benchmark 的话，我们的 Mercury 模型与 Haiku 模型、Flash 模型、OpenAI 的 Mini Nano 模型相当，同时速度快得多。
So our Mercury models are on par with the Haiku models, Flash models, Mini Nano models from OpenAI, if you look at benchmarks, while being significantly faster.
—— Stefano Ermon · [12:33]

指向原始笔记的链接

因为那样如果你的自由现金流达不到典型的行业基准，谁在乎呢，反正你拥有神。
Cause then if your free cashflow does not meet typical industry like benchmarks, who cares if you own God?
—— Ranjan Roy · [19:15]

指向原始笔记的链接

但有一个大问题摆在这里，基本上就是我每天睡前都在想的事情：基准测试、评估、智能体、以及你怎么配置它们，全都是协变的。
But one big problem presents itself, which is basically the thing that I go to bed thinking about every day, which is benchmarks, evaluations, the agents, and how you configure them are all covariant.
—— Zubin Aysola · [01:32]

指向原始笔记的链接

我需要承认的是，在某个时期，我真的认为追基准测试是有价值的，但坦白说，我不相信这个，这某种程度上是公司的心理战。
What I need to admit, at some point of time, I really was thinking that chasing benchmarks is valuable, but I don’t believe this is just sort of corporate psyops, frankly.
—— Alex Mashrabov · [23:23]

指向原始笔记的链接

他们开始以某种方式利用测试数据，用 LLM 作为裁判来训练模型，用各种花招来刷基准，拿季度奖金之类的。
They start to kind of use leverage test data to use LLM as a judge for training of the models, use all the various tricks to basically gain benchmarks, get like quarterly bonuses and so on.
—— Alex Mashrabov · [23:52]

指向原始笔记的链接

这里有一些基准测试，Jev 在平均走法上快了 10 倍，便宜了 4 倍。
it has some benchmarks over here that Jev was 10 times faster in the average move and it was four times cheaper.
—— John Lindquist · [21:41]

指向原始笔记的链接

我还没见过任何基准测试表明图数据库——原生图数据库——能在那些关系型数据库系统拥有正确的 SQL 或 API 构造、从而让你完全在服务器端做图遍历、不用来回往返的情况下，胜过关系型数据库系统。
I have not seen any benchmarks that show graph databases, the native graph databases, outperform the relational databases systems when those related systems have the right constructs in SQL or the API so that you can do graph traversal all on the server side and not go back and forth.
—— Andy Pavlo · [22:23]

指向原始笔记的链接

至于被高估的那一项，我大概说了一年多了，但我仍然认为它被高估了，那就是模型基准测试。
I think the over-hyped one, I think I’ve been saying this maybe for a year now, but I still think it’s over-hyped, model benchmarks.
—— Josh Woodward · [28:18]

指向原始笔记的链接

在我们看来，很明显 Muse 是被基准测试刷出来的，也就是说他们训练它让它在公开基准测试上看起来非常好看，但这些提升不一定能转化到你的日常工作中。
To us, it seems pretty clear that Muse is benchmarked, which means they trained it to look really good on public benchmarks, but those gains don’t necessarily translate into your everyday work.
—— 嘉宾 · [00:36]

指向原始笔记的链接

Gemini 4 是那种我一看基准测试就说“这个也是刷基准刷出来的”的模型，但它确实好。
Gemini 4 is one of those models where I saw the benchmarks and I was like, this is also really benchmark maxed, but it’s good.
—— 嘉宾 · [01:31]

指向原始笔记的链接

② 出现在这些集

2 集

③ 关联

点进去有真内容 —— 本页主要出口

智能体 · Max Junestrand · Lenny · Legora · Josh Woodward · 微调 · Google · 评估 · Google Labs · GPT 3.5

反向链接