① 他说过的话

14 条

我一直觉得那就是思考如何从无标注数据中学习的正确方式，也就是说，构建一个生成式模型，才是真正能确保你理解数据中结构的正确方法。
I always felt like that was going to be the, that was the right way to think about kind of like learning from unlabeled data, that like building a generative model is really the right way to make sure you understand the structure in the data.
—— Stefano Ermon · [01:48]

指向原始笔记的链接

因为它是 diffusion，因为你同时输出很多 token，我们能够比自回归模型快大约 10 倍地生成文本。
Because it’s diffusion, because you’re outputting many tokens at the same time, we were able to generate text like 10x faster compared to the autoregressive model.
—— Stefano Ermon · [05:36]

指向原始笔记的链接

在 Inception，我们押注 diffusion 模型，因为我们相信最终重要的是推理时扩展。
At inception, we’re betting on diffusion models because we believe that what matters eventually will be inference time scaling.
—— Stefano Ermon · [08:01]

指向原始笔记的链接

这种工作负载是极其受内存限制的。你把大部分时间花在内存层次结构之间搬运权重上，而你做的算术运算非常少。
That kind of workload is extremely memory bound. You’re spending most of your time moving around weights across the memory hierarchy and you’re doing very little arithmetic.
—— Stefano Ermon · [09:12]

指向原始笔记的链接

更好的教训是，更具并行性的方案就是最终会胜出的那个。
And the better lesson is that the more parallel solution is the one that is eventually going to win.
—— Stefano Ermon · [11:07]

指向原始笔记的链接

看 benchmark 的话，我们的 Mercury 模型与 Haiku 模型、Flash 模型、OpenAI 的 Mini Nano 模型相当，同时速度快得多。
So our Mercury models are on par with the Haiku models, Flash models, Mini Nano models from OpenAI, if you look at benchmarks, while being significantly faster.
—— Stefano Ermon · [12:33]

指向原始笔记的链接

你没法在 VLLM 或 SGLang 上运行这些基于扩散的 LLM。我们必须构建自己的服务引擎，而且我们能处理真实生产工作负载的很多复杂性。
You cannot run these diffusion-based LLMs on VLLM or SGLang. We had to build our own serving engine, and we can handle a lot of the complexity of real production workloads.
—— Stefano Ermon · [13:01]

指向原始笔记的链接

我怀疑我们还没有发现构建这些系统的最佳方式。肯定存在替代方案，肯定存在构建这些模型的其他方式。
I doubt we’ve discovered the best way of building these systems. There’s got to be alternatives. There’s got to be other ways of building these models.
—— Stefano Ermon · [15:25]

指向原始笔记的链接

而且我觉得一旦你习惯了快的模型，就很难回去了。有点像宽带，对吧？
And I think once you get used to a fast model, it’s hard to go back. It’s kind of like broadband, right?
—— Stefano Ermon · [17:20]

指向原始笔记的链接

令人兴奋的地方在于，在某种程度上，你从软件中获得的收益，与你从硬件中获得的收益是相乘的。
That’s the exciting piece, is that to some extent, the gains that you get from the software, they are multiplicative with the gains you get from the hardware.
—— Stefano Ermon · [19:18]

指向原始笔记的链接

从根本上说，每当你训练一个生成式模型时，你所做的，无论是自回归模型还是某种程度上甚至是扩散模型，都是你试图通过本质上构建一个压缩方案来识别数据中的结构。
And fundamentally, if you think about whenever you train a generative model, what you’re doing is, whether it’s an autoregressive model or to some extent even a diffusion model, is you are trying to identify structure in the data by essentially building a compression scheme.
—— Stefano Ermon · [22:34]

指向原始笔记的链接

所以至少在学术文献中有大量证据表明，扩散模型更容易控制，而且有不同的引导方式——那些是自回归模型根本做不到的。
And so there is a lot of evidence in the academic literature, at least, that diffusion models are easier to control and there are different ways of steering them. That are just not possible with autoregressive models.
—— Stefano Ermon · [26:53]

指向原始笔记的链接

所以这也是我们决定不完全开源的原因之一，真的是为了把这些 IP 更多地留在我们自己手里，不把它开放出去。
So it’s one of the reasons we decided not to open source everything, was really to keep the IP a little bit closer to us and not opening it.
—— Stefano Ermon · [31:24]

指向原始笔记的链接

学术界的一个好处是，它允许你下这些反共识的赌注。
One of the nice things about academia is that it allows you to take these contrarian bets.
—— Stefano Ermon · [37:12]

指向原始笔记的链接

② 出现在这些集

1 集

• 《扩散模型能不能取代GPT？扩散模型之父下场创业》 — 作为嘉宾

③ 他谈到的

点进去有真内容 —— 本页主要出口

inception 阶段 · Mercury · 扩散 · 自回归模型 · 推理 · Transformer · 服务引擎 · Cerebras

④ 也在聊「创业与行业」的人