Charles Frye
① 他说过的话
8 条
我认为，这里是工程工作如此关键的落点、作为工程师在这里工作如此酷的根本原因是：归根结底，训练是成本中心，推理是收入中心。
指向原始笔记的链接
I think the fundamental reason why this is such a critical place for engineering work to get done and why it’s such a cool place to work as an engineer is that in the end, training is a cost center and inference is a revenue center.
—— Charles Frye · [04:52]
人们普遍无法把直接向别人出售模型权重的生意做成，所以你没法真正把一个训练部门变成一个能自我支撑的系统。
指向原始笔记的链接
People have not generally been able to make businesses work where they sell model weights to people directly, and so you can’t really turn a training setup into a proper self-supporting system.
—— Charles Frye · [05:11]
一旦你在乎性能，它就是有状态的，而这几乎是所有情况。
指向原始笔记的链接
It’s stateful once you do care about performance, which is pretty much every case.
—— Charles Frye · [20:03]
你试图在 GPU 资源上管理锁或协调所带来的复杂度增加，远大于收益，因为即使你能把它降到微秒级，工作也要等到 GPU 上前一个工作完成后才会开始。
指向原始笔记的链接
The complexity increase that you get from trying to manage locks or coordination on the GPU resources, that complexity increase is much greater than the gain, because even if you were able to get it down to a microsecond, the work wouldn’t start happening until the previous work on the GPU had finished.
—— Charles Frye · [24:08]
我把那本质上视为引擎的一种失败状态，如果我到了出现 KV cache 抖动或者东西被刷到磁盘的地步，那说明我把引擎部署得不对。
指向原始笔记的链接
I think of that as essentially like a failure state of the engine that we, like I’ve deployed my engine improperly if I get to the case where I have KV cache thrashing or like things being flushed to disk.
—— Charles Frye · [45:46]
这是每秒 PB 级的内存带宽。
指向原始笔记的链接
That’s petabyte per second scale memory bandwidth.
—— Charles Frye · [51:23]
你能对推理引擎做的许多其他优化，都像是寸土必争的百分比游戏，每次百分之五、百分之十，而且你必须为每一个百分点做非常困难的性能工程。
指向原始笔记的链接
So many other optimizations that you can do on your inference engine are like, it’s like a game of inches of a percent, five percent, 10 percent each time, and you have to do the very difficult performance engineering for each of those percentage points.
—— Charles Frye · [52:31]
所以这已经成为让你能够在 NVIDIA 硬件上以每秒约 1,000 个 token 运行相当大的模型的关键技术之一，而不必去用像 LPU 或 Cerebras 这样的新型加速器。
指向原始笔记的链接
So this has emerged as one of the key techniques to allow you to operate, say, 1,000 tokens per second on NVIDIA hardware for actually fairly large models without having to go to a novel accelerator like an LPU or Cerebris.
—— Charles Frye · [53:06]
② 出现在这些集
1 集
- 《大模型背后：推理引擎到底在干什么》 — 作为嘉宾
③ 他谈到的
点进去有真内容 —— 本页主要出口
Modal · SGLang · vLLM · PyTorch · NVIDIA · Cognition · 推理 · 推理引擎 · 调度器 · 预填充