跨国深谈← 返回←Flash Attention 之前的注意力 (attention)

集里怎么说它

• 《把 100 万 token 用出 5 万的体验：SubQuadratic 的稀疏注意力》(00:00起):本集的出发点:注意力让每个 token 与其他所有 token 互相关照,1000 个 token 有 100 万个两两关系,成本随上下文长度平方增长——上下文翻倍,算力大约贵四倍。

① 提到它的金句

6 条

现在，我主要受限于注意力。与 tokens 或计算不同，我不能简单地增加更多的注意力。
Now, I’m primarily constrained by attention. And unlike tokens or compute, I can’t simply add more of it.
—— 嘉宾 · [21:19]

指向原始笔记的链接

如果 Transformer 是有利可图的,如果你可以花费更多的努力和资源来扩大 Transformer 以在下一个季度获胜,那么很难把大量的注意力和精力投入到可能在一两年内更好或重新定义该领域的事情上。
If Transformer is profitable and if you can spend more efforts and more resources scaling Transformer to win in the next quarter, it’s very hard to put at least a lot of attention and a lot of energy to work on something that will maybe better or maybe will redefine the field in a year or two.
—— Jerry Tworek · [13:28]

指向原始笔记的链接

因为，你知道，初创公司不直接瞄准现有企业，而现有企业只是不注意。
And because, and the, and because, you know, the startups don’t aim straight at the incumbents, and the incumbents just don’t pay attention.
—— Steven Sinofsky · [49:54]

指向原始笔记的链接

如果你是一个美国人,你希望我们在汽车生意上打败中国人,你们都该密切关注——但不一定是 Tesla。
If you’re an American and you want us to beat the Chinese in the car business, you’re all going to want to pay attention, not necessarily to Tesla.
—— Jim Farley · [12:11]

指向原始笔记的链接

有一场争夺注意力的战争。如果你造了个没人用的东西，那它就跟没被造出来一样。
There’s a war for attention. If you build something that nobody uses, it might as well have not been built.
—— Jordan Tigani · [01:00]

指向原始笔记的链接

你必须对未来有一个猜测。然后你必须有毅力和力量，在猜测错了时去注意并纠正。
You’ve got to have a guess. And then you have to have the fortitude and the strength to pay attention when your guess is wrong and correct.
—— Andrew Anagnost · [14:22]

指向原始笔记的链接

② 出现在这些集

1 集

• 《把 100 万 token 用出 5 万的体验：SubQuadratic 的稀疏注意力》 — 作为概念

③ 关联

点进去有真内容 —— 本页主要出口

Alexander Whedon · SubQuadratic · 稀疏注意力 · 上下文工程 · 智能体 · RAG · 预训练 · DeepSeek Sparse Attention · KVCache · Transformer