Kimi K3 Architecture Explained: How the World’s Largest Open Source AI Model Works | Kimi K3 架构解析：全球最大开源AI模型是如何运作的

Introduction 引言

EN: Moonshot AI, one of China’s leading AI labs, has just released Kimi K3 — the world’s first open-weight model to enter the 3-trillion-parameter class. The full model carries 2.8 trillion parameters, yet only 104 billion are active for any given token. Scale alone isn’t the story here: Moonshot’s technical report claims roughly 2.5x better scaling efficiency than its predecessor, K2. This post walks through the video’s breakdown of how Moonshot achieved that efficiency jump — not by simply making K2 bigger, but by re-engineering three core pieces of the architecture.

中文： 中国领先的AI实验室之一 Moonshot AI（月之暗面）刚刚发布了 Kimi K3——这是全球首个进入3万亿参数级别的开放权重模型。该模型总参数量达2.8万亿，但每个token实际激活的参数仅为1040亿。这里的重点并非单纯的”规模”：Moonshot的技术报告称，相比前代K2，其扩展效率提升了约2.5倍。本文将梳理视频中对这一效率飞跃背后原理的拆解——Moonshot并非简单地把K2做大，而是对架构中的三大核心模块进行了重新设计。

Features and Concepts: What’s New in This Version

新版本特性与核心概念

1. From Generation to Vision-in-the-Loop Debugging

从”生成”到”视觉闭环调试”

EN: Give Kimi K3 a prompt, and it can generate an entire 3D world — terrain, forests, villages, weather systems, all built with Three.js and WebGPU. What separates K3 from K2 isn’t code generation itself, but vision-in-the-loop debugging: the model boots a browser, inspects live screenshots of its own output, spots broken lighting or physics, and rewrites its own shader code until the scene renders correctly.

On benchmarks, K3 scores 88.3 on coding tests (Terminal-Bench, Frontier SWE, ProgramBench), putting it alongside GPT-5.6, Sonnet, and Fable 5. It shows similar strength on agent benchmarks like BrowseComp, Automation Bench, and Spreadsheet Bench.

中文： 只需一句提示词，Kimi K3 就能生成一整个三维世界——地形、森林、村庄、天气系统，全部基于 Three.js 和 WebGPU 构建。K3 与 K2 的真正区别不在于代码生成本身，而在于”视觉闭环调试”（vision-in-the-loop debugging）：模型会启动浏览器，实时查看自己输出的画面截图，发现光照或物理效果的错误，然后自行重写着色器代码，直到场景正确渲染为止。

在基准测试方面，K3 在编程类测试（Terminal-Bench、Frontier SWE、ProgramBench）中得分88.3，与 GPT-5.6、Sonnet 和 Fable 5 处于同一梯队；在 BrowseComp、Automation Bench、Spreadsheet Bench 等智能体基准上也表现相当。

2. Configuration: Stable Latent Mixture-of-Experts (MoE)

架构配置：稳定潜空间混合专家系统 (Stable Latent MoE)

EN: A 2.8-trillion-parameter model can’t run every token through every parameter — so K3 uses a sparse MoE. For each token, a router activates just 16 of 896 routed experts, plus 2 always-on shared experts. That’s 1 in 56 routed experts per token, dramatically cutting compute.

Sparsity solves compute, but at this scale it creates a communication problem: experts are distributed across GPU clusters, so a token’s 7K-dimensional hidden representation has to travel across the network to reach each selected expert. Moonshot’s fix — the core “configuration change” of this version — is a learned down-projection that compresses the hidden state into a compact latent vector before it crosses the network. Experts compute entirely in latent space, and only the final result is projected back up. This alone cuts inter-GPU communication by roughly 50%.

Two supporting mechanisms keep this stable:

• RMSNorm on the latent pathway — prevents activation drift across K3’s 93 stacked layers.
• Sigmoid-Tanh-Unit GLU (replacing SwiGLU) — bounds activation growth with a soft cap (beta = 4 for the gate branch, 25 for the value branch), capping maximum magnitude at 100 and stabilizing FP8 training.
• Quantile balancing — computes a per-expert bias from the quantiles of that expert’s routing-score distribution, so load stays balanced across all 896 experts without altering the learned routing function itself.

中文： 一个2.8万亿参数的模型不可能让每个token都经过全部参数计算，因此K3采用了稀疏混合专家（MoE）架构。对每个token，路由器仅从896个可路由专家中激活16个，再加上2个始终激活的共享专家——即每个token仅激活1/56的可路由专家，大幅降低了计算量。

稀疏化解决了计算问题，但在如此规模下又带来了”通信”难题：专家分布在庞大的GPU集群中，每个token 7K维的隐藏表示都需要通过网络传输到被选中的专家所在的GPU。Moonshot 的解决方案——也是本版本最核心的”架构配置”改动——是引入一个可学习的降维投影（down-projection），先将隐藏状态压缩为紧凑的潜向量，再进行网络传输。专家的全部计算都在潜空间中完成，仅在最后才将结果重新投影回原始维度。仅此一项改动，就将GPU间通信量降低了约50%。

两项配套机制保证了这一过程的稳定性：

• 潜空间路径上的 RMSNorm —— 防止激活值在93层堆叠网络中逐层漂移；
• Sigmoid-Tanh-Unit GLU（替代 SwiGLU） —— 通过软上限（gate分支beta=4，value分支beta=25）约束激活值增长，将最大激活幅度限制在100以内，提升FP8训练的数值稳定性；
• 分位数负载均衡（Quantile Balancing） —— 根据每个专家路由得分分布的分位数计算偏置，在不改变已学习的路由函数本身的前提下，让896个专家的负载保持均衡。

3. Kimi Delta Attention (KDA): Configuring for Million-Token Context

Kimi Delta Attention (KDA)：面向百万级上下文的配置方案

EN: Standard attention gives perfect recall but its KV cache grows linearly with sequence length — unworkable at 1 million tokens. K3’s answer is Kimi Delta Attention, evolved through a clear lineage:

• Linear attention replaces the growing KV cache with a fixed-size recurrent state, keeping memory constant.
• DeltaNet adds an “eraser” — it recalls a value with the incoming key, compares it to the new value, and writes only the difference, avoiding blind overwrites.
• Gated DeltaNet adds a learnable forget gate (alpha) alongside the write gate (beta), letting the model decide how much of the past to keep vs. overwrite.
• KDA decouples decay and erasure into an explicit sequence — decay stale context first, then erase conflicting memory, then write. This avoids wasting eraser capacity on content already being decayed, and — because state updates are sequential — eliminates the need for positional encodings like RoPE.

Example / config in practice: K3 uses a hybrid 3:1 layer ratio — three KDA layers followed by one gated Multi-Head Latent Attention (MLA) layer. The ablation study is the “why” behind this exact number: full attention every layer wastes memory without improving perplexity; a 7:1 KDA-heavy ratio degrades long-range reasoning; 1:1 doubles memory and latency. 3:1 hits the lowest perplexity for the least cost. The result — dubbed “Kimi Linear” — matches standard MLA accuracy (84.3% on the 128K RULER benchmark) while delivering a 4x decoding speedup at 1M tokens, rising to 6x overall.

中文： 标准注意力机制可以实现完美的信息召回，但其KV缓存会随序列长度线性增长——在百万token级别下完全不可行。K3给出的方案是 Kimi Delta Attention（KDA），其演化脉络十分清晰：

• 线性注意力（Linear Attention） 用固定大小的循环状态矩阵取代不断增长的KV缓存，使内存占用保持恒定；
• DeltaNet 引入了”擦除器”机制——先用新输入的key召回当前状态中的一个值，再与新的value对比，仅写入两者之间的”差值”，避免盲目覆盖旧记忆；
• Gated DeltaNet（门控DeltaNet） 在写入门（beta）基础上加入了可学习的遗忘门（alpha），让模型自主决定保留多少历史信息、覆盖多少新信息；
• KDA 将”衰减”与”擦除”解耦为明确的先后顺序——先衰减过时的上下文，再擦除冲突记忆，最后写入新信息。这样既避免了对即将被衰减内容浪费擦除容量，又因为状态更新是严格顺序化的，从而无需再使用RoPE等位置编码。

实际配置示例： K3 采用了3:1的混合层比例——每三层KDA之后接一层门控多头潜注意力（MLA）。消融实验解释了为何选择这一比例：每层都用全注意力会大幅增加内存却无法提升困惑度；7:1的KDA主导比例会损害长程推理能力；1:1比例虽能恢复精度，却使内存占用和解码延迟翻倍。而3:1比例在成本最低的情况下实现了最低的困惑度。这一设计被称为”Kimi Linear”，在128K RULER基准测试中精度可达84.3%，与标准MLA持平，同时在百万token场景下解码速度提升4倍，综合提速可达6倍。

4. Attention Residuals: Configuring Depth Across 93 Layers

注意力残差：93层网络的深度配置方案

EN: In a standard transformer, each layer only communicates with the one before it via the residual connection — by the final layer, much of the earliest information is diluted. K3 introduces attention residuals: each layer generates a query, computes soft attention weights over normalized representations of earlier layers, and aggregates them into a weighted sum. RMSNorm on the candidate keys prevents high-magnitude layers from dominating.

Example / config in practice: Storing all 93 individual layer states would be too costly, so K3 groups the 93 layers into 8 compressed blocks (~11–12 layers each) and runs attention residuals across block-level summaries instead of individual layers — an 11.6x reduction in memory and communication overhead while preserving deep feature retrieval.

中文： 在标准Transformer中，每一层仅通过残差连接与前一层通信——到达最后一层时，早期层的信息大多已被稀释。K3 引入了**注意力残差（Attention Residuals）**机制：每一层都会生成一个查询向量，对更早层的归一化表示计算软注意力权重，并将其加权汇总。对候选key进行RMSNorm可防止激活幅度较大的层主导注意力分数。

实际配置示例： 若存储全部93层的独立状态，开销将过于庞大，因此K3将93层划分为8个压缩区块（每块约11-12层），注意力残差机制在区块级摘要而非单层输出上运行——在保留深层特征检索能力的同时，将内存与通信开销降低了11.6倍。

Kimi K3: Installation, Setup, Configuration & Simple Showcase

1. Installation | 安装

Option A — Hosted API (recommended for most users) | 方案A——托管API（推荐大多数用户使用）

EN: The fastest path to using K3 is through Moonshot’s own API, the same way you’d call OpenAI or Anthropic’s API. No GPU required on your end.

bash

pip install openai  # Moonshot's API is OpenAI SDK-compatible

python

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_MOONSHOT_API_KEY",
    base_url="<https://api.moonshot.ai/v1>"  # verify current endpoint on Moonshot's docs
)

中文： 使用 K3 最快捷的方式是通过 Moonshot 官方API，调用方式与调用 OpenAI 或 Anthropic 的API类似，无需在本地配置GPU。

bash

pip install openai  # Moonshot 的API 兼容 OpenAI SDK

python

from openai import OpenAI

client = OpenAI(
    api_key="你的_MOONSHOT_API_KEY",
    base_url="<https://api.moonshot.ai/v1>"  # 请以 Moonshot 官方文档中的最新地址为准
)

Option B — Self-hosted (large-scale cluster only) | 方案B——自行部署（仅适用于大规模集群）

EN: Because K3 is a 2.8T-parameter MoE model, self-hosting realistically requires a multi-node GPU cluster with high-bandwidth interconnect (the video’s whole discussion of latent-space compression exists precisely because inter-GPU communication is the bottleneck at this scale). If Moonshot follows the K2 pattern, weights would be distributed via HuggingFace, served through an inference engine with MoE and long-context support:

bash

# Illustrative only — confirm exact package/repo names against Moonshot's release
pip install vllm  # or sglang
huggingface-cli download moonshotai/Kimi-K3 --local-dir ./kimi-k3

bash

vllm serve ./kimi-k3 \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size N \
  --max-model-len 1000000 \
  --trust-remote-code

中文： 由于K3是一个2.8万亿参数的MoE模型，自行部署在现实中需要具备高带宽互联的多节点GPU集群（视频中反复强调的”潜空间压缩”设计，正是因为在这个规模下GPU间通信才是真正的瓶颈）。若 Moonshot 延续K2的发布模式，权重文件将通过 HuggingFace 分发，并通过支持MoE与长上下文的推理引擎进行部署：

bash

# 仅作示意——请以 Moonshot 官方发布信息核实确切的包名/仓库名
pip install vllm  # 或 sglang
huggingface-cli download moonshotai/Kimi-K3 --local-dir ./kimi-k3

bash

vllm serve ./kimi-k3 \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size N \
  --max-model-len 1000000 \
  --trust-remote-code

2. Setup: Environment & Prerequisites | 环境准备与前置条件

EN:

• Python 3.10+
• CUDA-compatible GPUs (self-hosting only) — enough VRAM aggregated across nodes to hold 2.8T parameters even at low precision (K3’s Sigmoid-Tanh-Unit GLU design is specifically built to make FP8 training/inference stable, so FP8 serving is a realistic target)
• Fast inter-GPU networking (NVLink/InfiniBand) — the video’s entire “latent MoE” design exists because this is the actual bottleneck
• For API usage: just an API key and standard internet access

中文：

• Python 3.10 及以上版本
• 支持CUDA的GPU（仅自部署需要）——需要跨节点聚合足够的显存以承载2.8万亿参数（即便在低精度下）。K3 的 Sigmoid-Tanh-Unit GLU 设计正是为了让FP8训练/推理更稳定，因此FP8部署是较为现实的目标
• 高速GPU间网络互联（NVLink/InfiniBand）——视频中详细讲解的”潜空间MoE”设计，正是为了应对这一真正的瓶颈而生
• 若使用API方式：仅需API密钥及正常的网络连接即可

3. Configuration: Key Parameters to Understand | 关键配置参数解读

EN: Even when calling K3 via API (i.e., you don’t touch these directly), understanding them helps you reason about cost, latency, and output quality:

中文： 即便你是通过API调用K3（也就是说不会直接接触这些底层参数），理解它们也有助于你判断成本、延迟与输出质量：

4. Simple Showcase: A Minimal Example | 简单实战示例

EN: Here’s a minimal illustrative call, showing K3 used as a coding/agent assistant — the use case the video benchmarks most heavily (Terminal-Bench, SWE-Bench-style tasks):

python

response = client.chat.completions.create(
    model="kimi-k3",  # confirm exact model string on Moonshot's docs
    messages=[
        {"role": "system", "content": "You are a careful coding assistant."},
        {"role": "user", "content": "Write a Python function that checks if a string is a palindrome, and explain your reasoning."}
    ],
    temperature=0.3
)

print(response.choices[0].message.content)

中文： 下面是一个最简示例，展示如何将K3用作编程/智能体助手——这也是视频中benchmark测试最集中的使用场景（Terminal-Bench、SWE-Bench类任务）：

python

response = client.chat.completions.create(
    model="kimi-k3",  # 请以 Moonshot 官方文档核实准确的模型标识字符串
    messages=[
        {"role": "system", "content": "你是一位严谨的编程助手。"},
        {"role": "user", "content": "写一个判断字符串是否为回文的Python函数，并说明你的思路。"}
    ],
    temperature=0.3
)

print(response.choices[0].message.content)

EN: For the “generate-a-3D-world” showcase from the video, the workflow shown is agentic rather than a single API call: K3 writes Three.js/WebGPU code, launches a headless browser, screenshots the render, evaluates it against the prompt, and iterates — closer to a coding-agent harness (similar in spirit to how you already use Claude Code) than a single completion.

中文： 至于视频中展示的”生成三维世界”案例，其工作流程本质上是智能体式的，而非单次API调用：K3 编写 Three.js/WebGPU 代码，启动无头浏览器，对渲染结果截图，再根据提示词评估效果并迭代——这更接近于一种编程智能体的工作方式（在思路上与你已经在使用的 Claude Code 有相似之处），而非单次的文本补全。

Video about Kimi K3 Architecture | 关于 Kimi K3 的视频

相关内容：后训练与部署

EN: Pre-training alone doesn’t make a capable agent. K3’s post-training pipeline uses reinforcement learning across three domains — general tasks, general agents, and coding agents — each trained at three reasoning-effort levels (low/high/maximum), producing 9 specialist teacher models. For long-context RL, asynchronous partial rollouts let completed trajectories trigger training immediately while unfinished ones pause in microVMs, avoiding wasted GPU idle time. Finally, multi-tier on-policy distillation compresses all 9 teachers into a single student model that learns when to reason like each specialist, rather than simply averaging them.

中文： 仅靠预训练无法造就一个真正胜任的智能体。K3的后训练流程采用强化学习，覆盖三大领域——通用任务、通用智能体、编程智能体，每个领域又在三种推理强度（低/高/最大）下分别训练，最终产出9个专家教师模型。针对长上下文强化学习，”异步部分回合”机制让已完成的轨迹立即触发训练，未完成的则暂停于微虚拟机（microVM）中，避免GPU闲置浪费。最后，通过多层级同策略蒸馏（on-policy distillation），将9个教师模型压缩为一个学生模型，使其学会何时以哪位”专家”的方式进行推理，而非简单地对九者取平均。

Conclusion & Key Takeaways | 结论与核心要点

EN:

• Stable Latent MoE — solves width/parameter scaling via latent compression, RMSNorm stabilization, bounded activations, and quantile-based load balancing.
• Kimi Delta Attention — solves sequence-length scaling via a decoupled decay-then-erase state update and a 3:1 hybrid KDA/MLA layer ratio.
• Attention Residuals — solves depth scaling via cross-layer retrieval compressed into 8 block-level summaries.
• Most people should start with Moonshot’s hosted API — self-hosting 2.8T parameters is a cluster-scale undertaking.
• Configuration choices (context length, reasoning effort, agent mode) map directly to the architectural features covered in the main review: KDA for long context, distilled teacher tiers for reasoning depth.
• The flagship “generate a 3D world” showcase is an agentic loop, not a single prompt-response call — plan your integration accordingly if you want to reproduce it.
• Please verify exact package names, model strings, and endpoints against Moonshot’s official K3 documentation before deploying.

中文：

• 稳定潜空间MoE —— 通过潜空间压缩、RMSNorm稳定化、有界激活函数与分位数负载均衡，解决了宽度与参数规模的扩展问题；
• Kimi Delta Attention —— 通过”先衰减后擦除”的解耦状态更新机制，以及3:1的KDA/MLA混合层比例，解决了序列长度的扩展问题；
• 注意力残差 —— 通过将跨层信息检索压缩为8个区块级摘要，解决了网络深度的扩展问题。
• 大多数用户应从 Moonshot 官方托管API入手——自行部署2.8万亿参数模型属于集群级工程。
• 配置选项（上下文长度、推理强度、智能体模式）与主视频中介绍的架构特性直接对应：长上下文依赖KDA，推理深度依赖蒸馏后的多档教师模型。
• 视频中”生成三维世界”的标志性演示属于智能体循环，而非单次的提示词-响应调用——若想复现该效果，需据此规划集成方式。
• 正式部署前，请务必以 Moonshot 官方K3文档核实准确的包名、模型标识字符串与接口地址。

References 参考资料

• Moonshot AI — Kimi K3 technical report