- Total Parameters
- 1T
- Activated Parameters
- 32B
- Layers
- 61
- Dense Layers
- 1
- Attention Hidden Dimension
- 7168
- MoE Hidden Dimension (per Expert)
- 2048
- Attention Heads
- 64
- Experts
- 384
- Selected Experts per Token
- 8
- Shared Experts
- 1
- Vocabulary Size
- 160K
- Attention Mechanism
- MLA
- Activation Function
- SwiGLU
- Vision Encoder
- MoonViT
- Vision Encoder Parameters
- 400M