Looping Beyond Twice: Một công thức có thể mở rộng cho các chuyên gia

Đoạn trích bài viết

Looped Transformers introduce recurrent depth as a new scaling axis for LLMs: by repeatedly applying shared Transformer blocks, they increase effective depth without increasing parameter count. However, the benefits of looping remain unclear for large MoE LLMs under FLOPs-matched comparisons.