Researchers at Moonshot AI and UCLA will release a 3B/16B parameter mixture of exper (MOE) model trained with 5.7T tokens using Muon Optimizer.

Training large-scale language models (LLM) has been central to advancement in artificial intelligence, but it is not without challenges. As model sizes and datasets continue to grow, traditional optimization methods (particularly Adamw) will be able to demonstrate limitations. One of the main challenges is managing computational costs and ensuring stability throughout the expansion training run. [...]

The post Researchers at Moonshot AI and UCLA will release a 3B/16B parameter mixture of exper (MOE) model trained with 5.7T tokens using Muon Optimizer. first appeared on Versa AI hub.

from Blog - Versa AI hub https://versaaihub.com/researchers-at-moonshot-ai-and-ucla-will-release-a-3b-16b-parameter-mixture-of-exper-moe-model-trained-with-5-7t-tokens-using-muon-optimizer/
via IFTTT