More Open-Source Goodness: Snowflake’s “Arctic” MoE Model

5/19/24

Editorial team at Bits with Brains

Snowflake Arctic is a newly launched large language model (LLM) developed by Snowflake, a company best known for its data cloud services.

Snowflake Arctic is a newly launched large language model (LLM) developed by Snowflake, a company best known for its data cloud services. The model is designed to be highly efficient and open, specifically targeting enterprise applications. Snowflake Arctic aims to provide top-tier intelligence for tasks such as SQL generation, coding, and instruction following. The model is part of Snowflake's broader strategy to integrate AI capabilities into its data cloud platform, enabling businesses to leverage AI for more effective data analysis and decision-making. The launch of Arctic marks a significant step for Snowflake as it positions itself against other major players like OpenAI and Databricks.

Snowflake Arctic features a unique Dense-Mixture-of-Experts (DMoE) hybrid transformer architecture. This architecture combines a 10 billion parameter dense transformer model with a residual 128×3.66 billion MoE Multi-Layer Perceptron (MLP). That is, 128 smaller “experts” for a model total of 480 billion parameters, with a top-2 gating mechanism that activates 17 billion active parameters during inference. This design allows Arctic to achieve high performance while maintaining cost-efficiency. The architecture is optimized to handle the all-to-all communication overhead typically associated with MoE models, making it more resource-efficient. This innovative approach enables Arctic to deliver superior performance in enterprise tasks without the high computational costs usually required for such large models.

As we mentioned, Snowflake Arctic is specifically tailored for enterprise use cases, excelling in tasks such as SQL generation, coding, and instruction following. The model has demonstrated high accuracy in benchmarks like the Spider benchmark for SQL generation, HumanEval+ and MBPP+ for code generation, and IFEval for instruction following. These benchmarks highlight Arctic's capability to understand and generate complex SQL queries, produce accurate code, and follow detailed instructions, making it a valuable tool for businesses. The model's performance is further enhanced by its ability to operate efficiently, allowing enterprises to deploy high-quality AI solutions without incurring prohibitive costs. This makes Arctic an attractive option for companies looking to integrate AI into their operations to improve efficiency and decision-making.

One of the standout features of Snowflake Arctic is its commitment to openness. The model is released under an Apache 2.0 license, providing ungated access to its weights and code. This open-source approach allows developers and researchers to use, fine-tune, and share improvements to the model freely. Snowflake has also made available its data recipes and research insights, further promoting transparency and collaboration within the AI community. This move aligns with a broader trend in the AI industry towards open-source models, which are seen as a way to accelerate innovation and democratize access to advanced AI technologies. By making Arctic open source, Snowflake aims to foster a community of developers who can contribute to and benefit from the model's capabilities.

Snowflake Arctic is often compared to other leading LLMs such as Databricks' DBRX, Meta's Llama 3, and Mistral's Mixtral models. In terms of architecture, Arctic's Dense-MoE hybrid transformer is unique, offering a balance between performance and efficiency. Benchmark comparisons show that Arctic excels in enterprise-specific tasks, often outperforming proprietary stet-of-the-art models that require significantly higher compute budgets. Additionally, Arctic's cost-effective training process, which took less than three months and cost under $2 million, sets a new standard for efficiency in the development of large language models. This makes Arctic not only a powerful but also a practical choice for enterprises looking to implement AI solutions.

Sources:

[1] BIG win for Open Source AI | Snowflake Arctic 128 Experts MoE, "Cookbook" create world-class models" https://youtu.be/5Km-6zZP8IQ?si=PwQiuWXFbukVhhmM

[2] https://developer.nvidia.com/blog/new-llm-snowflake-arctic-model-for-sql-and-code-generation/

[3] https://www.snowflake.com/blog/arctic-tilt-compact-llm-advanced-document-ai/

[4] https://techcommunity.microsoft.com/t5/ai-machine-learning-blog/introducing-snowflake-arctic-in-azure-ai-model-catalog/ba-p/4121556

[5] https://www.together.ai/blog/snowflake-artic-llm

[6] https://investors.snowflake.com/news/news-details/2024/Snowflake-Launches-Arctic-The-Most-Open-Enterprise-Grade-Large-Language-Model/default.aspx

[7] https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/

[8] https://www.chaosgenius.io/blog/snowflake-arctic-vs-dbrx/

[9] https://www.techtarget.com/searchbusinessanalytics/news/366581900/Snowflake-targets-enterprise-AI-with-launch-of-Arctic-LLM

[10] https://timesofstartups.com/artificial-intelligence-2/snowflake-launches-arctic-the-most-open-enterprise-grade-large-language-model/

[11] https://www.techzine.eu/blogs/analytics/119158/snowflake-enters-the-llm-war-with-introduction-arctic/

[12] https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions

[13] https://www.chaosgenius.io/blog/snowflake-arctic/