Elon Musk's artificial intelligence company, xAI, has released the weights and architecture of their 314 billion parameter Mixture-of-Experts model, Grok-1, under the Apache 2.0 license. This follows a pledge by Musk last Monday to make Grok freely available.
This week, @xAI will open source Grok
— Elon Musk (@elonmusk) March 11, 2024
xAI says the opensourced Grok-1 is the raw base model checkpoint from the pre-training phase, which concluded in October 2023. This means the model is not fine-tuned for any specific application (such as conversation and dialogue). The weights and architecture are available on GitHub, along with instructions for getting started with the model.
The model was trained from scratch by xAI within just 3 months of its founding. Unlike some open-source releases, such as Gemma and Llama, which come with limitations on use, Grok's open-source release offers broad access to the underlying technology. However, it falls short of the complete transparency provided by models like Pythia, Bloom, and OLMo, which include training code and the datasets used to train the models.
magnet:?xt=urn:btih:5f96d43576e3d386c9ba65b883210a393b68210e&tr=https%3A%2F%2Facademictorrents.com%2Fannounce.php&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2FannounceTo download the model, users can access a magnet link for a torrent file that is approximately 300GB and contains 773 files.
This release comes at a time of increasing tension between Musk and OpenAI. Musk, who left OpenAI in 2018 over concerns that it prioritized profits over safety, has been increasingly vocal about his criticism of the company. He recently filed a lawsuit against OpenAI, alleging that it had backtracked on its founding pledge of creating an open-source model in favor of prioritizing shareholders.
Chris McKay
Musk has also been critical of other AI companies, including Google. He recently criticized Google's Gemini AI image generator after it produced historically inaccurate and misleading content regarding race.
The release of Grok-1 is a significant step for xAI, which Musk launched last March. The company's first product, the Grok chatbot, was released in November and made available to paid subscribers on X (formerly Twitter). Musk claimed the chatbot was "designed to answer questions with a bit of wit and has a rebellious streak."
As the AI landscape continues to evolve rapidly, the open-sourcing of models like Grok-1 is likely to have a significant impact (despite the large computing resources needed to run the model). While this is certainly a direct jab at OpenAI, it will be interesting to see how the AI community responds to this release and how it influences the ongoing debate around the responsible development and deployment of AI technologies.