What We Can Learn From Thinking Machines’ Initial Inkling

Thinking Machines has released the full weights of its new multimodal AI model, Inkling, under the open-source Apache 2.0 license. This is a notable departure from industry norms, where models are often released with restrictions or only as API access, and signals a shift toward greater transparency and ownership in AI development.

The Inkling model is a mixture-of-experts transformer with 975 billion parameters and supports a 1-million-token context window. It was pretrained on 45 trillion tokens across text, images, audio, and video, and is natively multimodal, processing input from multiple modalities without additional adapters. The full model weights were published on Hugging Face under Apache 2.0, allowing users to download, modify, and deploy independently.

In addition to the large model, a smaller version, Inkling-Small, with 276 billion total parameters, was also released. It reportedly matches or exceeds the larger model’s performance on several benchmarks, thanks to an improved pre-training approach. The training process involved hybrid optimization and over 30 million reinforcement learning rollouts, with performance improvements logged during training.

However, the release came with some caveats. While the weights are openly available, the training data and full training pipeline have not been published, which is typical industry practice but limits full transparency. Additionally, reports suggest that Thinking Machines maintains a separate Model Acceptable Use Policy that restricts certain applications, such as surveillance and automated decision-making, raising questions about the true openness of the release.

Implications of Open-Weight Model Release

The release of Inkling’s full weights under an open license is a significant development in the AI industry. It enables organizations and researchers to fine-tune, inspect, and deploy the model independently, fostering transparency and control. This move contrasts with many industry players that restrict access through APIs or closed models, potentially setting a new standard for open AI development.

However, the existence of a separate usage policy that may limit certain applications complicates the narrative. It highlights ongoing tensions between openness and responsible use, especially in areas like surveillance and automated decision-making. This development raises questions about how truly open such models are and what responsibilities developers and users bear.

Industry Norms and Recent Open Model Releases

Historically, most large AI models are released either as API services with usage restrictions or with closed weights, limiting independent scrutiny. The recent trend has seen some organizations releasing models with open weights, but often with caveats or additional policies restricting use. The case of Inkling stands out because the full weights were released first, with an emphasis on transparency and ownership, reflecting a shift toward more open practices.

Previous open releases, such as Meta’s Llama 2 or EleutherAI models, have demonstrated the benefits of open access for research and development. However, concerns about misuse and proprietary data remain persistent. Inkling’s release is notable because it combines open weights with a licensing and policy framework that may restrict certain applications, illustrating the complex balance between openness and responsibility.

“Our goal is to empower users with full ownership of the model, while maintaining responsible use through our policy framework.”

— Thinking Machines spokesperson

Unresolved Questions About Model Use and Data Transparency

It is still unclear how the Model Acceptable Use Policy will be enforced in practice or how it might limit certain applications. The training data and pipeline have not been published, raising questions about the model’s transparency and reproducibility. Additionally, the actual impact of the open weights on misuse or proprietary concerns remains to be seen.

Next Steps for Model Evaluation and Industry Impact

Researchers and developers will begin to examine the model’s capabilities, safety, and compliance with the stated policies. Independent benchmarks and testing will clarify its strengths and limitations. Industry observers will monitor whether other organizations follow suit with open releases, potentially shifting norms around model transparency and ownership.

Key Questions

What makes Inkling different from other AI models?

Inkling is released with full weights under an open license (Apache 2.0), allowing independent use and modification. It is a large, multimodal, mixture-of-experts transformer with extensive training on diverse data types.

Does open release mean the model can be used freely for any purpose?

While the weights are openly available, reports suggest that Thinking Machines enforces a separate Acceptable Use Policy that restricts certain applications, such as surveillance and automated decision-making. Users should review this policy carefully.

What are the risks of releasing such a large model openly?

Open access can lead to misuse, including malicious applications or misinformation. It also raises concerns about proprietary data and intellectual property, especially if the training data remains undisclosed.

Will the training data be made public?

No, the training data and full training pipeline have not been published, which limits full transparency and independent verification of the model’s training process.

What is the significance of the licensing choice?

The Apache 2.0 license grants users broad rights to use, modify, and commercialize the model, fostering innovation but also requiring responsible use under the company’s policies.

Source: ThorstenMeyerAI.com