In a notable development for AI in Japan, Bit192, Inc. has teamed up with CoreWeave to utilize powerful GPU resources for enhancing natural language processing (NLP) models specifically designed for the Japanese language. This collaboration aims to strengthen the AI infrastructure in Japan, as Bit192 embarks on an ambitious project to train a 20-billion-parameter Japanese model, separate from existing frameworks such as GPT-NeoX-20B.
A Shift in AI Development
Bit192, Inc. began as a provider of contract services for Japanese publishing houses. Under the leadership of CEO Yasu Seno, the company has transformed into a significant player in the open-source AI sector, fueled by Seno's enthusiasm for video games and interactive storytelling. The firm is now dedicated to developing advanced NLP models and exploring multimodal AI capabilities, which include image generation through diffusion models using Japanese CLIP.
A key challenge for Bit192 is the lack of publicly available, high-quality datasets for training Japanese models. While English datasets like The Pile are plentiful, resources for Japanese remain scarce. To address this, Seno and his team have been actively crawling the internet, curating ebooks, and enhancing the C4-Japanese datasets provided by Google. They have also developed a proprietary validation dataset focused on creative writing, comprising 1.5GB of professionally written content.
CoreWeave's Role in Innovation
Bit192's search for a dependable cloud service led them to CoreWeave, which they discovered through the EleutherAI community. CoreWeave's cloud infrastructure has become essential, enabling Bit192 to accommodate its growing user base of 400,000 unique visitors each month while driving continuous innovation.
"The availability and reliability of CoreWeave’s service allowed us to serve our current models and continuously build and test new ideas," Seno stated. This adaptability has been crucial for Bit192, especially as they leverage NVIDIA A40 GPUs, which deliver superior performance compared to the more commonly used A100 GPUs in distributed training.
CoreWeave's deployment of NVIDIA A40 GPUs provides exceptional capabilities, significantly accelerating AI workloads. With features like Tensor Float 32 (TF32) precision and support for structural sparsity, these GPUs enhance both training throughput and inference speed, making them well-suited for Bit192's ambitious projects.
AI Novelist: Bridging Creativity and Technology
One of Bit192's latest projects, AI Novelist, was originally created as a chatbot engine for a video game but has since evolved into a versatile writing assistant. It has gained popularity among professional writers, editors, and casual users, assisting them in generating a wide range of content, from stories to haikus.
Seno shared a touching moment when he demonstrated AI Novelist to his grandmother, who was delighted to see the AI produce a haiku, illustrating the technology's ability to inspire across generations.
The Future of Japanese NLP
Looking forward, Bit192, Inc. anticipates a considerable expansion of consumer-grade AI applications in Japan, which they believe will spark a community-driven wave of innovation. With AI still in its early stages in Japan, Seno is optimistic about the potential for personalized AI solutions that could eventually integrate into everyday devices like smartphones and home computers.
As the partnership between CoreWeave and Bit192, Inc. develops, it is poised to advance AI capabilities in Japan, promoting progress in NLP and beyond. This collaboration not only emphasizes the significance of infrastructure in AI development but also highlights the increasing momentum behind open-source AI initiatives in the region.
