The introduction of Together AI's new benchmark marks a shift in measuring the performance of large language models (LLMs) designed for coding agents. This benchmark simulates the high-demand scenarios that developers face, emphasizing throughput and responsiveness in production environments. The results show that the latest iteration can achieve 625 tokens per minute (TPM) per GPU, significantly surpassing existing benchmarks and establishing a new standard for performance metrics in the field.
Addressing Flaws in Traditional Benchmarks
Standard benchmarks have often struggled to reflect the realities of production AI workloads, especially in situations with multiple concurrent requests. Traditional methods usually concentrate on peak performance but overlook how these systems manage sustained loads. Together AI's new benchmark aims to correct this by stressing LLMs under conditions that mimic real-world usage, specifically coding agent workloads that involve extensive input contexts, often between 45,000 and 200,000 tokens.
The benchmark's design prioritizes critical metrics such as time to first token (TTFT), tokens per second per user (TPS), and overall TPM, highlighting the need for efficiency in developer experiences. The TTFT, which measures the time it takes for a user to see the first output, is especially important; delays can greatly impact developer productivity and satisfaction.
The Impact of Concurrent Requests on Performance
In a typical coding environment, requests may involve tens of thousands of tokens, including files, conversation histories, and snippets of code. As multiple developers send requests at the same time, competition for resources like memory bandwidth and KV cache increases. This added pressure can lead to performance degradation, particularly in TTFT, if not managed properly.
Together AI's benchmark is specifically designed to assess how well LLMs handle these concurrent, long-context submissions. The results indicated that while traditional engines may struggle under high loads, Together AI's Inference Engine maintained a TTFT of under one second, providing a crucial advantage for developers.
Cost Efficiency and Performance Benefits
The benchmark results also highlight significant cost advantages. For instance, the Kimi K2.6 model on Together AI showed a cost of $0.108 per request, compared to the $0.451 cost associated with Claude Opus 4.6. This represents a remarkable 76% reduction in inference costs. For engineering teams, this could translate to savings of around $440,000 annually.
This efficiency is particularly significant amid rising costs associated with AI development. As companies work to optimize their operational budgets, Together AI's benchmark offers a clear framework for assessing real-world performance and cost-effectiveness.
Looking Ahead
As this benchmark is still in its first version, there are plans for continuous updates to reflect ongoing optimization gains. Together AI aims to create a transparent standard that can be widely adopted across the industry, ensuring developers have the tools necessary for efficient coding agent integration. The implications of these advancements could reshape AI infrastructure, especially for teams dependent on high-performance coding agents in their workflows.
Together AI's benchmark reveals the limitations of traditional models while establishing a new standard for performance and cost efficiency in the evolving field of coding agents. As the demand for more capable and affordable AI solutions increases, this benchmark could play a key role in guiding future developments in the industry.
