C.
His “tokenomics” comment is in reference to the company’s claims that the Groq 3 LPX and Vera Rubin NVL72 racks, when connected and running in tandem, can significantly speed up the rate at which trillion-parameter models can produce tokens. These models can range in use case from chatbots with reasoning capabilities to coding assistants, producing multitudes of text or code as tokens depending on the application.
In one example, Nvidia said, the two systems can boost inference throughput for a 1-trillion-parameter GPT model by 35 times for every megawatt consumed by the two systems in comparison to the previous-generation Grace Blackwell NVL72 platform.
This would result in the model producing 300 tokens per second per megawatt as it serves 500 tokens per second for every user, which is expected to help large and influential AI model developers provide more expensive, premium services.
“The primary message [at GTC] was around token factories or high-end hyperscalers or AI natives whose business is built around tokens,” Weinstein said. “But as enterprises scale and tokenomics become even more important to those enterprises, that LPU system is going to be super important, [providing] low-cost inference that allows that enterprise to do millions if not billions of tokens to scale their token factories.”
The eventual channel play would apply to the “many” partners that are already delivering and implementing Nvidia’s rack-scale platforms such as the Grace Blackwell NVL72, according to Weinstein, who said one unnamed partner is handling 18,000 racks a year. These kinds of server racks often require liquid cooling and hundreds of kilowatts in electricity, limiting the hardware to customers that have the proper infrastructure for such products.
“These partners are in the rack-scale game, either in the enterprise or for neoclouds. Many of them even partner and do business with hyperscalers. The responsibility of the build side of that becomes a core ingredient of the services portfolio that those partners are building,” he said.
Weinstein said there is no timeline for when partners could start handling Groq LPX deployments, noting that the company will approach the channel “opportunistically.”
Lenovo, Mark III Execs See Limited Groq Enterprise Interest—For Now
While no OEM support has been announced for Groq 3 LPX, the sales leader of Lenovo’s data center business, Vlad Rozanovich, said his company would consider offering the rack if it starts seeing demand from customers, even if there isn’t a big enterprise play yet.
“We’ve heard about interest in Groq in places like Saudi Arabia,” he said in an interview with CRN last week, citing interest from the Middle Eastern country’s state-backed AI company, Humain, which had been using Groq chips prior to Nvidia’s announcement.
Elsewhere, Lenovo is seeing interest from “many of the same companies that are looking at large language models, but it’s not something that all enterprises are asking for,” said Rozanovich, who is senior vice president of Lenovo’s Infrastructure Solutions Group.
Andy Lin, CTO and vice president of strategy and innovation at Houston-based Nvidia systems integration partner Mark III Systems, told CRN that he thinks the Groq 3 LPX will find appeal among cloud service providers, neoclouds and AI-native companies “that really differentiate themselves on providing a great user experience for models.”
However, he admitted that those companies represent a “very small subset” of his company’s customer base because taking advantage of new chip architecture like Groq’s will require organizations that have a “certain amount of scale and advanced capability” with respect to development and integration resources.
As for Groq’s potential among enterprise customers, Lin said he sees limited appeal.
“I think it will require a special type of enterprise to want to do the work to integrate it into their current pipelines,” said the solution provider executive, whose company has won Nvidia Partner Network awards for multiple years in a row, including this year. “I would say it’s probably [going to appeal to] the small minority of enterprises because enterprises find it challenging enough to spin up a true AI Center of Excellence or AI factory.”