INTEL GAUDI 2 REMAINS ONLY BENCHMARKED ALTERNATIVE TO NV H100 FOR GENAI PERFORMANCE

MLCommons published results of the industry-standard MLPerf v4.0 benchmark for inference. Intel’s results for Intel^® Gaudi^® 2 accelerators and 5th Gen Intel^® Xeon^® Scalable processors with Intel^® Advanced Matrix Extensions (Intel^® AMX) reinforce the company’s commitment to bring "AI Everywhere" with a broad portfolio of competitive solutions. The Intel Gaudi 2 AI accelerator remains the only benchmarked alternative to Nvidia H100 for generative AI (GenAI) performance and provides strong performance-per-dollar. Further, Intel remains the only server CPU vendor to submit MLPerf results. Intel’s 5th Gen Xeon results improved by an average of 1.42x compared with 4th Gen Intel^® Xeon^® processors’ results in MLPerf Inference v3.1.

Building on its training and inference performance from previous MLPerf rounds, Intel’s MLPerf results provide customers with industry-standard benchmarks to evaluate AI performance.

The Intel^® Gaudi^® software suite continues to increase model coverage of popular large language models (LLMs) and multimodal models. For MLPerf Inference v4.0, Intel submitted Gaudi 2 accelerator results for state-of-the-art models Stable Diffusion XL and Llama v2-70B.

Due to strong customer demand for Hugging Face Text Generation Inference (TGI), Gaudi’s Llama results used the TGI toolkit, which supports continuous batching and tensor parallelism, enhancing the efficiency of real-world LLM scaling. For Llama v2-70B, Gaudi 2 delivered 8035.0 and 6287.5 for offline and server tokens-per-second, respectively. On Stable Diffusion XL, Gaudi 2 delivered 6.26 and 6.25 for offline samples-per-second and server queries-per-second, respectively. With these results, Intel Gaudi 2 continues to offer compelling price/performance, an important consideration when looking at the total cost of ownership (TCO).




Following hardware and software improvements, Intel’s 5th Gen Xeon results improved by a geomean of 1.42x compared with 4th Gen Intel Xeon processors’ results in MLPerf Inference v3.1. As an example, for GPT-J with software optimizations including continuous batching, the 5th Gen Xeon submission showed about 1.8x performance gains compared with the v3.1 submission. Similarly, DLRMv2 showed about 1.8x performance gains and 99.9 accuracy due to MergedEmbeddingBag and other optimizations utilizing Intel AMX.

5th Gen Xeon processors and Intel Gaudi 2 accelerators are available for evaluation in the Intel^® Developer Cloud. In this environment, users can run both small- and large-scale training (LLM or GenAI) and inference production workloads at scale, manage AI compute resources and more.