What is TOPS and why is it important for AI?
Quick answer: TOPS (Tera Operations per Second) is a term used to measure, simplify, and advertise Neural Processing Unit (NPU) performance in AI PCs. TOPS is not a perfect metric, and many variables contribute to how well a system can handle AI tasks. It nevertheless provides a quick reference to an NPU's speed and how it compares to its competition.
As we wade into the world of AI PCs, plenty of new terms and acronyms are popping up that explain the new hardware and abilities.
You might have noticed that many modern laptops — and now desktops with AMD's Ryzen PRO 8000 announcement — are shipping with processors (CPU) that include a Neural Processing Unit (NPU) for a boost in AI-specific performance.
With the NPU comes a need for a new metric to measure performance, and that is where the term "TOPS" comes into play. Here's why you'll be hearing a lot more about TOPS in the future as AI PCs continue to saturate the market.
What is TOPS?
TOPS is not a new term in the tech world, but it's certainly getting a lot of mainstream attention recently with the rise of high-end AI PCs (the definition is still murky, but it boils down to a PC with modern CPU, GPU, and NPU). Windows Central Editor Rebecca Spear wrote a great article about how NPUs differ from GPUs, which you can reference for more information.
TOPS denotes how many trillions of operations a system can perform per second. TOPS can be used to measure overall system performance, but it can also be split up for specific hardware.
For example, NVIDIA's RTX 4090 desktop graphics card (GPU) can offer more than 1,300 TOPS of performance, whether for gaming or to accelerate AI tasks. It's one of the most powerful GPUs on the market today, yet that number is still slightly baffling. Even NVIDIA's mobile GPUs can deliver up to 1,000 TOPS of power for AI workloads.
Intel Meteor Lake and Lunar Lake
Intel's Core Ultra "Meteor Lake" CPUs for laptops offer 34 TOPS overall (including the CPU, integrated GPU, and NPU), with about 11.5 TOPS carved out of the total for the NPU alone. Next-gen Intel "Lunar Lake" mobile chips are expected to pump what it calls "NPU 4" up to 48 TOPS.
With its Lunar Lake announcement, Intel introduced new terms to better denote performance. Platform TOPS, or "PTOPS," is used to calculate the total of the CPU, GPU, and NPU. Peak TOPS, or "pTOPS," is used to denote individual part performance (as from the NPU).
With that in mind, Intel says its Lunar Lake chips will deliver an NPU with 48 pTOPS while the entire chip will have 120 PTOPS with all components taken into account.
At the 2024 Intel Tech Tour in Taipei, Taiwan, Intel also alluded to Copilot+ features being able to run on Meteor Lake NPUs (which hit about 11.5 pTOPS) in the future if requirements come down. That seems like a tall order, and only time will tell.
AMD Ryzen 8040 and Ryzen AI 300
AMD's Ryzen 8040 mobile CPUs offer NPU performance up to 16 TOPS, slightly higher than Intel's "Meteor Lake" chips. There are also AMD Ryzen PRO desktop APUs with NPU for professionals, also sporting an NPU with 16 TOPS. These chips are available now, but anyone interested in AI performance should no doubt wait to check out the AMD Ryzen AI 300 hardware announced at Computex 2024.
AMD's two new mobile CPUs built on the Zen 5 architecture include the flagship Ryzen AI 9 HX 370 and AMD Ryzen AI 9 365, each with an NPU with 50 TOPS of power.
Laptops with Ryzen AI 300 chips are expected later this year from Acer, ASUS, HP, Lenovo, and MSI.
Qualcomm Snapdragon X and Copilot+
Microsoft and Qualcomm teamed up to introduce Copilot+ on May 20, 2024. Copilot+ is an umbrella term for a set of advanced AI features – including Windows Recall, Live Caption, and local image and text creation – coming to Windows 11. The Copilot+ PC brand is used for laptops capable of running the features locally with the help of a powerful NPU with at least 40 TOPS.
On Qualcomm’s side, May 20 was the unveiling of a plethora of Copilot+ laptops powered by Snapdragon X Elite and Snapdragon X Plus chips. These Windows on ARM devices from nearly all major laptop brands have a Hexagon NPU with 45 TOPS of power, and they’ll be the first Copilot+ PCs to hit the market starting June 18, 2024.
Intel’s next-gen “Lunar Lake” and AMD’s Ryzen AI 300 chips will get access to Copilot+ features at a later date.
On the subject of upcoming Copilot+ devices, it was also announced at Computex 2024 that NVIDIA’s RTX-enabled devices are expected to get access sometime later this year. NVIDIA showed off five RTX laptops from ASUS and one from MSI that will get Copilot+ access, with the features debuting on RTX 40-series mobile hardware.
Why is AI performance measured with TOPS?
TOPS certainly does not represent the whole picture when analyzing NPU performance. Chip makers are leaning into TOPS to advertise their products, and it's mainly to simplify performance and help buyers understand what they're getting.
Like the TFLOPs craze in 2020 when next-gen consoles were still on the horizon, TOPS is being used to simplify and advertise system performance to more people. An NPU capable of two or three times as many TOPS as its competitor looks great on paper, but the metric ultimately lacks nuance.
It's a "Big number good, small number bad" situation, at least at first glance, but there are many more factors at play that can influence an NPU's performance. Theoretically speaking, a specific NPU with, say, 100 TOPS might be tuned for a specific task, in which it can outperform a separate NPU with 200 TOPS that is tuned for a different task.
Measuring a PC's overall performance generally boils down to concrete benchmark numbers and abstract observations. In our laptop reviews, for example, we provide plenty of hard numbers using specialized testing software, but we also provide a more general "feel" of how the system performs in real-world scenarios.
Testing an NPU is also something that needs to be worked out as we move forward. There are benchmarks available that target the NPU alone, and they're a good start on the way to measuring power in terms of specific tasks.
So, even if TOPS isn't the best metric to judge an NPU's performance, it does give buyers a standard with which they can roughly compare AI PCs.
Should you care about TOPS?
Measuring TOPS is a quick and easy way to compare NPUs or to judge whether or not an NPU is capable of handling certain tools. However, TOPS should not be considered the end-all measurement when it comes to NPU power. There's a lot of nuance involved, and you want to be sure that you're getting the right hardware for the job ahead.
Should you care about TOPS? It depends. If you're a casual PC user who handles email, web browsing, and productivity, you can no doubt continue without worrying about NPUs and the methods used to measure their performance. If you're shopping for a new laptop, you might still want to pay attention to the requirements for certain AI tools that can help make your life easier.
If you are, on the other hand, interested in the emerging world of AI PCs, NPUs and their capabilities measured in TOPS are likely already familiar topics. Just be sure to consider that TOPS is not the perfect metric and is instead an easy way for chip makers to compare and advertise their products.
