Benchmarks are synthetic tests that are designed to mimic real-world performance. They are supposed to help us compare the performance of different smartphone processors, also known as system-on-chips (SoCs). These should help to make our lives easier by giving us simple and objective scores that we can use to make informed decisions. They are supposed to be the fairest and most reliable way to evaluate SoCs.
But are they really?
In this article, I will show you how we are most probably getting benchmarks wrong, and why we should not rely on them blindly. I will also show you how to use benchmarks properly and wisely, and what other methods we should consider when comparing SoCs.
Peak vs Sustained Performance
One of the ways we get benchmarks wrong is judging long-term performance by looking at the highest score, the peak performance. This is the score that we see on the websites, the ads, and the reviews. This is the score that we use to brag about our phones and bash other phones. A lot of people believe that this is the score that matters the most.
But does it really?
The truth is, most SoCs only operate at this level for a limited amount of time, from anything between 5 minutes to 30 minutes tops. After that, the performance usually takes a nosedive, due to various factors, such as heat, battery, and software. This means that we will only get that absolutely peak number for a short while, before it drops significantly.
So, when we are looking at peak performance, we should reduce about 15-20% from the peak score, to get the sustained performance. This is the performance that we will be working with most of the time, not the peak performance.
For example, if an SoC scores 1,000,000 on AnTuTu, we should adjust it by 15% and get around 850,000 of sustained performance. Because that’s mostly what we will be dealing with, not the 1,000,000.
These adjustments are mostly for Snapdragon SoCs, because they are fairly stable and efficient. With other SoCs, such as Exynos, MediaTek, Kirin, and Unisoc, the performance may completely fall off a cliff into the ocean. You may see as much as a 50% performance difference between peak and sustained performance. So much so that an SoC that scores 1,000,000 may start behaving like an SoC that scores 500,000 on AnTuTu.
This also applies to other benchmarks, such as GeekBench and 3DMark. And don’t forget, as your battery level falls, so does your performance levels. If you play games like me, you’ll notice that your performance levels will absolutely die when your phone gets to below 20%.
CPU vs GPU Performance
Another way we get benchmarks wrong is by using GeekBench alone. This is a popular method among Bionic stans, who love to flaunt their single-core scores and mock other SoCs. GeekBench only tests CPU performance, and tends to favor short burst workloads needing peak performance. This is why long-term workloads needing sustained performance, such as multi-core tests, are considered to be more important.
But even then, CPU performance is not the only thing that matters. The CPU cores does not work alone. It needs the GPU, the graphics processing unit, to handle the graphics tasks, such as games and videos. Some SoCs have very decent CPUs but crappy GPUs, and vice versa. This means that using GeekBench alone is not enough to compare SoCs. We need to consider other aspects, such as GPU performance, AI performance, ISP performance, and connectivity performance.
This is why I always separate my SoC comparison into different areas, such as:
– Performance
– Gaming
– AI (new)
– ISP and photography
– Connectivity
– Possible battery life
Picking only CPU performance and using it to crown an SoC as the better one over another SoC is ignorant at best and highly mischievous at worst. It is like judging a book by its cover, or a person by their height. It is not fair, and it is not accurate.
Benchmarks vs Real-World Performance
The final way we get benchmarks wrong is by forgetting that benchmarks are not the same as real-world performance. Benchmarks are synthetic tests, which means that they are artificial and simulated. They may not reflect the actual performance of SoCs in real-world scenarios, such as browsing, gaming, photography, and multitasking. They may use unrealistic or outdated scenarios, or may not account for the optimization and compatibility of the software and the hardware. These tests may also be influenced by external factors, such as the temperature, the network, and the background apps. They may also be manipulated or cheated by some manufacturers, who may tweak their SoCs to perform better on specific tests, or use different versions of the tests.
Therefore, benchmarks should not be the only way to compare SoCs, but rather one of the ways. They are useful and objective tools, but they also have limitations and drawbacks. These tests should be used with caution and skepticism, and not with blind faith and fanaticism. They should be complemented by other methods, such as user reviews, expert opinions, and hands-on experience. These methods can provide valuable and complementary insights on the performance of SoCs, and can help us make a more comprehensive and balanced comparison.
Conclusion: How to Compare SoCs Properly and Wisely
In conclusion, we have seen how we are most probably getting benchmarks wrong, and why we should not rely on them blindly. We have also seen how to use benchmarks properly and wisely, and what other methods we should consider when comparing SoCs.
Here are some tips and guidelines to help you compare SoCs in a fair and reliable way:
- Don’t look at the peak performance, look at the sustained performance. Adjust the benchmark scores by 15-20% for Snapdragon SoCs, and by 50% for other SoCs, to get a more realistic estimate of the performance.
- Don’t use GeekBench alone, use other benchmarks as well. Consider the GPU performance, the AI performance, the ISP performance, and the connectivity performance, as well as the CPU performance. Use benchmarks such as AnTuTu, 3DMark, GFXBench, and DXoMark, to get a more complete picture of the performance.
- Don’t rely on benchmarks alone, use other methods as well. Read user reviews, watch expert opinions, and try hands-on experience, to get a more accurate and personal impression of the performance.
- Use benchmarks as a reference, not as a verdict.
By following these tips and guidelines, you can compare SoCs in a more fair and reliable way, and you can make more informed and wise decisions. You can also avoid being fooled or misled by the benchmarks, and you can enjoy your smartphone more.
I hope you enjoyed this article, and I hope you learned something new. Thank you for reading, and happy comparing. 😊
Please leave a comment if you found this helpful and remember to:
- Subscribe to our YouTube channel
- Follow on Facebook
- Follow on WhatsApp
- Join our Telegram community
- Participate on Reddit
- Find us on Quora