If you have been around tech people in general or smartphone people in particular, the term “benchmark” is something that you must have heard by now. Benchmarks have caused a lot of arguments on many social media pages.
What is a benchmark and why is it so contentious?
What is a benchmark?
A benchmark is a standard or reference point against which other things are compared. It is mostly used as a tool of comparison. Benchmarks are used to compare anything and everything.
In the smartphone area, benchmarks are used to compare everything from the performance of entire phones to their components such as cameras, SoCs (CPUs and GPUs), storage speed, battery life, etc.
Recommended: How to read Smartphone processor specs
Why do we use benchmarks?
Benchmarks provide a fair and universal way of comparing the performance of different smartphones and their components. This is because there are many specs associated with smartphones and these are all built differently.
A good example where this can be seen is in CPUs. Many specs are associated with CPUs such as the number of cores, clock speed, architecture, transistor sizes, etc. This has given rise to many misconceptions of CPU performance amongst users.
A very good misconception is the megahertz myth where people believe that “faster clock speeds equals better performance”.
With benchmarks, you do not have to understand these details. You just have to look at the benchmark performance and see how good it is.
How are Benchmarks done?
A benchmark usually consists of a series of standardized tests designed to test the performance of the thing to be compared. The results of these tests are universal. They are designed to be uniform for all items being tested.
Let’s use the popular battle royale game, PUBG as a benchmark. We will assign the following grades to phones that can do any of the following:
| Test | Score | |||||
|---|---|---|---|---|---|---|
| Ability to max out all settings (both graphics and fps) | A | |||||
| Able to play the game at high settings with low fps | B | |||||
| Able to install game and play at medium settings | C | |||||
| Able to install game and play at low settings | D | |||||
| Able to install but unable to run the game | E | |||||
| Unable to install game | F |
Any phone that can complete the tasks as set out above gets the stated score.
Benchmarking is usually associated with assessing the ability of a CPU and its ability to run software. It provides a fair way to compare the performance of two or more CPUs/SoCs.
Principle
To be considered as a standard for measuring or comparing performance, a benchmark has to be:
- Relevant: the tasks set out for use in the test must be useful in day-to-day use.
- Fair: the tasks set out must be the same for everyone.
- Transparent: the processes for assigning scores must be fair and open for all to see.
- Verifiable or repeatable: anyone should be able to repeat the same test and get the same results.
- Cost-effective: it should be cheap
- Representative/Scalable: the tests should be designed for every device in its catchment area with no exceptions.
Common types of Benchmarks (smartphones)
The major generally accepted benchmarks for smartphones include Antutu (CPU, GPU, RAM), Geekbench (CPU), 3D mark (GPU), PC mark (GPU), GFX (GPU), etc.
Recommended: What is a GPU?
Problems
Benchmarks have not been without their problems. They (as earlier mentioned) have been a highly contested subject for decades now. Some of the reasons include:
- Different interpretations: different benchmarks can sometimes put out different scores. Thus it is sometimes left to the user to interpret these scores by themselves or to rely on those who can. This situation makes it possible for people to misinterpret benchmarks.
- Vendors can and will optimize SoCs to run better on benchmark software. These SoCs would then score high in benchmark tests but would perform poorly on other tasks.
- Outright cheating: SoC companies have been known to add code to the firmware of their SoCs. If these codes detect a benchmark test, they are designed to overclock the SoC to reach maximum performance. This way, they can get unrealistically high benchmark scores which they would then use for advertising.
- Overly streamlined tests that focus only on processing power and ignore other stuff like heating levels, power consumption, throttling, etc.
- User perception: Sometimes average day-to-day use does not reflect benchmark scores. It could be possible for a phone that did well in benchmarks to flop in real life. This has led to some distrust of benchmark scores.
- No account for CPU/OS load: Most phones are usually tested using the best possible conditions. They usually do not take into account what would happen if the storage of a phone filled up to a certain point with several apps running in the background.
Conclusion
Despite their shortcomings, benchmarks are still the fastest and most objective way that we can compare SoCs. Until another method is devised, they are the best that we can rely on for now so they are still very useful. The organizations owning these benchmarks have to step up and keep their benchmarks free and fair as well as look for ways to eliminate cheating.
Please leave a comment if you have any difficulty and remember to:
- Subscribe to our YouTube channel
- Follow on Facebook
- Follow on WhatsApp
- Join our Telegram community
- Participate on Reddit
- Find us on Quora