If you have been around tech people in general or smartphone people in particular, the term “benchmark” is something that you must have heard by now. Benchmarks have caused a lot of arguments on many social media pages.
What is a benchmark and why is it so contentious?
What is a benchmark?
A benchmark is a standard or reference point against which other things are compared. It is mostly used as a tool of comparison. Benchmarks are used to compare anything and everything.
In the smartphone area, benchmarks are used to compare everything from the performance of entire phones to its individual components such as cameras, SoCs (CPUs and GPUs), storage speed, battery life etc.
Recommended: How to read Smartphone processor specs
Why do we use benchmarks?
Benchmarks provide a fair and universal way of comparing the performance of different smartphones and their components. This is because there are many specs associated with smartphones and these are all built differently.
A good example where this can be seen is in CPUs. There are many specs that are associated CPUs such as the number of cores, clock speed, architecture, transistor sizes etc. This has given rise to many misconceptions of CPU performance amongst users.
A very good misconception is the megahertz myth where people believe that “faster clock speeds equals better performance”.
With benchmarks, you do not have to understand these details. You jus have to look at the benchmark performance and see how good it is.
How are Benchmarks done?
A benchmark is usually consists of a series of standardized tests designed to test the performance of the thing to be compared. The results of these tests are universal. They are designed to be uniform for all items being tested.
Lets use the popular battle royale game, PUBG as a benchmark. We will assign the following grades to phones that can do any of the following:
| Test | Score | |||||
|---|---|---|---|---|---|---|
| Ability to max out all settings (both graphics and fps) | A | |||||
| Able to play the game at high settings with low fps | B | |||||
| Able to install game and play at medium settings | C | |||||
| Able to install game and play at low settings | D | |||||
| Able to install but unable to run the game | E | |||||
| Unable to install game | F |
Any phone that can complete the tasks as set out above gets the stated score.
Benchmarking is usually associated with assessing the ability of a CPU and it’s ability to run software. It provides a fair way to compare the performance of two or more CPUs/SoCs.
Principle
To be considered as a standard for measuring or comparing performance, a benchmark has to be:
1. Relevant: the tasks set out for use in the test must be useful in day to day use.
2. Fair: the tasks set out for must be the same for everyone.
3. Transparent: the processes for assigning scores must be fair and open for all to see.
4. Verifiable or repeatable: anyone should be able to repeat the same test and get the same results.
5. Cost effective: it should be cheap
6. Representative/Scalable: the tests should be designed for every device in its catchment area with no exceptions.
Common types of Benchmarks (smartphones)
The major generally accepted benchmarks for smartphones include: Antutu (CPU, GPU, RAM), Geekbench (CPU), 3D mark (GPU), PC mark (GPU), GFX (GPU) etc.
Recommended: What is a GPU?
Problems
Benchmarks have not been without their problems. They (as earlier mentioned) have been a highly contested subject for decades now. Some of the reasons include:
1. Different interpretation: different benchmarks can sometimes put out different scores. Thus it is sometimes left to the user to interprete these scores by themselves or to rely on those who can. This situation makes it possible for people to misinterpret benchmarks.
2. Vendors can and will optimize SoCs to run better on benchmark software. These SoCs would then score high in benchmark tests but would perform poorly on other tasks.
3. Outright cheating: SoC companies have been known to add code to the firmware of their SoCs. If these codes detect a benchmark test, they are designed to overclock the SoC to reach maximum performance. This way, they can get unrealistically high benchmark scores which they would then use for advertising.
4. Overly streamlined tests that focus only on processing power and ignore other stuff like heating levels, power consumption, throttling etc.
5. User perception: Sometimes average day to day use does not really reflect benchmark scores. It could be possible for a phone that did well in benchmarks to flop in real life. This has led to some distrust for benchmark scores.
6. No account for CPU/OS load: Most phone are usually tested using the best possible conditions. They usually do not take into account what would happen if the storage of a phone if filled up to a certain point with several apps running in the background.
Conclusion
Despite their shortcomings, benchmarks are still the fastest and most objective way that we can compare SoCs. Until another method is devised, they are the best that we can rely on for now so they are still very useful. The organizations owning these benchmarks have to step up and keep their benchmarks free and fair as well as looking for ways to eliminate cheating.
Please leave a comment if you have any difficulty and remember to:
Subscribe to our YouTube channel
Follow on Facebook
Join our Telegram community
Follow us on Instagram
Check out the Clubhouse as well
We remain Inquisitive Universe
