DxOMark isn't as reliable as you think

Whenever a new top-range phone is released, such as an Apple, Samsung or Google, we see it fly past other competitors in the DxO list. Just like that, a device that was ranking towards the top will have to settle for fifth or sixth place. Does the device take bad pictures because the brand isn't keeping up with the times? Take the Mi Mix 2S for example, why doesn't it deliver more than the iPhone 8 Plus?

First let's find out what DxOMark is, and why it dictates the market. This is how they define what they do:

The DxOMark is the industry's trustworthy standard for camera and lens image quality measurements and ratings. For years we have been recognized for providing the most rigorous hardware testing using industry-grade laboratory tools in our analyzes to establish the most comprehensive reference database with our thousands of camera test results and lenses.

In each test, they evaluate elements such as exposure, contrast, dynamic range, color saturation, hue, white balance, color shading, texture, noise, auto focus, artifacts (including frame softness, distortion, vignetting, chromatic aberration, touch, reflexes, ghosts, aliasing, moiré patterns, flash, zoom at varying distances, bokeh effect (otherwise known as portrait mode) and these are also tested for video recording, too. You can learn more about their process below:

All this is done with thousands of tests with photos. As you saw in the video, they have special labs where they take the same pictures in the same light situations, but also go outside to do new tests. Until 2017, the test didn't take some things into consideration. Now, new items started to integrate smartphone cameras, so the evaluation has changed somewhat since last year.

It's not without problems

But, the question remains: why are there devices with questionable quality in the ranking? One of the problems lies in the latest changes, and another in the weight that the evaluator gives to new developments in features like zoom and bokeh effect. In many cases, good cameras that don't have these items, end up losing to others that do not have the same quality, but can offer these extras that don't always add value for the individual user.

• Which devices do you think had the best camera last year?

Many other producers and content analysts I've spoken to have come up with an important point: manufacturers have figured out how to score well in DxOMark. Looking at other tests, it's not too difficult to figure out which aspects are the focus of the company's attention, like zoom and bokeh as I mentioned, as well as contrast and sharpness.

Important factors such as wide lenses, black and white photos and RAW format are not considered in the running, although they are important for the photographic quality. Because these important aspects don't carry that much weight in the scoring, good cameras are unable to score highly, and manufacturers are paying attention only to the factors that are rated, and could even be using software that DxO sells itself to analyze cameras.

The third problem is that DxOMark is not an NGO: it's not just a camera analysis company. The main activity of the company is consulting for companies that work with cameras, so the tests and ranking are nothing more than an advertisement for their own primary service, which is to advise hardware companies to improve their cameras. Do companies that receive advice from DxO get better places in the ranking?

It's impossible for a company not to have its biases, but these factors are a little worrisome. Combined with the fact that there are no other large companies carrying out technical tests like DxO, we have a problem. Many consumers unfortunately rely on the ranking to decide what to buy. The company does, however, try to put our minds at rest by stating that it does independent testing, and wouldn't tarnish its reputation so willingly.

In the end, all that matters is your taste and your budget

There are many benefits to using the DxO website, and it certainly knows what consumers are looking for, so it is useful to check it out before making a purchase. Always take the score with a pinch of salt though, and trust your own judgement with what you consider to be decent photographs, as well as making sure you look at reviews and photo tests before making your final decision. Take into account what you consider to be best without leaving the final decision to the ranking. Zooming to you may not be as important, but having RAW could be for example.

• What are the upsides of having a dual camera on your smartphone?
• Discover how to take incredible photos using manual mode

Ideally, you should always have as much information as possible, and then build your own opinions based on your needs and how much cash you have to splash on a new smartphone.

Do you trust the DxOMark ranking? Tell us in the comments!