Samsung caught cheating on benchmarks

[KeyboardDemon]

I'm not sure if this has been posted before, I couldn't see it in the News section and a quick glance through these pages didn't look like it was here either.

Samsung cheats

[kalniel]

It was in a post under an unrelated topic, but definitely deserves it's own. Not good at all, feels like we're back to the old 3dmark days.

[NecronomicoN]

Wait a minute, because the Samsungs have power scaling, this is cheating? Seems like a cool feature if it could be freely accessed. Downscale when you need battery, upscale when you want performance.

Regardless, the source doesn't exactly inspire trust. See if I can dig up an unbiased report.

[kalniel]

Wait a minute, because the Samsungs have power scaling, this is cheating? Seems like a cool feature if it could be freely accessed. Downscale when you need battery, upscale when you want performance.

This is more upscale when detecting certain specific benchmarks by name, not when you want performance. As the anandtech article points out - generic or even user-controlable upscaling would be fine.

[NecronomicoN]

This is more upscale when detecting certain specific benchmarks by name, not when you want performance. As the anandtech article points out - generic or even user-controlable upscaling would be fine.

Okay... so because the phone has plug-n-play profiles for specific applications... it is 'cheating'? I'm still not getting this.

My PC detects when I load an audio CD and automatically launches Winamp. My nVidia drivers detect when a game doesn't support SLI and automatically disables it. My CPU allows my AV to take priority over other programs. I'd be hard pressed to put any of these actions into a 'cheating' context.


This seems to be setting a new precedent. In a game with no rules. Are the programmers at competing companies just too lazy to create such profiles? Or are they green with envy, because they didn't think of it first? To say Samsung is "cheating", because they changed the way the imaginary game is played, is juvenile.

[kalniel]

Okay... so because the phone has plug-n-play profiles for specific applications... it is 'cheating'? I'm still not getting this.

Yes it's cheating if it detects a benchmark by name and boosts the CPU for this, while not boosting the CPU for other tasks that would benefit from increased CPU speed. Note it is not boosting a class of tasks, like say audio CDs. It's boosting specific, named, applications.

This seems to be setting a new precedent. In a game with no rules. Are the programmers at competing companies just too lazy to create such profiles? Or are they green with envy, because they didn't think of it first? To say Samsung is "cheating", because they changed the way the imaginary game is played, is juvenile.

Sadly there's nothing new about it, as I mentioned. The practise was rightly called out on the PC and consumer pressure & more vigilant benchmarking practises eventually led the cheaters to be more transparent.

If Samsung want, they can implement far better algorithms for boosting that would benefit the user - TDP caps, downthrottleing, condition controlled boost states etc. have all been shown to work well in the PC field and their transparency resolves any accusations of cheating - they are in fact a selling point, not something you need to hide away.

[scaryjim]

Okay... so because the phone has plug-n-play profiles for specific applications... it is 'cheating'?

Yes, because it runs benchmarks at speeds that you cannot access in real world usage. Although "lying" might be a better word - as Samsung never published any details of this. If they publish a clock speed of 480MHz, but run benchmarks as 523MHz without telling anyone, they're basically trying to claim that the performance at 523MHz is really the performance at 480MHz - which obviously it's not.

Imagine AMD added a patch either through drivers or to microcode, that made their Piledriver FX CPUs run a particular single-threaded benchmark - say PiFast since that's the one that Hexus uses - at 5GHz, adjusting the core voltage to make sure it ran stably. They could probably do this because they can work out that the benchmark will only run for a few seconds, and it's safe to run their processors at those kinds of speeds and voltage for that short a period. It would make AMDs single-threaded performance look much better compared to Intels. But it'd be a lie; they'd be boosting their single threaded performance by over 20% just to look good on benchmarks.

It's easily possible that Samsung are doing the same. They know the power circuitry and cooling in the phone can't cope with the GPU running at 523MHz all the time, but if you can predict the length of time it will be running at that speed (e.g. when running a fixed length benchmark test) it can do so safely. It looks great in benchmark charts, but it doesn't reflect real world performance: in that benchmark the Samsung would appear be, say, 5% ahead of its competition, instead of 5% behind.

It's far from new - it used to happen a lot with GPUs and a bit with CPUs - but I believe this is the first phone to be shown to deliberately exaggerate its benchmark performance.

[KeyboardDemon]

Thanks scaryjim, I couldn't have put it better myself.

Companies like Samsung know that many customers consider benchmark results as published online by tech forums and enthusiast reviews before buying a phone, if they see that a product performs at a certain level they might have the expectation that buying that item should mean that they are getting something that will perform that well in all tasks at all times. But in reality the only time their new device would work at the specified performance level is when they are running that performance test and they will never be able to see those performance levels unless all they plan on doing is running that test all the time, the end user could justifiably argue that they have been cheated in to buying the phone/tablet/whatever because of this deceptive practice.

[NecronomicoN]

This is being discussed in the official thread:
http://forums.hexus.net/hexus-news/2...rkbooster.html