Features - Ian Drew Blog: Benchmarking for Smartphones and Tablets

Quote:

---

This is a guest blog by Ian Drew, Chief Marketing Officer and Executive Vice President Business Development for ARM. The views expressed in this blog are his and his alone. We invited Ian to share some of his thoughts on a hot topic of late: benchmarking for smartphones and tablets. Do let us know in the forums if you agree or disagree with his take on events.

---

Read more.

* removed by myself *

Gosh, what a huge insight into ARM's approach to (cough advertising cough) benchmarking.

I think it's only right when Intel claims to be faster than all it's other competition in the mobile market, that other competitors get to counter-punch those claims.

I quite like what this guy has to say, I trust it's not BS and it gives a fair comparison of ARM Vs x86 on android devices.... what we don't know is how much of this is due to the issues of running android on x86 architecture....in addition and in Intel's defence, this ATOM APU is old now, the new stuff will improve on the comparisons here.

A really great insight into ARM's benchmarking, though they haven't compared OS's that use the chipsets. As I'm guessing there could be a difference into performance.

I'm sorry but as far as I'm concerned graphs and charts without numbers are also graphs and charts without value, nothing more than marketing guff when we have to take it on trust the bars are accurately proportioned (they too often aren't). If the raw numbers are so amazing then use them and don't assume the public are too stupid to comprehend anything other than brightly coloured bars, frankly it's insulting - especially when other sources have a quite different story to tell.

I agree that using actual shipping devices is the correct approach to benchmarking but then you also need to use many devices to be fair and ensure you haven't picked a single very good implementation of one SoC and a single very bad implementation of another. You harp on about using a variety of benchmarks but then fail to apply them to a wide variety of devices...

Obviously using many benchmarks on many devices and publishing the raw numbers would make a much longer article but that's the key problem, it is too heavily dumbed down and summarised to the point of being almost useless for any objective analysis.

It's also a shame to see so many synthetic benchmarks and known outdated ones such as SunSpider. What about some real game frame rates, playing time (per battery capacity unit) until battery dead or measuring page load performance in a browser? The eternal problem with synthetic benchmarks is that users don't use their phones like that, it's all too often a poor indicator of how it will actually feel to use but this is rarely acknowledged in this type of marketing.

What other sources have 'a quite different story to tell'? Anandtech? The test essentially run by Intel with a few best/worst-case examples? I'm not saying something demonstrated by ARM themselves is going to be impartial either, but these results agree with other stuff I've seen around the web e.g. Phoronix, with the A9 a fair bit ahead of even the desktop Atom for example. Some numbers on the graphs would be nice though, to show they start at zero at least. TBH they do look about right in terms of percentage though.

Game frame-rates would be more dependant on the GPU than CPU a lot of the time, and the current Atom SoCs have fairly poor GPUs even vs entry level ARM SoCs of smaller die size (the Atom core isn't that small vs Bobcat/Jaguar/ARM cores).

All you have to do is look at the market TBH. Integrators aren't easy to lie to (you won't win them over with a few bars on a presentation ;)), they will pick what suits their needs best based on cost/performance/power/etc. The Atom SoC platform has been around for plenty long enough now and we have, what, two or three phones? Versus how many ARM based phones? Or tablets? IMO it's even more telling considering this is Intel we're talking about - they've frequently held the lion's share of a market despite being behind on performance e.g. Athlon days.

I wonder if many of these companies would prefer not to buy from Intel based on previous history. Having more sourcing options and control its better than Intel gouging then in the future.

True, with something like ARM (or MIPS) you're free to go with another SoC MFR without having to change much - the MFRs know that, which encourages competition. However, not all the OEMs will have had previous experience, meaning they'd be looking at things like price, performance, power. Unless you mean just their reputation?

Quote:

---

Originally Posted by watercooled

What other sources have 'a quite different story to tell'? Anandtech? The test essentially run by Intel with a few best/worst-case examples? I'm not saying something demonstrated by ARM themselves is going to be impartial either, but these results agree with other stuff I've seen around the web e.g. Phoronix, with the A9 a fair bit ahead of even the desktop Atom for example. Some numbers on the graphs would be nice though, to show they start at zero at least. TBH they do look about right in terms of percentage though.

Game frame-rates would be more dependant on the GPU than CPU a lot of the time, and the current Atom SoCs have fairly poor GPUs even vs entry level ARM SoCs of smaller die size (the Atom core isn't that small vs Bobcat/Jaguar/ARM cores).

All you have to do is look at the market TBH. Integrators aren't easy to lie to (you won't win them over with a few bars on a presentation ;)), they will pick what suits their needs best based on cost/performance/power/etc. The Atom SoC platform has been around for plenty long enough now and we have, what, two or three phones? Versus how many ARM based phones? Or tablets? IMO it's even more telling considering this is Intel we're talking about - they've frequently held the lion's share of a market despite being behind on performance e.g. Athlon days.

---

I essentially meant Intel (and any tech company) who also produce this kind of generalised, number deficient marketing guff (I used the word story quite deliberately with connotations of fiction). If you want to counter marketing guff you have rise above it with actual evidence, numbers and scientific rigor. Guff vs guff is like robot wars, daft creations bashing each other until one slips up by accident and gets caught out or the clock runs out in an anticlimax.

I'm not here defending Intel, I'm criticising this output from ARM as no better. If they want to show they have the better solution then they need to present better data.

Besides I agree with Kumagoro, ARM's dominance of the ultra mobile segment is based on more than raw performance. OS support, industry politics etc. Let's also not forget profitable competitors like Qualcomm and Samsung who unlike AMD aren't short of R&D and executive lunch budgets...