I'm James, the Manager of Performance Analysis at HEXUS.
I'd just like to elaborate on these issues, to help better explain the current problems with us using Crysis for benchmarking.
We first investigated using Crysis back in October last year (at the time the 8800 GT was being launched), and ran several cards on the Demo version then available.
After much frustration - and having to drop the maximum resolution we benchmark at, down to 1680x1050 - we managed to get a set of results that were reasonably consistent (<3% variance on two out of the three cards).
However, no matter how many times we ran Crysis, it wasn't possible to get usable results at 1680x1050 from the GeForce 8800GTS 320MB.
Subsequent to this, in our testing of the Radeon HD 3870 (still using the demo at this point), it was discovered not only that there was a high degree of variance between runs, but also complete sets of runs would produce wildly variable sets of results, than from before.
E.g. a set of 3 results from one day of 18.12, 18.25, 21.2, might produce a set of 3 results of 21.15, 23.15, 21.5 the next day, with nothing been changed on the system - the exact same game settings, OS build, and hardware!
Initially I put this down to problems with the Demo, however once the full game was released the problems persisted.
Therefore these issues, in addition to the games inability to properly scale on multi-GPU platforms - mainly on AMD GPU based systems, and eventually much later on fixed in the 1.1 patch - led us to abandon the use of Crysis in our testing.
The hopes of being able to use Crysis were once again dashed, when even upon the release of the 1.1 patch - and also with the assistance of the excellent Crysis Benchmark Tool:
Crymod Modding Portal | Finished Assets | Benchmarking Tool (released) - the inconsistencies remained, albeit to a lesser extent as it now only seemed to affect some cards, rather than all of them.
I doubt I need to tell anyone on here that these kind of variances would result in us being forced to make inaccurate, and misleading, judgements of a card’s performance, and this is something that neither myself or any other HEXUS member of staff would be willing to accept.
Still I've noticed the recent release of yet another Crysis patch, which I hope this time, in conjunction with making our own time-demo to benchmark, will result in consistent and therefore trustworthy and relevant comparisons.
Aez, to answer your query:
This is something we've always been looking at doing since the move to Vista, so expect to see a couple more games being benchmarked soon - once I've found a couple more that meet our exacting standards.