Real World Testing vs Benchmark Testing

[Chai]

Seriously, I don't mind testing the 1600x1200 results if the readers insist, if readers can't find what they want to read, what's the point? But the 'real world' testing is a no-no for me.

[PsYkHoTiK]

Originally Posted by Papercut

Fair enough but 640x480 is like giving the best case scenario...at the other end you have 1600x1200 and whatever AA/AF for GPU testing, but I feel that those settings are closer to what some may actually use (maybe on a 24" moneytor?), whereas 640x480 merely gives you results on paper, so to speak.

Just my $0.02

Well it's hard to know what most people use.

And again, if we used something inbetween, we can't see the limits of the hardware. If I test a graphics card and ran tests at lets say 1024x768 then at the higher IQ settings there wouldn't be too much difference if the graphics card is a powerful one because you would be seeing a system bottleneck skewing the results.

It's for that reason that NVIDIA recommended that reviewers use 1600x1200 and even higher resolutions for SLI testing so that the results wouldn't be skewed.

I'll bench a lil and show you what I mean (later on.)

[Chai]

Originally Posted by PsYkHoTiK

Well it's hard to know what most people use.

And again, if we used something inbetween, we can't see the limits of the hardware. If I test a graphics card and ran tests at lets say 1024x768 then at the higher IQ settings there wouldn't be too much difference if the graphics card is a powerful one because you would be seeing a system bottleneck skewing the results.

It's for that reason that NVIDIA recommended that reviewers use 1600x1200 and even higher resolutions for SLI testing so that the results wouldn't be skewed.

I'll bench a lil and show you what I mean (later on.)

Then again, readers will accuse of Nvidia trying to show the performance advantage that we might not see in normal circumstances...

[Papercut]

Originally Posted by Chai

Then again, readers will accuse of Nvidia trying to show the performance advantage that we might not see in normal circumstances...

Exactly!
The way I see it, there is no perfect method of testing...both the "real world" and "benchmark" approaches have their merits...but both will have people arguing against it for various reasons.

[PsYkHoTiK]

Originally Posted by Chai

Then again, readers will accuse of Nvidia trying to show the performance advantage that we might not see in normal circumstances...

I know that but I think there is the logic in it. I mean would you see much difference if you ran a a pair 7900GTX in SLI but test at 1280x1024? It would be really be under utilized.

And define what is normal. To me it was normal for me to play HL2 at 1600x1200 with 4AA and 8AF. I would crank it higher if I had SLI. Same with Doom3. Same with Serious Sam II (although no AA for HDR.)

Thats why I stand by the way we test it. It is impossible to catalogue the various "normal" settings people use. Just use the objective way that shows the results that can be replicated. By doing that, we can show the limit of the various parts we test.

NVIDIA writes their recommendation. I don't use it at all, but I do use that resolution not because thats what they wrote, but to me thats what made sense. It's not like the benchies are made by NVIDIA or neither are the ones we use synthetic benchies (remember the nvidia & 3dmark fiasco a while back?) We use actual games.

[Olle P]

Originally Posted by Papercut

...both the "real world" and "benchmark" approaches have their merits...

Which is why I prefer to have both presented! Why skip on one out of two good?

Benchmarking is the only useful way to compare the absolute performance of product A vs product B, whereas the "real world" approach is generally better if I want to know wether the performance of A and/or B is good enough for my needs.
A might have better benchmark performance, while B still provide sufficient "real life" performance and some extra features to that.

Cheers
Olle

[Chai]

Originally Posted by Olle P

Which is why I prefer to have both presented! Why skip on one out of two good?

Benchmarking is the only useful way to compare the absolute performance of product A vs product B, whereas the "real world" approach is generally better if I want to know wether the performance of A and/or B is good enough for my needs.
A might have better benchmark performance, while B still provide sufficient "real life" performance and some extra features to that.

Cheers
Olle

Ok, define your 'real world' satisfactory level then.

What AA, AF, resolution, details, minimum fps, average fps, maximum fps? If 1/2 the users here have similar requirements, then we can set the standard. If not, what's the purpose of 'real world' results? Who is the one to set the standards? You? Me? Do you prefer AA or HDR? Since Nvidia can only do one at a time? Which is your priority? AA or AF? Personally, I think 16xAF should come first before any AA, but I don't think everyone agrees with that.

[PsYkHoTiK]

Exactly what I meant. No way to cater to individual "norms"

I have two other brothers here. To me 1600x1200 with 4aa 8af (except with HDR) is normal. My other brother plays at 1280x1024 or 1024x768 whichever runs best. My other brother couldn't be bothered what setting its on. Papercut prolly runs his stuff at 1280x1024... Chai thinks that 16AF should come first. I think that 4aa 8af is good enough for any game and I refuse to run a game at something other than full details.

The possibilities are limitless..

[Papercut]

Originally Posted by PsYkHoTiK

Exactly what I meant. No way to cater to individual "norms"

I have two other brothers here. To me 1600x1200 with 4aa 8af (except with HDR) is normal. My other brother plays at 1280x1024 or 1024x768 whichever runs best. My other brother couldn't be bothered what setting its on. Papercut prolly runs his stuff at 1280x1024... Chai thinks that 16AF should come first. I think that 4aa 8af is good enough for any game and I refuse to run a game at something other than full details.

The possibilities are limitless..

Ok I see your point now

[Max_87]

Originally Posted by Chai

Ok, define your 'real world' satisfactory level then.

What AA, AF, resolution, details, minimum fps, average fps, maximum fps? If 1/2 the users here have similar requirements, then we can set the standard. If not, what's the purpose of 'real world' results? Who is the one to set the standards? You? Me? Do you prefer AA or HDR? Since Nvidia can only do one at a time? Which is your priority? AA or AF? Personally, I think 16xAF should come first before any AA, but I don't think everyone agrees with that.

Haha! exactly! Especially the minimum FPS part. Different games require different levels of FPS to achieve a satisfactory gaming experince, and different people would have different FPS level requirements. How low is the minimum satisfactory FPS level? 30FPS? 40FPS? 50? 60? Another reason why I don't like the way HardOCP tested the cards.