Real World Testing vs Benchmark Testing

[Chai]

Another reason why I'm interested in the actual 'raw performance' (640x480) is because that's the real potential of the CPU when there's no boundaries. 1600x1200 depends on your graphics card, your settings in games, graphics optimisations on your own systems, which is very subjective and most likely very different from the reviewers anyway, and it's up to the readers to buy their own graphics card with their own budget, unlike most reviewers with unlimited resources on hardwares. It will never tell you how well your 'real world' performance is going to be.

640x480 is just a lazy way to make sure that the reviewers won't hit the GPU limit without noticing. Most games today are still CPU limit at 1024x768 if you have an high end card.

[goldfries]

Originally Posted by Chai

Then again, readers will accuse of Nvidia trying to show the performance advantage that we might not see in normal circumstances...

LOL. darn users - what is a review? wouldn't it be great find out the full potential of the card? i know that for me - i'd like to find out the full potential of a card instead of the assumed normal use.

to me, if the card could do 30fps with 1600x1200 4xAA 8xAF, then i'm assured that i wouldn't have problem with it in an real-world usage already.

[Adrian Wong]

TBH, I see points on both sides of the camp. So, I don't think both are totally wrong. Neither are they totally right.

I would agree with the Firing Squad in that it's still useful to isolate the performance of the graphics card via a timedemo versus the addition of physics, etc. and letting the CPU influence the scores more.

That's because we cannot possibly simulate the overall experience for other systems out there. The most we can do is tell people how it performs in our system. And if that system is CPU-limited, well, just how accurate will the results be? Just how representative are our results?

But HardOCP is also correct in that real-world testing can better give a reader an idea of just how much a change in the system can affect its overall performance.

They are also right in that there's really no use to have pages of benchmark results without any analysis. Anyone can run benchmarks. We need critical minds to analyse and interprete the results.

IMHO, I think it all depends on the angle on which you wish to approach a particular story. Granted, the topic is on how best to accurately determine and differentiate performance. But I think the best way would be to make use of one or both approaches, depending on the situation, instead of sticking to one.

Incidentally, I had the opportunity to discuss with ZuePhOk about game benchmarks just two days ago. I think we came to the conclusion that instead of just showing the average frame rate, it would be better to also show the lowest and highest frame rates. The most important, of course, would be the lowest frame rate as this determines the worst case performance you can expect.

Perhaps instead of a single bar chart, we can start using a line chart showing the min, max and mean values...

[goldfries]

Originally Posted by Adrian Wong

Incidentally, I had the opportunity to discuss with ZuePhOk about game benchmarks just two days ago. I think we came to the conclusion that instead of just showing the average frame rate, it would be better to also show the lowest and highest frame rates. The most important, of course, would be the lowest frame rate as this determines the worst case performance you can expect.

that would be subjective to individual preference too. some people just like to know what is the maximum potential of a card (bragging rights?).

for me, i do agree that it'll be great to know the lowest frame rate.

[Chai]

But not every benchmark shows the lowest frame rate. Of all the game benchmark used in our reviews, Quake4 and Doom3 doesn't have any lowest frame rate report.

Maybe you are missing my point. It's ok to do the min, average, max via a graph chart, but the problem is the variability of the AA and AF settings, it makes it not comparable at all, even with the same review.

He said that X card can do 4xAA 8xAF, while the Y card can do 2xAA 8xAF at similar fps. But as all we know, some cards do better at AA or AF, while another card is stronger without any AA or AF. This is all due to different algorithms. Maybe it's wiser to use something like 2xAA 16xAF instead of 4xAA 8xAF for example. It creates way too much variabilities to make any comparison fair using HardOCP's method. And don't forget user preferences mentioned in my post earlier.

We shouldn't depend everything on the reviews. That's why even with the benchmark charts, it's only a gauge how well it's going to perform. For me, benchmark results are good enough to tell which card is the card to buy. Users should do their own part by optimising the graphics detail on their own. With charts covering most of the common settings like AA, AF etc, you should know which is the card to buy. I wouldn't buy a known slower card if the price is similar, even if it's not going to improve the performance of your existing PC.

I just can't see the point of not buying Conroe because A64 does the same at GPU limited situation even if Conroe is clearly faster.

[jamotto]

Originally Posted by Adrian Wong

Incidentally, I had the opportunity to discuss with ZuePhOk about game benchmarks just two days ago. I think we came to the conclusion that instead of just showing the average frame rate, it would be better to also show the lowest and highest frame rates. The most important, of course, would be the lowest frame rate as this determines the worst case performance you can expect.

Perhaps instead of a single bar chart, we can start using a line chart showing the min, max and mean values...

Admittedly it is harder to calculate, but I think instead of using Min Max values one should really use Standard deviation. This way you know what the real range is for framerate instead of the Min Max range which may only be valid 1% of the time, Just my 2 cents.

[Adrian Wong]

Sorry, the min fps was really a tangent point. Just something I thought I should just pop in, since this topic's on benchmarking graphics cards.

Yup, of course, it wouldn't work with games that do not report minimum frame rates. But if possible, I think it would be great to see some minimum frame rates.

Hmm.. I get your point. Yeah, would that really matter to most readers? What if we assume wrongly and our readers really prefer to just use 4xAA 8xAF? To say that a card would perform even better at 2xAA 8xAF would be quite pointless.

I guess it does serve to differentiate the performance of the cards compared but I think it's only useful if you pit one card against another. I don't see how it's possible or even useful in a multi-card comparison.

Plus, different cards use different ways of doing AA. Even cards from the same generation will have different ways of doing AA. 4xAA in an ATI Radeon is not exactly 4xAA in the NVIDIA GeForce. So, no matter which method we use, the results are just rough guides.

But I would like to add that benchmarks will only tell one part of the story. Granted, in graphics card reviews, they are an important part of the review. But as a consumer, I would be interested in other aspects of the card, like components used, accessories included, etc.

[Chai]

Actually the main topic of this thread is about CPU benchmarking, which is a bit more complicated. Some users want to see how well their graphics card perform with their cards, while it's not easy because the reviewers normally use very high end cards, while not all readers can afford them.

Normally, I would just test the raw CPU performance because that's what I normally want to know. When I see reviews with 1600x1200 results, I would just skip it. But since there are quite a number of readers wanting to know 1600x1200 results, I might change how I'm going to test in the next CPU reviews in the future.

Personally, I would just use the results as a gauge, because I know I won't be getting the same fps even if I have the same processor or graphics card. If I know X is faster than Y, I would just get the X. I don't think it's possible to simulate real world conditions.

[Adrian Wong]

Originally Posted by Chai

Actually the main topic of this thread is about CPU benchmarking, which is a bit more complicated. Some users want to see how well their graphics card perform with their cards, while it's not easy because the reviewers normally use very high end cards, while not all readers can afford them.

Normally, I would just test the raw CPU performance because that's what I normally want to know. When I see reviews with 1600x1200 results, I would just skip it. But since there are quite a number of readers wanting to know 1600x1200 results, I might change how I'm going to test in the next CPU reviews in the future.

Personally, I would just use the results as a gauge, because I know I won't be getting the same fps even if I have the same processor or graphics card. If I know X is faster than Y, I would just get the X. I don't think it's possible to simulate real world conditions.

Yup, CPU benchmarking, but involving the graphics cards. Haha.. That's why I said it was a tangent. Sorry.

Anyway... I agree with you. It would be better to just test the individual component, rather than the whole system. It would be great to know how the overall system performs, BUT that's really not representative of, or replicable by the readers.

Everyone has a different configuration. What will a report on the overall performance of our testbed tell them? Can they replicate the results? Not really.

[Olle P]

Originally Posted by Adrian Wong

Perhaps instead of a single bar chart, we can start using a line chart showing the min, max and mean values...

Please do that! It's only low actual framerates that cause irritation when using it, and knowing how much time will be spent below 20 fps is far better than knowing the average or upper limit.
(I'd rather play games with the framerate fixed at 25 fps than have the framerate average at 60 fps with slowdowns to 10 fps when there's intense action going on.)

/Olle