Real World Testing vs Benchmark Testing

[Chai]

As you have probably known, HardOCP has switched to a very radical method of testing CPU or graphics card known as the 'real world' testing, while most of the websites these days are still using the traditional benchmarking testing using various test suites, games etc. including ARP. Firing Squad has launched an 'attack' on HardOCP, and vice versa here.

http://enthusiast.hardocp.com/articl...50aHVzaWFzdA==

http://firingsquad.com/hardware/jaku..._benchmarking/

Personally, as a tester of many graphics card as of late, I'm definitely sticking with the old benchmarking method, because there's no true definition of real world testing, it's very difficult to compare performance. Defining performance based on variable testing configuration like setting AA or AF is extremely subjective. It's like different people having different taste or preference when choosing a car. Do you like a comfortable car with soft springs, or do you like a powerful car with responsive handling? So would you choose 2xAA with 16xAF or, 4xAA with 8xAF?

Testing graphics card and CPU is getting very complex these days, due to the fact that CPU and GPU bottleneck happens all the time.

So what I do is, when testing GPU, I will only test at high resolution to show the differences of various graphics card. And likewise, when I test the CPU, I will choose lower resolution. If I were to include 1600x1200 results to show GPU bottleneck results for a CPU review, what's the point of testing? It's like trying to push the limits of super grippy F1 tyres with an underpowered compact car, it's a waste of time. If you want to know how well a GPU does with certain CPU, having a separate article like "CPU scaling" or "GPU scaling" would show what HardOCP is trying to tell in the latest Core 2 Gaming review, but in my opinion, CPU scaling article is much more straightforward. HardOCP's article tells us the GPU is not scaling well with today's CPU, end of story. But with a GPU scaling article, we will know where's the limit of a CPU.

The interesting thing is, HardOCP has done that in the past. http://www.hardocp.com/article.html?art=NjMy

What's your say? Which do you prefer? Do you like our benchmark testing or HardOCP's real world testing?

[Papercut]

Personally I feel that benchmarking a game at 640x480 is also quite useless. It may show the differences in CPU power but who the hell plays games at 640x480?
So in that sense it's not relevant at all...
How about testing at something like 1280x1024? Would that show the differences in CPU power whilst still reflecting a setting that people actually use?

[PsYkHoTiK]

I prefer the way we've been doing it all along. Can't say I quite like how HardOCP's benchies are. I mean its not like we're using stuff like 3dMark. Ours are games as well.

We isolate stuff to show the scaling and thats that.

For CPU we isolate any advantage the card may have by using low resolutions. Therefore the bottleneck is not the graphics but the system itself.

If it's graphics, we use 1600x1200 to make the GPU the bottleneck.

I personally do it that way so I can reflect on the changes in performance. I mean no one really plays at low res but not everyone plays at high res either. But regardless, by isolating the piece of hardware that we are going to test is essential to show the performance differences with other alternatives.

[goldfries]

prefer Benchmark testing.

[Max_87]

I still prefer benchmark testing.

Personally, as a tester of many graphics card as of late, I'm definitely sticking with the old benchmarking method, because there's no true definition of real world testing, it's very difficult to compare performance. Defining performance based on variable testing configuration like setting AA or AF is extremely subjective. It's like different people having different taste or preference when choosing a car. Do you like a comfortable car with soft springs, or do you like a powerful car with responsive handling? So would you choose 2xAA with 16xAF or, 4xAA with 8xAF?

Very true.

[Chai]

Originally Posted by Papercut

Personally I feel that benchmarking a game at 640x480 is also quite useless. It may show the differences in CPU power but who the hell plays games at 640x480?
So in that sense it's not relevant at all...
How about testing at something like 1280x1024? Would that show the differences in CPU power whilst still reflecting a setting that people actually use?

The purpose of low resolution test is not to show how well you can game at low res, but how well the CPU does if the GPU bottleneck is removed.

That's exactly what we are doing with our GPU reviews. I'm testing at 1600x1200 32xAA 16xAF not because we are able to game at that resolution, but to show how well it does when compare to other cards, or how badly the card does at higher details.

We all know that GPU is going to be bottleneck at high detail, while CPU will be bottleneck at low detail. Isn't that quite obvious already?

1280x1024 is probably going to be 1/2 GPU bottleneck, 1/2 CPU bottleneck, so it's even harder to tell which is the bottleneck, which CPU is faster.

The problem with real world testing is, how does it tell you that you are going to be able to game at the settings set by the reviewer? It doesn't. Real world testing creates far too many variables for any sort of comparisons. Proper benchmark testing can eliminate a lot of variables. Remember the fight between HardOCP and Procooling? Do you want intensive temperature testing, or do you want a testing with a lot of variabilities?

When I go to HardOCP's review, I get some fps numbers with tons of various AA and AF settings. So the only thing I can understand is, one card will have higher details settings, and that's supposed to be faster than the other one will lower details? Hmm...

[Max_87]

LOL!.... HardASSes......

Compare this http://enthusiast.hardocp.com/articl...50aHVzaWFzdA==

and this http://sg.vr-zone.com/?i=3338&s=8

HardOCP tells me that there isn't much difference between the performance of 7600GT and X1600XT, which is really misleading. (Like Chai said, there are a lot of variables, and at that settings, it is probably half CPU and half GPU bottleneck.)The fact is, 7600GT is a lot faster than X1600XT when the situation becomes very very GPU limited, let's say 90% GPU and 10% CPU or when CPU is not a bottleneck at all.

When I buy a GPU, what I wanted to know is the true potential of the GPU! Not the performance when it is both CPU and GPU limited! Would you buy a X1600XT when you know 7600GT is actually a lot faster when it is not bound by CPU limitation? They are both similarly priced.

You will never know that one day you might upgrade your CPU/mobo, which reduces the CPU bottleneck and the difference between the performance of the cards would become much more evident.

[Papercut]

Originally Posted by Chai

The purpose of low resolution test is not to show how well you can game at low res, but how well the CPU does if the GPU bottleneck is removed.

That's exactly what we are doing with our GPU reviews. I'm testing at 1600x1200 32xAA 16xAF not because we are able to game at that resolution, but to show how well it does when compare to other cards, or how badly the card does at higher details.

We all know that GPU is going to be bottleneck at high detail, while CPU will be bottleneck at low detail. Isn't that quite obvious already?

1280x1024 is probably going to be 1/2 GPU bottleneck, 1/2 CPU bottleneck, so it's even harder to tell which is the bottleneck, which CPU is faster.

The problem with real world testing is, how does it tell you that you are going to be able to game at the settings set by the reviewer? It doesn't. Real world testing creates far too many variables for any sort of comparisons. Proper benchmark testing can eliminate a lot of variables. Remember the fight between HardOCP and Procooling? Do you want intensive temperature testing, or do you want a testing with a lot of variabilities?

When I go to HardOCP's review, I get some fps numbers with tons of various AA and AF settings. So the only thing I can understand is, one card will have higher details settings, and that's supposed to be faster than the other one will lower details? Hmm...

Fair enough but 640x480 is like giving the best case scenario...at the other end you have 1600x1200 and whatever AA/AF for GPU testing, but I feel that those settings are closer to what some may actually use (maybe on a 24" moneytor?), whereas 640x480 merely gives you results on paper, so to speak.

Just my $0.02

[Chai]

Looks like you guys prefer "real world" usage than the true performance of the CPU. We can always include the 1600x1200 results, still as a benchmark comparison. But seriously, showing all the GPU limited results isn't going to interest me at all, since it only tells me that the results are GPU limited, nothing else.

Having the 640x480 as best scenario is exactly what I want to know. I'm testing at 640x480 not to show how well it does at gaming, but how well the CPU processes. I'm merely using these games as a gauge just like what you guys do with SuperPi. If you guys don't find the need to know 640x480 results, there's no need for CPU reviews already.

If you truly want to know the results of 1600x1200, then it should be in a separate article concentrating on CPU scaling only.

[Papercut]

Again, it's just my opinion
For the record, I'm not completely supportive of [H]'s methods either.
Kind of sitting on the fence