Supreme Commander Benchmarking Guide

[Dashken]

Supreme Commander is a great benchmark for CPUs. Although it is a CPU-limited game, is also a good benchmark for graphics cards, as it shows just how much effect the graphics card has on CPU-limited games.

As we did with our Crysis Benchmarking Guide, we will be taking a look at how you can benchmark your CPU and/or graphics card in Supreme Commander. We also throw in some Supreme Commander benchmark results at the end.

Here's a quote from the guide:-

"Benchmarking using Supreme Commander is very easy. The creator, Gas Powered Games, has conveniently included a built-in benchmark that allows you to run a time demo with four opponents battling it out. At the end of the time demo, you will be given a SupComMark composite score as well as a large bunch of benchmark data, the most important of which is the average frame rate."

Link : Supreme Commander Benchmarking Guide

[goldfries]

nice guide but i find it incomplete.

i browse page 1 to 3 to 1 to 3 and to 1 again, i don't see which area mentions PC specs used in the benchies.

and based on the screenies, it looks like it's on Windows Vista. So I think it's also good if you highlight that the framerate would vary (should be higher) on Windows XP.

same thought applies to the Crysis guide.

[Adrian Wong]

Originally Posted by goldfries

nice guide but i find it incomplete.

i browse page 1 to 3 to 1 to 3 and to 1 again, i don't see which area mentions PC specs used in the benchies.

and based on the screenies, it looks like it's on Windows Vista. So I think it's also good if you highlight that the framerate would vary (should be higher) on Windows XP.

same thought applies to the Crysis guide.

Yeah, forgot to add that in. Will be doing so in the next revision. The Crysis guide doesn't have any benchmarks yet though... although that's slated for the next revision as well.

[Dashken]

The guide has just been updated!

Supreme Commander is a great benchmark for CPUs. Although it is a CPU-limited game, is also a good benchmark for graphics cards, as it shows just how much effect the graphics card has on CPU-limited games.

As we did with our Crysis Benchmarking Guide, we will be taking a look at how you can benchmark your CPU and/or graphics card in Supreme Commander. We also throw in some Supreme Commander benchmark results at the end.

Here's a quote from the guide:-

"Benchmarking using Supreme Commander is very easy. The creator, Gas Powered Games, has conveniently included a built-in benchmark that allows you to run a time demo with four opponents battling it out. At the end of the time demo, you will be given a SupComMark composite score as well as a large bunch of benchmark data, the most important of which is the average frame rate."

Link : Supreme Commander Benchmarking Guide Rev. 2.0

[moshpit]

I hate benchmarks based on the built in test. For one, it doesn't nearly test the CPU as hard as the game engine can, and doesn't show the CPU scaling this game really can show. For one, only 4 AI's used in the built-in benchmark don't stress 4 cores enough to show them off, and the built-in benchmark also uses a medium sized map instead of a proper 80x80 map that will work the CPU even harder.

The results of built-in testing only tell a player what a medium load from the game will show. That's inaccurate. The game WILL show a marked difference using quad core on a 7 AI map with 80x80 size, over dual core. Additionally, nearly 2 hours of game time are needed for maximum number of units for each AI and the player to be reached. If you expect the built-in benchmark to tell you how well your CPU can handle 7 AI's on an 80x80 map, it won't and you'll be disappointed by the end result of this scenario even though the built-in benchmark gave you a nice score on a dual core.

Supreme Commander is a GREAT CPU benchmark, but not the way it's shown in this article. Until somebody gets this point, every SupCom benchmark I see online is considered inaccurate and untrustworthy. That's not to say the author is in any way TRYING to pull anything, I know that's not the case at all. I'm simply saying the lack of experience with what the SupCom engine can really do to differentiate a quad over a dual isn't doing the article written any justice.

The method I outlined though is VERY time consuming. And difficult to accomplish, 7 AI's on an 80x80 map is insanely difficult for most players to even survive the necessary 2 hours to achieve maximum load. A savegame of this scenario made once is almost a must. That way it can be loaded up on any machine, started at the 2 hour point the save was made at, and then observed for a 5 minute fraps run, would be 10x more accurate, both on CPU performance, memory amount and performance.

Please don't take this insultingly, it's NOT intended that way at all. It's simply a matter of experience with this game that you find the built-in benchmark not giving an accurate picture of the games highest loads on the CPU.

[Adrian Wong]

Originally Posted by moshpit

I hate benchmarks based on the built in test. For one, it doesn't nearly test the CPU as hard as the game engine can, and doesn't show the CPU scaling this game really can show. For one, only 4 AI's used in the built-in benchmark don't stress 4 cores enough to show them off, and the built-in benchmark also uses a medium sized map instead of a proper 80x80 map that will work the CPU even harder.

The results of built-in testing only tell a player what a medium load from the game will show. That's inaccurate. The game WILL show a marked difference using quad core on a 7 AI map with 80x80 size, over dual core. Additionally, nearly 2 hours of game time are needed for maximum number of units for each AI and the player to be reached. If you expect the built-in benchmark to tell you how well your CPU can handle 7 AI's on an 80x80 map, it won't and you'll be disappointed by the end result of this scenario even though the built-in benchmark gave you a nice score on a dual core.

Supreme Commander is a GREAT CPU benchmark, but not the way it's shown in this article. Until somebody gets this point, every SupCom benchmark I see online is considered inaccurate and untrustworthy. That's not to say the author is in any way TRYING to pull anything, I know that's not the case at all. I'm simply saying the lack of experience with what the SupCom engine can really do to differentiate a quad over a dual isn't doing the article written any justice.

The method I outlined though is VERY time consuming. And difficult to accomplish, 7 AI's on an 80x80 map is insanely difficult for most players to even survive the necessary 2 hours to achieve maximum load. A savegame of this scenario made once is almost a must. That way it can be loaded up on any machine, started at the 2 hour point the save was made at, and then observed for a 5 minute fraps run, would be 10x more accurate, both on CPU performance, memory amount and performance.

Please don't take this insultingly, it's NOT intended that way at all. It's simply a matter of experience with this game that you find the built-in benchmark not giving an accurate picture of the games highest loads on the CPU.

LOL! There's no need to inject apologies into every paragraph! It's okay to disagree as long as it's constructive.

Okay, first off, this is really just a guide on how to benchmark Supreme Commander. Like the Crysis benchmark, it does not tell the whole story. If you take a look at the Crysis benchmark, it may record a frame rate of say 30 fps but in the actual game, it's really something like 25 fps.

This is exactly why we no longer consider 30 fps as the minimum acceptable frame rate. In fact, if you check our reviews, we specifically recommend a minimum frame rate of 40 fps.

Like it or not, built-in benchmarks make it much easier for hardware testers to produce consistent, repeatable results. Without them, we would have to rely on third-party apps like Fraps, which also affect the results.

The best kind of benchmark would be a time demo, like what we used to test in Doom 3. They are more representative of actual gameplay as they include enemies and actual firefights. Unfortunately, most games do not support time demos.

With that said, the SupCom benchmark is representative of single-player campaign gameplay although not multiplayer gameplay. As such, the benchmark is still useful. It is not inaccurate or untrustworthy. Just because there can be even more AI enemies and far more units on a much larger map, that does not mean that it is always that case.

You will note that even in our Intel Core 2 Processors Performance Comparison Guide, we did not intentionally reduce the graphics settings to increase the effect of the CPU on the game. Our purpose is to deliver a more realistic evaluation of the effect of the CPU on a game like Supreme Commander.

So, I personally prefer not to modify the settings of any game merely to demonstrate a skewed effect of the processor or graphics card on actual game play. It's like setting the graphics settings low and physics low when testing the CPU, and then ramping the graphics settings up high and the physics low when testing the graphics card. That's not how people play games, and that's why we don't test it that way.

[moshpit]

I can understand your points. I guess the point I was making was that as a CPU test, SupCom has the potential to be one of the best modern games in several years to test with, but by default doesn't push nearly as hard as it can. A savegame based on the scenario I described would be highly beneficial for determining the actual peak usage the game can generate.

So if the purpose of the test is only to see how a basic game of SupCom will run on a set of hardware, then your outlined methods are perfect. But all too often I've seen people take that internal benchmark as proof SupCom doesn't make good use of memory or CPU resources (the result being the false conclusion that dual core is as useful as quad in this game). This limited testing leaves readers with the belief that they can see nearly maximum performance with a dual core as easily as they would with a quad. And if all they ever play is medium maps, that conclusion will be correct.

But maps like Earth Conquest/World Domination will quickly cause that reader to have to rethink that conclusion, and these maps are VERY popular in online play nowadays. It would be more accurate to define the limit point between duals and quads in this game, though nobody has done any such testing in any reviews that I've seen. This ends up leaving most readers in the dark about how far they actually CAN go with this game.

Thank you for your answers though, I understand your methodology much better now and appreciate the time you took to answer. Perhaps an additional article later down the road about this subject could be considered

[Adrian Wong]

Originally Posted by moshpit

I can understand your points. I guess the point I was making was that as a CPU test, SupCom has the potential to be one of the best modern games in several years to test with, but by default doesn't push nearly as hard as it can. A savegame based on the scenario I described would be highly beneficial for determining the actual peak usage the game can generate.

So if the purpose of the test is only to see how a basic game of SupCom will run on a set of hardware, then your outlined methods are perfect. But all too often I've seen people take that internal benchmark as proof SupCom doesn't make good use of memory or CPU resources (the result being the false conclusion that dual core is as useful as quad in this game). This limited testing leaves readers with the belief that they can see nearly maximum performance with a dual core as easily as they would with a quad. And if all they ever play is medium maps, that conclusion will be correct.

But maps like Earth Conquest/World Domination will quickly cause that reader to have to rethink that conclusion, and these maps are VERY popular in online play nowadays. It would be more accurate to define the limit point between duals and quads in this game, though nobody has done any such testing in any reviews that I've seen. This ends up leaving most readers in the dark about how far they actually CAN go with this game.

Thank you for your answers though, I understand your methodology much better now and appreciate the time you took to answer. Perhaps an additional article later down the road about this subject could be considered

Yup, I do agree with that. The current test more accurately reflects the single-player mode. It isn't easy to test the other maps because SupCom does not allow us to choose a different map for the benchmark. Perhaps, it would be good if they could have included two different tests.

In any case, we generally prefer not to rely on a single benchmark but rather provide a range of different benchmarks to show the effect of the CPU (or other components) on the performance of a game or application. That should give us a better idea of how much of a performance boost we can expect from a CPU in real world applications.

No problem! I really appreciate your feedback. We strive to keep improving our articles, which is why you see us revising most of them over time. We are not perfect and it's good for readers like you to help us improve along the way. +rep!

[SamODJ]

Just been giving this guide a going through, testing my own system (A new system). Much to my dismay however, my results really aren't as high as I was expecting.

I averaged out at 36.339, on my q6600, ATI 4870, 4GB RAM system. That surprises me, as simply looking at firstly 40.7 average fps for the q6600 at 1600x1200. Secondly looking at the card high of 42.8 (The 8800 GTS 512mb), a card weaker than my own.

I'm stuck. My knowledge of pc workings is growing, but still limited, so any help/advice you could give would be much appreciated.

SamODJ

[Chai]

You should never compare system with different setups.

There are many factors, like different OS, different image quality settings.