Supreme Commander Benchmarking Guide

Discussion in 'Reviews & Articles' started by Dashken, Jan 10, 2008.

  1. Dashken

    Dashken Administrator!

    Supreme Commander is a great benchmark for CPUs. Although it is a CPU-limited game, is also a good benchmark for graphics cards, as it shows just how much effect the graphics card has on CPU-limited games.

    As we did with our Crysis Benchmarking Guide, we will be taking a look at how you can benchmark your CPU and/or graphics card in Supreme Commander. We also throw in some Supreme Commander benchmark results at the end.

    Here's a quote from the guide:-
    [​IMG]

    Link : Supreme Commander Benchmarking Guide
     
  2. goldfries

    goldfries www.goldfries.com

    nice guide but i find it incomplete.

    i browse page 1 to 3 to 1 to 3 and to 1 again, i don't see which area mentions PC specs used in the benchies.

    and based on the screenies, it looks like it's on Windows Vista. So I think it's also good if you highlight that the framerate would vary (should be higher) on Windows XP.

    same thought applies to the Crysis guide.
     
  3. Adrian Wong

    Adrian Wong Da Boss Staff Member

    Yeah, forgot to add that in. Will be doing so in the next revision. The Crysis guide doesn't have any benchmarks yet though... although that's slated for the next revision as well.
     
  4. Dashken

    Dashken Administrator!

    The guide has just been updated! :wave:

    Supreme Commander is a great benchmark for CPUs. Although it is a CPU-limited game, is also a good benchmark for graphics cards, as it shows just how much effect the graphics card has on CPU-limited games.

    As we did with our Crysis Benchmarking Guide, we will be taking a look at how you can benchmark your CPU and/or graphics card in Supreme Commander. We also throw in some Supreme Commander benchmark results at the end.

    Here's a quote from the guide:-
    [​IMG]

    Link : Supreme Commander Benchmarking Guide Rev. 2.0
     
  5. moshpit

    moshpit Newbie

    I hate benchmarks based on the built in test. For one, it doesn't nearly test the CPU as hard as the game engine can, and doesn't show the CPU scaling this game really can show. For one, only 4 AI's used in the built-in benchmark don't stress 4 cores enough to show them off, and the built-in benchmark also uses a medium sized map instead of a proper 80x80 map that will work the CPU even harder.

    The results of built-in testing only tell a player what a medium load from the game will show. That's inaccurate. The game WILL show a marked difference using quad core on a 7 AI map with 80x80 size, over dual core. Additionally, nearly 2 hours of game time are needed for maximum number of units for each AI and the player to be reached. If you expect the built-in benchmark to tell you how well your CPU can handle 7 AI's on an 80x80 map, it won't and you'll be disappointed by the end result of this scenario even though the built-in benchmark gave you a nice score on a dual core.

    Supreme Commander is a GREAT CPU benchmark, but not the way it's shown in this article. Until somebody gets this point, every SupCom benchmark I see online is considered inaccurate and untrustworthy. That's not to say the author is in any way TRYING to pull anything, I know that's not the case at all. I'm simply saying the lack of experience with what the SupCom engine can really do to differentiate a quad over a dual isn't doing the article written any justice.

    The method I outlined though is VERY time consuming. And difficult to accomplish, 7 AI's on an 80x80 map is insanely difficult for most players to even survive the necessary 2 hours to achieve maximum load. A savegame of this scenario made once is almost a must. That way it can be loaded up on any machine, started at the 2 hour point the save was made at, and then observed for a 5 minute fraps run, would be 10x more accurate, both on CPU performance, memory amount and performance.

    Please don't take this insultingly, it's NOT intended that way at all. It's simply a matter of experience with this game that you find the built-in benchmark not giving an accurate picture of the games highest loads on the CPU.
     
  6. Adrian Wong

    Adrian Wong Da Boss Staff Member

    LOL! There's no need to inject apologies into every paragraph! It's okay to disagree as long as it's constructive. :D

    Okay, first off, this is really just a guide on how to benchmark Supreme Commander. Like the Crysis benchmark, it does not tell the whole story. If you take a look at the Crysis benchmark, it may record a frame rate of say 30 fps but in the actual game, it's really something like 25 fps.

    This is exactly why we no longer consider 30 fps as the minimum acceptable frame rate. In fact, if you check our reviews, we specifically recommend a minimum frame rate of 40 fps.

    Like it or not, built-in benchmarks make it much easier for hardware testers to produce consistent, repeatable results. Without them, we would have to rely on third-party apps like Fraps, which also affect the results.

    The best kind of benchmark would be a time demo, like what we used to test in Doom 3. They are more representative of actual gameplay as they include enemies and actual firefights. Unfortunately, most games do not support time demos.

    With that said, the SupCom benchmark is representative of single-player campaign gameplay although not multiplayer gameplay. As such, the benchmark is still useful. It is not inaccurate or untrustworthy. Just because there can be even more AI enemies and far more units on a much larger map, that does not mean that it is always that case.

    You will note that even in our Intel Core 2 Processors Performance Comparison Guide, we did not intentionally reduce the graphics settings to increase the effect of the CPU on the game. Our purpose is to deliver a more realistic evaluation of the effect of the CPU on a game like Supreme Commander.

    So, I personally prefer not to modify the settings of any game merely to demonstrate a skewed effect of the processor or graphics card on actual game play. It's like setting the graphics settings low and physics low when testing the CPU, and then ramping the graphics settings up high and the physics low when testing the graphics card. That's not how people play games, and that's why we don't test it that way.
     
  7. moshpit

    moshpit Newbie

    I can understand your points. I guess the point I was making was that as a CPU test, SupCom has the potential to be one of the best modern games in several years to test with, but by default doesn't push nearly as hard as it can. A savegame based on the scenario I described would be highly beneficial for determining the actual peak usage the game can generate.

    So if the purpose of the test is only to see how a basic game of SupCom will run on a set of hardware, then your outlined methods are perfect. But all too often I've seen people take that internal benchmark as proof SupCom doesn't make good use of memory or CPU resources (the result being the false conclusion that dual core is as useful as quad in this game). This limited testing leaves readers with the belief that they can see nearly maximum performance with a dual core as easily as they would with a quad. And if all they ever play is medium maps, that conclusion will be correct.

    But maps like Earth Conquest/World Domination will quickly cause that reader to have to rethink that conclusion, and these maps are VERY popular in online play nowadays. It would be more accurate to define the limit point between duals and quads in this game, though nobody has done any such testing in any reviews that I've seen. This ends up leaving most readers in the dark about how far they actually CAN go with this game.

    Thank you for your answers though, I understand your methodology much better now and appreciate the time you took to answer. Perhaps an additional article later down the road about this subject could be considered :D
     
    1 person likes this.
  8. Adrian Wong

    Adrian Wong Da Boss Staff Member

    Yup, I do agree with that. The current test more accurately reflects the single-player mode. It isn't easy to test the other maps because SupCom does not allow us to choose a different map for the benchmark. Perhaps, it would be good if they could have included two different tests.

    In any case, we generally prefer not to rely on a single benchmark but rather provide a range of different benchmarks to show the effect of the CPU (or other components) on the performance of a game or application. That should give us a better idea of how much of a performance boost we can expect from a CPU in real world applications.

    No problem! I really appreciate your feedback. We strive to keep improving our articles, which is why you see us revising most of them over time. We are not perfect and it's good for readers like you to help us improve along the way. +rep! :beer:
     
  9. SamODJ

    SamODJ Newbie

    Just been giving this guide a going through, testing my own system (A new system). Much to my dismay however, my results really aren't as high as I was expecting.

    I averaged out at 36.339, on my q6600, ATI 4870, 4GB RAM system. That surprises me, as simply looking at firstly 40.7 average fps for the q6600 at 1600x1200. Secondly looking at the card high of 42.8 (The 8800 GTS 512mb), a card weaker than my own.

    I'm stuck. My knowledge of pc workings is growing, but still limited, so any help/advice you could give would be much appreciated.

    SamODJ
     
  10. Chai

    Chai Administrator Staff Member

    You should never compare system with different setups.

    There are many factors, like different OS, different image quality settings.
     
  11. SamODJ

    SamODJ Newbie

    True enough, but guides like this and others do often seem to be advising you as to what you should perhaps come to expect from your machine, at least thats what the graphcs and such come across as.

    Eitherway, any ideas?
     
  12. Adrian Wong

    Adrian Wong Da Boss Staff Member

    You must understand that different systems perform differently from other systems because game performance is NOT only linked to the performance of the graphics cards. There are other factors, like CPU performance, memory size and throughput, and even whether you have fully patched up your games or not.

    This is why we post the full details of our testbed, listing not only its specifications but also the details of the game versions and settings. Your results are only DIRECTLY comparable if you use the same testbed and settings. Even then, you can always count on a certain small percentage of variation in the results.

    Do note that Supreme Commander is more reliant on CPU performance than the graphics card. So your system is more CPU-limited than our testbed which uses the Intel Core 2 Extreme QX9650 processor. For a comparison between the two processors, please take a look here - Tech ARP - Intel Core 2 Processor Performance Comparison Guide Rev. 2.6

    You will see that there's a marked difference in their performance even using the same graphics card.

    If you actually test and compare your Radeon HD 4870 against the GeForce 8800 GTS 512MB, you will see that the Radeon HD 4870 is actually faster. Much faster. For the results on OUR testbed, please take a look at this review - Tech ARP - ASUS EAH4870 Radeon HD 4870 Graphics Card Review

    Hope that helps you some! :D
     
  13. SamODJ

    SamODJ Newbie

    Yes very helpful. Just to note, when I said graphics in my previous post, I was referring the the graphs and charts rather than GPU.

    I will still pursue this issue though. I'm hoping to get the best out of my new system, although it could simply be a case of "new PC" syndome. ;)

    One thing I will note however, in your processor test your Q6600 results end up in just over an average of 40 FPS. Again, I respect I do not have the testbed system, however, given my system having the increased speed of the 4870 including more RAM i would've expected at least a similar average of 40 on my own system. (1600x1200) I find its a case of drawing the line when comparing systems, between my own and the tested one. Could just be my knowledge failing me. ;)
     
  14. Chai

    Chai Administrator Staff Member

    Even if you have 4870X2 Crossfire, if you have a slow processor, the game will still have to wait for the processor. That is why you have to ensure that the system is balanced, fast processor, fast graphics card.

    Same games are more processor limited, some games are more graphics card limited.
     
  15. goldfries

    goldfries www.goldfries.com

    The only time that SupComm relies on GC is when you're zoomed into the scene.

    On the huge overview, any card works the same. I play SupComm often, especially Forged Alliance.

    One thing to note that processor power affects a lot on your gaming experience, especially if the game involves the AIX.

    The game itself slowdown significantly faster when you're against just an AIX or 2 compared to like say 4 or 6 AI.
     

Share This Page