For years, Futuremark’s (the company formerly known as Mad Onion) 3DMark series has been considered the leading synthetic benchmark for graphic card performance.  Some of the benchmark?s success is due to marketing, with a dash of ease of use, a dollop of consistency, and an ounce of entertainment value added in for spice.  Discussions and arguments have swirled around the 3DMark series ever since 3DMark 2003, when ExtremeTech discovered NVIDIA had released drivers that flat-out cheated when rendering the benchmark.  The resulting flurry of PR?s between FutureMark and NVIDIA fooled no one, and the ?we?re best friends again? scenario that resulted did nothing to restore credibility to either side.  To date, FutureMark only allows certain ?approved? optimizations in its ?official? driver recommendations, which raises the murky issue between drivers and cheats all over again.

Lost in all the shouting, however, is a different (and perhaps more-important question):  Does the 3DMark series actually predict game performance?  Is it a reliable benchmark for measuring video cards in today?s games?   

Almost anyone whose considered buying a gaming video card has seen the 3DMark scores for it and its competitors.  Whether that’s going back to 3DMark 2000, or all the way up to the current 05 version, it’s pretty ubiquitous in online reviews.  But what do those numbers mean?  Do they relate to anything current, or hint at future performance?  Is it just a gimmick to sell ad space to Alienware or Falcon Northwest?

According to Futuremark themselves, 3Dmark is designed to stress the various parts of the most current DirectX version.  That means programmable pixel and vertex shaders, in accordance with the various shader models used by both nVidia and ATI.  While not currently using any specific game engine, or closely following any type of game that is popular at the moment (in other words, first person shooters), they create models and tests themselves that are used to approximate performance (at least in the "future".  The obvious problem we can see with this approach before even starting, is that the differences in who the graphics card makers associate with as far as games go (and the included optimizations) are not going to be included here.  After the flap from a few years ago about including driver optimizations (cheats is probably a better word in this particular case) by nVidia to increase performance in the test by decreasing the visual quality, drivers are now regulated to prevent any kind of foolery.  The problem is, we aren’t sure what kind of optimizations COULD be made, as anything that flags the benchmark  is no longer.  Games, on the other hand, are starting to have many of these types of things, especially with advances, such as SLI, which are based on optimizing the drivers for specific game engines. We?ve also seen 3D manufacturers explicitly connecting themselves with specific game engines (think ATI + Half Life 2, or Doom 3 + NVIDIA).

This causes the most problems for a 3rd party benchmark like 3DMark.  After all, how do you go about including a GPU maker in your process of creating the benchmark without annoying either of them for not optimizing enough for their technology, or keeping the press and public from roasting you over hot coals for those same optimizations and showing either one a preference? With a game, those same questions aren’t quite as present, as a user of that specific card will at least enjoy a good gaming experience.  A benchmark, after all, is just that.  Other than those in the review community, people obviously aren’t running a benchmark more than games themselves.

next >

---

• 3DMark 05 Design, Actual Performance Scores. (1 of 5)
• Half Life 2 Results Compared. (2 of 5)
• Far Cry Results Compared. (3 of 5)
• Doom 3 Performance Comparison. (4 of 5)
• Conclusion (5 of 5)