[H]ard Journalism

By Cyrris

Since the launch of Intel’s impressive new Core 2 line of CPUs, I’ve read a flurry of reviews to get a good idea of the performance increase people are likely to see. Make no mistake - Intel is back in the game, and at least for the time being, AMD can only resort to huge price cuts to keep up. What caught my eye recently though is the online stoush between two hardware review sites which I frequent. On the left we have [H]ard|OCP, a hardware enthusiast website which is pretty well known and well regarded. On the right we have FiringSquad, a gamer-centric site which focuses heavily on both hardware and the games you play on it. Also well known, also well regarded. This isn’t the first time I’ve seen two hardware sites at each others throats - the last time I recall was between Tom’s Hardware and the now defunct Amdmb.com. That got pretty ugly, and frankly isn’t the kind of thing I like to be reading about when I go to a hardware review site. This time however, I am a little more interested in the subject at the centre of the debate.

Let’s get started with a quote from Kyle Bennet in HardOCP’s article on Core 2 gaming performance:

You will see a lot of gaming benchmarks today that just simply lie to you. That is right, you will see frames per second numbers that are at best total BS, and at their worst a terrible representation of what difference a new Intel Core 2 processor will make in your gaming experience. The old ways of video game benchmarking do little to tell you about exactly how a new CPU will affect how you play your games or what experience your system supplies to you. Having more CPU power is a very cool thing, but being able to utilize it is not an easy thing to do nowadays.

The article then goes on to desrcibe HardOCP’s method of how it evaluates gaming performance with the processor. The phrase “real world gaming” was employed several times - [H] likes to test their games at the settings they think will be represntative of how people actually like to play. That is (of course), eye candy on, res high up. The idea is that Kyle and the [H] team believes that canned benchmarks - things such as timedemos being run at different resolutions on the different hardware, are no longer relevent. Apparently they don’t tell us how well the games will run on our systems. Of course it makes sense if you think about it for a second: a brand new CPU won’t help you if all your games are limited by your video card. And a new video card might not help much if you are installing it on your old AthlonXP system which doesn’t even have DDR RAM.

Step in, FiringSquad. [H]’s introduction left them feeling a bit upset, and Jakub decided to make a response detailing why he thought Kyle was out of line in calling his canned benchmarks a terrible BS lie. It’s quite a long read, but it did make some good points on why HardOCPs own testing methods aren’t exactly top notch either. For the sake of brevity, I will outline the main points here:

• People want to know how the CPU itself performs in gaming applications, so isolating its performance from other variables (such as the video card) is necessary
• “Real World gaming” is so different for everyone, that it is ludicrous for HardOCP to narrow it down to just a couple of tests and come out with a definitive answer
• Showing a CPU review which has the video card as the limiting factor in every test case does not provide any useful information to the reader
• Demonstrating that the CPU is video card limited in all current games does not help those who may want the CPU now, and want headroom for a new video card later, which may not be so limited. It also fails to take SLI/CrossFire into account

Now, all of these are fair points. Fair enough in fact for Kyle to make an article in response. In it, he explains how [H] now seems to transcend benchmarking in games. They’re not going for the raw numbers, they’re going for the gamer experience. HardOCP has other reviews to determine Core 2’s performance in situations such as movies, music and the like. The article in question though, focused solely on gaming - which according to them is something Core 2 is currently not beneficial for, over the current crop of high-end CPUs. This quote generally sums it up:

So what exactly is HardOCP doing? We test using real gaming scenarios like you would find yourself in at home (when we are testing games) and give our readers feedback on our experiences with the hardware. We tell our readers what their expectation of their computer experience should be based on our own experiences. Firing Squad gives you pages of canned benchmarks (Although they are following suit on encoding) without any or very little analysis, then declares a winner.

He then goes on to deride FiringSquad further, in what is a rare display of stupidity which I have to say I didn’t expect to see coming from Kyle, whose statements I have usually found myself agreeing with.

So, now we have all the background information, why did I bother presenting all this? Well, I think both sides are a bit off. Well, OK - I think Kyle is way off. The canned benchmark is indeed not as indicative of game play as it once was, but when I look at the Core 2 gaming reviews on both HardOCP and FiringSquad, it is FiringSquads which I find to actually have some merit to it.

Let’s go over HardOCP’s review: It outlines the test system. Only 3 CPUs are used, one AthlonFX and two Core 2 chips. All are run on a high-end 7900GTX video card. Two RPGs, three FPSs and a racing game are all run on high detail. In each and every case, as any hardware enthusiast would already guess, the video card was the limiting factor and there was virtually no difference shown. Except for Oblivion, all games were run at 1600×1200. All had either AA or AF or both running, and the minimum, max and average FPS rates were summarised with a subjective analysis following.

Well, gosh, that’s great. What did I learn from this review? Yep - nothing. It was completely worthless to me in every possible way. Being a hardware enthusiast (which I think is HardOCP’s target audience… just a guess), I already know that running recent games on high detail, in a high res, is going to make the situation GPU-dependent. This is not something I needed to waste time viewing over the course of an 11 page review, thanks Kyle.

I also can’t run anything in 1600×1200 - as my LCD’s native res is only 1280×1024. Sure - your apples to apples comparison at the very end demonstrated that res for the games, and on medium quality too - so why did you only include 3 graphs on one page at the end for this extremely important aspect of the review? And why are you only comparing 3 high-end CPUs? Hell, it would have been nice if you at least included Intel’s previous Pentium EE flagship, and maybe a regular Athlon 64. Not everyone has an FX-62 and not everyone wants one.

And what about if I am playing a singleplayer game of Battlefield 2? I have a fair few bots and it’s a bit of a strain on my CPU. I would love to know how much better the Core 2 is at running that. Oh, and Civilization 4 as well - bigger maps have some pretty long gaps in between turns which I’m sure would be a lot shorter with a Core 2 processor. And don’t forget CnC Generals! It’s an oldie but it’s shoddy AI has such poor pathfinding code that big maps with enough units slow to a crawl. It’s not really GPU bound even with all details maxed out. I still play that game, and I’d love to know if Core 2 would make it run better.

And [H]… your music/images/movies performance review suffers as well. I would like to see the difference there between the Core 2’s and the old Pentiums, which were always good at that stuff. I’d like to see the gap in performance contrasted between the two chips, both in gaming and in multimedia - as we know the Pentiums suck at the former but are good at the latter, while Core 2 is good at both. So where is the review for Pentium owners looking to upgrade?

FiringSquad’s review on the other hand shows me a fair bit of useful information. It’s supposed “total BS” leaves no doubt that when the video card is taken out of the equation, the Core 2 is the supreme processor for gaming. Now of course I don’t game at 800×600 but it does give a good indication of which processor to go for if you plan on getting Nvidia’s G80 or ATI’s R600 when they first come out - and take all those games that were tested in their stride. Or will you also re-review the Core 2 then, to let me know, Kyle?

FS’s review also shows how the single-core FX-57 and the Pentium EE compares to the Core 2’s and the FX-62, instantly making it more comprehensive and useful than both your gaming and your multimedia reviews combined. It also includes CrossFire results which do begin to show the Core 2 pulling a substantial lead over the other CPUs even at high res - why didn’t you try that, [H]?

Make no mistake - FiringSquad’s reviews are not perfect. I would like them to show or at least mention the minimum and maximum framerates, not just the averages. I would also like 1280×1024 to always be in the testing suite because there are so many users with 17″ and 19″ LCDs who have that as their native res. What I would also like to see though, is perhaps some older CPUs being used in reviews such as this - so people with older machines can see the benefits they would get if they decided to do an upgrade. Afterall, if I have a high-end FX-62 already, I am not as much in need of an upgrade as I am with an older 3000+. Or a Pentium, as I mentioned before.

However, the information I am able to extrapolate from FiringSquad’s benchmarks with a larger variety of different CPUs, settings, and CrossFire results, makes for a more interesting, useful and valuable review. I don’t need a 2-3 paragraph long subjective analysis at the bottom of each page when really, it doesn’t tell me anything more than what the graphs are saying already. If there were issues that the graphs didn’t necessarily portray, I want to hear about it, but otherwise, FiringSquad is not doing anything wrong by letting the graph results speak for themselves. I can make my own decisions from that, thanks.

I should also note, that FiringSquad is actually demonstrating some initiative to improve their reviews even further: adding some strategy games to the benchmarks is something I would find quite valuable - I always appreciated it when Anandtech did that.

Now, all up, I can see where HardOCP is coming from in a way. Kyle stated their philosophy quite concisely:

We looked at the canned benchmarks we were claiming to our readers to be a good evaluation tool and decided that the industry was moving beyond these simple tools. Were canned benchmarks really telling our readers what we thought they needed to know? No, they were not, so we decided to change the ways we were doing things.

So this begs the question: What do you think we need to know? Ah, here we are:

I wanted to know what Core 2 meant to many of our readers that already own high end Athlon 64 gaming machines. As our Core 2 Gaming Performance spells out, there is not much of a reason for many current gamers to upgrade.

What about what Core 2 means to everyone who doesn’t own an Athlon FX-62? Somehow I think you have more readers without one of those, than with. This is a very narrow perspective to be making a CPU review from, and instantly limits it’s usefulness to the majority of readers.

That fact is, HardOCP’s gaming review gives no useful information to the vast majority of gamers. It was not broad enough, both in terms of hardware tested, the types of games tested, and the number of different settings tested. To publish this half-assed review whilst firing off at other sites and calling their material useless is ironic to say the least.