Ask Immensely: Reviewing and re-reviewing MMOs

Back in September, Star Trek Online Producer Daniel Stahl gave an interview that proposed to tell game journalists how to do their jobs. That's only fair; we're always telling developers how to do their jobs, right? Stahl told [a]listdaily,
 "The whole game rating business doesn't necessarily do a great justice to MMOs. MMOs are designed to grow over time and get better with every major release. It might be better if sites like Metacritic could find a way to rate MMOs by releases instead of just the initial day one . . . There are plenty of MMOs that have made huge strides since day one and some that have even gotten worse. Until then, we will continue to offer the game for free and ask for people to try it out and decide for themselves."

Quipped Massively reader Matthew12, "If only there were MMO gaming blogs and websites that keep up to date with the MMOs and their updates... oh wait; there are."

For as long as I've worked at Massively, the site has eschewed formal reviews and ratings. It's not a top-down policy; it's just the way we've always done things and the way that's always made the most sense to us.

Ratings, for example, bring with them a whole slew of problems. How do you assign a number to an entertainment form that can take months or even years to fully grasp and complete? We're not talking about a two-hour movie or a 20-hour single-player title that really has but one path to the finish. How do you overcome our natural inclination to misconstrue what the numbers or stars or whatevers mean? Is a 5/5 an A or an A+? Is a 5/10 an average C or a failing F? How do you avoid number-creep and the desire to compare one title to another and push the rating up just a tad until everything is a 99/100?

But Stahl is right in saying that reviews themselves are a problem, even when they don't come with a handy score you can plug into Metacritic. MMOs change over time. We get angry when they don't change and improve over time. Heck, many of my favorite gw2 gold games were pretty wretched at launch and needed a few more years to bake before I fell in love with them. Nevermind the fact that no one person can legitimately "review" every imaginable launch feature in an MMO in the timely manner expected by readers!

So traditionally, Massively has stuck to impressions and re-impressions pieces. We stream older games; we cover older games in Rise and Shiny and The Game Archaeologist and even in Choose My Adventure on occasion; we cover our favorite long-running MMOs in dedicated game columns; and we check in on the most popular MMOs for their birthdays and anniversaries. We're even working on a new column specifically to revisit games that have fallen off our collective radar. Once you realize that so much of our news is about games that are already out in the wild, you have to admit that a site like Massively is primarily focused on updating its readership about existing games and their patches and expansions. Even if we don't do formal reviews or hand out ratings, we are already exactly what Stahl is proposing -- just not necessarily for the reasons he proposed it.

Commenter JamesKephart countered Stahl's complaint with a really good reason not to re-review old MMOs:

 "I really have a problem with the concept of giving MMOs a free pass to officially release a substandard title and give them time to fix it. Granted, an MMO is more complex than your latest generic brown military shooter, but if some AAA studio released a game that was half as buggy as some MMO launches, they would get poor reviews and that would be that. For games like STO to ask for the system to be changed just for them is to say that they deserve special treatment. MMOs can change over a long time, but so can all the single-player titles. With DLC and free patches, many titles have changed notably in their lifetime. Essentially, if you can't put together a decent enough gw2 gold game to grab people initially, then you really don't deserve a free second review when you put a new coat of paint on your outhouse."

There's also the tl;dr version by lucidrenegade: "If you wanted great reviews, you should have shipped a product that earned them." That might not be realistic, as reader Porculasalvania argued, as "sometimes you have to launch with what you have before you run out of money," but I think James and lucid are on to something.