CW's Guide to RPG Reviewing
Let's start off by considering what a score actually means. It has always been my opinion that the number assigned to a review is far less important than what the review has to say about a game. One person's perfect score is another's average product, and some folk who will pass up a game because of an average score might be missing out on something they would love. Unfortunately, years of experience has shown me that publishers and fans are often very concerned with scores. With that in mind, let's consider how a reviewer should generally approach scoring.
There are two major ways to approach review scoring. The more common method is to use an above average average. What do I mean by this? Many reviewers assign scores to games that result in their average score being quite a bit higher than the average for the numerical system used. For example, on RPGnet's dual 5 point scale some reviewers will maintain an average of above 4 while a true average for the score number is 3. Of course, this could be a simple result of the reviewer carefully choosing what they review so that they only review products they already have an interest in. Even then, though, few reviewers will assign as many minimum scores as they will maximum scores. Compare the frequency of 5/5 scores at RPGnet with 1/1s and I'm sure you'll have an idea of what I mean here.
There's nothing wrong with a reviewer taking this approach so long as they're aware of what they're doing. Higher scores on reviews tend to invoke less criticism, which for shy reviewers can be a good thing as they need to defend the review a little less. Publishers are also more likely to prefer a reviewer who tends to publish higher scores. While the publisher won't pay any mind to the reviewer's average, they will note the scores given to their products. On the down side, it's more difficult to build a reputation as being a very critical reviewer when scores tend to be consistently high.
The other approach to scoring, and the one I attempt to take, is to try and shoot for an average closer to true average. Actually hitting true average would be artificially generated, as the reviewer is likely to be requesting material that they already have an interest in or are otherwise disposed to like. Scoring this way becomes more a matter of being willing to assign lower scores to products that fail the reviewer's evaluation criteria. Questions such as “Should a minimum score be as common as a maximum score?” come up, and it can involve some involved consideration for those reviewers really concerned about the fairness of their scoring.
The big advantage to this technique is that readers are more likely to view the reviewer as critical in a positive way. I, personally, have received many messages of thanks from readers for a tough, low scoring review of a product over the years. However, there are plenty of disadvantages. Some publishers will become quite unhappy over even a single product receiving a low score, especially when the reviewer doesn't have a large base of reviews. Fans are easily the biggest problem here, though, as nothing is as un-fun for reviewing as irate fans who want to express their disagreement in as nasty and personal a way as possible. How dare the reviewer say that one of a dozen products from a game line is merely an average book! On the whole I still prefer this method, and while it causes me a little more trouble I have some pride in trying to evaluate products with a critical eye.
Scores will mean different things to different people. For some a max score means that the product is flawless. These reviewers will reserve a max score for only the rarest, most exceptional products. Others view a max score as a great product. It can have flaws, but on the whole it's very good and recommended. I have never found a product that had no flaws at all in my reviewing, but I've certainly encountered many that were exceptional in a variety of ways. As with the note above on how critical a reviewer might be with scores, keep in mind that different boards will gravitate towards different values to mean average. On RPGnet a 3/3 is presented as average, but 4/4 seems to be rewarded far more frequently for average games. I have observed some readers react accordingly, taking an “average” point value and considering it as a below average rating. My advice to other reviewers is to just be aware of this when writing your score. You don't need to write to the site's average (I don't), but understanding this can help with disconnects between the score you awarded and the feedback you're receiving from the community.
I already mentioned that fans and publishers may be commenting on the scores the reviewer awards. What's important to understand is how to react to this. There are two primary, legitimate disagreements that a reviewer will find some people put forth. The first is that a score is just wrong based on the objector's own criteria. Perhaps the objector is more familiar with the game system than the reviewer, or maybe it's just a matter of difference of opinion. In either case, the best response is to acknowledge the objector's perspective, briefly restate the major factors for the score, and move on. Remaining positive at all times is essential to being professional and encouraging positive feedback.
The other legitimate disagreement is that the objector does not observe a connection between the review text and the score. This is especially likely to happen when the reviewer had some good or bad thoughts in mind that just didn't make it into the review, and by taking a step back the reviewer may very well see what the objector is talking about. Once again the best policy is to be polite and engage the objector. Restate reasons, or apologize for leaving a piece out and quickly summarize the reasoning behind the score.
There are also illegitimate disagreements that some may put forth. I have had the pleasure of irate fans telling me I'm a terrible person based on a review of a product they have never read, where they have also never read the review. Depending on the review there are a variety of other criticisms that are probably not worth engaging. Accusations of fanboyism, judging a product outside of its historical context, being a shill for a company, hating a certain topic, or anything that does not involve directly engaging the review are all possibilities. In my experience these more extreme comments tend to be less interested in seriously discussing the merits of the product than in starting a negative interaction with the reviewer. My recommendation is to thank the person for the contribution, acknowledge their points, and then to move on. It never hurts to suggest that the person contribute a review of their own as well. After all, if they feel so passionately about the topic then hopefully they'll also contribute some content for everyone to enjoy!
As mentioned, one legitimate objection is that the score was not well supported by the review text. A bad review will include a score that would not have been predicted by the review text. A good reviewer should take a moment when deciding the product score to make sure that that's really consistent with the review. This is a great opportunity for self reflection, as it forces the reviewer to examine their own biases and to make certain that the score issue is really deserved based on what they had to say and isn't merely what they want the product to receive. Having a specific part of the review set aside to clearly note the big positives and negatives is one way many reviewers quickly check to make sure their score logically follows from the review text.
Scores will vary based on the review system a reviewer is using. RPGnet's dual 1 to 5 scoring system is a little odd, with other sites 1 to 10 being more intuitive in many respects. ENWorld currently uses an involved multi-step 1 to 10 scoring system that offers a great deal of precision in scores. Each site is different and has different expectations for scores based on the value range and typical scores assigned by reviewers. From my perspective, anything more than a 1 to 5 is simply unnecessarily. Great games should receive a 5, poor games should receive a 1, and the rest should be a shade between. Unfortunately, most systems expect more detail than that which leaves it up to the reviewer to determine what the difference really is between a 6 and a 7 on a ten point scale. Fortunately, consistency of scores can help with that.
Of final note is consistency of scores. As a reviewer writes more and more reviews it may become difficult to keep the scores consistent across products. What the reviewer considered an average product fifty reviews ago may now be considered below average as the reviewer has had far more exposure to products. I recommend occasionally looking back over past review scores to remember why different products were evaluated as they were. If scoring considerations have changed that's ok, but striving for consistency of scoring across all reviews should be a priority. Not only does it cut down on potential criticism of reviews, but more importantly it aids those who follow the reviewer. Many readers will develop a sense of what the reviewer's own biases are, and consistency of reviews further serves them in determining whether or not they may want to pick up a given product.
Scoring is an important part of review writing, and while it's not as important as the text itself any skilled reviewer should keep in mind that it's the first thing many readers see and it will set the tone of the review for them. Supporting the score with appropriate text, or vice versa, maintaining consistent scores, and using consistent scoring criteria are all important to establishing a reliable review system for readers to enjoy again and again.
