As I’ve summarily explained in a previous article, having no framework leads to unfair review practices in a very technical sense of the word. On top of this we should consider that unlike systems that are powerful in that they express their elements truthfully, those that approximate a temporary molding together of personal influence and bias alongside sentiment and guessing are easier to use because humans do it all the time. They act as heuristics by which people operate in the world in a correct manner. But if the whole point of a process is to measure quality then that sort of ambiguous heuristically-driven approach doesn’t fare well at all. This is not difficult to realize. Find any reviewer on the web and find a review they made on a game level. They may have described their experience in playing the level. More analytical approaches provide elaborate essays detailing particular events and things that stood out. But in the case wherein no defined framework is provided you cannot trace their reasoning back in a logical way to conclude how they went from a description of sensory experience to a number, which so happens to be the very element of ambiguity. This is true both practically and logically; English is an ambiguous language and I don’t know if there has ever existed a human-developed language that is not ambiguous, in reference to the definition of ambiguity used in formal languages, a linguistic and computer-scientific study.

We can at least begin with a framework that has an outer structure. In the cases where each aspect of which the framework is composed of is well defined, readers and observers can more easily decrypt the reviewer’s reasoning and isolate particular components in order to more adequately figure if the reviewer’s rating makes sense, which is to determine the degree to which the rating coincides with the reasoning supporting it, which is collectively an approximation of the quality of that particular element, and being the very point in question, it suffices to say it is the whole challenge motivating the framework in the first place.

In order to build a framework however, we need to figure out what we want to get out of a work of art, which roughly translates into the need to define quality. This is dangerous territory because so many fields of study come to mind with partial answers. I’ll provide two different answers. The first is a general rule that may give a thoughtful and intuitive sense of what the goal for this framework is. I hold the philosophical presupposition that art ought to express and evoke the human spirit. The author ought to express it, and should endeavor to construct their art in order to evoke the spirit of others. This should be taken with a grain of salt, because what I mostly mean by this is more grounded in scientific concepts. This is where the second answer comes along, and it’s a long winded one. Beauty –the beauty we speak of, or perhaps artistic quality– is a property constructed by human interpretation. Hence we have fields of knowledge in every direction which may contribute to the problem. Beauty is undoubtedly influenced by phenomena studied in psychology, evolutionary biology and neuroscience. It has also been well-established, for instance, that certain stimuli in controlled environments yield a deterministic response from the human brain. Change blindness is a popular example. Scientific experiments have demonstrated that changing a particular element on a still-image in a sufficiently slow manner does not catch the attention of the human viewer, provided there are no disruptions [1]. There have also been quite a few many studies done documenting human male preferences of females based on deciding their level of beauty. Physical appearance is a culturally-independent property relevant to humans in seeking a mate, and certain features are found to be attractive across cultures [2]. It has also been demonstrated that combining visual and auditory stimuli resulted in activity in a particular cortical region [3], although it is more difficult to scientifically extrapolate more meaningful conclusions out of such experiments. Given that there are particular stimuli that evoke deterministic or chaotically deterministic responses in the human brain and that beauty is heavily influenced by human evolution we can at at least be certain that there is an human-contextual objective description of beauty, and so a framework which seeks to measure beauty as a quality of artwork ought to aim to identify aspects which reflect the degree to which beauty is properly expressed.

At this point you can imagine that interactive quality has a similar background, although perhaps there is a more distinct difference in male and female brains when it comes to interaction and play as opposed to visual and auditory stimulation. Nevertheless this collectively provides us a foundation onto which we can explore more proper methods by which we can quantify quality, and with this kind of unexplored and mysterious challenge we’re sure to modify some things every so often, but the whole point is to get closer to a more universal reading of design quality.

The initial template we use is as follows and is given out of its sum value of 10 points:

• Architecture (2)
  • Structure (1)
  • Innovation (1)
• Atmosphere (2)
  • Visual Immersion (0.75)
  • Auditory Immersion (0.75)
  • Detail (0.5)
• Gameplay (2)
  • Entertainment (0.5)
  • Intensity (0.5)
  • Novelty (0.5)
  • Flux (0.5)
• Visual Impact (2)
  • Concept Impression / Grandness (1)
  • Visual Awe (0.5)
  • Visual Creativity (0.5)
• Storyline
  • Character Development (0.5)
  • Plot Development (0.5)
  • Depth (1)

I will break down each component individually and elaborate on it in future articles. You may notice that Storyline doesn’t have the kind of immediate experimental support that other aspects hold, so it also begs investigation on a psychological level, although that is an entire topic that can be written about in the future.

References:

[1] “Change blindness in the absence of a visual disruption”. Simons, Daniel J. Franconeri, Steven L., Reimer, Rebecca L. Perception. 14, July 2000. Vol. 29 pp. 1143-1154 DOI:10.1068/p3104

[2] “Maxims or myths of beauty? A meta-analytic and theoretical review”. Langlois JH, Kalakanis L, Rubenstein AJ, Larson A, Hallam M, Smoot M Psychol Bull. 2000 May; 126(3):390-423.

[3] “Toward A Brain-Based Theory of Beauty”. Ishizu, T., & Zeki, S. (2011).  PLoS ONE, 6(7), e21852. http://doi.org/10.1371/journal.pone.0021852

Have you ever read or listened to a game review, or a review for anything for that matter, and wondered how on Earth the reviewer came up with their diagnosis? Maybe they gave it their highest rating, but you find yourself baffled at how they might have been able to reason it through in the first place. Perhaps you saw a mod with a “Best Of” award, although the mod didn’t seem particularly outstanding. In the overwhelming majority of cases I’ve seen, mod reviewers in particular employ a practice of judgment and reasoning which is only as sound as the extent to which they can properly judge and reason, which is not saying very much given that it’s difficult to say that a review based off of reason does much more than discreetly display the reviewer’s biases and preferences. Not to mention that they might have never exposed their methodology which they use in order to come to a decisive conclusion in their review. The collection of these problems makes things confusing.

Firstly, since there is no sufficiently rigid framework for what defines quality, a reviewer can “feel” as though one mod is better than the next. The reasoning by which they say one is better than the other can hardly be said to be identical if they can’t describe the framework through which they felt one was better than the other. The result in effect is two different ratings based off two different (un)reasonable conclusions. It would be the equivalent of doing the following: looking at two different works and judging the first solely on its visual quality, and the second solely on its auditory quality. The first gets a far better rating than the other. That’s not a fair rating because, technically, what the reviewer actually did is give one a rating based off System A, and the other based off System B, whose components are totally independent, so that there is no comparison to say which one is better, yet one got a better rating. What many reviewers do is similar. They have no rubric and as a result, they can step into all sorts of mistakes that skew their review process. It’s not particularly credible. In the end if you are comfortable with that kind of phenomenon, it’s fine, but realize it’s simply the “Person’s Opinion Awards” and not an endeavor for true quality control, measurement, and comparison.