Just some 2am thinking.
We often try to rate or rank things relative to each other, and often just describing or giving opinions on any given thing is enough but really it’s useful sometimes to think of things relative to other things. We have a bunch of common ways of doing this. Most of them can also be thought of either in terms of what you think is the best or your favorite, but that’s kind of a separate thing that coexists and the distinction doesn’t matter here so I won’t bore you with my opinions on how we use these words.
Anyway you can for example think of thing y as your favorite thing in category a, and perhaps say that you also have a second and third favorite. This is simple, it’s just a ranking of your favorite things. If you actually try to do this with any kind of accuracy you will quickly notice how difficult it is, at least if you are anything like me. It feels like any given entry to the list could easily be shifted up or down a bit, and it becomes very difficult to make a final decision that you have a huge amount of confidence in, and at some point you just need to pick something at random or via even more arbitrary criteria than you are already using. This is a core problem to all ways of rating things relative to each other (commonly used ones that I know of, at least).
Tier lists are a common and slightly more engineered way of rating/ranking, and in a lot of cases they are an improvement over a simple ordered list. The distinction between tiers is often useful. There appears to be a split between defining what qualifies a thing for any given tier vs just being an ordered list of a single metric, but split into larger groups where the divide between them is given some amount of significance. The former can at times drift away from being an ordered list at all, which makes it more of an exercise in categorization (than ranking or to a lesser extent rating) as there isn’t a clear order to the tiers (they may as well be mutually exclusive buckets/sets/whatever). Some tier lists do and some don’t have ordered things within tiers. Either way you still have a similar problem to what I mentioned with ordered lists, although it’s a bit smaller especially with non ordered tiers.
Often people ‘reviewing’ things on an ongoing basis will use a rating system where each thing is given a rating for example between 1 and 10 at the time of review. The number means nothing without the context of the reviewer’s other ratings, so the main point is to rate things relative to each other. Of course, comparing across different things based on this single number is rather reductive, so you get a lot of people disliking this form of rating. Honestly I’m not sure if it really serves much of a purpose in most cases besides just giving a numbered rating to things being fun. It kind of dodges the problem of being hard to decide the positions of things by allowing them to share a given rating similar to unordered tiers. Additionally ratings are often done without as much consideration to every other rated thing as they are done one by one on an ongoing basis which removes some of the analysis paralysis. However, you can and will still end up in situations where it’s impossibly difficult to give an accurate rating to things relative to the other things you have already rated.
I suppose at this point I should describe what I mean by “accurate”. When I am making a tier list for example, or listing my favorite trackmania mappers, there is some kind of pressure to ‘answer’ in a way I think is correct or right (or least wrong). It feels as if my true opinion must exist in some sense, but I am unable to reach it at the given moment due to for example lack of information or perspective. I end up basically having to answer at “random” (usually not actually random, more randomly picking something in the moment, which could be described more as what I feel like picking at the moment than random in a certain way, but that’s kind of irrelevant so I don’t know why I’m typing this.) I may also expect to be more likely to change these ratings in the future that I am currently unsure of.
The way I see it this essentially constitutes a kind of margin of error for all ratings and rankings of individual items/things. HOWEVER no rating or ranking system that I know of accommodates for this variable. I’m going to try to think and suggest some right now, but I am honestly hoping that the reader will think of something for me xd.
You could assign a margin of error (number?) to all items. Just as you would give a 1/5 star rating, you give a 1/5 confidence rating, or any other distribution. (Or perhaps -2 to +2 would be more to my taste for unrelated reasons that are outside the scope of this Chatting.) This could be applied to an ordered list or a tier list or even ratings done on an ongoing basis. Very easily applied to any format. I’m not sure how helpful it would be in practice though.
For example, how do you define what the number means? You could say it’s just an arbitrary scale, 1 is least confident you would be 5 is most, kinda like asking how much pain you are in. You could go a bit further and say 1 is roughly the bottom 20% of your confidence ratings and 5 is roughly the top 20% of your confidence ratings.
A more useful option would be to define the rating relative to the list you are making. For example a rating of 0 means you are pretty confident the item belongs in it’s tier, rating of 1 means it could move up or down a tier, then 2, 3 etc follow accordingly. When it comes to making tier lists this seems like a relatively simple and effective change to the ranking system, even if I don’t feel that it is the genius solution I am hoping to find to this whole conundrum.
Maybe you can make a kind of three dimensional tier list to represent the axis of higher/lower rating and higher/lower confidence in that rating. Ecks dee dee tier solution I would say.
Perhaps it’s impossible to find a truly elegant and universal solution for the simple reason that ordered lists and ratings are always very reductive, and ultimately overused (in most situations unnecessary). Still seems useful to attempt to improve the methods since we will continue to use them in some capacity anyway.
Anyway if you have any good ideas dm me on discord @skifreak.
Edit: You would probably also want a way of saying something has a margin of error but only really in one direction, like you may want to put it higher but definitely not lower, for example.
Suggestions recieved:
Agrabou:
- There’s ways you can do a large set of 1v1 direct comparisons between maps and end up with an ordered ranking based on your answers. I would think that would lead to easier decisions only because it lets you make decisions you would otherwise think might be wrong or be unable to decide on. I think this would be useful for making an initial list to then edit though.
More: Posts