Making A List, Checking It Twice: Algorithmic List Generation vs. Handcrafted List-Making

For nearly two decades now, I have been compiling year-end rankings of my favorite films of the previous year.

Only for the last half a decade has Flickchart been around to help me with that task. Thus far, I have not sought out the help… even though Flickchart allows you to filter the movies you’ve seen from a given year, and shows your exact order of preference of those movies.

This has partly to do with the fact that I just plain like making lists. If I handed over the responsibility to Flickchart, it would deprive me of one of my favorite pastimes. Then there are the practical reasons an organic list seems preferable, like the fact that I have a pretty subjective method of determining which movies I organically rank. The list usually includes at least 20 films that are “technically” from the previous year, either because that’s the year they were released in their country of origin, or because they played a bunch of festivals before their general theatrical release. At the moment, Flickchart can only compare movies with others from the same release year, if you are going to filter by release year.

This is not to say I have not been curious about the discrepancies between my organic list and the rankings I have given these movies on Flickchart. If I am truly consistent in my thinking, they should correspond pretty well. Of course, I am not consistent in my thinking, and that’s what complicates any attempt to compare them. I thought it was time to do the comparison, though, just to see which version of my 2014 rankings really seems like the correct one.

A few things you need to understand going in

1) My organically compiled 2014 rankings were finalized on January 15th, the morning the Oscar nominations were announced. They represent a snapshot of how I felt about those movies at that exact moment. Even a week later, some movies would surely have moved up or moved down.

2) My Flickchart rankings of those same movies were finalized on March 18th, the day I finally ranked the last movie (Love is Strange) I watched before finalizing my organic list. The delay was due to my personal policy of letting a month elapse between seeing a movie and adding it to my chart. I view this as a “cool down period” that allows me to gain a more realistic assessment of how I really feel about it. I got a bit behind, so more than a month elapsed.

So there is a built-in problem with this comparison since it presupposes no change in the way I feel about these movies from January 15th to March 18th. As I’ve said above, I know that’s not the case. However, it’s still only two months, so I’m much closer to having the same feeling about those movies than I will be six months from now.

My Top Ten from the organic list

1. Birdman or (The Unexpected Virtue of Ignorance)
2. Like Father, Like Son
3. Ida
4. The Skeleton Twins
5. Edge of Tomorrow
6. Whiplash
7. Love is Strange
8. Boyhood
9. Enemy
10. Under the Skin

The Top Ten as I ranked them on Flickchart

with my overall ranking (out of 4100) listed in parentheses:

1. Ida (176)
2. Like Father, Like Son (185)
3. Love is Strange (188)
4. Birdman or (The Unexpected Virtue of Ignorance) (191)
5. Boyhood (230)
6. Edge of Tomorrow (241)
7. Whiplash (271)
8. The Skeleton Twins (286)
9. Snowpiercer (356)
10. Under the Skin (357)

Pretty interesting. Nine out of the ten are the same, as are two of the top three. However, my #1 from the first list drops down to #4 on the second, and my #4 drops down to #8. Perhaps most noteworthy is the one movie from outside the organic top ten that snuck into the Flickchart top ten, which is Snowpiercer at #9. Snowpiercer had dropped down to #15 on my organic list, the result of a second viewing that left me less enthused about the movie than the first. Since I don’t re-rank a movie after each viewing, it keeps its lofty Flickchart ranking even though I now have less love for it.

While the top ten is more or less the same but in jumbled order, some real discrepancies show up as we move down the list. Mistaken for Strangers is #26 on my organic list, but all the way up at #14 on my Flickchart list. Like Snowpiercer, that has to do with some initial passion for the movie cooling over the course of the year. So really, if I wanted those movies to more accurately reflect my changing opinion of them, I needed to re-rank them at that time. Otherwise, the only way for them to move down is gradually, as other movies beat them and knock them down one spot at a time.

Sometimes Logic Isn’t Logical

Then there’s the case of The Hobbit: The Battle of the Five Armies, which was only 53rd on my organic list but is all the way up at 31st on my Flickchart list. That movie was both seen and ranked recently, so the same logic as above doesn’t apply. As I was ranking that movie and trying to get it to the correct spot — more on that phrase in a moment — I granted it a win in a duel that I knew would thrust it higher on the chart than perhaps it deserved to be. Once it had beaten that movie, it was going to end up higher than I imagined it ending up, no matter what happened from then on. A similar phenomenon can be observed elsewhere between the two lists.

So, “trying to get it to the correct spot” is something that artificially factors into the process. Once I already know where I’ve ranked The Hobbit organically, the later Flickchart ranking process is a bit tainted, because there’s that small part of me that knows where it’s “supposed” to end up on the chart. In this and other scenarios, I’m trying to get the dueling process to conform to some preconceived notion of the movie’s worth, which means I’m not letting the site’s methodology operate as purely as it should be operating.

The Results Can Be Surprising

There are blocks of list, though, that are incredibly accurate. From slots 100 to 105 on the list, no movie is off by more than two positions between the two lists. In fact, from positions 100 through the 136th and final spot, only one film is off by more than ten slots — the Nicolas Cage vehicle Rage, whose #100 ranking on my organic list was likely too generous. The #117 ranking on Flickchart is probably more in line with the film’s real quality.

The one interesting thing I noted is that 76 of the 136 movies are in the top half of my Flickchart, meaning that they’re ranked #2050 or higher. When you are adding By Title and the movie comes up against the movie in the exact middle of your chart, you are not only dueling against that one particular movie — you are also deciding whether the film you’re adding will end up in the top or the bottom half of your chart. So in effect, I made a decision that I felt more positive than negative about 76 of the movies I saw last year, and more negative than positive about the other 60.

This perhaps hurt some of the guilty pleasures on my list. Take Walk of Shame. This is one where I ranked it lower on Flickchart than I have it ranked on my organic list, #74 vs. #65. At the time I was ranking it, I knew that if it belonged in the top half of my movies, it should be only just barely. So ultimately I punished it for its perceived status as a movie I shouldn’t have liked.

Offline vs. Online

The unavoidable difference between composing my list offline and composing it on Flickchart is simple: other movies. When I’m adding a new movie to my organic list, I am essentially using a Flickchart model to do so — I am choosing its spot based on the fact that it’s worse than the movie above it, but better than the movie below it. However, in that scenario, my universe is limited only to the movies from that year. When there are movies from 2007 and 1954 and 1987 and 1922 mixed in, I have to consider how I feel about those movies in addition to how I feel about the 2014 ones. Impressions of older movies are even less trustworthy than impressions of newer ones.

I suppose the real way to do an accurate comparison would be to create a Flickchart comprised only of 2014 movies. However, it’s the other movies adding random interference that even makes them two different types of lists in the first place. If you were focusing only on movies from last year, then it really would be no different than composing the list organically offline.

So which list is “more me”? I’m going to be cagey and not provide an answer. While there’s no doubt Flickchart has revealed some preferences that I didn’t know I had, the organic list compensates better for unconscious, up-to-the-moment changes in my feelings about a film. I’ll tinker with the relative placement of films on an organic list more than I’d tinker with them on Flickchart, apparently. Perhaps that’s because it’s a smaller, more manageable universe, while once you start re-ranking movies on Flickchart, you may be starting a project without an end.

Ultimately, as useful a tool as Flickchart is in building filtered lists of preferences, I will probably never give up the organic list, for one simple reason: Being a list maker is what drew me to Flickchart in the first place. I want Flickchart to enhance my list-making tendencies… not replace them.