The Dark Knight Falls

20 May
2012
avatar

by

Flickchart at its core is about two big things: helping you discover and rank movies to develop a coherent chart of your favorites, and mashing everyone’s individual charts into a globe-straddling list of the greatest movies of all time. All the other stuff — social features, recommendations, and so on — flow from these overarching pair of concepts. If either are broken, it’ll ripple through the site in ways small and large.

You’ve all been ranking movies like mad on Flickchart, some of you for years and some of you hundreds of thousands of times. We’ve kept an eye on the global charts and tweaked things here and there in response to feedback or our own gut sense and analysis of how those rankings were being aggregated. But there comes a time to pause and give deeper thought to how the machine ticks; how is “The Best Movies of All Time” shaping up? Let’s look at what it’s been telling us up to today:

  1. The Avengers (2012)
  2. The Dark Knight (2008)
  3. Star Wars (1977)
  4. The Shawshank Redemption (1994)
  5. Pulp Fiction (1994)
  6. The Empire Strikes Back (1980)
  7. Inception (2010)
  8. Fight Club (1999)
  9. Drive (2011)
  10. Raiders of the Lost Ark (1981)

What it’s telling us is that we’re on the heels of perhaps the greatest era of film-making in history. Count ‘em: FOUR movies within the past four years that are among the 10 greatest of all time. What a run! Something must have finally clicked with Hollywood to spur them to leave behind such dross as Rear Window, Casablanca, The Godfather, Ghostbusters, and The Shining.

Or… you could subscribe to the notion (as we do) that although excellent, movies like The Avengers vault up the charts on their newness, and that given time they’ll drop in the rankings to a more reasonable level. This is, after all, a list of the greatest movies of all time, and movies released within the past few years aren’t of a sufficient vintage for us to forecast their longevity.

The most obvious flaw in weighting new movies equally with older ones is selection bias: the folks who are way into superhero movies watch The Avengers in the theater, and the ones who love the movie hit Flickchart to rank it and build a digital monument to its greatness. The folks who don’t love it don’t rush to the site in equal measure, meaning that our global stats are drawn disproportionately from the subset of users who are Avengers fans. This effect will lessen as more ambivalent users rank The Avengers in random matchups, but it’s a gradual process — and we want the chart to be as accurate as possible all along the way.

With all that in mind, we’ve significantly changed the algorithm used to compile the global chart. We already use Bayesian inference to express uncertainty about a movie’s true average ranking based on the number of users who have ranked it (a movie ranked by 10,000 users is more “solidly” accurate than one ranked by only twenty); now, the new algorithm also expresses an analogous uncertainty based on how old the movie is. Has it stood the test of time? Does it find an audience beyond the confines of its own generation? The algorithm now defers those questions to time itself.

The end result is that the “The Best Movies of All Time” on Flickchart will consist of fewer films from the 1990s, 2000s, and 2010s. The change in distribution in the top 250, for instance, looks like this:

NOTE: Your personal Flickchart will not be affected by these changes! We’re only talking about the globally aggregated charts here.

Rest assured that we’ll keep our fingers on the pulse of the global rankings going forward, and we’ll continue to refine the algorithm as needed. Let us know what you think, and have fun Flickcharting!

  • HedgeHog

    What a GREAT tweak. As an “old guy” and regular rater I’m quite pleased at the shakeup.

  • Anonymous

    I’m not a fan of this change. I thought the point of this site was to compile everyone’s subjectivity into one official master list. Ignoring subconscious biases like recency and selection undermines that. As it was before, only more rankings from more users was necessary to make a movie’s ranking more accurate. Now by introducing time into the equation, you’re making the list more objective and therefore less reflective of the true pulse of movie fans.

  • http://www.the-frame.com Jandy

    I liked watching the other list rise and fall well enough, but it was by no stretch of the imagination “the best films of all time.” If you’re going to keep calling it that, then the new algorithm is better. “The true pulse of movie fans” is basically the current zeitgeist, which the older list was better at capturing, it’s true. I may be just arguing semantics, but “the best films OF ALL TIME” is an entirely different concept than “people’s favorite films RIGHT NOW.”

    Thought – I wonder if Flickchart could display both lists? Use the old algorithm to show what people are ranking the highest RIGHT NOW regardless of the movie’s age, and the new one to show what films stay at the top over time.

  • Nathan Chase

    Jandy – We may very well approach that idea, especially as it pertains to movies in theaters, or recent releases. The more charts and ways we can display the data, the better.

  • Andrew F

    I must admit that I am rather pleased with this change. The fact of the matter is that Flickchart and IMDb are heavily used by a rather small but passionate demographic: men from 18-25. If we want Flickchart to be something more than a ‘fanboy ranking site’, this kind of statistical analysis is necessary. Because let’s face it: “The Avengers” and “The Dark Knight” being the two greatest films ever made?

    I suppose then what we are really seeing with Flickchart is a conflict between deciding what films one considers the “greatest” (i.e. influential, artistically and aesthetically significant, etc.) and the films we consider our “favourites” (i.e. sentimental value). Can this site ever even achieve a balance between these, when so many of its users are simply –and excuse my academic snobbery here!– not versed in a large selection of cinema? Probably not. Flickchart is thus perhaps resigned to the fact that it can only represent a skewed, demographically biased ranked selection subject to the limited availability of certain kinds of films and the inevitable cultural bias of its users. Yay!

  • Nathan Chase

    Andrew – keep in mind that the global aggregate also favors users who have both seen more movies and ranked those movies more often. A user with 5,000 movies ranked 50,000 times influences those movies’ scores in the aggregate with more weight than a user with only 100 movies ranked 1000 times.

    You’re correct that we can only reflect the choices of those who choose to visit the site. We’ll continue to do what we can to make the barrier to entry and enticement such that anyone can (and should) have a Flickchart of their very own.

  • http://22crossroads.in/PG Kumar Ritwik

    Hugely disappointed.
    While I do not mind tweaks in algorithms here and there, if your new algo makes your list (almost) identical to IMDB Top250, I think it loses its point.
    Also, The Dark Knight was number 1, now its not even in Top 20? How does one believe this algo makes sense?

  • http://Flickchart Justin

    I see it like this. New movies don’t stand a chance of being in the top 10. The new way is all about older movies because they are considered classics. Give some of the new release movies some credit. They are really good. They just got thrown to the side with this new list. I love the old list

  • CONMAN999

    I’m not really sure if I like the changes or not. I definitely do appreciate the active approach to try to find just the right algorithm. I hope you guys continue to tweak it and get it right where you want it.

    @Nathan Can you speak on exactly how much more weight a user gets for having seen more movies? I think the algorithm might be better if it favors these people even more than it does now. This way, a user who has seen 10,000 movies has probably seen movies from the 50′s as well as The Dark Knight and The Avengers. Then the rankings would heavily rely on these people ranking one over the other rather than movies just getting “points” for being old.

  • http://www.flickchart.com/DaveintheCave David Pawlowski

    Is it just me or have animated films gotten hit particularly hard with this change.

  • Mike Osciak

    My biggest problem with it is that there still isn’t a solid balance. Before, newer movies had the advantage and now, older “classics” hold all the cards. It went from one extreme to the other.

  • http://www.flickchart.com/DaveintheCave David Pawlowski

    Is this still in the works? Cause I noticed that The Good the Bad and the Ugly moved from 21 to 20 in the last 12 hours.

  • Guitarman

    I like the new list as far as better representing the history of cinema. I mean, I loved The Avengers too but best movie of all time? I don’t think so.

    I do have a question about the algorithm. If a user adds a movie to their list, will this automatically cause that film’s rank to rise or can its rank fall if it is added low down on a user’s list?

  • Anonymous

    The approach of adding uncertainty to newer films seems like a great way to establish which movies are timeless masterpieces, however, I would like to see a ranking system made of two parts: how a movie ranks against all movies and how that movie ranks against movies from its same year. Combining these two factors might give a better balance between classics and great new films.

  • Jeremy Thompson

    Kumar: I suspect that the users who take a lot of time rating movies on IMDb and those who spend a lot of time ranking on Flickchart have a big overlap in that aforementioned 18-25yo male demographic (or at least 18-35). For our lists to end up somewhat similar shouldn’t be terribly surprising.

    That said, the new chart still reflects uniqueness in the Flickchart user base; Ghostbusters, for instance, is at #23 for us, but doesn’t appear anywhere in IMDb’s Top 250. Monty Python is at #80 for them and #14 (!) for us.

    The Dark Knight and Inception used to be in the Top 20 on *both* Flickchart and IMDb, and now specifically due to the algorithm change, ours are much further down the list than theirs.

    Finally, it just doesn’t work to compare our current chart to the old one, and conclude that this one is broken because TDK shot down the list so far. If it was drastically wrong in some cases before (and that’s our contention), that’s exactly what we would expect to happen. If our original chart had had Glitter at #1 and it had gone to its current position of almost 17K, you could use the same logic to say “it’s obviously wrong!!!”

  • Jeremy Thompson

    Justin: Good points; we don’t want to cast new movies to the side just because they’re new. The Avengers is currently at #82, though, meaning that despite being less than a month old and ranked by a large proportion of fans rather than a true cross-section of FC users, it’s being declared as one of the 100 greatest films in history.

    It’s arguable that The Dark Knight went down too far, though. We’re evaluating how heavy to weight the age portion, and we might ease off a bit depending on how things feel. It’s all subjective, unfortunately.

  • Jeremy Thompson

    Conman: We’re still tweaking. I’ve yet to hear a convincing enough argument that age shouldn’t be weighed in the algorithm for “The Best Movies of All Time”, but just how much weight to give it is subjective.

    The amount of prevalence a user has in the globals is based on the number of movies in his or her chart. We look not at absolute number when calculating, but at the percentile position a movie has in your chart (e.g. 90th percentile == top 10%). The #1 movie for someone with 10 movies is in the 90th percentile, whereas the movie merits a 99th percentile rating for someone with 100 movies.

    I’m experimenting with weighting prolific movie watchers more heavily, because at the mid-to-high range that effect is less pronounced. Thanks for the comment.

  • Jeremy Thompson

    Mike: You’re not the only user who’s commented on it seeming like a swing from one extreme to the other. As I commented above, we’ll be looking deeply into all our parameters to judge the intensity of the recency weight.

  • Jeremy Thompson

    David: That was an organic change. We haven’t tweaked the algorithm since its deployment yet.

  • Jeremy Thompson

    Guitarman: Another user adding a movie to his list won’t have an inherently positive or negative effect on the movie’s global position. As you say, it will depend on where the film is on that user’s chart: we aggregate the percentile ranking of each movie across all users.

  • Jeremy Thompson

    Anonymous: We agree with you entirely, and we’re working on more contextual charts that are less holistic so you don’t have to see those same old classics year after year.

    That said, you can always look at the 2012, 2011, or 2000s chart by filtering the global chart accordingly.

  • becks711

    This is a knee-jerk reaction because avengers went no.1 after in its first mth of release. I dont think it is the fact that it is a new movie that has got flickchart so bothered, but that FC is of the opinion that it is an ordinary movie at no.1 (I too added it at no.838 on my account). But is not for FC or me to decide which is the best movie, but everyone that uses FC. If more people like this than star wars, so be it. Its their choice, not FC’s. That is what made this site so good. Now it feels like the chart is manipulated too much. I liked it just the way it was…

  • Derek Armstrong

    I think the point is, any system for determining which is the best is subjectively created using subjective criteria. Becks, you say it is not for you or FC to decide what is the best movie — but in a way, FC is in fact deciding it by using the particular algorithm they use to weight certain factors over other certain factors. Who is to say when evidence will come along that indicates that the factors they previously thought were effective to determine the best movie were, in truth, not the right factors? The algorithm was chosen based on a logic that seemed to make sense, but sometimes certain elements (such as The Avengers) come along that challenge the notion of whether that logic actually does make sense. I think it’s fair to say that a movie that is only a month old cannot really be the consensus pick of any group of people as the best movie of all time, simply because too few people will have seen it yet. To me, that’s the kind of thing that should cause the guys at Flickchart to reconsider whether their criteria was correct.

  • Derek Armstrong

    And I see your point when you say the reaction is knee-jerk — in an ideal world, you wouldn’t want any single film to play a role in changing an entire paradigm for how things are done. However, sometimes a single film is what causes you to realize the system is broken. And when you do realize something is broken, you sometimes want to fix it at knee-jerk speed — in other words, as fast as you can.

  • bill

    tdk is a pile of poo anyway. ofc the avengers is going to be rated the best. only ppl who would like that movie would go to see it. i will probably never see it, therefore it will not be on the bottom of my list next to tdk.

  • Sean

    I have to call shenanigans on this change. First off, seems like part of the problem comes from FC harping on the list being called “The Best Movies of All Time” and wanting it to live up to that name. Which is ridiculous since the “best” of anything is subjective, really the name of the chart in question should just be changed to “The Favorite Films of Flickchart Uers”.

    Secondly, just because a movie is old doesn’t instantly make it better than a newer movie. Apparently now with this new formula nothing from the 2000’s is worthy of being in even the Top 30 “best” films of all times, seems this new formula is even more bias than the old one supposedly was.

    As if to prove my point lets take a look at the films currently sitting at #33-#35:

    33. Rashomon (1950): 23% Have Seen It; 4,072 Total Users Who’ve’ Ranked It; 9 Have it At #1; 229 Have it in Their Top 20.

    34. Double Indemnity (1994): 24% Have Seen It; 4,539 Total Users Who’ve Ranked It, 13 Have It at #1; 239 Have it In Their Top 20

    35. The Dark Knight (2008): 91% Have Seen It; 65,120 Total Users Who’ve Ranked It; 2,474 Have It At #1; 21,192 Have it in Their Top 20

    I will admit my bias towards The Dark Knight; I have it at #2 on my list. However, I ask everyone to put personal preferences aside and look at those stats I listed above. To me it seems pretty clear the age of a film is being relied on too heavily on in this formula over what the users of this site actually list as liking.

  • http://www.flickchart.com/DaveintheCave David Pawlowski

    To all the naysayers, it’s pretty clear from the diagram that the list is FAR more balanced than it was before.

  • MysticSpoon

    I completely agree with Sean on this one. What does ‘the best movies of all time’ mean, anyway. I think it should be ‘the most liked movies’ too. The new system relies too heavily on the age of the movie, whereas the old system relied too heavily on how popular a movie is. It should be somewhere in between, with a small tendency towards popularity. I think the win/lose ratio and the seen/not seen-ratio are more important than the number of people that have liked a movie (no wonder Dark Knight was number 1) and the age of the movie (no wonder Citizen Kane and Monty Python and the Holy Grail are now so high on the list). But yeah, it’s tough to make the list as accurate as possible.

  • MysticSpoon

    Or in short: the old system was democratic, but was also heavily influenced by popularism. The new system gets rid of the popularism, but is hardly democratic anymore. There should be a balance: a democratic system that ignores a certain % of the popularism.