The 25,000 Vote Minimum for the Top 250 is Indefensible!

  • 5
  • Question
  • Updated 5 years ago
  • Answered
Who made this decision?

I just used Advanced Search to find the 17 movies that have 20,000 - 24,999 votes and a score of 8.1 or better.

There were 17 films that fit this criteria.

I looked at the number of votes from Top 1000 voters as a proxy for the "regular voters" whose votes determine the Top 250. Nine of the 17 had 361 such votes or fewer and would clearly be extreme long-shots to make the Top 250 once you exclude the non-regular voters (Underground, Tae Guk Ki, The Cove, Departures, The King of Kong, Lagaan, My Sassy Girl, Senna, Rang De Basanti).

The other eight films had between 476 to 588 votes from Top 1000 voters, and all but two had scores of 8.2 or higher. In order of raw score:

Ikiru, 8.3, 24153 votes, 476 Top 1000 votes
The Wages of Fear, 8.3, 21496, 563
Diabolique, 8.3, 23689, 580
Judgment at Nuremberg, 8.3, 22955, 588
The Best Years of Our Lives, 8.2, 24467, 558
The Battle of Algiers, 8.2, 22004, 518
Fanny and Alexander, 8.1, 23805, 495
The Cabinet of Dr. Caligari, 8.1, 24097, 553

Rank of these films among *critics* in the top 1000 of all time at TheyShootPictures, on order: 83, 180, 576, unranked, 131, 74, 75, 148. So with one or two exceptions, they are also regarded by critics as slam-dunk all-time classics.

Someone really thought the IMDB Top 250 would have more credibility and usefulness if it excluded these 8 films and included Jurassic Park, Papillon, 3 Idiots, Big Fish, Castle in the Sky, The Untouchables, Arsenic and Old Lace, and The Help instead? Seriously?

Put another way, when it takes an 8.0 to make the Top 250, how can a film that has been seen by more than half the Top 1000 voters (more or less) and grab an 8.2 or higher (for the most part) be considered unworthy?

Please pass this on to the powers that be. The vote minimum needs to be dropped back down to 20,000. Otherwise, it's just an embarrassment to the web site.
Photo of Eric M. Van

Eric M. Van

  • 24 Posts
  • 37 Reply Likes

Posted 7 years ago

  • 5
Photo of Dan Dassow

Dan Dassow, Champion

  • 13950 Posts
  • 14546 Reply Likes
Eric,

You are not the first nor only one to question the 25,000 vote threshold. See this thread for more details:
Top 250 Chart change announcement.

You may also be interested in this list::
Highly Rated Films Needing More Votes to Make the IMDb Top 250
Photo of Eric M. Van

Eric M. Van

  • 24 Posts
  • 37 Reply Likes
Oh, I'm well aware that people have been complaining. Complaints about IMDB ratings of any and all sorts at the site proper have always been ignored by staff, so I thought I'd give it a shot here, where there is an actual IMDB employee whose job it is to respond to complaints.

Folks have been complaining about the nonsensical weighted averages given to films with a small number of votes (e.g., < 1000) for years. (I know, because I was one of them, and not even on behalf of films I knew anything about.) Nothing ever gets done. I've come to the conclusion that the various algorithms they boast about for making the ratings fair and free of manipulation were invented, put in place, and have never been subsequently reviewed, even though they've been revealed to be horribly flawed, and that there is no one at IMDB whose job it is to attempt to improve then, or, indeed, to pay any attention to the ratings at all.

In the current case, someone made a decision to raise the minimum number of votes from 3,000 to a greater number. I don't think that was necessarily a bad idea. But it's painfully, pathetically obvious that the 25000 vote minimum was pulled out of an orifice with perhaps ten or twenty seconds of thought. It took me just an hour with only the Advanced Search tool to figure out what films would be included if the limit were 20,000 votes instead, and the results were above. It's impossible to think that more than an hour was spent on this decision.

Personally, I've got an impressive background in creative data analysis (four years as a Baseball Operations Consultant with the Red Sox, current chair of the Society of American Baseball Research's Science and Baseball committee) and I'd love to get my hands on the IMDB database and improve the ratings across the board. (I even wonder if they've done things like normalize each voter's votes for how easy or tough a grader they are. And rather than simply using a Bayesian estimate to adjust for vote totals, that could be done empirically by analyzing how ratings drop as films gain exposure. Etc.) They could hire me or someone like me for a sum that would be completely invisible to their bottom line, but would add a wealth of value and a great deal of credibility to the site. From a business point of view, it's a no-brainer. But it doesn't appear that there's even anyone in a management position whose job it it to care.
Photo of Dan Dassow

Dan Dassow, Champion

  • 13950 Posts
  • 14546 Reply Likes
Eric,

As someone who is an operations research analyst, I respect the work of qualified sabermetrics analysts such as you. The primary problem with the IMDb ratings is that it is effectively a self selected poll. I would also find it interesting to do proper statistical tests on the raw data.
Photo of Rob Underhill

Rob Underhill

  • 9 Posts
  • 2 Reply Likes
All of my films in fledgling status and unreleased have been score-bombed by top 1000s voters. They simply doll out "1"s. It's pretty ridiculous. And no way they've seen the work (unreleased). I'm probably going to get more "1"s for complaining! One film had a bunch of 9s and 10s (sure by family, friends, etc of a preview screening) but one "1" score comes along and the overall score becomes 3.0
Photo of Emperor

Emperor, Champion

  • 6418 Posts
  • 3004 Reply Likes
The vote minimum needs to be dropped back down to 20,000.


From tinkering with the searches and excluding low vote numbers, then you get the film's with obviously manipulated scores dropping out around 5,000 votes so it might even be possible to drop the limit to around 10,000 without there being much of a problem and it would bring in a lot of excellent films that would benefit from the extra attention.

However, it is always going to be a balancing act and I assume the limit was raised because there are dodgy types out there selling IMDB votes by the thousand (of course, this might just be a scam) but I don't think lowering the limit to a reasonable level would open the door to these master manipulators.
Photo of brmcoyle

brmcoyle

  • 1 Post
  • 0 Reply Likes
"I looked at the number of votes from Top 1000 voters as a proxy for the "regular voters" whose votes determine the Top 250. "  This sounds like you looked at how many times each of the Top 1000 voters voted.  The next sentence:  "Nine of the 17 had 361 such votes or fewer and would clearly be extreme long-shots ..." Now you're referring to films.  Huh?
Photo of Eric M. Van

Eric M. Van

  • 24 Posts
  • 37 Reply Likes
brmcoyle, I'm not sure what you're confused about. There were 17 highly rated films with between 20K and 25K votes. For each film, I looked at the number of votes it had from Top 1000 voters. Nine of the 17 had been seen by fewer than 40% (actually, 37%) of the Top 1000 voters and thus were clearly niche films whose vote totals were artificially high. These films could hence not be expected to maintain their high rankings while still having 20K votes from "general voters." The other eight had been seen by at least 45% of Top 1000 voters and thus could be expected to hold their high rankings when only "general voters" were included. And in fact 7 of the 8 films are considered all-time classics by critics.

This conversation is no longer open for comments or replies.