Simpsons Episode Ratings Hacked?

  • 4
  • Problem
  • Updated 4 years ago
  • Solved
Hello!

I like to look through The Simpsons episode ratings here while I'm watching the show.  It always seemed to be pretty much on par with what you'd expect ratings-wise if you're a fan.  However, lately this has not been the case.

Recently, there has been a drastic ratings fluctuation on certain episodes. 

I found it funny that now, all of a sudden, some of the most widely-accepted "best episodes" are rated quite poorly (Or at least very 'average.').  Additionally, I've been seeing certain episodes that have been widely panned, rated very highly.


RATINGS DROP
Let's start with the episode "Marge Vs. The Monorail."

This is one of the most universally acclaimed Simpsons episodes of all time, and it's a rarity to not see it on anyone's top 10 list.

This used to be rated a 9+.  Now it's a 6.9.

As I started to investigate, I checked the ratings breakdown graphs. [http://www.imdb.com/title/tt0701173/ratings?ref_=tt_ov_rt]

If you look at the second graph, with the red and blue, you'll notice that most of the female age groups voted very low, around 2.5 average.  If you look at the episode ratings graph immediately before or after the "Monorail" episode, you'll see the male and female groups are more in line with each other and much less lopsided.  

I would bet that if an admin were to investigate this, you would find a point in time where this episode rating got flooded with "1" and "2" ratings in a short amount of time.


RATINGS SPIKE
I've also noticed this tampering for episodes with previously low ratings.  Let's use "The Principal And The Pauper" for this example.

Previously, this was rated a low "6."  Now it's an 8.4.  Once again, if you look at the second graph on the ratings breakdown page, [[http://www.imdb.com/title/tt0701257/ratings?ref_=tt_ov_rt]] you'll see that the female age groups all rated very highly, a 9.6 average. (Much higher than the male average)  And also, once again, if you look at the breakdown for the episode immediately before or immediately after this one, the ratings breakdown is more even and less lopsided between genders.

And again, I would bet that if an admin were to investigate this, you would find a point in time where this episode rating got flooded with "9" and "10" ratings in a short amount of time.

It probably goes without saying that most of these episodes that have been tampered with also seem to have 400-500 more "votes" than most episodes.

LIST OF (Possibly) TAMPERED EPISODES
Here is a list of just some of the episodes that seem to have something fishy going on.

DROP IN RATINGS:
El Viaje Misterioso de Nuestro Homer - http://www.imdb.com/title/tt0701094/
Sunday Cruddy Sunday - http://www.imdb.com/title/tt0701220/
Homer At The Bat - http://www.imdb.com/title/tt0701114/
Marge Vs. The Monorail - http://www.imdb.com/title/tt0701173/
You Only Move Twice - http://www.imdb.com/title/tt0701298/
Homer The Great - http://www.imdb.com/title/tt0701115/?ref_=fn_al_tt_2

SPIKE IN RATINGS
Alone Again, Natura-Diddly - http://www.imdb.com/title/tt0701052/?ref_=fn_al_tt_1
Two Bad Neighbors - http://www.imdb.com/title/tt0779680/?ref_=fn_al_tt_1
The Principal And The Pauper - http://www.imdb.com/title/tt0701257/?ref_=fn_al_tt_1
Homer Defined - http://www.imdb.com/title/tt0701110/?ref_=fn_al_tt_1
Homer The Smithers - http://www.imdb.com/title/tt0779667/?ref_=fn_al_tt_1


For the record; I don't 'hate' the Principal And The Pauper episode, and surely don't consider it the 'worst episode ever.'  I do however, find it hard to believe that it's rated a whole point and a half higher than a classic such as "Marge Vs. The Monorail," when it used to be the other way around just a short time ago.

This post isn't about which episodes are the best or the worst.  I've just noticed some suspicious things lately while going through them.  I could be completely crazy for bringing this up, and everything really is how it should be.  I don't think I AM crazy though, because I've noticed these drastic changes without even looking for them.  Maybe it's worth someone looking into; this is all I'm saying.  Thanks for your time.
Photo of NekPants

NekPants

  • 3 Posts
  • 0 Reply Likes

Posted 4 years ago

  • 4
Photo of Nobody

Nobody

  • 1455 Posts
  • 706 Reply Likes
Others have also noticed the changing trends in ratings.
Here are two related discussions:
http://www.imdb.com/title/tt0096697/board/nest/246361403
http://www.imdb.com/title/tt0096697/board/nest/246770581


Although there is no way to see a full history of ratings,
some partial history can be gleaned from page snapshots
via the archive.org Wayback Machine.

For example, "Marge vs. the Monorail"
was rated at about 9.0 until 2012.
By mid-2013 it declined to 8.6.
https://web.archive.org/web/20130727091536/http://www.imdb.com/title/tt0701173/

By late 2014 it had dropped to 7.0
(due apparently to the influx of many "1" votes).
https://web.archive.org/web/20141109174808/http://www.imdb.com/title/tt0701173/

And now it is 6.9.
(Edited)
Photo of NekPants

NekPants

  • 3 Posts
  • 0 Reply Likes
Great info, thanks.  For the record, I did do a search first.  I figured I couldn't have been the first person to notice this!  Thanks for the reply.
Photo of ljdoncel

ljdoncel, Champion

  • 771 Posts
  • 1553 Reply Likes
Hi, NekPants:

I agree with you. I believe that it's pretty clear that there has been a deliberate manipulation of the ratings and here are some supporting evidence of it.

(My English is not good enough to explain it perfectly, but I'll try my best)

As Lucus has noted, while it is true that we don't have a full history of ratings, there are several snapshots of the episode list (with ratings and number of votes) of the series at the Wayback Machine, and some interesting analysis can be done. A first inspection suggests that the fraudulent voting probably started at around mid-2013. I've collected data of the first 9 seasons of the series, ignoring web captures dated earlier than 9-10 years after the ninth season aired (aiming for a "steady state" far from the initial hype). Then, I've defined two periods of study:
  • Period 1: From March 2008 (03/09/2008) to December 2011 (12/01/2011).
  • Period 2: From December 2011 (12/01/2011) to November 2014 (11/10/2014).
For each time point we have the average rating and the number of votes/users until that point, so we can calculate the number of new users between two time points and the average rating of those new users during that period:



The following graph represents, for every episode of The Simpsons, the number of new votes between december 2011 and november 2014 (period 2) in terms of percentage over the users at the beginning of the period (it also displays the percent variation of average ratings during that time, but I'll talk about that in the next paragraph):


We see that the increase of new users during those 3 years has been more or less similar for every episode (median 98%), except for several noticeable peaks that suggest an "abnormal excess" of votes in some episodes. Well, as this could be just a coincidence or a passing fad, we should assess whether this "bump" has an effect on the average rating of the episode or not. A possible approach is to measure the percent variation of the average rating within period 2 and compare it with the percent variation of the rating within (base) period 1.



I've selected the episodes with a percentage of new users higher than the median (those whose bar surpasses the horizontal line that indicates 98% in the graph from above) and will calculate the percent variation of average rating during period 1 (as a reference for a "non-hacked" period) and period 2. An episode with a high percent variation (in absolute value) will be highly suspicious of having been strained, specially if it has received a suspiciously greater amount of votes. Two preliminar comments:
  • You Only Move Twice (#8.2) (marked with a in the first graph) is the episode with the highest drop in average rating between period 1 (9.2) and period 2 (5.1) (percent variation -44%). However, although having been clearly manipulated, it didn't have a vote increment of over 98% (95.8%) because at the beginning of period 2 (December 2011) this was the second most voted episode (1,102 users; after Simpsons Roasting on an Open Fire (#1.1) [2,271 users], and ahead of Marge vs. the Monorail (#4.12) [861 users] and The Springfield Files (#8.10) [834 users]) so it'd take many more abusing votes to overcome the 98% threshold.
  • Since most nine first seasons' episodes have user ratings higher than 7 (often than 8) is much more difficult to raise an average rating than to lower it (the impact of some 1-votes on an episode rated 7.9 is greater than the same number of 10-votes). Thus, episodes like Homer the Smithers (#7.17), clearly boosted (over 220% new votes between 2011 and 2014, with an average rating of 9.7), don't exhibit pronounced changes in the final average rate (barely +15% in this case).
The following graph represents, for the episodes with a greater-than-median increase in votes, the percent variation of average ratings during March 2008-December 2011 and December 2011-November 2014:


There's a notable difference between both periods. Between March 2008 and December 2011 the average rating of all the episodes (mysteriously, except for El Viaje Misterioso de Nuestro Homer (#8.9) Inaudito) hasn't varied more than 4.5% (and half of them don't even change more than 1%). This find is consistent with the expected behaviour of a non-biased voting system. However, percent variations of average ratings during the second period are much larger (60% of cases show changes higher than 10%), suggesting a fraudulent distribution of votes during this time.

I've defined arbitrary thresholds to get a list of episodes with a fabricated rating (although I believe that there are more than these):

Many episodes from seasons other than 1-9 have suspicious ratings as well, but I didn't want to elaborate so much. In conclusion, I concur that IMDB should investigate this (factual data invites to it).

HolaHola
(Edited)
Photo of Nobody

Nobody

  • 1455 Posts
  • 706 Reply Likes
[For "Marge vs. the Monorail"]
"... you'll notice that most of the female age groups
voted very low, around 2.5 average. ..."
Also, FWIW, note the differences between the "US" vs. "Non-US" rating patterns for that episode.  (Of 2184 votes up to the time of this reply, 1796 are regionally identified as US or Non-US.)   A majority of the "1" votes for that episode are from US users.

"US"   (weighted average 4.3, arithmetic mean 5.1):
http://www.imdb.com/title/tt0701173/ratings-usa
    

"Non-US"   (weighted average 9.0, arithmetic mean 8.9):
http://www.imdb.com/title/tt0701173/ratings-international
    
(Edited)
Photo of Col Needham

Col Needham, Official Rep

  • 6834 Posts
  • 4812 Reply Likes
Thanks for the detailed reports.  The appropriate team have been notified.
Photo of Col Needham

Col Needham, Official Rep

  • 6834 Posts
  • 4812 Reply Likes
This has been addressed now, thanks for the problem report. 
Photo of NekPants

NekPants

  • 3 Posts
  • 0 Reply Likes
Great, thank you!  It would be a shame to steer new fans in the wrong direction.  :)