Ratings: Give the regular voter weighting to the regular reviewers

  • 15
  • Idea
  • Updated 2 years ago
  • (Edited)
This was suggested by Surendra Mohan but needs its own thread as it could be very useful:https://getsatisfaction.com/imdb/topi... there is a concern that the weighting given to regular voters is being used for evil by a number of users by adding 1 star votes to films with a small number of votes, that they presumably think has a suspicious voting pattern, hoping to "correct" for any possibly "vote stuffing":https://getsatisfaction.com/imdb/tags... highlights a problem with the system - the weighting "rewards" people for being able to hammer in as many votes as possible. While a lot of the regular voters are film and TV buffs merrily consuming media by the bucketload there is a small minority abusing the system, either just chasing the Top 1,000 Voter status (which you can see when you check your votes, even if you don't get a badge for it) or because they think they are some kind of champion, protecting poor old IMDB from nasty vote stuffers (even if I suspect that the algorithm already corrects for this, so the regular voter weighting leads to a double whammy downgrading of the weighted averageI believe that the weighting is still important, as it gives the votes of the most knowledgeable voters more significance which would help give a consistency to the ratings of films. So I wouldn't want to just get rid of the weighting but the score bombing is a problem that needs fixing.As the title says, the suggestion is to give the weighting to Top Reviewers or regular reviewers (the equivalent number people with the highest number of reviews, in the way the regular voters seems to be a larger group than the Top 1,000 Voters, even if it is the latter than gets flagged up on the voting breakdowns, which allows us to see if the 1 stars are coming from regular voters). This would have the effect of:* Rewarding the people who take the time and effort to write reviews explaining their ratings.* Making these votes visible through the person's review history, which has the effect that it can keep people honest (if, for example, someone tried to game the system to get their weighting back by adding in reviews copy and pasted from elsewhere, it' be easy enough to spot and report) - as I've argued for other suggestions:https://getsatisfaction.com/imdb/topi...* It would also remove the incentive for people to just hammer in these 1 votes - it no longer gets you the weighting and no longer has as much impact in cases of perceived vote stuffing.So win, win, win - the triple winner.Some have suggested only allowing people to rate films when they review them (as is done on Amazon) or at least only have them count if you have a review but I don't think that'd be very useful here (it'd be no more help in addressing the score bombing than this suggestion), it would cause a riot from users, it'd undermine the idea of the wisdom of crowds that powers the IMDB rating and it might actually make vote manipulation easier (as we see from the score bombing, in most cases you can only influence a score when the votes are low).There are a few things that need to be fine-tuned:* Should the weighting apply only to ratings where they have also added a review? or:* Should the weighting apply to all of the ratingsHowever, I'm sure staff can figure that out - the former would encourage more reviews but the latter would make the weighting work better if the aim is to try and get more consistency in the voting across a range of film and TV. Personally, I'd be fine with the second option, but if there was evidence of abuse it could be switched to the second one.
Photo of Emperor

Emperor, Champion

  • 6418 Posts
  • 3004 Reply Likes

Posted 6 years ago

  • 15
Photo of Manu

Manu

  • 8 Posts
  • 1 Reply Like
Yes!!!!!
Photo of Dan Dassow

Dan Dassow, Champion

  • 13729 Posts
  • 14180 Reply Likes
This is a great idea. I will have to think about the details to refine the idea.

One possibility I've considered suggesting is that users can only rate a film 1 or 10 if they write a review. If the review is simply random text, other user could report the review and have the rating and review deleted.
Photo of Emperor

Emperor, Champion

  • 6418 Posts
  • 3004 Reply Likes
Interesting idea, but I suppose all this would do is make people post 2 and 9 stars instead ;) It might end up slightly skewing the results and I wonder if film-makers will come on here complaining that their fans now can't vote 10 for their films because they are the very best thing ever!!!
Photo of Bilal

Bilal

  • 62 Posts
  • 34 Reply Likes
"One possibility I've considered suggesting is that users can only rate a film 1 or 10 if they write a review."

I don't like this idea. IMDb Registered users can cast a vote (from 1 to 10) on every title in the database. If you make something like that, nobody will use IMDb anymore. Just you should replace Top 1000 voters with Top 1000 Reviewers.
Photo of Norry Niven

Norry Niven

  • 11 Posts
  • 0 Reply Likes
This is all very, very frustrating for us, our film dropped half based on these weird changes. It has NOT been released yet either and our score went from 8.5 to 4 in one day with no new votes...very, very frustrating.
Photo of bluesmanSF

bluesmanSF, Champion

  • 10815 Posts
  • 6429 Reply Likes
It changed after votes hit 100, which is when "weighting" now kicks in. It just didn't show until it refreshed on the page. It would'nt change without that vote moving from 99 to 100.
Photo of Doihaveto Givethis

Doihaveto Givethis

  • 8 Posts
  • 1 Reply Like
Top Reviewers -- Excellent idea!

* Weight toward Top Reviewers (or number of reviews a voter has written)

* Also weight toward the length of time a voter has been and IMDB member. Then we know they didn't just create an account to vote for one film.

* Do NOT assume that votes are padded either high or low, as you can't know that, and it's unfair when there's no vote-padding.
Photo of Emperor

Emperor, Champion

  • 6418 Posts
  • 3004 Reply Likes
Yeah, I suspect part of the problem is they adjust for vote stuffing, then someone puts in a 1 vote to adjust for vote stuffing - the two combined, plus the weighting of the Top Voter and it throws the whole system out. It is all combining to make a mess - changing it to Top Reviewers would not only remove the weight of these 1 votes, but it'd remove any incentive to do it. So it might be the vote stuffing weighting could stay, but put them all together and the system falls apart for films with low numbers of votes.
Photo of bluesmanSF

bluesmanSF, Champion

  • 10815 Posts
  • 6429 Reply Likes
Heaviest weight to those that pay a small fee...just throwing it out there as a way to kick butt over those throwing fake 1's out there.

And, yes I've had a couple beers...
Photo of Peter

Peter, Champion

  • 6438 Posts
  • 7754 Reply Likes
I disagree because I'm not a regular reviewer.
Photo of Victoria Masina

Victoria Masina

  • 36 Posts
  • 6 Reply Likes
I love the idea. It seems the fairest compared to anything else. It's a reward and compliment for people who take the time writing reviews and are serious cinephiles, and it would be easier to determine whether a voter has watched the movie or not. I don't understand the complaints of the few nay sayers and their abusive remarks regarding a new system for voting, especially now with the changes - in particular this Bilal individual who replied to a comment of mine in another thread that my movie is very bad etc. - not because he has seen it, because he hasn't, but just because he can't stand the thought of having to "allow" the average score. It's not like people suddenly compare the little films with the big ones! We understand it's a completely different league. You can see that on the amount of voters. It's the same with any other product. I take a product with a thousand or more reviews way more seriously than one with a handful. I also don't understand the weird need to be a Top 1000 voter and troll films in order to make that happen - what kind of deranged power trip is that? lol
Photo of Emperor

Emperor, Champion

  • 6418 Posts
  • 3004 Reply Likes
Yes, while my other suggestion to fix the weighting issue has been negated by developments:

https://getsatisfaction.com/imdb/topi...

I think this still has legs.

As you say, it not only becomes more difficult to spam votes onto films (especially the 1 star voters, who seem to work through lists hammering on votes - I suspect some not only haven't seen the films but may not even visit the film's entry) and partly removes the incentive (yes some people are sad enough to want to chase the Top Voter status to get more weight for their votes) but there are other checks and balances - it becomes obvious if someone is just adding filler to a review in order to hammer out cookie cutter reviews in order to get a Top Reviewer status and people have caught reviewers out copy and pasting other people reviews, which can then be reported.
Photo of bluesmanSF

bluesmanSF, Champion

  • 10815 Posts
  • 6429 Reply Likes
Yes, it's still relevant because this new change only affects the view while a film has low vote total. It's still a good idea overall.
Photo of Emperor

Emperor, Champion

  • 6418 Posts
  • 3004 Reply Likes
Good point.
Photo of Victoria Masina

Victoria Masina

  • 36 Posts
  • 6 Reply Likes
Not to display votes till there are 100 accumulated would be a way to go, but then a lot of films will never get there. Maybe blockbusters and low budget and rather unknown films should be judged in different categories, then people like Bilal or whatever the name was can't complain about the new system. It could be done by simply giving the stars a different color. If there would be a color system in place then people would understand immediately the differences and the trolls don't need to get their panties up in a bunch. Let's say the big blockbusters will keep the yellow stars. Smaller indie films with low vote count could be red and shorts could be blue. We can even add green for documentaries.That prevents people from comparing scores cross-category and should satisfy everybody's concerns.
Photo of Believe Again

Believe Again

  • 15 Posts
  • 0 Reply Likes
I like your idea - change the star color. Like you said, we can't compare big blockbusters and smaller films or shorts.
Photo of Victoria Masina

Victoria Masina

  • 36 Posts
  • 6 Reply Likes
Maybe we should put it in the suggestion box, well, make it a separate thread as an idea. I think Emperor should do this. I'm on IMDB for over 7 years but new to this get-satisfaction bb. Emperor would have more people being interested in taking a look at the suggestion and discussing it. :)
Photo of Dan Dassow

Dan Dassow, Champion

  • 13729 Posts
  • 14180 Reply Likes
Emperor,

I've liked your idea since you suggested it. It addresses a fundamental flaw with the current Top 1000 voters, providing an incentive to simply vote for a film rather than watch and thoughtfully rate it.

In addition to the Top Reviewers, we may also wish to consider the Top Contributors as part of that group. It would be reasonable to expect that someone who spends time to research, update and correct the data in IMDb would also rate films with due diligence.
Photo of Emperor

Emperor, Champion

  • 6418 Posts
  • 3004 Reply Likes
I've liked your idea since you suggested it.


Although, strictly speaking it isn't my suggestion - I just reposted Surendra Mohan's idea as it seems a good one.

Not having ever posted a review (which I will have to rectify) I don't really have a dog in this fight, which is why I can be a little more subjective on this.

In addition to the Top Reviewers, we may also wish to consider the Top Contributors as part of that group. It would be reasonable to expect that someone who spends time to research, update and correct the data in IMDb would also rate films with due diligence.


Yeah I'd be up for that (again never having troubled the Top Contributors slot, and if not this year then I never will, I don't have a dog in this fight either) - they are clearly know their onions and would be expected to provide a carefully considered vote.

If so it might be an idea to reduce the weighting, as you would be doubling the number of people given weighting (although what about the crossover between the two groups?) - this actually might be also considered an argument for this, as it does seem like the weighting given to Top Voters might be a little too much (I think another idea was to reduce the weighting for Top Voters, and this would have the benefit of also doing that).
Photo of Peter

Peter, Champion

  • 6438 Posts
  • 7754 Reply Likes
Terms like "regular voters", "top voters" and "top 1000 voters" seem to be thrown around as if people know which votes are weighted. I don't know how the weighted ratings work, or whether they work as intended, but here is IMDb's own statement on the purpose of the weighting:

"IMDb publishes weighted vote averages rather than raw data averages. Various filters are applied to the raw data in order to eliminate and reduce attempts at 'vote stuffing' by individuals more interested in changing the current rating of a movie than giving their true opinion of it."

According to this, the intention is not to weigh a narrow group of voters higher than others, but to filter out abuse. One sure consequence of weighting votes by top reviewers is that you leave thousands of voters out of the weighting who use the voting system as it was intended.
Photo of Dan Dassow

Dan Dassow, Champion

  • 13729 Posts
  • 14180 Reply Likes
pbn, I've quoted this phrase frequently:
"IMDb publishes weighted vote averages rather than raw data averages. Various filters are applied to the raw data in order to eliminate and reduce attempts at 'vote stuffing' by individuals more interested in changing the current rating of a movie than giving their true opinion of it."
I do not advocate weighing the votes from Top 1000 voters any more than any other regular user. I do, however, advocate changing the criteria for who IMDb considered a Top 1000 voter to reduce the incentive for voting titles to gain that status.

I would hope that IMDb filters out votes from user who have unusual voting patterns, such as assigning a large percentage of 1s and/or 10s.

Filtering data has the advantage of excluding outliers from the distribution. However, excluding data from very small data sets can lead to non-sensible results.
Photo of Peter

Peter, Champion

  • 6438 Posts
  • 7754 Reply Likes
The top 1000 voters are the voters who have voted for the most titles, but IMDb have said before that this specific group is given no special weighting in the voting averages.
Photo of Victoria Masina

Victoria Masina

  • 36 Posts
  • 6 Reply Likes
I don't know if you already read my idea about the color system in one of my replies - what I was saying was that maybe blockbusters and low budget and rather unknown films should be judged in different categories, then people can't complain about the new non-weighed system for little votes films. It could be done by simply giving the stars a different color. If there would be a color system in place then people would understand immediately the differences and the trolls don't need to get their panties up in a bunch. Let's say the big blockbusters will keep the yellow stars. Smaller indie films with low vote count could be red and shorts could be blue. We can even add green for documentaries.That prevents people from comparing scores cross-category and should satisfy everybody's concerns.

Maybe we should put it in the suggestion box, well, make it a separate thread as an idea. I think Emperor should do this. I'm on IMDB for over 7 years but new to this get-satisfaction bb. Emperor would have more people being interested in taking a look at the suggestion and discussing it. :)
Photo of Emperor

Emperor, Champion

  • 6418 Posts
  • 3004 Reply Likes
I think Emperor should do this. I'm on IMDB for over 7 years but new to this get-satisfaction bb. Emperor would have more people being interested in taking a look at the suggestion and discussing it. :)


Well it is nice of you to say so (although looking through my suggestions doesn't really support that ;) ) but really it is about the quality of the idea and how many people like it, not the person who started it. So go for it.
Photo of Victoria Masina

Victoria Masina

  • 36 Posts
  • 6 Reply Likes
I did it! Baaahhhh, I'm scared... I'm not feeling in a controversy mood right now, I hope people will like it :) https://getsatisfaction.com/imdb/topi... ... nervous...
Photo of Giancarlo Cairella

Giancarlo Cairella, Official Rep

  • 1108 Posts
  • 1042 Reply Likes
I'm marking this idea as 'under consideration' because there isn't a better choice in the system but the truth is that since we do not disclose if, how and when we make changes to the weighting system, even if we do something like this we probably wouldn't say that we did :-)

Let's just say that all good ideas about improving the rating system are carefully weighted (no pun intended) when we make changes.
Photo of Emperor

Emperor, Champion

  • 6418 Posts
  • 3004 Reply Likes
Say no more, nudge, nudge, wink, wink.
Photo of Victoria Masina

Victoria Masina

  • 36 Posts
  • 6 Reply Likes
I posted my color system idea here: https://getsatisfaction.com/imdb/topi...

Please let me know what you all think. I am trying to come up with a solution that everybody will be happy with, because I'm worried there will be too many unhappy people with the new scoring system (non-weighed for low vote films) and they'll change it back. I also don't want 100 votes being the cut-off point and then it switches back to weighing. That's still too few votes IMO. With the color system we can keep the non-weighed up to a thousand or for ever. :)
Photo of Emperor

Emperor, Champion

  • 6418 Posts
  • 3004 Reply Likes
When trying to divine the motivation of the 1 star spammers I came up with a couple of suggestions:

there is a small minority abusing the system, either just chasing the Top 1,000 Voter status (which you can see when you check your votes, even if you don't get a badge for it) or because they think they are some kind of champion, protecting poor old IMDB from nasty vote stuffers


However, during the discussion on fake reviews:

https://getsatisfaction.com/imdb/topi...

I did some checking and found someone offering fake IMDB reviews:

http://www.prfastmarketing.com/imdb-m...

What really struck me though was:

We are offering REAL WEIGHTED VOTES


Which feels like the last piece of the puzzle fitting into place - some people are hammering through films throwing out 1 star votes in order to get weighted votes, that they can then sell in batches.

This makes it more important to address the weighting of votes. Of course, this situation could lead to more fake reviews, but, as I say (here and over on the other thread) the reviews are public and easier to analyse for suspect behaviour (I'm sure Amazon are working on detecting this kind of thing).
Photo of Gordon Michaels

Gordon Michaels

  • 17 Posts
  • 3 Reply Likes
The fact that a cohort of TOP 1000 voters are voting every film and TV show a 1 (out of 10) is a clear sign that this is automated voting using the same piece of Bot software. If this voting was originating without collusion from multiple independent voters who wanted to achieve TOP 1000 voter status, then some people would be rating everything a 5 or a 6 or a 7 (which would certainly be less conspicuous) rather than ranking everything a 1.

So, the problem with the voting on IMDB is not just a group of independent voters all trying to achieve TOP 1000 Voter status. There is an element of cooperation or collusion in what is occurring. That is considerably more . . . sinister,

This type of malware voting should be easily delectable by IMDB staff in many ways, and the offending voters should be tagged or badged such that their votes are given zero weight in any average votes that are posted by IMDB.

It is not appropriate to ignore this problem because "TOP 1000 Voters are not weighed more heavily than any other voters." First of all, IMDB posts the average vote of TOP 1000 voters. If IMDB does not think the average vote of TOP 1000 Voters is important enough to fix, then it is not important enough to post in the first place. Basic business rule: no organization should post information that they believe to be corrupted by malware.

Secondly, many large sites, including video game sites, have on-going staff efforts to detect and prevent instances of automated software (BOTS) interacting with their sites. It has become a standard practice, and if IMDB has not already adopted some anti-BOT strategy then you need to consider doing it. The trust and respect of your user community is a priceless asset, and you should guard it zealously.

I apologize for sounding preachy, and hope IMDB is already on top of this problem.
Photo of Jennifer O'Neill

Jennifer O'Neill

  • 11 Posts
  • 13 Reply Likes
Bumping this thread, since the problem described by @Emperor is still very much alive, as noted here: https://getsatisfaction.com/imdb/topics/numerous_fake_glowing_reviews_for_otherwise_panned_film?utm_...
Photo of bluesmanSF

bluesmanSF, Champion

  • 10815 Posts
  • 6429 Reply Likes
Yes..they'd need to correct the "fake review" issue before giving those folks "regular user" status so they could then control ratings more. Without getting rid of those accounts and somehow blocking them from coming back, giving them more power doesn't help anything. It's kind of a two pronged issue. Thanks for mentioning the other here, Jennifer!
Photo of Stan Guingon

Stan Guingon

  • 14 Posts
  • 1 Reply Like
This thread has had a lot of responses but there is NOT ONE PERSON from IMDB making a comment on it... and please don't post that FAQs link to give an answer.

It's all generic answers and the top people from IMDB should not be turning thier heads away from so many complaints!!!
Photo of Glover Steve

Glover Steve

  • 2 Posts
  • 0 Reply Likes
Our little documentary in very limited release (currently in Australia only), has a total of 38 votes, two thirds of these have been 6 star ratings or more.

It has received 4  1 star votes, all of these have been from top 1000 voters, two of who are US voters where the film has never been seen. Is it a coincidence that only IMDB top 1000 voters have rated the film 1 star?
(Edited)
Photo of Believe Again

Believe Again

  • 15 Posts
  • 0 Reply Likes
I have an exact same experience. That is harassing vote. I got 40 1star votes. 40 1 STARS for a small film!!! Can you believe it? My film was premiered only in the U.S. But I got a lot of 1 stars from non-US user. Obviously harassing vote. I found out most of votes are mainly composed of friends/fan vote and harassing vote, especially in case of small films. Because of the harassment, no future for novice film makers. Sucks rating system.
I also have an experience that one day Amazon delete all good reviews and leave only one bad review on my book. After the case, no one can give a review on my book, even not good review. I often hear that Amazon delete a lot of earnest reviews with gonzo reason. We should not believe "stars". Most of them are crooked!!! No rating is fine anymore. "Star" is no meaning.
Photo of Believe Again

Believe Again

  • 15 Posts
  • 0 Reply Likes
One more thing. I don't feel good, nay, feel very bad of course, but I noticed spending times to bother about rating is waste of time. What we can do is just concentrate to make better things, see future and ignore the sucks vulgar rating...!