Advanced Title Search page problem

  • 1
  • Problem
  • Updated 4 weeks ago
  • In Progress
The Advanced Title Search results are shown in pages of 50 movies.  Page 2 onward appears to have a problem.  Page 2 contains results already shown in page 1.  Page 3 contains results already shown in pages 1 and 2.  And so on.

For example, take the following search
https://www.imdb.com/search/title/?title_type=feature&year=2007,2019&num_votes=1000,&use...

For me, page 1 contains the following results:
43. Zombieland (2009)
44. Ex Machina (2014)
45. John Carter (2012)
...
49. Spectral (2016)
50. Guardians of the Galaxy (2014)

I then click "Next »" and proceed to page 2 (results 51 - 100).

The first 5 results shown on page 2 are:
51. Zombieland (2009)
52. Ex Machina (2014)
53. John Carter (2012)
54. Spectral (2016)
55. Guardians of the Galaxy (2014)

Further, if I refresh page 2 the results completely change to what I believe is the correct results (i.e. no repeats of page 1 movies):
51. The Hunger Games (2012)
52. The Mist (2007)
53. Transformers (2007)
54. Prospect (2018)
55. The Purge: Anarchy (2014)

If I then refresh page 2 a second time the results return to the incorrect list (i.e. Zombieland, Ex Machina, etc.).

Continuing to refresh page 2 reveals that the results switch back and forth between the incorrect and correct results alternately.

If you try this on page 3 it gets even stranger.  I have so far counted 4 distinctly different pages of results for page 3.  I have not tried this on page 4 onward.
Photo of Craig

Craig

  • 14 Posts
  • 15 Reply Likes

Posted 1 month ago

  • 1
Photo of Oswald

Oswald

  • 1428 Posts
  • 2426 Reply Likes
Yes, I have noticed this.

It is to do with the "order by" criterion - the bug is most noticeable when there are a small number of options to be ordered by ... such as "my rating" which is just one of 10 discrete values.

When the database is queried, the results of the query are not cached in any way - leading to the "volatile" results you see.

Sometimes I wonder where iMDB finds its backend developers.
(Edited)
Photo of bderoes

bderoes, Champion

  • 2447 Posts
  • 4057 Reply Likes
The sort criteria on this link is Your Rating, meaning YOURS, so no one else will see these results precisely. 

I have to do a different query to replicate your problem (changed the genre and lifted the date restriction). For me the titles that change their sequence have the same rating from me. So, really, they ARE interchangeable in this sort.

I did observe both that some of them repeated from page 1 to page 2, and the refresh of either page changing, then changing back. (I used Compact mode, btw.) If anything, I find it amazing the refresh doesn't have even MORE variety.

Is it possible we only see "duplicates" when we've done a page refresh (or use the browser's Fwd/Back links)? I just opened the Next page in a new tab, and do NOT see duplicate titles there.
Photo of Oswald

Oswald

  • 1428 Posts
  • 2426 Reply Likes
Nope, the problem still occurs.

(I used the same query as the OP, since I've rated plenty of titles that meet his criteria.)

The duplication issue still occurs when using opening "Next" in a new tab:

From page 1:



And again on page 2:

Photo of Craig

Craig

  • 14 Posts
  • 15 Reply Likes
I also see duplicates when I open the page 2 in a new tab.
Photo of bderoes

bderoes, Champion

  • 2447 Posts
  • 4057 Reply Likes
Wow.
What if you increase the Count to 100 (and therefore only have 6 pages of results; just append &count=100 to the url), do you still see dupes? And does the 6th page have just 11 titles (total retrieved is 511 right now)? 'Cause if you only see 511 total, but it displays duplicates, then are you actually seeing all the results?
 
Does the same thing happen if you restrict to only titles you've Seen? Here's the parameter, so you don't have to start from scratch:
&my_ratings=restrict
Photo of Oswald

Oswald

  • 1428 Posts
  • 2426 Reply Likes
Rose, I have done this before while restricting to only titles I have seen; and the bug still appears.

But I'll demonstrate it anyway, give me a sec.
Photo of Craig

Craig

  • 14 Posts
  • 15 Reply Likes
Thanks for your responses.

The titles that are repeating for me all have the same rating. So, as you say, technically they are interchangeable in this sort.  My point is though that really they shouldn't be interchangeable.  The search results should be consistent every time.

I'm a software developer, so reading between the lines here what I believe is occurring is the SQL "order by" clause is missing one or more "default" criterion that should added when certain sort options are chosen.

For example, selecting sort by "Your rating" should also silently (i.e. in the programming background, NOT in the URL) add "Date of your rating".  So the SQL "order by" clause would end up something like "order by my_ratings desc, your_rating_date desc".

Doing this would ensure that titles that all have the same rating are always sorted the same - therefore consistent results.

(Notice the inconsistent prefixes of "my_" and "your_" even though they both refer to the user)
Photo of Oswald

Oswald

  • 1428 Posts
  • 2426 Reply Likes
Yes Craig, teach them.

(Though using the title's primary key - it's unique title ID - would be better, as some users rate multiple movies on the same day.)
Photo of Craig

Craig

  • 14 Posts
  • 15 Reply Likes
Ha!  You beat me to it.
Photo of Oswald

Oswald

  • 1428 Posts
  • 2426 Reply Likes
I'm retired.

Good to know I have not totally lost it - yet. :)
Photo of Craig

Craig

  • 14 Posts
  • 15 Reply Likes
I've just realised that my example solution above could also result in the same problem, although probably on a smaller scale.

If you happen to rate 10 different movies with the same rating on the same day then those 10 will be interchangeable in the search results.

So, in my example there should probably be a third "default" criterion added which might be "A - Z".  So the SQL "order by" clause would end up something like "order by my_ratings desc, your_rating_date desc, alpha desc".
Photo of Ed Jones(XLIX)

Ed Jones(XLIX)

  • 23311 Posts
  • 27775 Reply Likes
I just realized that my brain was not having deja-vu!
I have had this problem the past week also.
Photo of Oswald

Oswald

  • 1428 Posts
  • 2426 Reply Likes
It has existed for years.

I just don't bother to complain about these things on the boards.
Photo of Oswald

Oswald

  • 1428 Posts
  • 2426 Reply Likes
Our entire website and nearly all of our internal-use tools are created with open source software such as Apache, and the usual collection of GNU and Linux utilities. The software that runs the database itself was completely developed in-house.  

^^ with an unstable underlying sorting algorithm, likely some variant of randomized quicksort.

Craig, look what you started.
(Goes back to my drinking.)
Photo of Craig

Craig

  • 14 Posts
  • 15 Reply Likes
Oh, Oswald, you're making me laugh.  :-)

Photo of Craig

Craig

  • 14 Posts
  • 15 Reply Likes
So... (speaking to the IMDb support staff in general)...

Is there any likelihood that someone from IMDb will decide to update the code to fix this issue?

Or will it remain no more than a dream?
Photo of Taylor

Taylor, Official Rep

  • 767 Posts
  • 625 Reply Likes
Hi Craig,

Thank you for take the time to flag this. I've forwarded this information to our technical team for further investigation.
Photo of Craig

Craig

  • 14 Posts
  • 15 Reply Likes
Thank you Taylor.

I'm happy to assist with testing if needed.