Netflix Prize: Forum

Forum for discussion about the Netflix Prize and dataset.

You are not logged in.

Announcement

Congratulations to team "BellKor's Pragmatic Chaos" for being awarded the $1M Grand Prize on September 21, 2009. This Forum is now read-only.

#1 2008-12-10 05:35:09

prizemaster
Administrator
From: Netflix HQ
Registered: 2006-08-29
Posts: 181
Website

Netflix Progress Prize 2008 awarded to team BellKor in BigChaos

It is our great honor to announce the winner of the Netflix Progress Prize for 2008 as team BellKor in BigChaos for their verified just-in-time submission on Sept 30 at 21:17:40 UTC achieving a 9.44% improvement over Cinematch.  We congratulate the team of Yehuda Koren, Robert Bell and Chris Volinsky of AT&T Research Labs combined with Andreas Töscher and Michael Jahrer of Commendo Research for their superb work integrating many significant techniques to achieve this result.

In accord with the Rules the team has prepared a system description consisting of two papers, which we both make public below.  We will be awarding the Prize in a presentation at the Netflix offices in Los Gatos on December 17, 2008 at 4pm.  Andreas Töscher and Michael Jahrer will present a public talk at that time about their Prize algorithm. We will post a video of that presentation via the Forum.

We have also updated the Prize leaderboard to reflect the award of the 2008 Progress Prize. Again, in accord with the Rules, the sole remaining Prize level is the Grand Prize reflecting a 10% improvement over the original Cinematch accuracy level.

The Grand Prize remains available to all qualified teams.  We are pleased to note team BellKor in BigChaos continues to submit prediction sets as do many other teams.  We look forward to everyone learning from what team BellKor in BigChaos has achieved and working to exceed the Grand Prize level.  Good luck!

The BellKor in BigChaos papers submitted to the judges can be found below.  These papers build on, and require familiarity with, several previous papers published by the teams. Those papers are available below or via the team BellKor website.

R. Bell, Y. Koren, C. Volinsky, "The BellKor 2008 Solution to the Netflix Prize", (2008).

A. Töscher, M. Jahrer, “The BigChaos Solution to the Netflix Prize 2008", (2008).

A. Töscher, M. Jahrer, R. Legenstein, " Improved Neighborhood-Based Algorithms for Large-Scale Recommender Systems"SIGKDD Workshop on Large-Scale Recommender Systems and the Netflix Prize Competition  (KDD’08) , ACM Press (2008).

Offline

 

#2 2008-12-10 07:58:10

Aron
Member
Registered: 2006-10-02
Posts: 186

Re: Netflix Progress Prize 2008 awarded to team BellKor in BigChaos

oh goody! Been waiting for this. Dying to see if there were model breakthroughs or just brute force ensemble engineering.

/ Runs off to get coffee..

Offline

 

#3 2008-12-10 09:35:29

CS1
Member
From: San Jose, CA
Registered: 2006-10-02
Posts: 151

Re: Netflix Progress Prize 2008 awarded to team BellKor in BigChaos

Congratulations!

I'm very impressed that with just ~22 "basic" models BigChaos managed to beat Cinematch.  My favorite variable is "stringlengthMovie Title", for a once basis point improvement.  smile  In fact, that 1 BP improvement is right at the dividing line between RMSE = Cinematch and RMSE < Cinematch.  I wonder if the ordering in the table is an inside joke.  wink

Leave no BP unturned.  big_smile  Good luck!

CS1

Last edited by CS1 (2008-12-10 09:36:42)

Offline

 

#4 2008-12-10 10:01:32

Aron
Member
Registered: 2006-10-02
Posts: 186

Re: Netflix Progress Prize 2008 awarded to team BellKor in BigChaos

It doesn't surprise me that many of the benefits are coming from the time dimension. I've been suspecting this would be the last frontier given it had not been addressed significantly last year, and has some excellent common sense justification. I found a decent approach there myself, though its not clear yet to me if the ones they have (which are different) are superior. My challenge is figuring out how to logically blend models that may or may not have been made time-sensitive. I think if you get lower RMSEs with time-sensitive models, the blending power of others in your ensemble gets weaker rather rapidly.

It strikes me as interesting how little these teams come at these problems from a 'probabilistic' standpoint. Putting together models via gradient descent is much simpler I suppose, as I don't even really know how one converts some of these more complicated models to a probabilistic formula.

Offline

 

#5 2008-12-11 10:46:35

Newman!
Member
From: BC, Canada
Registered: 2006-12-26
Posts: 168
Website

Re: Netflix Progress Prize 2008 awarded to team BellKor in BigChaos

CS1 wrote:

Congratulations!

I'm very impressed that with just ~22 "basic" models BigChaos managed to beat Cinematch.  My favorite variable is "stringlengthMovie Title", for a once basis point improvement.  smile  In fact, that 1 BP improvement is right at the dividing line between RMSE = Cinematch and RMSE < Cinematch.  I wonder if the ordering in the table is an inside joke.  wink

Leave no BP unturned.  big_smile  Good luck!

CS1

They should translate movie titles to different languages and run the same algorithm. This will recover different aspects of the data. I suspect German, with its long words, is very suitable to this algorithm.


When you control the mail, you control... information !

Offline

 

#6 2009-01-04 04:02:41

pds
Member
Registered: 2009-01-04
Posts: 3

Re: Netflix Progress Prize 2008 awarded to team BellKor in BigChaos

I have a question, perhaps for prizemaster. I see the winning team used huge number of Quiz set RMSE's to dramatically improve their scores. It appears that without this final blending the prize level would not have be achieved (by far, which is surprising).

Is there a discussion or an official statement from Netflix about "legality" of the training on the Quiz set RMSE's? The 2008 prize seem to indicate it is OK.

Note that allowing for the Quiz set RMSE's to be used for training ("blending") implies  allowing submissions whose sole purpose is to find out the rating structure of the Quiz set. For example, there is no way to infer correlation between predictor ratings and the Quiz set ratings needed for linear regression blending used in the 2008 winner solution without having submissions whose sole purpose was to get required info from the Quiz set. So looks like hacking the Quiz set rating structure through for-that-purpose-designed submissions is a fair game, or maybe I am missing something?

Offline

 

#7 2009-01-04 11:24:19

YehudaKoren
Member
Registered: 2007-09-23
Posts: 54

Re: Netflix Progress Prize 2008 awarded to team BellKor in BigChaos

Pds,

I see that you simultaneously posted three similar variants of this question. I will try to answer here shortly those posts.

pds wrote:

How important is it to blend with Quiz set RMSE's (as 2008 winning team did) as opposed to use the probe set for blending? Any quantitative idea?

If you read the Progress Prize report carefully, you will understand that Probe blending allows possibilities that are significantly more accurate than Quiz blending. So if you start from scratch, I would say, go with Probe blending which will yield better results.

pds wrote:

If this is a way to win then this gives huge advantage to teams who accumulated high number of submissions, which puts latecomers at significant disadvantage. I wonder if there is a good way to level the playing field there without Netflix making public the actual structure of the Quiz set for all to be able to train on it...

Yes, latecomers are put in a different situation than older competitors. They do not need to invent all those ideas which were published by many papers by the early comers, who actually had to work hard to get them right while sifting through many details and less useful ideas. As a side result we accumulated some predictors, of course we don’t complain on this :) However, if you are a latecomer you start at a far more favorable point that what we used to have.

Yehuda

Offline

 

#8 2009-01-04 16:03:28

Petar Simic
Member
Registered: 2006-11-08
Posts: 2

Re: Netflix Progress Prize 2008 awarded to team BellKor in BigChaos

Yehuda,

I agree that the latecomers indeed have more info to start with as result of generosity of those who shared their methodology, as your team did. I join the expression of gratitude to your team and all the others who shared their ideas so far.

This would be the whole story and a very positive one *if* the game of winning the context would remain the game of discovery of a better forecast model. However, by allowing the Quiz set to be used in training (which is what the 2008 blending winner does), Netflix allowed and is risking the game to be changed from the one of developing significantly new data mining and forecast methodology, to the one of exploring if the prize can be won at this point in time by a short-cut: the data fitting the Quiz set by an assemble of improved (but still grossly insufficient) models.

I wish they kept the original spirit of the challenge intact and found some other way of dealing with a possible shortage of new ideas or insufficient information in the original training set.

I do not know if you agree with this assessment and if you would like to share your thoughts on this here...

Also, I am curious:

(1) what was the best score that your team achieved on models that where trained exclusively with the training set (and probe set, of course), that is, where the Quiz set info was not used in any way? (the result of 0.8643 quoted in your paper seems already a result of a blend with Quiz set RMSE's as target, so this is not it, no?)

(2) Did your 2007 entry also blend using the Quiz set as a target, or was it constrained to blending using the training & probe set only?

Once more, thank you and congratulations on your team's progress so far!

Petar.

Offline

 

#9 2009-01-05 01:55:55

gavin
Member
Registered: 2006-10-17
Posts: 53

Re: Netflix Progress Prize 2008 awarded to team BellKor in BigChaos

Petar

Petar Simic wrote:

Yehuda,

This would be the whole story and a very positive one *if* the game of winning the context would remain the game of discovery of a better forecast model. However, by allowing the Quiz set to be used in training (which is what the 2008 blending winner does), Netflix allowed and is risking the game to be changed from the one of developing significantly new data mining and forecast methodology, to the one of exploring if the prize can be won at this point in time by a short-cut: the data fitting the Quiz set by an assemble of improved (but still grossly insufficient) models.

Petar.

I think a point may have been missed here.  The competition has been very carefully designed.  The submission scores that are reported are not the scores that are used to calculate the rmse for the $1million prize or for the progress prizes.  If you look (carefully) at the rules, Netflix only reports the score that was achieved on half the submission.  The scores that are used to calculate for the prizes are the scores on the other half (not reported) of the submission.

The scores that are reported are, however, the scores that are used to trigger the run-off, so as soon as someone hits 10% on the reported scores, we all have one month to try and get as good a result as possible. 

This leaves open the possibility that someone could (theoretically) game their way to the finish post and then not have a good enough score to claim the $1million prize.  I'm not sure what happens in that situation.

I did some experiments a while back on my method of combining the submissions using the quiz results and concluded that I would have to have an rmse of .0004 or so better than the target result to be reasonably confident that  I could achieve the target RMSE on the hidden portion.  Others will have to do their own experiments.

So I think the story remains a very positive one.

Gavin (Just a guy in a garage)

Offline

 

#10 2009-01-05 02:50:45

YehudaKoren
Member
Registered: 2007-09-23
Posts: 54

Re: Netflix Progress Prize 2008 awarded to team BellKor in BigChaos

Petar Simic (or pds),

My personal opinion is that this competition has kept a good balance between scientific novelty and lower level tricks. Might be luck, but more probably thanks to a good design of the competition and the dataset.

Regarding your other questions, as I answered earlier, now we know that linear Quiz blending is inferior to the more sophisticated schemes possible only by Probe blending (as far as I know). This makes Quiz blending a non-issue and those questions less relevant imho.

Regards,
Yehuda

Offline

 

#11 2009-01-05 13:43:24

Lazy Monkey
Member
Registered: 2007-12-13
Posts: 93

Re: Netflix Progress Prize 2008 awarded to team BellKor in BigChaos

If Netflix sees a divergence between the quiz and test scores of the leading teams they will know that the teams are tuning against the Oracle.

Offline

 

Board footer

Powered by PunBB
© Copyright 2002–2005 Rickard Andersson