Netflix Prize: Forum

Forum for discussion about the Netflix Prize and dataset.

You are not logged in.

Announcement

Congratulations to team "BellKor's Pragmatic Chaos" for being awarded the $1M Grand Prize on September 21, 2009. This Forum is now read-only.

#1 2007-11-13 14:58:51

prizemaster
Administrator
From: Netflix HQ
Registered: 2006-08-29
Posts: 181
Website

Netflix Progress Prize 2007 awarded to team KorBell

It is our great honor to announce the winner of the Netflix Progress Prize for
2007 to team KorBell (aka BellKor) for their verified just-in-time submission on
Oct 1 at 23:29:20 UTC achieving an 8.43% improvement over Cinematch.  We
congratulate the team of Yehuda Koren, Robert Bell, and Chris Volinsky of AT&T
Research Labs for their superb work integrating many significant techniques to
achieve this result.

In accord with the Rules Yehuda and the team have prepared a system description
paper, which we both make public below.  We will be awarding the Prize in a
presentation in at the Netflix offices in Los Gatos on November 19th, 2007 at 4pm. 
Yehuda Koren will present a public talk at that time about their Prize algorithm. We
will post a video of that presentation via the Forum.

We have also updated the Prize leaderboard to reflect the award of the 2007
Progress Prize and have established the new accuracy requirement to qualify for
the 2008 Progress Prize.  Again, in accord with the Rules, the new Prize level
reflects a 1% improvement over team KorBell's verified submission, requiring a
9.34% improvement over the original Cinematch accuracy level.

This Prize and, of course, the Grand Prize remain available to all qualified
teams.  We are pleased to note team KorBell continues to submit prediction sets
as do many other teams.  We look forward to everyone learning from what team
KorBell has achieved and working to exceed the Prize levels.  Good luck!

The KorBell paper submitted to the judges can be found below.  This
paper builds on, and requires familiarity with, several previous papers
published by the team.  Those papers are available below or via the team
KorBell website.

R. Bell, Y. Koren, C. Volinsky, "The BellKor solution to the Netflix Prize", (2007).

R. Bell, Y. Koren, C. Volinsky, "Modeling relationships at multiple scales to improve accuracy of large recommender systems",
Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD'07), ACM Press (2007).

R. Bell, Y. Koren, "Improved Neighborhood-based Collaborative Filtering",
KDD Cup and Workshop (KDD'07), ACM Press (2007).

R. Bell, Y. Koren, "Scalable Collaborative Filtering with Jointly Derived Neighborhood Interpolation Weights"
IEEE Conference on Data Mining (ICDM'07), IEEE (2007).

Offline

 

#2 2007-11-28 20:30:23

winsteps
Member
From: Chicago
Registered: 2006-10-03
Posts: 269
Website

Re: Netflix Progress Prize 2007 awarded to team KorBell

Congratulations, KorBell!
What was the lowest RMSE reported here that would indicate the best that Netflix could obtain on a brand-new set of predictions for data that has yet to be collected? This would not have any blending or such like.

Offline

 

#3 2007-11-29 07:03:54

YehudaKoren
Member
Registered: 2007-09-23
Posts: 54

Re: Netflix Progress Prize 2007 awarded to team KorBell

Thanks Winsteps.
It is good to see that you are still lurking around here.
Sorry, but I couldn't understand your question.

Offline

 

#4 2007-11-29 13:38:12

winsteps
Member
From: Chicago
Registered: 2006-10-03
Posts: 269
Website

Re: Netflix Progress Prize 2007 awarded to team KorBell

Thanks for the response YehudaKoren. Am returning after a long intermission on other projects, but now have a Conference Paper to present in January 2008 about the Netflix Prize, so am trying to catch up ...

The KorBell PDF has about 107 RMSEs. Many are the result of blending or taking advantage of previous Quiz RMSEs or the probe dataset.

In real-life prediction, we have no previous Quiz RMSE nor probe dataset. My question is "Which is the best of your 107 RMSEs that corresponds to real-life prediction?"

In other words, the prediction method you would use if Netflix said, "Here's a new Prize dataset everyone. There's no probe dataset, and you only have one submission allowed. (Its RMSE is kept secret till the competition closes). Best RMSE wins!"

Last edited by winsteps (2007-11-29 14:01:35)

Offline

 

#5 2007-11-29 14:31:32

YehudaKoren
Member
Registered: 2007-09-23
Posts: 54

Re: Netflix Progress Prize 2007 awarded to team KorBell

I think that the ensemble approach is practical for real life situations.
It allows concentrating on relatively simple models, which are fast to code and to run. The result is also more robust against programming bugs and overfitting.
Since we are measured on the Test set (not the Quiz or Probe), the situation is not that different than the one you suggest.

I will not recommend just one of the predictors. You want at least to explain the data at multiple scales (local+regional) and also to taste the "binary view".

Anyway, using 107 predictors is probably overkill for real life situations. There are diminishing returns. Now, we know how to reach improvement beyond 8% using 8 predictors. In fact, the 107 results mostly reflected an ongoing research, and most of them were accumulated because they do no harm, but also no good...  We have found that RMSE=0.8712 could be achieved with less than 50 predictors.

Best,
Yehuda

Offline

 

#6 2007-12-10 09:15:59

Curious George
Member
Registered: 2007-12-10
Posts: 1

Re: Netflix Progress Prize 2007 awarded to team KorBell

"We will be awarding the Prize in a
presentation in at the Netflix offices in Los Gatos on November 19th, 2007 at 4pm. 
Yehuda Koren will present a public talk at that time about their Prize algorithm. We
will post a video of that presentation via the Forum"

Is the video available already?

Offline

 

#7 2007-12-13 07:45:25

prizemaster
Administrator
From: Netflix HQ
Registered: 2006-08-29
Posts: 181
Website

Re: Netflix Progress Prize 2007 awarded to team KorBell

We are working on some quality issues with the video.  It should be available "soon".

Offline

 

#8 2008-01-17 04:03:24

winsteps
Member
From: Chicago
Registered: 2006-10-03
Posts: 269
Website

Re: Netflix Progress Prize 2007 awarded to team KorBell

Thank you, YehudaKoren, for your explanation. Such pains-taking diligence ....

Offline

 

#9 2008-02-07 08:25:46

prizemaster
Administrator
From: Netflix HQ
Registered: 2006-08-29
Posts: 181
Website

Re: Netflix Progress Prize 2007 awarded to team KorBell

At long last we make available a recording of Yehuda's Netflix Progress Prize 2007 acceptance talk at the Netflix offices in Los Gatos.  We apologize for the delay, which is solely our problem.  We tried to recover the few, general questions that were asked of Yehuda in the public session but that part of the recording was of poor quality and so, unfortunately, it is not included. We are still trying to recover the Q&A and will post it here if we are successful.  Meanwhile, enjoy Yehuda's presentation.

Offline

 

#10 2008-03-21 17:52:04

erdal
Member
Registered: 2008-03-21
Posts: 2

Re: Netflix Progress Prize 2007 awarded to team KorBell

Is it just me or the video does not have sound? Can I find this video somewhere else?

Thank you!

Offline

 

#11 2008-03-22 08:40:47

prizemaster
Administrator
From: Netflix HQ
Registered: 2006-08-29
Posts: 181
Website

Re: Netflix Progress Prize 2007 awarded to team KorBell

The video does have sound.

Offline

 

#12 2008-03-23 13:43:33

erdal
Member
Registered: 2008-03-21
Posts: 2

Re: Netflix Progress Prize 2007 awarded to team KorBell

That was a problem on my browser. I can hear it now. Sorry about that!

Offline

 

#13 2008-09-07 23:29:03

EnjoyYoung
Member
Registered: 2008-09-05
Posts: 2

Re: Netflix Progress Prize 2007 awarded to team KorBell

I'am sorry,but some papers about your work aren't available for free.

Offline

 

#14 2008-10-06 06:51:51

dinc3r
Member
From: evden eve nakliyat
Registered: 2008-10-06
Posts: 1
Website

Re: Netflix Progress Prize 2007 awarded to team KorBell

The KorBell PDF has about 107 RMSEs. Many are the result of blending or taking advantage of previous Quiz RMSEs or the probe dataset
good thenkss

Offline

 

Board footer

Powered by PunBB
© Copyright 2002–2005 Rickard Andersson