Netflix Prize: Forum

Forum for discussion about the Netflix Prize and dataset.

You are not logged in.

Announcement

Congratulations to team "BellKor's Pragmatic Chaos" for being awarded the $1M Grand Prize on September 21, 2009. This Forum is now read-only.

#1 2009-08-06 18:31:12

prizemaster
Administrator
From: Netflix HQ
Registered: 2006-08-29
Posts: 181
Website

Netflix Prize 2 (Yes, a sequel!)

This is Neil Hunt, Chief Product Officer at Netflix.

To everyone who participated in the Netflix Prize: You've made this a truly remarkable contest and you've brought great innovation to the field. We applaud you for your contributions and we hope you've enjoyed the journey. We look forward to announcing a winner of the $1M Grand Prize in late September.

And, like so many great movies, there will be a sequel.

The advances spurred by the Netflix Prize have so impressed us that we’re planning Netflix Prize 2, a new big money contest with some new twists.

Here’s one: three years was a long time to compete in Prize 1, so the next contest will be a shorter time limited race, with grand prizes for the best results at 6 and 18 months.

While the first contest has been remarkable, we think Netflix Prize 2 will be more challenging, more fun, and even more useful to the field.

Stay tuned for more details when we announce the winners of Prize 1 in September.

--

Additional information added March 12, 2010: Please see the Netflix Prize Update.

Last edited by prizemaster (2010-03-12 09:53:48)

Offline

 

#2 2009-08-06 18:37:13

Aron
Member
Registered: 2006-10-02
Posts: 186

Re: Netflix Prize 2 (Yes, a sequel!)

Just when I thought I was out, they pull me back in...

http://www.youtube.com/watch?v=UPw-3e_pzqU

Last edited by Aron (2009-08-06 18:38:09)

Offline

 

#3 2009-08-06 19:46:16

Newman!
Member
From: BC, Canada
Registered: 2006-12-26
Posts: 168
Website

Re: Netflix Prize 2 (Yes, a sequel!)

Please, no more ! I don't want to spend more time on these contests, but I probably can't resisit !

Please just hire some programmers to hack it out.


When you control the mail, you control... information !

Offline

 

#4 2009-08-06 20:00:31

CS1
Member
From: San Jose, CA
Registered: 2006-10-02
Posts: 151

Re: Netflix Prize 2 (Yes, a sequel!)

"The Sparse Matrix: Reordered" and "The Sparse Matrix: Factorizations".  big_smile

I can see it now: they will release the data for the users from 2006-2009.  smile 

Seriously, this is great news.  It's also very interesting vis-a-vis everyone else who's been thinking of having a similar competition: they now have to compete with Netflix, which is souping up their competitions.

Offline

 

#5 2009-08-06 20:34:21

Bold Raved Tithe
Member
Registered: 2006-11-17
Posts: 115

Re: Netflix Prize 2 (Yes, a sequel!)

That's good news. I was having withdrawal symptoms and I was looking for a some new competitions to fulfill my needs smile

Offline

 

#6 2009-08-07 04:33:40

quadcore
Member
Registered: 2008-03-30
Posts: 40

Re: Netflix Prize 2 (Yes, a sequel!)

I think the most interesting question from a business perspective would be "Who will cancel the service". The "churn" rate for a business has a huge impact on profitability. Knowing who is likely to cancel will allow targetted offers to retain customers.

-Richard Epstein (quadcore)

Offline

 

#7 2009-08-07 06:16:22

chef-ele
Member
Registered: 2006-10-31
Posts: 124

Re: Netflix Prize 2 (Yes, a sequel!)

Well, kudos to Netflix for continuing to spur innovation.  I certainly hope the problem for NFP2 is significantly different, and that it uses a new dataset and a new success metric (instead of RMSE).  All of these would keep the technical problem fresh & interesting and force the use of new algorithms.   

The time-limits of the next contest are also a great idea. Deadlines make interesting things happen, as we saw in NFP1.

Anyway,  I also think I need to rest for a while first, before before even considering NFP2!

Offline

 

#8 2009-08-07 12:41:20

greenmatter
Member
Registered: 2009-08-07
Posts: 1

Re: Netflix Prize 2 (Yes, a sequel!)

Aron wrote:

http://www.photosnag.com/img/4420/n09x0302vnsn/clear.gifhttp://www.photosnag.com/img/4210/n09x0302vnsn/clear.gifJust when I thought I was out, they pull me back in...

http://www.youtube.com/watch?v=UPw-3e_pzqU

LMAO, bring on the battle smile

Offline

 

#9 2009-08-07 12:46:45

HaPpY
Member
Registered: 2009-01-23
Posts: 6

Re: Netflix Prize 2 (Yes, a sequel!)

I posted a thread requesting this continuation. You all have me to thank big_smile

Offline

 

#10 2009-08-07 23:19:27

Itchy Kitty Software
Member
Registered: 2008-05-01
Posts: 55

Re: Netflix Prize 2 (Yes, a sequel!)

prizemaster wrote:

While the first contest has been remarkable, we think Netflix Prize 2 will be more challenging, more fun, and even more useful to the field.

Heh.  I don't know that "more challenging" is really such a good thing!  It seems that everyone had pretty much run out of ideas during the last year of this contest, and it was only through the formation of super-teams that the last two prizes were won.  In other words, this contest was already hard enough to stump the best and brightest; how much harder does it need to be?  smile

At the very least, I hope the sequel is not a strict continuation of Netflix Prize 1.  About the only thing I see coming out of that is "Super BellKor's Pragmatic Ensemble of Chaos +5, Turbo Edition."  They could largely avoid that, and get more useful algorithms to boot, if they simply banned blending!

Offline

 

#11 2009-08-08 00:18:51

edr2
Member
Registered: 2009-07-02
Posts: 58

Re: Netflix Prize 2 (Yes, a sequel!)

Itchy -

It will be fun to see what the problem is!! I also hope that it is a different one!!   

It will be interesting to see if blending is even applicable to the next challenge. For predicting a rating, you would expect a blend of loosely correlated estimates to get you a better estimate just as a statistical kind of thing. Many problems are sufficiently non-linear so that a linear blend of results wouldn't buy you much (thinking about some genetic/evolvable algorithms things I worked on). 

Blending a *couple* of methods could result in a practical system, but the kind of blending I suspect won the prize (dozens or hundreds of solutions) is numerical virtuosity - cool to do, cool to see, wins $1000000, but possibly of not all that much practical use.  I am not sure I would ban blending, but maybe limit it to something reasonable (5 algorithms?).  Maybe a category for 'best single algorithm'  would be useful - although you could conceivably 'blend' a number of algorithms into a single executable to make a super-algorithm.  At some level it could become a tough problem to define what consitutes a 'single' algorithm.

It was fun to see some of the creative team names though smile

edr

Offline

 

#12 2009-08-08 15:41:01

Itchy Kitty Software
Member
Registered: 2008-05-01
Posts: 55

Re: Netflix Prize 2 (Yes, a sequel!)

Well, if you're somehow combining multiple stand-alone submission files, that definitely counts as blending.

The area is gray when you start feeding algorithms the residuals from other algorithms.  The obvious example here is k-NN + global effects.  Personally, I'm okay with that.  It seems to better encourage finding complementary algorithms.  It also seems to be much harder to improve one's score through residuals training than through blending, which should similarly encourage stronger stand-alone algorithms.

(Blending reminds me a bit of the old study where they averaged faces together and people tended to prefer the faces with the most averaging.  The conclusion drawn was that averageness = beauty.  What was later found was that the averaging process tended to remove skin imperfections--a lot like airbrushing does--and that people tend to prefer clear, even-colored skin.)

Offline

 

#13 2009-08-09 04:39:48

Got it
Member
Registered: 2009-07-29
Posts: 21

Re: Netflix Prize 2 (Yes, a sequel!)

edr2 wrote:

Itchy -
I am not sure I would ban blending, but maybe limit it to something reasonable (5 algorithms?). 
edr

I think that NetFlix want to have just the FINAL FORMULA, the simple final form and not the complex details of the formula build up. If that's correct, then reinstating the final formula should anyway hide the blending…

A wise person told me once that the best Poets are those that have the fullest wastebasket…  The same probably goes to us, where the blending is used and thrown away, leaving behind just the essence.

What I'm saying is that blending is not present in the final answer, so let it stay free of restrictions.

Edith

Offline

 

#14 2009-08-10 07:47:03

CS1
Member
From: San Jose, CA
Registered: 2006-10-02
Posts: 151

Re: Netflix Prize 2 (Yes, a sequel!)

Hey this made Slashdot!
http://tech.slashdot.org/story/09/08/10 … ng-Contest

For some reason, they quoted my inane suggestions for a name for the sequel.  My opinion of Slashdot just went down the intertubes.   lol

CS1

Last edited by CS1 (2009-08-10 07:47:33)

Offline

 

#15 2009-08-10 12:49:38

Dave the Rave
Member
Registered: 2007-01-27
Posts: 33

Re: Netflix Prize 2 (Yes, a sequel!)

Coincidentally, I saw the article on Slashdot and found my way back here for the first time in a couple of weeks.

Another contest sounds like a great idea, looking forward to it !

Offline

 

#16 2009-08-10 20:32:57

edr2
Member
Registered: 2009-07-02
Posts: 58

Re: Netflix Prize 2 (Yes, a sequel!)

I think it is fair to say that anyone who put any effort into this competition filled their wastebaskets with failed algorithms many times over.. blending or no blending.. I know I did..

Got it wrote:

I think that NetFlix want to have just the FINAL FORMULA, the simple final form and not the complex details of the formula build up. If that's correct, then reinstating the final formula should anyway hide the blending…

Edith

That is the essence of my comments on blending. The results of the blend are a numerical answer which NetFlix already has - the purpose of the 'answer' is to validate how well the 'formula' that generated it works - what they really want is the formula. From both practical and 'aesthetic' standpoints, the simpler the formula the better (as long as it gives superior results).  Blending muddies the waters in that it allows you to get a better answer for the purpose of scoring through potentially enormous increases in the complexity of the formula (and asociated computation). It also makes it much more difficult to define what the final formula was - one reason why although they knew who had the winning score 2 weeks ago, it is going to take them another month to validate all the work.

If a forecasting system can get better results by blending the results of a few methods, you may still have a practical basis for a prediction system. If hundreds of methods need to be blended, this could result in a final 'formula' so complicated that it has little practical benefit.

One might also note that the blending methods that were publicly described (using least-squares to figure out  the proper mix) are actually  pretty elegant solutions in their own right to the problem of getting a little more accuracy.  It's just that sometimes you can have too much of a good thing.

EdR

Offline

 

#17 2009-08-10 22:30:08

Infinite Improbability
Member
Registered: 2009-07-20
Posts: 20

Re: Netflix Prize 2 (Yes, a sequel!)

edr2 wrote:

It also makes it much more difficult to define what the final formula was - one reason why although they knew who had the winning score 2 weeks ago, it is going to take them another month to validate all the work.

You underestimate their intelligence and underestimate how long it takes to explain this stuff to lawyers.  The whole "Do you really own the rights to this algorithm you're licensing to them/us?  And, by the way, what is this algorithm?" takes forever when you're talking to someone who bills by the hour.  That's why law firm partners make a million bucks year in and year out, regardless of whether they improved anything.  lol

Offline

 

#18 2009-08-10 23:47:37

Bored Bitless
Member
From: Leamington Spa, UK
Registered: 2007-02-22
Posts: 154

Re: Netflix Prize 2 (Yes, a sequel!)

edr2 wrote:

Itchy -
Blending a *couple* of methods could result in a practical system, but the kind of blending I suspect won the prize (dozens or hundreds of solutions) is numerical virtuosity - cool to do, cool to see, wins $1000000, but possibly of not all that much practical use.  I am not sure I would ban blending, but maybe limit it to something reasonable (5 algorithms?).

It all comes down to computer resources, CPU time, memory and disk space. It probbaly doesn't make sense to limit the number of blends because you could get into argumnets about what constitutes a blend. However you could define a prize whereby you try to get the best RMSE for a given set of CPU resources, either as a fixed limit or as an alternative scoring systsem, e.g. score = rmse / resources.

I wouldn't be surprised if this is the basis of contest II.

Offline

 

#19 2009-08-11 00:09:18

CS1
Member
From: San Jose, CA
Registered: 2006-10-02
Posts: 151

Re: Netflix Prize 2 (Yes, a sequel!)

I think avoiding blending just serves whiners.  An ensemble algorithm uses a large toolbox in order to solve a problem.  Who hires a carpenter who only has a screw driver?  Is there only one guidance program on an explorer sent to Mars - just code for getting off of earth?  Do people here drive cars that can only go straight, in one gear?  How boring is that.

I think most people complaining about blending didn't work hard enough to see how different models fit different subsets of the data.  Fine, choose a model and let it serve all people the same stuff, rather than adapt a model based on newer users, high or low ratings density users/movies, etc.

Better results are good for Netflix and its users.  Would the anti-blenders like to call in the Handicapper General?  (From the story "Harrison Bergeron")  That's like asking, in the original Longitude competition, for the requirement that a watch be made out of a single gear or not include the various checks and balances that John Harrison identified when he built the H3 clock.

Similarly, some of you may be familiar with Richard Feynman.  In his book "Surely You're Joking, Mr. Feynman", he referred to exploring many different subjects and building a larger mental toolbox than his peers.  After awhile he had few peers, because of his extraordinary breadth of intellectual resources.

To blend is to build an ensemble and use more data and models in order to reduce error.  Either you're with us (and the forces of good predictions) or you're with error and bad predictions!  smile  Make your choice, but don't be dishonest about being lazy or dishonest about not understanding how sophisticated blending can be.  The history of science and engineering favors the multi-talented.

CS1

Offline

 

#20 2009-08-11 02:34:04

tlp
Member
Registered: 2008-04-21
Posts: 36

Re: Netflix Prize 2 (Yes, a sequel!)

Maybe they can penalise based on the number of parameters in the model or set a maximum number of parameters per user/movie or something. Then you could still blend if you want to.

Offline

 

#21 2009-08-11 08:18:13

DandA
Member
Registered: 2008-10-31
Posts: 22

Re: Netflix Prize 2 (Yes, a sequel!)

prizemaster wrote:

a new big money contest

I love it!

prizemaster wrote:

with grand prizes for the best results at 6 and 18 months.

Great idea!  No quasi-arbitrary performance threshold this time.

prizemaster wrote:

more challenging

I hope that doesn't mean "requiring larger computers"... might as well hand it to ATT/Yahoo again hmm

Last edited by DandA (2009-08-11 08:19:06)

Offline

 

#22 2009-08-11 08:18:30

LMV
Member
Registered: 2008-05-24
Posts: 46
Website

Re: Netflix Prize 2 (Yes, a sequel!)

CS1 wrote:

I think avoiding blending just serves whiners.

No it serves to avoid Rube Goldberg "solutions"  with ZERO practical value.
As I mentionned earlier instead of real research this turned to some sort of F1 racing with the teams with the largest budgets winning by a hair, even Pragmatic Theory, they reported at least 60k lines of source code, how much "spare time" and running on which CPU?
And it was still tinkering not research since blending allows any little crumb however minuscule to be added to the pot.
May be there is no way to avoid that within the scheme of a "prize".

Offline

 

#23 2009-08-11 08:24:38

DandA
Member
Registered: 2008-10-31
Posts: 22

Re: Netflix Prize 2 (Yes, a sequel!)

About blending: 

I wonder if an RBM with 1,000 hidden units should be considered a blend of 1,000 little dumb models? 

I wonder if a KNN scheme with 17700^2 regression coefficients should be considered a blend of 17,770 little dumb regression models?

I wonder if an SVD model with 1,000 components should be considered a blend, too?

Hmm...  Honestly, I really don't know!

Offline

 

#24 2009-08-11 10:04:57

LMV
Member
Registered: 2008-05-24
Posts: 46
Website

Re: Netflix Prize 2 (Yes, a sequel!)

DandA wrote:

Hmm...  Honestly, I really don't know!

I think you are misusing the word "honestly" or may be you don't know what it means.
Hairsplitting about what is or is not blending is just as silly as asking what is porn:you know it when you see it.
I must add that I am not "whining" about my own participation, except for one dummy submission to test the submission protocol I DID NOT sent any file and since I am not an academic and furthermore are retired and have no career concern I am not fighting for a rank in the leaderboard. tongue
What I fear is that among the few 100 best scores there is a handfull of truly innovative ideas worthy  of further development but that they are buried in the mass of copycats "solutions" and that they never had a chance to win anything since the first draft of an idea is rarely finely tuned enough or so outstanding as to overcome the "state of the art".
For short, the rules of the contest were not really effective to incentivize creativity.

Offline

 

#25 2009-08-11 15:51:18

LiyeZhang
Member
Registered: 2009-04-01
Posts: 49

Re: Netflix Prize 2 (Yes, a sequel!)

What you are missing is that any amount of blending is still one algorithm. It is just that you are calculating it in different pieces instead. For me, at least, you can generate my entire blended result with a single script, and I fail to see why it is not a single algorithm.

Offline

 

Board footer

Powered by PunBB
© Copyright 2002–2005 Rickard Andersson