Netflix Prize: Forum

One of the conjectures in the literature is whether, and what, extra data about the movies (or the people) might help improve prediction accuracy.  There are tantalizing studies on this topic.  You are free to pursue this line of investigation, of course, to see if those results scale.  However, to be clear we are not supplying that data in the dataset and we didn't use anything like it when we computed the Prize RMSE scores.   And using outside sources of data does not preclude you winning the prize.

Excellent question!  I guess I'll stop rating my movies since it does not affect my recomendations.

prizemaster wrote:

...we are not supplying that data in the dataset and we didn't use anything like it when we computed the Prize RMSE scores.

Maybe that's the whole problem!

Right on, CriterionDA. You just saved them $1million.

M@

Okay, so Cinematch doesn't use movie genre, MPAA rating, etc., but clearly a lot of people believe it could help their algorithm.  Wouldn't it be a lot easier to just release a more extensive movie file since Netflix already has all this information?  And I don't just say "easier" to mean for the contestants.  It sounds like lots of people are talking about scrapping the Netflix website, which would cause unwanted traffic.

Please see the Forum FAQ discussion on this topic:

http://www.netflixprize.com/community/v … .php?id=98

Why not just not use it, get in the game and win some money?
The only reason you are stuck with this extra data notion is that you, currently, can't visualize the game without it. You don't need it. It is worse than useless.

The rules of the contest are set

A cursory look over any of the 'proposed' extra data will show you that it will produce false positives. If Hollywood could get their hands an a prediction system as accurate as Cinematch they would pay somewhat in excess of $1m.

It is worse than useless plus it will keep you out of the game.

He already answered you, the rules are set.  I mean seriously, you think they are going to change the rules after the fact?  That would expose them to all kinds of lawsuits.

Regardless, the rules seem to say you are allowed to use additional data.  If its imdb data, and it works, and you are legally using it (from imdb's point of view), knock yourself out.

I think the reason willakawill is saying you will be "kept out of the game" is because he is predicting (as I am) that you won't win if you are going down that path --- because you are going to spend a lot of time trying to meaningfully extract and balance gobs of relatively unstructured data, while the ratings data that is supplied is thousands of times richer in terms of what it contains.

This reply is aimed at everyone who is wondering about extra data.

Please see my previous post on this subject of Extra Data

You can waste your time looking for extra data, but once you got it, cleaned up, and linked it to the netflix dataset, what exactly are you going to do with it?  The amount of processing required to simply work with the current dataset is huge and thats only with a one-dimension link between user and movie.  If you add so many data dimensions your processing requirements are going to go up exponentially.

All the genre information is already there, you just have to mine for it.  Getting artificial data about actual genre will only degrade the dataset.  If you don't understand this, you're not understanding the contest.

You may be thinking "Ah, but you don't know what I'm going to do with my data.  I have a cunning plan." You may be serious, you really do have a cunning plan that decreases the cross-links between data dimensions (both in just one movie, and between all movies linked by a user)*.

If still want to go ahead with the extra idea, consider this plan:

1) Get all the current data into a database
2) Using just the standard data, get your RMSE to 0.9514 (Cinematch)
3) Improve your RMSE to 0.9413 (The Thought Gang)
4a) Then either look for room for improvement in your current algorithm, or
4b) Go hunting for extra data.  If you've got here and you still think extra data would help, you'll know exactly which data you need and how to use it.
5) Win contest, fame and fortune.

Good luck!

Paul


*For the small proportion of you that do not understand what I wrote about data-dimenson cross-links here: your plan is not serious.

Barry wrote:

It sounds like you may be saying that you have to put all processing power into the provided data and only use other data if it doesn't have a processing impact, which I don't think is true.

That's not actually what I'm saying, obviously using more data is going to increase the processing impact.  Processing large amounts of extra data, as any sane team-member will tell you, is going to make the current task with the given data-set, seem trivial.  Unless you can decrease your relationships between the data-dimensions - but reducing the relationships would decrease the whole point of using the extra data.

To use an example from a previous posting of mine, a user may like films with Clare Danes in, where Clare plays a girly-girl, in a drama based on a classic book or play.  The user also rates films highly where she plays characters in apocolypse films.  But in anything contemporary, the user feels she overracts and doesn't rate the film. 

Can you imagine processing this level of data?  Analysing all the relationships between all the different types of data?  It really would make the current task a walk in the park.

I'm not saying it wouldn't be a fun walk in the park.  I'm the first person up for a programming challenge.  But people should be realistic about the problem domain.  Achieving Cinematchs score is a very very hard problem.  The teams competing here are no doubt mostly amazing programmers, but only three teams have beaten Netflix's RMSE score.  Analysing more data is a much harder task.  It would be better to start with what Netflix has given and reach their score, then look at which extra data is really needed.  It may be that an accurate year of release to DVD is relevant, though I'd prefer the difference in DVD release and theatre release.  Genre would be pointless.  Director and script complexity would be good to have, but how do you judge script complexity?

Going back to school to learn programming techniques would help for people having problems with basic coding techniques, the actual manipulation of data.  I wouldn't think 2 years though, perhaps an intensive course.  Though this challenge is probably as good a way as any to improve your coding.  Inexperienced coders shouldn't be afraid to ask for help though, and listen to people who have tackled problems of similar magnitude before.

Experience in analysing complex problem domains would be an essential requirement here.  Not just experience, I should really say successful experiences.  Writing highly scalable systems is also crucial.  I bet the guys at google at all laughing at us

willakawill wrote:

The rules of the contest are set

A cursory look over any of the 'proposed' extra data will show you that it will produce false positives. If Hollywood could get their hands an a prediction system as accurate as Cinematch they would pay somewhat in excess of $1m.

It is worse than useless plus it will keep you out of the game.

No data is worse than useless, if you know your math.