Netflix Prize: Forum

I should have probably explained it better in my write-up..

1) A 'unique features' is simply a unique integer id which maps some item to that id. In case of an actor, the name is mapped to an id. For a genre, the name of the genre is mapped to an id. So the 'space' is a union of all of these unique id's, which happens to include directors, writers, producers, etc.

2) If two movies only share the director, then they will only have one common feature - namely, director's id mapped to a unique integer id. When you query the hash-map, all you get is a list of integers which represents presence or absence of that 'feature' in that movie. For example, if we query movie '23' (netflix movie_id), we might get an array: [44,89,1044]. Interpret this as a 'sparse' representation of a binary vector. Assuming that the entire 'space' consists of 1100 features, column 44, 98 and 1044 are 1, and the rest is 0. What feature 44 stands for exactly is not important, you should only care about it being there. (Given that 44 always stands for the same 'thing', be it a director or year of release. And it does.)

3) Well, that's up to you really. The simplest way would be to compute the hamming distance between each binary vector (xor them), or even cosine similarity. A better way might include dimensionality reduction. You might want to look at my writeup of SVD system in Ruby.

Hope that clears it up a bit.

Last edited by igrigorik (2007-01-28 05:52:46)

Ah, that's a good point. Assigning different weights to different components can give you even higher granularity, but that's not something I was interested at the time. If there is strong-enough demand, I can probably produce a weighted version of the space also...

I didn't crawl IMDB, that would have turned an already tedious task into a multi-day adventure! I downloaded their entire database, loaded it into a local SQL server and then used IMDbPY on the backend to put together my own script to do the dirty work of correlating netflix and imdb data.

Frankly, I think you're thinking too hard about this. Remember, you're not trying to find 'good' movies based on this data, you're trying to find similar movies.

Hint: User 25 rated movies x and y and now you have to predict the rating for movie z. Without semantic knowledge you can look at 'similar users' and see how they rated z when they rated x,y just as you did (or close to you). Alternatively (or as an additional component), you can also look if x and y share any of the same semantic features with z and predict solely on that. So if you find that x and y have very similar casts, and so does z, you can be fairly confident in your prediction for z without looking at different users.

P.S. Essentially, this data can tell you 'why' the user gave a rating x to movie y. Given enough movies, you can construct a good preference vector for every user and weigh every prediction against it.

Last edited by igrigorik (2007-01-28 20:26:55)

Hello igrigorik,

where can I get the datafile that has the correlated Netflix movies (in the dataset) against IMDB.

I am insterested in using that dataset with my existing data.

do you have it sorted by movies and the the features for each movie or did I miss something.

In following your link, there appears to only be some python stuff and I was expecting to be able to download the datafile with the features.

Thanks,
Lonnie