Netflix Prize: Forum

I’ve created some visualizations that show the interrelationships between 5000 of the most popular movies in the Netflix dataset.  You can see them on Flickr at the following links:

The visualizations of the entire set of  5000 movies are located here. 
Smaller close-ups of  interesting areas in the visualization can be found right here.

(  NOTE:  The first set of images above are very large, high-resolution .PNG images  (about 7000 x 7000 pixels, and >5MB), so you have to pan/zoom through them to see all the detail.  Select 'All Sizes'  above the image & select the largest size;  otherwise, the default preview size makes the plots slightly fuzzy, and you won’t be able to see the details & the movie titles.   Also note that the graph is the same in all these images, though the background color & zoom changes between the images, and in some the movie titles are omitted. )

In the images, each movie is represented by a dot/circle.  Similar movies are connected by lines. The line colors indicate the strength of the similiarity.  Colors closer to red represent weaker similarity, and colors closer to yellow indicate stronger similarity.

You can see a lot of themes emerge in certain regions of the images. Some regions show many movies connected due to similarities in genre( e.g.  groups of horror or sci-fi movies).  Other regions show similarities in the cast or  directors  (e.g. Robert DeNiro films, Kubrick films).    Also, certain movies seem to be “hubs,"    which share similarities with a lot of other movies.

Next, here are the details about how these visualizations were made:

1.    The 5000 most popular movies in the Netflix data set were selected for use in the visualization.  This subset was used to keep the graph legible, and to reduce runtimes of the algorithms that follow.  Although the 5000 movies is less than the 17,770 in the NetflixPrize data,  93% of all ratings in the dataset  were for those movies.

2.    A  movie-movie correlation matrix was calculated using these 5000 movies.   Pearson correlation was used  on ratings from customers who saw both movies in a movie-movie pair.  Corrections for support were made by using the lower bound of Fisher's Z'-transformed confidence interval for the correlation.   A confidence interval of 3 standard deviations was used, and the edge of the interval closest to 0 correlation was used.  (Note: in the future,  other distance metrics could be used to see if the results change).

3.    The correlations were fed to a minimum-spanning-tree (MST) algorithm.  The negative value of the correlations were used to indicate inter-movie  distances. That way, the MST would pick the most correlated movies when building the tree connecting the movies.

4.    The MST  definition was  fed to a graph-visualization package for plotting.

I made some similar plots over two years ago (as indicated in this old post). However, these updated images use a different algorithm to determine the lines/links, and the images themselves are  larger & higher resolution  (also, I just think they look better!). Previously, I was using a single-nearest-neighbor algorithm, which tended to form small clusters.

Enjoy!

(PS  Sorry for the long post. )

Last edited by chef-ele (2009-08-06 17:43:54)