Netflix Prize: Forum

I got very similar results to those published in the paper by Bell and Koren.  By "weights," are you referring to shrinkage parameters?  If so, check out http://www.netflixprize.com/community/v … php?id=772.

(edit: sorry, I had misread the OP)

Last edited by Paeva (2008-01-24 13:46:04)

Dan, yes I could duplicate the results reported by BellKor and Bored Bitless.

Gavin, do you care to share what other global effects you have used ?

Personally, I thought of 2 additional effects:

1. Day of the week effect: for example, do people tend to rate higher on Saturdays and lower on Mondays ? I actually coded this but it only improved my probe RMSE by 0.0001. Similarly, there might be Christmas day effect, new year's day effect, Valentine's day effect, etc. Due to large number of ratings for these effects, I think shrinkage is unnecessary.

2. Menstruation effect: women's ratings are affected by their mood, which varies with their menstrual cycles. To detect this reliably, we need large number of ratings over long period of time for each viewer, which we don't have, so shrinkage should be applied. I haven't implemented this yet, but it sounds just politically incorrect enough to worth a try.

gavin wrote:

So, for example, when calculating the means it is important to realise that just taking the average score on the training set is biased.  What I mean by this, is that some customers may only have seen lousy movies and so they have a lower average score.  This will give the impression that they tend to mark downwards.  However, if they had seen averagely good movies in the training set, their average score would be higher.  If you take just this effect into account then you can reduce the quiz score using movie and customer means from .9826 to .9799.   

Gavin

As I recall in the Bellkor global effects approach, each effect is trained on the residuals from the step before it.  So, for example, if the first effect is the "movie mean" effect, then the second effect - the main customer effect - will be calculated on residuals from the first effect.

This approach should already address the point you are making (if a customer has rated lousy movies then the raw mean of his ratings could lead you to the false impression that they "mark downwards").

gavin wrote:

To be honest, its not so much the effects as an underlying approach of trying to think what is really going on in the situation that helps.  You just can't get everything just from performing a straight statistical analysis.

I couldn't agree more.  You can go pretty far using a variety of "blind" numerical optimizations and/or statistical analyses, but you'll eventually hit a wall.  After that you need to start thinking about what makes the people who rate movies tick.  I like your idea about "feel good" and "feel bad" days (from a global perspective) - I think I'll take a look at the ratings for the weeks following 09/11/01 (and possibly some other important dates) just to see if there's a discernible pattern.

Then I'm back to my "blind" numerical approach.  I haven't hit that wall yet.

...one tiny step at a time...

Yes. Tiny.  The "no free lunch" rule clearly applies.

Clueless wrote:

- I think I'll take a look at the ratings for the weeks following 09/11/01 (and possibly some other important dates) just to see if there's a discernible pattern.

That would be very interesting - perhaps we could find some mood indicator (the change in stock prices?) and correlate ratings to that.  The beauty with this dataset is that its so large, that it is possible to discern trends that would be undetectable in smaller datasets -I think we need to persuade more psychologists involved in the competition to complement the mathematical approach.

I should declare that I am a psychologist - running as "Just a guy in a garage".  Are there any others out there?

Gavin

YehudaKoren wrote:

I don’t expect that improving global effects will have much impact on the final RMSE. Latent factor models were proven much more effective at recovering the same information and much beyond. Of course, I might be wrong about this...

Judging from my (haphazard and fledgling) experiments I have to agree - although feeding the global-effects-adjusted ratings to even a half-decent nearest-neighbor implementation gives decent results.

Personally I have been playing with "global effects" primarily to get a better gut feel of the data - plus it's kinda fun.

Based on this discussion, it seems like there's a whole lot of small psychological factors out there to consider...

I remember finding that the closer you get to the weekend, the higher the average rating of the movies rated.  The effect was small (~0.01stars) , but I'd suspect it also has a psychological basis (at least for me )

And maybe a weather index might be useful, too. There was a study from a while back that showed that the stock market returns have a weak but statistically significant correlation with the amount of sunshine. (see this article )   The theory was that sunshine makes people slightly more optimistic, and therefore slightly more bullish.  But optimism could also boost movie ratings...

Back to the stock market index idea:  It might also be useful to use the percentage change in the index over a short period (1 day, 1 week, 1 month), rather than the index itself.  It's the swings in the market that make the news & make people nervous, not the value of the index itself.   Anyway, there's lots of free historical market data at Yahoo, for example, this.

And you know, I wonder if only people who watched "Wall Street" might show this 'stock market effect' !

Last edited by chef-ele (2008-01-26 07:01:02)

chef-ele wrote:

I wonder if only people who watched "Wall Street" might show this 'stock market effect' !

I was curious about this, so I did a very quick-and-dirty analysis of S&P500 monthly returns vs the average monthly movie ratings of two movies: "Wall Street" and "Broadcast News".  I picked Broadcast News because it has a different theme (& perhaps audience) than Wall Street, so it would provide some contrast. It also had a similar number of ratings (~20k) & came out in the same year (1987).

A plot of the results is here. It's hard to tell much from the plot, but the correlations between the monthly SP500 market returns & "Wall St" ratings was indeed higher than the correlation with "Broadcast News"  (0.211 for Wall St, 0.179 for Broadcast News). But there were only 63 months of data that I could use, so its not a statistically significant difference. 

I was surprised to see that both movies had  correlations of around  0.2 ; it's not high, but I thought it might be much closer to zero. If I do some Fisher-transforms & statistical tests,  both correlations are different from zero at a >90% confidence level.  Still, I wouldn't have thought both movies might be correlated with the stock market...maybe I should try a childrens movie next. Or perhaps the market might be a proxy for something else (time, etc.). Interesting...

Last edited by chef-ele (2008-01-26 17:05:00)

Newman! wrote:

Yehuda, one thing you didn't mention in your paper is how you determine the order of the effects ?

The order that we used is the one shown in the table within the paper (from top to bottom). It is quite arbitrary. Of course you want to start with the main effects, as they are so strong and overwhelm anything else. Beside this, other orders might have worked better.

In fact, there is a more fundamental approach to this that enable squeezing extra points from the global effects, but I'm pessimistic about any effect this would have on the final RMSE, so why bother with this

Yehuda

After a somewhat longer analysis, I'm less sure about how valid these stock market correlations are. If you look at the average of all ratings submitted in  a given week &  the SP500 for each week, you get a plot like this..  You can see how the major trends of the S&P (falling then rising) line up with the rating trends (rising slowly, then quickly).  This creates a net positive correlation, but they're both just a function of time.  So using the average rating around a certain date would probably be a better predictor than the SP500.

For the correlation between [%change in the SP500 in the week] vs. [average of ratings submitted in the week], there's the problem of how to deal with the rapid shift in the mean rating around 1Q2004. Generally, I've found it created some misleading correlations,

To get around the shift in the mean rating, I tried using the CHANGE in the average rating. I calculated the correlation of the  [%change in weekly avg rating] with [% change in SP500], but found only about 0.02 correlation, which doesn't seem all that statistically significant. Oh well.

YehudaKoren wrote:

In fact, there is a more fundamental approach to this that enable squeezing extra points from the global effects, but I'm pessimistic about any effect this would have on the final RMSE, so why bother with this

Yehuda,

Please tell us what this fundamental approach is. I managed to get my global effects RMSE down to 0.9534. I basically added a viewer-week-day effect and re-computed the effects a few times. As you mentioned earlier, this doesn't help SVD at all, but helps KNN a lot. Running my KNN on the residue reduced its RMSE by 0.005. So I'm very intrigued by all sorts of global effects now.

Paeva wrote:

I would be curious to find how well these improved global effects + kNN methods blend with everything else.  Not that this is an easy thing to quantify.

So far my combined blending improvement is about 0.002 on probe data, from 0.8954.

Last edited by Newman! (2008-01-29 21:05:39)