Tuesday, September 22, 2009

sparse data, complex model and netflix prize

The bellcore team won the Netflix prize. They have spent about an year to work on this problem and then developed a complex model with millions of features for the Netflix data recommendation. Quite a few interesting papers are published. One million dollar is roughly equivalent to the cost of hiring 5/6 Engineers for an year. This is quite a gain for Netflix and it's a gain for research community as well. Yehuda's KDD paper got a best paper award and it is a very interesting paper. The use of temporal modeling for similar problems for web applications or predicting CTR for advertisements is not new. But, it is interesting to identify how closely we can model the intuition and mood to understand human behavior even from a sparse set of data.
This is a more than million dollar information. An open question could be however if the model necessarily need to be complex or if there is a simpler mathematics that can explain it.

No comments: