Chanel’s Blog

I decided a couple of nights ago to see how well I would do in the Netflix Prize. This is a competition from the company Netflix, an online movie rental site, that gives two gigs worth of user ranking data to see if anyone can improve their own ranking algorithm, Cinematech, by at least 10%. Many have tried, but few have come close.

Recommendations are a fairly difficult problem. At Grooveshark, we have our own recommendation system using various statistical techniques that have been fine-tuned over the years. They are not perfect, but they do come close to what Pandora, and Last.fm have to offer in certain instances. I’m sure the techniques Grooveshark uses are no way near as sophisticated as Google or Amazon, but we try our best. Early Google has shown that simple algorithms using the right data can be more successful than advanced statistical tools. But even with Google, their algorithms and tools have grown more sophisticated over time. In my opinion, simple tools using the correct insights can be very powerful as proven by a psychologist who has jumped very high in the leaderboard (Just a guy in a garage).

Overall, this project gives a lot of goodwill to Netflix for being so open and providing a great competition for researchers and joe-schmoes alike. I really just want to apply some of the new techniques I have learned in an environment other than music (not surprising when you spend 60 hours a week thinking about music). Here’s some of the books I have read or currently reading:

On Intelligence by John Hawkins: hierarchical Markov Models FTW

Collective Intelligence by Toby Segaran: leveraging simple statistical tools to add intelligence to web applications

Predictably Irrational by Dan Ariely: more psychology than statistics/intelligence

Pattern Recognition by Theodoridis and Koutroumbas: never finished – a little over my head for right now

Probabilistic Reasoning in Intelligent Systems by Judea Pearl: not finished, but find the language more understandable than “Pattern Recognition”

Along with these books, I have kept up a large collection of recommendation and music information retrieval papers. I have read a lot of them, but most of them are on my to-read list. If you would like, check out my document subversion repository at: svn://cmunezero.com/docs.

Also, here’s a pretty good presentation by somebody at Netflix talking about the challenges and issues they face. Now for some muzik:

Grooveshark, Netflix, music, programming books, Grooveshark, grooveshark music, intelligence, mir, movie recommondations, movies, music, music information retrieval, music recommendations, Netflix, netflix challenge, netflix leaderboard, pattern, pattern recognition, probability, rec, recommendations, recs, statistics, stats, subversion

Over the weekend, I really got into music information retrieval (MIR). Its basically grabbing meta-information of an audio file by analyzing its waveform. This type of information is really valuable, especially for a music company (ie: Grooveshark). If I ever have time, this would be a really fun side project. A really good source of information about this topic is this bibiography page (too bad it hasn’t been updated since August, 2007). A list of up and running MIR systems can be found here.

What makes MIR systems so important is that for music sites, they can generate a lot of useful data without anyone having to enter it by hand. For iTunes, this is not a problem because labels give them all the information they need, but for sites where song files can come from anywhere and anyone, there’s no way you can handle the variability in data quality and availability. By having a system that could automatically fetch the required info, within certain bounds of error, you create a vast collection of information that you can use to generate recommendations, provide more accurate searches, and create better categorization of all that music.

The problem with MIR systems is that they require large amounts of storage space and processing power. The cost of both storage and processing are dropping everyday which is great for the future of MIR systems. Processing power is the largest inhibiting factor, especially when you try to analyze millions of songs. The only companies that could probably do a project like this on a large scale would be Google, Amazon and their ilk. Currently, I’m very hopeful that a startup with the right mix of programmers, hardware, and music can compete with the big boys

Grooveshark, music, programming Grooveshark, hardware, itunes, itunes music, mir, music, music information, music information retrieval, programming, songs