Winsteps

This month I'm teaching an on-line Rasch Course to 40 students around the world. It could easily be a 24 hours a day, 7 days a week activity. But that would lead to mental burn-out and physical collapse. All of us must take time off. We must have rest. "Rest" has been a central theme in Western Christianity, but it is fading away under the pressures of modern life. An excellent sermon on the spiritual purpose and necessity of rest is http://download.redeemer.com/rpcsermons/storesamplesermons/Work_and_Rest.mp3

Elizabeth Newton's (1990) Stanford Ph.D. dissertation "The rocky road from actions to intentions" discusses the Curse of Knowledge - the paradox that being too much of an expert makes it difficult to explain elementary concepts to a beginner.
The on-line Winsteps course Practical Rasch Measurement is reminding me of this. Participants in the Course have pointed out to me that some of my supposedly obvious and comprehensive instructions in the Winsteps Help file assume too much of the beginning Winsteps user. Thank you participants! All feedback is welcome!
But this also reinforces my conviction that Ken Conrad and Barth Riley are better instructors for the Introduction to Winsteps in-person workshops than I could be. They are Winsteps users, not Winsteps specialists. They know what users need to know to be productive, and they know how to teach that knowledge.
The advantage of knowledge with the on-line course is that I can answer those obscure technical questions produced by the 40 participants over a period of four weeks. This is the fourth time for the Practical Rasch Measurement Course. Each time has seen improvements in the teaching material (at least from my perspective).

It's heading towards the Made to Stick: Why Some Ideas Survive and Others Die ideals of the Heath brothers,

1. Simplicity - every time better, but still a way to go.
2. Unexpectedness - there are always surprises; the challenge is to make them all good ones!
3. Concreteness - Rasch is mathematically abstract. So solid examples are emphasized.
4. Credibility - do the results make sense?
5. Emotions - O frabjous day! Callooh! Callay! He chortled in his joy ... Rasch analysis should be fun ....
6. Stories - that's the crux of it all. If our findings don't communicate a great story, who will remember them? What use will they be?

Ben Wright would love the acronym of those 6 points, SUCCES, but he would have completed it with another "S", perhaps "7. Sharing". We can have wonderful ideas with all the 6 attributes, but they will quickly die unless we share them with others. Rasch analysis appears to be an abstruse statistical technique to be performed by the initiated almost in secret, but its purpose is to produce knowledge that can be shared, that is useful to as many people as possible. SUCCESS, that's what we want!

William Fisher reminds me "there is nothing so practical as a good theory"*. But what is good theory for the Netflix challenge, essentially predicting 3 million new ratings based on 100 million previous ratings? It turns out there are several challenges.

1. Predicting the 1.5 million "quiz" ratings ... for which a summary statistic of success so far is supplied. These decide the "leaders".
2. Predicting the 1.5 million "test" ratings ... for which there is no feed-back until the competition is over. These decide the "winner".
3. Constructing a Netflix-acceptable method for achieving 2 in order to be eligible to win.

It would seem that 3. comes first. But no! There is no need to consider whether a prediction method is acceptable until one is in a winning position. When one has a winning solution, then one can commence the effort to develop an acceptable method of obtaining that solution (or better).

So, since we won't know how 2. went until after the challenge is over, the task becomes success on 1. That success is shown on the Netflix Leaderboard: http://www.netflixprize.com/leaderboard - As of this writing, I'm about 30th of the 1,352 teams who have submitted answers (multiple submissions are allowed - as often as one per day, but only the best result of the top 40 teams is shown). Most of the 14,994 registered teams have not submitted an answer.

So what theory is good? Initially, I tried complicated methods. But my results were only so-so. A couple of elementary insights improved my answer sets - but still far behind the leaders. Then Simon Funk, an early leader, published his approach http://sifter.org/~simon/journal/20061211.html . It is a simple raw-observation SVD (single value decomposition). Conclusion: my approach was too elaborate.

So it's back to the algebraic drawing board. It's not a matter of predicting the most likely set of ratings, but rather of predicting a particular set of ratings. I've tried many different ideas ... but nothing brilliant so far. Two methods appeared to be winners .... but they fell by the wayside. Happily, there are more ideas to try ....

What have I learned? (a) How to manipulate a dataset of 100 million ratings. (b) The virtue of 2GB RAM memory. (c) Slick methods of computing in C++. (d) Additional features to add to Facets.

Will brains beat brawn in the Netflix Challenge? Will the winner be the team with the keenest insights or the one with the greatest computer power? Keen insights, i.e., good theory, seem to be the way to go ....

* Lewin, K. (1951). Field theory in social science: Selected theoretical papers (D. Cartwright, Ed.). New York: Harper & Row, p. 169.

"Companies are beginning to realize that these questions of 'How can I accomplish more good in the world?' and 'Where is the market opportunity?' are essentially the same question," says Jeff Hamaoui, founder of Origo Inc., a consulting firm that helps both nonprofits and for-profits navigate this blended arena of social enterprise. "Simply put, good business design maximizes opportunity and resources, now and for the future."
http://www.fastcompany.com/magazine/111/open_socap-intro.html

According to a book review at 800ceoread.com/blog/archives/006561.html:

"The key to Toyota's success is the nature of constant improvement. Innovation to them is not invention or artistry. It is gaining deep understanding of the work at hand. It is about having a strong engagement in the work you do. It is about tinkering and trying new things."

Tinkering is something I do a lot of ... Someone always wants little extras, and there are the extras I want myself!

Modeling each of the 17770 movies in the Netflix database to have its own rating scale (the Rasch-Masters partial-credit model) produces worse predictions than modeling them all to share the same rating scale! And modeling each of the 480,189 customers to have a unique rating scale is even worse!!
This accords with the Rasch proposition that "betterdescription of the local dataset can result in worse inference for other data sets." I was already skeptical of the accidental nature of many partial credit analyses, particularly those with low category frequencies. The Netflix data confirm my skepticism.

Asia is rushing into hard science research Time magazine story, but seems to have overlooked soft science: psychology, sociology, etc. But it is advances in soft science that they need in order to solve the huge societal problems they are experiencing, and that are likely to become worse as populations grow and the migration to the cities continues.

Only two of the 30+ enrolled in my current online Rasch Course give Asian countries as their addresses. Yet Rasch measurement is a powerful tool in the advance of soft science because it demands that you know what you are doing. Books on Rasch have already been published in Korean and Chinese. Please email me links to where those Rasch books (or any in non-English languages) can be obtained so that I can announce them on this website.