I’d like to change the world. Who’s with me?

Cameron,

In response to your comment above, I feel the need to address what I feel to be a “cult of quantification”. Although I can’t speak with much certainty, I feel like young, educated professionals today place a gratuitous premium on quantifiable data points, while discounting the impact that certain non-quantitative measurements may have on the predictive capability of mathematical models.
This desire to quantify can manifest itself in many different ways. The most ubiquitous example is making simplifying assumptions that allow for simple, quantifiable data to explain mind-bogglingly complex situations. Whether in the theoretical realm (simplifying assumptions of free market theory: perfect competition, access to markets), or in real world applications (Li’s Gaussian copula and its use in the valuation of securitized debt), people seem to be suckers for a simplified, mathematical explanation in lieu of getting their hands dirty and dealing with the complexities and specificities of the real world.

While the over-reliance on simple mathematical models for complex phenomena is dangerous, even more dangerous is what I feel to be the movement to ill legitimize qualitative data and analytics. Going back to the banking example (if any one industry is long quantitative metrics and short qualitative metrics, this is it!), bankers used to approve loans based on metrics much like we had today, but with the addition of one more: character. Local bankers were able to assess the character of a customer and make an intuitive (instinctive!) decision as to whether they should extend a loan or not. But how does that instinct fit into the model? How does it get normalized over hundreds or thousands of individuals? It can’t be (without some fudging), and that information is subsequently lost and forgotten about. While I understand the need for quantitative metrics, I am sad that so many people view them as diametrically opposed with qualitative measurements – I believe that the best models should be incorporating both.

So that was my rant about the “cult of quantification”, but it really isn’t directly applicable what you were talking about. So you are probably wondering, where am I going with this? I see two major problems with your friend’s model.

The first problem is that it seems like he is recruiting test takers and not teachers. I would have a hard time believing that the best test takers are going to be the best teachers. Is someone who scores a 39 on their MCAT really going to be significantly more qualified than someone who scores a 37? I am hoping that the online assessment is not more heavily weighted than, say, things that actually have to do with teaching (like teaching experience!). This brings me to my second fault, which is admittedly one that I am assuming exists: I do not believe that teacher experience should be a linearly weighted variable in this model. I believe that it is probably some kind of exponential function, where two years of experience is at least two times as valuable as one year of experience, and that there should be an upper-bound around the ten-year mark (probably better to get the advice of an actually professional in the industry and ask them when more teaching experience stops making a difference in the quality of education provided). Anyways, I think that many of these excel “models” are little more than a simple ranking system, and that more thought should be given to the treatment of each individual piece of the dataset in order to get a more precise correlation than might be otherwise provided.
I would offer some advice to your friend. In order to judge who WILL be the best teacher, you have to know who IS the best teacher. Run the same kind of statistical analysis using the variables you mentioned above, and see what impact those variables had on the differences in student’s test scores (pre/post tutoring) – those are the results you want to capture and analyze. My guess is that his dataset might be too small to make any statistically significant discoveries, but it is a starting point.
It is possible that your friend has an awesome model for picking applicants. It is possible that this model weights each variable appropriately. It is possible that your friend has not omitted any variables that have an impact on the teaching ability of said candidate. But my guess is if he was able to do a really good job at that he would sell his company and start working on the startup that you proposed: giving HR departments across the world a kick-ass way of beginning the hiring process.
In closing, while I agree that there is a place for a more quantitative analysis of individuals during the hiring process in some situations, I feel that it is often done poorly and in a rather unscientific manner. Once again, this may not be the case at your friend’s company, and this rant is more about the general “cult of quantification” than your friend’s methods for choosing teachers, but it is something I feel strongly about. On that note, I would love to talk to Wes directly (even though he is probably super-busy) to see how he runs his model — I am just geeky like that. Why else would I be writing a two-page comment on a blog at 2:00am on a Friday night?

Written with Love,
Dan