Identifying The Blind Spots

On Model Building

Some people think building a good machine learning classifier is easy. You just grab some data, the thinking goes, do some feature selection, if necessary, pump the data through some black box classifier, such as a naive bayes classifier or a decision tree, and presto, you have yourself a model.

In fact, nothing could be further from the truth. Building a good predictive model is often painfully hard work. The more common scenario I find in the business world is you have gobs of data, more than you would ever know what to do with. Not only that but you have an incredibly large number of potential predictors you can draw from, often in the hundreds, sometimes in the tens of thousands.

Where things get difficult is that most of the data, in its raw form, is often unusable, containing little predictive value. So what is required is knowledge of the domain and a sometimes painstaking process of examining the variables you have and figuring out the correct representation, i.e. how to transform those variables into variables that are predictive. In my opinion, this process is not only the hardest part, but also is critical in squeezing every last bit of accuracy into the model and taking what would otherwise be barely better than using some common sense heuristic rules into one that is gold-standard, leaps and bounds better than anything that could every built manually without using machine learning techniques.

Unfortunately, assuming you have all the right variables in place, you're barely half way done. In reality, you have to go through many iterations of of tuning the classifier. There are many different kinds of tuning that may be required. They include making sure the model does not overfit the data and making sure the learning gradient is as steep as possible so training finishes as quickly as possible.

Actually the type of tuning most people forget to do is by far the most important one: calibrating the model to the real world. This means, for one, making sure the predicted class distribution matches the actual class distribution. Novices often forget this, run their model against a validation set, see that they get a very high accuracy and think they have a winning model when in fact all they have done is learn to say "yes" on every decision where 95% of all decisions are yes.

Beyond that, you also have many other things to take into account, including the cost of different types of incorrect predictions. Sometimes these costs are equal but not always. The 5% of cases that are no, in the last example, could be worth a lot more than the yeses in terms of cost however it's defined.  Building a good cost model can in some cases be extremely difficult and can require a lot of tedious analysis.

Of course, all the satisfaction comes at the end once you've built the model, deployed it into production, and are able to verify that yes the damn thing actually works.

The Nature of Incumbents

There is a telling paragraph at the end of the first chapter of "Seattle and the Demons of Ambition: A Love Story" written by Fred Moody (same guy who wrote "I Sing the Body Electric") that speaks to the question of how revolutionary, young startups are often perceived and actually change once they become the incumbents they sought to replace, having succeeded in their quests to radically transform the marketplace:

"I was standing within canister's throw of four Northwest companies - Nordstrom, Nike, Starbucks, and AT&T Wireless that had  all at one time  been brash, romantic startups determined to rebel against the status quo in their businesses and deliver something previously forbidden to the beleaguered and deprived consumer-citizen. Now, all four were reviled as oppressors of customers, competitors, employees, former employees, contracted third-world employees, or all of the above. I remembered too that Microsoft, Amazon.com, and McCaw Cellular (before AT&T bought it and turned it into AT&T Wireless) had once been popular Seattle startups, freedom fighters in the corporate age, wresting power over information and communication from the hands of previously indomitable corporations and putting it in those of ordinary citizens."

It's an interesting point that Moody makes and I think it can be broadened outside of the Northwest.  In the same way people forget that every religion was once a cult, people often forget that virtually every large company that people so despise for one reason or another was once a revolutionary startup cheered on by the masses to replace the incumbents that came before them.

In many cases, I think, it's the story of youthful idealism being corrupted by the forces of power and greed and the way companies are perceived aptly mirrors the reality of the situation. Think Walmart, Microsoft (at least leading up to the antitrust case), Enron, and so forth.

But then there are the other cases, where perceptions are misaligned with the reality on the ground. Take Amazon, for example, a company I happen to work for. There is a lot of resentment by people against Amazon because they have supposedly driven many small bookstore owners out of business. But what these people may not realize is that they allow businesses of any size to sell their products on Amazon's website so a customer can choose to buy a book (or any other product) directly from Amazon or alternatively pay a lower price for a used or new one  from a small mom and pop. Perhaps Amazon is evil in other ways I am unaware (that they have flaws is not in question), but at least in this regard there is a large gap between perception and reality.