Some people think building a good machine learning classifier is easy. You just grab some data, the thinking goes, do some feature selection, if necessary, pump the data through some black box classifier, such as a naive bayes classifier or a decision tree, and presto, you have yourself a model.
In fact, nothing could be further from the truth. Building a good predictive model is often painfully hard work. The more common scenario I find in the business world is you have gobs of data, more than you would ever know what to do with. Not only that but you have an incredibly large number of potential predictors you can draw from, often in the hundreds, sometimes in the tens of thousands.
Where things get difficult is that most of the data, in its raw form, is often unusable, containing little predictive value. So what is required is knowledge of the domain and a sometimes painstaking process of examining the variables you have and figuring out the correct representation, i.e. how to transform those variables into variables that are predictive. In my opinion, this process is not only the hardest part, but also is critical in squeezing every last bit of accuracy into the model and taking what would otherwise be barely better than using some common sense heuristic rules into one that is gold-standard, leaps and bounds better than anything that could every built manually without using machine learning techniques.
Unfortunately, assuming you have all the right variables in place, you're barely half way done. In reality, you have to go through many iterations of of tuning the classifier. There are many different kinds of tuning that may be required. They include making sure the model does not overfit the data and making sure the learning gradient is as steep as possible so training finishes as quickly as possible.
Actually the type of tuning most people forget to do is by far the most important one: calibrating the model to the real world. This means, for one, making sure the predicted class distribution matches the actual class distribution. Novices often forget this, run their model against a validation set, see that they get a very high accuracy and think they have a winning model when in fact all they have done is learn to say "yes" on every decision where 95% of all decisions are yes.
Beyond that, you also have many other things to take into account, including the cost of different types of incorrect predictions. Sometimes these costs are equal but not always. The 5% of cases that are no, in the last example, could be worth a lot more than the yeses in terms of cost however it's defined. Building a good cost model can in some cases be extremely difficult and can require a lot of tedious analysis.
Of course, all the satisfaction comes at the end once you've built the model, deployed it into production, and are able to verify that yes the damn thing actually works.
