ch1

p.3 Why Predictive Models Fail

There are a number of common reasons why predictive models fail, and we address each of these in subsequent chapters.

 

The common culprits include

(1) inadequate pre-processing of the data,

(2) inadequate model validation,

(3) unjustified extrapolation (e.g., application of the model to data that reside in a space which the model has never seen), or, most importantly,

(4) over-fitting the model to the existing data.

 

Furthermore, predictive modelers often only explore relatively few models when searching for predictive relationships. This is usually due to either modelers’ preference for, knowledge of, or expertise in, only a few models or the lack of available software that would enable them to explore a wide range of techniques.

 

p.4 The Case that Accuracy is more important than Explainability 

Notice that the focus of this type of modeling is to optimize prediction accuracy. For example, we don’t really care why an e-mail filter thinks a message is spam. Rather, we only care that the filter accurately trashes spam and allows messages we care about to pass through to our mailbox. As another example, if I am selling a house, my primary interest is not how a web site (such as zillow.com) esti- mated its value. Instead, I am keenly interested that zillow.com has correctly priced the home. An undervaluation will yield lower bids and a lower sale price; alternatively, an overvaluation may drive away potential buyers.

 

p.5~6 It's not enough to work solely. we should combine knowledge & statistical support

Ayres (2007) extensively studies the interplay between expert opinion and empirical, data-driven models makes two important observations bolstering the need for problem-specific knowledge.

 

Firstly, 

“In the end, [predictive modeling] is not a substitute for intuition, but rather a complement”

Simply put, neither data-driven models nor the expert relying solely on in- tuition will do better than a combination of the two.

 

Secondly, 

“Traditional experts make better decisions when they are provided with the results of statistical prediction. Those who cling to the authority of traditional experts tend to embrace the idea of combining the two forms of ‘knowledge’ by giving the experts ‘statistical support’ . . . Humans usually make better predic- tions when they are provided with the results of statistical prediction.”