Incremental validity is a predictor’s ability to explain an outcome, beyond all other predictors. For example, assume predictor A accounts for 25% of the variance in an outcome of interest and, when entered separately, predictor B also accounts for 25% of the variance. Because their influences most certainly overlap, it is also important to understand the amount of variance each predictor explains when considered in conjunction with the other. One scenario is that predictor A and predictor B account for much of the same variance, so predictor B can be said to have low incremental validity because it adds very little new information to the prediction equation. Another scenario is that their variances overlap very little, so predictor B can be said to have high incremental validity.
Although incremental validity is conceptually simple and straightforward, it becomes difficult to estimate in practice, for several reasons. First, it is impossible to know all possible predictors of a given outcome, so it is imperative to draw from relevant, well-developed theory when defining a subset of predictors to examine. Second, the incremental validity of a given predictor necessarily fluctuates from situation to situation. For example, the incremental validity of athletic ability over intelligence is likely quite large when attempting to predict the performance of an NFL quarterback but is likely nonexistent when predicting the performance of an accountant.
Estimating Incremental Validity
The most widely used method of assessing incremental validity is hierarchical multiple regression, which allows researchers to assess the amount of variability explained by the predictors, after previous predictors have already explained their share of variance. Adding predictors to a model necessarily increases its predictive power, so an important question becomes, How do you determine when incremental validity ceases to be significant?
Arguably, the most important consideration is parsimony—that is, defining a model that is as simple as possible, without sacrificing substantial predictive power. One way to do this is by examining a model’s adjusted R2, which indicates the proportion of variance explained in the outcome after the estimate has been slightly decreased for each predictor included in the model. This is done because adding predictors to a model always increases its predictive power. So, to increase the adjusted R2 value, the incremental validity of each added predictor must outweigh its subsequent decrement in the adjusted R2.
Applying Incremental Validity
Of all forms of validity (e.g., construct, criterion), incremental validity is the most applied, in that it is typically used to better predict valued outcomes in real-world settings and often influences (and is influenced by) other considerations, such as time, money, and effort. This is especially true in the area of personnel psychology, where incremental validity is given more attention than in “purer” forms of psychology. Some have even developed utility analyses, which are used to translate organizational incremental validity estimates into financial savings.
One particularly prevalent application of incremental validity has been predicting applicants’ future job performance. Here, research has led to the nearly unequivocal conclusion that general mental ability (GMA, or g) is the best predictor of job performance across nearly all occupations. However, tests of GMA do not predict performance perfectly, so an important finding is that, under certain circumstances, predictors such as personality inventories, structured interviews, and work samples add incremental validity to the prediction of job performance.
Concluding Thoughts
Incremental validity is a relatively straightforward concept that can cause many practical difficulties. However, many complications can be avoided by remembering that (a) incremental validity varies from situation to situation (e.g., across occupations), so one should refrain from making absolute statements about the incremental validity of any predictor; (b) there is a nearly infinite number of predictors of a given outcome, so a subset of predictors should be chosen from well-developed theory; and (c) a point of diminishing returns is typically reached after including a relatively small number of predictors, so models should be as parsimonious as possible.
References:
- Garb, H. N. (1984). The incremental validity of information used in personality assessment. Clinical Psychology Review, 4, 641-655.
- Hunsley, J., & Meyer, G. J. (2003). The incremental validity of psychological testing and assessment: Conceptual, methodological, and statistical issues. Psychological Assessment, 15, 446-455.
- Sechrest, L. (1963). Incremental validity: A recommendation. Educational and Psychological Measurement, 23, 153-158.