One of the three basic experimental design types used in empirical research in industrial-organizational psychology and related disciplines is quasi-experimentation. Quasi-experimental designs are different from both randomized experimental designs and nonexperimental designs. In the process of describing the nature of quasi-experimental designs, we make reference to a number of issues having to do with the validity of inferences stemming from research.
Attributes of Quasi-Experimental Designs
Quasi-experimental designs have a number of features. Taken together they serve to differentiate such designs from designs of the experimental and nonexperimental varieties.
Types of Quasi-experimental Designs
There are five major varieties of quasi-experimental designs, as noted by W. R. Shadish, T. D. Cook, and D. T. Campbell (2002). They differ from one another in terms of the use of comparison conditions, the use of pretests, and the degree to which they are time-series based. The term comparison conditions is used here to refer to either a no-treatment control condition or a condition that has a different level of the independent variable than the focal condition.
Single-Group Designs without a Control Condition
In terms of single-group designs without a control condition, the weakest ones, with respect to the criterion of internal validity, have only a posttest, for example the one-group posttest-only design. Slightly stronger designs have both a pretest and a posttest, such as the one-group pretest-posttest design. Even stronger designs have multiple pretests and posttests. For designs of the latter type, internal validity can be enhanced through such means as introducing treatment at one time period and removing it at a later period as in the removed treatment design. This process can be repeated several times, as with the repeated-treatment design.
Designs with a Control Condition but No Pretest
A second type of quasi-experimental study uses a control condition but no pretest. One example of such a design is the posttest-only design with a nonequivalent control condition. In general, internal validity is quite problematic with studies that use this design because the researcher typically has no knowledge about any pretreatment differences on a host of variables, making it difficult, if not impossible, to attribute posttest differences to the treatment.
Designs with Control Conditions and Pretests
A third type of quasi-experimental design uses both pretests and posttests. The simplest example of this is the untreated control condition design with both pretest and posttest measures. The addition of pretests in the treatment and comparison conditions serves to improve internal validity, such as by ruling out selection as a rival explanation of posttest differences on the dependent variable.
Time Series Designs
A fourth type of quasi-experimental design uses time series data to assess how the introduction of a treatment affects measures of dependent variables. For time series designs there must be a large number of observations, such as 50 or more, of such variables prior to and following the introduction of a treatment. In some cases time series designs involve both the introduction of a treatment and its subsequent removal. Internal validity is enhanced to the degree that these changes have expected effects on the dependent variables and the changes cannot be attributed to such confounds as history-based fluctuations. In addition, internal validity can be further enhanced through the addition of a nonequivalent control condition to the basic time series design.
Regression Discontinuity Designs
A fifth type of quasi-experimental design is the regression discontinuity design. In the most basic of such designs, the researcher uses scores on a pretest to assign individuals in a single group to treatment and control conditions. Individuals who have pretest scores at or above a given level are assigned to one condition, such as treatment, and those with scores below that level are assigned to the other condition, such as control. After the treatment, posttest scores are regressed on pretest scores. The effect of the treatment is indexed by a discontinuity in the regression lines for the treatment and control groups.
Manipulation of Variables
In quasi-experiments the values of independent variables (X,,X2, … XI) are manipulated by the researcher. This is important in terms of the criterion of internal validity. More specifically, the fact that the manipulations precede the measurement of dependent variables serves to strengthen inferences about the effects of the manipulated variables on one or more dependent variables.
The simultaneous manipulation of several independent variables is common in randomized experiments that are of the factorial variety. However, it is rare to encounter quasi-experiments with more than one independent variable. This is, at least in part, a function of the fact that most quasi-experiments are conducted in non-special purpose settings (see the Experimental Designs entry for more on setting-related issues). Relative to special purpose settings, in non-special purpose settings, researchers typically have relatively little control over the number and nature of manipulations to which units, such as individuals or groups, are exposed. In addition, it is often difficult to deliver treatments to units in a uniform manner, reducing statistical conclusion validity. What’s more, because individuals other than the experimenter, for example managers, have control over many features of the settings in which quasi-experiments are conducted, the same individuals may behave in ways that serve to reduce the construct validity of treatments. For example, to lessen research-related inequalities, managers may deliver desirable treatments of their own to units in study conditions that are scheduled to receive less desirable research-related treatments than others. This can lead to compensatory equalization-based threats to the construct validity of the study’s independent variable(s).
Nonrandom Assignment of Units to Conditions
A key characteristic that differentiates quasi-experimental designs from randomized experimental designs is that in the case of the former, there is no capacity to randomly assign units to treatment conditions. Rather than being randomly assigned to conditions by the experimenter, the units may be routed to treatment and control conditions through such means as self-selection or assignment by an administrator such as a manager. An important implication of this is that, in most cases, it is difficult, if not impossible, to rule out both selection as a threat to internal validity and selection by treatment interaction as a threat to external validity.
Measurement of Dependent Variables
As is true of all other experimental design types, in quasi-experiments, dependent variables are measured. A number of techniques can be used for this purpose (see the Experimental Designs entry for examples). However, it should be added that because most quasi-experiments are conducted in non-special purpose settings, the experimenter often has diminished control over the timing of measurement and the conditions under which it takes place. In addition, many quasi-experiments use data from archival records such as those of the organizations participating in a study. As a consequence, there are often problems with the construct validity of measures of dependent variables.
Control Over Confounding or Extraneous Variables
As noted earlier, in the case of quasi-experimental research, there is no capacity to randomly assign units to treatment conditions. As a result, there may be pre-treatment differences between or among the conditions on variables that are related to the dependent variables of a study. For example, consider a hypothetical study in which a researcher tested the effects of job enrichment on job satisfaction in an organization having two geographically separate production facilities. Individuals in one facility experienced job enrichment, whereas workers in the other served as no-treatment controls. Because workers were not randomly assigned to these facilities in advance of the quasi-experimental study, they might have differed from one another at the pretreatment period on such variables as age, tenure, pay, and job satisfaction, reducing the study’s internal validity.
In quasi-experimental research, assumed confounds are controlled through statistical means. For example, in a study using an untreated control condition and a pretest, a researcher might regress posttest scores on pretest scores and measures of a set of assumed confounds. Regrettably, the statistical controls used in quasi-experimental studies are typically a poor substitute for the control that can be achieved in randomized experiments, as noted by Shadish and colleagues (2002) and Eugene F. Stone-Romero (2002).
References:
- Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton Mifflin.
- Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
- Stone-Romero, E. F. (2002). The relative validity and usefulness of various empirical research designs. In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 77-98). Malden, MA: Blackwell.