A little thought indicates why the procedure of selecting extremes is troublesome. Even without the intervention of a training program, how would the participants redistribute themselves if measured again? Recognizing all measuring devices are imperfect, there will be some change in the individual scores or ratings. Persons at the extremes on the first measurement have no place to go on the second measurement except back toward the middle. For example, you take a vocabulary test this month and score a maximum 200. Next month, when the test is repeated, you can not score over 200 and are much more likely to score less than the maximum--because any measure is somewhat unreliable. This is known as regression. The use of extreme groups in "test-retest" evaluation studies is, therefore, poor practice.
The above error sources rarely occur in isolation. Rather, there is much interaction. Mortality problems, for example, usually reflect maturation and history problems as well. Selection and regression are paired frequently. Indeed, it is possible to cite discredited evaluation efforts which illustrate all error sources at once. The only effective protection against these errors is the elusive goal of perfect randomization,. And, there are, if one is to scan the plethora of literature on the subject, many good reasons why randomization remains a rather ethereal prescription.
The experimental model then, despite the unassailable logic upon which it is based, is more often the "impossible dream" in program settings. There may be ethical and moral grounds as well which prohibit the formation of a control group (e.g., imagine refusing the best available treatment to a patient in a health clinic on the basis that he is assigned to a control!). In most cases, therefore, the particiapnts should be the ones most likely to benefit from the program-- and not selected "randomly." With elaborate control mechanisms also come "halo effects." These effects occur whan a positive result in the treatment group is due simply to the attention the treatment group members received.
The experimental model has also been attacked not only for being infeasible, but because it dfoes not foster continual improvement to the program -- since the model provides information only after the program is over (i.e., to interfere with an on-going program would violate the control requirements and add a new, "non-program" variable to the evaluation). Furthermore, the rigidity of the control requirements often make the program so aseptic that it becomes unrelated to the manner in which things are done in the "real world."
These criticisms have point and weight. However, they are not inevitable consequences of the controlled experiment. The outcomes, for example, can be studied at shorter intervals -- monthly, weekly, whatever. There is no intrinsic need to wait until the program is completed for the feedback of information. The experimental model itself says nothing about requiring a stable program. We may be interested in the outcomes of a program under developing or changing conditions. Successive modifications of a program can be evaluated against each other -- with each modification serving as a control, so to speak, for the modification. Experimental techniques need not be applied by-the-book, without flair or imagination.
The basic issue is fitting the design to the objectives of the evaluation. Experimental design then is simply on way -- albeit, an elegant one -- to find out how well the program is reaching its goals. In cases where one objective of a program -- or single objective to a program -- is important enough to justify the strict contols necessary for a relatively unquestionable conclusion, this design may be warranted. For example, decisons about the continuation or discontinuation of a program require the greatest confidence in the results. Hence, the evaluation should probably come the closest to the experimental design (called by some, the "Cadillac" of program evaluation). Yet, many restrictions (money, time, mangement) impinge upon all evaluations. Hence, the experimental model usually is abandoned in favor of slightly less rigorous designs.