Go To Sources of Error Page

Most evaluation is done on one program at a time. Study is done to determine whether the participants are better off after the program than they were before it. If control groups are used, the evaluation will determine whether results are real or extraneous. Evaluation research can be designed to compare the effectiveness of several programs that have same objectives, but different content by using the same set of outcome measures (Weiss, 1972).

Even within a single program, it is possible to do a comparison study. For example, many programs continue through a series of smaller projects and these smaller units can be evaluated with respect to differing levels of success or goal attainment. Even when the projects have the same goals and ways of delivering a program, important information can be uncovered. A cross-program study might call attention to the findings otherwise unnoticed. This increased information will be promote generalization of results and an increase in the specifications of which strategy provides better results.

Comparative cross-program evaluation is based on assumptions about the program and the people it serves. Weiss (1972) gives the following example:

In its most advanced form, comparative cross-program study will be based on theoretical assumptions about the nature of people's needs and the kinds of help that programs offer. Thus a program of vocational training for high school drop-outs might have three patterns worth testing, each based on somewhat variant theories about why traditional programs have not worked in the past. The first pattern is a three-component program--basic education, skill training, and counseling. It is based on the "deficit" supposition--that drop-outs are deficient in education and training and that personal counseling can motivate the individual to benefit from the opportunities offered. The second program is the male model variant. The belief here is that the drop-out's key need is for effective male figures with whom to identify. Therefore, while offering the same three components, it uses dynamic and able male teachers. The other variant is the community hookup project, which sees the problem in the lack of visible coordination between school courses and the real world of work. Thus, while also giving the same program, its distinguishing feature is skill training in an actual on-the-job setting.

Each of these project variations is introduced into a number of communities, preferably of different types--large and small, urban and rural, North and South. Trainees are recruited into each program by the same procedures. Then the evaluator observes the effects of each. Let us say he chooses as an indicator of program success the percentage of trainees earning at least X dollars weekly three months after the conclusion of training. He collects the data and fills in the cells of Figure 4-5. He can now compare the effectiveness of each program type both overall and in each kind of community. He has the basis for quite specific recommendations on which program type is most likely to be successful in communities with different characteristics. By further analytic elaboration--for example, classifying types of trainees (male/female, under 21/21-45/over 45)--he can make his conclusions increasingly specific. If he has enough cases to study, he can deal with a number of variables at a time.

This type of comparative study can have great power. Unfortunately, however, programs are rarely set up with variations so nicely arranged for the evaluator to study. Too, the evaluator rarely has the power to influence conditions conducive to good design. Vanecko, in studying one hundred community action agencies, classified their major goals into three groups: those that emphasized educational and social service goals, those with community organization goals and those with employment goals. To present his analysis of program activities and effects in a vocational setting, he offered a three-way classification (Vanecko).

Program Type

Many large programs have diverse methods of addressing the same problem and within the program there is often different emphasis on different strategies. In this sense, comparative designs are often not elegant, but can make a start to begin to understand what works and what does not work under given conditions.

There are, of course, many problems with comparative evaluation. One problem is: uncontrollable variation--projects are not assigned randomly and participants are not assigned randomly to program types. Another problem is that program strategies, while often identifiable are difficult to ensure maintenance or adherence. Still another problem is that evaluators are often at cross-purposes. For example, the evaluator may want to find out how well the different programs work, i.e. summative evaluation while the program administration may want to know how to make the program work better, (i.e. formative evaluation). Finally, comparative evaluation can present a political problem. For example, policy makers may contract for a cross-program evaluation to provide information for their decisions, while effective control programs often rests with local managers and administration.

Cross-program study tends to be expensive. Despite this it has the potential for making important contributions to understanding of social programs. Carol Weiss offers the following conditions where this expense is justifiable:

When the issues are real, when policy makers are faced with vital decisions among alternative strategies of action. When the alternative programs are relatively well-defined, with substantially similar aims but clearly differentiated strategies for attaining them. (Each alternative program may have some goals the others do not share, but they should have a set of core goals in common). When there is evidence that the programs have the viability and strength to offer some likelihood of success. Embarking on the demanding course of comparative evaluation appears foolhardy for weak, low-intensity, or untried programs. Comparative evaluation should probably be a second-stage effort after earlier study shows the probability of some positive outcome.

Go To Sources of Error Page