Conclusion


This workbook offers several simulation models that are appropriate for use of three common research designs for evaluating program effects. The logic of these simulations can be easily extended to other relevant research contexts. For instance, many agencies routinely conduct sample surveys to identify needs and target populations, assess services that are provided, and compare agency functioning with the performance of other similar agencies or with some standard. One would construct simulation models for survey instruments for the same reasons that they are constructed for evaluation designs--to improve teaching and general understanding, to explore problems in implementing the survey (such as non response patterns), or to examine the probable effect of various analytic strategies. The key to doing this would again rest on the statistical model used to generate hypothetical survey responses. A "true score" measurement model is useful, at least for simple simulations, but may have to be modified. For instance, assume that one question on a survey deals with client satisfaction with a particular service and that the response is a 7-point Likert-type format where 1=very dissatisfied, 7=very satisfied, and 4=neutral. The analyst could make the assumption that for some sample or subsample the true average response is a scale value equal to 5 points (somewhat satisfied), and that the true distribution of responses is normal around these values, with some standard deviation. At some point, the analyst will have to convert this hypothetical underlying continuous true distribution to the 7-point integer response format either by rounding or by generating normally distributed random integers in the first place. Such a variable could then be correlated or cross-tabulated with other generated responses to explore analytic strategies for that survey. Similar extensions of the models discussed here can be made for simulations of routinely collected management information system (MIS) information, for data for correlational studies, or for time-series situations, among others.

Simulations are assumptive in nature and vary in quality to the degree that the reality is correctly modeled. When constructing a simulation, it is important that the analyst seek out empirical evidence to support the assumptions that are made whenever this is feasible. For instance, it should be clear that the simulations described here could be greatly enhanced if we had more specific data on how much and what type of attrition typically occurs, what type of floor or ceiling effects are common, what patterns of misassignment relative to the cutoff value typically arise for the RD design, what the typical test-retest reliabilities (for use in reliability-corrected ANCOVA) might be, and so on. Although some relevant data will be available in the methodological literature, all of these issues are context specific and demand that the analyst know the setting in some detail if the simulations are to be reasonable.

One way to approach the assumptive nature of the simulation task is to recognize that reality conditions or constraints in the models need to be examined systematically across a range of plausible conditions. This implies that multiple analyses under systematically varied conditions that are based upon principles of parametric experimental design are needed in state-of-the art simulation work. This point is made well by Heiberger et al. (1983:585):

The computer has become a source of experimental data for modern statisticians much as the farm field was to the developers of experimental design. However, many "field" experiments have largely ignored fundamental principles of experimental design by failing to identify factors clearly and to control them independently. When some aspects of test problems were varied, others usually changed as well--often in unpredictable ways. Other computer-based experiments have been ad hoc collection of anecdotal results at sample points selected with little or no design.

Heiberger et al. (1983) go on to describe a general model for simulation design that allows the analyst to control systematically a large number of relevant parameters across some multidimensional reality space, including the sample size, number of endogenous variables, number of "key points" or condition values, matrix eigenvalues and eigenvectors, intercorrelations, least squares regression coefficients, means standard errors, and so on.

Although rigorous, experimentally based simulations are essential for definitive analysis of complex problems, they will not always be feasible or even desirable for many program evaluation contexts. Instead, it is important to recognize that simulations are generally a useful tool that can be used to conduct more definitive statistical studies. However, simulations in program evaluation can provide the analyst with the means to explore and probe simple relevant data structures for the purposes of improving the instruction of research, examining research implementation issues and pilot testing analytic approaches for problematic data.


Simulation Home Page
Copyright 1996, William M.K. Trochim