Quasi-Experimental Designs
Trochim's Knowledge Base
References
Other Tutorials
The following web pages are designed as a tutorial on the NEDV-Pattern Matching design for social program evaluations. Wherever possible, layman's terms are used in a question and answer format to facilitate understanding of the subject for the novice user. However, this tutorial assumes the reader is familiar with basic evaluation/experiment terminology, such as may be used in an undergraduate psychology or human service studies course.
The user should note that this information can be accessed through a variety of ways:
linking or "jumping" to other parts of this page or other pages (see
Quick Jump Site) , keyword linking to this and other pages, or
sequentially by reading through the pages (keep clicking on browser "down"
button). Learners are encouraged to "jump" via underlined links to topics that
interest them or that need more clarification.
References are listed at the end of the page along with a description of the work, so that the reader may go directly to the hardcopy for reference.
Keywords: (jump to topic in this
document)
quasi-experiment random assignment versus random selection
variable reliability corrected ANOVA
NEDV-Pattern Matching Design
Introduction
The nonequivalent dependent variables design with pattern matching is somewhat of an obscure design, but nonetheless useful in certain situations where funds restrict the researcher to evaluation of only one group of participants. Simplistically, it is composed of a regular nonequivalent dependent variables design with a pattern matching component.
What is a nonequivalent group design?
In an ideal world, experimental designs would be used for all evaluations because they are the strongest in internal validity due to the random assignment. However, randomization is not always possible in this world, which leads to quasi-experiments. The term "quasi-experiment" is applied to situations in which the researcher does not control the assignment (see random assignment versus random selection) for any reason. This lack of control in turn labels the units(people) "nonequivalent". The nonequivalent group design is the most commonly used design in evaluations of human social service programs because of the inherent impossibility of randomization. In reality, the groups may indeed be different, or they may be statistically equivalent; however, the researcher does not know, and so they are assumed to be different on all pretest variables. This difference is always adjusted for via statistical manipulations (reliability corrected analysis of covariance) because those differences may affect the posttest variables resulting in a biased outcome.
The nonequivalent group design is notated:
N O X O
N O O
Where:
N=nonequivalent groups (nonrandomly assigned)
O=test (pre or post)
X=treatment
(time moves from left to right)
The major threats to validity
of quasi-experiments are the selection threats:
history, maturation, testing, instrumentation, regression, mortality. These are the
strongest threats because of the unknown assignment variable. Whereas in randomized
experiments the threats are expected to affect both treatment and control groups equally,
this may not be true with nonequivalent groups, because they are not random. Hence, the
threats are may act differently on each group because of their pre-existing nonrandom
characteristics.
The important thing to realize with nonequivalent groups is that these differences between
groups may affect the posttest variables and thus the outcomes of the treatment.
Why Wouldn't the Researcher Control
Assignment?
In most social service programs, it is not possible to assign humans randomly to program and control groups. This may be due to ethical or logistical reasons. In educational programs for example, children are already in schools as determined by location or some other factor. With this situation, it is not possible for the researcher to switch them around in order to randomize the kids. So, in this case a nonequivalent groups design must be utilized because the researcher can not control assignment.
What is Nonequivalent Group Dependent
Variables Design?
The NEDV design is one in which one nonequivalent group of people is split into two groups that receive different pretests, both groups receive the treatment, and both receive posttests that correspond to the pretest. In it's most simple form, it may be notated:
N O1 X O1
N O2 X O2
(note: the Ns are from the same group, the treatment X is the same, and the Os are different variables or measurements.)
The idea of the NEDV design is that the program targets one particular variable (either O1
or O2) to change (difference between pre and posttest) and not the other. The NEDV design
uses variables (Os) that are similar and would be affected similarly by all of the
threats. In essence, the nontargeted variable group acts as a control group to the other
group. If the variables are optimally similar, the threats (eg. history, maturation,
regression) are expected to act similarly on both groups. Hence, the difference observed
in the posttests (post-O1 and O2) will be attributable to the treatment or caused by the treatment. Of course, if the
variables are not similar, a critic will argue that all posttest differences are due to
differences in the variable. The challenge, then, is to find and use variables that are
maximally similar, but yet will be differentially affected by the treatment. None too
easy!
To it's credit, the NEDV design is ethically
strong, because it doesn't deny potentially beneficial treatment to any people of the
group.
Pattern matching is just what the name implies: a match between patterns. In the case of evaluation, the patterns are those of the theory as visualized and the outcome revealed in the posttests. In other words, the researcher attempts to match the pattern of the described theory and the pattern of the observed outcome.
The first question that comes to mind is how do we get the
theoretical pattern? This may be accomplished in a number of ways, including
conceptualization or the equivalent of a "nomological
net". Of course, the next natural question is "conceptualization" by
who and of what? Conceptualization of the program outcome may be accomplished by the
researcher or the participants in the study. Typically, they conceptualize or envision
what they hope and expect the treatment or program outcomes to be. Conceptualization is
the holistic perspective, that is manifested in the implementation of the program, also
called operationalization (see contruct
validity).
After conceptualization has "mapped out" what the parties want to happen, the
program is implemented. Then the outcome results may be observed and analyzed for a match
or lack thereof between the original theory and the posttests. Under optimal
circumstances, this difference between groups of posttest results is what represents the
program effect. Depending on the original conceptualization, the program effect may
represent a successful or failed operationalization. The reasons for either may or may not
be apparent in the posttest results.
As the conceptualized pattern or theory becomes more complex and unique, the odds of
corroboration or matching between pre and posttests being purely due to chance become
slimmer. But if there is a close match, the result will be more internally valid due to
the small odds of the match being due to sheer chance. To put it
another way, the chances that the outcome will turn out exactly as conceptualized in the
beginning completely due to chance are very small. Therefore, if there is a strong match
the researcher can make the assumption that the outcome is in fact due to the treatment or
program.
Using the diagram from the NEDV design:
N O1 X O1
N O2 X O2
we notice that the pre and posttests across groups are different. This is to say that the
tests may be on different scales or metrics and still be comparable. Therefore, to compare
across groups is possible, given the above criteria of maximally similar variables.
What is NEDV and Pattern Matching combined?
When these two ideas are combined, a stronger design (than NEDV alone) is created. Basically, the design is that of nonequivalent dependent variables with the outcomes (variables) predicted by a theory (conceptualization). The actual outcomes (posttests) are then compared to the original prediction, and the extent of the match or "fit" is analyzed. The two key factors that increase the probability of ruling out threats are the number of dependent variables (Os) and the specificity of numerical or sign predictions. If the outcomes indeed match the predictions, given high similarity of variables, then that represents a program effect that is also strong in internal validity.
Cook and Campbell explain a little more technically: "(The) nonequivalent dependent
variables designs based on multiple variables and multiple measurement waves will often
permit ruling out all plausible threats to internal validity. But if these threats are not
made explicit before data collection begins, it is unlikely that all the variables will be
measured that are required for matching obtained data with a pattern of relationships that
logically rules out threats to valid causal inference."
The researcher can increase the internal validity even more by randomizing assignment
within program groups (variables). By random assignment of subjects (participants) to
different program groups, threats to validity are also randomly spread out over the
groups.
What are the uses
of NEDV-Pattern Matching?
As the trend toward participatory methods increases in many fields, so does the apparent usefulness of the NEDV-pattern matching design. While the NEDV design assures that no participants will be denied potentially beneficial treatment, the conceptualization process dictates that the group is involved with the program design and molding of expectations. Hence the marriage of the two components includes the benefits of both participation with the strength of a recognized design methodology.
Generally, the NEDV design can be used when evaluation and/or program resources are
limited and will only allow for measurement of one group.
What are the limitations of the
NEDV-Pattern Matching design?
One of the limitations of the combined design format requires many variables so that the effects of the threats can be relegated. If proper variables can not be found then the design is relatively uninterpretable, and another one must be used.
When used alone without pattern matching or any other complimentary design, NEDV is one of
the weakest interpretable quasi-experiments. This is why the NEDV combined with the
pattern matching is preferable to NEDV alone.
Back
to top.
Campbell, Donald T. 1966. Pattern matching as an essential in distal knowing, in KR
Hammond (ed.) The Psychology of Egon Brunswik. Holt, Rinehart & Winston, New
York, NY. The beginnings of pattern matching theory are laid out here.
Cook, Thomas and Donald Campbell. 1979. Validity. Chapter 2 of Quasi-Experimental
Design and Analysis. Jossey-Bass, San Francisco, CA. This chapter from a
"bible" of evaluation techniques provides clear, comprehensive explanations of
the four types of validities and the relationships among them.
Kalyalya, Denny, Khethiwe Mhlanga, Ann Seidman, Joseph Semboja. 1988. Aid and
development in southern Africa: evaluating a participatory learning process. Africa
World Press, Inc., Trenton, NJ. This handbook outlines the year long pilot program, the
Southern Africa Pilot Learning Process. Among the highlights are the importance of
participation in evaluation, and evaluation in aid (both in money and action) projects.
Liguori, Laura. 1987. An example of pattern matching in program evaluation. Cornell
University, Ithaca, NY. This thesis provides clear explanations of structured
conceptualization, the NEDV design, and pattern matching. The remainder of the manuscript
is focused on the actual conceptualization and pattern matching in practice. Ms. Liguori
presents the evaluation of a mental health program in the Ithaca area.
Marquart, Jules Maree. 1988. A pattern matching approach to link program theory and
evaluation data: the case of employer-sponsored child care. Cornell University,
Ithaca, NY. This thesis provides another evaluation example of employer-sponsored child
care using a theory-based evaluation based on administrator conceptualization and pattern
matching on observations of employee satisfaction.
Patton, Michael Quinn. 1978. Utilization-focused evaluation. Sage Publications,
Beverly Hills, CA. Patton provides a rounded explanation of the practical evaluation
practice. Especially useful is the section on evaluation as it is process-oriented. This
is directly related to the conceptualization and pattern matching approach as a learning
tool.
Setze, Rose Jusko. 1994. A nonequivalent dependent variables-pattern matching approach
to evaluate program outcomes: the case of a supported employment program for chronically
mentally ill individuals. Cornell University, Ithaca, NY.
Trochim, William MK. 1985. "Pattern matching, validity, and conceptualization in
program evaluation," in Evaluation Review, vol 9 no 5. Sage Publications,
Inc., Beverly Hills, CA. This is a synthesis of the three ideas as applied to evaluation.
Trochim, William MK. 1989. "Outcome pattern matching and program theory," in Evaluation
and Program Planning, vol 12. Pergamon Press, USA. This chapter outlines the different
types of pattern matching and how they can be used in theory-based research.
Jump to a nifty painting by Pablo Picasso from 
Go to my favorite
team's home page.
Check out Dali's Disintegration of the Persistence of
Memory from the Dali Virtual Museum of
Art.
Browse through my first web attempt, Evaluation Techniques in International
Development Programs.
Back
to top.
Send comments to 
alv4@cornell.edu
