Program evaluation of social services set the stage for a variety of research opportunities. One of the primary goals of evaluators conducting this type of social research is to construct research designs that are reliable and valid so high quality evaluations can be conducted while enhancing scientific knowledge.
A concern for most evaluators of human service programs is the complex nature of the phenomena under study, the human experience. Multiple perspectives are required in order to reflect the richness of these complexities. In addition, due to the fluid nature of human behavior rigorous attention must be directed toward threats to Internal Validity in social science research endeavors .
This WEB site will examine the merits of integrating Qualitative and Quantitative research methodologies in the form of Triangulation, in order to strengthen the Internal Validity of social scientists research.
Research
Engine for the Social Sciences Social Science Data Archives and Data Libraries
Social Science Resource
Center
Some academians claim that the heated debates between the bi-polar quantitative and qualitative methodological encampments are passe, however it appears that the literature continues to contain many works by social science researchers that accept one epistemological perspective to the exclusion of others. In addition few WEB sites have been constructed discussing integration of qualitative and quantitative methodologies via Triangulation.
>From the perspective of a fledgling social researcher I think it is time that we realize the complex nature of the context in which we aspire to conduct our research. Human phenomenon cannot be completely controlled or isolated in a sterile environment. Quantitative research designs including measurement, prediction and causal inference do not always fit in isolation with the world of social science where perceptions, feelings, values, and participation are frequently the variables we are attempting to measure.
By omitting qualitative methods, the social science researcher may overlook many phenomena that occur within the context of the setting. Campbell notes that quantitative measurements rests on qualitative assumptions about which constructs are worth measuring and how constructs are conceptualized. (Shaddish & Cook, 1991).
By omitting quantitative methods casual relationships between variables as well as quantification and analysis of those variables to determine statisti cal probabilities and certainty of a particular outcome will be flagrantly absent.
I propose that the inherent differences between Quantitative and Qualitative research methodologies be used to the advantage of the social science researcher. By combining the different perspectives a more comprehensive research design can be constructed.
The following suggestions were adapted from an article by Dr. Mary Duffy in 1987. Dr. Duffy was actually outlining differences in methodologies, however I will combine the perspectives in order to illustrate the benefits of multiple methods in the study of human phenomenon.
While the Quantitative design strives to control for bias so
that facts can be understood in an objective way, the Qualitative approach is striving to
understand the perspective of the program stakeholders, looking to firsthand experience to
provide meaningful data.
The accumulation of facts and causes of behavior are addressed
by quantitative methodology as the qualitative methodology addresses concerns with the
changing and dynamic nature of reality.
Quantitative research designs strive to identify and isolate
specific variables within the context (seeking correlation, relationships, causality) of
the study as the Qualitative design focuses on a holistic view of what is being studied
(via documents, case histories, observations and interviews).
Quantitative data is collected under controlled conditions in
order to rule out the possibility that variables other than the one under study can
account for the relationships identified while the Qualitative data are collected within
the context of their natural occurrence.
Both Quantitative and Qualitative research designs seek reliable
and valid results. Data that are consistent or stable as indicated by the researcher's
ability to replicate the findings is of major concern in the Quantitative arena while
validity of the Qualitative findings are paramount so that data are representative of a
true and full picture of constructs under investigation.
By combining methods, advantages of each methodology complements the other making a stronger research design with resulting more valid and reliable findings. The inadequacies of individual methods are minimized and more threats to Internal Validity are realized and addressed.
Qualitative MethodsAn interesting argument for the utilization of qualitative methods and how this methodology relates to reality experienced by human beings. The author offers a provocative paper based on social research as discovery. He uses the philosopher, Kleining who questions the issue of how we, as human beings find out about the world we live in. The author talks generally about how research methods are a matter of development from the every-day strategies of openness and discovery rather than from closure and interpretation. This is an excellent paper in that it assists the researcher in developing skills in thinking about reality and the importance of incorporating qualitative methods into research designs in order to better capture the reality of human beings.
The Human Side of being Human Harvey Jackins provides background insights into reality and how humans function in this reality. This site provides text which incites a broader view of reality and the human being's position within.
Pitfalls of Data Analysis This is an excellent site on pitfalls of data analysis. Clay Helberg of the University of Wisconsin informs readers how to avoid "Lies and Damned Lies". The Web Site is written in an informal, chatty fashion. Trickier aspects of applied data analyst are reviewed as Clay talksabout sources of bias, errors in methodology, and problems with interpretation. The graphics are quite good although black and white only. Included is a great list of areas for potential problems when using statistical analysis of data. Clay offers a good discussion on the importance of the researcher understanding the conditions for causal inference. If a causal inference needs to be made a random sample should be drawn. As is the case in some social research, random samples are not possible. In this instance added effort is needed to discover causal relationships by using a variety of approaches.
Qualitative MethodsThis site is a qualitative methods course description created by Professor Peggy Beranek. Its utility is the extensive reference list which provides the WEB user with literature documenting the utilization of qualitative methods.
Introduction to Validity This site provides an indepth look at validity in social science research.The Knowledge Base serves to organize the different types of validity in a clear manner that makes complex concepts more tangible. Superb graphics, well constructed and organized. A must for any WEB user with questions or areas of confusion relating to validity. Some sections are under construction so even greater information benefit is forthcoming.
Once the social science researcher has identified variables that covary, the next major
step is to determine whether or not there is any causal relationship between the two. If
causality is established, the researcher must then decide whether the direction of
causality is from the independent variable on the dependent variable or vice versa. In
social research this is often a major challenge since human phenomenon typically does not
occur in neat little boxes. We cannot make an epistemological assumption that the social
world behaves consistently so that objective forms of measurement can be used in
isolation. Further, we should not strip the data completely from their natural context but
rather strive to understand human behavior from the stakeholder's own frame of reference.
Knowledge of a time sequence is of vital importance in order to ascertain the direction of causality . The social science researcher might use a preprogram questionnaire, indepth interview or a combination of both methods prior to program interventions {Time 1}, then once again among the same participants after a predetermined length of time in the program {Time 2}. In a perfect world, the social science researcher could establish a causal relationship between program intervention and changes in participant behavior from the above described method, however there might very well be a third-variable lurking in the shadows that can cause the researcher to assume incorrectly that the program has no effect{false negative/Type II Error} or that the program has an effect when it actually does not {false positive/Type I Error}.
"Accounting for third-variable alternative interpretations of presumed A-B relationships is the essence of internal validity". (Campbell & Stanley, 1963, p. 50). A strategy that can be used to illuminate third-variable alternative interpretations is Triangulation.
Triangulation is a term used in navigation to describe a technique whereby two known or visible points are used to plot a third. Campbell (1956) was the first to apply the term "triangulation" to research methodology. (Breitmayer, 1993). "Triangulation combines independent yet complementary research methods to:
enhance the description of a process or processes under study
identify a chronology of events
provide evidence for internal validity estimates
serve as a corroborating or validating process for study
findings. Thus, an expanded understanding and contextual representation of the studies
phenomena result". (Hinds and Young, 1987, p. 195).
Methodological triangulation can be classified as simultaneous or sequential. "Simultaneous triangulation is the use of the qualitative and quantitative methods at the same time. In this case, there is limited interaction between the two datasets during the data collection, but the findings complement one another at the end of the study. Sequential triangulation is used if the results of one method are essential for planning the next method. The qualitative method is completed before the quantitative method is implemented or vice versa". (Morse, 1991, p. 120).
Determination of the specific research problem is the first step in qualitative-quantitative triangulation. This can be accomplished by identifying whether the theory that drives the research is developed inductively from the social science researcher her/himself or deductively as is characteristic in quantitative inquiry. Mitchell (1986) suggests that triangulation offers flexibility and an in-depth approach that single method designs cannot provide. Four principles must be adhered to, however according to Mitchell in order for triangulation to be used effectively:
"1. the research question must be clearly focused
2. the strengths and weaknesses of each chosen method must complement each other
3. the methods should be selected according to their relevance to the nature of the phenomenon being studies, and
4. continual evaluation of the approach should be under-taken during the study." (Corner, 1990, p. 721).
Benefits of triangulation have been identified by several social science researchers. Madey (1982) discusses using exploratory interviews and/or observations in improving the sampling framework. Data collection using observation and exploratory interviews can provide information about the receptivity and frames of reference of program participants prior to the construction of quantitative survey instruments. As a result, better instruments are created as well as improved methods of instrument administration.
Mary Duffy, (1987), cites nine benefits associated with Triangulation:
The conceptual framework, which provides the theoretical base of
the study, can be developed in whole or in part from qualitative methods. In areas where methods produce information overlap, certain quantitative
results can be verified by results obtained through qualitative methods.
Qualitative data gained from interviews and/or observations can
be used as the basis for selecting survey items to be used in instrument construction.
External validation of empirically generated constructs can be
obtained by comparison with interview and/or observation data: where discrepancies exist,
additional probing can be done to determine whether the mismatch was because of a weakness
in the instrument or to misinterpretation by the individuals taking the test.
Case studies can be used to illustrate statistically derived
models.
Clarification of ambiguous and provocative replies to individual
questionnaires can be observe by reexamining field notes.
Quantitative data can provide information about program
stakeholders who were overlooked initially.
The use of a survey instrument that collects data from all
program stakeholders in the study may serve to correct the qualitative research problem of
collecting data only from an elite group within the system being studies.
Using quantitative assessment can correct for the "holistic
fallacy"; (the perception by the researcher that all aspects of a given situation are
congruent, when in fact only those persons interviewed by the researcher may have held
that particular view). Also the use of quantitative instruments can verify observations
collected during informal field observations. (p. 132).
Although triangulation moves the social science researcher closer to convergence, corroboration and correspondence of results across different method types, threats to Internal Validity must be recognized and minimized.
Decision MatrixThis site provides a superb analogy using the proceedings from the OJ Trial and the possibility of making a Type I or Type II error. While this site may seem to use an issue that is quite controversial, the example assists the WEB user in thinking about making inferences based upon research findings and potential errors that are inherent in most social science research projects.
Donald Campbell thought very deeply about construct validity and actually came up with a complex technique that involves the use of a multi trait multi method matrix. The multi trait multi method matrix requires convergent and discriminate validity as conditions for naming something. Campbell espouses multiple operationalism, the belief that many measures are needed to triangulate on a single construct.
While the MTMM is a systematic approach to assess construct validity, no such approach has been developed to assist the researcher think about threats to internal validity. Threats vary depending upon the context of each individual research environment. Specific threats to internal validity have been identified by many and totally conquered by few.
Multi-Method Matrix
Dr. Trochim from Cornell University once again provides very readable text with vivid graphics that discuss and illustrate the concepts of the Multi-Method Matrix. Trochim describes the validational process of utilizing a matrix of intercorrelations among tests representing at least two traits, each measured by at least two methods.Designs A brief illustration of various types of experimental and quasi-experimental designs are provided in an abridged form. Specific threats to internal validity are numbered 1-8. With each design, potential threats are listed. Very useful during the planning process of a research project. The social science researcher can trouble shoot depending on the particular design she/he decides to use.
Design Should Meet Certain Criteria A succinct definition of internal validity is provided by the Florida Agriculture Information Retrieval service. Examples are given outside the context of Social Science (i.e., environment, plant growth) however illustrations are straight forward and provide a different context in which to think about design issues. The page appears to be under development, so further discussions concerning design might be forthcoming.
Introduction to Internal and External Validity This site does a very nice job discussing internal validity along with various threats that can interfere. Dr. William Huitts of Valdosta University has established this site for his students thus explaining the psychological bent noted in the the writings. The discussion of internal validity is in clear terms with good discussions of 8 threats to internal validity. Language of the text is helpful because Dr. Huitt links independent and dependent variables in his discussion of internal validity and potential threats. Good as a secondary resource.
In order to provide a point of reference, I will offer a hybrid research design that illustrates the use of Triangulation as a means of addressing several common threats to Internal Validity.
Let us assume that we wish to study the effects of a prenatal program called Healthy Beginnings on pregnant adolescent females self-care abilities. Healthy Beginnings is a program consisting of nursing interventions that occur at the same time as the prenatal visits. The intervention is by and large counseling, referral to social services such as WIC, home visits before and after delivery and contraceptive information and prescriptions)
Initially concept mapping would be conducted among program planners, program implementers, teens, nurse midwives involved in seeing participants during prenatal visits and social workers working with teens. The goal of creating this concept map is to assist the Healthy Beginning stakeholders to work collectively as a group while maintaining their own individual perception of the program. Concept mapping is a structural process, focused on the construct of interest (Self-Care perception of adolescent mothers). Input from a range of program stakeholder is required in order to create a conceptual map of ideas and meanings of the program. This would be particularly important among the teen participants, who are readily 'put-off' by vacuous, pretentious terminology.
Two randomly selected groups of teens would be selected, half participating in the Healthy Beginnings program, the other half seen by the Obstetric physician for prenatal care, only. The two groups would be similar demographically and developmentally. Dissimilar strategies and methods would be used within the same research design. Triangulation of methods that are different would:
reflect the theorized mulitdimensionality of the construct of
self-care
provide more detail about the meaning teenagers attach to the
phenomenon of self-care as it relates to pregnancy, parturition, and motherhood
index a process of change in the perception of self-care
abilities as was theoretically predicted
methods will be diverse and independent of each other
methods will be suitable for use in the field setting.
Dissimilar strategies including observations, structured interviews, self-report questionnaires, and document review would be employed. These methods could be viewed as compensatory, as the limitations of one are offset by the strengths of the other. For example, review of documents (patient chart, prenatal information sheets, postpartum discharge summary) could be used to counterbalance the reactive influence of the social science researcher's presence on the adolescent female's self-report data. Observational data, which can become contaminated by the researchers bias could be compared with or checked against an adolescent's questionnaire (measuring perceptions regarding self-care) and interview responses. Interviews could include question that were open-ended so that the predetermined, defined and limited foci of the questionnaire would be offset. The questionnaire could be administered prior to participating in the Healthy Beginnings Program as well as a follow-up questionnaire at the 6 weeks post-partum visit. In addition a random sample of teens could be interviewed post-partum from both the treatmentment and non-treatment groups.
This selection of methods would hopefully have dissimilar biases and therefore result in less systematic effects of participant and investigator based errors leading to problems with the internal validity of the research.
Teen Pregnancy
This site is very provocative in that it shares actual stories of teenage women who are pregnant and how their lives are changed as a result. The text is fairly simplistic, however the case study approach in describing the teen's situations (along with color pictures of the young women) is an effective mechanism for keeping the reader interested and heightens awareness concerning the social problem of teenage pregnancy.Concept MappingThis site provides excellent graphics with understandable text and examples. Included are additional links for specific examples of concept mapping in use and extensive general informationThis process would express the conceptual framework in the language of the participants rather than in the rhetoric of the social science researcher.
In theory the use of Triangulation seems like a logical way to strengthen the Internal Validity of social science research. Researcher, however must never rest on their laurels and depend on the methodology alone to insure solid, internally valid work. The following caveats are important to keep in the frontal lobe as we endeavor to conduct social science research in a dynamic never static environment.
Unit of Analysis: a common unit of analysis is required to guide
the data collection and analysis. The common unit needs to be a part of all aspects of the
triangulation. Time and money constraints: the time and money
required to combine different approaches of data collection and analysis is likely to be
considerable.
Investigator demands: the investigator who wants to use multiple
triangulation successfully needs a broad theoretical perspective and a broad knowledge
base of research methodology, including both quantitative and qualitative methods. Also
required is the ability and desire to deal with complicated design, measurement and
analysis issues.
Data Analysis: analysis of data generated by multiple
triangulation is a difficult problem that has yet to be solved. This would be particularly
important among the teen participants, who are readily 'put-off' by vacuous, pretentious
terminology. Dissimilar strategies including observations, structured interviews,
self-report questionnaires, and document review would be employed. These methods could be
viewed as compensatory, as the limitations of one are offset by the strengths of the
other. For example, review of documents (patient chart, prenatal information sheets,
postpartum discharge summary) could be used to counterbalance the reactive influence of
the social science researcher's presence on the adolescent female's self-report data.
Observational data, which can become contaminated by the researchers bias could be
compared with or checked against an adolescent's questionnaire and interview responses.
Interviews would include question that were open-ended so that the predetermined, defined
and limited foci of the questionnaire would be offset. The methods selected would
hopefully have dissimilar biases and therefore result in less systematic effects of
participant and investigator based errors leading to problems with the internal validity
of the research.
The literature provides few guidelines. Numerous questions are generated by the analysis issue such as:
How to combine numerical data, linguistic and textural data
How to interpret divergent results between numerical and
linguistic data
What to do with overlapping concepts that emerge from the data
and are not clearly differentiated from each other
Whether and how to weight data sources
Whether each different method used should be considered equally
sensitive and weighted equally
Methodological triangulation is not the panacea for every social science research project. We as social science researchers should be mindful however, that one methodology can narrow a researcher's perspective and can deprive him/her of the benefits of building on the strengths inherent in a variety of research methodologies. Triangulation can maximize the strengths and minimize the weakness of each individual approach while strengthening research results and contributions to theory and knowledge development. The benefits of triangulation also serve to enrich and deepen our understanding of the research environment while seeking convergence, corroboration, and correspondence of results across the different method types. This framework highlights the integrative potential of these strategies, and underscores their potential power not only to incorporate qualitative and quantitative analyses, but also vice versa, and, even beyond, to spiral iteratively around the different data sets, adding depth of understanding with each cycle. (Caracelli & Greene, 1993). Through this process threats to internal validity can be recognized and addressed.
While methodological triangulation can enhance, illustrate, and clarify research
findings, the researcher should keep in mind that use of multiple methods can also lead to
the discovery of paradox and contradiction. In addition to considering the caveats listed
above, at the onset of the research project, the social science researcher must
meticulously develop a comprehensive conceptual framework for methodological triangulation
which includes planning for data analysis along with planning the design of the study. The
analysis of research findings from one methodology can then provide a set of substantive
categories that is used as a framework applied in analyzing the remaining research
findings. (i.e. Indepth interviews or concept mapping to inform questionnaire
development).
Breitmayer, B.J. (1993). Triangulation in Qualitative Research: Evaluation of Completeness and Confirmation Purposes. IMAGE:Journal of Nursing Scholarship 25(3), 237.
Campbell, D.T. & Fiske, D.W. (1959). Convergent and discriminant validation by the multi-trait-multi-method matrix. Psychological Bulletin, 56, 81-105.
Caracelli, V. & Greene, J. (1993). Data Analysis Strategies for Mixed-Method Evaluation Designs. Educational Evaluation and Policy Analysis, 15(2), 196.
Corner, J. (1990). In search of more compete answers to research questions. Quantitative versus qualitative research methods: is there a way forward? Journal of Advanced Nursing, 16, 718-727.
Duffy, M.E. (1887). Methodological Triangulation: A Vehicle for Merging Quantitative and Qualitative Research Methods. IMAGE: Journal of Nursing Scholarship,19(3), 130-133.
Hinds, P. & Young, K. (1987). A Triangulation of Methods and Paradigms to Study Nurse-Given Wellness Care. Nursing Research, 36(3), 195.
Mady, D. (1982). Benefits of Qualitative and Quantitative methods in program evaluation, with illustrations. Educational Evaluation and Policy Analysis, 4, 223-236.
Mitchell, E. (1986). Multiple Triangulation: A methodology for nursing science. Advances in Nursing Science, 8, 18-26.
Morse, J. (1991). Approaches to Qualitative-Quantitative Methodological Triangulation. Nursing Research, 40(1), 120.
Patton, M. (1990). Qualitative Evaluation and Research Methods. SAGE Publications, Newbury Park. 464.
Shaddish, W, Cook, T. Leviton, L. (1991). Foundations of Program Evaluation: Theories of Practice. SAGE Publishing Company. Newbury Park.
Trochim, W. (1982). Designing Designs for Research. The Researcher, 1(1), 195-200.
Trochim, W. (1989). An Introduction to Concept Mapping for Planning and Evaluation. Evaluation and Program Planning, 12, 1-16.
Kathryn A. BowenCopyright ©: 1996, Kathryn A, Bowen, Revised April 1, 1996