Research Design and Mixed-Method Approach

A Hands-on Experience

This tutorial web site provides hands-on experience integrating different research methods into a research strategy. The use of mixed-method is likely to increase the quality of final results and to provide a more comprehensive understanding of analyzed phenomena. Hyperlinks will connect you to other sites where you can learn more about research design.

Getting Started
Case Study: "Bolsa-Escola"
Evaluation: Audience, Purpose, and Issues
Research Design
Why Mixed-Method
References
Read More About

Getting Started

The definition of what is a good evaluation varies according to initial assumptions, values, and philosophical positions shared by the evaluator and based on the intended uses of the results of an evaluation. One dimension that unites evaluators, however, is a particular concern regarding the quality of their work. In some sense, that might explain why research methodology is a topic to which evaluators pay so close attention and even fight for it.

Within the so-called quantitative tradition, quality standards have been defined using the concept of validity (Cook and Campbell, 1972). This concept is a cumulative process with four steps. The initial steps are to assess whether a relationship exists between two variables (conclusion validity) and to determine if this relationship is causal (internal validity). The third examines if the theoretical model is well depicted by the means through which it was operationalized (construct validity). Finally, external validity examines if, and to what extent, findings can be generalized to other groups, places, and times.

This conceptualization of validity has been very influential even within the so-called qualitative tradition, wherein a solid approach to assess the quality of interpretative inquiry is the truthworthiness criteria (Lincoln and Guba, 1985; Guba and Lincoln, 1989). Besides the critiques to the classical approach of validity, these criteria include the notions of credibility and transferability that are parallels to the concepts of internal validity and external validity, respectively.

These parallels suggest that the dichotomy--quantitative versus qualitative--might not be so incompatible as purists from both sides have argued. More than that, studies using mixed-method have shown that integration of these traditions within the same study can be seen as complementary to each other (Greene and Caracelli, 1979; Caracelli and Greene, 1997).

So, to give you hands-on experience, let us examine a case study and a research strategy to evaluate it. This strategy is placed within a mixed-method approach, and potential benefits of such an approach are highlighted. This will put into perspective your knowledge of research design.

Case Study: "Bolsa-Escola"

[Children - Bolsa-Escola] Four years ago, the Secretary of Education of Brasília, the federal capital of Brazil, started an innovative educational program called "Bolsa-Escola" (Basic School Fellowship). According to specialists, "Bolsa-Escola" is not a simple educational or welfare program for poor families. It is a program that addresses children's education. It also helps their families, whose difficulties are at the root of childhood neediness. The program provides an additional monthly income of approximately $100 U.S. (current minimal monthly wage in Brazil) for poor families that have all their 7-14 year-old children enrolled and regularly attending classes in the nearest public school.

To be eligible, children in the family must attend a public school, the family has to live in Brasília for at least five years before the actual enrollment in the program, and the family must be considered "poor" according to a scale. This scale takes into account per capita income, type of dwelling (owned or rented), number of children in the household, whether father and/or mother are employed, and the number and type of electric/electronic devices in the house. Instead of food stamps, benefits in goods or services (e.g. clothes, shelter), the program gives mothers, but not fathers, money to spend the way they want. They can buy groceries, pay bills, or drink cachaça (the Brazilian vodka or tequila) in a bar. Two unjustified absences of a child from school in a month are sufficient to cancel the benefit for that month (Policarpo Jr. and Sandra Brasil. 1997).

Evaluation: Audience, Purpose, and Issues

Within an evaluation, we must be selective in terms of the audience, purpose, and issues to be addressed. In this particular case, the audience comprises policy makers, professionals of the Secretary of Education, and managers directly involved with the program. The main purpose of the evaluation is to assess the efficacy of the program in order to make decisions about its future.

The main issues the evaluation will address are as follows:

Enrollment/Withdrawal
- What was/is the difference in school enrollment/withdrawal in Brasília before and after the program started?
- What is the difference in the number of children abandoning school since the program started (in general and within the program)?
- How does program participants' class attendance compare to that of nonparticipants?
Performance
- What was/is the difference in students' performance at the end of the school year before and after the program started?
- How does program participants' performance compare to that of nonparticipants?

Research Design

Research design refers to the strategy to integrate the different components of the research project in a cohesive and coherent way. Rather than a "cookbook" from which you choose the best recipe, it is a means to structure a research project in order to address a defined set of questions (Trochim and Land, 1982).

Considering the nature of the "Bolsa-Escola" program, types of research design, and specific strengths of the major quasi-experimental designs, we decided to adopt the regression-discontinuity design. The major assumptions of this design and its relation to our case study are:

The Cutoff Criterion. Children and their families are assigned to the program based on a defined socioeconomic scale, creating two distinct groups: a) children belonging to low-income families (program group) who, therefore, will receive financial support (treatment); and b) children belonging to families above this income level who, therefore, will not receive any additional benefit (control).

The Pre-Postprogram Measures. The major sources of information for both issues-- enrollment/withdrawal/attendance and students' performance--are official school records. Complementary data come from application forms and initial interviews with parents before the child/family is formally enrolled in the program. For both issues, two dimensions are considered before and after the program was implemented, as well as during implementation of the program (program group versus control group).

Statistical Issues. We will assume that the requirements regarding the statistical model are fully met, including statistical power (42,000 children enrolled in the program).

Program Implementation. We will assume that the program is implemented according to the guidelines and there is no major delivery discrepancy.

[notation - r-d design] The figure expresses the regression-discontinuity design in notation form. The letter "C" indicates that groups are assigned by means of cutoff (not randomly); the "O" indicates "pre-postprogram" measures, and the "X" indicates the treatment. The first line refers to the program group and the second to the control group.

Why Mixed-Method

Though regression-discontinuity is strong in internal validity and can parallel other non-equivalent designs in terms of validity threats, interpretation of results might be difficult. Outcomes might be the result of combined effects of factors (e.g. better training of teachers, improvement in school facilities) that are not exactly related to the program per se. Depending on the statistical results, it might also be difficult to assess the efficacy of the program. Adding qualitative flesh to the quantitative bones is a good strategy to overcoming some of these problems.

Among the purposes for mixed-method evaluation design, Green et al. (1989) highlight five major ones that might enhance the evaluation as follows:

Triangulation. tests the consistency of findings obtained through different instruments. In the case study, triangulation will increase chances to control, or at least assess, some of the threats or multiple causes influencing our results.

Complementarity clarifies and illustrates results from one method with the use of another method. In our case, in-class observation will add information about the learning process and will qualify the scores and statistics.

Development results from one method shape subsequent methods or steps in the research process. In our case, partial results from the preprogram measures might suggest that other assessments should be incorporated.

Initiation stimulates new research questions or challenges results obtained through one method. In our case, in-depth interviews with teachers and principals will provide new insights on how the program has been perceived and valued across sites.

Expansion provides richness and detail to the study exploring specific features of each method. In our case, integration of procedures mentioned above will expand the breadth of the study and likely enlighten the more general debate on social change, social justice, and equity in Brazil and the role of the public and private sector in this process.

In sum, the examination of this case study helps us see that a research strategy integrating different methods is likely to produce better results in terms of quality and scope. In addition, it encourages us to probe the underlying issues assumed by mixed-method. Two of them are professionals with broader technical skills and financial resources to cover "extra" activities.

Mixed-method is a way to come up with creative alternatives to traditional or more monolithic ways to conceive and implement evaluation. It is likely that these alternatives will not be able to represent radical shifts in the short run. However, they are a genuine effort to be reflexive and more critical of the evaluation practice and, ideally, more useful and accountable to broader audiences.

Cited References

Caracelli, Valerie J. and Greene, Jennifer C. 1997. "Crafting mixed-method evaluation design." In J. C. Greene and V. J. Caracelli (eds.), Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. New Directions for Program Evaluation, No. 74. San Francisco, CA: Jossey-Bass, pp. 19-32.

Cook, Thomas D. and Campbell, Donald T. 1979. "Validity." In T.D. Cook and D.T. Campbell. Quasi-experimentation: Design and analysis for field settings. Boston, MA: Houghton Mifflin, pp. 37-94.

Guba, Egon G. and Lincoln, Yvonne S. 1989. "Judging the quality of fourth generation evaluation." In E.G. Guba and Y. Lincoln. Fourth generation evaluation. Newbury Park, CA: Sage, pp. 228-51.

Greene, Jennifer C. and Caracelli, Valerie J. 1997. "Defining and describing the paradigm issue in mixed-method evaluation." In J. C. Greene and V. J. Caracelli (eds.). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. New Directions for Program Evaluation, No. 74. San Francisco, CA: Jossey-Bass, pp. 5-18.

Greene, Jennifer C., Caracelli, Valerie J. and Graham, Wendy F. 1989. "Toward a conceptual framework for mixed-method evaluation design." Educational Evaluation and Policy Analysis, 11(3), pp. 255-74.

Lincoln, Yvonne S. and Guba, Egon G. 1985. Naturalistic inquiry. Beverly Hills, CA: Sage.

Policarpo Jr. and Sandra Brasil. 1997. "Casa e escola: Governo do Distrito Federal ajuda crianças pobres - e também os seus pais." Veja, edition 1516 (October, 8, 1997), 30 (40), São Paulo, SP: Editora Abril, pp. 74-6.