There are several ways to strengthen your analysis. The first, discussed in Reliability and Validity: What's the Difference?, is to address as many threats to the validity of your research as possible. In addition, employing the methods discussed to achieve an accurate estimate of the reliability of your measure is also a good idea. This section will provide you with a brief overview of two other ways in which you can strengthen your analysis: basic statistics and research design.

Statistics

There are three primary ways to improve conclusion validity:

1.Good reliability of your measure, as discussed in the first section of this web page;

2.Good implementation usually achieved by standardizing the way in which your program is administered; and

3.Good statistical power.

I'd like to focus a bit on statistical power. There are four primary components of statistical power: the sample size, the effect size, alpha level and power.

Sample size is simply the number of people or units available to be studied.

Effect Size is simply the ability to detect an effect relative to the other factors that appear in your study.

Alpha level refers to the likelihood that what you observed is due to chance rather than your program.

Power is the likelihood that you will detect an effect from your program when it actually happens.

For additional terms that are related to statistical inferences go to the Glossary of Terms. Finally, remember that the purpose of most research is to assess or explore relationships among a set of variables, and that the use of some straightforward, thought-out basic statistical methods can really enhance the strength of your findings.

Research Design

The other way to strengthen your analysis is through a thought-out research design. There are three primary types of research designs:

1. Randomized or true experimental designs, which are the strongest design for establishing a cause and effect relationship, as random assignment is used - and the groups involved are considered equivalent. This helps to reduce all of the multiple group threats to internal validity discussed earlier.

2.Quasi-Experimental designs are those which employ multiple measures or a control group without randomly assigning participants to group. The ability of these designs to establish a cause effect relationship is dependent upon the degree to which the two groups in the study are equivalent.

3. Non-experimental designs do not employ multiple measure, do not use a control group and do not use any random assignment in its design. These are usually descriptive studies conducted using a simple survey instrument only once. While they are useful in their own right - they are weak in establishing cause/effect relationships.

For a comprehensive discussion of the variety of research designs that fit into all these categories, check out Bill Trochim's Center for Social Research Methods. Finally, a good research design is realistic in its scope relative to the resources of your study, appropriate for the context in which your treatment will occur, and involves a few good steps to measure your effect efficiently.

Back to Home Page