Securing Internal Validity - How to Avoid the Threats Securing Internal Validity - How to Avoid the Threats
Validity is the best available approximation to the truth of a given proposition, inference or conclusion. In other words, when we make some claim, does the evidence support our conclusions. Validity is divided into four categories. They are as follows: conclusion, construct, external and internal.
Internal validity focuses on cause and effect relationships. The notion of one thing leading to another is applicable here. ( Event A leads to Event B). For example, it is believed that listening to Selena twice a day causes adolescent girls to crave fajitas and develop an uncontrollable urge to attack pinatas. However, if we want to test such a claim for internal validity, then we must be able to measure this unusual phenomenon for causality.
As we stated earlier, causality is critical to internal validity. Now, let's see how we might be able to help our Selena fans. There are three components necessary to establish causality.
We must have evidence that the cause precedes the effect.
Here simply we must note a change that demonstrates cause and effect. If the treatment or program is effective then there is a change. On the other hand, if the program is withheld then there is no change observed.
On occasion a treatment plan is implemented and a change is noted, but the change is not the result of the treatment plan. In social science, this is referred to as the third variable or missing variable problem. Despite the fact there is a relationship, it is not clear that there is cause and effect. This brings us to focus of our study of threats to internal validity.
Now, that we understand the notion of cause and effect do you think our Selena hypothesis meets the criteria? You do the math. For more information on the criteria for establishing a cause-effect relationship check out Trochim's home page.
Threats can be defined as issues or concerns that will arise when there are inconsistencies in the data that compromise the causal relationship. These threats fall into seven major categories:
I'm sure you are wondering what does the Magna Carta have to do with cause and effect. Well, you're right - Nothing!! It's not that sort of history we're dealing with, but a more garden type variety. In order to achieve internal validity, we must be careful to account for events that occur during the course of the program that might impact the final outcome. For example, what if you are implementing a health education program in the public schools to create awareness to reduce the incidence of teenage pregnancy. During the program, one of the students becomes pregnant, she eventually drops out of school and commits suicide. Clearly, this event would have a profound effect on the students awareness. So when we measure the effectiveness of the program and we note an increase in awareness, we must be sure to factor in the impact of the suicide on student's awareness when assessing program effectiveness.
This is simply what is sounds like. The everyday human activities that lead to growth that occur naturally as the treatment program is being implemented are potential threats. These include maturation due to age, experience, physical development or anything that leads to increase an knowledge and understanding of the world which can affect program results.
As the old adage goes, "Experience is the best teacher". This is precisely what happens in programs that utilize pre and post test measures. One of the simplest examples is that of testing. For example, just by virtue of having had the experience of taking the GRE once, without any additional preparation, you are more likely to improve your score on a re-take.
The instrumentation threat is caused by inconsistencies with the testing instrument i.e., interviewer, grader, or the test itself. Although a student's math performance may remain unchanged during the course of the program, if the measurement tool is altered, it can show an improvement that may be attributable to the revised test and not the program.
The phenomenon of regression shows that there is a natural tendency for individuals who score on the outer extremes (either very high or very low) of the grade continuum, when retested will score closer to the mean. For example, Sheena, a low scorer on the high school achievement exam upon retaking the exam will improve her scores even without the intervention of any study program due to the regression effect.
Participants dying in your program would definitely not achieve the internal validity you seek. However,luckily you need not worry about that because we are not referring to a death that leads to loss of life, but instead to a common program problem - the issue of participants dropping out of the program. Unfortunately, the loss of program participants can create a false treatment effect that appears to be the result of your program. For example, you want to evaluate a pre-school readiness program for 3-4 yr olds. To determine the effectiveness of the program, you use a pre-post measurement, but during the course of the study several of the low scorers drop-out. The loss of low end scorers will artificially raise the post-test measures creating the impression of an program effect when in fact there is none.
The above internal validity threats are considered single group threats because there is no control group. However, multiple groups designs are also common and statistically stronger and are subject to the same threats as single groups. The difference is simply due to differences between groups. A selection threat is generated when there are inconsistencies in the comparison groups. Below is a list of the multiple group threats. I will not discuss them in detail as they are very similar to single group threats, but will again refer you to the Trochim Home page for more details about multiple group threats. Below you will find a list of the selection multiple group threats.
Selection-HistorySelection Maturation
Selection-Testing
Selection-Instrumentation
Selection-Mortality
Selection-Regression
In addition to single and multiple group threats there is another category of threats known as social interaction threats. Because social research does not take place in a vacum and the human element underpins all research activity, the occurence of social interaction between researcher and subject or among subjects is inevitable. It is this social interaction that can lead to misinterpretation of the cause-effect relationship. The following are the major social interaction threats.
Let's assume for the sake of this exercise that you are studying the impact of art education on student achievement. You have randomly selected two sixth grade classrooms for study in Boise, Idaho. One class recieves arts instruction once a week and the comparison group receives no arts instruction. And since your research is interesting and valuable, you have a large budget from the National Endowment for the Arts and the Dept. of Educationthat which includes a trip to New York City for the program group. Now, let's see what could happen in such a scenario.
This threat occurs if the control or comparison group finds out about the program. Keeping a program like this quiet will be next to impossible so there will be some sharing of information and experience. Also, the students and teachers in an effort to minimize differences may try to implement a similar curriculum for the other group.
Undoubtedly such a program will produce some jealousy (who wouldn't want to spend there afternoon painting and a free trip to NYC) and this may translate into the comparison group developing a competitive spirit. The comparison group may work harder than usual to show up the lucky artsy-fartsy types.
The reverse of the compensatory rivalry threat may also occur. Here rather than competing with the NYC group the comparison group may become angry and upset, and out of frustration stop trying altogether.
Now, this is where you get administrators and parents involved in the process. Naturally, in the above scenario there will be some hurt feelings. Parents will call to complain and demand to know why there child was not included in the new program. In an effort to conduct damage control, a similar or compensatory program is implemented to appease angry parents.
Finally, since social science research is not a perfect science and there are externalities that can effect the outcome of programs, the ability to design a project that limits outside factors is crucial. Therefore having a clear understanding of the threats during the design phase of your program can help you to strengthen internal validity and avoid the nasty comments from critics who review your work.
Save
I-V-Y (Internal Validity - Yes!)Next, to test your knowledge of internal validity threats, play this exciting game to see if you can save IVY. IVY is a together career girl. She's a graduate of Cornell's School of Human Ecology and is a Program Evaluator working for a Washington, DC - based Public Policy Institute. Currently, she is involved in a project that is studying the impact of an after school computer program in Baltimore. When evaluating this project, IVY must be sure to be on the lookout for threats - that is internal validity threats. Help IVY identify the threats and secure Internal Validity. GO!!
1. After reviewing some documentation, IVY learns a test was administered both before and after the program. But the tests were different. Find the threat.
a. History
b. Mortality
c. Testing
2. About half way through your program, the three most at-risk students, and consequently the lowest scores on the pre-test, are arrested and subsequently leave the program. Find the threat!!
a. Maturation
b. Testing
c. Mortality
3. Since all of the students are at-risk and the primary criteria for participation in the program is poor academic performance, when assessing the effectiveness of the program based on pre and post test measures, IVY must be careful of what threat to internal validity? Find the threat.
a. Mortality
b. Regression
c. Testing
4. Since these are the only computers on campus, other students want access as well so they tell there parents. Parents begin calling complaining and demanding to know why their child can not use the computers. Bowing to pressure from the parents and the School Board, the principal provides access other students during school hours. Find the threat.
a. Diffusion or Imitation of Treatment
b. Compensatory Equalization of Treatment
c. Mortality
5. The program is one year project Over the course of the program the students are students naturally developing and growing and increasing their knowledge of the world. Find the threat.
a. Regression
b. Mortality
c. Maturation
Copyright @ 1997, Paul A. Burns
Trochim Home Page