**Also note: The events and characters portrayed here are purely fictional. If anyone or any situation resemble you or your own situation in any way -- join the club.
On to the topic at hand. Picture this if you will:
A bubbly, over-eager, graduate student, we'll call her Susie, comes bouncing into her first committee meeting. Enthusiastically, she recites her ideas for her masters thesis:
"I am planning on doing a study which looks at increasing the level of parents' involvement in their children's' lives, particularly parents of Head Start students. I figure I will take a couple of Head Start classrooms in the area and provide the parents with a series of parenting classes. The topics of the parenting classes could include things like: practicing positive discipline, cooking with kids, how to handle problematic peer relationships, and multicultural curriculum. I could measure their involvement with their children both before and after the classes and see if my program made a difference. Sounds great, Eh?!"
The committee members glanced at each other with furrowed brows. Dr. Doolittle was the first to comment: "What about validity Susie? How do you plan on addressing validity issues in your study?"
"Validity? What do you mean exactly?", said Susie, slumping down like a deflated balloon.
"Well," said Dr. Doolittle, "Maybe we should start at the beginning. Validity is the best available approximation to the truth of a given proposition, inference or conclusion (click here for a quick overview of validity -- this will be necessary if you have little exposure to the topic of validity). Generally, validity is subdivided into four parts: construct validity, conclusion validity, internal validity, and external validity. Now, ...".
"Um, excuse me Dr.D," interjected Dr. Muffy, a sweet, older woman who felt as if she should slow down the good doctor before he gave her favorite graduate student an information overload. "Why don't we just start with construct validity today. In fact why don't we even narrow it down further to the THREATS TO CONSTRUCT VALIDITY?"
"Sure, sure," replied Dr. Doolittle. "Where was I? Construct validity refers to generalizing from your program or measures to the concept of your program or measures; it is an issue of labeling. In other words, when you deal with the concept of construct validity you can ask yourself the question -- DID I MEASURE WHAT I THOUGHT I MEASURED?
There are many issues and topics to consider when addressing concept validity. There are the types of measurement validity, the nomological network, the multitrait-multimethod matrix, and then of course there is pattern matching...."
"But we agreed that we are going to focus on the THREATS to construct validity today --RIGHT DR. DOOLITTLE????" Dr. Muffy was getting a bit impatient at this point. She knew what a complicated topic validity was, and how overwhelming it could be for someone who was first starting out.
"Right, threats."
"And only hit the major threats, please," Dr. Muffy continued, "I would like to go to lunch sometime this year." Susie shot her a grateful look and settled back to listen to Dr. D spew about these "threats".
"Threats refer to questions and issues that may be raised by critics (both of the friendly and non-friendly variety) of your research. There are 10 threats we will address today. The first one is INADEQUATE PREOPERATIONAL EXPLICATION OF CONSTRUCTS...."
"I'm going to operate on who, why?" Susie interrupted.
"No, dear," said Dr.M, "that's just a fancy way of saying that you didn't do a very good job of operationally defining your constructs. In your case, you will need to think through concepts such as 'parent involvement'. What do you mean by parent involvement? Do others share your view? In order to address this threat, it often helps to illicit expert opinions and use specific methods (such as concept mapping) to better define the construct."
"Right, as I was saying," interjected Dr.D, always insisting on being the center of attention, "the next two threats are MONO-OPERATION bias and MONO-METHOD bias..."
"Basically, the mono-operation threat refers to using only one version of your treatment. In the case of your proposed study this would refer to having only one version of parenting classes; critics may argue that any conclusions drawn may refer to your particular version of the program, not the actual construct. In order to avoid this threat, you could employ multiple versions of your parenting program.
On the other hand, a mono-method threat is basically the same thing as a mono-operation threat, except that it refers to the inadequacy of using a single measure to look at a particular concept. Your critics may ask you how you can be sure you are measuring parent involvement, if you are only using one measure. The answer? Use several methods!"
"That doesn't sound too bad, " said Susie, "but I have a feeling that's not the whole story..."
"That's not even the half of it!" exclaimed Dr. Doolittle. "Oh no indeed, next there are the INTERACTION THREATS. The first of these refers to interactions between different treatments. You need to be sure that the results you are obtaining are a result of your parenting classes and not some combination of activities in which your parents are involved. For example, maybe the parents in your study are also participating in activities at church or through a neighborhood organization. These activities may encourage more involvement with their children as well. It may be that the reinforcement of both programs working in conjunction with each other is what prompts them to increase their involvement with their children.
The other type of interaction that may occur is an interaction of the testing and the treatment. Simply, by giving your parents a pretest you may heighten their awareness and sensitivity to parent involvement. Because you have made them aware of the time they spend with their children, it may cause parents to reflect and increase involvement levels. When you label your parenting classes the program, you are leaving out the pre-test, which also may be influencing the construct."
"Very well said Dr. Doolittle, "commented Dr.M, "Take a break and I will continue with RESTRICTED GENERALIZIBILITY and CONFOUNDING CONSTRUCTS."
"I think I should have stayed in bed this morning..." replied Susie. "Am I ever going to get the hang of all this???? "
(If at this point YOU are considering bagging the whole thesis, dial 1-800-IWANT-MOM)
"Of course you are and since I have been in this business awhile, I even have a 'cheat sheet' for you to take home with you (Click here for a copy of Dr.M's Cheat Sheet). In the meantime, sit back and try to catch the basics. The next threat is restricted generalizability across constructs. This refers to unanticipated consequences. For instance, your parenting classes may in fact increase the level of parents involvement with their children , but they may also increase the number of arguments between spouses because one spouse (the one involved in the classes) may accuse the other spouse of not spending enough time with their children. This threat reminds you to be careful about whether the observed effects ("parenting classes are a good thing") could be generalized to other outcomes (marital satisfaction).
Finally, there is the confounding constructs and levels of constructs threat. Imagine that you carried out your project , and the data analysis revealed that the parenting classes really did nothing to increase involvement. You'd be bummed and your first reaction would probably be to chalk it up to another dissertation that proved nothing. However, it may not be that parenting classes are useless, it may be that you didn't conduct ENOUGH parenting classes to see the desired effect. Therefore, it is not appropriate to label parenting classes as a "waste". Get it?"
"I got it, or at least I think I am starting to get it. More importantly, did I hear you say FINALLY and did that mean we are close to the end of this madness for today?" Susie queried.
"Close," laughed Dr. Muffy. "The last three threats we will cover are known as 'social threats'. These threats all represent the joy of doing research with human beings. The first of these is HYPOTHESIS GUESSING."
"Let me guess, " joked Susie. "It is when the people in your study guess what you are looking at and their actions reflect their guess. In my case, that would mean that parents guess I am trying to measure involvement with their children and they purposefully get more involved, not because of my parenting classes, but because of their inference."
"Excellent!" cried Dr.Doolittle, who had been uncharacteristically quiet. "Why don't I finish up with the last two threats." Without waiting for an answer, he proceeded.
"EVALUATION APPREHENSION is next. Many people get really anxious about being in a study -- they are afraid they won't look good or smart, or in this case that they won't appear to be good parents. In their desire to look like the model participant their behavior and actions may not reflect reality. Again, this is a labeling problem because you label "increased participation" as a program effect (when really it is not).
FINALLY, for real this time, one must address the RESEARCHER EXPECTANCIES threat. Without knowing it, you may bias your study. For example, you may become really enthusiastic when you discuss parent involvement. This may send a message to parents that you think involvement is a "good thing" and may prompt them to act accordingly. Again, this means you will label the involvement as a program effect, when in truth it is your overwhelming enthusiasm. SO...what do you think Susie?"
"I think I am overwhelmed."
"Overwhelmed is understandable, but don't panic!" cautioned Dr. Muffy.
"Understanding validity is a long process and there are many sources out there to help you get a grip on things. As a matter of fact, another professor here at Cornell, Bill Trochim, has quite a bit of information on his web site that will be useful to you (click here to go to Bill's Page). In the meantime, I will leave you with my Top 5 Golden Rules of Validity:
1. Validity is something you argue -- not prove. You can't please all the people all the time.
2. Do the best you can with what you have, where you are. In other words, with limited time and resource (constraints of grad school and the real world) you aren't going to eliminate every single solid threat to construct validity. Try your best, and be able to back up what you did.
3. An ounce of prevention is worth a pound of Tylenol at your defense. In other words, think about threats to construct validity (and other kinds of validity) NOW rather than later.
4. Two heads are better than one when brainstorming threats to validity -- talk to others, get help!
5. There is no "end of the road" when learning about validity -- it's the process, not the final amount of knowledge -- that counts.
Send any questions or comments to Nicole M. Driebe. Good luck!