For each question in your survey, you should ask yourself how well it addresses the content you are trying to get at. Here are some content-related questions you can ask about your survey questions.
Is the Question Necessary/Useful?
Examine each question to see if you need to ask it at all and if you need to ask it at the level of detail you currently have.
- Do you need the age of each child or just the number of children under 16?
- Do you need to ask income or can you estimate?
Are Several Questions Needed?
This is the classic problem of the double-barreled question. You should think about splitting each of the following questions into two separate ones. You can often spot these kinds of problems by looking for the conjunction "and" in your question.
- What are your feelings towards African-Americans and Hispanic-Americans?
- What do you think of proposed changes in benefits and hours?
Another reason you might need more than one question is that the question you ask does not cover all possibilities. For instance, if you ask about earnings, the respondent might not mention all income (e.g., dividends, gifts). Or, if you ask the respondents if they're in favor of public TV, they might not understand that you're asking generally. They may not be in favor of public TV for themselves (they never watch it), but might favor it very much for their children (who watch Sesame Street regularly). You might be better off asking two questions, one for their own viewing and one for other members of their household.
Sometimes you need to ask additional questions because your question does not give you enough context to interpret the answer. For instance, if you ask about attitudes towards Catholics, can you interpret this without finding out about their attitudes towards religion in general, or other religious groups?
At times, you need to ask additional questions because your question does not determine the intensity of the respondent's attitude or belief. For example, if they say they support public TV, you probably should also ask them whether they ever watch it or if they would be willing to have their tax dollars spent on it. It's one thing for a respondent to tell you they support something. But the intensity of that response is greater if they are willing to back their sentiment of support with their behavior.
Do Respondents Have the Needed Information?
Look at each question in your survey to see whether the respondent is likely to have the necessary information to be able to answer the question. For example, let's say you want to ask the question:
Do you think Dean Rusk acted correctly in the Bay of Pigs crisis?
The respondent won't be able to answer this question if they have no idea who Dean Rusk was or what the Bay of Pigs crisis was. In surveys of television viewing, you cannot expect that the respondent can answer questions about shows they have never watched. You should ask a filter question first (e.g., Have you ever watched the show ER?) before asking them their opinions about it.
Does the Question Need to be More Specific?
Sometimes we ask our questions too generally and the information we obtain is more difficult to interpret. For example, let's say you want to find out respondent's opinions about a specific book. You could ask them
How well did you like the book?
on some scale ranging from "Not At All" to "Extremely Well." But what would their response mean? What does it mean to say you liked a book very well? Instead, you might as questions designed to be more specific like:
Did you recommend the book to others?
Did you look for other books by that author?
Is Question Sufficiently General?
You can err in the other direction as well by being too specific. For instance, if you ask someone to list the televisions program they liked best in the past week, you could get a very different answer than if you asked them which show they've enjoyed most over the past year. Perhaps a show they don't usually like had a great episode in the past week, or their show was preempted by another program.
Is Question Biased or Loaded?
One danger in question-writing is that your own biases and blind-spots may affect the wording (see Decisions About Question Wording). For instance, you might generally be in favor of tax cuts. If you ask a question like:
What do you see as the benefits of a tax cut?
you're only asking about one side of the issue. You might get a very different picture of the respondents' positions if you also asked about the disadvantages of tax cuts. The same thing could occur if you are in favor of public welfare and you ask:
What do you see as the disadvantages of eliminating welfare?
without also asking about the potential benefits.
Will Respondent Answer Truthfully?
For each question on your survey, ask yourself whether the respondent will have any difficulty answering the question truthfully. If there is some reason why they may not, consider rewording the question. For instance, some people are sensitive about answering questions about their exact age or income. In this case, you might give them response brackets to choose from (e.g., between 30 and 40 years old, between $50,000 and $100,000 annual income). Sometimes even bracketed responses won't be enough. Some people do not like to share how much money they give to charitable causes (they may be afraid of being solicited even more). No matter how you word the question, they would not be likely to tell you their contribution rate. But sometimes you can do this by posing the question in terms of a hypothetical projective respondent (a little bit like a projective test). In this case, you might get reasonable estimates if you ask the respondent how much money "people you know" typically give in a year to charitable causes. Finally, you can sometimes dispense with asking a question at all if you can obtain the answer unobtrusively (see Unobtrusive Measures). If you are interested in finding out what magazines the respondent reads, you might instead tell them you are collecting magazines for a recycling drive and ask if they have any old ones to donate (of course, you have to consider the ethical implications of such deception!).
Copyright ©2006, William M.K. Trochim, All Rights Reserved
Purchase a printed copy of the Research Methods Knowledge Base
Last Revised: 10/20/2006