SEO Marketing Research

SEO Marketing Research header image 2

Questionnaire design

659 Comments · Measurement instruments

Questionnaire design is key to both qualitative and quantitative research. In the former, even small samples can be investigated using semi-structured (or in other cases, unstructured) questionnaires to elicit answers and to probe interviewees’ responses.

The questionnaire in quantitative research is used as a survey instrument with larger samples, normally containing structured questions for ease of coding and analysis.

A questionnaire is a data collection instrument, formally setting out the way in which research questions should be asked.

Even simple questions need proper wording and organization to produce accurate information.

Consideration needs to be given to how questions should be worded, in the light of the objectives of the research, and the target group of respondents. Attention also needs to be given to the organization of the questionnaire and to testing.

There are limitations to what a questionnaire can measure, especially when it comes to product recalls because people can only accurately report upon the item they have bought for a limited time after the actual purchase.

A questionnaire has specific objectives:
? To translate the information needed into questions that the respondents can and will answer. Developing questions that will yield the desired information is difficult.

Two apparently similar ways of posing a question may result in different information. Hence, this objective is a challenge.

? A questionnaire must motivate the respondent to become involved in the interview, to co-operate, and to complete the interview.

Before designing a questionnaire, the researcher must evaluate “what is the respondent going to get out of this” and appreciate what respondents go through when approached and questioned.

Not all respondents are the same in what they seek from a questionnaire or interview process.

? A questionnaire should minimize response error. Such errors arise when respondents give inaccurate answers or when their answers are mis-recorded or mis-analyzed.

A questionnaire can be a significant source of response error and minimizing this error is an important objective of questionnaire design.

To develop a further understanding of questionnaire design, the process can be presented as a series of steps.

The process outlined below shows that the ten steps are interrelated and the development of a questionnaire involves much iteration and interconnection between stages.

Step 1: specify the information needed
The research process often begins when the international marketing manager, brand manager, or product development specialist has a need for decision-making information that is not available.

Five classes of information, generally useful for marketing decisions have been identified:

? Facts and knowledge: what are the beliefs, perceptions, and depth of knowledge of the survey respondents about, for example, specific products, services, industries, or organizations?

? Opinions: what are the existing attitudes towards products, etc., including an assessment of the strength with which these attitudes are held?

? Motives: what motivates buyers of various kinds of products or services?

? Past behaviour: what are the patterns of consumption over specified time periods? Insight will be given into factors such as brand loyalty.

Observational techniques, such as home audits, will help in verifying reported consumption behaviour.

? Future behaviour: indications of possible behaviour may be gleaned from sensitive questioning.

Included in this approach, of course, would be an evaluation of the nature of buying behaviour, which may be complex, and making a list of research objectives.

The first step in questionnaire design is to specify the information needed. It is helpful to review the components of the problem and the approach, particularly the research question, hypotheses and characteristics that influence the research design.

To further ensure that the information obtained fully addresses all the components of the problem, the researcher should prepare a set of variables, which would influence the decision-making problem.

The project’s research objectives should then be translated into information goals that are specific enough to guide question formulation, thinking ahead to the types of tables and graphs that can be used in the final report.

Clearly define the target respondents. The characteristics of the respondent group have a great influence on questionnaire design.

The wording and style of questions that may be appropriate for finance directors being surveyed about their IT needs may not be appropriate for retired persons being asked about their holidays.

The more diversified the respondent group, the more difficult it is to design a single questionnaire appropriate for all.

Step 2: determine the survey methods
A telephone interview often requires a rich verbal description of a concept to make certain the respondent understands the idea being discussed.

In contrast, in an interview, the researcher can show the respondent a picture or demonstrate the concept.

If you want respondents to react to physical stimuli, face-to-face interviews are best. Telephone surveys are very limited in this regard, though it may be possible to send materials to respondents in advance of a phone interview.

Step 3: determine question type and content
Two main types of question response formats are used in marketing research. Openended questions are those in which respondents can reply in their own words.

In other words, the researcher does not limit the response choices. The advantage of such responses is that they can provide the researcher with a rich array of information.

The respondent is answering from his own frame of reference. Advantages are described in “real world” terminology rather than marketing jargon.

Often this is helpful in designing promotional themes and campaigns. It enables copywriters to use the consumer’s language. This rich array of information can now be captured even in computer-assisted interviews.

Open-ended questions are not without their problems. Editing and interpretation of responses are problematic.

If too many categories are used, data patterns and response frequencies may be difficult for the researcher to interpret.

If the categories are too broad, the data are too general and meaning may be lost. Even if a proper number of categories is used, editors may have to interpret what the interviewer has recorded and force data into a category.

A related problem of open-ended questions is interviewer bias. Although training stresses the importance of verbatim recording of open-ended questions, it is often not practised in the field. Also, slow writers may miss important comments.

Open-ended questions also may be biased towards the articulate interviewee. A person with elaborate opinions and the ability to express them may have much greater input than a shy, inarticulate, or withdrawn respondent. Yet they could be equally likely prospects for a product.

So, a basic problem with open-ended questions lies in interpreting and processing data. In fact, a two-phase judgement must be made.

First, the researcher must decide on the proper set of categories and then each response must be evaluated as to which category it falls into.

In general, open-ended questions are useful in exploratory research and as opening or closing questions.

They should be chosen with care as their disadvantages can outweigh their advantages in a large survey.

Closed questions require the respondent to make a selection from a list of responses. The primary advantage of a closed question is simply the avoidance of many of the problems of open-ended questions.

Interviewer and coder bias are removed because the interviewer is simply checking a box, circling a category, recording a number, or punching a key.

The simplest form of a closed-ended question is the dichotomous choice. An example is:
Do you think that this year’s inflation will be greater or less than last year?

? Greater than last year
? Less than last year

Note that the respondent is limited to two answers. It is easy to administer and usually evokes rapid response.

Often a neutral or no opinion/don’t know is added to dichotomous questions to take care of those situations.

Dichotomous questions are prone to measurement error. Because alternatives are polarized, possible choices between the poles are omitted. Thus, question wording is critical to obtain accurate responses.

In the above question, response may vary depending upon whether greater than or less than is listed first.

These problems can be overcome using a split ballot technique. Onehalf of the questionnaires are worded with greater than listed first and the other half with less than first. This procedure will aid in the reduction of potential bias.

As well as open-ended and closed questions, there are multiple-choice questions. These overcome many of the disadvantages of open-ended questions because interviewer bias is reduced and the questions are administered quickly.

Also, coding and processing of data is much less costly and time-consuming. In self-administered questionnaires, respondent co-operation is improved if the questions are structured.

In multiple-choice questions, the researcher provides a choice of answers and respondents are asked to select one or more of the options given.

Sometimes the respondent is asked to only select one item. For instance, a question might ask the respondent “What is your preferred brand?

Alternatively, it could ask the respondent to indicate, say, three favourite brands. A third way would be to simply let the person make the choices that are regarded as relevant or that apply. Consider the following example:

Please indicate all the brands of soft drinks that you have consumed in the past week.
Please check all that apply.

1. Coca-Cola       –
2. Pepsi-Cola      –
3. Fanta              –
4. Seven Up       –
-
-
-
14. Dr pepper       –
15. Other
(please spsescify  –

Of concern in designing multiple-choice questions is the number of options and the order of potential responses, known as position bias.

The response options should include all possible choices. The general guideline is to list the important options and to include another labelled “Other (please specify),” as shown above. The responses should be mutually exclusive.

Multiple-choice questions are not without disadvantages. Considerable effort is required to design effective multiple-choice questions and qualitative techniques may be required to determine the appropriate response options.

It is difficult to obtain information on items not listed. Even if an “Other (please specify)” category is included, respondents tend to choose from the list.

In addition, showing respondents the list of answers produces biased responses. There is also the potential for order bias.

The choice between open- and closed-response questions is not necessarily an either/or distinction.

Open-response questions can be used with closed-response questions to seek additional information.

Using an open-response question to follow up a closed question is called a probe. Probes can combine some advantages of both open and closed questions.

They can be used for specific pre-chosen questions or to obtain additional information from only a subset of people who respond to previous questions in a certain way.

A common example of the latter is to ask respondents who choose “none of the above” a follow-up question to expand on their answer.

There are two general purposes for the use of probes in a questionnaire. The first is to pinpoint questions that were diffiult for respondents.

Adequate testing of questions reduces the need for probes. The second purpose is to aid researcher interpretation of respondent answers.

Answers to open-response follow-ups can provide valuable guidance in the analysis of closedresponse questions.

Step 4: establish question format
Every question in a questionnaire should contribute to the information needed or serve some specific purpose.

It is useful to ask some neutral questions at the start to establish involvement and rapport, particularly when the topic of the questionnaire is sensitive or controversial.

Sometimes filter questions are asked to disguise the purpose or sponsorship of the project.

For example, rather than limiting the questions to the brand of interest, questions about competing brands may be included. At times, certain questions may be duplicated to assess reliability or validity.

Once a question is deemed necessary, the researcher must make sure that it is sufficient to get the desired information.

Sometimes several questions are needed to obtain the required information in an unambiguous manner.

A double-barrelled question is really two different questions posed in one question. With two questions posed together, it is difficult for a respondent to answer either directly.

Consider a question asked of patrons at a restaurant “Were you satisfied with the food and service?” How does the respondent answer?

If they say “yes” does that mean they were satisfied with the food? The service? A combination?

The question would be much improved by asking about a single item: one question for food and another question for service.

Step 5: choose question wording
The wording of specific questions always take a significant amount of time. It is a skill developed over time and subject to constant improvement.

Questions are the raw material of questionnaires and vital to the quality of the research. Question wording is the translation of the desired question content and structure into words that respondents can clearly and easily understand.

Deciding on the wording is perhaps the most critical task in developing a questionnaire. If a question is worded poorly, respondents may refuse to answer it or answer incorrectly.

Even small changes in wording can shift respondent answers, but it is difficult to know whether or not a wording change will have such an effect.

Question phrasing depends on such factors as the information being sought, the characteristics of target respondents, and where the survey is administered. Good questionnaire writing requires that researchers follow these guidelines:

? The questions should be easy to understand. Ordinary words should be used in a questionnaire, and they should match the vocabulary and intellectual level of the respondents.

The most common pitfall is to use technical jargon or specialized terms. Special care must be taken to avoid words that have different meanings for different groups.

This can be readily appreciated in cross-cultural studies, where translation problems are profound.

One socio-economic group may refer to the evening meal as dinner, while others call this meal supper and have their dinner at noon.

Most respondents do not understand technical marketing words. Never forget that you are imposing your language upon respondents in the form of a questionnaire.

Your language communicates and puts respondents in a particular frame of mind as they answer the questions you pose.

Unless that language is meaningful to respondents, they will be put in a frame of mind that you may not intend, and be answering different questions from those you set.

? Questions should be focused on a single issue or topic. The researcher must stay focused on the specific issue or topic.

The question “What type of hotel do you usually stay in when on a trip?” is too vague.

A more focused version is “When you are on a family holiday and stay in a hotel at your destination, what type of hotel do you typically choose?”

? The question should be a grammatically simple sentence. A simple sentence is preferred over compound and complex sentences.

The more complex the sentence, the greater the potential for respondent error. To avoid these problems, the researcher should strive to use a simple sentence structure, even if two separate sentences are necessary for the question.

What is an appropriate length of a question? A common rule of thumb is to limit the number of words in any question to under 20.

Under certain circumstances, a question may have to be long in order to avoid ambiguity, but this should be the exception.

A questionnaire filled with long questions is tiring to answer and more difficult to understand.

Brevity will help respondents to comprehend the central question and reduce the distraction of wordiness.

? Avoid leading questions. A leading question is one that suggests the answer or reveals the researcher’s (or interviewer’s) opinion.

This can be done easily by adding “don’t you agree?” or “wouldn’t you say?” to a desired statement.

A loaded question introduces a more subtle bias. A common type of loading of possible responses is through failure to provide a full range of options, for example, by asking, “How do you generally spend your free time-watching television, or what?” Simply adding “I’m sure you agree” or “Don’t you think?” to a statement can bias responses.

Researchers have also found that respondents tend to agree with plausible propositions unless they have a strong opinion or choices are provided.

Even when options are offered, respondents tend to agree rather than disagree with plausible statements. Given this tendency, questions should be worded neutrally.

? Consider the ability of the respondent to answer the question. Asking respondents about a brand or store that they have never encountered creates a problem.

When a question is worded in such a manner that it implies that the respondent should be able to answer it, then often a reply will be forthcoming, but it will be nothing more than a guess.

This creates measurement error, since uninformed opinions are being recorded. A second problem is forgetfulness.

For instance, “What was the name of the last film you saw on TV?” “Who were the stars?” To avoid the problem of a respondent’s inability to remember, time periods should be kept relatively short.

? Consider the willingness of the respondent to answer the question. Reporting of an event is likely to be distorted in a socially desirable direction.

If the event is perceived as embarrassing, sensitive in nature, threatening, or divergent from one’s self-image, it is likely either not to be reported or to be distorted.

Embarrassing topics that deal with things such as borrowing money, personal hygiene, sexual activities, and criminal records must be phrased in a careful manner to minimize measurement error (Wrobel, 2002).

One technique is to ask the question in the third person. For example, “Do you think that most people spend more using their credit cards than they should?”

By asking about “most people” rather than about the respondents themselves, researchers may be able to learn more about the respondents’ attitude to credit and debt.

A third method for soliciting embarrassing information is to state that the behaviour or attitude is not unusual before asking the question.

For instance, “Millions of people suffer from hemorrhoids; do you or any member of your family suffer from this problem?”

This technique is called “using counter-biasing statements,” and makes embarrassing topics less intimidating for respondents to discuss.

Step 6: arrange the sequence and layout of questions
After questions have been formulated, the next step is to order them and develop a layout for the questionnaire.

Questions should be asked in a logical order. All questions that deal with a particular topic should be asked before beginning another topic.

When switching topics, brief transitional phrases should be used to help respondents switch their train of thought.

“Branching” or “skipping,” a procedure in which certain questions are not asked if they do not apply to a respondent, should be designed carefully.

Branching questions direct respondents to different places in the questionnaire based on how they respond to the question at hand.

These questions ensure that all possible contingencies are covered. A simple way to account for all contingencies is to prepare a flowchart of the logical possibilities and then develop branching questions and instructions based on it.

Branching is most easily done in computer-assisted telephone interviews or online surveys, where software can perform the branching. The logical order of a questionnaire could be:

? Use screener questions to identify qualified respondents. Most market research employs some variation of quota sampling.

Only qualified respondents are interviewed, and specific minimum numbers (quotas) of various types of qualified respondents may be desired.

A study on food products generally has quotas of users of specific brands, a magazine study screens for readers, a cosmetic study screens for brand awareness and so forth.

Thus, any demographics obtained provide a basis for comparison against persons who qualify for the study.

A long screener can significantly increase the cost of the study. It means that you are obtaining more information from every contact with a respondent.

Short screeners quickly eliminate unqualified persons and enable the interiewer to move to the next potential respondent.

Yet a long screener can provide important information on the nature of non-users, or persons unaware of the product or service being researched.

? After introductory comments and screens to find a qualified respondent, the initial questions should be simple, interesting, and non-threatening.

Income or age questions might be disastrous. These are often considered threatening. The initial question should be easy to answer without much forethought.

? Ask general questions first. Once the interview proceeds beyond the opening “warmup” questions, the questionnaire should proceed in a logical fashion.

General questions are covered first to get the person thinking about a concept, company, or type of product, and then the questionnaire moves to the specifics.

? Ask questions that require effort in the middle of the questionnaire. Initially, the respondent is only vaguely interested and understanding of the nature of the survey, until the process builds momentum and commitment to the interview.

When the interviewer shifts to questions with scaled-response formats, the respondent must be motivated to understand the response categories and options.

Build interest and commitment early to motivate the respondent to finish the questionnaire.

? Put sensitive, threatening, and demographic questions at the end. Sensitive topics include money, personal hygiene, family life, political and religious beliefs, and involvement in accidents or crimes.

In industrial surveys, sensitive questions may encompass much of what a company does, especially if it reveals strategy and plans.

Placing these questions at the end ensures that most of the questions will be answered before respondents become defensive or break off the interview.

Moreover, rapport has been established between the respondent and the interviewer by this time, increasing the likelihood of an answer.

Conclude the survey by thanking respondents for their time. You may also want to inform respondents how they can obtain the results of the survey.

The format, spacing and positioning of questions can have a significant effect on the results, particularly in self-administered questionnaires.

It is good practice to divide the form into parts. Several parts may be needed for questions pertaining to the basic information.

Each questionnaire should be numbered. This helps the control questionnaires in the field as well in coding and analysis.

Numbering makes it easy to account for the questionnaires and to determine whether any have been lost.

Finally, don’t make the questionnaire too long. Long questionnaires are tiring and overwhelm respondents.

As respondents exceed the time they have mentally set aside to complete the survey, their responses are no longer accurate. Furthermore, long questionnaires tend to have high non-response rates.

Step 7: obtain approval from relevant parties
Copies of the draft questionnaire should be distributed to all parties that have direct authority over the project.

The client is given the opportunity to comment during the client approval stage, in which the client reviews the questionnaire and assesses whether it covers all of the appropriate issues.

If questions are either inappropriate or perhaps can be improved, it is necessary for the client to convey these changes to the researcher.

This may cause changes, but it is important for the client to approve the questionnaire that will be used.

Client approval ensures that the client is aware of the survey’s progress, and the initialled questionnaire ensures that the researcher is protected against any later claims that the questions were incomplete or done incorrectly.

Step 8: test, revise and correct problems
Once approval has been obtained, the questionnaire should be tested to identify and eliminate problems.

The most basic test is to have as many people as possible look at drafts of the questionnaire as a sounding board. The worst problems will be uncovered by these reviews.

Ideally, a pilot test is done by the best interviewers who will ultimately be working on the job and is administered to target respondents for the study.

They are told to look for misinterpretations by respondents, lack of continuity, poor skip patterns, additional choices for pre-coded and closed questions, and general respondent reaction.

Testing could also involve a trial run of the questionnaire using a small sample of respondents, say five to ten, from the target population.

While the sample may be small, it should cover all subgroups of the target respondents. The goal of testing is to check that the questionnaire will capture the information sought by the researcher.

Testing helps refine the instrument and identifies errors that may be apparent only to the target.

Testing is usually done in two stages. The first stage is personal interviews, regardless of the way the questionnaire will later be administered, because researchers need to observe the behaviours of both the interviewers and the respondents.

Respondents’ reactions to the questions are the primary interest. The interviews can be carried out through protocol analysis or debriefing.

Protocol analysis is an interviewing technique in which respondents think aloud while responding to each question.

Debriefing is an interview conducted after respondents have completed the questionnaire. The respondents are then informed that the exercise was a test and are asked to share with the researcher their thoughts about the question, their answers, and any shortcomings of the survey.

The second testing stage involves administering the survey to a small sample in an environment as similar as possible to the one in which the questionnaire will ultimately be administered.

This stage often reveals problems that cannot be detected in personal interviews. In either phase, researchers should attempt to eliminate any problems and revise the questionnaire.

After the questions have been checked, a numerical code is allocated to each type of response to aid data processing.

All possible answers may be listed and coded in advance of the interview and, in surveys of any size, this is done wherever possible.

If responses cannot be allocated to a range of possible answers, coding can take place after the interview. Especially, pre-coding is relevant in quantitative surveys with closed questions.

Step 9: prepare final copy
Even the final copy phase does not allow the researcher to relax. Precise typing instructions, spacing, numbering, and pre-coding must be set up, monitored, and proof-read.

In general, the quality of copying and the paper used is a function of who will see the questionnaire.

Step 10: implementation
Most research interviewing is conducted by a field services department. It is their duty to complete the interviews and send them back to the researcher. In essence, field services are the production line of marketing research.

Supervisor’s instructions inform interviewers of the nature of the study, start and completion dates, quotas, reporting times, equipment and facility requirement, sampling instructions, number of interviewers required, and validation procedures. Detailed instructions are required for any taste test that involves food preparation.

The supervisor’s instructions are vitally important. Without clear instructions, the interview may be conducted in ten different ways in ten different countries.

Call record sheets are used to measure the efficiency of the interviewers. A form normally indicates the number of contacts and the results of the contact.

A supervisor can examine calls per hour, contacts per completed interview, average time per interview, and similar measures to analyze an interviewer’s efficiency.

If, for example, contacts per completed interview are high, the field supervisor should examine the reasons behind it. Perhaps the interviewer is not using a proper approach or the area may be difficult to cover.
What is the purpose of a questionnaire?

How would you determine whether a specific question should be included in a questionnaire?

What are the issues involved in designing multiple-choice questions?

What are the guidelines available for deciding on question wording?

What is a leading question? Give an example.

Once a questionnaire is developed, what other factors need to be considered before giving it to interviewers?

Why is testing a questionnaire important?

Keywords: Questionnaire, semi-structured Questionnaire, unstructured Questionnaire, Openended questions, Closed questions, split ballot technique,

Tags:

659 Comments so far ↓

Leave a Comment