How to Evaluate a Research Proposal

How to Evaluate a Research Proposal

Whenever you use research findings to make decisions, you need to evaluate the process used to generate the findings. If the research process is flawed then you need to be cautious about using the research findings. The SRC works with the state agency to conduct research and you have an obligation to determine if the research process will produce valid and reliable findings. The Statewide Consumer Needs Assessment and Customer Satisfaction Survey are just two examples of research projects that you and agency management undertake together. The findings from The Statewide Consumer Needs Assessment and Customer Satisfaction Survey are two of the most important inputs to decisions made by the SRC and agency management. It is imperative that any research studies you rely on to draw conclusions about consumers needs and the agency performance follow widely accepted guidelines for consumer market research.

The purpose of this SRC Tool is to:

1. Provide you with an overview of the market research process.
2. Familiarize you with some of the technical terms researchers use so that you can have more meaningful dialogue with them.
3. Help you make decisions about research studies you shape.
4. Help you evaluate the quality of research findings from studies conducted by others.

Market research is a profession. There is a considerable body of scientific knowledge that influences researchers decisions about every aspect of research design. Generally accepted practices guide researchers in making decisions about what data to collect, who to collect data from, and how and when to collect the data. These professional norms also determine if a census or a sample is appropriate. If sampling is used, the researcher needs to decide how to sample the population. Should a convenience or a random sample be drawn? Does a proportional sample best fit the objectives of the research or is there a reason to over sample some subgroups? A variety of analysis methods exist and the researcher must determine which methods are most appropriate for each study.

The research process determines the quality of the findings from a study. If you want to enhance your credibility and influence then you need to be disciplined about insisting on the use of research professionals to design your consumer research studies, to supervise the research and, to interpret the research findings. Remember that the evidence based decision making process is grounded in the appropriate collection, analysis and application of information.

When others don't agree with your thinking or don't like the findings from a research study, they usually attack the process you used in your research study. The research process needs to be absolutely defensible in that it is based on widely accepted research principles and guidelines for consumer research studies. Anything less will leave you open to criticism that will be difficult to neutralize.

Why Do Research?

Consumer research is conducted to support decision making. The Statewide Consumer Needs Assessment and Consumer Satisfaction Survey help the SRC and agency management make decisions about policies and practices. The research provides information useful in making decisions about what services to provide, how to make services available, what the service delivery system should look like, how to communicate the agency's services to consumers and referral sources, what to focus on to improve consumer satisfaction and other similar decisions.

Questions You Need To Answer

Six questions need to be answered before a decision to conduct a research study can be made. The SRC should work with the state agency personnel responsible for conducting prior studies on the subject or your research. You may also engage an independent research professional to help you answer these questions. Often, this is an iterative process, involving the client [You] and the Researcher. The client, which would be you and the VR agency, would explain to the Researcher what decisions you would like to make. The Researcher is responsible for translating your objectives into research terms. The questions that must be answered are:

1. What decision [or decisions] is this research study supposed to support?
2. What information is needed to make a good decision?
3. What do we already know?
4. What information do we need but don't have?
5. Can we get this information by conducting this research study?
6. Does the value of the information justify the cost of the research?

What is The Research Plan?

If you decide to conduct a research study then a research plan is developed that lays out in substantial detail how the data will be collected and analyzed. The research professional you are working with is responsible for developing the research plan. The plan describes all of the following elements:
Background [What Are the Circumstances That Led To the Study?]

• Management Decision[s] The Research Study Will Support
  
• Research Objectives
  
• Research Design
  
• Report And Presentation Plan And Format

Do the stated objectives refer to the management decision[s] the study is supposed to support? Do the objectives spell out what information will be gleaned from the research findings and how this information will help management make these decisions?

The researcher is responsible for translating management decisions into researchable issues, information needs and data requirements. This usually occurs through a series of discussions between the client [the SRC and the state agency] and the researcher. The researcher should be asked to prepare a written summary of the discussion that describes his understanding of your objectives and the specific management decisions that the findings from the research study will support. This written summary is reviewed by members of the SRC and agency management.

If you determine that the researcher does not fully understand your objectives then it is incumbent upon you to arrange for another discussion with the researcher to clear up any misunderstanding. The researcher should again be asked to prepare a written summary of the discussion that describes his understanding of your objectives and the specific management decisions that the findings from the research study will support. This written summary is reviewed by members of the SRC and agency management. This process continues until you are satisfied that the researcher fully understands your objectives and knows what information you expect the research findings to provide.

One important benefit of this process is that it helps members of the SRC and agency management clarify their own thinking and ensures a more precise set of research objectives. Quite often you will find that engaging in this iterative process with the researcher helps you deepen understanding of the research objectives among board members and management. It strengthens support for the research study and ultimately produces more useful information.

Research Target Audience

Who will we get this information from? It is important to understand how the target audience will be identified and reached. Will the research study target consumers who are or were served by the state agency? If so, the agency has a record for each consumer served. The record should include basic contact information for each consumer such as their mailing address, phone number and email address if they have one.

Most agencies verify the consumers contact information when the case is closed. Even so, the contact information for many consumers is incorrect. In part, this results from the lack of a permanent residence for some consumers. Another cause of dirty data or incorrect consumer contact information includes an inconsistent effort to gather this information at case closure. Data entry errors can also contribute to the problem of incorrect contact information. All databases contain some incorrect contact information. This only becomes a source of concern if it causes one of two problems.

The first problem is that time and money are wasted trying to reach consumers who can't be reached because the contact information is incorrect or missing. Of course any effort to clean up the database should be weighed against the cost of trying to reach consumers who are unreachable due to incorrect contact information. If the unreachable are a microcosm of the overall consumer population you are better off accepting this as a normal condition of any database.

The second problem is that a dirty data can skew the results if contact information is more frequently missing for a specific segment of consumers. For example, lets assume that African Americans represent 30 percent of total clients served but represent 65 percent of the consumers with incorrect contact information. In this case, African Americans will be under represented in the study. This happens because we can't reach almost two thirds of these clients. If were using a random sample, we may find that only about ten percent of the sample is African American. This is unacceptable and should be addressed.

The SRC can help improve the quality of consumer research by determining if the state agency has a problem with incorrect or missing contact information. Begin by asking about response rates in previous studies. Ask the agency to determine what percentage of non respondents could not be contacted and why. If the survey was mailed, the agency receives information on all undeliverable surveys. If the survey was conducted by telephone, the agency should have a record of how many consumers targeted for a study could not be reached due to an incorrect or missing telephone number. You should also ask agency management to review response rates for important segments of the consumer population to see if any group is underrepresented.

Research Design

The next step in the research process is to choose a research method best suited to meet the objectives of the research study. The researcher can select from a myriad of research design alternatives such as focus groups, individual depth interviews, mail, telephone, in-person or web based surveys.

In general, research studies are classified as either qualitative or quantitative. Qualitative research usually takes the form of focus groups. Focus group research helps illuminate how the target audience thinks about and makes decisions about the subject of interest. In the context of vocational rehabilitation, a focus group research study may help you understand how consumers think about their employment related needs and make decisions about them, including decisions to seek services from the state VR agency.

Focus group studies are often characterized as exploratory. These studies are useful in understanding why people do what they do. Focus groups give us insight into the motivations that drive consumers actions. Focus group studies are usually small, involving as few as 20 or 30 people. Participants are not chosen through a random sampling process where every member of the target population has an equal chance of being selected for the study. The research findings take the form of a narrative summary of how participants think and behave; they cannot be subjected to statistical analysis. Technically, the findings cannot be generalized to the total population of interest but often are in spite of the limitations of the research design.

Quantitative research studies are more useful in helping us understand what consumers think or do with regard to the subject of interest. Quantitative studies often take the form of surveys. Surveys are almost always designed to be representative of the population of interest. Some form of random sampling is employed and the data is subjected to statistical analysis. This sounds daunting but it isn't if you are a trained researcher. The chief benefit of survey research is that the findings can be generalized to the population of interest.

Often focus groups are conducted to gain insight into how the target audience thinks and makes decisions about the subject of interest as a prelude to a larger survey. The focus group research is exploratory and helps shape the survey objectives, sampling plan, questionnaire and other important aspects of the study. Survey research is often referred to as confirmatory. This does not mean that the survey research does confirm the findings from the focus groups. It means that the survey is capable of validating the focus group findings. The survey may confirm or disconfirm the findings from the focus groups.

Research Sampling Plan

Census or Sample When a research study is designed to include all members of the target population it is called a census. It is rarely advisable to conduct a census because sampling methods enable you to lower the costs of the research while maintaining a high degree of validity. This means that the research findings represent the population of interest and can be generalized to the entire population. In addition to being less costly, sampling is often faster.

When determining the sampling plan, a key consideration is how the data will be used. You should carefully consider the costs of making an incorrect management decision. In addition to monetary costs, there could be political or public relations fallout. When the costs of making a mistake are high, a higher degree of confidence is called for and a larger sample is required. Even so, it would be very difficult and often impossible to justify the cost of taking a census.

Determining the correct sampling plan can require extensive work. Laypersons may be left to engage in guesswork to make the correct choice. This guesswork, if incorrect, will yield survey results that inaccurately reflect the sentiments of the population, and therefore compromise any conclusions drawn based upon the results.

Many factors should be considered when determining a suitable sampling plan, such as the total population size, appropriate confidence level, anticipated participation level, and other technical issues.

A key consideration is whether survey results need to be interpreted at the sub-group level. Examples of sub-groups are cultural or racial minorities such as African Americans or Hispanics. Other examples could include consumers with a specific disability or combination of disabilities or other characteristics such as whether they have internet access or work history. Unless proper sampling strategies are implemented, sub-group results may not be representative of the views held by members of sub-groups. If subgroup results are not representative, developing and implementing effective strategies will be challenging at best.

Judgment Samples Focus group studies do not require use of a random sample because the objective is not to generalize the findings to the entire population. Rather, the goal of a focus group study is to help us deepen our understanding of why consumers behave as they do or think as they do about their rehabilitation and employment needs. The findings will help shape a larger survey. Participants are usually chosen from a judgment or convenience sample.

The SRC is well advised to consult an experienced survey researcher or statistician for advice on sampling issues. These are not decisions that members of the SRC are qualified to make, unless, of course, they are professional survey researchers.

Research Questionnaire Design

Questionnaire design is quite a science and should also be undertaken by a research professional. Experienced researchers understand the characteristics of marketing data, levels of measurement and consequences of improper manipulation of research data.

An experienced researcher will help you avoid common errors in questionnaire construction. The researcher will make decisions about what to ask and how, the flow of the questions, question structure and wording. The researcher will also make decisions about the use of open vs. close-ended questions. In general, you want to use open ended questions when doing qualitative research but closed ended questions on a survey.

Open ended questions allow participants to share their thinking and practices. In exploratory studies you want to encourage the participants to share in-depth. In contrast, surveys are structured to facilitate statistical analysis. Closed-ended questions yield more useful information and are subject to a wide array of statistical analyses. Open-ended questions yield limited information on a survey and are time consuming and costly to analyze. One open-ended question is fine but more than that is a poor use of resources.

It is best to let the researcher make decisions about alternative procedures for obtaining data from respondents; choosing multiple choice, ranking procedures, rating scales, constant sum and other procedures that will affect the types of analyses the data can be subjected to later.

You should insist on pre-testing the questionnaire to discover any problems with flow, question structure, wording and the ability of participants to understand what you are asking. Usually this can be accomplished with a small sample and is well worth the time and cost, both of which should be relatively minor.

Research Data Collection Method

Data collection methods differ for qualitative and quantitative research studies. Qualitative research relies on focus groups or in-depth personal interviews. Both have similar goals and use a similar method. A focus group is a group of 8 to 12 participants who share similar characteristics of interest. They may share similar needs or experience paths. One example of an experience path might include clients whose rehabilitation plan included education or training. A minimum of three focus groups should be conducted with each sub-group of interest. A trained but unbiased moderator should conduct and analyze the focus groups.

Quantitative research most commonly takes the form of a survey and the data can be collected using all of the following methods:

• Face-to-face
  
• Telephone
  
• Internet or email
• U. S. mail

Multiple data collection methods are needed when surveying people with disabilities and should be carefully considered when planning the research study. It is unlikely that you will use face-to-face as a primary data collection method in survey research due to the expense. This makes it imperative that you provide an 800 HELP telephone number that consumers can call for assistance in completing the questionnaire.

Biased Research

Bias can be introduced into a research study quite innocently, usually because lay people make decisions about research design without consulting a trained researcher. All of your great work can be negated if the research results are biased in some preventable way. The most common source of preventable bias is the use of state agency personnel to conduct the research and analyze the findings. One could argue that the use of state agency personnel to complete certain steps in the research process introduces more egregious bias. The steps most often mentioned as being particularly vulnerable to bias are data collection and data analysis. If the agency feels that it cannot avoid using staff to complete these steps in the research process, it is absolutely essential that direct service staff, especially counselors not be selected for this purpose. Staff asked to complete any part of a research study should be research professionals with the possible exception of staff used to answer the 800 HELP telephone line. These staff, however, must be trained to take the phone calls and may need to pass these calls to more skilled personnel.

The best approach avoids even the appearance of bias by limiting the use of state agency staff to design and conduct research studies. You need to take this issue seriously if you want to use the evidence based decision approach. You need to produce credible evidence that you can use to build a case of a defensible case to explain positions you take and decisions you make.
Research Report and Presentations

The research proposal should also describe how the report will be organized and distributed. If the research findings will be formally presented to you and or other audiences, the research proposal should describe the general presentation plan.
A Regional Approach

Lower Costs and Greater Insights

The SRC should evaluate the possibility of pursuing some research projects on a regional basis. The potential benefits include lower costs and greater insight into the subject of your research. Region I is already using a regional approach to conduct consumer satisfaction research. Region IV used a regional approach to conduct research with employers. Regional studies enable you to compare and contrast the findings within your state to other states and to the region. Regional studies often provide more insight into sub-groups of interest because the sample size is larger and supports more detailed and sophisticated analysis.

A Unified Voice

Using a regional or even a national approach to consumer research produces a more unified voice. A unified voice makes the entire vocational rehabilitation community more effective as an advocate for people with disabilities.

It also occurs in the field of research, possibly even more prevalently. This is where the client expects the vendor to create a research design to be delivered in a proposal for research delivery. It’s in that tangled context that I want to recommend a great (and somewhat hard to find) little resource....