USING RAPID APPRAISAL METHODS

1996, Number 5
http://www.usaid.gov/pubs/usaid_eval/ascii/pnaby209.txt


Performance Monitoring and Evaluation

TIPS

USAID Center for Development Information and Evaluation


USING RAPID APPRAISAL METHODS


What Are Rapid Appraisal Methods?


Rapid appraisal methods are quick, low-cost ways to gather data

systematically in support of managers' information needs, especially

questions about performance.


Rapid appraisal methods fall on a continuum between very informal

methods, such as casual conversations or short site visits, and highly

formal methods, such as censuses, surveys, or experiments.


Informal methods are cheap, "quick and dirty," and susceptible to

bias. They follow no established procedures, but rely on common

sense and experience. They do not generate systematic, verifiable

information, and thus may not be credible with decision-makers.


Conversely, formal methods are highly structured, following precise,

established procedures that limit errors and biases. They generate

quantitative data that are relatively accurate, enabling conclusions to

be made with confidence. Because they have high reliability and

validity, they generally have high credibility with decision-makers.

Weaknesses include their expense and requirements for highly

technical skills.


Between these two lie rapid appraisal methods. They are neither very

informal nor fully formal. They share some of the properties of both

and that is their strength as well as their weakness.


Strengths and Limitations


Strengths of rapid appraisal methods include the following:


They are relatively low-cost. Rapid appraisal studies are usually only

a fraction of the $100,000 to $200,000 often spent for a sample sur-

vey. They typically have a smaller sample size and narrower focus,

and they often require less technical and statistical expertise than

formal methods.


They can be quickly completed. Rapid appraisal methods can gather,

analyze, and report relevant information to decision-makers within

days or weeks. This is not possible with sample surveys. Rapid

appraisal methods are advantageous to decision-makers who seldom

have the option of holding up important decisions to wait for

information.


They are good at providing in-depth understanding of

complex socioeconomic systems or processes. Formal

methods, which focus on quantifiable information, lose

much in "operationalizing" social and economic phenomena.


They provide flexibility. Rapid appraisal methods allow

evaluators to explore relevant new ideas and issues that

may not have been anticipated in planning the study. Such

changes are not possible in sample surveys once the questionnaire is designed and the survey is under way.


Rapid appraisal's limitations:


They have limited reliability and validity. Information

generated may lack reliability and validity because of

informal sampling techniques, individual biases of the

evaluators or interviewers, and difficulties in recording,

coding, and analyzing qualitative data. Those using rapid

appraisal methods can minimize these problems, for example, by taking steps to reduce bias during data collection and analysis, or by using more than one method to

cross-check results (triangulation).


They lack quantitative data from which generalizations

can be made for a whole population. Most rapid appraisal

methods generate qualitative information. Even those that

generate quantitative data (such as minisurveys and direct

observation) cannot be generalized with precision, because they are almost always based on non-representative

samples. While a rapid appraisal method can give a picture of the prevalence of a situation, behavior, or attitude,

it cannot tell the extent or pervasiveness. For example, it

may show that many farmers are not using credit facilities,

but not the percentage of farmers.


Their credibility with decision-makers may be low. Most

decision-makers are more impressed with precise figures

than qualitative descriptive statements. For example, a

sample survey finding that 83 percent of local entrepreneurs were satisfied with technical assistance provided is

likely to carry more weight than the conclusion, based on

key informant interviews, that most entrepreneurs interviewed seemed satisfied with the technical assistance.



When Are Rapid Appraisal Methods

Appropriate?


Choosing between informal, rapid appraisal, and formal

methods of data collection should depend on balancing

several potentially conflicting factors:


purpose of the study ( importance and nature of

the decision hinging on it)

level of confidence in results needed

(accuracy, reliability, validity)

time frame within which it is needed (when decision must be made)

resource constraints (budget, expertise)

nature of information required


Regarding the last factor nature of the information

required rapid appraisal methods are especially useful

and appropriate:


When qualitative, descriptive information is sufficient for

decision-making. When there is no great need for precise

or representative quantitative data, rapid appraisal is a

good choice. When there is a need to understand complex

cultural, social, or economic systems and processes, qualitative information from rapid appraisal methods have an

advantage over formal methods for example, when

assessing organizations and institutions, socioeconomic

conditions of an area (communities, for example), or the

cultural patterns, behaviors, values, and beliefs of a group

or population.


When an understanding is required of the motivations and

attitudes that may affect behavior, for instance of a development activity's customers, partners, or stakeholders.

Rapid appraisal methods are successful in answering the

"why" and "how" questions. For example, key informant

interviews or focus group discussions are more likely than

sample surveys to provide insightful answers to such

questions as, "Why are farmers not adopting the recommended variety of seeds?" or "How are macroeconomic

policies being implemented?"


When available quantitative data must be interpreted.

Routinely generated quantitative data from activity records and performance monitoring data about financial

outlays, input and output volumes, products and services

provided to customers, customer usage, results targets

accomplished or missed may require explanation. Many

of the rapid appraisal methods are useful in interpreting

such data, resolving inconsistencies, and deriving meaningful conclusions. Suppose, for instance, performance

monitoring data show female farmers aren't using a technical package recommended by an agricultural development activity. Interviews with key informants and one or

two focus groups can shed light on this.


When the primary purpose is to generate suggestions and

recommendations. Often an evaluation is used to

solve a problem facing an activity. What is needed are

practical recommendations. For example, the manager of

a contraceptive social marketing activity may be concerned with finding ways to augment sales. The manager's

needs can be served by eliciting suggestions in interviews

or focus groups with doctors, pharmacists, medical workers, traders, and customers.


When the need is to develop questions, hypotheses, and

propositions for more elaborate, comprehensive formal

studies. Key informant and group interviews are widely

used for this purpose.


Common Rapid Appraisal Methods


The most commonly used methods include:


Key informant interviews. Involves interviews with 15 to

35 individuals selected for their knowledge and to refect

diverse views. Interviews are qualitative, in-depth and

semistructured. Interview guides listing topics are used,

but questions are framed during the interviews, using

subtle probing techniques.


Focus groups. Several homogeneous groups of 8 to 12

participants each discuss issues and experiences among

themselves. A moderator introduces the topic, stimulates

and focuses the discussion, and prevents domination of

discussion by a few.


Community interviews. These take place at public meetings open to all community members. Interaction is between the participants and the interviewer, who presides

over the meeting and asks questions following a carefully

prepared interview guide.


Direct observation. Teams of observers record what they

see and hear at a program site, using a detailed observation form. Observation may be of physical surroundings

or of ongoing activities, processes or discussions.


Minisurveys. Involves interviews with 25 to 50 individuals, usually selected using nonprobability sampling techniques. Structured questionnaires are used that focus on a

limited number of closed-ended questions. Generates

quantitative data that can often be collected and analyzed

quickly.



Each of these methods has particular situations in which

they are most appropriate or useful, as well as distinct

advantages and limitations. The matrix on page 4 summarizes this. For information on individual methods, see

additional Tips, or selected further readings below.














Selected Further Reading


Kumar, Krishna, Rapid, Low Cost Data Collection Methods for A.I.D., A.I.D. Program Design and Evaluation

Methodology Report No. 10. 1987 (PN-AAL-100)


Kumar, Krishna (editor), Rapid Appraisal Methods,

World Bank Regional and Sectoral Studies, 1993.


Kumar, Krishna, Conducting Key Informant Interviews in

Developing Countries, A.I.D. Program Design and

Evaluation Methodology Report No.13, 1986 ( PN-AAX-226)


Kumar, Krishna, Conducting Group Interviews in De-

veloping Countries, A.I.D. Program Design and Evalua-

tion Methodology Report No.8, 1987 (PN-AAL-088)


Kumar, Krishna, Conducting Mini Surveys in Developing

Countries, A.I.D. Program Design and Evaluation

Methodology Report No. 15, 1990 (PN-AAX-249)


Rapid Appraisal and Beyond, The Participation Forum

Workshop Notes, 1995.

COMMON RAPID APPRAISAL METHODS






METHODS





Useful for Providing



Advantages



Limitations



KEY

INFORMANT

INTERVIEWS


--general, descriptive data

--understanding of attitudes and

behaviors

--suggestions and

recommendations

--information to interpret

quantitative data

--provides in-depth, inside

information

--flexibility permits exploring

unanticipated topics

--easy to administer

--relatively inexpensive

--takes 4-6 weeks

--does not generate quantitative

data

--susceptible to interviewer and

selection biases




FOCUS

GROUP

INTERVIEWS


--customer views on services,

products, benefits

--information on

implementation problems

--suggestions and recommenda-tions for improving activities

--can be completed rapidly (5

weeks)

--very economical

--group discussion may reduce

inhibitions, allowing free

exchange of ideas


--does not provide quantitative

data

--discussion may be dominated

by a few individuals

--susceptible to moderator

biases




COMMUNITY

INTERVIEWS


--village/community level data

--views on activities and

suggestions for improvements


--permits direct interactions

between evaluator and large

numbers of individuals

--can generate some quantitative

data on community

characteristics, behaviors,

opinions

--participants tend to correct

each other, providing more

accurate information

--inexpensive and quick (5-6

weeks)

--can be manipulated by elites

or monopolized by individuals

--cultural taboos or norms may

inhibit discussion of certain

topics




DIRECT

OBSERVATION


--data on physical infrastructure,

supplies, conditions

--information about an agency's

delivery systems, services

--insights into behaviors or

events

--phenomenon can be examined