TOC | Previous | Next

Section Six
Evaluating a Coalition

"To be sure of hitting the target, shoot first and call whatever you hit the target."
Anonymous

Overview of Section

This section discusses the various types of evaluation, discusses models for evaluation, reviews literature on evaluation, and generally points to the long-term value of evaluation for coalitions.

B. Strategy of Evaluation

Goals of Evaluation

The goals of health-related coalitions are typically driven by long-term considerations. Asthma coalitions appropriately spell out their goals in terms of helping reduce morbidity and mortality of those afflicted with asthma and generally design programs to increase public awareness and promote prompt and proper diagnosis and treatment of the disease. These goals provide sufficient specificity that useful evaluation programs can be prepared and implemented.

However, the focus of the evaluation cannot be exclusively on achievement of long-term goals. Beginning the coalition is a difficult process in which not everything runs smoothly. While undertaking a goal-oriented evaluation is necessary, it need not be the only evaluation. Other aspects of evaluation provide useful information and should not be neglected.²⁵

It has been suggested that three perspectives of evaluation should be considered in framing the evaluation of health-related programs.²⁶

1. Evaluation for accountability (Do the programs achieve results?)
2. Evaluation for development (How can we strengthen the prevention and treatment programs and the capacity of program personnel to evaluate and improve their own programs?)
3. Evaluation for knowledge (What types of programs work well with what types of problem areas in what types of populations?)

There are multiple ways for a coalition to evaluate its programs and the subsequent impact on the community served by the coalition. Some of the types frequently described in the literature include:

1. Process Evaluation
2. Implementation Evaluation
3. Outcome Evaluation
4. Context Evaluation

The process of evaluation is a highly sophisticated one involving a number of options that are quite detailed. The following summary offers the briefest overview of these elements.

Process Evaluation

A process evaluation documents and analyzes the early development and implementation of a strategy or programs, assessing whether strategies were implemented as planned and whether expected outcomes were achieved. A thorough process evaluation will contain the following elements:

• description of the program environment and supplying data;
• description of the process used to design and implement the program;
• description of program operations, including any changes in the program;
• identification and description of intervening events that may have affected implementation and outcomes;
• documentation, such as meeting minutes, reports, memorandums, newsletters, and forms.

Process evaluation should begin during the program-planning phase and continue through program implementation. Process evaluation adds a qualitative dimension to the descriptive statistics and can be used to both describe and assess program materials and activities.

Implementation Evaluation

Implementation evaluation focuses on the core activities of the program or organization and asks whether these activities are individually and collectively effective in moving the program forward. No matter how much time and effort go into the planning phase, reality will usually necessitate changes. All who have ever done a project have done implementation evaluation when they engaged in activities such as the following:

• reviewing whether the available resources will be sufficient to carry out the project;
• measuring the performance of the staff against expected criteria;
• examining the interaction of staff and members with the various client groups;
• reviewing preliminary results against expected outcomes;
• reviewing the efficiency and effectiveness of the work of the steering committee and any of the work groups;
• documenting the satisfaction of client participants with the intervention strategy.

Implementation evaluation is designed to assist new or rapidly changing projects and focuses immediately on how the project is evolving and continually assessing whether changes need to be made to the plan. It allows documentation of these changes and the causes for them, e.g. failure of staff to carry out assignments, lack of interest on the part of some clients, overly ambitious timelines, etc.

Outcome Evaluation

Outcome evaluation assesses the short- and long-term results of a project and measures the changes brought about by the project. This type of evaluation will obviously measure the progress being made toward the achievement of the outcomes specified in the action plan. In addition, an outcome evaluation will focus on all changes brought about by the project, not just those that were intended. Other outcomes, positive and negative, unintended but real, are usually the result of any project. The later stages of outcome evaluation will provide documentation needed for funding agencies as well as demonstrate those parts of the project that were successful and those that did not meet expectations.

The questions involved in the outcome evaluation are essentially the same questions as coalitions pose to themselves at the beginning of the program.

• Who is the coalition going to serve?
• What outcomes is it attempting to reach for this target population?
• How will the coalition measure whether it has achieved these outcomes?
• What data will it collect and how will it be collected?
• How will the coalition use the results?
• What are the performance targets?

Context Evaluation

Context evaluation examines how a program or organization operates within the context of the community it seeks to serve. This type of evaluation continually reviews such items as:

• the appropriateness of the target group chosen by the coalition for intervention;
• the support being given by those portions of the community that were requested to give support, for example, a school or school district;
• the likelihood that the seed funding the program has received can be built upon with additional funding;
• the attitude of the health community toward the program initiatives.

Context evaluation should be continually done throughout every phase of a coalition's work. It assumes that the original plan is capable of constant improvement based upon comments from the community in which the coalition is operating.

Regardless of the kind of evaluation, all evaluations use data collected in a systematic manner. The data may be quantitative such as counts of emergency department admissions; number of asthma education programs provided prior to patient discharge; number of hospitals and physicians adopting asthma care practice guidelines or critical pathways; number of Open Airways programs initiated in schools; number of survey respondents indicating the presence of a community-based asthma program. They also may be qualitative such as descriptions of what transpired at a series of home visits after hospital discharge; local residents' understanding of asthma; description of comments made at a community meeting regarding environmental pollutants. Successful evaluations will combine quantitative and qualitative data collection.

The Essential Points

1. Serious consideration needs to be given to evaluation as a coalition begins to ensure the existence of a systematic plan and to prevent the loss of data critical for such evaluation.
2. An evaluation plan should be designed so that it is useful for internal analysis and decision-making as well as for submission to external organizations that might be providing funding or oversight.
3. There are multiple ways for a coalition to conduct its evaluation program; the most effective way that serves a variety of purposes is an evaluation plan that includes both quantitative and qualitative measures.
4. Evaluations of coalitions need to be able to address three fundamental concerns: a) Does the coalition achieve results; b) How can the coalitions strengthen and enhance what it is currently doing; and c) What are we learning from this program that will assist this coalition or other coalitions.

TOC | Previous | Next

3300 Dundee Road • Northbrook, Illinois 60062-2348 • (847) 498-1400 • (800) 343-2227
Copyright ©1999-2009 American College of Chest Physicians. All Rights Reserved.
ACCP Privacy Policy | ACCP Web Site Terms of Use