The Health Resources and Services Administration (HRSA), the Centers for Disease Control (CDC), and Public Health - Seattle & King County (PHSKC) require that King County agencies and organizations receiving Ryan White Title I and/or Prevention Education funding evaluate their programs. In conjunction with the Health Department staff that administer these funds, the Evaluation Committee of the HIV/AIDS Planning Council has developed the following standards in order to clarify the expectations that surround these evaluation requirements. The Committee recognizes that agencies exist primarily to serve their goals and objectives and that they often have limited resources with which to conduct evaluations. We offer these standards as a guide for constructing and implementing realistic and appropriate evaluation plans. Because of significant differences between Care Service and Prevention programs, some standards apply to all agencies and some apply only to either Care or Prevention.

All agencies

General

• The primary goal of program evaluation is to improve programs and/or to document program successes or challenges. Therefore, agencies should be asking and trying to answer questions that are important for their programs? mission, goals, and objectives.
  
  
• The Planning Council staff includes an Evaluation Coordinator because the Council recognizes that agencies do not necessarily have an evaluation expert on staff. Therefore, we encourage agencies to use this resource for technical assistance with any aspect of their evaluations.
  
  
• Evaluation plans, including data analysis, should be realistic to implement. The Committee would rather have agencies evaluate one or two questions well than many questions poorly. We encourage agencies to create and evaluate well-defined, near-term goals and objectives that are directly related to the services for which they have been funded.
  
  
• We recognize that smaller, less well-funded agencies may not be as capable of conducting complex evaluations as their larger, more well-funded counterparts.
  
  
• The Committee encourages agencies to share evaluation plans, strategies, instruments, data, and results.

Data

• All agencies should be able to answer the question, "Who are we reaching?". This means being able to report on the demographics and other important characteristics of the targeted community or communities, including level of risk behaviors for Prevention agencies.
  
  
• We recognize that, in some cases, scope of work activities are also outcomes (e.g., number of clients placed in housing, number of condoms distributed, amount of emergency financial assistance provided, etc.). Agencies are welcome to use these outcomes, but they should also make every effort to include other outcome measures in their evaluations.

Methods

• While client satisfaction surveys provide agencies with useful programmatic information, they are less important to the Planning Council and grantees for determining the usefulness and effectiveness of HIV/AIDS programs. Therefore, agencies must move away from relying solely on client satisfaction surveys in their evaluation plans.
  
  
• In many cases, this is an over-surveyed population. Therefore, the Committee encourages agencies to avoid client surveys as much as possible. Where client surveys are appropriate, agencies should craft the instruments carefully in order to limit their length.
  
  
• Whenever possible, agencies will include baseline measures and/or comparison groups in their evaluations.
  
  
• The Committee encourages agencies to find standards of success with which to compare their evaluation results. Technical assistance is available through the Evaluation Coordinator and staff at the Public Health HIV/AIDS Program Library to help agencies review available data for similar local programs or programs in other regions of the country.
  
  
• We encourage agencies to implement multi-year evaluation plans that will not only yield information about potential areas of improvement, but also assess whether programmatic changes have improved the delivery of the service or intervention.

Care service agencies

• Because these agencies receive Ryan White CARE Act funds, they will focus their evaluations on their Ryan White clients, rather than all clients who participate in their programs. This is particularly true for agencies that have a significant proportion of non-Ryan White clients in programs that receive a portion of their funding through the CARE Act.
  
  
• To the greatest extent possible, agencies will build their evaluations around measuring quantifiable outcomes.
  
  
• If possible, agencies will try to incorporate one or more of the HRSA evaluation questions into their evaluation plans (see below.)

Prevention agencies

• The CDC and this Committee recognize that evaluating prevention outcomes can be extremely expensive and complex. Therefore, the Committee invites those agencies that cannot reasonably measure outcomes to conduct formative and/or process evaluations. Formative evaluations can help agencies find out how best to reach their target population. Process evaluations can help them see whom they are reaching, if they are implementing their chosen model, and if their intervention seems relevant to their target population.
  
  
• We encourage agencies that are able to measure outcomes (such as changes in knowledge, attitudes, beliefs, and behaviors) to incorporate these measures into their evaluation plans.

1998 DRAFT RWCA EVALUATION QUESTIONS