Evaluation Perspective
Program evaluation and outcome measurement are crucially dependent on perspective. What makes sense to measure for the program overall might not be the best measurement for clients or CHWs. The most common perspectives in developing a measurement plan are:
CLIENT
This perspective centers the clients. Initial planning could include potential clients in a focus group or other planning committee to choose measures that are the most important for the clients. Measures using this perspective would center the client and be based on the individual, most likely comparing the same person at different points in time.
COMMUNITY HEALTH WORKER
This perspective might be the most useful for measuring performance of CHWs or making decisions about the allocation of personnel and resources across various sites or programs. Measures using this perspective would center the CHW and so be measured at that level, either comparing the same CHW over time, or comparing among different CHWs.
PROGRAM
This perspective would allow metrics to be calculated for an entire program. An important consideration is how different aspects of the program are combined. For example, some clients are enrolled for 30 days compared to another set enrolled for 90. Consistency in measurement, especially in the timing, is critically important. Having one program measure 30-day emergency room usage and another measure 90-day emergency room usage means that some measures may need to be adjusted (multiply the 30-day rate by 3) with notes in the report(s). Some outcomes measured from the client or CHW perspective may be aggregated to provide a program perspective. Others, such as costs, can only be meaningfully measured at the program level.
PAYER
This perspective is most likely to be used by funders of CHW programs. Using this perspective implies that outcomes important to funders, such as costs, savings, and missed appointments, are specific to the location or funder. Many of these measures will overlap with measures used for the program perspective but will be calculated differently. For example, if your program has two funders, measures of the percentage of missed appointments for all clients will need to be separated into funder groups.
COMMUNITY
This perspective uses all available information to assess the overall outcomes at a community level. For smaller programs, especially in more populous communities, this will be a more difficult perspective to use. Measurement can be of community-level outcomes, such as the percentage of people reporting unmet health care needs or the percentage of people waiting for housing. It can also take information from the other levels, such as program costs, payer costs/savings, and combine that with other information, such as reduced Medicaid expenditures, or lower unemployment payments.
Choosing the perspective is not a one-time-only decision. There may be reasons to use multiple perspectives. For example, programs might use client perspective measures when discussing the program with other community organizations, or the payer perspective when writing a report for the payer. Thinking through and writing out the different perspectives and when they will be used can have a dramatic impact on reducing workload later when programs must be justified to different stakeholders. After identifying the appropriate perspectives, program developers can move onto selecting, testing, and using different measures.
Evaluating programs is important for sustainability as well as improvement. The Online Resources section provides available resources for conducting evaluations of CHW programs. However, general principles to keep in mind are:
- Use a comparison group. Having a plan to compare outcomes across people who are/aren’t in the program and time can strengthen the impact of any evaluation.
- Measure the process. Measuring the process, numbers of visits, referrals, etc., will help assess how closely the program meets the plan. This is especially important for programs addressing social determinants of health since many community resources require extension coordination and time to manage.
- Measure the outcomes. Make sure that at least some of the measures are validated (have been used in other evaluations) to make comparisons easier.
HOW TO: CHOOSE AND USE A COMPARISON GROUP
- Best Case: Randomize who receives services for a defined length of time and measure the differences in outcomes between groups. Using a delayed waitlist for people not initially selected can be helpful.
- Best Case: Randomly “encourage” some potential clients to enroll. This can be done through a letter or other means. Compare the outcomes of the group who was encouraged to the group who wasn’t.
- Better Case: Find individuals who are similar to actual clients but who do not receive CHW services. Compare them to CHW clients.
- Good Case: Compare the same people over time.