Issues and Challenges of Outcome Measures
"Ask the Expert" Transcipts
"Issues and Challenges of Outcome Measures - Keeping Your Outcome Measures Simple" by Dawn Smart
July 7, 2004
Bill Freeman, NRC Moderator
Actual Transcript – Duration 60 Minutes
Moderator (Jul 7, 2004 2:56:38 PM)
Welcome. My name is Bill Freeman and I am the National Resource Center's Knowledge Manager, and today's Moderator.
Moderator (Jul 7, 2004 2:58:05 PM)
We are pleased and excited that you have agreed to participate in our fifth Ask the Expert online training on outcome measurements.
Moderator (Jul 7, 2004 2:58:23 PM)
First off, don't worry - this is an easy thing to do. There are three roles in this session: Dawn Smart is today's Expert; I am the Moderator; and you are the Discussion Participants.
Moderator (Jul 7, 2004 2:59:01 PM)
Dawn will take the lead and present the session. This includes providing information, answering questions and feeding discussions.
Moderator (Jul 7, 2004 2:59:26 PM)
As the Moderator, I am like a switchboard operator. When you ask your questions, they are presented to me, (you won't see them). I will then review your questions and send them on to Dawn. For most questions, I will post them live for all of us to see and for Dawn to respond.
Moderator (Jul 7, 2004 2:59:55 PM)
You are the discussion participants. This session is for you. When you want to ask questions, just type them in the bottom panel on your screen. Now, don't bother taking notes. This session has an afterlife! Following this discussion, we will email you a full transcript of the session (with simple edits).
Moderator (Jul 7, 2004 3:00:14 PM)
Ok, Dawn . . . Take it away!
Dawn Smart (Jul 7, 2004 3:00:28 PM)
Thanks for having me ...
Dawn Smart (Jul 7, 2004 3:00:37 PM)
Evaluation and outcome measurement often loom as daunting, complex tasks. Staff worries about the added burden it will create on their already heavy workloads. Management worries about the staff time and costs for development and implementation of evaluation systems. Outcome measurement does not have to be elaborate. Nor does it require a major expenditure of funds. Attention to a couple of key elements can keep organizations from designing elaborate evaluations they can neither afford nor sustain.
Dawn Smart (Jul 7, 2004 3:00:56 PM)
Outcome measurement can be a sustainable practice integrated into the organization. And perhaps most important -- developing a performance-based learning culture will help organizations seek and find the best and most efficient ways to manage outcome measurement. When there are expectations for results, data-based decision-making, and a focus on continuous improvement, everyone will have a stake in building an outcome measurement system that works and can be sustained.
Dawn Smart (Jul 7, 2004 3:01:38 PM)
So let's begin . Start with a clear focus! It is important to make a distinction between evaluation research and simpler, more straightforward outcome measurement. Evaluation research requires considerably more time and effort than a process meant to assess the extent to which a program has achieved its intended results.
Dawn Smart (Jul 7, 2004 3:02:14 PM)
Evaluation can provide information in great breadth and depth, but you have to ask, "Do we need it and how will we use it?" Focusing your outcome measurement work -- and pushing yourself to maintain that focus -- is at the heart of keeping your process manageable. What do you need to know about the program and why do you need to know it?
Dawn Smart (Jul 7, 2004 3:02:45 PM)
Here are five reasons to consider undertaking outcome measurement work and what each approach might require:
- To satisfy a funder's requirement or request for evidence that the program merits continued funding.
- To report on a program's impact as part of staff or board planning or budget-related priority setting.
- To make improvements in how the program is delivered.
- To identify what constitutes success and how it achieves that success.
- To determine whether the theory underlying the program is correct.
Dawn Smart (Jul 7, 2004 3:03:22 PM)
Do these ring true for you? Have you done evaluations for other reasons?
Dawn Smart (Jul 7, 2004 3:03:35 PM)
Anyone have a question for Dawn?
Marlene - Christian Community Health Fellowship (Jul 7, 2004 3:04:40 PM)
Those are all the reasons we've faced. The question for us is how to meet the various requests. The response is not always the same.
Dawn Smart (Jul 7, 2004 3:05:20 PM)
Meeting everyone's needs isn't easy. And it might mean doing more than one thing in your evaluation work.
Sarah - Father Joe's Villages (Jul 7, 2004 3:05:31 PM)
I teach a logic model/measuring program outcomes workshop. One of the things that I struggle with is the "procedures" part. Do you have any ideas where I can get some information on what procedures look like? Examples?
Dawn Smart (Jul 7, 2004 3:06:23 PM)
That's a complicated issue - not hard, but takes time to discuss. Maybe we can talk about it offline.
Dawn Smart (Jul 7, 2004 3:06:39 PM)
Having considered the purpose of outcome measurement, coming back to the question "what do you need to know about the program?" will help narrow the focus further. What questions are you trying to answer? What outcomes are of most interest to your key stakeholders and audiences?
Dawn Smart (Jul 7, 2004 3:07:00 PM)
For example, a housing program might want to know if the quality of housing meets established standards, is perceived by residents as satisfactory, and whether it increases housing stability for clients. Or a case management program might want to know whether clients make progress toward their goals and whether this increases their readiness to obtain and maintain permanent housing.
Dawn Smart (Jul 7, 2004 3:07:22 PM)
There may be many other interesting things to learn about these programs, but sticking to the most important elements will mean simpler evaluation design and data collection. Does this make sense?
Marlene - Christian Community Health Fellowship (Jul 7, 2004 3:08:03 PM)
Maybe someone on your board can do this or know someone who can. Often a for-profit company will give you an in-kind contribution of site development.
Dawn Smart (Jul 7, 2004 3:08:30 PM)
Procedures are the "how to" instructions about who will do what evaluation task, and when.
Dawn Smart (Jul 7, 2004 3:08:58 PM)
It's a good idea to have these written down in very explicit terms, so that others in your organization can fill in if needed.
Dawn Smart (Jul 7, 2004 3:09:31 PM)
For example, who will send out the customer satisfaction survey, to whom, when, and how will they track them when the surveys come back?
William - Development Publications (Jul 7, 2004 3:09:44 PM)
Can you comment on creating useful indicators of change?
Dawn Smart (Jul 7, 2004 3:10:25 PM)
Once you have identified the outcomes you're after, you can begin to brainstorm "how will we know that outcome has occurred?" How will you know? Can you see it by doing observation? Can you hear by doing an interview with people? Can you read by looking at a set of surveys?
Dawn Smart (Jul 7, 2004 3:11:50 PM)
These will give you ideas for the indicators of your outcome. And we'll come back to indicators in a minute.
Dawn Smart (Jul 7, 2004 3:12:04 PM)
The second step is to think about limits for your evaluation. A few, good outcomes are vastly better than a laundry list of many weak ones. Outcome measurement planning should start with brainstorming about potential outcomes and the creation of outcome chains to identify the sequence of short-term, intermediate, and long-term changes that may occur - what happens first, followed by what, leading to what?
Dawn Smart (Jul 7, 2004 3:12:31 PM)
Remember, an outcome is not what you do, but what changes for the people or groups you serve. In your board development work with sub-awardees, for example, an outcome might be "increased stability of the board." What you do, the number of hours you spend training or working with staff and board members, is an output, an important measure of your effort. But it is not an outcome.
Dawn Smart (Jul 7, 2004 3:13:03 PM)
CONTENT.
Dawn Smart (Apr 29, 2004 3:14:30 PM)
Once you have a list of possible outcomes, selecting the best and most useful ones will help simplify your approach. Use the following five criteria to choose among the possible outcomes:
- Are the outcomes relevant? Are they related to the "core business" of your program?
- Are the outcomes useful and meaningful? Will they help you understand whether or not your program is successful?
- Are your outcomes achievable? Are they realistic and attainable given the intensity and duration of service you deliver?
- Are the outcomes likely feasible and practical to measure? Will you be able to collect data related to them?
- Is it within your control to influence the outcomes for your clients?
Dawn Smart (Jul 7, 2004 3:14:04 PM)
Do these criteria make sense in helping you select the BEST outcomes?
Dawn Smart (Jul 7, 2004 3:14:56 PM)
Limiting the number of indicators used to measure each outcome also will help reduce complexity. Sometimes one indicator is all that is necessary. Careful thought about the best indicator is worth the effort. Remember, indicators define the outcomes and make them specific and measurable. For the outcome on board development noted earlier, -- increased stability of the board -- you may identify the indicator "#" and "%" of board members whose tenure on the board is at least two years.
Dawn Smart (Jul 7, 2004 3:15:35 PM)
Again, brainstorming potential indicators for each outcome is the right place to start, but selecting from among the possibilities will make the evaluation work easier. Consider these five criteria for selecting indicators:
- Does the indicator directly relate to the outcome? Does it define the outcome or capture an important characteristic of the outcome?
- Is the indicator specific?
- Is the indicator measurable or observable? Can it be seen (i.e., observed behavior), heard (i.e., participant interview), or read (i.e., client records)?
- Is it reasonable to think you can get access to the data on the indicator? Are the data available?
- Is it likely within your resources to collect the data on the indicator?
Marlene - Christian Community Health Fellowship (Jul 7, 2004 3:16:25 PM)
I found your definition of what an outcome actually is - changes to the people you serve not what you do to be very helpful. I have confused indicators and outcomes in the past as I sit down to write it all out.
Dawn Smart (Jul 7, 2004 3:17:05 PM)
There are many ways to define "outcome" and we use this one because we try to get people to focus on the benefit of their service for the people they serve.
Dawn Smart (Jul 7, 2004 3:17:47 PM)
In response to Marlene's question: the difference we see is this ... an outcome is the change and the indicator is more specific to the information we want to collect to document the change.
Dawn Smart (Jul 7, 2004 3:17:56 PM)
Does that make sense?
Mickey - Kairos Horizon Communities (Jul 7, 2004 3:18:55 PM)
Could you call the indicator the Short Range outcome and the outcome the Long Range outcome?
Dawn Smart (Jul 7, 2004 3:19:19 PM)
Actually, no. Because you still need indicators for the short range outcome as well as the long range one.
Dawn Smart (Jul 7, 2004 3:19:51 PM) Short-term, intermediate, and longer-range outcomes form a chain ... what happens to your clients first, then what happens, then what happens ...
Dawn Smart (Jul 7, 2004 3:20:07 PM)
Maybe we can look at a couple of outcomes and indicators ...
Mickey - Kairos Horizon Communities (Jul 7, 2004 3:20:30 PM)
Okay, thank you for the clarification on that.
Dawn Smart (Jul 7, 2004 3:20:42 PM)
Outcome: Increased awareness of basic financial management practices.
Dawn Smart (Jul 7, 2004 3:21:13 PM)
Indicators might be: # and % of organizations that can articulate the components included in financial management ... OR
# and % of organizations' boards that perform at least quarterly monitoring of the agency's financial performance ... OR
# and % of FBO/CBOs that comply with board-approved policies for capital and operating reserves.
Dawn Smart (Jul 7, 2004 3:22:56 PM)
So you can see that the outcome is the change ... in basic language ... while the indicators are specific and measurable and give you an idea of how you will collect the data.
Dawn Smart (Jul 7, 2004 3:23:17 PM)
Let's move on to design - Using a simple, straightforward evaluation design is essential to making outcome measurement manageable. This means thinking about when and from whom data will be collected and searching for the approach that is easiest, while still providing needed information.
Dawn Smart (Jul 7, 2004 3:23:30 PM)
It may mean measuring just once -- using post-only instead of pre- and post-measurement. You don't necessarily give up the ability to measure change. There is a way to administer a survey at the end of the program and still learn whether change has occurred. This is called a retrospective pre-test, which asks people what they know now, what they can do now, or how they feel now in comparison to when they began a program.
Dawn Smart (Jul 7, 2004 3:24:02 PM)
Retrospective pre-tests are a growing trend in evaluation and considered as reliable as pre/post methods of data collection.
Dawn Smart (Jul 7, 2004 3:24:23 PM)
Has anyone used a retrospective pre-test?
Dawn Smart (Jul 7, 2004 3:25:31 PM)
A straightforward design also may mean giving up control or comparison groups. Although the advantage of using control or comparison group designs is that they can provide powerful evidence of the effect of your program, they require considerable time and effort. It may not be necessary, however, since what you are trying to measure is "the extent to which the change you hoped or thought would happen for your clients actually occurred."
Dawn Smart (Jul 7, 2004 3:26:15 PM)
Limiting the number and type of data collection methods will also help you simplify. If one survey or interview approach provides all the information you believe you need, then use that data collection method. You may change your mind and revise your data collection strategy at a later date if you feel you didn't get all the data you needed.
Dawn Smart (Jul 7, 2004 3:26:46 PM)
There are times when combining two different data collection methods is useful and a good practice. For example, if all the data were to come from client self-report, concerns about validity and objectivity might arise. Adding observation of client behaviors or circumstances or seeking staff perspectives would strengthen the reporting of results.
Dawn Smart (Jul 7, 2004 3:27:17 PM)
But ... keeping it simple may mean living with the limitations of fewer data collection methods.
Dawn Smart (Jul 7, 2004 3:28:40 PM)
The best way to keep outcome measurement manageable is to build on what is already in place in your organization. It is always worth the time to examine existing forms, processes, and systems within the organization that can be adapted for evaluation purposes. Using existing opportunities to collect data can lessen the burden for everyone, including clients. It is the most efficient approach to data collection.
Dawn Smart (Jul 7, 2004 3:29:01 PM)
Once you have identified the outcomes and indicators for your evaluation, take the time to look at every form clients or staff now complete. You may be able to revise a form or add a question that will provide what you need. Look at every interaction clients have with your organization. You may be able to capture needed information during that interaction.
Dawn Smart (Jul 7, 2004 3:29:23 PM)
As an example, an information and referral service may now track the number of calls received and the type of information or referral requested. Two questions might be added to the tracking form to make it more useful: Were you able to provide the information/referral requested? Yes or No? (This provides data on increased client access to information on services.) When asked, did the client report that they received information that was useful? (This provides data on client satisfaction.)
Dawn Smart (Jul 7, 2004 3:30:28 PM)
Does anyone have an example of a way to use existing tools that they can share?
Dawn Smart (Jul 7, 2004 3:31:09 PM)
Another example: a staff member inspects the agency facility against quality and safety standards on a regular basis. S/he might use that opportunity to ask a few questions of clients as they are encountered during the inspection. Or s/he might be able to observe some interactions or behaviors that relate to outcomes of interest.
Marlene - Christian Community Health Fellowship (Jul 7, 2004 3:31:31 PM)
Would you design your evaluation process from the questions you ask the sub-awardees to respond to in your RFP?
Dawn Smart (Jul 7, 2004 3:32:18 PM)
Yes ... and you need to make sure that the questions in your RFP include something about outcomes. What change do they (or you) expect to happen?
Dawn Smart (Jul 7, 2004 3:33:04 PM)
You may be challenged on some of the changes you recommend. To ensure you get the support you need, it's wise to involve others in this process. You might put together a staff team with specific responsibility to look across the organization for opportunities and efficiencies. You may need to include people who can help build the buy-in you'll need to make changes to the "way things have always been done."
Dawn Smart (Jul 7, 2004 3:33:33 PM)
By changes, I mean changes you want to recommend in your existing systems.
Dawn Smart (Jul 7, 2004 3:33:50 PM)
Does anyone have an outcome measurement team in their agency?
Dawn Smart (Jul 7, 2004 3:34:39 PM)
Take advantage of current innovations in technology. Consider where electronic data collection or data entry could help you. The advances in technology are creating affordable solutions to data collection that may help create efficiencies. Hand-held data collection devices might be appropriate. Online data entry may save time, particularly if you have staff in more than one location.
Dawn Smart (Jul 7, 2004 3:35:04 PM)
Online surveying, for example, is now very accessible and affordable.
- SurveyMonkey (http://www.surveymonkey.com) and
- Zoomerang (http://www.zoomerang.com)
Dawn Smart (Jul 7, 2004 3:35:26 PM)
Has anyone used an online survey tool?
Dawn Smart (Jul 7, 2004 3:36:38 PM)
There are an increasing number of opportunities for nonprofit organizations to learn about and gain access to technology. For example, NPower (http://www.npower.org) provides training and some technology guides online that you can download, such as Technology Guide for Nonprofit Leaders: A Mission Support Tool for Health and Human Services
Dawn Smart (Jul 7, 2004 3:37:04 PM)
Also, a future "Ask the Expert" session will deal with data management systems. So stay tuned for more information on this topic!
Dawn Smart (Jul 7, 2004 3:38:17 PM)
Think about how you can simplify your reporting. Many funders tell you their reporting requirements, others do not. You should consider who needs what information -- Who is the audience? What do they care about or need to know? What decisions do they need to make? Answers to these questions will form the basis of a report.
Dawn Smart (Jul 7, 2004 3:38:36 PM)
Try simple reporting formats. Time and energy can be saved by utilizing a basic template. A model for a performance report might look like this one, adapted from the Oregon State University Family Policy Program:
The Facts:
- What, how, and from whom was the information collected?
- What are the results of the data analysis on your indicators?
- What evidence is there that you achieved your intended outcomes?
The Meaning of the Information:
- How do these findings relate to your outcomes and the program's goals?
- Why are these findings important?
Your Assessment: Given these findings and their meaning,
- To what degree have you been successful in reaching your outcomes?
- Has your work been successful for some stakeholders or in some venues, but not others?
Your Next Steps: Based on the facts, their meaning, and your assessment
- What course of action is possible?
- What changes are you considering that might improve your efforts and your performance?
Moderator (Jul 7, 2004 3:40:32 PM)
What is the biggest challenge you have found in reporting your results?
Dawn Smart (Jul 7, 2004 3:41:14 PM)
The first challenge goes back to the first question we had ... what happens when you have multiple funders with different outcome and reporting requirements?
Dawn Smart (Jul 7, 2004 3:42:00 PM)
The sad truth is ... there's no way around this problem. They are your funders' and may have requirements related to their higher-ups!
Dawn Smart (Jul 7, 2004 3:42:21 PM)
The second challenge is how to report on less than positive results …
Dawn Smart (Jul 7, 2004 3:43:18 PM)
The truth is in most social and human services ... little change is evident. So you have to report that finding. But that doesn't necessarily point to a negative result. I try hard with all of my clients to get them thinking about "how can we use our evaluation to help us improve our service delivery?" When you think of it this way, negative results can be as important as positive ones.
Dawn Smart (Jul 7, 2004 3:45:36 PM)
Does anyone want to ask a question specific to their organization or project?
Dawn Smart (Jul 7, 2004 3:46:35 PM)
I have a couple of thoughts on sustaining outcome measurement in organizations.
Dawn Smart (Jul 7, 2004 3:46:58 PM)
Integrating data collection and evaluation into agency routines will serve you best in building an outcome measurement system. Making it "standard operating procedure" helps ensure it will last over time.
Marlene - Christian Community Health Fellowship (Jul 7, 2004 3:47:58 PM)
Has Health and Human Services, Administration of Children and Families (ACF) outlined or stated what kinds of evaluation reporting they expect at the end of the grant? It has been a long time since I've read the Federal Register but I don't recall any specifics. When we submitted our proposal we identified certain outcomes that we expected. I presume we report that back to them.
Dawn Smart (Jul 7, 2004 3:49:21 PM)
The last I heard ... and I may not be current ... ACF was asking grantees to report on "what they were doing to evaluate their programs." But I think if you submitted outcomes and have some data to report, go ahead, report on your results.
Dawn Smart (Jul 7, 2004 3:50:10 PM)
Funders are always interested in results ... regardless of whether they specifically ask for them.
Dawn Smart (Jul 7, 2004 3:50:45 PM)
Another thought about sustaining your outcome measurement work …
Dawn Smart (Jul 7, 2004 3:50:52 PM)
Involving everyone in the organization will help create buy-in and institutional capacity and depth. Making outcome measurement a participatory process helps everyone feel they have a stake and a role in this work. Given the typically high turn-over rate of staff within nonprofit agencies, broad participation also means you won't lose capacity if someone leaves.
Dawn Smart (Jul 7, 2004 3:52:34 PM)
My final thought. developing a performance-based learning culture where there are expectations for information about results, data-based decision-making, and continuous improvement will help the organization seek and find the best and most efficient ways to manage outcome measurement. Does this bring up other questions?
Moderator (Jul 7, 2004 3:53:19 PM)
Our time is getting short; let's have just one more question. Any burning questions?
William - Development Publications (Jul 7, 2004 3:53:58 PM)
My interest is in community development, wherein community members do the planning, gauge the progress and interpret the results.
Dawn Smart (Jul 7, 2004 3:55:10 PM)
I am currently involved with neighborhood and community development projects where stakeholders create the design, collect the data, and interpret the results. It is very exciting to see people make it their own evaluation.
Dawn Smart (Jul 7, 2004 3:55:58 PM)
Some evaluators in the field worry about maintaining "objectivity" regarding the results. But the counter argument is ... "whose program is this anyway?"
Moderator (Jul 7, 2004 3:56:17 PM)
We want to provide the following glossary to outcome terms.
- Baseline
- Data gathered to provide a comparison for assessing program changes or impact.
- Comparative standard
- Data used as a comparison or a standard of achievement for a specific indicator or outcome.
- Compliance monitoring
- Tracking and reporting information on what and how much service a program delivers, the clients it services, how much money it expends and, possibly, the outcomes it achieved, in relation to what an organization has agreed upon, generally referring to contractual arrangements made between an organization and its grant maker on the use of funds.
- Data
- Information collected in a systematic way that is used to draw conclusions about a program or its outcomes.
- Evaluation
- The systematic application of social research procedures for assessing the conceptualization, design, implementation and utility of health or social interventions.
- Goal
- Broad statement of the ultimate aims of a program, generally beyond the ability of one organization to achieve on its own.
- Indicator
- The specific, measurable information that will be collected to track success of an outcome. Another commonly used phrase is "performance measure."
- Inputs
- The resources available to the program that allow and support service delivery, including, for example, money, staff, volunteers, materials or equipment.
- Instrument
- A tool used to collect data, including survey questionnaires, interview guides, observational checklists and written record extraction forms.
- Objective
- A specific, measurable accomplishment within a specified timeframe.
- Outcome
- The changes in the lives of individuals, families, organizations or the community as a result of this program; benefit for participants during or after their involvement in a program; the impact of the program on the people it serves.
- Outcome chains
- A logical progression of short-term, intermediate, and long-term outcomes that lead to a program's goals.
- Output
- The product delivered or the unit of service provided by the program, usually described numerically, such as number of people served or number of hours of service delivered.
- Outcome measurement
- A systematic way to assess the extent to which a program has achieved its intended results; generally used in the not-for-profit world.
- Quantitative data
- Numerical information gathered in a structured way.
- Qualitative data
- Descriptive or subjective information provided in narrative terms.
- Reliability
- The extent to which data collected are reproducible or repeatable.
- Sustainability
- Providing the best for people, organizations, and society now and in the indefinite future, or meeting the needs of the present without compromising the ability of those in the future to meet their needs.
- Validity
- The accuracy of information collected.
Moderator (Jul 7, 2004 3:57:56 PM)
Thank you for participating in today's session. As always, we are here to help you anyway that we can.
- Join us for our next Ask the Expert session on Wednesday, August 11th at 3:00 p.m. (EDT) on Using Email to Build Your Organization. Contact Yvette Green at 703-752-4331 or by email at ymg@daremightythings.com if you would like to participate.
Dawn Smart (Jul 7, 2004 3:58:09 PM)
Thanks for inviting me to play!
Moderator (Jul 7, 2004 3:58:11 PM)
If you would like to receive a transcript copy of today's session, please send me an email to wjf@daremightythings.com. We also recommend that you sign up for our free, bi-weekly e-newsletter packed with information on managing an effective organization. The subscription link is at www.ccfnews.org.
Moderator (Jul 7, 2004 3:59:18 PM)
Dawn, you rock! Thanks for your help. If you would like to contact Dawn directly for help you can reach her at 206-448-0878 or by email at dsmart@cleggassociates.com
Moderator (Jul 7, 2004 3:59:40 PM)
Goodbye!
If you have trouble reading this e-newsletter due to formatting issues, or visible HTML code, or if you would like to discuss content-related issues, please contact Bill Freeman, NRC e-Newsletter Editor at wjf@daremightythings.com.
Hypertext links and other references to non-CCF products and services are provided for information only and do not constitute endorsement or warranty, express or implied, by the CCF, DHHS, or U.S. Government, as to their suitability, content, usefulness, functioning, completeness, or accuracy.