Identifying Measures to Evaluate Network Progress: How to use both quantitative and qualitative data to tell your network’s story
This article was written by Esther Hammerschlag, MA, owner of East Chewuch Consulting, for the September 2017 edition of “Networking News.” The Network Technical Assistance Project is funded by the Federal Office of Rural Health Policy, Health Resources and Services Administration, U.S. Department of Health and Human Services through a contract to Rural Health Innovations, LLC, a subsidiary of the National Rural Health Resource Center.
Network development is an iterative and dynamic process that doesn’t typically follow a straight trajectory, meaning an evaluation plan which only tracks numbers won’t paint a complete picture of your network. I like to think of evaluation first as telling the story—a story that is then backed by numbers. Although people may argue the benefit of one type of data over another (quantitative v. qualitative), the reality is you need both to tell the full story of your network. While numbers provide measurements across time, contextual and anecdotal information bring your numbers to life. Finding this balance and artfully merging the two is where you can most successfully demonstrate your network’s value.
To accomplish this as a network evaluator, I like to administer surveys to members and/or stakeholders that include both closed-ended (e.g., multiple choice or scales) and open-ended questions. This allows me to:
Collecting and analyzing qualitative data
As an example, I recently completed a one-year follow-up evaluation of a relatively new, informal network that was working to improve its infrastructure and capacity. As one way to demonstrate value, I included an open-ended survey question asking members to describe how they had personally benefitted from participating in the network. I analyzed their responses using the same themes identified in last year’s survey, to allow comparison from one year to the next. By doing so, I found that an increasing number of members realized benefit from the network: almost half of members cited close ties to their own personal, professional, and organizational goals (nearly double than in 2016), and two-thirds indicated an increase in their own skills, knowledge, and awareness (an increase from 50% in 2016).
I then looked at responses to a multiple-choice question asking members how much they agreed or disagreed with the statement, “The goals of the coalition support the goals of my own organization.” More than double the percentage strongly agreed with the statement in 2017 than did in 2016. (Do you see the pattern emerging?)
Knowing organizations want to see numbers—and changes over time—I developed a graph showing progress in these indicators from 2016 to 2017. These indicators could continue to be measured in future years by asking the same questions, analyzing responses the same way each time. This graph shows how you can translate qualitative (descriptive) data into numerical displays of network value and progress over time.
Several members also provided valuable comments in their responses, especially as to how participation has helped them within their own organizations. One member said, “Having the opportunity for professional growth has benefited the work I do both inside and outside the coalition,” and another stated, “It has kept me in the loop about how these challenging issues are affecting our community and our agency's population.”
Turning data into a story
You can begin to see how this network’s data emerges into a story. Using this example, the following passage about member benefit could be used in newsletters, annual reports, or relevant funding applications (names were changed for the purposes of this article):
“Not only do an increasing number of our members feel the goals of the network support the goals of their own organization, but a growing number of our members indicate an increase in their own skills, knowledge and awareness. In fact, two-thirds of our members in 2017 cited an increase in their own skills, knowledge and awareness as a result of network participation, and more than double the number of members in 2017 as in 2016 strongly agreed that the goals of the network support the goals of their own organization. According to Joe Smith, CEO of Rural Community Hospital, ‘Having the opportunity for professional growth has benefited the work I do both inside and outside the coalition.’ Lucy Brown, the Executive Director of Family Planning Clinics states, ‘It has kept me in the loop about how these challenging issues are affecting our community and our agency's population.’”
You could develop similar vignettes for other elements of network value using the same approach. For example:
Identifying the right data to collect
Before you develop a survey to collection evaluation data for your network, make sure to clearly define your goal. For example, do you want to quantify benefits to the community? Do you want to quantify challenges your network has faced, and demonstrate how you’ve overcome these challenges? Or do you want to evaluate strengths and challenges related to your network’s infrastructure and leadership?
Some of my favorite open-ended questions to ask when evaluating networks are:
Responses to these types of questions can be sorted into themes and then quantified, as in the example I showed you. To provide additional measures that corroborate these themes, some of my favorite closed-ended questions use scales. For example:
Finally, think about developing dashboard tools as you write your surveys. How might responses to questions be turned into data points that can be tracked over time? Whether you want a big-picture snapshot of your network’s value, or you want to hone in on something more specific, using this approach will help you tell your story in a compelling way. Sharing a compelling network story is one of the best ways to maintain engagement and participation of your members and stakeholders, strengthen funding applications, and gain financial backing from your members by personalizing your request to make it more meaningful.
Esther Hammerschlag, MA, is the owner of East Chewuch Consulting and provides a variety of program planning and development services for health and social service programs, including strategic planning, community health assessment, and program evaluation. As a former network director, she has a special interest in working with health networks that are working to improve the health of rural communities. She brings over 17 years’ experience in the planning and implementation of a variety of health and social service programs from the perspectives of consultant as well as community facilitator, and holds a master’s degree in Rural Development from the University of Alaska Fairbanks with a focus on creating effective and meaningful collaboration to improve community health
The National Rural Health Resource Center (The Center) is a nonprofit organization dedicated to sustaining and improving health care in rural communities. Rural Health Innovations, LLC is a subsidiary of the National Rural Health Resource Center.