Three Ways to “Move the Needle” on CDFI Impact Measurement and Evaluation
Jamie McCall, Vice President of Economic Development Policy, Carolina Small Business Development Fund (CSBDF)
As a career community economic development researcher, I have no problems offering critiques of the industry. What we do is vital and transformative for the populations we serve. But I believe doing it well means we must engage in robust measurement and meaningful evaluation of our activities. I was fortunate to attend both OFN’s last pre-pandemic in-person conference and the 2022 conference. I was excited to see what looks like a groundswell culture change that is taking shape across the industry.
There seems to be a genuine desire by the CDFI community to now make progress on issues around research, impact, and evaluation. I was inspired by the conversations I saw about this at the 2022 conference, and I think we are starting to see a movement toward improving what we do in this area. I’m not an idealist – I know that this will take time and change will be incremental. But the willingness to assess how we can do better seems to be present now, and that was the critical first step. So how do we start walking toward that goal? Based on my conversations at the conference about our work at Carolina Small Business Development Fund (CSBDF), here are my three take-aways:
Pick a few things and measure them well, and don’t worry (much) about the rest.
Most CDFIs engage in a large array of activities with limited resources. So, on the data collection front, prioritize a few metrics and measure them exceedingly well. That is easier said than done: identifying a handful of metrics is no small task when everything you do seems important. Ideally, focus on items which are (1) reflective of strategic plan priorities, (2) possible to collect with pre-existing data systems (no need to reinvent the wheel!) and (3) get you closer to measuring actual outcomes.
But what about all the other data CDFIs need to collect? At least from an evaluation and impact measurement perspective, don’t worry about it. While it’s important good quality information be collected for compliance purposes, compliance reporting is not evaluation.
At CSBDF, one of the metrics we focus on is our role in rebuilding trust across community institutions. An example of a question we use to measure this construct is below. Our analyses of the data show these types of metrics are an important component of our impact, which we try to highlight via our peer-reviewed work and evaluations. If we’ve done our job well, we should see agreement in sentiments like the below hold steady or (preferably) rise across time.
Community needs should guide activities and evaluations, even if that means doing things differently than you’ve done before.
When you have a variety of stakeholders and receive funding from an array of sources, goal displacement is likely. Meaningful impact measure and evaluation is an ideal tool to keep the spotlight on what matters: meeting the needs of the populations you serve. For example, amidst the pandemic CSBDF offered 10 different loan and grant programs to North Carolina small business owners. Many of those were in partnership with local governments and North Carolina’s state government.
CSBDF engaged in an exhaustive evaluation of this project, even though no stakeholder required it, because we wanted to ensure we were meeting the community’s needs. We were proud to partner on this project with ResilNC, a non-profit research organization that focuses on black equity issues. At a high level, our data suggest both loan and grant programs were effective – but in very different ways. Loans were vital in promoting employment retention, which was a critical outcome amidst the COVID shutdowns. Concurrently, grant aid helped improve financial stability and was correlated with anticipating a need for capital in the next 12 months. We have already seen small businesses who received grant aid from CSBDF return for a loan, because they know we’re invested in their success. And that investment extends far beyond financial capital. We’ve been told that CDFI’s “don’t do grants” outside of an emergency. The data disagrees – we can, and we will – because that is what the community needs.
Sometimes things won’t work out as intended, but you don’t know if you don’t measure. And when you can, help others learn by sharing what works (and what doesn’t work).
In many ways, CDFIs are given an impossible task: cultivate development in marginalized communities while being highly adaptive and hyperlocal. Oh and, do all that with very little resources. But since CDFI’s are charged with doing so much with so little funding, how can you be efficient if you don’t measure and evaluate activities? Most CDFI staff are concerned that identifying what isn’t working will result in loss of funding. And when every dollar counts, that is an admittedly valid concern.
At CSBDF, our approach on this has three components: research, measure, and evaluate. But beyond that, we also share our work in this area with what you might call a painful level of transparency. All our program evaluations are posted on CSBDF’s website, yes, even those that don’t have positive findings. Anonymized versions of CSBDF’s data are posted and freely available, and we invite anyone to use it and confirm our findings. We do this because we believe there is no other choice – we must know what we do well and be willing to share that knowledge. There is no other way to ensure we are best serving the community.
And how does this work out in terms of funding? At least for CSBDF, funders have responded by telling us that they greatly appreciate our intentionality in this area. In our experience, grant and debt funders respect and value our desire to make effective use of their support. Nevertheless, it’s just not possible for every CDFI to copy this level of radical transparency. Having the resources to do this level of analysis is in and of itself a form of privilege. Still, we do think every CDFI can benefit from this approach – which we call research, measure, evaluate – in some way.
###
Jamie McCall is Vice President of Economic Development Policy for Carolina Small Business Development Fund (CSBDF), a CDFI serving North Carolina. Research and evaluation are a core part of CSBDF’s mission of finding holistic and sustainable interventions that bolster small business ecosystems.