Blog, News & Events

Measuring Impact on the Path to Scale: Take-Aways from a Monitoring and Evalulation Perspective

Wednesday, April 22, 2015
Laura Budzyna

The following is one of a series of blogs about presentations, workshops, and panels that were part of the 2015 MIT Scaling Development Ventures conference. Laura Budzyna is IDIN's Monitoring, Evaluation & Learning Coordinator. 

Ever since its founding, WeCyclers has dutifully measured the amount of trash it collects and recycles each week from households in Lagos. These tallies roll up into the company’s key outcome metric: kilograms of waste diverted from the city’s slums and landfills. 

But founder and CEO Bilikiss Adebiyi-Abiola realized that she could do more with this data than stick it in an investor report. Her team started marking down the label of each bottle they picked up – Pepsi, Sprite, Fanta – and graphed the number of bottles by the major soft drink companies that manufactured them. Armed with a damning pie chart, Bilikiss marched into the offices of Coca-Cola. Your company is one of the largest producers of trash in the city, she declared. Shouldn’t you be investing in a solution? 

It worked: Coca-Cola is now an official partner of WeCyclers. “I guess you could say that we use our data selfishly,” quipped Bilikiss.

A Different Conversation

The Measuring Outcomes and Impacts on the Path to Scale panel featured an all-star lineup: Kevin Starr of the Mulago Foundation, Kasia Stochniol of Acumen, Mike McCreless of Root Capital, and Bilikiss of WeCyclers. MIT D-Lab’s data-savvy Kendra Leith moderated. 

When I took my seat in the audience, I had expected to witness the tension between the investors and entrepreneurs: the former demanding more and more data, and the latter overwhelmed by the mounting requests. To be honest, I was looking forward to some good-natured sparring. 

But the conversation I heard was different. Judging from this panel, it seems that we’ve entered an era where data is valued and methods are lean. More social entrepreneurs see themselves as the primary users of the data they collect, and the cost is less and less burdensome for their bottom line.

Embracing “Selfish” Measurement

Bilikiss’s story is the perfect example of what Kasia Stochniol argues for every day. Since joining Acumen last year, Kasia has worked to show investees the potential of data to help their businesses grow (echoing Acumen’s call to action, "No, it’s not too much trouble to measure impact"). The same tools used to measure impact can also capture critical market insights: for instance, an SMS survey helped an Acumen investee in Kenya realize that their radio marketing strategy was way off the mark. Mike McCreless pointed out that the demand for impact data is rooted in competition: just like any business, social enterprises want to know how they’re performing relative to their peers.

Perhaps another indication of the shift toward “selfish” data is in the metrics themselves. A few years ago, the big question in impact measurement was how investors could compare the social performance of businesses in their portfolio. Enter: GIIRS, IRIS and a host of other standard metrics. But these indicators are given less importance as entrepreneurs begin to see themselves as the primary users of data. Even investor Kevin Starr was quick to deprioritize the comparability question: “First, we want to know if businesses are fulfilling their own mission. Then we can look at them against their competitors.” 

Go Lean or Go Home

Whether the demand is internal or external, a well-integrated data system has become a required part of doing business. Shrugging off concerns about cost, Starr asked, “Would you say it’s too expensive to have an accounting department?” 

Moreover, the costs are falling every day. Kasia, who works on Acumen’s Lean Data Initiative, emphasized that SMS surveys, automated voice response, and sensor technologies have dramatically decreased the cost of measurement. “You don’t need an army of enumerators anymore,” she laughed. “It’s not 1997.” 

Coupled with these low-cost data collection tools is a shift away from large, intermittent evaluations and toward real-time tracking. As Kevin put it, “It’s better to have a movie than an expensive snapshot,” especially if that snapshot is outdated by the time you see it.

Mike emphasized that a lean data approach doesn’t just cut costs; it also improves your relationship with your clients. Years ago, Root Capital found that the time-intensive in-person surveys were having negative effects. “Survey fatigue can turn into cynicism about the program,” he explained. Now, Root Capital takes a client-centered approach, collecting only the data they need and making sure to share results with respondents.

Keep Getting Better

When all is said and done, data collection is not the same thing as rigorous impact evaluation. The panelists advised that as a business grows, its impact measurement should also evolve and mature to include more rigorous methods (randomized controlled trials) and capture harder-to-measure concepts (behavior change, cost of impact). 

The WeCyclers team knows that these efforts are worth it. The more they uncover about their impact on customer behavior, the better they will be able to grow the business, benefit more people, and as their hashtag goes, #FollowTheRubbish.