CASE STUDY: A Nonprofit Lean Startup
We feel strongly that the challenges of innovation in uncertainty are felt not only by business startups, but by innovators in the nonprofit world, too. Rob Emrich, founder of multiple startups, shared with us his story about how he successfully iterated his nonprofit and how his vision evolved as he “got out of the building.”
Lean Entrepreneur: What’s the name of your nonprofit?
Rob Emrich: Roadoflife.org is our website; it’s the name of our program. The actual nonprofit is called the Keren Emrich Foundation. We provide health-education curricula to schools and teachers.
LE: How did you get started?
Emrich: When I was a lot younger I had a sister who passed away from cancer. She had a nonpreventable type of cancer called neuroblastoma. I was a real young kid when she died but it affected me profoundly. As I was older, in my last year of college, I had a cousin who died of a brain tumor and my mom had a breast cancer scare and my aunts had breast cancer.
So when I was a senior in college I decided I wanted to do something to make a real difference in the fight against cancer and raise a lot money for that cause.
LE: What did you do first?
Emrich: The first thing we did was what would be called “getting out of the building,” the equivalent of customer development in startups. We specifically made appointments with the head of the cancer division at the Ohio Department of Health, who was extremely helpful to us along the way.
What we found out really quickly—and it would have been easy to just be stubborn and go out and raise money—but we found out there’s not really a problem.
LE: Classic. In what way?
Emrich: I mean, there could always be more money that goes to cancer research, but even if we were tremendously successful and we raised $50 million for cancer research, that is a drop in the bucket on a year-to-year basis compared to what the federal government puts into cancer research, let alone all the other charitable foundations combined. We had no way that would have made us any better at giving away the money we raised.
LE: So you got out of the building and the feedback was, “Hey, guys, we appreciate your enthusiasm,” but then, “Here’s the bigger picture. Here’s how the ecosystem works. Money is not really the big problem for us.”
Emrich: The people at the Ohio Department of Health made it really clear the problem isn’t research and you’re also not in a very good position to make any significant difference in research, whether through money or ideas or better allocation of funds or whatever.
A lot of times you actually can get in the way; people who donate money get in the way because they donate to the wrong things that really don’t have that much promise. They told us the really big issue out there is prevention and specifically prevention for children. A large percentage of all cancer deaths are preventable, through really simple things like eating better, not smoking, and exercising more. There’s also a massive epidemic of childhood obesity, diabetes, and other diseases of excess. We ended up latching onto that a lot more.
What we found out is that there’s a really big challenge in prevention and specifically when it comes to children because by the time you’re an adult it’s harder to change your behavior.
Our next step was to start to investigate that as much as possible. We found out that there are actually quite a few curriculums that are out there—some from the American Cancer Society, some from other tobacco-prevention programs—but just look at the outcomes.
There are still a lot of obese kids, and that’s a near-term indicator for a long-term problem. Why isn’t this working?
LE: You actively looked at outcomes and thought, “Look, current methods don’t seem to be working.” So you actually took a sort of skeptical approach rather than just execute a good idea?
Emrich: Right, and as soon as we realized we needed to work with kids, we wanted to find out what was happening in the classrooms, so we actually started talking to teachers. That was like another example of us “getting out of the building.”
LE: How many teachers did you talk to?
Emrich: We went to dozens of teacher conferences. I would say we probably talked to 50 teachers during the course of the initial development. What we found out is some of them had been given these curriculums, but this is a critical point: They were never actually using them.
We couldn’t figure out why they weren’t using them, because we knew they had them. We would go into the classroom and we’d see a curriculum from the American Heart Association sitting on the shelf.
LE: How did the American Heart Association get the curriculum to the teachers?
Emrich: Usually they got a grant that put them into the schools. There are a lot of people that can say yes or no, so usually a district will make a decision, and this is an important point about how complicated getting a product into the nonprofit sector can be. A district can say no to a curriculum, a school principal can say no, or a teacher can say no.
What we decided to do, based on knowing how many levels of no’s we could get, was to just avoid going to the two levels that could’ve given us a no, and we went right to the teachers who could give us a yes or no answer.
LE: And this is based on your time spent talking to teachers.
Emrich: Yes, about how curriculum gets into classrooms, how they even are allowed to teach certain things. How they choose what they teach.
What we found out is that it is very individualized. Although the district believes they have a really specific set of standards, for the most part teachers did what they wanted and they were evaluated based on the tests that students take at the end of the year. We realized, after talking with so many teachers, that the reason they weren’t actually teaching the other curriculums that were out there was because they didn’t have time during the day and because they had an enormous amount of pressure to teach these academic concept standards.
This is a really critical thing; other nonprofits had probably talked to teachers, but they’d come in with a preconceived notion that they had to have a peer-reviewed program that’s been used in small cohort studies and demonstrated to be successful and then go in and give that to teachers. It’s hard for teachers to do a 50-lesson approach, however, so all the other curriculums that had been built, it was either you do all 50 of these lessons or you don’t do any.
We knew that that was completely unrealistic. So immediately we made a decision to sacrifice what’s called program fidelity versus adoption.
LE: What does that mean exactly?
Emrich: Like how often a teacher actually completes a lesson or how well the program is actually followed to the letter.
LE: So you find out about the teachers’ pains, and how they do their jobs every day. Then you say, “Okay, we want to do something. Something is better than nothing, so we’ll sacrifice some fidelity in order to get adoption.” Is that sort of the thought process with that?
Emrich: That was one of the thought processes, and the other is: We’re going to go directly to teachers to get an up-or-down answer from them right away. So we started to put stuff in front of them and say, “Is this enough?” What we found out was it was difficult to teach what we were putting out there because it wasn’t aligned to the academic content standards. So, at the time—and this is different now—but at the time every single state had its own academic content standards. It was part of the No Child Left Behind act passed by Congress to begin the process of standardizing what’s taught in schools.
So no teacher actually had time to teach any lessons during the day, because they only had to teach what was in the academic content standards and what was going to be in the test at the end of the year.
So our big insight was that we took our lessons and transformed them to align to the academic content standards for both national health education standards and math. This was like our big innovation that actually helped make our program much more successful.
This was a pivot that I think was critical. So instead of just teaching kids about health, we would talk to them about their body mass index (BMI), based on teaching them a lesson on long division. For all the things we wanted to teach, we found out where it could correspond to something that students already had to learn, and then we taught health in the context of the academic content standards.
LE: Why do you think that hadn’t happened before?
Emrich: I think no one actually got out of the building and talked to teachers and observed teaching in the classroom. I think the programs were designed by doctors who were much more interested in building small cohort studies that could be peer reviewed, and would demonstrate that the programs work in a vacuum. When a teacher is forced to actually teach all those lessons, it works very well.
Once we had built the program, it was online, and any teacher could use it, it still wasn’t being used much and we didn’t understand why. So we got back out of the building and they kept telling us, “Well, I like to have a book in my classroom.” We’re always reading about these new smart whiteboards that are in all the classrooms. It turns out teachers didn’t really know how to use them yet.
LE: So there’s this supposed technological innovation, these smart whiteboards and blackboards. . . .
Emrich: And computers. . . .
LE: And yet the adoption wasn’t very high; people weren’t actually using them. They were telling you, “Look, we want a physical paper book.”
Emrich: Right. So they actually wanted to be able to tear the pages out of a workbook, and photocopy them so they could give them to the kids, because that’s what they’re used to doing. They go through a lesson plan that they probably developed at home after hours or over the summer. They make photocopies on the copy machine at the school, and for some of them it was, “We don’t have a printer, but we do have this copy machine.”
LE: So this was after you had already created all the online content?
Emrich: Right. We did that in conjunction with teachers. Then we aligned it to all the state academic content standards and changed the nature of all the lessons. Then we found out that we still were missing part of the story: Younger teachers were teaching our curriculum because they were using this technology; they had been taught how to use it and were using it.
LE: So you guys had aligned the contents to the academic standards, you guys built an online platform, and younger teachers thought, “That’s pretty cool.” They were using it.
Emrich: Right, but the more experienced teachers, who make up the vast majority of teachers, weren’t.
LE: At that point was it actual conscious testing?
Emrich: There were two types of experiments that were running. One, about the actual changes that our curriculum produced, like what knowledge, attitudes, and behaviors had changed with the kids, and we did that with pre- and posttests. So that was one level of testing—that to be relevant on a long-term basis, we had to demonstrate that what we were doing mattered.
The other thing was just in terms of: How do we get more teachers to do this? That was the usability testing we did. That’s how we found out that older teachers were uncomfortable using a laptop in the classroom or a computer. It just didn’t make sense for them. So through surveys and talking to teachers directly at conferences, they would say, “I want it printed out,” and we would say to them, “It’s so easy to print out. This format automatically creates a PDF from the Wiki. We spent a lot of money using this technology; just print it yourself.”
LE: Their response?
Emrich: They didn’t do it. We thought they were going to do it but they didn’t. We didn’t see our adoption rates go up. They didn’t have printers, or they didn’t know how to use them, or they were just not comfortable doing it that way. We eventually listened to them and printed our own version through Lulu.
LE: You guys printed your own book through Lulu, the online self-publishing service.
Emrich: Right, and we had school districts buy it. So state governments and school districts bought our curriculum. We had teachers download the curriculum and then we asked them, “Would you be more likely to teach this if you had a physical copy?” Almost all of them, 99 percent of them, had said yes. So what we would do is match them up to a funder who was willing to pay for the paper curriculum, and in that way we became completely self-sustainable. So, this is significant; our program wasn’t paid by donations anymore. It was paid from either the school district or a teacher or anyone who wanted our curriculum, and they actually paid us for it. We stopped asking for donations. I mean we still got significant donations from Microsoft, from Aetna, from Dannon, from other big companies, and from the federal government, but we stopped asking.
LE: Being able to have the freedom to not ask for donations, how did that affect you guys?
Emrich: It allowed us to completely focus on delivering a better curriculum that would be used by more teachers, rather than creating this narrative about “all the great things our nonprofit is doing” in order to get more funding. That’s normally the paradigm. Every nonprofit tells a story and they come up with things that sound like facts about what they’re doing, like, “This is how many people slept in our homeless shelter during the course of the year. This is how many meals we actually fed to people.”
LE: Because they have different problems and it’s not just a smaller version of a big company.
Emrich: That’s probably one of the biggest problems. When you’re trying to make a change, there’s a whole bunch of people who are just telling you what big nonprofits have to do with these already. They’re not telling you, “Don’t worry about this now; when you get bigger you can worry about it.” All they’re saying is that you need to have the following checks and balances, and you spend all your time working on administration.
LE: Why don’t you give me some vanity metrics for fun, actually? How many downloads? How much curriculum has been distributed? What are the big numbers?
Emrich: For us we looked at the amount of curriculum we distributed, and the University of Maryland Business School did a study for us. This is specifically so that we could create a social ROI [return on investment] analysis of how much each curriculum was worth to each child. So we took all of our kids and multiplied them by this number that they came up with for us and that’s how we get our $70 million of curriculum distributed, and over 250,000 kids have used our program.