RCTs – what type of learning?

Could somebody do a rigorous impact evaluation of whether all the rigorous impact evaluations out there are actually having an impact on improving outcomes for people in developing countries?

Probably not, but this is essentially the question that Lant Pritchett is asking in his latest Center for Global Development blog post. In short, he argues that the value of writing more economics papers about the impact of projects is pretty much taken on faith, without showing any evidence that those papers are changing anything. His basic argument goes: “While it might be the case that RCTs could accelerate poverty reduction this was (and is) a faith-based, not evidence-based, claim.”

Pritchett’s irony is sharp, though it doesn’t necessarily prove anything either. Of course, he’s right that believing in the value of RCTs involves a degree of faith. All policy involves a degree of faith. Of course, the randomistas are also right that people can learn more from evidence.

But what type of learning? Perhaps this is the real question we should be asking – not the yes-or-no “is anyone learning anything?” question, but rather, “how are we learning?”

Pritchett’s implication, in organizational learning language, is that the RCT fad justifies itself based on Single-Loop Learning – that we are getting better at doing evaluations and RCTs. But, he implies, that there is no evidence that Double-Loop Learning is happening – aka learning about whether RCTs are actually making anything better.

To explore this question through learning language, again we must distinguish between single and double loop learning. The single loop question asks whether an RCT helped somebody do better something that was already being done. A good example is deworming. A lot of studies have showed that deworming in schools improves health and educational achievement of kids. And, a lot of development programs, and even governments, have taken up this model. J-PAL, when it helped the Indian state of Andhra Pradesh scale up a policy of deworming in schools, made its argument based on evidence from these RCTs.

That’s an easy one – deworming works pretty much the same everywhere. What about issues that are far more contextualized?

An example here could be gender and microfinance. Initial evidence praised the effects of microfinance on women’s empowerment, but a big study in 2009 from MIT questioned the gospel and soon many other critical studies emerged as well. They suggested that access to small loans can actually have a detrimental effect on women’s empowerment, for example by increasing women’s workload in the household, pulling girls from school to help run family businesses, or allowing men to spend their wives’ loans but still forcing the wife to pay it back. Of course, each of these studies happened in a specific context, and the results didn’t necessarily apply to every microfinance program. Still, it sent shockwaves through the whole development community.

The point is, people started asking new questions. Deeper, more critical questions. And these questions made their way, quickly, into the mainstream as practitioners and organizations realized they needed to take a more critical look at their assumptions and their data. And when assumptions are questioned, double-loop learning can occur.

Did we need RCTs to figure this out? Well, here I’ll take it on faith that RCTs have helped, at least to spread the news faster. No doubt the disadvantages of microfinance would have leaked into mainstream practice eventually, but the splash of a high-profile study can provide the shock it takes to spur critical reflection, and for tougher questions to spread more quickly.

Again from Lant, writing last November: “RCTs are one hammer in the development toolkit and previously protruding nails were ignored for lack of a hammer, but not every development problem is a nail.”

True words. Some are nails, others aren’t. The best answers to difficult questions often only lead to more questions – but even if they do, we know we are starting to ask the right questions, and engaging in deeper learning.



About developingnathan
I am a reflective person. I am an introvert, a friend, a brother and a son. I appreciate a well-crafted glass of beer, piece of music and turn of phrase. I am a professional of international development, a good pianist and a Green Bay Packers fan.

3 Responses to RCTs – what type of learning?

  1. Hi Nathan.

    Very nice blog. Nice to know you’re a fellow development doer and thinker.

    I haven’t been au courant on this topic specifically, and I have to read more to become more conversant. However, the issue of the propriety and usefulness of RCTs vs. other designs for evaluating public health impact is something I care about. There has been an excellent set of papers by JP Habicht and Cesar Evora, two epidemiologists, one of which I think of as an instant classic. It essentially has to do with the simple question of “How much proof do you need?”

    I will find you the citation now and provide it below. Maybe we can have an informal brown bag journal club amongst ourselves and discuss it? I have included them below with shortened links. Both are available Full Text PDF.

    1) Cesar G. Victora, Jean-Pierre Habicht, and Jennifer Bryce. Evidence-Based Public Health: Moving Beyond Randomized Trials. American Journal of Public Health: March 2004, Vol. 94, No. 3, pp. 400-405. http://is.gd/8k1Ahl

    2) Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int. J. Epidemiol. (1999) 28 (1): 10-18 doi:10.1093/ije/28.1.10 http://is.gd/NGvaVI

    Finally, here are some related articles: http://is.gd/Htk0kq

    Cheers, Barmak

    • Interesting. I’ll check them out – and would definitely be up for a brown bag! I think a lot of us are nerdy enough to find that a great discussion topic 🙂

  2. The name is Cesar VICTORA and not Evora 🙂 Madame Evora must be singing up in the skies….

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: