A concise guide to systems from Find What Works

A great, straightforward guide to systems from “Find What Works” (by Dave Algoso), that explains pretty well the different layers of thinking about systems. I summarize his post below about healthcare, but in the titles one could easily substitute just about any other field and add its own description. He distinguishes four levels of complexity, and refers to “zooming out” from Level 1 at the center to Level 4:

  • Level 1 – “Thinking about healthcare”: Focused on the patient, and the doctor-patient interaction that offers prevention or treatment.
  • Level 2 – “Thinking about healthcare systems”: Focused on all of the distinct elements that make that doctor-patient interaction possible (he cites: policy & governance; financing; physical facilities; patient access; workforce development; supply chains; health information systems)
  • Level 3 – “Systems thinking about healthcare systems”: Systems thinking adds complexity by considering not just the impact of each element on patient care, but also on all the other elements of the system. (eg: effects of policy on supply chains or workforce development; effects of healthcare information on policy; effects of financing on access or facilities or supply chains…)
  • Level 4 – “Systems thinking about healthcare”: At this level, Algoso says, we zoom out beyond the health care system and consider everything that affects health – environment, working conditions, water & sanitation, conflict & stability, food security, etc etc…

At each level, things get more and more complex, and interventions to improve healthcare will depend on what level of analysis we are using. I find this stuff fascinating, especially when it’s presented in such a clear, concise way. Next, I do a little thought exercise on using this analysis in a different context.

Should NGOs be developing tax systems for poor countries?

As my accounting professor used to say, “it depends.” Or perhaps a more accurate answer is that NGOs shouldn’t be doing it all on their own, but they may have a role to play in the process.

Oxfam is talking about income inequality*. A recent briefing paper (discovered through Duncan Green’s From Poverty to Power blog) makes the case that good public services can reduce inequality, as well as progressive tax systems. They cite analysis showing reduced economic inequality after taxes and public services, and note that services have been shown to reduce gender inequality as well. Arguing against austerity, Oxfam challenges governments to use reducing inequality as main policy goal.

In this paper, meant for policy advocacy, all the recommendations are for governments. But what does this suggest about the role of the international development community? Certainly, we’ve been in the business of providing public services in poor countries for a long time. The World Bank, INGOs, and donor agencies are all about getting more kids in school, more women and children surviving childbirth, more villages with access to potable water. Of course, times are changing (or at least goes the talk). Now international aid is increasingly going towards strengthening local systems for providing public services, instead of NGOs just building the schools and wells themselves. (Ok, it’s a slow transition…) Either way, NGOs are doing their part to reduce inequality and promote inclusive access to services, right?

Well, this paper doesn’t suggest there’s more than that, but the way they frame their analysis. It’s not just public services, and nor do taxes stand alone. Taxes, of course, help pay for public services, so a progressive tax system can have a multiplied impact on inequality.

This is an obvious point, but it’s one that is often left out when NGOs start asking how to improve public services in Country X or Y. They will be all over the free education policy and the water system administrator training – and increasingly, the social service workforce system, including stuff like helping build career paths and salary scales. But the sustainable financing of the whole thing – devising and implementing a system of progressive taxation – is pretty far out of non-profits’ areas of expertise, save for a few.

And, so it should be. That’s not where those donations or budget lines need to be going, hiring and training legions of tax experts to invade developing countries. Let the tax experts stay with the IMF and OECD, and a few specialized nonprofits.

But many NGOs are good at one thing that would be very useful in the process of developing / improving a tax system: facilitating citizen participation. Citizens should have a voice in these discussions, and since many NGOs have experience at facilitating citizen participation when it comes to public services, policy advocacy, or indigenous rights, they could bring this expertise to bear on the tax system as well. This could mean: hiring local staff with some knowledge of tax policy and using these staff to develop a clear, feasible advocacy ask that citizen groups could push for; convening national or provincial-level forums where marginalized groups could represent their interests to policymakers; or working closely with government and international agencies to facilitate participatory consultations. Also, NGOs could advocate themselves to governments that reducing inequality be a significant goal of tax policy.

If more people have more ownership of tax policy, there is likely to be a greater sense of ownership over how those taxes are spent – aka public services. NGOs have traditionally been focused on the public services side, but it seems they may be able to effectively influence the financing of those public services too, to ensure it is fair to the poor whose interests they have at the heart of their missions. Maybe some NGOs are doing this already, and I would love to see how this could work in practice!

Just food for thought.

*An aside – good for them! Here in DC, most development orgs act more on their interest to not be seen stirring politically controversial pots…

RCTs – what type of learning?

Could somebody do a rigorous impact evaluation of whether all the rigorous impact evaluations out there are actually having an impact on improving outcomes for people in developing countries?

Probably not, but this is essentially the question that Lant Pritchett is asking in his latest Center for Global Development blog post. In short, he argues that the value of writing more economics papers about the impact of projects is pretty much taken on faith, without showing any evidence that those papers are changing anything. His basic argument goes: “While it might be the case that RCTs could accelerate poverty reduction this was (and is) a faith-based, not evidence-based, claim.”

Pritchett’s irony is sharp, though it doesn’t necessarily prove anything either. Of course, he’s right that believing in the value of RCTs involves a degree of faith. All policy involves a degree of faith. Of course, the randomistas are also right that people can learn more from evidence.

But what type of learning? Perhaps this is the real question we should be asking – not the yes-or-no “is anyone learning anything?” question, but rather, “how are we learning?”

Pritchett’s implication, in organizational learning language, is that the RCT fad justifies itself based on Single-Loop Learning – that we are getting better at doing evaluations and RCTs. But, he implies, that there is no evidence that Double-Loop Learning is happening – aka learning about whether RCTs are actually making anything better.

To explore this question through learning language, again we must distinguish between single and double loop learning. The single loop question asks whether an RCT helped somebody do better something that was already being done. A good example is deworming. A lot of studies have showed that deworming in schools improves health and educational achievement of kids. And, a lot of development programs, and even governments, have taken up this model. J-PAL, when it helped the Indian state of Andhra Pradesh scale up a policy of deworming in schools, made its argument based on evidence from these RCTs.

That’s an easy one – deworming works pretty much the same everywhere. What about issues that are far more contextualized?

An example here could be gender and microfinance. Initial evidence praised the effects of microfinance on women’s empowerment, but a big study in 2009 from MIT questioned the gospel and soon many other critical studies emerged as well. They suggested that access to small loans can actually have a detrimental effect on women’s empowerment, for example by increasing women’s workload in the household, pulling girls from school to help run family businesses, or allowing men to spend their wives’ loans but still forcing the wife to pay it back. Of course, each of these studies happened in a specific context, and the results didn’t necessarily apply to every microfinance program. Still, it sent shockwaves through the whole development community.

The point is, people started asking new questions. Deeper, more critical questions. And these questions made their way, quickly, into the mainstream as practitioners and organizations realized they needed to take a more critical look at their assumptions and their data. And when assumptions are questioned, double-loop learning can occur.

Did we need RCTs to figure this out? Well, here I’ll take it on faith that RCTs have helped, at least to spread the news faster. No doubt the disadvantages of microfinance would have leaked into mainstream practice eventually, but the splash of a high-profile study can provide the shock it takes to spur critical reflection, and for tougher questions to spread more quickly.

Again from Lant, writing last November: “RCTs are one hammer in the development toolkit and previously protruding nails were ignored for lack of a hammer, but not every development problem is a nail.”

True words. Some are nails, others aren’t. The best answers to difficult questions often only lead to more questions – but even if they do, we know we are starting to ask the right questions, and engaging in deeper learning.

 

The most important questions – are they answerable?

The short version: Questions without clear answers are still worth asking. In international development, as elsewhere. 

The longer version: 

The Sheffield Institute for International Development has launched “ID100″, an initiative to determine “the 100 most important questions in international development.” I do appreciate them crowdsourcing this. But sadly, their name is misleading. Its proper name should be, “100 of the most important questions in research into international development.” 

Sure, that’s not as exciting or marketable. But they limit their questions  to only those that have a “factual answer” that can be “addressed by a research team” and that “must not be answerable by ‘it all depends’.” (See the full criteria below.) By bounding the “most important questions” with these constraints, they imply that unanswerable questions, “it depends” questions and non-factual questions (eg political, cultural, …) are so unimportant, they don’t even belong in the first 10 pages of your Google results. These types of questions may not make for neat and tidy research. I don’t deny or diminish the importance of research in development. But, neither should SIID diminish the importance of the many relevant questions that don’t have clear answers, which development researchers, practitioners, policymakers (global and national) can choose to ask, or not to ask.

These unanswerable questions are immensely valuable: they bring new perspectives to the debate, they lead to more relevant questions that may have otherwise been buried, and they allow us to take highly context-specific information into account. If, as SIID implies, we should only be asking the questions with clear answers, we are missing all of this, and deluding ourselves that A should always lead to B.

But we have a choice – to ask, or not to ask. And if we choose not to ask, if we decide that these questions aren’t important, we lose

Some examples of highly relevant questions that don’t meet SIID’s criteria:

What political constraints does evidence-based policymaking face in development settings?
How do my personal biases – based on gender, culture, upbringing, religion, etc – affect my professional judgment?
How does the cultural relevance of curricula affect how much, or how well, students learn?

Why don’t they meet the criteria? Because the answer depends on so many factors, and because those factors and their interactions are constantly changing, aka there is no factual answer. However, these questions are very relevant to those researchers, practitioners, and policymakers – in understanding the context of their work and the impact that they can have.

Their criteria for submitted questions are below.

  • Must be answerable through a realistic research design,
  • Must be of a spatial and temporal scope that reasonably could be addressed by a research team
  • Must not be formulated as a general topic area
  • Must have a factual answer and must not be answerable with ‘it all depends’
  • Except if questioning a precise statement (e.g. ‘does the earth go round the sun?’), should not be answerable by ‘yes’ or ‘no’
  • If related to impact and interventions, must contain a subject, an intervention, and a measurable outcome

Some music to help us ponder: “The Unanswered Question” by Charles Ives. (If you don’t know it, listen for the trumpet and the flute. Actually, don’t listen for anything – just listen.)

Power of Marketing – and not just for the product

This ad sells Congo-Brazzaville an image of itself. For Guinness, of course the most important message is, Drink Guinness and Feel Empowered. But that’s not the only message that’s there. Be Congolese and Feel Empowered. Express Yourself and Feel Empowered.

Of course, “Empowered” is the development way to phrase it. Someone else might say, “Feel Powerful”.

Is Guinness making people feel more powerful for their own gain? Sure. Is that a bad thing? The way they do it, I don’t think so. It’s highlighting a positive empowerment – not negative towards any person or group, and not making feel bad for not doing anything. The “be proud to express yourself as working-class Congolese” and “be proud to drink Guinness” messages are intertwined, but that doesn’t diminish the first one, which is an important one in its own right.

In development, when we “do empowerment work”, it’s easy to forget that empowerment is about… power. People having more power to do what they want. To empower people, we try to remove the constraints that they face (eg. cultural restrictions on women moving outside the house) and give them more useful tools (eg. new savings products). But often they are the ones that are easily seen. And it’s hard to see how people feel.

There’s a quote from Game of Thrones (of which many development nerds who are also real nerds may be aware): “Power resides where men believe it resides.”

If we remove constraints and provide tools, but people don’t feel powerful – don’t believe in their own power – how much can we really empower them?

Not that the first two aren’t important, nor that they can’t lead to the third. But that third part – belief in one’s own power – is the clincher, I think, and something we should consider more.

DC act of kindness

Snow was falling, and sidewalks were already slick with a half-inch of wet snow. A woman walked with her cane up to the crosswalk, clearly working hard not to fall, and assessed her options for getting across once the light turned. A young woman came up behind her, and the light turned. The young woman, seeing the other struggling, reached out her hand, and said a word. A connection was made. The one helped the other across the puddle against the curb, across the street, and over the opposite puddle. All the way, two mouths moved and two heads nodded, affirming human connection and human dignity with friendly words. Then they were on the far sidewalk. The initial reason for their connection gone, I thought I saw the young woman hesitate for a split second, as if making a decision. Just a split second. Then, her choice made, the two figures continued slowly together, the connection still visible even through the snow, until they were out of sight. 

When I moved to DC, I was worried that everyone would be focused on themselves and on the “who you know” game. I didn’t expect there was much room for simple kindnesses to strangers.

Perhaps this was unfair. Last night waiting for the bus at Cleveland Park, watching this scene unfold, my belief in the good inherent in human nature was reinforced. Even in this town.

 

Benefits of being top of the class?

Is that arrogant, smiling guy or girl giving your graduation speech actually going to be more successful? An interesting study from the UK suggests that, maybe they will, but perhaps not for the reasons you think… or maybe we all just knew it all along.

Essentially, they found that being a big fish in a small pond is better than being a equally big fish in a pond full of huge fish. The study finds that (take a breath…) kids who were in the top of their class in primary school had better test scores in secondary school, than kids who had the same scores in primary school but were in better schools, putting them in the middle of their class.

The money quote: “Non-cognitive skills such as confidence, perseverance and resilience have big effects on achievement.”

But. I would also be very interested to see how they controlled for the “inherent motivation” factor – the question of whether that confidence is inherent and not a result of their environment, and those kids were more inclined to achieve better test scores in both primary and secondary school anyway. I guess I should read the whole paper…

Follow

Get every new post delivered to your Inbox.