Skip navigation

Malaria No More Talks about Measuring Impact

Help

Malaria No More Talks about Measuring Impact

Monday, Mar 19
High
General Task
5 years 39 weeks ago

Children and bed nets

NightWatch is a communications campaign that airs nightly in malaria-endemic communities across Africa. The platform broadcasts creative and compelling multimedia content about malaria prevention, diagnosis and treatment (featuring popular local and international celebrities) across every available distribution channel, including television, radio, SMS, concerts, school curriculum, and billboards. Through this program, Malaria No More and partners Lalela Project and United Against Malaria aim not only to increase malaria awareness and education, but also to engage partners from every segment of society – from corporates and community leaders to politicians and musicians - to create a strong culture of taking action against malaria. The goal is threefold: contribute to individual behavior change, facilitate social mobilization around malaria, and engage leaders in a way that makes them more accountable for results in the fight against malaria. For more information on Nightwatch click here.

In an ideal world, organizations like ours would be able to document and prove a straightforward chain of events: people hear or see a message about malaria; they learn key facts about prevention, diagnosis, or treatment; they become empowered by this knowledge; they take action to protect themselves from malaria and put pressure on public institutions to provide the support they need; and malaria deaths drop (hopefully, to zero by 2015) as a result of this individual and collective action. While we believe NightWatch is essentially following this process in places like Cameroon and Senegal, the chain of events is never quite that simple, and measuring it is even more complex.

As many in the health field would agree, communications work calls to mind the adage that “not everything that counts can be counted, and not everything that can be counted counts.” In the case of NightWatch, success metrics - the complex indicators which truly reflect our overall goals - include:
• Program effectiveness: Are our messages reaching enough people? Are they memorable?
• Individual behavior change: Are people learning something? Do they feel more empowered? Are they taking action to protect themselves against malaria?
• Social mobilization: How many influential members of society are engaged? How strong is the culture of action?
• Leadership: Are leaders committed to, and accountable for, effective policies and programs?

Once we’ve defined “what counts,” we still face several hurdles in actually doing the counting. While stakeholders demand evidence that investments in communication pay off, compiling and presenting concrete indicators of success presents a number of challenges. For example:
Logistical challenges: Many elements of communication are intangible, like whether a message resonates with people. We therefore use qualitative research to collect evidence from focus groups or in depth interviews. These results are not generalizable, but they do validate our strategies (or highlight problems in a strategy that need to be addressed). Nationally representative quantitative data, on the other hand, are generalizable but harder to come by. The research infrastructure in many of the areas we work is limited; interviews generally have to take place face-to-face in the absence of the phone databases, widespread internet access, and media ratings technology that are the staples of quantitative research on mass communication in the U.S. and Europe. To address this challenge, we generally rely on local research agencies whose interviewers and data processing teams are experienced in producing high-quality data with limited infrastructure.
Budgetary constraints: By working with experienced local research agencies, we are able to overcome logistical challenges… but at a price. Conducting a nationally representative survey with a local research agency is cost-effective, but not cost-free. Naturally, we feel a responsibility to keep research costs to a minimum while still collecting all the necessary evidence. Therefore, though we conduct national surveys when appropriate, we have also developed a short set of questions on the most bare-bones indicators of success (message reach, knowledge, and behavior) that we can add to existing omnibus surveys at lower cost. We also strive to share findings with other partners so that our collective research dollars go further in identifying areas of success – and areas that need improvement.
Analytical limitations: The effects of communications interventions are notoriously difficult to isolate. It’s generally not possible to randomly assign exposure and conduct a randomized control trial, so we have to be content with a less definitive methodology. We may not be able to prove causality, but we can and should report on correlations and trends over time. We can also estimate the extent to which exposure to our communications is associated with more or less knowledge (or more or less action to protect against malaria), controlling for observable variables like access to media, socioeconomic status, education, etc. For example, a regression of malaria knowledge on possible explanatory variables in recent national survey data from Senegal showed a statistically significant positive impact of exposure to NightWatch messages on knowledge of key facts about malaria. While this is not a perfect measure, it does allow us to gauge whether our communication efforts are contributing to our overall goals.
Political obstacles: Both the research process and its results can stir up political trouble that stands in the way of further communication on malaria. For example, testing the effectiveness of a program in a small pilot might be a good idea on paper, but selecting a pilot site to receive program benefits can be fraught with difficulties and delays that erase the benefits of starting small. In terms of results, we could find evidence that communication is driving citizens to hold their leaders more accountable for public spending on malaria and health systems. While that is good news for the fight against malaria, it may not be welcomed by the political establishment.

As a community of organizations investing in malaria communications, we all face these challenges. We know that communication counts, so it is important for us to keep counting – using qualitative studies, cost-effective quantitative surveys, and careful analysis, recognizing the limits of the data. We must be sensitive to the politics of malaria programming, but not shy away from presenting evidence of how communication encourages individual and collective action to fight malaria.

Soul Beat Africa: Malaria Network
Need help?

Case Tracker

The case tracker gives you a way to track progress on your projects and assign cases to yourself and others.

  • Add projects to keep your cases organized.
  • Add cases to assign tasks or assignments to yourself and others for completion.
  • Cases can be reassigned, postponed, and closed, among other actions.
  • The history of a case - who it's been assigned to, its status and its priority - can be tracked viewing its comment thread.