[Originally posted on the CGIAR Research Program on Policies, Institutions, and Markets website.]
“Agricultural (research) programs without rigorous impact evaluation tend to focus on rapid testing (and rollout) of technologies with high-probability adopters to provide swift feedback to donor constituencies. Such programs also tend to favor numeric accomplishments over a deeper understanding of complex development processes with the unintended consequence of promoting solutions without strong evidence of impact or cost-effectiveness.”
This rather stark observation comes from a recently published paper on the targeting and early effects of the Africa RISING program in Malawi. The study, conducted by one of the program’s partners, the International Food Policy Research Institute (IFPRI), reveals multiple challenges in evaluating the impact of participatory agricultural interventions, especially in programs that work with self-selected farmers and program-selected treatment areas. The authors illustrate how the program's approach, while useful in strengthening the innovative capabilities of participating farmers, may not provide much insight on the efficacy, scalability, and poverty reduction potential of the tested technologies.
Photo credit: S.Malyon/CIAT
In this blog, we asked the authors Beliyou Haile, Carlo Azzarri, and David Spielman to summarize the main discoveries of their study and answer a few questions about the lessons learned and the potential use of this work for future participatory agricultural interventions.
Background
The current world population of 7.3 billion is expected to reach 9.7 billion in 2050, with Africa accounting for more than half of this growth. Will the world be able to feed this growing population sustainably? Are the right technologies being developed to allow smallholders to be part of the solutions?
The USAID-funded Africa Research in Sustainable Intensification for the Next Generation (Africa RISING) program aims to identify, test, and scale up successful technologies and management practices for core farming systems in Africa south of the Sahara, covering Mali, Ghana, Malawi, Tanzania, Zambia, and Ethiopia. Africa RISING technologies are best described as sustainable intensification solutions: an integrated set of inputs, practices, and technologies designed to increase productivity and conserve natural resources. The new approaches also build resilience to frequent, complex, or unpredictable climatic shocks. So, in a sense, Africa RISING is one of the projects exploring the intersections between productivity and sustainability that are a big part of our future.
The study
The study focuses on the Africa RISING program in Malawi, where researchers have been testing several agricultural technologies and practices since 2012. The program is being implemented in the Dedza and Ntcheu districts of the Central region, and the technologies include fertilized maize, improved legume varieties, as well as maize-legume and legume-legume intercropping. The study examines the targeting and early effects of the technologies on maize yield and harvest values based on data collected in 2013 from three groups – non-randomly selected households testing the technologies (beneficiaries), randomly selected households from program-target villages not participating in the program (non-beneficiaries), and randomly selected households from non-program villages with similar agroecological conditions as program villages, measured by elevation and precipitation (controls). Researchers used alternative matching techniques to construct the missing counterfactual for beneficiaries and, in addition, conducted placebo tests that estimated the extent of targeting bias, if any. Three comparisons were conducted – beneficiaries versus control (out-of-village comparison), beneficiaries versus non-beneficiaries (within-village comparison), and non-beneficiaries versus control (placebo comparison).
Results
The study finds that beneficiaries are systematically different, i.e. better-off than the other two groups along several socioeconomic dimensions including education, land size, and asset-based wealth, which suggests program targeting of better-off households. After taking into account these observed differences through matching techniques, the out-of-village comparison shows significant and positive effects on maize yield and harvest values while the results from the within-village comparison are statistically insignificant (or marginally significant) and much smaller in magnitude. While the statistical insignificance of the latter estimates is linked to small sample, a weaker magnitude implies possible upward biases of the out-of-village estimates. In other words, program beneficiaries could have attained higher yield and harvest values than control households even without the tested technologies due to the observed (and possibly unobserved) socioeconomic differences. This finding is further supported by the placebo tests results, where belonging to the program target villages is associated with a significant positive “effect.” The authors acknowledge that the analysis is based on non-experimental methods and one wave of data, with the consequent limitations with cross-section based analysis.
To read more discussion on this study and to learn more about Policies, Institutions, and Markets, please visit the PIM website.
