Most evaluation studies take a “black box” approach. They assess the impact of an intervention on a small number of outcome indicators and they provide policy effects in a specific setting. In order for a monitoring and evaluation system to be credible and convincing, research as well as our own experience suggest it is useful to find measures of intermediate impact further up the causal chain, closer to point of the intervention itself. For example, in education, much of the policy analysis is based on the “production function” approach, in which inputs such as school physical facilities, family attributes, teacher attributes are linked to student achievement. But the effect of input variables on learning outcomes often is mediated by contextual variables such as time spent learning outside classrooms, curriculum coverage, teacher skill, teacher motivation, student motivation and engagement attention.
Our approach in design of an evaluation and monitoring study is to consider relevant contextual factors (e.g., economic, cultural, and political) and explore the impact on intermediate or structural variables. We use a graphical modeling mechanism known as Bayesian networks to represent such variables, their inter-relationships, and their impact on the final outcome. Bayesian networks are particularly suited for specifying causal relationships among numerous variables in a complex system to analyze their interrelationship and the performance of the entire system. A Bayesian network is a graphical representation of variables and their probabilistic inter-dependencies.