EDI Impact EvaluationsSetting the standard for evidence based decision making
Impact evaluations conducted by EDI typically involve baseline and follow-up surveys, where treatment and comparison groups are constructed through randomised experiments, or through econometric modelling. We are adept at working with clients who are serious about building this counterfactual, as well as understanding the channels through which the impact takes effect.
Our Impact Evaluation experience covers a broad range of sectors including agriculture and livestock, economic development, education, health, social welfare, water and sanitation, and other cross cutting areas. With the methodologies we deploy we can easily tailor our operations to meet the needs of the particular project or unit of observation.
Approximate Number of Households Interviewed by EDI
Impact evaluation of water infrastructure upgrades
This impact evaluation was undertaken on behalf of the Millennium Challenge Account Tanzania (MCA-T) , part of the Millennium Challenge Corporation (MCC) in collaboration with Social Impact, with the aim of assessing water sector projects in Dar es Salaam and Morogoro. These projects are intended to improve the supply and quality of water in the two regions, and the survey aimed to measure the impact of the water works on several outcomes of interest, such as water consumption, time spent fetching water, sources of water (domestic/commercial), water expenditures, water quality at point of consumption, incidence of water-borne disease (particularly incidence of diarrhoea among children under five years), investments in physical and human capital, time allocated within the household to productive activities, and household income.
About 5000 households were sampled across Morogoro and Dar es Salaam and an extensive baseline survey was conducted spanning both the dry and wet seasons. The Morogoro sample had an additional short survey conducted prior to the longer baseline survey. All households were sampled to continue, after the baseline, for another 3 rounds of phone surveys. This implies up to 5 rounds of data for close to half the sample. Frequent data collection is useful in order to obtain more accurate measures of water availability (as water availability varies significantly over time). Water samples were collected and tested at the household, community and system levels. Rainfall data was collected from weather stations as rainfall affects water availability. Monthly reports were also obtained from the public utilities, to provide supplementary data. Finally, the survey included a qualitative component in both the survey regions, with focus group discussions and structured interviews being conducted among a range of relevant water users and suppliers, such as female water users in low-income areas, water-related business owners, various types of water sellers, community managed water source operators, and district health and education officers.
The project will exploit differences in treatment exposure across households to measure the effects of the programme. Because treatment exposure will not be random Generalised Propensity Score Matching (GPSM) and Difference-in-differences (DD) regressions will be used to control for observables that jointly determine treatment and outcomes. A secondary identification strategy is to use distance to the distribution network as an instrument for treatment.
Experiments to Improve Learning Outcomes in Tanzania
Twaweza is collaborating with J-PAL to design a randomized control trial to evaluate and understand the effects of two interventions at 350 randomly selected public primary schools across 10 districts in Tanzania. The interventions include a school grant as well as teacher incentives, which aim to improve the channelling of resources to public primary schools for quality improvement of public education, and the learning outcomes of primary school students respectively. The baseline survey in early 2013 collected school and teacher information as well as student and household information from randomly selected classes, teachers, and students from standard 1 to 3. Participating schools were then be randomly allocated to one of four study arms, and the interventions were be rolled out in the treatment groups. In May/June and September 2013, two rounds of monitoring visits to schools collected information on teacher attendance and activities within schools. An end-line survey during late 2013 again collected teacher and school data, as well as information from students and households interviewed and tested during baseline.
EDI continued to work with TWAWEZA and IPA/JPAL on these experiments throughout 2014 and we are just embarking on the experiments for 2015 with them.
Promotion of Women’s Entrepreneurship
This was a randomised control trial of a virtual business incubator project. A survey of 832 women, a random selection of whom have been given specialised business training through a virtual business incubator project. This project, implemented on behalf of the World Bank, aimed to establish whether lack of business skills is a constraint to development and emancipation of female entrepreneurs and, if so, whether specialised training can make a difference.
Assessing the Impact of a Supplier Incentive Scheme on Coverage and Uptake of Subsidized ACTs in Remote Retail Shops
This RCT, conducted on behalf of the Clinton Health Access Initiative (CHAI), the Havard School of Public Health (HSPH) and the Affordable Medicines Facility-malaria (AMFm) was conducted over 2 years and comprised 7 rounds of retail audits visiting over 400 Accredited Drug Dispensing Outlets (ADDOs), 5 rounds of exit interviews at these ADDOs and 3 rounds of socio-economic surveys in 1,000 households
Community Based Assistance to Vulnerable Groups
A Randomised Impact Evaluation This 5 year panel survey, commissioned by the World Bank and TASAF, follows a treatment and comparison groups of vulnerable people, who participate in a community driven development project. Impacts will be identified through the staggered introduction of the project in 50 of the 100 sampled communities. This project involves a complicated listing exercise in which a short questionnaire is administered to everyone in the village (around 50,000 households in total) to assess the targetting efficiency of the programme as well as to stratify the sample within the village and oversample vulnerable groups.
Impact evaluation of the KNCU Health Plan in Tanzania
The Amsterdam Institute for International Development (AIID), together with the Amsterdam Institute for Global Health and Development (AIGHD) are implementing an impact evaluation of the Kilimanjaro Native Cooperative Union Health Plan (KNCU HP), one of the Health Insurance Fund programs rolled out by the PharmAccess Foundation in four countries in Sub-Sahara Africa. As part of this impact evaluation, in 2013 EDI conducted a large-scale baseline survey that collects socio-economic, health and biomedical information from around 1,500 coffee-growing households in the Kilimanjaro region. Some of these farmers will be provided the insurance product and other not. A follow-up survey with the same households is being conducted in early 2015. The study aims to quantify the impact of the project by comparing differences between treatment and control groups. Take-up is a key concern in micro-insurance and this study aims to study it by comparing farmers in treatment villages who did not purchase insurance to those that did. The study intends to make a contribution to the literature on micro-insurance as well as providing immediate feedback to the implementing partner of the KNCU Health Plan.
Entrepreneurship Networks and Manufacturing Firm Performance in Tanzania
In collaboration with the Centre for the Study of African Economies (CSAE) at Oxford university, EDI has implemented several rounds of surveys on African Competitiveness in light manufactured goods. They were initiated and funded by the World Bank’s Research Department and the Africa Region’s Financial and Private Sector Development Department. The survey was conducted in Tanzania, Ethiopia, Zambia and China. The objective is to understand why African countries import instead of producing some of the most simple manufactured products that they consume. About 200 young entrepreneurs presented their business ideas before a committee of experienced managers. The best 16 applications received a start-up grant of US$1,000. Surveys track the senior committee members in addition to the successful and unsuccessful young candidates.
Randomised Impact Evaluation of Handwashing with Soap and Total Sanitation and Sanitation Marketing Projects
Handwashing with soap at critical times—such as after contact with faeces and before handling food—has been shown to substantially reduce disease incidence, and results in direct and indirect health, developmental, and economic benefits by breaking the faecal-oral transmission route. Despite this benefit, rates of handwashing with soap at critical times are very low throughout the developing world. This research project exploits the randomised implementation of a handwashing campaign in Tanzania to study the effect of the campaign on the rural poor. The findings will be used to inform the scaling up of the programme to national level. Conducted on behalf of the Ministry of Health and Social Welfare and the Water and Sanitation Project at the World Bank. Download end-of-project presentation
Rural Road Upgrading
This project will run over at least 5 years to assess the impact of the upgrading of rural roads in Tanzania. EDI designed the evaluation strategy and implemented the baseline fieldwork. Our teams visited around 4,200 households in 280 communities across mainland Tanzania and Zanzibar as part of a baseline survey conducted prior to the start of the construction work. This survey is undertaken on behalf of the Millenium Challenge Account Tanzania (MCA-T) part of the Millenium Challenge Corporation (MCC). The MCC website provides more information on the evaluation and as well as a public release dataset. It is expected that during 2015, a further survey is going to be implemented to assess the impact of the upgrading
Conditional Cash Transfers
A Randomised Impact Evaluation This longitudinal survey commissioned by the Government of Tanzania evaluates a pilot of a conditional cash transfer programme of the Government of Tanzania. Cash transfers were disbursed to households assessed vulnerable by both the community and a survey instrument administered to all households in the village. Eligible (vulnerable) households needed to abide by certain conditions, such as children’s school and clinic attendance in order to remain in the transfer programme. The impacts are identified by a staggered introduction of the programme in 40 out of the 80 villages. The outcome of this project aimed to provide a basis for the government to decide how to scale this programme up to national level. The study was commissioned by TASAF and supported by the World Bank’s Evaluation Team