Impacts assessments without true baselines: assessing the relative effects of training on the performance of water user associations in southern Tajikistan
Conducting rigorous evaluations of whether the process of creating new institutions affects their performance of mandated duties presents several challenges. Not only is assignment to process often not random, but when the process of creating new institutions starts, outcomes and other performance-i...
| Main Authors: | , , |
|---|---|
| Format: | Journal Article |
| Language: | Inglés |
| Published: |
World Scientific Publishing
2018
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10568/92141 |
| _version_ | 1855517459243597824 |
|---|---|
| author | Balasubramanya, Soumya Price, Joseph P.G. Horbulyk, Theodore M. |
| author_browse | Balasubramanya, Soumya Horbulyk, Theodore M. Price, Joseph P.G. |
| author_facet | Balasubramanya, Soumya Price, Joseph P.G. Horbulyk, Theodore M. |
| author_sort | Balasubramanya, Soumya |
| collection | Repository of Agricultural Research Outputs (CGSpace) |
| description | Conducting rigorous evaluations of whether the process of creating new institutions affects their performance of mandated duties presents several challenges. Not only is assignment to process often not random, but when the process of creating new institutions starts, outcomes and other performance-influencing covariates are not measurable because the yet-to-be created institutions are not functioning at baseline. This paper compares the performance of 74 ‘treated’ water user associations (WUAs) in Tajikistan that were created using a longer training process with 67 ‘control’ WUAs that were created using shorter training, to assess the impact of training on WUA performance of mandated duties. First, propensity scores were constructed to estimate the probabilities of being ‘treated’ by treatment status. These results guided the application of the difference-in-difference technique with right-hand side covariates in a context where field measures of outcomes and other performance-influencing covariates were made after the new institutions were created and functioning. The first measures were taken within 12–18 months of the new institutions being functional and the second measures were taken 24 months after the first. This choice of methods introduces a bias due to measurement error causing an underestimate of the treatment effects, while controlling for biases due to time-invariant and time-varying unobservables. An alternative method that only compared the differences in outcomes at a single point in time after the new institutions were created would have provided an inaccurate estimate of the effects of the intervention. This is a context in which methods such as synthetic controls are impossible to employ due to the nature of the intervention, other macroeconomic structural changes, and severe data restrictions. The methodology employed here generates evidence that, while biased toward generating an underestimate of effect, can still be useful and informative for policy and management purposes, and for evaluating the impact of process on the functioning of new institutions in transition settings. |
| format | Journal Article |
| id | CGSpace92141 |
| institution | CGIAR Consortium |
| language | Inglés |
| publishDate | 2018 |
| publishDateRange | 2018 |
| publishDateSort | 2018 |
| publisher | World Scientific Publishing |
| publisherStr | World Scientific Publishing |
| record_format | dspace |
| spelling | CGSpace921412025-12-08T10:29:22Z Impacts assessments without true baselines: assessing the relative effects of training on the performance of water user associations in southern Tajikistan Balasubramanya, Soumya Price, Joseph P.G. Horbulyk, Theodore M. impact assessment water user associations training performance evaluation water delivery water distribution irrigation water infrastructure Conducting rigorous evaluations of whether the process of creating new institutions affects their performance of mandated duties presents several challenges. Not only is assignment to process often not random, but when the process of creating new institutions starts, outcomes and other performance-influencing covariates are not measurable because the yet-to-be created institutions are not functioning at baseline. This paper compares the performance of 74 ‘treated’ water user associations (WUAs) in Tajikistan that were created using a longer training process with 67 ‘control’ WUAs that were created using shorter training, to assess the impact of training on WUA performance of mandated duties. First, propensity scores were constructed to estimate the probabilities of being ‘treated’ by treatment status. These results guided the application of the difference-in-difference technique with right-hand side covariates in a context where field measures of outcomes and other performance-influencing covariates were made after the new institutions were created and functioning. The first measures were taken within 12–18 months of the new institutions being functional and the second measures were taken 24 months after the first. This choice of methods introduces a bias due to measurement error causing an underestimate of the treatment effects, while controlling for biases due to time-invariant and time-varying unobservables. An alternative method that only compared the differences in outcomes at a single point in time after the new institutions were created would have provided an inaccurate estimate of the effects of the intervention. This is a context in which methods such as synthetic controls are impossible to employ due to the nature of the intervention, other macroeconomic structural changes, and severe data restrictions. The methodology employed here generates evidence that, while biased toward generating an underestimate of effect, can still be useful and informative for policy and management purposes, and for evaluating the impact of process on the functioning of new institutions in transition settings. 2018-07 2018-04-24T04:40:21Z 2018-04-24T04:40:21Z Journal Article https://hdl.handle.net/10568/92141 en Open Access World Scientific Publishing Balasubramanya, Soumya; Price, Joseph P. G.; Horbulyk, Theodore M. 2018. Impacts assessments without true baselines: assessing the relative effects of training on the performance of water user associations in southern Tajikistan. 28p. (Online first) doi: 10.1142/S2382624X18500078 |
| spellingShingle | impact assessment water user associations training performance evaluation water delivery water distribution irrigation water infrastructure Balasubramanya, Soumya Price, Joseph P.G. Horbulyk, Theodore M. Impacts assessments without true baselines: assessing the relative effects of training on the performance of water user associations in southern Tajikistan |
| title | Impacts assessments without true baselines: assessing the relative effects of training on the performance of water user associations in southern Tajikistan |
| title_full | Impacts assessments without true baselines: assessing the relative effects of training on the performance of water user associations in southern Tajikistan |
| title_fullStr | Impacts assessments without true baselines: assessing the relative effects of training on the performance of water user associations in southern Tajikistan |
| title_full_unstemmed | Impacts assessments without true baselines: assessing the relative effects of training on the performance of water user associations in southern Tajikistan |
| title_short | Impacts assessments without true baselines: assessing the relative effects of training on the performance of water user associations in southern Tajikistan |
| title_sort | impacts assessments without true baselines assessing the relative effects of training on the performance of water user associations in southern tajikistan |
| topic | impact assessment water user associations training performance evaluation water delivery water distribution irrigation water infrastructure |
| url | https://hdl.handle.net/10568/92141 |
| work_keys_str_mv | AT balasubramanyasoumya impactsassessmentswithouttruebaselinesassessingtherelativeeffectsoftrainingontheperformanceofwateruserassociationsinsoutherntajikistan AT pricejosephpg impactsassessmentswithouttruebaselinesassessingtherelativeeffectsoftrainingontheperformanceofwateruserassociationsinsoutherntajikistan AT horbulyktheodorem impactsassessmentswithouttruebaselinesassessingtherelativeeffectsoftrainingontheperformanceofwateruserassociationsinsoutherntajikistan |