Economic Impact Evaluation of Road Improvements Using Alternative Quasi-Experimental Designs: The Case of MCC’s Road Improvements in the Republic of Georgia

Benjamin Linkow NORC at the University of Chicago 4350 East-West Highway, 8th Floor Bethesda, Maryland 20814 [email protected] John Felkner, (Corresponding Author) Florida State University Department of Urban & Regional Planning 113 Collegiate Loop Tallahassee, Florida 32306 Tel: (1) 850-645-1352, Fax: (1) 850-645-4841 [email protected] Hyun Lee University of Chicago Department of Economics 5757 S. University Avenue Chicago, Illinois 60637 [email protected]

ABSTRACT This paper presents the results of the evaluation of the impacts of MCC’s US$ 200 million improvement of 220 km of highway in the Republic of Georgia using a rigorous quasiexperimental design. Three evaluation methods were implemented: a difference-in-difference method, a dose-response continuous treatment approach that estimated project impacts across geography, and a matched difference-in-difference using propensity scores. The study finds that the road improvements led to a 27% increase in the number of industrial facilities in impacted communities, a 4.2% increase in traffic volumes and a 24.4% increase in vehicle speeds, and that food prices in local markets were also significantly impacted in a complex way. Notably, no strong evidence is observed on income, consumption, asset ownership, employment, or utilization of health and education services. Both the results and methodology contribute to the growing literature on infrastructure impact evaluations with policy implications regarding both the impact magnitudes and affected sectors from rural road improvements.

Keywords: development effectiveness, poverty alleviation, infrastructure impact evaluation, experimental methods, road improvements, Republic of Georgia

ACKNOWLEDGEMENTS We would like to thank The Urban Institute and our colleagues in Georgia who supported this work: Miranda Berishvili, Giorgi Giorgadze, Sophia Chiburdanize, and Dea Tsartsidze, as well as our data collection partners CRRC, Gorbi, and ACT. We are also indebted to our colleagues at the Millennium Challenge Corporation and Millennium Challenge Georgia for their guidance and support. In particular, we thank Celeste Tarricone, Marc Shapiro, Ryan Moore, Mamuka Shatirishvili, Tengiz Chumburidze, and Leon Pietsch. Special thanks to Safaa Amer, Fritz Scheuren and Mawadda Damon at NORC without whom this evaluation would not have been possible. For helpful comments, we thank Lasha Mgeladze, Mamuka Shatirishvili, Zurab Kviriashvili, Tengiz Tsekvava, Maximo Torero, Adam Rothenburg, and staff at MCC. This research was funded solely by the Millennium Challenge Corporation (www.mcc.gov).

INTRODUCTION Spatial economic theory going back to the Von Thunen (1826) land rent model has argued that spatial access to markets, controlled by transportation costs, is crucial for economic development. Where road networks are poorly developed, transportation costs are high, affecting households, reducing real incomes, and limiting access to health, education, and employment. High transportation costs can inhibit investment, innovation, and the competitiveness of the private sector in export markets. Poor road infrastructure also limits the flow of information and access to markets, leading to imperfect competition and misallocation of productive resources (World Bank 2009). Given the economic importance of road network improvements for economic development in developing countries, it is not surprising that often the majority of foreign aid provided is for road related projects: for example, the African Development Bank invests almost 80% of its portfolio in infrastructure. Since the onset of the global financial crisis in 2008 and with rising road materials costs, fiscal pressures on official infrastructure development assistance support have greatly intensified, while demands from external stakeholders and developing countries in a world of rapid urbanization have only increased. At the same time, maintaining public support for development aid given continued uncertainty about the effects of development projects and a climate of increasing development budget cuts has resulted in a global emphasis on results-based evaluation systems (Hansen et al 2011). Consequently, focus on results-based evaluation systems for road infrastructure to provide tangible evidence that the designed infrastructure has met and is meeting the objectives originally set for it, such as poverty reduction, is unprecedented (Nangia 2013). Rigorous results-based evaluations of programs including infrastructure improvements offer the potential to identify which types of interventions work, which projects have succeeded in their goals, and lessons learned which in turn can inform improved donor spending (Smith et al 2013). This is particularly true for informing decisions on road improvements as they are traditionally (and in most cases today) made using cost-benefit measures usually focusing on Vehicle Operating Cost (VOC) savings1 (Van de Walle 2009). In this context, the use of experimental design evaluations (Randomized Control Trials (RCTs) or quasi-experimental designs) are seen as a “gold standard” and are in increasing demand because of their robust ability to reduce estimation bias and more precisely estimate the magnitude of programs on intended outcomes (Smith et al 2013). Experimental or quasiexperimental evaluation approaches have been identified as crucial for road projects because of their rigor and their robust ability to address issues such as the endogeneity of road placements (Binswager et al 1993; Jalan and Ravallion 1998), their ability to better identify and match comparison or control groups in treatment/control approaches, and their ability to control for fixed time-invariant factors or common trends affecting both treatment and control groups (Ravallion 2007; Hansen et al 2011; Smith et al 2013; Nkonya et al 2014). Yet despite this consensus, there is little reliable empirical evidence on both the types and magnitudes of the socio-economic benefits of road improvements (Smith et al 2013). A big part of the challenge related to this is in identifying and defining both treatment and control groups. Unlike other aid interventions where there may be a very clearly identified set of program beneficiaries, as for example a set of farmers receiving loans or grants, road improvements do 1

As for example within the Highway Development Model (HDM) used widely in road maintenance departments and ministries in developing countries.

not reduce travel times and costs for a well-defined set of beneficiaries in a uniform way. Rather, the effects of such improvements are spatially mediated: the extent to which travel times and costs are reduced depends on the particular location of the beneficiary, with some experiencing greater reductions than others depending on the importance of the road in travelling to various destinations of interest. Relatively few road evaluations attempt to quantify how the improvement benefit varies across geography – the spatial spillover effects of the intervention – for different beneficiaries at different locations. For example, in a typical binary treatment variable in a standard treatment and control set-up (e.g. using difference-in-differences), often the “treatment” group may be selected from within an arbitrary fixed radius – say 5 kilometers – of the project roads. In this paper, we conduct an evaluation of the socio-economic impacts of The Millennium Challenge Corporation (MCC) US$ 200 million improvement of 220 km of highway in the Republic of Georgia, completed 2008-2010, using a rigorous quasi-experimental design that includes an approach that estimates quantified variation in project improvement benefits across geography and space. The improved highway provides significant trunk connectivity in Georgia (see Figure 1), as it connects the capitol, Tbilisi, to both Turkey and Armenia to the south, and travels through one of the poorest and least accessible regions of Georgia. Three distinct evaluation methods are implemented to allow for a robust comparison of estimated outcomes: (a) a difference-in-difference approach, (b) a matched difference-in-difference using propensity scores, and (c) a “dose-response” continuous treatment approach that estimated project impacts across geography, quantifying the variation in impact for individual communities at individual locations. The study takes advantage of an extensive set of data to measure impact that was collected ex-ante, during the project, and ex-post, including a) 36 rounds of a quarterly household survey in more than 300 communities across Georgia from 2003-2011, b) a community-level survey of approximately 1000 Georgian communities in 3 rounds, and c) 12 rounds of a quarterly traffic survey conducted from 2009-2012 on both treatment and comparison roads that measured traffic volumes and vehicle speeds. We also take advantage of highly detailed and accurate Georgian road network GIS data, supplied by the Georgian roads ministry (RDMED). The study finds that the road improvements led to a 27% increase in the number of industrial facilities in impacted communities, a 4.2% increase in traffic volumes and a 24.4% increase in vehicle speeds, and that food prices in local markets were also significantly impacted in a complex way but which we explain in accordance with theoretical expectations. Notably, no strong evidence is observed on income, consumption, asset ownership, employment, or utilization of health and education services, although the lack of findings of a significant impact on some of these welfare outcomes has precedents in the literature Cuong (2011) and Gachassin et al (2010) and may well be related to insufficient time ex-post improvements for measurement (Mu and Van de Walle 2011). Both the results and methodology contribute to the growing literature on experimental infrastructure impact evaluations, through the development of an innovative method for quantitatively estimating spatial variation in road improvement impacts across geography, and with policy implications regarding both the impact magnitudes and affected sectors from rural road improvements.

PROJECT BACKGROUND AND GOALS In September 2005, the Millennium Challenge Corporation (MCC), a US government aid agency , signed a five-year, $395.3 million Compact with the Government of Georgia3 to improve two main barriers to economic growth in rural areas of Georgia: a lack of reliable infrastructure and the slow development of businesses, particularly agribusiness. The SamtskheJavakheti Roads (SJ Roads) Activity (see Figure 1) was a primary component of the Compact designed to address chronic infrastructure challenges, with a particular focus on rehabilitating key regional transport routes. To assess the efficacy of its investments in this area, MCC commissioned a rigorous impact evaluation of the SJ roads activity. Beginning in June 2006, NORC at the University of Chicago was engaged in the process of evaluating the SJ roads project. This report presents the final results of the impact evaluation. 2

Samtskhe-Javakheti Roads Activity The Samtskhe-Javakheti region is one of the poorest regions of Georgia, with a per capita income significantly below the national average and a high dependency on subsistence agriculture. More than 40 percent of the population had a daily income of $2 or less in 2009 (Care International 2010). Agriculture consists primarily of potatoes but also cabbages, carrots and grain, with some cattle breeding for export (dairy products are mostly consumed or sold locally) (Wheatley 2004). Industrial production in the region has declined dramatically during the post-communist era: According to figures provided by the Governor’s Office of SamtskheJavakheti, between the first half of 1996 and the first half of 2001, industrial and agricultural production fell by 49.7% in the two rayons4 of Javakheti. The business sector is underdeveloped and employment opportunities limited. In July 2010, only 2.2 percent of Georgian businesses in the National Business Registry of the National Statistics Office of Georgia were from the Samtskhe-Javakheti region. The region was particularly hard hit by the Russia-Georgia War in 2008, with the GDP in the region dropping 11.3 percent from 2008 to 2009 compared to a more modest decline of 5.9 percent across all of Georgia (IOM 2010). Infrastructure connectivity within and to areas outside the region is notoriously poor, and significant deterioration of road conditions in the region, due in part to the elimination of previous Russian funds for periodic road maintenance pre-1991, have greatly reduced mobility in the region. The SJ road – the primary road linking Tbilisi to the SJ region and providing connectivity to Turkey and Armenia – deteriorated significantly during the 1990s and early 2000s. By 2005, the SJ road condition has deteriorated to the point that some sections measured International Roughness Index (IRI) as high as 16.6, with a total travel time from Tbilisi to Turkey of more than 8 hours. With high costs to transport produce out of the region, regional farmers have been unable to compete with farmers from other regions. Moreover, the poor road infrastructure also created significant obstacles to importing high quality agricultural inputs and other goods. Due to the very difficult local economic conditions, local households frequently

2

See www.mcc.gov The MCC Compact in Georgia entered into force in April 2006 and was completed March 29, 2011. 4 A rayon is a Soviet-era administrative unit 3

have functioned on a non-cash barter economy, trading food, stone, timber and, in particular, oil products and gas5.

Figure 1: SJ Road Improvement Sections in Georgia

Consequently, the MCC Samtskhe-Javakehti Road Rehabilitation Activity (S-J Road), with an initial budget of $203.5 million, was one of the three Regional Infrastructure Rehabilitation Project activities in the MCC Georgia Compact. The S-J Road project goal included the rehabilitation and/or construction of approximately 220 km of the main road that traverses the isolated Kvemo-Kartli and Samtskhe-Javakehti Regions and extends to the Turkish and Armenian border (MCA-Georgia 2009).6 Figure 1 shows the location of the SJ improvement Lots in Georgia. Rehabilitation of the roads under the SJ Roads Activity included paving the roads, as well as constructing bridges and drainage systems. Construction and rehabilitation of the roads was carried out by a number of international and local construction firms on the basis of a competitive procurement process. Construction began in the spring of 2008, and was largely completed by December 2010, with some additional minor work on some of the road segments proceeding through the following few months. Implementation Issues While the planned rehabilitation of the roads was completed in December 2010, a number of difficulties, that are important considerations in the context of the evaluation, arose during the 5

The existence of local barter economies was verified by NORC personnel during informal interviews with local residents conducted in July 2006. 6 The original scope of the SJ road was planned for 245 km of improvements, but subsequent alternations to the scope resulted in the final improvement length of 218 km, 880 m.

implementation. Originally, the project planned to rehabilitate 245 km of roads, but the initial bids to carry out the construction that were submitted in April 2007 exceeded the available budget. As a result, the scope was reduced to a total of 170 km and the construction contracts were re-bid. In late 2008, however, MCA-Georgia made an additional $60 million available for the project, and in the course of the Compact an additional $50.7 million was re-allocated to the project from other Compact activities. These funds were used in part to improve additional 47 km of roads, increasing the total length by 22%. Hence, at a relatively late stage the scope of the project was significantly increased. The GAO report also cites delays in the initial feasibility studies, as well the need to terminate and reassign the work of one of the contractors due to poor performance, as further factors that contributed to the compressed timeframe. In the context of the present evaluation, these concerns suggest the importance of investigating the extent to which the activities under the project did in fact successfully improve the roads. In order for the project to positively impact socioeconomic outcomes such as household income, employment, and investment, it must be successful at improving travel conditions. Thus, in addition to the impact of the project on socioeconomic outcomes of interest, it is important to also consider the impact of the project on intermediate outcomes such as traffic counts, vehicle speeds, and travel times. In so doing, we can address the question of whether the roads successfully improved travel conditions, as distinct from the question of how these improved travel conditions led to broader outcomes. METHODOLOGICAL APPROACHES AND DATA Outcome Categories The outcomes that the evaluation considers stem from the existing literature on economic and social impacts of rural infrastructure improvements in developing countries, and the available data. Improving the quality of the road reduces the time and costs associated with transportation. As a result, demand for transportation increases, resulting in more trips along the improved road, as well as lower prices for goods that are transported along the road. At the household level, real consumption increases as a result of both lower prices for consumer goods, as well as increased earnings from employment and business opportunities due to the intervention. At the firm level, lower transportation costs mean increased real returns on investment, and thus we expect higher levels of investment. Finally, these impacts lead to economic growth and reduced poverty. These outcomes are comprised of five categories as follows: Transportation-Improvement Indicators. We measure this using traffic counts, vehicle speeds, and self-reported travel times to points of interest; a wide range of previous studies have considered similar outcomes (e.g. Bakht 2000, in Bangladesh; Levy 1996, in Morocco; Lucas, Davis and Rikard 1996, in Tanzania). Transportation Costs. In addition, the road improvement should reduce transport costs, and thus potentially increase access to public transportation if suppliers of transport respond to market prices. We thus consider both household-level expenditure on transportation, and the frequency of minibuses to various locations. Reductions in transportation costs have been found by Cuánto (2000) in Peru, Khandker et al (2009) in Bangladesh, as well as by Bakht (2000), Gollin & Rogerson (2010), Levy (1996), Lucas et al (1996), and Storeygard (2013). Investment, employment, and land use. Reduced transportation costs should spur productive investment as investors take advantage of lower transactions costs and generate employment as a

result. Increases in investment have been found by Ahmed and Hossain (1990), Dercon and Hoddinott (2005), Mu and Van de Walle (2011), Shrestha (2012), and Smith et al (2001). A number of studies have found beneficial impacts of roads improvements on off-farm employment, including Corral and Reardon (2001) in Nicaragua, de Janvry and Sadoulet (2001) in Mexico, Escobal (2000) in Peru, as well as by Escobal and Ponce (2004), Lanjouw et al (2001), Levy (1996), Mu and Van de Walle (2011) and Rand (2011). Additionally, we would expect changes in land use towards the new investment activities, as well as potential changes in cropping patterns to favor crops with higher transportation costs, as these become relatively cheaper to produce, as for example found by Levy (1996). The related outcomes that our evaluation considers are the number of industrial facilities at the community level, householdlevel employment in the form of the percentage of adults with regular work, as well as whether the most common land use in the community has changed and whether each of wheat, corn, grapes, or vegetable planting are reported as widespread. Market prices. Another observable channel by which the roads improvements can affect outcomes is through the market prices of goods that are transported along the roads, as observed by Ahmed and Hossain (1990), Khandker et al (2009), and Minten (1999). These effects are somewhat complex, as roads improvements could either increase or decrease prices depending on the structure of the market and who bears the transaction costs. A useful discussion of the theory behind these effects is provided by Casaburi et al (2012). In any market, there are transportation costs that are paid by both sellers, who transport the good from where it is produced to the market, and buyers, who must travel to and from the market to conduct transactions. Casaburi et al (2012) refer to the impact of fall in transportation costs on sellers as the “supply effect,” which acts as a reduction in costs and thus tends to drive prices down. Conversely, the impact on buyers is termed the “demand effect,” and here a fall in transportation costs exerts upward pressure on prices. This is because the effect of reducing buyers’ costs is effectively a reduction in the net price of the good that buyers are facing. Thus, the demand curve shifts to the right, driving prices up. Further, imperfect competition can potentially mitigate these effects. To the extent that sellers are able to exert market power, the “supply effect” will be muted. Monopolistic or oligopolistic sellers will reap the surplus of the fall in transaction costs, rather than passing it on to buyers in the form of lower prices. Thus, where this is the case we would expect less downward pressure on prices than in competitive markets. We build on this framework by incorporating the observation that these effects are also conditioned on the location of the market for a good relative to the location where it is produced. The relative incidence of transportation costs for buyers and sellers may change depending on the location of the market. Generally, we would expect more distant markets to result in relatively lower costs for sellers, increasing the relative importance of the supply effect and causing greater downward pressure on prices than on local markets. In addition, lower transportation costs will also tend to encourage sellers to sell in more distant markets rather than locally, which should act to increase the price in markets closer to where the good is produced. Location could also be an important determinant of imperfect competition. Sellers in distant markets where a good is not widely produced may have market power, where they would not on local markets where production of the good is common. As we discuss in the presentation of our results, these effects may also be impacted by characteristics of the particular good such as the potential for spoilage and the need for centralized, refrigerated storage. While our price data is limited, our data do allow us to analyze the impact of the project on consumer prices of wheat, beef, milk, potatoes, and honey.

Household welfare. Ultimately, the benefits of the road improvement should be observable in terms of benefits to households. The most direct measure of economic impact at the household level is household income, which we consider here. A number of previous studies have found positive impacts on income (Ahmed and Hossain (1990), Escobal and Ponce (2004), Ravallion and Chen (2005), Fan et al (2000)). However, income can be difficult to measure and many studies thus focus alternatively or additionally on household consumption, including Escobal and Ponce (2004) and Dercon et al (2006). Thus, we consider total consumption per household member as well. Notably, however, other studies have reported no measurable impacts on income or consumption expenditures (Cuong (2011) and Gachassin et al (2010)), which may be due to those effects taking a much longer time to be measurable than other impacts. For example, in a study of market-related outcomes of road improvement project in Viet Nam, Mu and Van de Walle (2011) find no impact on most outcomes 27 months after project completion, but positive impacts on many outcomes two years later. Moreover, both consumption and income are susceptible to measurement error and bias, and in some cases focusing instead on ownership of durable assets can be a more reliable alternative. Positive impacts of rural roads on asset ownership have been observed by Gannon and Liu (1997), Jacoby (2000), and Escobal and Ponce (2004). We thus use an asset ownership index as an additional measure of household welfare. The asset ownership index is calculated using survey data on 28 types of durable assets that are typically owned by Georgian households, including refrigerators, televisions, and stoves. The first principal component is used to construct a weighted average of each type of asset as a proxy for wealth. Such an approach was proposed by Filmer and Pritchett (2001) and Zeller et al (2006), and has since been widely used in the literature. Access to health and education. Where transaction costs related to travelling to schools or hospitals falls, utilization should increase with a corresponding improvement in long run outcomes. Improvements related to health services have been observed by Ahmed and Hossain (1990), Bell and Van Dillen (2012), Levy (1996), and Porter (2002). In terms of education, Levy (1996) finds improvements in primary education enrollment rates in Morocco, while Mu and van de Walle (2011) identify increases in primary school completion rates in Vietnam. Additional impacts on positive educational outcomes were found by Bell and Van Dillen (2012) and Khandker and Koolwal (2011). The outcomes we consider are the likelihood that treatment in a clinic, household, or from a doctor’s visit is obtained in the case of illnesses at the household level, as well as the likelihood that all primary school aged children in the household attend school. Outcomes under each of these five categories and the indicators that we use to measure them are presented in Table 1. The indicators are derived from our traffic survey, community-level survey, and household survey data, which we discuss in detail below.

Table 1: Outcomes, Indicators, and Data Sources

Outcome

Indicators

Data Source

Transportation-related outcomes Traffic counts and vehicle speeds

# of vehicles per day by vehicle type (regular car, jeep, minivan, bus, truck used for road construction, other truck without trailer, other truck with trailer, other), average speed along the road

Traffic survey

Travel times

Travel time to Tbilisi, main market in Region, main market in District, main local market by car and minibus

Community-level survey

Availability of public transport

Periodicity of minibuses to Tbilisi, main market in Region, main market in District

Community-level survey

Transportation expenditure

Household transportation expenditure

Household survey

Investment, land use, and employment Investment

Number of industrial facilities within settlement

Community-level survey

Land use

Most common use of land, widespread planting of corn, wheat, grapes, vegetables

Community-level survey

Employment

% of adults in household with regular employment

Household survey

Prices of wheat, beef, milk, potato, honey in village

Community-level survey

Income

Total household income per person

Household survey

Consumption

Total household consumption per person

Household survey

Asset ownership

Index of durable assets owned

Household survey

Health care access

% of households experiencing an illness that sought medical care at a hospital or clinic or were visited by a doctor

Household survey

Education access

Probability that all school age children attend school

Household survey

Market prices Commodity prices Household welfare

Access to health and education

Evaluation Methodology A central consideration of any impact evaluation is its approach to establishing the causal impact of the intervention. This is because for most interventions, the outcomes of interest are affected not only by the intervention itself but by a range of other factors as well. The approach of the evaluation to inferring causality is referred to in the economics literature as the “identification strategy;” the broader evaluation literature sometimes uses the term “attribution.” The literature on roads evaluations generally takes one of two main identification strategies: difference-in-difference, or continuous treatment. We summarize each of these as follows: Difference-In-Difference. A common approach in the literature is to use a binary treatment variable in the context of a standard treatment and control set-up. At the outset of the program, a selection of similar comparison roads is selected. A treatment sample of individuals or communities is selected from within a certain radius of the improved roads, with another comparison sample drawn from within the same radius (for example, within 5 kilometers) of the comparison roads. The comparison sample thus represents the counterfactual, that is, what would have happened to the treatment sample in absence of the program. The treatment variable is a dummy set equal to “1” for the treatment group and “0” for the comparison group. Thus, individual observations are defined as having either been treated or untreated, and impact evaluation calculates an Average Treatment Effect (ATE) over the treated population, sometimes incorporating propensity score matching techniques to reduce the potential for selection bias (program placement and self-selection bias). This is likely the most widely used approach in the roads evaluation literature; some recent examples of papers that have used these methods include Mu and Van de Walle (2011), Montgomery and Weiss (2011), Datta (2012), Rand (2011), Bell and Van Dillen (2014), Cuong (2011), Nkonya et al (2014). Continuous Treatment. A much rarer approach in the literature is to define treatment in terms of continuous variables. The justification for this approach is that the effects of roads improvements are spatially mediated. Unlike some other kinds of interventions, roads improvements do not reduce travel times and costs for a well-defined set of beneficiaries in a uniform way. Rather, the extent to which travel times and costs are reduced depends on the particular location of the beneficiary, with some experiencing greater reductions than others depending on the importance of the road in travelling to destinations of interest. This approach distinguishes between treated individuals in terms of having received higher or lower “doses” of the treatment depending on their location. Using a continuous treatment approach also requires that the measure of treatment be specified. One option is to use GIS data to calculate approximate travel times between each beneficiary’s location and the location of “treatment,” or the improved road, or to destinations of interest, such as markets, cities, or international borders. The “dose” of treatment is then measured by the extent to which the project has reduced an individual’s travel time. Continuous treatment approaches are much rarer in the infrastructure impact literature, as this is an evolving methodology. Notable examples include Datta (2012) and Khandker et al (2009). The principal advantage of the difference-in-difference approach is that provides coefficients that are more readily interpretable, within a framework that is easy to understand and to communicate. Thus, our main results use difference-in-difference estimates. We also include continuous treatment estimators as a way of corroborating the results. This is important in this case because to the extent that benefits are not experienced uniformly across space, a noncontinuous difference-in-difference approach can obscure the true impact. By contrast, a

continuous treatment approach can take advantage of highly detailed and accurate GIS data to measure variation in the degree of impact with increasing accessibility. In addition, while the continuous treatment and difference-in-difference estimates are both subject to potential selection bias, these potential sources differ in each case, and thus provide an important robustness check. Lastly, we also include a matched difference-in-difference approach as a further means of corroborating our findings. This approach is to estimate our difference-in-difference model, but with the sample adjusted using propensity score matching (PSM). This approach is described in detail in Gertler et al (2011). Selection of Comparison Roads Using propensity score matching, we identified eight comparison road segments to be included in the analysis. The comparison roads were selected from an inventory of 117 road segments for which data on a variety of characteristics were available from RDMED, the Georgian government roads agency. Our application of PSM in this case is to estimate a logistic regression model of the probability that a road is part of the treatment group as a function of observable characteristics. We then calculate the predicted probability (or propensity score) that a road segment is part of the treatment group for each of the eight treatment roads and 117 potential comparison roads using these estimated regression coefficients. Finally, each of the eight treatment roads is matched to a single comparison road for which the propensity score is the closest in value from among the 117 potential comparison roads. The observable characteristics that were used in the propensity score specification were as follows: average daily traffic volume per road segment, 2008 (RDMED); length of road segment; mean elevation of segment; standard deviation of elevation of segment (topographic variation); maximum elevation of segment; minimum elevation of segment; mean spatial density of Georgian communities within 20 kilometers of segment; travel time to nearest large city; travel time to nearest large town; travel time to Tbilisi. The locations of the selected roads are shown in Figure 3. Data To address these research questions, the evaluation makes use of three different datasets: a traffic dataset including vehicle counts and speeds, the Settlement Information Survey (SIS) to measure outcomes at the level of the community, and finally the Integrated Household Survey (IHHS) to investigate outcomes at the household level. Volumetric Traffic Survey. Our volumetric traffic survey was designed to record vehicle counts on a network of traffic stations on the SJ treatment roads, as well as the comparison roads. Traffic survey stations are show in Figure 2.

Figure 2: Volumetric Traffic Station Locations

The data were collected quarterly, to record variation in seasonal traffic counts, which were conducted for 4 quarters over a 3-year period for a total of 12 waves. The first wave was conducted in September 2009, with the final wave completed in May 2012. Each wave comprised one full week of data collection, for all seven days of the week. Vehicle counts were collected for 8 different vehicle types heading in either direction along the roads. Vehicle counts were recorded in most cases by visual counting. However, for roads with very high traffic volumes where visual counting was not feasible, the traffic was recorded by video and then later played back slowly to provide accurate vehicle counts. For nighttime data collection, count locations were selected considering adjacency to nearby major population centers. Vehicle speed was measured by conducting a test drive using a typical saloon car at each of the last 11 waves of data collection. The times to traverse each road segment were recorded, and the average speed calculated using the length of the segment. Traffic speed data was collected on the project roads only. The total vehicle counts for the both the treatment and comparison roads as well as the average speed along the treatment roads measured at each wave is presented in Table 2. We note significant fluctuations in both traffic counts and vehicle speeds based on seasonality. In addition, the total vehicle counts are substantially higher along the comparison roads at all time periods.

Table 2: Traffic Counts and Vehicle Speeds By Wave

No. of vehicles,

No. of vehicles,

Avg. vehicle speed,

comparison roads

treatment roads

km/h

Wave 1, Sept 2009

346,861

97,555

n/a

Wave 2, Dec 2009

312,712

87,145

47.5

Wave 3, Feb 2010

261,960

92,751

45.1

Wave 4, May 2010

309,350

135,601

50.1

Wave 5, Aug 2010

454,187

164,639

53.0

Wave 6, Nov 2010

347,348

153,185

82.3

Wave 7, Feb 2011

236,815

110,217

61.3

Wave 8, May 2011

299,368

144,659

82.0

Wave 9, Sept 2011

381,562

180,512

77.7

Wave 10, Dec 2011

304,802

145,100

78.6

Wave 11, Feb 2012

220,995

88,629

62.5

Wave 12, May 2012

341,574

170,902

75.5

Settlement Information Survey. The Settlement Infrastructure Survey (SIS) was a panel survey designed for the evaluation to measure outcomes at the level of the village or town. SIS collected information on accessibility to key locations and social services as well as economic conditions of the settlement. The survey was conducted in three rounds: a baseline in 2007, a second round conducted in 2010, and a third round was August 2012. The data are longitudinal, i.e. the survey is administered to the same settlement at different points in time. The sample for the first round used the 2002 Census to identify a sampling frame of 732 settlements around either the project or comparison roads, of which 690 were surveyed. The sample size was increased for the second and third rounds, which conducted surveys in all settlements that met at least one of the following criteria: settlements along the SJ Road; settlements along comparison roads where traffic counts are conducted; or settlements included in the Integrated Household Survey (IHHS) that the evaluation uses to evaluate household-level outcomes, as well as any other village that was included in the baseline. The second and third rounds each included 960 settlements. The scope of baseline survey covered a variety of aspects of the settlements infrastructure and was aimed at being a monitoring and evaluation tool for multiple components of the MCC program in Georgia. These include: Geographic and demographic information; Utilities; Transport; Roads; Agriculture; Markets and their accessibility; Prices; Industry and construction; Savings and credit; Education; Health care; Programs implemented in the settlement; and Climatic and environmental conditions. To collect the data, enumerators travelled to each settlement and worked with local authorities to identify a small group of individuals who were identified as knowledgeable about conditions in the settlement. Each survey question was posed to the group to reach consensus on the most appropriate response. Survey questions were designed to be objective and straightforward for respondents to answer, so that any bias or inaccuracies resulting from the particular respondents who were selected would be minimal. The locations of the SIS communities in relation to the treatment and comparison roads are shown in Figure 3.

Figure 3: Location of SIS Sampled Communities, and Locations of Comparison Roads

Descriptive statistics for all variables used in the analysis as well as some additional settlement characteristics from the second round of the survey are presented in Table 3. We include the full sample, as well as separate samples reflecting the treatment and comparison designations that we use for the difference-in-difference analysis (settlements within 30 minutes travel time of SJ improved roads (treatment) and within 30 minutes travel time of a comparison road).

Table 3: SIS Descriptive Statistics, 2nd Round

Full Sample

Treatment

Comparison

Mean

Std. Dev.

Mean

Std. Dev.

Mean

Std. Dev.

Settlement elevation, meters

759

560

1373

522

477

310

Industrial facilities within settlement

1.69

2.41

0.72

1.66

2.47

2.71

Variable

Agriculture very common, %

77.9%

82.0%

81.5%

Wholesale/retail trade very common, %

14.4%

8.8%

18.5%

Incidence of significant natural disasters, %

79.4%

78.0%

80.8%

Poor road maintenance %

16.9%

20.0%

16.6%

Travel time to nearest project road, minutes

160.5

117.7

14.1

9.4

203.3

86.7

Travel time to nearest comparison road, minutes

78.9

57.3

136.4

24.8

15.7

8.7

Travel time to Tbilisi by car, minutes

196.1

123.3

193.4

118.8

169.6

117.4

Travel time to regional center by car, minutes

65.3

53.1

86.6

57.3

43.3

29.5

Travel time to district center by car, minutes

34.6

29.4

36.8

26.0

27.9

20.0

Travel time to Tbilisi by minibus, minutes

167.8

107.5

139.2

91.9

141.0

100.4

Travel time to regional center by minibus, minutes

61.1

49.8

84.1

40.0

49.6

31.5

Travel time to district center by minibus, minutes

34.5

23.8

35.6

20.4

29.8

17.3

Periodicity of minibuses to Tbilisi, daily

5.3

7.9

6.1

10.6

4.9

6.7

Periodicity of minibuses to regional center, daily

5.9

6.2

3.6

3.7

6.1

5.4

Periodicity of minibuses to district center, daily

6.0

7.2

4.5

7.0

6.0

5.4

Wheat planting common, %

13.1%

16.5%

17.3%

Corn planting common, %

54.2%

12.8%

66.1%

Grape planting common, %

31.2%

5.3%

48.0%

Vegetable planting common, %

58.6%

33.0%

69.3%

Price of wheat, GEL/kg

0.51

0.14

0.48

0.11

0.50

0.11

Price of beef, GEL/kg

6.38

1.16

6.36

1.44

6.43

0.98

Price of milk, GEL/liter

0.79

0.33

0.50

0.16

0.91

0.27

Price of potatoes, GEL/kg

0.50

0.17

0.35

0.13

0.54

0.14

Price of honey, GEL/kg

10.90

2.69

11.17

2.29

11.29

2.87

Number of observations

1,082

205

271

In terms of the sample as a whole, we note that settlements are predominantly agricultural, with agriculture described as “very common” in 77.9% of the settlements. However, industry is common as well, with an average of 1.69 industrial facilities (defined as factories, agricultural processing facilities, etc.) within the settlement. Settlements were located an average of 196.1 minutes from Tbilisi by car7. The treatment and comparison communities were broadly similar in terms of agricultural orientation, incidence of natural disasters, and distance from Tbilisi. However, the treatment and comparison groups show important differences in two ways. First, industry is significantly more prevalent in the comparison areas, with an average of 2.47 factories, processing plants, or canneries in comparison communities as compared to just 0.72 of treatment communities. Secondly, vineyards are an important industry in 48.0% of the comparison communities, but only 5.3% of the treatment communities. Prices of wheat, beef, and honey are similar, but prices of milk and potatoes differ substantially. Integrated Household Survey. Finally, household level outcomes are measured using the Integrated Household Survey (IHHS), a dataset that is collected by the by National Statistics Office of Georgia (GeoStat) for a variety of government and research purposes. The IHHS has been collected quarterly since 1996, and includes a range of information on household characteristics. The evaluation includes IHHS data collected between the first quarter of 2003 and the fourth quarter of 2011, a total of 36 rounds. The IHHS sample follows a two-stage procedure that is designed to produce a nationally representative sample. First, a sample of approximately 200 primary sampling units (roughly corresponding to communities) was selected in 20038. Within each primary sampling unit, a sample of households is drawn for the survey, stratified according to demographic characteristics. Each household is interviewed four times over the course of a year, and then households are re-sampled from the primary sampling units again for the following year. Thus, while the communities remain consistent over time, individual households are only interviewed quarterly over one year each. The IHHS survey instrument gathers extensive information about a broad range of household and individual characteristics, covering the following topics: Household roster and demographic data; Characteristics of dwelling, asset ownership, and land use; A weekly diary of consumption expenditures; Quarterly consumption expenditures; Employment; Additional sources of income; and Health and education outcomes and service utilization. Figure 4 shows the location of the IHHS communities in relation to the treatment and comparison roads.

7

Note that all travel time variables in the SIS survey are reported by the respondents. Thus, to the extent that the respondents may not be perfectly informed about travel times, there will be some degree of error in these variables 8 An additional 100-150 communities were added for the period from 2008-2010, which were subsequently dropped in 2011

Figure 4: Location of IHHS Communities

Descriptive statistics for all variables used in the analysis as well as some additional settlement characteristics are presented in Table 4. As above, we also divide the sample into households within 30 minutes travel time of a project road and households within 30 minutes of a comparison road.

Table 4: IHHS Descriptive Statistics

Full Sample Variable

Mean

Std. Dev.

Treatment Mean

Std. Dev.

Comparison Mean

% rural

62%

73%

76%

km to nearest city

57.1

88.3

66.0

km to nearest hospital

6.1

8.5

6.7

Household size

3.71

1.92

3.93

2.03

3.72

Std. Dev.

1.98

HH head education: secondary

46.0%

54.6%

48.7%

HH head education: higher than secondary

40.1%

27.0%

36.1%

HH head age

60.11

Primary occupation: agriculture, %

45.3%

Transport expenditure in past 3 mos., GEL

31.64

132.89

24.33

86.04

27.32

133.31

Per capita HH income in past 3 mos., GEL

152.52

196.72

130.39

133.66

153.26

190.54

Per capita HH consumption in past 3 mos., GEL

125.73

119.41

107.96

90.09

123.31

113.31

0.80

0.70

0.79

0.63

0.73

0.66

Household asset index

14.61

59.61

14.97

55.4%

61.33

14.18

50.0%

% of adults in HH with regular employment

39.0%

47.6%

40.7%

% of households experiencing illness

30.1%

15.4%

33.8%

% of HHs with illness obtaining treatment

77.6%

88.3%

74.9%

% of HHs with all primary school aged children in school

95.3%

93.3%

95.2%

Travel time to nearest project road, minutes

170.9

112.9

14.6

11.1

213.3

88.1

Travel time to nearest comparison road, minutes

61.8

46.3

137.5

17.3

14.7

8.7

Number of observations

127,347

7,296

31,111

For the sample as a whole, households are split between urban and rural areas, with 62% located in rural areas, and agricultural the primary occupation for 45.3%. Household heads average 60 years in age, with just over 86% having completed at least secondary school. 30.1% have experienced an illness in the past year, with just over three-quarters having sought treatment at either a hospital, clinic, or by having a doctor visit the home. Per capita income and consumption are low, corresponding to just US$365 and US$302 per year in 2012 currency respectively. This is likely due to under-reporting by households and casts some doubt on whether these outcomes can be accurately measured. As discussed earlier, measurement of income and consumption is often problematic, and as a result we have included a measure of asset ownership as an alternative means of assessing livelihoods.

Econometric Models Here we present the empirical models that we use to estimate the impact of the project on the outcomes described previously, as well as the results of the statistical analysis. Modeling Traffic Outcomes. Our outcomes using the traffic data are vehicle counts along the roads, and the average vehicle speed during a test drive along the project roads. For the vehicle counts, we use a difference-in-difference model as follows: 𝑇𝑟𝑖𝑡 = 𝛽0 + 𝛽1 𝑉𝑖𝑡 + 𝛽2 𝛿𝑡 + 𝛾(𝛿𝑝𝑜𝑠𝑡 × 𝛿𝑡𝑟𝑒𝑎𝑡 ) + 𝛼𝑖 + 𝜀𝑖𝑡 Where: 𝑇𝑟𝑖𝑡 is daily traffic count on segment i during wave t of the traffic survey 𝑉𝑖𝑡 is a control variable for vehicle type 𝛿𝑡 is a vector of time dummies representing 11 of the 12 survey waves 𝛿𝑝𝑜𝑠𝑡 is a dummy equal to 1 for observations after construction was completed 𝛿𝑡𝑟𝑒𝑎𝑡 is a dummy equal to 1 for a segment in the treatment group 𝛼𝑖 is a segment-level fixed effect 𝜀𝑖𝑡 is a random error term The 𝛽𝑖 are parameters to be estimated, and 𝛾 is estimate of the average treatment effect (ATE) For average speed, we only have data along the treatment roads9. Our model is: 𝑆𝑖𝑡 = 𝛽0 + 𝛽1 𝑇𝑡 + 𝛽2 𝑄𝑡 + 𝛾(𝛿𝑝𝑜𝑠𝑡 ) + 𝛼𝑖 + 𝜀𝑖𝑡 Where: 𝑆𝑖𝑡 is average speed on segment i during wave t of the traffic survey 𝑇𝑡 is a yearly time trend 𝑄𝑡 is a vector of quarterly dummies to reflect seasonality 𝛿𝑝𝑜𝑠𝑡 is a dummy equal to 1 for observations after construction was completed 𝛼𝑖 is a road segment-level fixed effect 𝜀𝑖𝑡 is a random error term The 𝛽𝑖 are parameters to be estimated, and 𝛾 is estimate of the program impact Modeling Community-Level Outcomes. We measure community-level outcomes of interest using the Settlement Information Survey (SIS) dataset. Our modeling approach includes three specifications: difference-in-difference, continuous treatment, and matched difference-indifference. Our empirical model for the difference-in-difference estimation is as follows: 𝑌𝑖𝑡 = 𝛽0 + 𝛽1 𝑋𝑖 + 𝛽2 𝛿2007 + 𝛽3 𝛿2010 + 𝛽4 𝛿2012 + 𝛾(𝛿2012 × 𝛿𝑡𝑟𝑒𝑎𝑡 ) + 𝛼𝑖 + 𝜀𝑖𝑡 9

We note that unlike the other models that we estimate, the impact on average speed is not relative to the comparison roads but instead relative to average speeds on the treatment roads before the project. We do not view this as a major shortcoming, however, as we would not expect significant changes in average vehicle speeds to have occurred for reasons other than roads improvements.

Where: 𝑌𝑖𝑡 is outcome Y for community i at time t 𝑋𝑖 is a vector of community control variables 𝛿2007 , 𝛿2010 , 𝛿2012 are time dummies corresponding to each survey round 𝛿𝑡𝑟𝑒𝑎𝑡 is a dummy equal to 1 for the treatment group 𝛼𝑖 is a community-level fixed effect 𝜀𝑖𝑡 is a random error term The 𝛽𝑖 are parameters to be estimated, and 𝛾 is estimate of the average treatment effect (ATE) The impact of the project is measured using the coefficient on the interaction term 𝛾 between treatment status and time period. Thus, in effect we ask how outcomes differ for the treatment group following the treatment, as compared to outcomes for the treatment group prior to the treatment and the control group both before and after the treatment. Here, the treatment group is defined as all settlements within 30 minutes travel time of one of the improved roads, while the comparison group is defined as all settlements within 30 minutes travel time of a control road. Using the same notation, the continuous treatment model is: 𝑌𝑖𝑡 = 𝛽0 + 𝛽1 𝑋𝑖 + 𝛽2 𝛿2007 + 𝛽3 𝛿2010 + 𝛽4 𝛿2012 + 𝜑(𝛿2012 × 𝐷𝑖 ) + 𝛼𝑖 + 𝜀𝑖𝑡 Where 𝐷𝑗 is the travel time between cluster j and the nearest project road. Here our estimate of impact is given by 𝛽4 + (𝜑 × 𝐷𝑖 ). For an outcome that is increasing in the treatment, we expect 𝛽4 to be greater than zero and 𝜑 to be less than zero, reflecting the fact that the impact is smaller as 𝐷𝑖 increases. The matched difference-in-difference approach uses the difference-in-difference model above but on a restricted sample to reduce the potential for selection bias. An important consideration in PSM is the specification of the first stage regression. There is no clear consensus in the literature on what criteria to use to select the independent variables in the first stage regression. Our approach has been to experiment with a variety of specifications using variables that appear to be potential drivers of selection bias. Our final specification uses the following variables: population, elevation, distance to nearest city, distance to the nearest treatment or comparison road, and dummies reflecting whether agriculture or trade are major sources of employment, the presence of industrial facilities, and recent incidence of natural disasters. Our Community-Level Outcome models make use of the panel nature of the data by using a fixed effects specification. This specification includes a dummy variable for each community, and thus effectively controls for any characteristics of the communities that do not change over time, but that might affect outcomes. Thus, we do not need further controls for any community-level characteristics that remain constant across the three survey rounds, such as elevation, land area, etc. We include a number of control variables to account for factors that could change over time and affect our outcomes. Our dataset contains a large number of potential control variables, which we experimented with in the course of developing our model. Our final selection of

control variables is based on those factors that theory or intuition suggest would be important, as well as those that tended to show up as statistically significant in our various specifications. These are as follows: log of population; incidence of natural disaster; road maintenance; whether or not almost all of the settlement is engaged in agriculture; and whether or not most of the settlement is involved in wholesale or retail trade. In addition, we include dummies for whether the settlement is a major producer of various crops in our price regressions, and a dummy indicating whether there is an industrial facility present in the settlement except where the number of industrial facilities is the dependent variable. Modeling Household-Level Outcomes. As in the previous case, we include difference-indifference, continuous treatment specifications, and matched difference-in-difference specifications. The structure of the dataset has important implications for the modeling approach. For the purposes of the evaluation, the structure of the data is a repeated cross-section, but with the longitudinal aspect of the data imposing additional correlation between outcomes measured at different points in time from the same household. As a result, the appropriate empirical specification is a mixed effects model with a fixed effect for the cluster and a random household effect. The empirical model for the difference-in-difference model is as follows: 𝑌𝑖𝑗𝑡 = 𝛽0 + 𝛽1 𝑋𝑖𝑗 + 𝛽2 𝛿𝑡 + 𝛽3 𝛿𝑝𝑜𝑠𝑡 + 𝛾(𝛿𝑝𝑜𝑠𝑡 × 𝛿𝑡𝑟𝑒𝑎𝑡 ) + 𝛼𝑗 + 𝜏𝑖 + 𝜀𝑖𝑗𝑡 Where: 𝑌𝑖𝑗𝑡 is outcome Y for household i in cluster j at time t 𝑋𝑖𝑗 is a vector of household controls 𝛿𝑡 is a vector of time dummies 𝛿𝑝𝑜𝑠𝑡 is a dummy equal to 1 in the post-treatment period 𝛿𝑡𝑟𝑒𝑎𝑡 is a dummy equal to 1 for the treatment group 𝛼𝑗 is a cluster-level fixed effect 𝜏𝑖 is a household-level random effect 𝜀𝑖𝑗𝑡 is a random error term The 𝛽𝑖 are parameters to be estimated, and 𝛾 is estimate of the average treatment effect (ATE) Note that because the treatment is defined at the cluster level, we do not control for treatment status alone, since this is subsumed in the cluster level fixed effect. Using the same notation as above, the continuous treatment model is: 𝑌𝑖𝑗𝑡 = 𝛽0 + 𝛽1 𝑋𝑖𝑗 + 𝛽2 𝛿𝑡 + 𝛽3 𝛿𝑝𝑜𝑠𝑡 + 𝜑(𝛿𝑝𝑜𝑠𝑡 × 𝐷𝑗 ) + 𝛼𝑗 + 𝜏𝑖 + 𝜀𝑖𝑗𝑡 Where 𝐷𝑗 is the travel time between cluster j and the nearest project road. In this case the impact of the project is given by 𝛽3 + (𝜑 × 𝐷𝑗 ). As above, for an outcome that is increasing in the treatment, we expect 𝛽3 to be greater than zero and 𝜑 to be less than zero, reflecting the fact that the impact is smaller as 𝐷𝑗 increases. Our matched difference-in-difference specification follows the same approach as described above. We match at the community rather than the household level, since households remain in the sample for one year only. Here, the variables in our first-stage regression are:

settlement elevation, settlement population, distance to the nearest treatment or comparison road, and distance to the nearest major city. We opt for a limited set of household-level control variables in our models because of endogeneity concerns. Many of the household characteristics that could explain variation in the levels of our outcome variables could also be affected by the treatment. For example, we would like to control for household wealth, but we expect the roads improvement to increase levels of wealth for households in neighboring areas. Since our data are not panel, controlling for household wealth would eliminate our ability to detect any impact. Thus, our control variables are as follows: household size; age and education of the household head; and a dummy variable indicating whether any member of the household is employed in agriculture. Interpretation of coefficients for community and household level models. For the difference-in-difference and matched difference-in-difference models, the interpretation of the coefficients is straightforward. For both of these models, the value of the coefficient 𝛾 on the interaction term between the treatment dummy and the post-treatment period time dummy is an estimate of the average treatment effect (ATE), the average change in the outcome in question that results from the SJ roads improvement. For the continuous treatment model, the interpretation is more complex, as the magnitude of the impact for a particular community or household depends on its distance from the nearest treatment road. As shown in the methodology section, the model’s estimate of the magnitude of the impact is given by 𝛽 + (𝜑 × 𝐷𝑖 ), where 𝛽 is the coefficient on the dummy variable indicating the post-treatment period, 𝜑 is the coefficient on the interaction between distance from the treatment road, and 𝐷𝑖 is the distance of the particular community or household from the nearest treatment road. The intuition is that 𝛽 reflects the impact of the treatment for a household located immediately along a treatment road, while for every minute of travel time for a community to the nearest treatment road, the impact of the SJ road improvement changes by 𝜑 units. For an outcome that is impacted by the treatment, we thus expect the two coefficients to be of opposite signs. If 𝛽 is positive- meaning the treatment increases the value of the outcome in question- 𝜑 should be negative, since a higher value of 𝐷𝑖 should result in a smaller impact. Conversely, where the treatment causes the value of the outcome to decrease, we would expect the impact to get closer to zero as distance from the road increases, so that 𝜑 will be positive. In our presentation of the results below, for brevity we omit the results for the control variables and include only the coefficients on our impact measures. For the difference-indifference and matched difference-in-difference model, we report 𝛾, the average treatment effect. For the continuous treatment model we report 𝛽 and 𝜑, the coefficients on the treatment time period dummy and the treatment time dummy interacted with distance from the nearest project road. The results include the associated t- or z-statistics calculated from robust standard errors10. RESULTS Traffic Outcomes Our traffic count and speed outcomes are presented in table 5. In both cases, the results are of the expected sign and statistically significant. Following the completion of construction, the average number of vehicles increased by 44.2 vehicles per day, while the average speed 10

As a robustness check, we replicated the main SIS results using standard errors clustered at the village level rather than robust standard errors. In no cases do the reported significance levels change.

recorded along the project roads was higher by 13.6 km/h. This corresponds to a 4.1% increase in daily traffic compared to the mean across the pre-treatment periods for the treatment roads, and a 24.4% increase in average vehicle speed. Table 5: Traffic Impacts

Dependent variable

Coefficient

Average number of vehicles per day

44.20414*** (3.35)

Average vehicle speed

13.55***

(5.72) t-statistics in parentheses *, **, and *** denote statistical significance at 10%, 5%, and 1% respectively

Community-Level Outcomes Our community level outcomes include first transportation-related outcomes, consisting of the travel times to various points of interest reported by our informants, as well as availability of public transportation. A second set of impacts relates to changes in investment and land use as result of improved transportation access. Finally, we consider the impact on local prices of wheat, beef, milk, potatoes, and honey. The estimates of impact on transportation-related variables are shown in Table 6. As we would expect, we see a significant impact of the project in reducing travel times. In particular, travel times to Tbilisi as well as to local markets are reduced substantially; the difference-indifference estimate indicates a reduction in driving to Tbilisi of 39.6 minutes, and a 43.6 minute reduction in driving time to local markets. The continuous treatment results show similar impacts though of somewhat different magnitudes. The continuous treatment model finds that a community located 30 minutes from the treatment road would experience reductions of 83.6 in the travel time to Tbilisi, and 25.9 minutes in the travel time to the local market. We see a smaller reduction in travel time by minibus to the district center. Other travel time variables are mostly statistically insignificant, however. In addition, the availability of public transportation as measured by the frequency of minibuses has not increased as a result of the road improvements.

Table 6: Transportation-Related Impacts

Dependent variable

Difference-in-difference Matched diff.-in-diff.

Continuous Treatment: PostTreatment Dummy

Continuous Treatment: (Distance * PostTreatment Dummy)

-85.53***

0.0636***

Travel Time to Tbilisi by Car

-39.62*** (-7.13)

-33.64*** (-5.63)

-13.3

-3.64

Travel Time to Tbilisi by Minibus

-24.11**

-13.50*

-29.89***

0.0385

(-3.09)

(-1.83)

(-6.02)

-1.7

11.55

11.73*

-21.76***

-0.0151

-1.86

-1.9

(-3.49)

(-1.02)

-8.591

-8.736

-9.97***

0.0105

(-0.90)

(-0.92)

-2.28

-0.79

-4.216

-2.136

12.76***

0.0171*

(-1.80)

(-0.89)

-5.63

-2.28

-4.590*

-4.516*

-4.316**

0.00486

(-2.35)

(-2.13)

(-3.11)

-0.89

-43.61***

-39.45***

-29.09***

0.107***

(-4.42)

(-3.95)

(-4.83)

-4.82

-74.87***

-59.24***

-61.89***

0.217***

(-5.65)

(-4.14)

(-6.17)

-5.36

0.517

9.158

-23.28

0.0833

-0.04

-0.85

(-1.15)

-1.15

0.476

0.777

2.484

-0.00629

-0.29

-0.43

-1.65

(-1.17)

9.455

9.587

8.085*

-0.0225

-1.06 2.126

-1.08 2.412

-2.16 3.34

-1.63 -0.0139

-0.69

-0.66

-1.54

(-1.48)

Travel Time to Region's Major Market by Car Travel Time to Regional Center by Minibus Travel Time to District's Major Market by Car Travel Time to District Center by Minibus Travel Time to by Car to the Basic Market at which the village is selling its products Travel Time by Hired Automobile to the Basic Market at which the village is selling its products Travel Time by Bus to the Basic Market at which the village is selling its products Periodicity of Minibus to District Center Periodicity of Minibus to Regional Center Periodicity of Minibus to Tbilisi t-statistics in parentheses

*, **, and *** denote statistical significance at 10%, 5%, and 1% respectively

Impacts of the project on outcomes related to investment and land use at the community level are shown in table 7. We see a substantial impact of the program on the number of industrial facilities within the settlement, with an increase due to the program of 0.51 facilities and a slightly weaker finding using the matched difference-in-difference model. This finding is confirmed by the results using continuous treatment, which finds that a community located 30 minutes from the nearest road would have an expected increase of 0.65 facilities. Given the

importance of this finding, we undertake some further related analysis and discussion at the end of this section. In terms of the other land use variables, we find no significant impact of the project. Table 7: Investment and Land Use Impacts

Dependent variable Number of Industrial Facilities within Settlement Change in most common usage of land Wheat planting common Corn planting common Grape planting common Vegetable planting common

Continuous Treatment: (Distance * Post-Treatment Dummy)

Difference-indifference

Matched difference-indifference

Continuous Treatment: Post-Treatment Dummy

0.512***

0.455***

0.716***

-0.00230***

(3.69)

(3.09)

(7.97)

(-6.08)

0.0690

0.0537

-0.245***

-.000188

(1.32)

(0.96)

(-8.12)

(-1.27)

0.0263

0.0261

-0.0239

-.0000138

(0.88)

(1.17)

(-1.09)

(-0.20)

0.0496

0.0564

0.0432

-0.000108

(1.16)

(1.21)

(1.64)

(-1.05)

0.0333

0.0424

-0.0251

0.0000364

(0.87)

(1.04)

(-1.17)

(0.33)

0.0879

0.0773

0.0548*

-0.000171

(1.72) (1.44) (1.69) (-1.20) t-statistics in parentheses *, **, and *** denote statistical significance at 10%, 5%, and 1% respectively Binary dependent variables use linear probability models; count variables use negative binomial regression

Finally, we consider impacts on commodity prices in the settlement. As described in the methodology section, the impact of the road improvement on the price of a commodity may be different in communities where that commodity is produced as compared to those where it is not. Thus, we include both the treatment variable and the interaction of the treatment variable with a dummy indicating whether or not production of the commodity is common in the community. The interpretation is that the coefficient on the treatment variable reflects the impact in communities where the commodity is not produced, while the sum of the two coefficients shows the impact in communities where the commodity is produced. Note that our data on production do not include potatoes or honey specifically, so we proxy these using vegetable production. The results for commodity prices are shown in Table 8. All three models indicate statistically insignificant results for wheat. The price of beef shows a statistically significant increase in all three models, but with a large and negative coefficient on the interaction term for both difference-in-difference models. Thus, in areas where livestock production is common the road improvement tends to induce a fall in the price of beef. The results for milk show the same pattern, though they are mostly not statistically significant. The price of potatoes appears to fall

slightly, with no difference in vegetable producing areas. Finally we see the largest impact on the price of honey, with our difference-in-difference estimate illustrating a 1.62 Lari/kg decrease. The data suggest that this decrease is smaller in areas where honey production is common, though the statistical significance is not strong. The continuous treatment model does not confirm this finding. Table 8: Impacts on Commodity Prices

Dependent variable Price of wheat

Difference-indifference

Matched difference-indifference

Continuous Treatment: Post-Treatment Dummy

Continuous Treatment: (Distance * PostTreatment Dummy)

0.0362

0.0375

0.0139

-0.0000258

(1.11)

(1.91)

(1.91)

-0.012

-0.0198

-0.000261**

(-0.31)

(-0.48)

(-2.73)

Price of beef

0.812*

0.846**

3.184***

(2.51)

(2.31)

(25.84)

Price of beef * livestock producing area

-0.896*

-0.905**

-0.000257

(-2.45)

(-2.13)

(-0.42)

0.106

0.104

0.190*** (8.29)

Price of wheat * wheat producing area

Price of milk Price of milk * livestock producing area Price of potatoes Price of potatoes * vegetable producing area Price of honey Price of honey * vegetable producing area

(-0.44)

-0.000845 (-1.30)

0.0000371

(1.46)

(1.35)

-0.134*

-0.155**

-0.00016

(0.2)

(-2.03)

(-2.11)

(-0.65)

-0.0623

-0.0956**

-0.142***

(-1.61)

(-2.44)

(-10.47)

0.038

0.0677

0.000202*

(0.89)

(1.52)

(2.14)

-1.618***

-1.675***

2.388***

(-4.98)

(-4.70)

(10.96)

0.814

1.257**

0.00232

(1.75)

(2.49)

(1.48)

0.000206** -3.21

0.00108 (0.68)

t-statistics in parentheses *, **, and *** denote statistical significance at 10%, 5%, and 1% respectively

Household-Level Outcomes The results of the household-level analysis are presented in Table 9. Overall, we do not obtain strong findings at the household level. There are no household level outcomes for which all three models indicate impact. Our models yield some significant regression coefficients for household income and consumption, but these are not of the expected sign and likely reflect data problems related to measurement or endogeneity rather than any true relationship. We do obtain significant findings on the index of asset ownership from both the difference-in-difference and continuous treatment models, but this is not confirmed by the matched difference-in-difference

model. Given the apparent concerns with the household level data that the other findings suggest, we do not draw conclusions from this result. Table 9: Household Level Impacts

Dependent variable Transport expenditure Income per capita Consumption per capita Asset index % employed Probability of health care utilization Probability of all children in school

Matched difference-indifference

Continuous Treatment: PostTreatment Dummy

Continuous Treatment: (Distance * Post-Treatment Dummy)

2.499945

8.601852

-1.989435

-.0324185*

(0.52)

(0.80)

(0.38)

(1.94)

-24.20239**

-29.77346**

-14.27958*

.0046024

(-2.48)

(-2.30)

(-1.74)

(0.16)

-12.07077**

-9.459688

-13.86415***

.0060832

(-1.92)

(-0.99)

(2.85)

(0.35)

.0088079**

.0060364

.0057474*

-.0000349***

(1.90)

(0.79)

(2.07)

(2.92)

.0375272**

.0287014

-.0116898

.0000771*

(1.94)

(0.78)

(-0.98)

(1.78)

.0000979

.0013508

-.0000001

.0000000

(0.69)

(1.19)

(-0.08)

(0.04)

-.0000739

-.0000619

.0001483

.0000001

(0.92)

(-1.05)

Difference-indifference

(1.51) (0.99) z-statistics calculated from robust standard errors in parentheses *, **, and *** denote statistical significance at 10%, 5%, and 1% respectively

Tests of Assumptions and Robustness Checks. Test of “Common Trends” Assumption. An important characteristic of difference-indifference modeling is that it implies what has been termed the “common trends” or “parallel paths” assumption. That is, the validity of the comparison between treatment and comparison groups requires that in the absence of the project, outcomes would have evolved in the same way. If in fact the treatment and comparison groups were on different trajectories, difference-indifference estimates of treatment effects will be biased, reflecting not only the impact of the project but also the underlying difference in trends. Our data allow us to explore this assumption for our key findings described in the previous section. Where multiple rounds of pre-treatment data are available, the common trends assumption can be tested by comparing trends in outcomes prior to treatment (Angrist and Krueger 1999). In our case, we can use the fact that we have two pre-treatment rounds to undertake some additional investigation of the common trends assumption for the key findings using the SIS data. To do so, we implement our difference-in-difference and matched difference-indifference models for our main findings using only the two rounds of pre-treatment data. Our placebo “treatment effect” is the coefficient on the interaction term between the treatment area dummy, and the second round time dummy. As there was no treatment that took place over this

time period, this coefficient gives us a measure of whether the trend (conditional on the control variables) in the two pre-treatment periods differed between the treatment and control groups. To the extent that the common trends assumption was valid in the pre-treatment period, we expect the coefficients reflecting the measurement of the treatment to be statistically insignificant, or at minimum not of the same sign as the estimates of impact. Where the results of this exercise would cast our findings into doubt is if we were to observe a trend in the pretreatment period that mirrors the direction of our estimate of impact. In that case, we would suspect that our estimate of impact might reflect a continuation of the pre-treatment trend in that outcome, rather than the causal impact of the program. The results of this exercise are presented in Table 10, and tend to support the validity of the main findings. In no cases do we find pre-treatment trends running in the same direction as the evaluation findings. The coefficients on the transportation related impacts are insignificant, as expected. We do find significant coefficients on the number of industrial facilities variable. However, these are of opposite sign from our evaluation findings. The interpretation is that in the period from 2007-2010, the number of industrial facilities was increasing faster in the comparison areas. Thus, our finding of project impact may in fact understate the true impact. Similarly the coefficients on milk prices are significant, but do not mirror the evaluation findings. Table 10: Test of Common Trends Assumption in Pre-Treatment Periods

Dependent variable Travel time to Tbilisi by car Travel time to Tbilisi by minibus

Travel time by car to local market Travel time by hired car to local market Number of industrial facilities within settlement Price of wheat Price of wheat * wheat producing area

Price of beef Price of beef * beef producing area

Price of milk

Difference-indifference n/a

Matched diff.-in-diff. n/a

-8.897

0.0739

(-1.03)

(0.01)

23.441

25.248

(1.02)

(1.07)

29.543

38.639

(1.28)

(1.62)

-0.447**

-0.382*

(-2.23)

(-1.86)

0.524*

0.0387

(1.76)

(1.24)

-0.105**

-0.107**

(-2.44)

(-2.50)

0.625

0.602

(1.52)

(1.46)

-0.799

-0.560

(-1.78)

(-1.27)

-0.337***

-0.334**

(-2.86)

(-2.40)

Price of milk * milk producing area

0.095 (0.81)

(0.59)

Price of potatoes

-0.007

0.0162

(-0.12)

(0.25)

-0.024

-0.0404

(-0.40)

(-0.60)

-0.378

-0.429

(-0.89)

(-0.91)

-0.535

-0.204

(-0.86)

(-0.32)

Price of potatoes * potato producing area Price of honey Price of honey * honey producing area

0.0787

t-statistics calculated from robust standard errors in parentheses *, **, and *** denote statistical significance at 10%, 5%, and 1% respectively Note that travel time to Tbilisi by car could not be estimated because of low variation over time

Range of Robustness Checks. In addition to the results using the modeling specifications presented above, we attempted a wide range of alternative specifications in order to explore whether our findings were sensitive to the particular assumptions that we used. These focused on the Community Level Outcomes, since our above analysis has obtained the most interesting results using these data. For these, we estimated an additional 27 plausible specifications where we vary the statistical assumptions that we have employed, including the following:  15- and 90-minute travel time cutoffs: as discussed in the text, our difference-indifference analyses include the communities in our dataset that were located within 30 minutes travel time of either one of the improved roads, or a comparison road. We also obtained difference-in-difference estimates using 15- and 90-minute travel time cutoffs respectively, and using random effects and clustered OLS in addition to the fixed effects models presented in the text. The findings from our preferred specifications for transportation-related impacts and investment and land use are highly robust to varying the cutoff used to define treatment and control groups. For commodity prices, we tend to obtain somewhat weaker results, particularly using the 15-minute cutoff.  Propensity score specification: here we investigate the robustness of our matched difference-in-difference results using alternative specifications of the propensity score. We thus replicate our matched difference-in-difference model using 13 alternative propensity score specifications. Findings for the transportation-related impacts and commodity price effects are highly robust to changes in the propensity score. Conversely, our finding on the impact on the number of industrial facilities is more sensitive to how the propensity score is specified. When distance to Tbilisi and distance of the settlement to the nearest road are included in the propensity score specification, the coefficient on industrial facilities tends to become insignificant, although we obtain significant results when we use alternatives such as distance to nearest city or distance to the nearest primary road. Overall, our robustness checks give us confidence in the main findings that emerge from our preferred specifications, though less so with respect to the finding that the number of industrial facilities increased in the areas of the SJ road improvements. Since this is a particularly important finding, we provide a detailed examination of this result in light of the above robustness checks as well as some further analysis.

Further Discussion of Impact on Investment. Our preferred specification indicates a large and statistically significant impact of the SJ road improvement on investment: the difference-indifference coefficient of 0.512 corresponds to a 30.3% increase in the number of industrial facilities, with a p-value of less than .01. The continuous treatment and matched difference-indifference models corroborate this result, although the matched model shows slightly weaker results. Conversely, our robustness checks indicate that alternative specifications of the propensity score lead to statistically insignificant results in some cases. In addition, while our test of the common trends assumption indicated a pre-treatment trend in the opposite direction, the fact that we did find a significant difference between treatment and control groups suggest that there are factors that drive investment that we have not been able to observe and control for. To further investigate this relationship, we performed some exploratory spatial regression analysis on the number of industrial facilities outcome. Spatial regression is a technique that is used for controlling and estimating for spatial dependencies, or spatial autocorrelation, in the data. Thus, in our case if the number of industrial facilities in a community is determined in part by the number of industrial facilities in nearby communities, our spatial model is an approach to controlling for this tendency. The results tend to support our main result. Using the inverse of distance squared as a spatial weight yields significant coefficients for both difference-indifference and matched difference-in-difference models, suggesting that when we control for spatial patterns related to investment our finding of impact due to the project holds. Further analysis along these lines could potentially provide additional insights into the robustness of the investment finding. Our conclusion is that our findings likely reflect a strong impact of the project on investment, but that some of the impact may be due to selection bias. While the evidence of impact is convincing, we cannot completely rule out the possibility that other factors unrelated to the project may have contributed to the relative increase in investment in SJ project area as well. As a result, we take our more conservative matched difference-in-difference coefficient of 0.455, corresponding to a 26.9% increase, as our estimate of the impact of the SJ road improvement.

CONCLUSIONS In this section, we summarize the findings of the previous section and discuss interpretation and implications. Our discussion of the size of the effects is based on the difference-in-difference or matched difference-in-difference results. Thus, these results can be interpreted as our estimate of the causal impact of the program on a household or community within 30 minutes travel time of one of the improved roads, relative to a household or community within 30 minutes travel time of a comparison road. Key Results The roads improvements successfully improved travel conditions along the project roads. Our evidence shows that despite the concerns about construction and maintenance, the project has had a substantial impact on traffic conditions. Following the completion of construction, the volume of traffic on project roads increased by an average of 44.2 vehicles per day relative to the comparison roads, an increase of 4.2% over pre-treatment levels. The average speed along the roads increased by 13.6 km/h, a 24.4% increase. In addition, there were large and significant reductions in self-reported travel times both to Tbilisi and to the local market where farmers sell their products. We do not find a corresponding increase in either the availability of public transport, nor in travel times to regional or district centers, with the exception of a small decrease in travel time to the district center by minibus. Our data show a significant impact of the project on increasing industrial investment in communities near the improved roads. At the community level, we observe an increase in the number of industrial facilities (i.e. canneries, factories, agricultural processing facilities, and similar enterprises) in the settlement, with our matched difference-in-difference model showing an increase of 0.46 facilities per community after the project. Given that the mean at baseline is 1.69 facilities, this corresponds to a 26.9% increase. As discussed in the previous section, some caution must be exercised in interpreting this finding because of the possibility of selection bias. We do not find evidence of impact on household-level outcomes such as income, consumption, or utilization of health and education services. In no cases do our models show consistent findings for any of the household-level variables. One possible explanation is that the SJ roads improvements did not impact these outcomes. However, in light of the significant findings we observe at the community level, a more likely explanation is that we are unable to detect impacts due to limitations of the household dataset, which is not ideally structured for the evaluation. Typically, evaluations make use of panel data where the same households are interviewed both before and after the project. This approach allows for the analysis to control for household level characteristics. Since the IHHS dataset used in the evaluation re-samples households after four quarters, our analysis must compare a set of household prior to the intervention to a different set of households after the intervention. The resulting high degree of variability in income and consumption across households could potentially mask any impact of the program11. Moreover, the final round of IHHS data that was available for the evaluation was collected less than one year after the completion of construction.

11

It is important to bear in mind that this limitation of the data is not a shortcoming of the IHHS dataset itself, which is collected for a range of social science research purposes. Rather, it is a difficulty with using the IHHS dataset for the purposes of the SJ roads evaluation.

Commodity Prices. As discussed previously, the relationship between prices and transport costs such as those induced by a road improvement can be complex. Our results bear out these complexities, as we observe statistically significant findings in a number of cases, but with different tendencies for different products. These impacts differ depending on whether or not the product in question is produced in the local area. The strongest findings are for honey and beef. The project tended to cause honey prices to fall overall, but increase in areas where honey is produced. For beef, we observe the opposite tendency: improving the road tended to lead to a decrease in prices where beef is produced, and a rise in prices elsewhere. Other commodities show weaker results but there are still some discernible tendencies. Milk shows a similar tendency to beef, while the project tended to increase wheat prices while decreasing potato prices. For both wheat and potatoes, the effect tended to be smaller in areas where these products are widely produced, though not statistically significantly so. A full explanation of these results would require further data on market and production conditions for each product. However, we can offer some potential explanations by discussing the findings that we observe in light of the theoretical issues, and how these might reflect different market conditions for the different commodities. Honey (large decrease in distant markets, smaller increase in local markets): The price response for honey provides an illustration of the straightforward impact that what we expect in the absence of other mitigating factors. The project reduces transportation costs, which results in a price decrease in distant markets for which transportation costs are greater. These lower transportation costs will induce sellers to sell a greater share of their output in more distant markets, which on local markets will have the effect of reducing supply and thus increasing prices on local markets. While we would expect prices in local markets to remain lower than in more distant markets, the difference between the two should be reduced as result of the project. Beef and milk (large increase in distant markets, small decrease in local markets): The opposite tendency that we observe for beef and milk has two potential explanations. First, there may be imperfect competition in markets distant from cattle producing areas. If this is the case, the “supply effect” whereby cost decreases for producers are passed on to consumers is muted, so that the “demand effect” driving up prices as a result of transportation costs for buyers would dominate. If local markets are more competitive, demand effects are counteracted by supply effects and thus we would expect the pattern that we observe in the data. Another explanation could be that buyers face substantially higher transportation costs in distant markets as compared to local ones, possibly due to the potential for spoilage of these commodities. Since milk and meat sold on distant markets must be stored in refrigerated facilities, fewer retail outlets are able to sell them relative to commodities that are easier to store. As a result, consumers tend to pay higher transportation costs as a result of the need to travel further to these outlets, driving up prices in response to the road improvement. Meanwhile, in cattle producing areas consumers may be able to purchase milk and meat from local producers before it needs to be refrigerated. Thus, transportation costs could be lower in this case and the upward pressure on the price is reduced. Potatoes (small price decrease in distant markets, smaller decrease on local markets): The price response for potatoes also conforms to simple economic theory, similar to the case with honey. The price falls in distant markets to reflect lower costs associated with transporting potatoes there, as was the case with honey. On local markets, the price decrease is smaller since transportation costs are smaller. Since the magnitude of the price change is smaller than was the

case with honey, we do not observe sellers switching to more distant markets to the same extent and thus there is less upward pressure on prices locally. Wheat (small price increase in distant markets, smaller increase on local markets): The fact that wheat prices move in the opposite direction from potato prices could be explained by two possible factors. First, as we discussed in the case of beef and milk, if sellers have market power in distant markets this would mute downward pressure on prices. It could also be the case the incidence of transportation costs could fall more heavily on buyers in the case of wheat as compared to potatoes. This could be the case if, for example, processors such as bakeries purchase wheat from wholesale markets and then transport it to processing facilities, while purchasers of potatoes tend to be consumers who pay lower transportation costs. Limitations In interpreting these results, some important limitations to the analysis must be borne in mind. Roads projects present inherent difficulties from the standpoint of evaluation, and the present evaluation is to some extent inhibited by these constraints. In this section, we review these difficulties and discuss their implications for the results. First, selection bias as a result of endogenous placement is a potential concern. Selection bias is a concern in any evaluation that makes use of control or comparison group to represent the counterfactual. If the treatment and comparison groups are not identical, then observed differences in outcomes between the two groups may be due to factors other than the impact of the project. In the context of roads evaluations, this is a particular concern (van de Walle 2009), since roads improvements are typically targeted to specific locations for particular reasons. The most apparent potential difficulty here is that the SJ roads lead to the Turkish and Armenian borders, whereas the comparison roads do not. As a result, if access to the border would have caused outcomes along the project roads to improve faster than outcomes along the comparison roads, then we may be misattributing the changes we observe to the impact of the program. It is important to bear in mind however that the Kartsakhi checkpoint on the Turkish border was not open during the evaluation period, and thus outcomes would not have been affected by access to the Turkish border. As described in the text, our comparison roads were carefully selected using a statistical matching procedure to ensure the most accurate comparison possible. While this approach reduces the potential for selection bias, we cannot completely rule out the possibility that this bias may be driving some of our results. A second limitation is that the timeframe over which the evaluation has taken place may be insufficient for the full impacts to have occurred. The final round of data collection occurred just 20 months after the completion of construction in the case of the SIS settlement survey, and less than one year after construction in the case of the IHHS dataset. However, the literature suggests that the full impacts of roads improvements may take several years to unfold (e.g. Mu and Van de Walle (2009). In addition, while the border crossing with Turkey was not open during the evaluation period, it is planned to open in the future. The SJ road could then become an important export route in the future as producers take advantage of the improved road to access the border. Thus, our evaluation is likely capturing only a limited portion of the full impact of the project on the populations that we have studied12. Third, there are likely to be broader impacts of the project that the evaluation is not able to capture. The evaluation has focused on beneficiaries living adjacent to the project roads, but 12

Conversely, impacts may also decrease over a longer timeframe in the event that the roads are not properly maintained, or further construction defects emerge.

the project causes transportation costs fall for all users of the roads. This includes producers who may be located some distance from the project roads, but who use these roads to transport their goods. This could include a substantial amount of goods shipped along the road. The resulting benefits will not be captured by the evaluation.

REFERENCES Ahmed, R., & Hossain, M. (1990). Developmental impact of rural infrastructure in Bangladesh. Washington, D.C.: International Food Policy Research Institute. Angrist, J. & Krueger, A. (1999). “Empirical strategies in labor economics,” in Handbook of Labor Economics, ed. by O. C. Ashenfelter, and D. Card, vol. 3A, pp. 1277-1366, Elsevier Press Bakht, Z. (2000). “Poverty Impact of Rural Roads and Markets Improvement & Maintenance Project of Bangladesh.” Paper presented at the World Bank South Asia Poverty Monitoring and Evaluation Workshop, June 8-10, 2000, India Habitat Centre, New Delhi. Bell, C., & van Dillen, S. (2014). How Does India’s Rural Roads Program Affect the Grassroots? Findings from a Survey in Upland Orissa. Land Economics,90(2), 372-394. Binswanger, H. P., Khandker, S. R., & Rosenzweig, M. R. (1993). How infrastructure and financial institutions affect agricultural output and investment in India. Journal of development Economics, 41(2), 337-366. Care International (2010), “Care Caucasus USPV Survey”, 16 June 2010. [www.carecaucasus.org.ge/en/main.php?id=1276500236]. Casaburi, L., Glennerster, R., & Suri, T. (2013). Rural Roads and Intermediated Trade: Regression Discontinuity Evidence from Sierra Leone. Available at SSRN 2161643. Bell, C., & Van Dillen, S. (2012). How does India's rural roads program affect the grassroots? findings from a survey in Orissa. Findings from a Survey in Orissa (August 1, 2012). World Bank Policy Research Working Paper, (6167). Corral, L., & Reardon, T. (2001). Rural nonfarm incomes in Nicaragua. World Development, 29(3), 427-442. Cuánto, I. (2000). “Perú: Informe Final de Evaluación del Proyecto de Caminos Rurales.” Reporte preparado para la Dirección de Caminos Rurales. Lima: Ministerio de Transporte, Comunicaciones, Vivienda y Construcción. Cuong, N. V. (2011). Estimation of the impact of rural roads on household welfare in Viet Nam. Asia-Pacific Development Journal, 18(2), 105-135. Datta, S. (2012). The Impact of improved highways on Indian firms. Journal of Development Economics, 99(1), September 2012: 46-57. de Janvry, A., & Sadoulet, E. (2001). Income strategies among rural households in Mexico: the role of off-farm activities in poverty reduction. World Development, 29(3), 1043-1056.

Dercon, Stefan, Daniel Gilligan, John Hoddinott, and Tassew Woldehanna. 2006. “The Impact of Roads and Agricultural Extension on Consumption Growth and Poverty in Fifteen Ethiopian Villages.” CSAE WPS 2007-01, University of Oxford, UK Dercon, S. and J. Hoddinott. 2005. “Livelihoods, growth, and links to market towns in 15 Ethiopian villages.” FCND Discussion Paper 194, International Food Policy Research Institute, Washington DC. Dercon, S., Gilligan, D. O., Hoddinott, J., & Woldehanna, T. (2009). The impact of agricultural extension and roads on poverty and consumption growth in fifteen Ethiopian villages. American Journal of Agricultural Economics, 91(4), 1007-1021. Escobal, J and Ponce, C. (2004). “The Benefits of Rural Roads: Enhancing Income Opportunities for the Rural Poor.” GRADE Working Paper 40, Lima Peru. Fan, S., Hazell, P. & Thorat, S. (2000). “Government spending, agricultural growth and poverty in rural India.” American Journal of Agricultural Economics 82(4): 1038-1051. Filmer, D. & Pritchett, L. (2001) Estimating Wealth Effects Without Expenditure Data—Or Tears: An Application To Educational Enrollments In States Of India. Demography, 38(1), 115132. Gachassin, Marie, Boris Najman, and Gaël Raballand. "The impact of roads on poverty reduction: a case study of Cameroon." World Bank Policy Research Working Paper Series, Vol (2010). Gannon, C., & Liu, Z. (1997). Poverty and Transport. Washington, D.C.: The World Bank. INU/TWU Series Transport Publications. TWU-30. General Audit Office (2012), Millennium Challenge Corporation. Georgia and Benin Transportation Infrastructure Projects Varied in Quality and May Not Be Sustainable, Report to Congressional Committees, GAO-12-630, June 2012. [http://www.gao.gov/assets/600/591956.pdf] Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. (2011). Impact evaluation in practice. World Bank Publications. Gollin, D., & Rogerson, R. (2010). Agriculture, roads, and economic development in Uganda (No. w15863). National Bureau of Economic Research. Hansen, H., Andersen, O. W., & White, H. (2011). Impact evaluation of infrastructure interventions. Journal of Development Effectiveness, 3(1), 1–8. doi:10.1080/19439342.2011.547659 International Fund for Agricultural Development (2012), Report on Rural Poverty in Georgia, [http://www.ruralpovertyportal.org/web/guest/country/home/tags/georgia].

International Organisation for Migration (2010), “Regional Labour Market Survey in SamtskheJavakheti”, Tblisi: Czech Republic Development Cooperation. [http://jcp.ge/iom/pdf/engLMS.pdf] Jacoby, H. C. (2000). Access to markets and the benefits of rural roads. Economic Journal, 110(465), 713-737. Jalan, J., & Ravallion, M. (1998). Are there dynamic gains from a poor-area development program?. Journal of public economics, 67(1), 65-85. Khandker, S., Bakht, Z. & Koolwal, G. (2009). “The Poverty Impact of Rural Roads: Evidence from Bangladesh.” Economic Development and Cultural Change 57(4) pp. 685-722 Khandker, S. and Koolwal, G. (2011) "Estimating the long-term impacts of rural roads : a dynamic panel approach," Policy Research Working Paper Series 5867, The World Bank. Lanjouw, P., Quizon, J., & Sparrow, R. (2001). “Non-agricultural earnings in peri-urban areas of Tanzania: evidence from household survey data” Food Policy, 26, 385-403. Levy, H. (1996). Morocco: Socioeconomic Influence of Rural Roads. Impact Evaluation Report 15808, Operations Evaluation Department, World Bank, Washington, DC. Lucas, K., Davis, T., & Rikard, K. (1996). Agriculture transport assistance program: impact study. Dar es Salaam: Project Number 621-0166. USAID/Tanzania. MCA-Georgia (2009), Georgia Program Monitoring and Evaluation Plan, December 2009 [http://www.mcc.gov/documents/reports/me_plan_-_Georgia2.pdf]. Minten, B. (1999). Infrastructure, Market Access, and Agricultural Prices: Evidence from Madagascar. International Food Policy Research Institute 2033 K Street, N.W.Washington, D.C. 20006 U.S.A. Montgomery, H., & Weiss, J. (2011). Can Commercially-oriented Microfinance Help Meet the Millennium Development Goals? Evidence from Pakistan. World Development, 39(1), 87–109. Mu, R., & Van de Walle, D. (2011). Rural roads and local market development in Vietnam. The Journal of Development Studies, 47(5), 709-734. Nangia, R. Evaluations Matters: From the Director’s iPad. eVALUatiOn Matters, 3, 2013. Nkonya, E., Phillip, D., Mogues, T., Pender, J., & Kato, E. (2012). Impacts of Communitydriven Development Programs on Income and Asset Acquisition in Africa: The Case of Nigeria. World Development, 40(9), 1824–1838. Retrieved from http://www.sciencedirect.com/science/article/pii/S0305750X12000988

Porter, G. (2002). Living in a walking world: rural mobility and social equity issues in SubSaharan Africa. World Development, 30(2): 285-300. Rand, J. (2011). Evaluating the employment-generating impact of rural roads in Nicaragua. Journal of Development Effectiveness 3(1): 28-43. Ravallion, M. (2007). Evaluating anti-poverty programs. Handbook of development economics, 4, 3787-3846. Ravallion, M. & Chen, S. (2005), Hidden Impact: Household Saving in Response to a Poor Area Development Project, Journal of Public Economics, 89: 2183-2204. Shrestha, S. A. (2012). Access to the North-South Roads and Farm Profits in Rural Nepal. Working Paper, National University of Singapore. Smith, D., Gordon, A., Meadows, K., & Zwick, K. (2001). Livelihood diversification in Uganda: patterns and determinants of change across two rural districts. Food Policy, 26, 421.435. Smith, L. C., Khan, F., Frankenberger, T. R., & Wadud, a. K. M. A. (2013). Admissible Evidence in the Court of Development Evaluation? The Impact of CARE’s SHOUHARDO Project on Child Stunting in Bangladesh. World Development, 41, 196–216. doi:10.1016/j.worlddev.2012.06.018 Storeygard, A. (2013). Farther on down the road: transport costs, trade and urban growth in subSaharan Africa. World Bank Policy Research Working Paper, (6444). Torero, Maximo (2009) Impact Evaluation Design for MCC Connectivity Interventions in El Salvador, Report Prepared for Social Impact Van de Walle, D. (2009), Impact evaluation of rural road projects, Journal of Development Effectiveness, 1(1), 15-36. von Thünen, J. H. (1966). Isolated state: an English edition of Der isolierte Staat. Pergamon Press. Wheatley, J. (2004), Obstacles Impeding the Regional Integration of the Javakheti Region to Georgia, ECMI Working Paper #22, September 2004. World Bank (2009) World Development Report 2009: Reshaping Economic Geography, Washington DC: World Bank Zeller, M., Sharma, M., Henry, C., & Lapenu, C. (2006). An operational method for assessing the poverty outreach performance of development policies and projects: Results of case studies in Africa, Asia, and Latin America. World Development, 34(3), 446–464.