Professors A. Giraldo & M. Castiglioni How to measure the impact of LD Improving widows’ inclusion at the community level (IWIC) Caterina Dadá, Janine Gölz, Marc Tevini
28/12/2015
1. Description of the problem In the patriarchal Indian society, widows are deeply discriminated. Since they are considered the cause of their husbands’ death, most widows are disowned by their families and are forced to abandon their properties and families, thus suffering what UN defines a “social death” (UN Women2000, 2001, p. 6). More specifically, widows’ life in India is marked by a high exposition to violence, exclusion from the community, lack of access to basic social service and a deep lack of self esteem. Moreover, because of the above mentioned discrimination attached to widows and of the low value traditionally attached to women’s knowledge and skills, widows are excluded from the legal labour market. They can find only degrading and underpaid jobs, which in turn furtherly worsen their selfesteem and their status in the community. Widows’ conditions are even more extreme in rural area, where the social norms are very tight and the awareness of widows is lower. In the rural village of Gangadevipally, in the state of Telengana, 35% of widows (Sopar Balavikasa, 2015) reported having seriously considered suicide due to being single, poor, helpless and socially humiliated. The gravity of widows situations urge a local action to be taken, that allows to challenge the social norms at the basis of the discrimination in an innovative and effective way.
2. The intervention Numerous programs and intervention have been implemented to improve living conditions of widows but government interventions has failed up to reach the majority of widows. Indira Gandhi pension schemes, for example, was introduced for widows living below the poverty line, but only 0.1% of widows are actually benefiting of the pension (Indian Census and the National Social Assistance Programme). Moreover, most projects delivered both by the government or by international NGOs, do not tackle the roots of the problem, they merely try to reduce the negative consequences of discrimination. Four years ago, in the area of Gangadevipally, Balavikasa association involved widows in a project meant to empower women in the community through Self Help Groups. The project succeeded in enhancing widows’ selfesteem and participation in community life. Nonetheless, the relations between widows and men are still few and degrading and this concur to keep widows excluded from the legal labour market. For this reason the present project has specifically been designed to enhance widows’ integration in the labour market through activities that require an ongoing active collaboration between widows and men. The cooperation between the two groups will be the base for the participants to challenge those societal norms that prevent a full rehabilitation of widows in the community. The above mentioned objective will be reached through an organic farm that will be established and managed by both widows and farmers. The organic farming has a key role in the project since it allows to arise the interest and participation of the farmers, which are having troubles with the traditional intensive farming; it has a direct beneficial impact on the whole community and, especially, it allows to gather and enhance the skills and knowledge already possessed by both men and widows. 1
The action consists of three main activities: training course and establishment of an organic farm, in order to enhance participants knowledge and synergy; daily agricultural activities in the farm, in order to provide the participants with a decent and upgrading employment and make widows establish cooperative relations with men; the sale of the agricultural products, in order for the participants to have a monthly revenue and increase the contacts of widows with the community. For a more detailed insight of the project, it is possible to refer to the following SWOTanalysis and the result chain in annexe. Through the implementation of these activities we intend to improve the relations between men and widows and to increase the integration of widows in the labour market, which correspond to an improvement of the role of widows in the community of Gangadevipally. On the level of a long term impact, this will contribute to the challenge of the social norms at the base of widows discrimination and to their inclusion in society. Table 1. SWOT analysis of the intervention
STRENGTHS ● ● ●
The widows that were already empowered by previous projects act as a bridge to attract the new widows and the farmers The widows are involved in an profitable and innovative activity The organic farming is a special kind of activity that requires both thee skill f men and women
WEAKNESSES ● ● ● ●
●
OPPORTUNITIES ● ● ● ● ● ● ●
● ●
providing knowledge to build up on in the future Improvement of social inclusion of the widows in the community Improvement of the the economic independence of widows Enhancement of the widows’ selfesteem The intervention is easily transferable to other communities in India Successful implementation of the project brings positive social change to the entire community Economic integration of widows and their children makes the community more competitive because they have more human capital available than other regions Widows are able to provide a better education to their children impact also on future generations can enhance cooperation between widows and men in the future and also transferred better relation between men and women in general
The success of the activity relies also on meteorological conditions The selection of the participants is a very delicate phase and could arise conflicts within the community The low selfperception of the widows prevents them from involving in the project widows and male farmers selected start from different basic knowledge. Men already have practical experiences in farming whereas widows do not have this experience. This could lead to a hierarchy not desired The intervention is limited to a certain number of widows
THREATS ● ● ●
Widows are finally not the effective users of the earned money No acceptance of the ongoing social change by the inhabitants of the community Organic products are generally more expansive and consumer might not be willing to pay a greater amount for their food
2
Rules of eligibility The project we are trying to evaluate has very clear assignment rules. In fact, the geographical scope of the project is limited to the single area of Gangadevipally where social change towards widows can be tried out, observed and measured. Rules of eligibility for the selection of the 20 widows : Be a widow of the community of Gangadevipally (50% of them has have already participate to the previous project of Balavikasa association for the water management). Rules of eligibility for the 15 selected farmers : Be a male farmer of the community of Gangadevipally suffering financial troubles due to the crisis of the ongoing traditional agriculture in the state of AndraPradesh. Priority is given to the those farmers who already kept in contact with Balavikasa association.
3. The evaluation design: Matched DifferenceinDifferences The present impact evaluation will investigate the causal effect (α) of the establishment and management of the organic farm together with widows and men (P) on widows’ role in the community (Y). Widows role in the community (Y) is composed by the following indicators : no. of widows reintegrated in their families; no. of widows that are effectively raising their children. This will be helpful in order to understand whether a project proposing activities based on the collaboration between widows and men is effective to increase widows’ integration in the labour market and in the community. The following Conceptual Framework aims at visualizing the overall indicators measurable in the project, and the focus of the present evaluation design (highlighted in red) :
3
The dimension that will be investigated for the impact evaluation is the interaction of widows with the community members. Our work will specifically focus on widows relation with their families. How many widows live back with their families as a result of the project? How many of them do raise their own children? These indicators will be compared before (average values measured before the start) and after the completion of the project, both in the treated group of widows of Gangadevipally and in a comparison group of widows of the nearby village of Mogilicherla where the project will not be implemented. The widows from Mogilicherla, nevertheless, participated to the previous project of Balavikasa for women empowerment. This will ensure that the difference between the comparison group and the treated is not correlated to the impact of the previous Balavikasa project, since both of them participated in it. To reduce the risk of bias and have a more robust estimated counterfactual, the Differenceindifferences method will be combined with the Matching. This will allow us to take into consideration any unobserved characteristics that are constant across time between the two groups. First, the matching method will be applied on observed baseline characteristics that are relevant for the project. Secondly, the Differencesindifferences method will be applied to take into account what will occur to the control group during the implementation of the project. Finally, the average of the mean impact will be calculated upon the matched groups and compared through the Differencein differences analysis. Matching The evaluation design is first based on matching in order to observe the expected increased inclusion of the widows in their community. Matching is highly recommended when the treated group is established on observable variables. In this project, the treatment is focused exclusively on the widows that are specifically living in the area of Gangadevipally. As already mentioned, the indicators that will be measured are: the number of widows growing their children and the number of widows reunified with their families.
4
The scale of intervention, the reduced number of participants, and the selection criteria for the target group should, in theory, avoid the risk of the socalled ‘Dimensionality problem’. Indeed, the project we are trying to evaluate has very clear assignment rules that explain why some participants are enrolled and others not. The internal validity of an impact is, in the bestcase scenario, ensured through the process of randomized assignment of treatment. Nonetheless, thanks to the peculiar condition of this project (being the target group only women that are widows and that come from a single village) the internal validity of the matching is increased by selecting the comparison group with the same rules that have been applied for the treated group (units must be widows and they must come from one single village). The controlgroup making up the units for the matching is the group of widows of the nearby village of Mogilicherla. The units will be matched according to the following characteristics, already available in the data at hand (detailed in Point 4, Data collection): Age
Income level
Number of years that they are widows
Number of children
Number of months unemployed
with or without children
living with their families or not
Hereby, some hypothetical matching:
Differenceindifferences The evaluation aims at comparing to what extent the project will increase widows’ inclusion in the community of Gangadevipalli, in comparison with widows of surrounding villages. Differenceindifferences allows to compare the beforeandafter change in outcomes for widows of Gangadevipally and for the controlgroup of widows of Mogilicherla, that did not enroll in the program but were exposed to the same set of environmental conditions and that, thanks to the matching, are very similar in characteristics to the widows of Gangadevipally.
5
Figure 1: hypothetical example of possible measured impact with DD technique of the project on the relations of widows with their families, with Y measured by the % of widows living back together with their families and % of widows growing their children. The main difficulty in this type of evaluation is, first of all, to be sure that the outcome trends are similar in the comparison and treatment groups before the intervention and second, that the only factors explaining changes in outcomes between the two groups are constant over time. As it has been stated in the introduction, widows’ conditions are more extreme in rural areas, where the social norms are very tight and the awareness of widows is lower. Consequently, the diffindiffs method is applied in a region where social norms and the social habits are very strong, meaning that the indicators we will measure are, in theory, constant over time. Nonetheless, in order to verify if the outcomes of the treatment and comparison groups will not change before the intervention, we will have two rounds of data available, one before the start of the program, and one after the end of the projects, and test to see if any difference in trends appears between the two groups. Also, in order to increase the reliability of the evaluation, falsification test will be realized to verify that the outcomes would have moved in tandem in the treatment and comparison groups in the absence of the program. At the end, attention will be paid to identify other factors that could have affected the difference in trends between the two groups, in order to prevent any bias in the evaluation. Mean impact The question we ask to conduct the impact evaluation is “What is the impact of the treatment (establishment and management of organic farming in cooperation with men) on the improved role of widows in the community of Gangadevipally?”. How this treatment lead to :
Widows raising their children Widows reunification with their families
As mentioned above, the mean impact will be calculated by doing the comparison of the matched units, according to the baseline characteristics, within the difference in differences comparison.
4. Data collection
6
(see also section 3 Conceptual Framework) Data collection will primarily focus on the data needed to measure the two indicators selected in the evaluation design (see graph 1: Conceptual Framework red circle), namely Widows reunification with their families Widows raising their children. Nonetheless, being the issues addressed by the project multidimensional, also other indicators of the conceptual framework will be gathered when easily available and not costly, thus allowing to establishing a broad and more holistic view of the impact of the project. These supplementary data, nonetheless, will be analyzed only with the most common and superficial statistical tools (mean, frequency distribution,..) The following part describes how we will access the data needed to start our impact evaluation. Being the focus of the project on the change at the community level (inclusion of widows in the legal labour market of Gangadevipally village), the data to be collected are mostly at the local level and microlevel, with the units being the widows of Gangadevipally. In order to measure the impact in the most objective way, most data will be quantitative. Nonetheless, dealing the issue of widows with aspects as inclusion or selfesteem that fit very little with mathematical categories, also some qualitative data will be gathered, thus allowing a deeper understanding of the expected change. Different methods of data collection will be applied according to the different actions to measure. The general collected data will be mapped and filtered and adapted to our needs. As mentioned above, to investigate the selected priority dimension of the relation of widows with their families, data about the percentage of widows growing their children and of widows living back in their families needs to be gathered. The data source will be an adhoc survey addressed directly to the widows both of the treated and control group. The results will be integrated and compared with the Census Report of Telengana State (Government of Telengana, 2015) and the Census Report of the Government of India (Government of India, 2011). Since also the indicators describing the financial situation of widows require a survey, we will take advantage of the already needed survey to also investigate this dimension. Question will be added to the survey about widows increased income and their working conditions (scaled from 1bad and 10very good). This data will be matched and integrated with the internal management documents of Balavikasa describing monthly wage of widows at the organic farm and with reports of Telengana State Info (Government of Telengana, 2015) about employment rate. The measurement of selfesteem, even if not mentioned in the Conceptual Framework because not directly related to the identified dimensions, will be addressed. Firstly, it does not rise significantly the cost of evaluation since it would be conducted through the addition of some questions to the already needed survey; secondly, selfesteem is a very significant qualitative indicator to assess widows empowerment, and consequently their ability and will to challenge in first person the stigma associated to their status. Awared that it is a very critical factor to be objectively assessed and measured, we think that nonetheless an open questions investigating their opinion on their ability to bring about change will help to gain a deeper insight of the impact of the project on widows status. The dimension of widows’ relationship with men, instead, will not be analyzed and included in aside information since both its two indicators require a big effort in term of resources and time To know the reported cases of violence against widows a monitoring system in the health centers should be 7
established, while to measure the number of activities carried out by both men and widows an indepth research into the associations, economical activities and committees of the village should be conducted, with occurring evident high cost in both cases.
5. Organization of the evaluation study First, general data from official sources as mentioned above will be conducted and mapped. Afterwards, the monitoring system will be designed and implemented. Third, local governments will be asked for collaboration, especially concerning the data needed to answer the question of how many activities are carried out by men and widows. Fourth, the questionnaire for the survey will be designed and delivered to the widows of the the treated and control group. The first survey will be sent out before the starting of the project (t=0), and the second 3 years after the completion of the project (t=1). In the process, some threats could hinder a successful evaluation study. First, a lack of data might be occurring. For example, it is possible that not every widows will answer all the questions of the survey. Moreover, the data from Balavikasa association, being the one delivering the project, could overestimate the impact of the project in widows revenue. Secondly, the indicators of widows income could not be enough to verify the real economic independency of widows since it is very possible do not actually have direct control over their incomes but other family members do that at their place. But this issue should be the object of another project; as for the evaluation of the present project, we will limitate to highlight the degree of increased in their wage after the project. For the project as a whole the most dangerous risk are the weather conditions. If there will not be the right conditions, the organic farm does not produce enough outputs to allow widows to sell products and to finally earn money. Last but not least, a risk is also the lack of willingness and readiness of male farmers to work with widows. Therefore the selection of our population is one of the most important issues of the project and the starting point for a successful implementation.
6. Conclusion The importance of a well composed evaluation and monitoring system is key for future development projects and their success. Our long term goal is a more equal society that values every individual and their own rights, not only in Gangadevipally but also in other parts of the world and especially in India. The evaluation report, then, is essential to decide whether it worths to scale it up to and to enlarge it to other communities.
8
7. Annexes Result chain:
9
Example of a possible baseline: ID widows
Age
Treatme nt status
Level income (in Indian rupee)
1 2 3 4 5 6 7 8 9 10 ...n
36 43 19 24 46 39 26 35 50 41
T NT T NT NT T T T NT NT
0 0 200 170 0 100 200 0 200 120
Since how many years the person is widow 5 0 0 5 3 15 7 7 14 9
Number of children
Number of month unemployed
3 4 2 5 5 3 0 6 5 0
12 30 25 34 3 12 34 16 18 12
Timing : Year 1
Months
Implementing body
Activity
1
2
3
4
5
6
7
8
9
10
11
12
Preparation of the
Activity 1 Establish
an organic farm Execution of Activity 1
Execution .Activity 1.1.Selection of the participants Execution Activity 1.2.Training course Execution Activity 1.3.Monthly visits to successful organic farms Execution Activity 1.4.Coplanning of the farm structure and management
coapplicant and/or affiliated entity
Center of sustainable agriculture Partners associations
10
Execution Activity 1.5. Coimplementation of the organic farm Preparation of the
Activity 2 Conduct
daily agricultural activity Execution of Activity 2 Execution .Activity 2.1. Daily work on the
field, organized according to the scheduled coplanned in point 1 with the Execution Activity 2.2.Management of a space dedicated to participants’ children Execution Activity 2.3.Draw ongoing report of the activities on the field
Year 2
Months
Implementing body
Activity
1
2
3
4
5
6
7
8
9
10
1 1
12
Execution of Activity 2
Execution .Activity 2.1. Daily work on the
field, organized according to the scheduled coplanned in point 1 with the continuous support of the staff Execution Activity 2.2. Management of a space dedicated to participants’ children Execution Activity 2.3. Draw ongoing report of the activitieson the field Execution Activity 2.4
Organize monthly visit to the new implemented
Partners association
11
organic garden for partner associations
Execution of the activity 3
Execution activity 3.1
Preparation of the Activity 3. Organise the sales of the agricultural products made
Repartition of the role and task for the sale of the products. execution activity 3.2 Build partnerships with neighbours villages/associations
execution activity 3.3 Daily sales of the organic product in Gangadevipally execution activity 3.4 Distribution of income
Year 3
Months
Activity
1
2
3
4
5
6
Execution of the activity 2 Execution of the activity 3
Implementing body
8. References Census 2011 (2011) Report India. Government of India Ministry of Home Affairs. Retreived from: http://www.censusindia.gov.in/2011census/population_enumeration.html [14.12.2015]
12
Mastey Nimi (2009) Examining Empowerment among Indian Widows; A Qualitative study of the Narratives of Hindu Widows in North Indian Ashrams. Journal of International Women’s Studies, article 12. Volume 11, Issue 2 Paul J. et al (2011) Impact evaluation in practice, World bank Reddy, P. Adinarayana (2004) Problems of Widows in Indi. N ew Delhi, India: Sarup & Sons UN Women (2014) Empowering widows: an overview of policies and programmes in India, Nepal and Sri Lanka UN Women (2001) Widowhood: invisible women, secluded or excluded. United Nations Division for the Advancement of Women, Department of Economic and Social Affairs Government of Telengana (2015). Telangana state portal. Retrieved from http://www.telangana.gov.in/ [13.12.2015]
Sopar Balavikasa (2015) Women and Widows. Retreived from: http://www.soparbalavikasa.org/blog/cause/womenandwidows/ [01.12.2015]
13
