� សា�ស� គេ្រមាង វធ សរុបេដម ងអំ ណានបែន�មស្រមាប់កមា ុ រ ើ ្បព្រង� ី ី TOTAL READING APPROACH FOR CHILDREN PLUS (TRAC Plus) គាំ្រទថវ ិកេដយ:
្របតិបត�ិេដយ:
Early Grades Reading Analysis (EGRA) Report for TRAC+ Year 2. EGRA Endline Analysis.
Report: September/October 2016 th FINAL DRAFT 3: 06 October 2016
(Cover photo: MoEYS EGRA training, March 2016)
1
Contents: Section 1 List of acronyms___________________________________________________________________________ 2 Executive summary________________________________________________________________________ 2.1 Aim of report______________________________________________________________________ 2.2 Introduction to TRAC+_______________________________________________________________ 2.3 EGRA sample, test and analysis_______________________________________________________ 2.4 Summary of main findings___________________________________________________________ 2.5 Summary of recommendations_______________________________________________________ 3 Introduction to TRAC+ and Aim of report______________________________________________________ 3.1 Introduction to TRAC+______________________________________________________________ 3.2 Aim of report______________________________________________________________________ 4 EGRA testing______________________________________________________________________________ 4.1 EGRA protocol_____________________________________________________________________ 4.2 TRAC+ effect on EGRA_______________________________________________________________ 5 Limitations of the EGRA_____________________________________________________________________ 5.1 Limitations of the EGRA test content___________________________________________________ 5.2 Use of the Tangerine app____________________________________________________________ 5.3 Absent students___________________________________________________________________ 5.4 Inconsistencies in the EGRA administration_____________________________________________ 5.5 Alterations in the EGRA test between baseline and endline________________________________ 6 Sampling and limitations on analysis__________________________________________________________ 7 Methods of analysis________________________________________________________________________ 8 Main findings_____________________________________________________________________________ 8.1 Grade 1___________________________________________________________________________ 8.1.1 Gender factors at Grade 1____________________________________________________________ 8.2 Grade 2___________________________________________________________________________ 8.2.1 Gender factors at Grade 2____________________________________________________________ 8.3 Grade 3___________________________________________________________________________ 8.3.1 Gender factors at Grade 3____________________________________________________________ 8.4 NGO factors_______________________________________________________________________ 8.5 Promotion of struggling students______________________________________________________ 9 Recommendations for TRAC+ year 3 and beyond________________________________________________ 10 Annexes_________________________________________________________________________________ Annex 1 Sample population for testing______________________________________________________ Annex 2 Schools________________________________________________________________________ Annex 3 School level participants trained in EGRA orientation___________________________________ Annex 4 DTMT Guidelines for Basic Education [extract]_________________________________________
Page 3 4 4 4 4 5 8 9 9 9 10 10 11 11 11 12 12 13 13 13 14 15 16 17 18 20 21 23 24 25 29 31 31 34 35 36
2
1.
List of Acronyms
ADP BL BMC C CA CEC CFS EGR EGRA EL F IMP (or Imp) KTM LC LPM M MoEYS PVH RRS SD SRP T TKO ToT TTS TRAC(+) USAID WEI WPM WVI
Area Development Programme (World Vision International administrative area unit) Baseline Banteay Meanchey (Province) Comparison (school without TRAC+ intervention) Correct Answers Cambodia Economic Census Child-Friendly School Early Grades Reading Early Grades Reading Assessment Endline Female (for participant numbers) Improvement (difference between baseline and endline, i.e. EL – BL, or written BLEL) Kampong Thom (Province) Literacy Coach Letters (read) Per Minute Male (for participant numbers) Ministry of Education Youth and Sports Preah Vihear (Province) Rapid Response System School Director Siem Reap (Province) TRAC+ school (Intervention school) Takeo (Province) Training of Trainers Thoun Thean Suksa (Translates as: learning resources – TTS make educational games. Also referred to as ‘learning toolkit’) Total Reading Approach for Children (Plus) United States Agency for International Development World Vision, Inc. (NGO) Words (read) Per Minute World Vision International (NGO) 1
EGRA sub-component abbreviations : LNR Letter Name Recognition FWR Familiar Word Reading NWR Non-Word Reading SR Sentence Reading ORF Oral Reading Fluency RCO Reading Comprehension LCO Listening Comprehension
1
For the purposes of graphical display.
3
2.
Executive Summary
2.1 Aim: •
Aim of Report
2.2
Introduction to TRAC+
The aim of this report is to evaluate the current performance of the Total Reading Approach for Children Plus (TRAC+) project interventions based on data provided by the Early Grades Reading Assessment (EGRA) endline survey performed at the end of the 2015-2016 academic year. Purpose: • The purpose of this report is to identify any strengths and weaknesses of the TRAC+ intervention that are displayed by the EGRA analysis and to use these to improve project outcomes for TRAC+ year 3. Outcomes: • Improved practice for TRAC+ year 3 at strategic (project management) level and at local (school, district, provincial) level. • Improved practice in EGRA for TRAC+ year 3. th • Relevant content disseminated for TRAC+ Mid-Term Evaluation (MTE) meeting planned for 27 September 2016. Target Audience: • TRAC+ collaborative partners: World Education, Inc. (WEI) and World Vision International (WVI). • Summary findings will be presented to the Ministry of Education, Youth and Sports (MoEYS) as appropriate and after agreement between WEI and WVI.
The TRAC+ project is a collaborative work between WEI, WVI and MoEYS. TRAC+ provides support for early grades students (Grades 1-3) in 138 primary schools in five provinces in Cambodia. TRAC+ aims relevant to this report are: Project goal: Improve early grade literacy outcomes Project outcomes: # 1: Schools with TRAC+ interventions report better reading achievement than comparable schools that do not use TRAC+ # 4: Project activities are effectively monitored+ To meet these targets, TRAC+ uses a system of Reading Benchmarks to help teachers evaluate the progress of their students as they work through MoEYS textbooks. Students who are unable to pass a benchmark test are given remedial support in the form of the Rapid Response System (RRS). Teachers are assisted in their testing of students and in the interventions by a TRAC+ appointed Literacy Coach (LC). This report focuses on the results from the EGRA tool used to assess specific reading competencies of students at a baseline (BL, November 2015) and endline (EL, July 2016) of the academic year. EGRA is administered using the specifically designed Tangerine App. TRAC+ is a three-year project, with the final year starting in October 2016 and project completion due in September 2017. The total number of students expected to benefit over the lifetime of the project is 19,822; with a project budget of $2.84m, this works out at a cost per beneficiary of $143.27. 2.3
EGRA sample, test and analysis
TRAC+ has 138 TRAC+ (T) Intervention schools, which are split into Group A (73 schools with support for Grades 1-3 that receive an additional mLearning intervention) and Group B (65 schools with support for Grades 1-2 and no mLearning intervention). These schools are compared with a selection of 50 Comparison (C) schools from within the five target provinces. Group A schools were 2 selected as schools that were ‘likely to respond better with TRAC+ interventions’ which presents difficulties in comparing Group A and Group B school learning outcomes. Comparison schools were selected according to those with similar EGRA BL figures to TRAC+ Intervention schools. The BL comparisons were made using incorrectly calculated EGRA scores and selection process was not rigorous enough to ensure a valid Comparison sample. All TRAC+ Intervention and Comparison schools undertook the EGRA, using sample of four children per Grade 1 2
Internal WEI communication
4
class and three each for other target classes. The total sample size used was 9.5% of the eligible student population (2346 students tested altogether for the endline). The EGRA test has seven sub-components as shown below: Sub-component
Early reading skill
Skill demonstrated by student’s ability to:
Output unit
1. Letter Name Recognition (LNR) 2. Familiar Word Reading (FWR) 3. Non-Word Reading (NWR) 4. Sentence Reading (SR)
Letter recognition
5. Oral Reading Fluency (ORF)
Oral reading fluency
Read a narrative text with accuracy, little effort, and at a sufficient rate
6. Reading Comprehension (RCO)
Reading comprehension
Respond to literal and inferential questions about the text read
Correct answers (CA)
7. Listening Comprehension (LCO)
Comprehension
Respond correctly to literal and inferential questions about a text read to the student
Correct answers (CA)
Alphabetic principle/decoding skills Alphabetic principle/decoding skills Reading fluency
Provide the name of graphemes presented in random order Use letter-sound correspondence to read a list of familiar words Use letter sound correspondence to read a list of non-words Read unrelated sentences fluently
Letters per minute (LPM) Words per minute (WPM) Words per minute (WPM) Words per minute (WPM) Words per minute (WPM)
TRAC+ interventions (the Rapid Response System (RRS)) focus on the first three sub-components of the EGRA test that relate to letter recognition and decoding words. The current version of the EGRA test has a number of problems which may be summarised briefly as: • • • • • •
Almost one-third (7/23) dependent vowels missing from sub-component 1 (LNR). Lack of alignment of content with MoEYS textbooks and omission of high-frequency phonetically decodable words. Inconsistent selection of words on sub-component 3 (NWR). Lack of standardisation between baseline and endline testing content. Sub-component 7 (LCO) contains questions that do not distinguish students who understand from those who do not. Inaccessibility of large parts of the test for the majority of Grade 1 students assessed.
EGRA results are analysed by sub-component and comparisons are made between endline scores (T vs. C) and by improvement (BLEL) made over the course of the school years (T vs. C).Gender differences will also undergo analysis. Analysis is by two-tailed t3 test with an α-value of 0.05. The second stage of analysis will seek to look at effect size using Cohen’s d test where a d-value of more than 0.2 may be considered a small effect; a value of more than 0.5 may be considered a medium effect; and a value of more than 0.8 may be considered a large effect. 2.4
Summary of main findings
Limitations on findings: All findings listed in this report should be considered as indications of the current situation; the level of data integrity available for analysis is insufficiently robust to deliver firm conclusions. Some trends based on these indications may later lead to valid conclusions, but any EGRA taking place in TRAC+ year 3 will need to be delivered with greater rigour. The following concerns should be taken into context with regard to any conclusions reached in this report: • Sampling was not performed according to the EGRA guidelines. • Comparison schools were not matched according to an accepted protocol. • EGRA training was of insufficient standard and duration to permit sufficient rigour in gaining test data. • The MoEYS-sanctioned EGRA tool contains too many basic errors to be considered a reliable indicator for a number of subcomponents. • Allocation of ‘better’ schools to Group A precludes analysis of Group A compared with Group B using the small sample size. Overall: • EGRA testing shows some improvements (BLEL) in TRAC+ Intervention schools compared with Comparison schools. 3 http://www.uccs.edu/lbecker/effect-size.html
5
• •
•
• • •
o Improvements are most marked in EGRA sub-components (1-3) relating to letter recognition and decoding words; this is observable across all three grades sampled and is consistent with the type of intervention applied. Grade 1 Intervention schools show significantly (P<0.001) higher results on all EGRA reading sub-components (1-6) compared with Comparison schools; the effect size is small in all instances (Cohen’s d = 0.28-0.39). Grade 2 Intervention schools show higher results and better improvements on all EGRA phonics sub-components (1-3) compared with Comparison schools; the difference is significant for LNR (#1, P<0.05) and NWR (#3, P<0.01) and the effect size is small in all three instances (d = 0.22-0.45). Grade 3 Intervention schools show higher results and better improvements on all EGRA sub-components (1-7) compared with Comparison schools; the difference in improvement is significant for LNR (#1, P<0.01) and NWR (#3, P<0.001). Effects in improvements are shown as: small in sub-components FWR (#2, d = 0.35), ORF (#5, d = 0.26) and RCO (#6, d = 0.24); medium in LNR (#1, d = 0.53); and large in NWR (#3, d = 0.83). Girls out-performed boys consistently in EGRA reading sub-components (1-6) of the EGRA test, regardless of interventions. The level of reading ability shown by many students is insufficient for a promotion rate in line with MoEYS expectations. The improvements in reading ability shown by TRAC+ interventions to date are relatively small.
Grade 1 (EGRA Endline only): • TRAC+ Intervention schools out-perform Comparison schools in all seven sub-components in EGRA endline. • The improvements in reading sub-components (1-6) were all highly significant (P<0.001) and the effect size was ranked as ‘small’ in all instances (d = 0.28-0.39). • Assessed Grade 1 students were unable to sufficiently access sub-component 6 (RCO) (over 90% of students scored zero). o Sub-components 2-5 were also inaccessible (zero scores) to over two-thirds of the sampled population. • The level of reading ability in Grade 1 students is so low that only letter recognition can be reliably analysed; this is a shortfall relative to the curriculum expectations where students should be able to read compound words at the end of Grade 1. • Girls out-performed boys in reading sub-components (1-6) of the EGRA test, regardless of interventions. Graphical results for Grade 1 (nb. ‘zero’ baseline assumed):
600%
Grade 1: EGRA EL results T vs C (C normalised at 100% for comparison)
Although percentage scores (T vs. C) are high, the absolute scores achieved by sampled students are very low. LNR (T) = 18.8 LPM LNR (C) = 13.3 LPM
500% 400%
Word (and non-word) reading (T) range is 2.7-3.4 WPM across subcomponents. Word (and non-word) reading (C) range is 1.2-1.3 WPM across subcomponents.
300% 200% 100% 0% LNR
FWR NWR SR TRAC+ Intervention
ORF RCO Comparison
LCO
Note that the very high (T) column for RCO is based on a very low absolute score (0.15 correct answers (T), 0.0125 correct answers (C)).
Grade 2: • TRAC+ Intervention schools out-perform Comparison schools in 6/7 sub-components in EGRA endline. • The endline scores in two phonics sub-components (1 and 3) were significantly better (P<0.01) in Intervention cf. Comparison. • TRAC+ Intervention schools out-perform Comparison schools in 4/7 sub-components in improvements BLEL. o The improvement scores in two phonics sub-components (1 and 3) were significantly better (#1 P<0.01, #3 P<0.05) in Intervention cf. Comparison. o The three phonics sub-components (1-3) showed a small effect size (d = 0.22-0.45) in improvements for Intervention cf. Comparison. • Results across provinces, although not statistically robust, showed large variations: o Comparison schools in Siem Reap and Kampong Thom performed well when compared with TRAC+ Intervention schools. o TRAC+ Intervention schools in Takeo province performed well against Comparison schools. o Further work can be done to determine if the difference in results between provinces depends on other NGO support. • Comparing Grades 1-3, Grade 2 shows the weakest impact of TRAC+ intervention within the three target grades. • Girls out-perform boys in sub-components 1-6 in both TRAC+ Intervention and Comparison schools.
6
Graphical results for Grade 2:
140% 120% 100% 80% 60% 40% 20% 0%
Absolute scores achieved by sampled students are low.
Grade 2: EGRA Endline IMPROVEMENTS T vs C (C normalised at 100% for comparison)
LNR
FWR NWR SR TRAC+ Intervention
LNR (T) = 32.8 LPM and LNR (C) = 28.8 LPM Word reading (T) range is 13.6-18.8 WPM across sub-components. Word reading (C) range is 13.2-19.3 WPM across sub-components. NWR (T) = 7.8 WPM and NWR (C) = 5.8 WPM
ORF RCO LCO Comparison
The effects of the late start of TRAC+ interventions in the 2015-2016 academic year would be particularly noticeable in Grade 2, where students had finished the part of the curriculum that was most linked to the TRAC+ interventions by the time TRAC+ year 2 began implementation.
Grade 3: • TRAC+ Intervention schools out-perform Comparison schools in all seven sub-components in EGRA endline (and 6/7 at BL). • TRAC+ Intervention schools out-perform Comparison schools in all seven sub-components in improvements BLEL. o The improvement scores in two phonics sub-components (1 and 3) were significantly better (#1 P<0.01, #3 P<0.001) in Intervention cf. Comparison. • The effect sizes showed small effects in FWR (#2, d = 0.35), ORF (#5, d = 0.26) and RCO (#6, d = 0.24); a medium effect in LNR (#1, d = 0.53); and a large effect in NWR (#3, d = 0.83). • TRAC+ year 3 can investigate whether the above results are directly due to project intervention; length of intervention; sampling bias or other external factors. • Results across provinces, although not statistically robust, showed large variations: o Comparison schools in Siem Reap province performed well when compared with TRAC+ Intervention schools. o In the four other target provinces, TRAC+ Intervention schools performed well when compared with Comparison schools. • Girls out-perform boys in sub-components 1-6 in both TRAC+ Intervention and Comparison schools (as with Grade 2); this effect is significant (P<0.05) for BL and EL figures. There are too many variables to compare improvements with any reliability. Graphical results for Grade 3: Grade 3 improvements are hard to assess because of the sampling of only Group A schools which were selected at the start of TRAC+ as being potentially higher attaining schools. Baseline figures at Grade 3 are higher for TRAC+ Intervention schools than control schools in 6/7 sub-components. Brief analysis suggests that this is because of school selection; group A schools create most of the baseline inequality.
Grade 3: EGRA Endline Improvements T vs C (C normalised at 100% for comparison) 250% 200% 150% 100% 50% 0% LNR
FWR NWR
SR
TRAC+ Intervention
ORF
RCO
LCO
LNR (T) = 47.2 LPM and LNR (C) = 37.0 LPM Word reading (T) range is 23.5-32.1 WPM across subcomponents. Word reading (C) range is 17.8-26.7 WPM across subcomponents. NWR (T) = 13.8 WPM and NWR (C) = 8.4 WPM
Comparison
NGO support
7
The question of how much NGO support effects the results of TRAC+ is beyond the scope of this report. Figures from the 2011 4 Economic Census of Cambodia show that Siem Reap (SRP) Province has 32 times the level of NGO staff per head of population compared with Takeo. (SRP:Takeo (TKO) = 32.44:1, SRP:Banteay Meanchey (BMC) = 1.22:1, SRP:Kampong Thom (KTM) = 2.77:1, SRP:Preah Vihear (PVH) = 5.35:1). Promotion of struggling students Detailed analysis of the EGRA figures compared with MoEYS expectations of 90%+ or 93%+ promotion rates shows that students are being promoted to higher grades without sufficient reading ability to access the next class. Analysis of data includes the findings: • Grade 1: Only 60% of TRAC+ Intervention students and 51% of Comparison students sampled were able to correctly identify ten Khmer characters in a minute. Only 85% of students in TRAC+ Intervention schools and 78% of students in Comparison schools were able to identify a single letter. • Grade 2: Only 67% of TRAC+ Intervention students and 58% of Comparison students sampled were able to correctly read 20 Khmer letters in one minute at the end of Grade 2. • Grade 3: Only 69% of TRAC+ Intervention students and 63% of Comparison students sampled were able to correctly read 30 Khmer letters in one minute at the end of Grade 3. Considering the weakest students, 10% of TRAC+ Intervention students and 17% of Comparison students sampled are unable to correctly read 10 Khmer letters in one minute by end Grade 3. The MoEYS presented a long-term aim in 2014 that: “Every child who successfully completes Grade 3 will be able to read
with understanding at a level of 45 – 60 words per minute.” 5 The results shown in this sample suggest that this aim is a long way from being achieved. 2.5
Summary of recommendations
The current MoEYS-sanctioned EGRA tool being administered has serious weaknesses; however, there is little or no chance of a new tool being created, tested and standardised before TRAC+ year 3 EGRA baseline is required to commence. ● Recommendation 1: EGRA for TRAC+ year 3 to use improved EGRA test IF such a resource is available in time. Comparison schools were not selected with sufficient rigour and the sample size (especially for Comparison schools) did not allow sufficient sampling for a meaningful comparison. Re-selection of Comparison schools is a very time-consuming task and would not be possible before the start of TRAC+ year 3. ● Recommendation 2A: No change in school selection for TRAC+ year 3. ● Recommendation 2B: Increase sample size in Comparison schools. ● Recommendation 2C: Comparison school selection issues noted in subsequent EGRA or other performance analysis. TRAC+ interventions are focussed very directly on phonics/decoding reading skills. Support for higher-level reading skills is not yet a part of the project intervention strategy. ● Recommendation 3A: Additional ‘connected text’ resources (e.g. mini-libraries) to be provided as RRS interventions. ● Recommendation 3B: Additional levelled readers to be distributed in multiple sets to allow paired or group reading activities. Students are being promoted with insufficient reading ability to access the curriculum at a new grade; this becomes more noticeable at higher grades (Grade 2 revises most of Grade 1 in the first 31 lessons) and is partly down to the pressure on Directors to achieve MoEYS-mandated pass rates. ● Recommendation 4: TRAC+ to present relevant EGRA data to MoEYS with priority of stopping promotion of students who do not have the requisite competencies to access the next grade. Some crucial RRS interventions were missed in the early part of the academic year 2015-2016 because of the delay in beginning the implementation of TRAC+ year 2 activities. ● Recommendation 5: TRAC+ activities to start promptly at the beginning of the 2016-17 academic year. All recommendations are made on behalf of WEI as a basis for further discussion with stakeholders in EGR in Cambodia.
4
Ministry of Planning (2013) Economic Census of Cambodia 2011 Provincial Reports (#01 BMC, #6 KTM, #13 PVH, #17 SRP, #21 TKO) National Institute of Statistics supported by JICA 5 Kann Puthy (2014) Variations in early reading assessment results between rural and urban populations presented to UNESCO Institute for Statistics conference ‘What works in oral assessments of early reading?’ 21 July 2014, Montreal, Canada on behalf of MoEYS, Cambodia (http://tinyurl.com/hop9538 retrieved 09-Oct-2016)
8
3.
Introduction to TRAC+ and Aim of Report
3.1
Introduction to TRAC+
6
The Total Reading Approach for Children Plus (TRAC+) project scales up the work from the Total Reading Approach for Children (TRAC) project (2012–2014), which was funded under the All Children Reading Grand Challenge, a consortium of donors including USAID, World Vision International (WVI), and Australian Aid. TRAC+ scales up the TRAC project from the pilot project of eight schools up to 138 schools in five provinces. TRAC+ is a three-year project, with the final year starting in October 2016 and project completion due in September 2017. The total number of students expected to benefit over the lifetime of the project is 19,822; with a project budget of $2.84m, this works out at a cost per beneficiary of $143.27. Under the TRAC+ project, World Education, Inc. (WEI) works closely with the Ministry of Education, Youth and Sports (MoEYS) and in partnership with WVI, the donor partners, to improve reading outcomes of Grade 1, 2 and 3 students in Cambodia using an integrated approach comprised of various interventions, including: • Reading Benchmarks WEI collaborated with MoEYS to create the now endorsed Reading Benchmark Standards for the national curriculum. These include specific tasks and skills that students should master at designated intervals. The benchmarks include regular interval assessments which divide reading skills in specific competencies, enabling teachers to deliver targeted remedial interventions. • Literacy Coaches TRAC+ selects and trains school-based Literacy Coaches (LCs) to help teachers administer the benchmark assessments and the Rapid Response System (RRS) interventions. Literacy Coaches also act as the liaison between schools and parents. • Rapid Response System Literacy Coaches implement the RRS after struggling students are identified by means of the benchmark interval assessments. The RRS includes parental engagement, peer tutoring, a reading toolkit, and (in the 73 Group A schools) an mLearning application. The toolkit and mLearning application are aligned with the MoEYS national curriculum textbooks and the Reading Benchmark system. TRAC+ has the following targets which are related to this report: Project goal: Improve early grade literacy outcomes Project outcomes: # 1: Schools with TRAC+ interventions report better reading achievement than comparable schools that do not use TRAC+ # 4: Project activities are effectively monitored TRAC+ has three potential ways to assess project progress through reading outcomes. These are: (1) Standardised EGRA testing using the Tangerine app or paper substitute. (2) Interval test results using the Reading Benchmarks (internally administered by schools with support from TRAC+). (3) Internal end-of-semester tests from schools. This report focuses on the first of these areas: the EGRA test which is externally administered using a Tablet device fitted with the specifically designed Tangerine app. 3.2 Aim: •
Aim of Report The aim of this report is to evaluate the current performance of the TRAC+ project interventions based on data provided by the EGRA endline survey performed at the end of the 2015-2016 academic year.
Purpose: 6
Adapted from website: http://cambodia.worlded.org/projects/total-reading-approach-for-children/
9
•
The purpose of this report is to identify any strengths and weaknesses of the TRAC+ intervention that are displayed by the EGRA analysis and to use these to improve project outcomes for TRAC+ year 3. Outcomes: • Improved practice for TRAC+ year 3 at strategic (project management) level and at local (school, district, provincial) level. • Improved practice in EGRA for TRAC+ year 3. th • Relevant content disseminated for TRAC+ Mid-Term Evaluation (MTE) meeting on 27 September 2016. Target Audience: • TRAC+ collaborative partners: WEI and WVI. • Summary findings report will be presented to MoEYS as appropriate and after agreement between WEI and WVI. 4. 4.1
EGRA Testing EGRA protocol
The Early Grade Reading Assessment (EGRA) is a tool used to measure students’ progress towards learning to read. It is a test that is administered orally, one student at a time and takes about 15 minutes to administer. Details about the testing protocol can be found in the Khmer language book: េតស�អំណានថា�ក់ដំបូង Early Grade Reading Assessment (EGRA) which details the assessment 7
protocol for administering the test using the Tangerine app. Administration of the EGRA tests was co-ordinated by experienced WEI staff with the support of MoEYS and WVI counterparts. Some school-level counterparts also received orientation training on administration of the EGRA test. A total of 34 assessors administered the EGRA, of whom 21 were WEI staff. Training in EGRA orientation and use of the tangerine app was delivered to the following groups as shown in Table 4.1.1 below. Table 4.1.1: Participants trained for administration of EGRA test Group
#participants
MoEYS (Departments of: Primary Education, Curriculum Development, Teacher Training, Information Technology) School level (SDs, Deputy Directors, Teachers, LCs, officials at District and 8 Provincial level) WVI Trainers WVI Staff
Type of training
Duration
10
Orientation
2 days
203
Orientation
1 day
12 9 23
ToT Capacity Building General Training
3 days 2 days
10
In accordance with international norms (RTI, 2011 ) the Cambodian EGRA test is split into sub-components to identify specific reading and language competencies. These are as shown in Table 4.1.2 below. Table 4.1.2: Sub-components of the EGRA test Sub-component 1. Letter name recognition (LNR) 2. Familiar word reading (FWR) 3. Non-word reading (NWR) 4. Sentence reading (SR) 7 8
Early reading skill
Demonstrated by student’s ability to:
Letter recognition
Provide the name of graphemes presented in random order Alphabetic principle Use letter-sound correspondence to read Decoding skills a list of familiar words Alphabetic principle Use letter sound correspondence to read Decoding skills a list of non-words Reading fluency Read unrelated sentences fluently
# items on test: BL 100
EL 100
50
50
50
50
55
57
Output unit Letters per minute (LPM) Words per minute (WPM) Words per minute (WPM) Words per minute
ករែណនាំEGRAនិងរេបៀបេធ�េតស�EGRAេដយេ្របTangerineស្រមាប់អ�កវយតៃម�
Full list in Annex 3 To avoid double-counting, the total number of WVI staff trained was 28. 10 RTI International and International Rescue Committee (2011) Guidance Notes for Planning and Implementing EGRA page vii 9
10
(WPM) 5. Oral reading fluency (ORF)
Oral reading fluency
Read a narrative text with accuracy, little effort, and at a sufficient rate
82
73
Words per minute (WPM)
6. Reading Comprehension (RCO)
Reading comprehension
Respond to literal and inferential questions about the text read
5
4
Correct answers (CA)
7. Listening Comprehension (LCO)
Comprehension
Respond correctly to literal and inferential questions about a text read to the student
3
3
Correct answers (CA)
Total:
345
11
337
Able students may achieve scores above the EGRA ‘ceiling’ scores. For example, a student reading 100 letters correctly in subcomponent 1 (LNR) in 30 seconds would have a score of 100 * (60/30) = 200LPM. Similarly, a student correctly reading 50 words correctly in sub-component 2 (FWR) in 50 seconds would have a score of 50 * (60/50) = 60WPM. 4.2
TRAC+ effect on EGRA
TRAC+ directly addresses sub-components 1-3. Sub-components 4-6 are indirectly addressed, i.e. the reading skills directly addressed by the RRS should be expected to improve learning outcomes on other sub-components. Sub-component 7 (LCO) might almost be considered to be a ‘Control Item’ within the EGRA as it does not actually require any reading skills. However, in this EGRA the low number of questions (three) and the poor question selection means that this control provides little real data. Table 4.2.1 below shows how directly the TRAC+ RRS addresses the seven EGRA sub-components. Table 4.2.1: Level of TRAC+ interventions relating to specific sub-components of the EGRA Subtasks
Early reading skill
Level of TRAC+ intervention
1. Letter name fluency
Letter recognition
VERY DIRECT
2. Familiar word fluency
Alphabetic principle/decoding skills
DIRECT
3. Non-word reading fluency
Alphabetic principle/decoding skills
VERY DIRECT
4. Sentence reading
Reading fluency
INDIRECT
5. Oral reading fluency
Oral reading fluency
INDIRECT
6. Reading Comprehension
Reading comprehension
VERY INDIRECT
7. Listening Comprehension
Comprehension
NOT ADDRESSED
5.
Limitations of the EGRA
5.1 Limitations of the EGRA test content 12
The first EGRA test in Cambodia was implemented by MoEYS in 2010 . The complexity of the Khmer language and the lack of available standardised data means that all current available versions of Khmer language EGRA tests are limited in their effectiveness. The EGRA version used here was endorsed by MoEYS and the most suitable available at the time of testing but has a number of problems that can be quickly identified and will need to be addressed. Some limitations noted by WEI are specified by EGRA subcomponent as shown in Table 5.1.1 below. The relevant departments within MoEYS are aware of the problems with the current EGRA tools and have commissioned Research Triangle Institute (RTI) to collaborate in improving the EGRA tools. A new EGRA tool is unlikely to be available for administration of a TRAC+ year 3 BL. Table 5.1.1: Some limitations of the EGRA test by sub-component as analysed by WEI 11
The total number of items does not represent a ‘total’ score for a test that could be expressed, for example, as a percentage. This is because there is no clear weighting of sub-components, which are considered individually. 12 Further info. see e.g.: RTI International (2015) Assessment of Early Grade Reading in the Education Sector in Cambodia Report for USAID page 64
11
Subtasks
Issue to be addressed:
1. LNR
The seven Khmer dependent vowels that are placed above a consonant are missing from this part of the test. (Vowels: (◌ិ) (◌ី) (◌ឹ) (◌ឺ) (◌ំ) (◌ំ◌ុ) (◌ា◌ំ)). (7/23 vowels)
2. FWR
The section is not aligned with MoEYS textbooks. The most high-frequency words are not included in this part of the test. Obvious high-frequency words excluded which can be decoded phonetically include: I, is, with, and, this, that, have, which (ខ�ុំ , ជា , ជាមួ យ , និង , េនះ , េនាះ , មាន , ែដល ).
3. NWR
Some words are open to multiple possible pronunciations (e.g. ែខស has two possible readings).
4. SR
No obvious problems with this sub-component.
5. ORF
The first line of Version B (EL) uses a technically correct but relatively uncommon plural for ‘students’ that children in early grades would be unlikely to know, much less read. This could be replaced with a simple alternative without affecting meaning: (សិស្សោនុសិស្ស change to សិស្ស).
6. RCO
Version A (BL) of this sub-component uses more repetitive language than Version B (EL); this would appear to make Version A more accessible to students able to infer word meanings from context.
7. LCO
Only three questions. In Version A (BL), the second question does not discriminate students who understand from those that do not understand; the wording needs to be changed.
5.2
Use of the Tangerine App
Assessors report problems with the Tangerine app, in particular the auto-stop feature showing inconsistencies. Another problem was the removal of one question in sub-component 6 (RCO). Although conversations with the assessors suggest that these issues did not affect the overall EGRA results, these problems should be noted and addressed for future implementation of the EGRA. 5.3
Absent students
The intention of the Year 2 EL was to re-assess the students who undertook the Year 2 BL. In the unavoidable case of student absences, EGRA administrators selected an alternative student at random; however, the author’s opinion (backed-up by questioning at school-level) that it is more likely that the more able students are present at the end of the academic year, meaning any detectable bias from re-selection would be expected to be in favour of higher results in the EGRA EL scores. Full analysis is beyond the scope of this report, but the raw data will remain available. Attrition rates are show in Table 5.3.1 below. Table 5.3.1: Attrition rates for students in G2 and G3 Group
# Student BL tested
Students present BL and EL
Attrition rate
Grade 2 (T)
514
`75%
25%
Grade 2 (C)
179
75%
25%
Grade 3 (T)
476
74%
26%
Grade 3 (C)
180
74%
26%
Attrition rates were the same at for Intervention and Comparison at each grade level. An attrition rate of 25-26% is another factor that creates confusion over comparisons of BL and EL data. 5.4
Inconsistencies in EGRA administration
12
The training for EGRA EL administration was more rigorous than that for the EGRA BL, EL training having been conducted based on previous field experience and with new knowledge about how EGRA is applied. Thus, in the opinion of WEI, it is possible that the results for the EL are more reliable than those for the BL. Where alterations in school selection occurred between BL and EL, leading to incomplete figures, the data from those schools has been excluded from analysis. However, this data will be shared with TRAC+ provincial level staff when appropriate to inform on schools that require priority interventions (for example, some schools with incomplete data had very low BL or EL scores; these schools may need extra monitoring and support from TRAC+ field staff). 5.5
Alterations in the EGRA test between baseline and endline
Between the BL and the EL, a number of test items were altered as shown in Table 4.1.2 (page 10) and these also created alterations in the number of test items available. There is no evidence that the number of test items had any effect on the EGRA results as corrections can be made very simply and tested students were not achieving ceiling scores. Further testing would be required to establish whether the different versions of the tests are directly comparable. 6.
Sampling and limitations
In TRAC+ schools are divided into TRAC+ Intervention (T) schools and Comparison (C) schools with no intervention. TRAC+ Intervention schools are further divided into Group A schools – which receive the additional mLearning (Aan Khmer App) support – and Group B schools which receive all the interventions except for the mLearning. All TRAC+ Intervention schools support Grades 12 and Group A schools additionally support Grade 3 students. Five schools (one TRAC+ Intervention, four Comparison) were re-classified between Years 1 and 2 of the TRAC+ project. The schools analysed for the Year 2 EGRA endline are as shown in Table 6.1 below. Table 6.1: Number of TRAC+ Intervention and Comparison schools analysed Group Group A Group B TRAC+ total TRAC+ Intervention schools Comparison schools Total
Intervention TRAC+ Grades 1-3 and mLearning TRAC+ Grades 1-2 and no mLearning No intervention
Number of Schools 73 65 138 50 188
The sampling procedure uses a randomly selected group of students from Grades 1-3 in all TRAC+ Intervention and Comparison schools. The sampling is shown in Table 6.2 below, as well as the percentage of the total eligible students per grade. Note that the sampling is per class not by grade: i.e. a school with three Grade 1 classes would have 3 * 4 = 12 students sampled. Full sampling details are in Annex 1.
13
Table 6.2: Sample sizes for TRAC+ Intervention and Comparison schools by grade Grade
Grade 1 Grade 2 Grade 3
Baseline No. of TRAC+ Intervention No. of Comparison students sampled students sampled (% population) (% population) 0 0 n/a (assumed zero) n/a (assumed zero) 3 per class 3 per class (8.6%) (8.4%) 3 per class 3 per class (8.3%) (9.3%)
Endline No. of TRAC+ Intervention students sampled (% population) 4 per class (10.1%) 3 per class (9.2%) 3 per class (9.2%)
No. of Comparison students sampled (% population) 4 per class (8.8%) 3 per class (9.4%) 3 per class (10.3%)
The overall sample size is 9.5% of the eligible student population (2346 students tested altogether for the endline). The sampling 13 method was adapted from CheckMarket Sample Calculation which requires a sample size of >7% of the total student population for valid analysis according to the TRAC+ Year 1 Annual report. This figure is incorrect. Comparison schools were matched with TRAC+ Intervention schools according to whether those schools achieved an equivalent comparable baseline score with the TRAC+ Intervention schools. All data analysis in this report will have to take into consideration the fact that Comparison school selection was insufficiently rigorous to allow proper analysis. The sample size was also insufficient to provide a valid Comparison population. Matching took place based on an aggregated total EGRA score without consideration of sub-component weighting or geographical issues (district baselines were not matched). Other variables that might affect school selection but were not considered include: •
•
• •
Size of school (e.g. cluster core school or satellite school): o This is potentially very significant as the level of LC support is consistent at one LC per school, regardless of the number of classes requiring support; this varies between two classes and 13 classes. Level of past or current NGO intervention in EGRA or related field: o The most significant NGO intervention observed was that of the NGO ‘Room to Read’ who provide levelled reading materials training and technical support to their intervention schools. Accessibility of schools: o The term ‘remote school’ has been officially removed from MoEYS technical vocabulary; however, matching schools in terms of their accessibility for support and monitoring is a significant issue. Library facilities: o Includes whether the school has: a library, a separate space for the library, and a full-time librarian.
The analysis will nevertheless look for factors where comparisons can be made – remembering that the primary purpose of the analysis is to look for ways to improve the TRAC+ project in the final year of implementation and to achieve maximum sustainability following the conclusion of the project. The purpose of this report is not to evaluate the performance of individual schools or districts, nor is it a method to identify weak teaching. 7.
Methods of Analysis
The EGRA analysis is a key part of the project goal and the related project outcomes #1 and #4 as shown in 3.1 (page 9). The EGRA results are aggregated by: • raw student scores aggregated for school scores at Grade level (Grades 1,2 and 3) • school scores are aggregated for district / ADP results • district / ADP results are aggregated for provincial results • finally, provincial results are aggregated for a project-level analysis.
13
https://www.checkmarket.com/market-research-resources/sample-size-calculator/
14
This report will focus primarily on project-level analysis. Further analysis at lower levels will be provided to assist field staff in their work, but the sampling techniques used preclude statistically valid conclusions. 14
To decide if TRAC+ project outcome#1 is being met, there are two forms of analysis that may be considered: (1) • (2) •
Endline score comparison Are TRAC+ Intervention schools performing better than Comparison schools on the EGRA EL results? Baseline-Endline improvement comparison Are TRAC+ Intervention schools making larger improvements between BL and EL when compared with Comparison schools?
Differences in TRAC+ year 2 are targeted to show a ‘measureable difference’, but statistical analysis will be used to confirm if TRAC+ Intervention/Comparison differences may be considered significant. This analysis will use a two-tailed t-test with an α-value of 0.05. 15 The second stage of analysis will seek to look at effect size using Cohen’s D test where a d-value of more than 0.2 may be considered a small effect; a value of more than 0.5 may be considered a medium effect; and a value of more than 0.8 may be considered a large effect. The following measures will be taken in an effort to maintain data integrity: (1) Test monitoring by staff from outside WEI. (2) Flagging to check data with: a. Absences. b. Areas where BL > EL. c. Exceptionally high or exceptionally low scores at BL and/or EL. d. Inconsistencies within data sets, e.g. where NWR is strong but LNR is weak. e. Incomplete data (e.g. from changed school selection – schools lacking either BL or EL figures will be removed, but data will be kept). Data from aggregated figures checked in (3) above will be checked against original raw data at school or student level as required. 8.
Main findings
16
Limitations on findings: All findings listed in this report should be considered as indications of the current situation; the level of data integrity available for analysis is insufficiently robust to deliver firm conclusions. Some trends based on these indications may later lead to valid conclusions, but any EGRA taking place in TRAC+ year 3 will need to be delivered with greater rigour. The following concerns should be taken into context with regard to any conclusions reached in this report: • Sampling was not performed according to the EGRA guidelines. • Comparison schools were not matched according to an accepted protocol. • EGRA training was of insufficient standard and duration to permit sufficient rigour in gaining test data. • The EGRA tool was changed between BL and EL without standardisation and testing. • The MoEYS-sanctioned EGRA tool contains too many basic errors to be considered a reliable indicator on many sub-components. • Allocation of ‘better’ schools to Group A precludes analysis of Group A compared with Group B using the small sample size. Overall: • EGRA testing shows some improvements (BLEL) in TRAC+ Intervention schools compared with Comparison schools. o Improvements are most marked in EGRA sub-components (1-3) relating to letter recognition and decoding words; this is observable across all three grades sampled and is consistent with the type of intervention applied. • Grade 1 Intervention schools show significantly (P<0.001) higher results on all EGRA reading sub-components (1-6) compared with Comparison schools; the effect size is small in all instances (Cohen’s d = 0.28-0.39). • Grade 2 Intervention schools show higher results and better improvements on all EGRA phonics sub-components (1-3) compared with Comparison schools; the difference is significant for LNR (#1, P<0.05) and NWR (#3, P<0.01) and the effect size is small in all three instances (d = 0.22-0.45).
14
TRAC+ outcome #1: Schools with TRAC+ interventions report better reading achievement than comparable schools that do not use TRAC+ 15 http://www.uccs.edu/lbecker/effect-size.html 16 In the following data analysis some figures may appear to be inaccurate at the least significant digit. This is due to rounding errors where calculations are made carrying figures forward and then displayed in a readable form (one decimal place for most figures).
15
• Grade 3 Intervention schools show higher results and better improvements on all EGRA sub-components (1-7) compared with Comparison schools; the difference in improvement is significant for LNR (#1, P<0.01) and NWR (#3, P<0.001). Effects in improvements are shown as: small in sub-components FWR (#2, d = 0.35), ORF (#5, d = 0.26) and RCO (#6, d = 0.24); medium in LNR (#1, d = 0.53); and large in NWR (#3, d = 0.83). • Girls out-performed boys consistently in EGRA reading sub-components (1-6) of the EGRA test, regardless of interventions. • The level of reading ability shown by many students is insufficient for a promotion rate in line with MoEYS expectations. • The improvements in reading ability shown by TRAC+ interventions to date are relatively small. Detailed analysis by Grade (1-3) follows. 8.1
GRADE 1
Summary: • TRAC+ Intervention schools out-perform Comparison schools in all seven sub-components in EGRA endline. • The improvements in reading sub-components (1-6) were all highly significant (P<0.001) and the effect size was ranked as ‘small’ in all instances (d = 0.28-0.39). • Assessed Grade 1 students were unable to sufficiently access sub-component 6 (RCO) (over 90% of students scored zero). o Sub-components 2-5 were also inaccessible (zero scores) to over two-thirds of the sampled population. • The level of reading ability in Grade 1 students is so low that only letter recognition can be reliably analysed; this is a shortfall relative to the curriculum expectations where students should be able to read compound words at the end of Grade 1. • Girls out-performed boys in reading sub-components (1-6) of the EGRA test, regardless of interventions. Grade 1 students were assessed only at EL. There is no BL for Grade 1 students: the assumption is that Grade 1 students start their academic education with zero knowledge of reading; as the coverage of pre-schools increases, this assumption may become less valid. In six of the seven EGRA sub-components, TRAC+ Intervention schools perform significantly better than Comparison schools at P<0.05. On the seventh sub-component (LCO) TRAC+ Intervention outperforms Comparison at P<0.1. Table 8.1.1 shows the comparison between TRAC+ Intervention (T) and Comparison (C) schools on the seven EGRA sub-components. Table 8.1.1: TRAC+ Intervention (T) against Comparison (C) schools for EGRA sub-components at Grade 1 Endline SubComponent
T or C
Raw Score (unit)
T>C
T > C Raw Score
T>C% difference
Cohen’s ‘d’ 17 (effect)
% students score >0
1. LNR
T C
18.9 (LPM) 13.3 (LPM)
T>C TRUE ***
+5.6 (LPM)
+42%
0.37 (SMALL)
84.7% 78.0%
2. FWR
T C T C T C T C
3.1 (WPM) 1.3 (WPM)
T>C TRUE *** +1.8 (WPM)
+143%
T>C TRUE *** +1.7 (WPM)
+174%
3.4 (WPM) 1.2 (WPM)
T>C TRUE *** +2.2 (WPM)
+190%
3.2 (WPM) 1.2 (WPM)
T>C TRUE ***
+2.0 (LPM)
+162%
30.8% 20.5% 31.5% 20.0% 28.4% 22.0% 20.1% 11.0%
+10.3%
2.7 (WPM) 1.0 (WPM)
0.35 (SMALL) 0.39 (SMALL) 0.28 (SMALL) 0.29 (SMALL)
0.15 (CA) 0.025 (CA)
T>C TRUE *** +0.125 (CA)
+500%
0.31 (SMALL) T>C TRUE (#) +0.12 (CA) +8% 0.13 (NULL) * = P<0.05 ** = P<0.01 *** = P<0.001
9.0% 1.0% 86% 83%
+8%
3. NWR 4. SR 5. ORF 6. RCO
T C 7. LCO T C Levels of significance:
17
1.64 (CA) 1.52 (CA)
(#) = P<0.1
http://www.uccs.edu/lbecker/effect-size.html
T>C difference score >0 +6.7%
+11.5% +6.4% +10.1%
+3%
16
The results of the EGRA EL analysis in Grade 1 schools show that students in TRAC+ Intervention schools are achieving significantly better results than students in the selected Comparison schools in the reading sub-components (1-6) of the EGRA test. From these figures, testing for effect size shows that for each area, after one year of intervention, the effect is small (d-values from 0.28-0.39). Regardless of the effects of the intervention, the absolute scores are so low (2.7-3.4 WPM (T) and 1.2-1.3 WMP (C)) that the effectiveness of education must be seriously questioned for all schools in the sample. Students are being taught how to read compound words during the later stages of the Grade 1 curriculum and the results show that the vast majority are unable to do this. Graph 8.1.1 shows the tabular data of TRAC+ Intervention school scores against Comparison school scores in bar chart form where the Comparison school figure has been normalised to 100% across all sub-components so the levels of attainment can be observed. Graph 8.1.1: EGRA results of Grade 1 EL Grade 1: EGRA EL results T vs C (C normalised at 100% for comparison)
EGRA EL results T vs C: scores and zeros 100%
500%
80%
400%
60%
300%
Zero
40%
200%
Score LCO (T)
LCO (C)
RCO (C)
ORF (C)
RCO (T)
ORF (T)
SR (T)
LCO
SR (C)
RCO
NWR (T)
FWR NWR SR ORF Treatment Control
NWR (C)
LNR
FWR (C.)
0%
0%
FWR (T)
100%
LNR (T)
20% LNR (C.)
600%
Graph 8.1.2: EGRA results of Grade 1 EL: scores and zeros
The largest percentage improvement is in RCO where all scores are exceptionally low. For that particular sub-component, WEI suggests that the data should not be taken as a significant improvement in reading ability. The improvements in sub-components 1-5 are a positive result for the TRAC+ interventions in Grade 1, but these are still very low figures as shown by number of children failing to register marks on each sub-component and the absolute reading ability of the students tested. Graph 8.1.2 above shows the number of children achieving a positive score. It is only in sub-components 1 (LNR) and 7 (LCO) where a majority (or even over a third) of children can access the task set. However, for all seven sub-components, the TRAC+ Intervention schools show a higher level of attainment than the Comparison schools. 8.1.1
Gender factors at Grade 1
Consideration of gender factors is also possible from the disaggregated data. As show in Table 8.1.1.1 and the associated Graph 8.1.1.1 girls out-perform boys in sub-components 1-6 in both TRAC+ Intervention and Comparison schools. The exception is in subcomponent 7 (LCO) where boys in TRAC+ Intervention schools achieve higher scores than girls. Table 8.1.1.1
Grade 1 Endline gender comparison Category
LNR (LPM)
FWR (WPM)
NWR (WPM)
SR (WPM)
ORF (WPM)
RCO (CA)
LCO (CA)
TRAC+ Intervention Comparison
18.7
2.6
2.4
3.1
2.9
0.14
1.70
Male
11.5
0.6
0.8
0.5
0.8
0.00
0.42
TRAC+ Intervention Comparison
19.1
3.5
2.9
3.7
3.6
0.16
1.33
Female
15.2
2.0
1.2
1.9
1.7
0.05
0.51
Graph 8.1.1.1
Grade 1 Endline gender comparison
17
Normalised score where M(T) = 100
Grade 1 Endline Gender Comparison
8.2
160 140 120 100
M (T)
80
F (T)
60
M (C.)
40
F (C.)
Graph 8.1.1.1 (left) shows the tabular figures in graphical form. The figures are normalised to show the male students TRAC+ Intervention (M (T)) scores at 100% and other figures in comparison to that with the aim of showing the relative ability of girls being higher than boys regardless of intervention.
20 0 LNR
FWR
NWR
SR
ORF
RCO
LCO
GRADE 2
Summary • TRAC+ Intervention schools out-perform Comparison schools in 6/7 sub-components in EGRA endline. • The endline scores in two phonics sub-components (1 and 3) were significantly better (P<0.01) in Intervention cf. Comparison. • TRAC+ Intervention schools out-perform Comparison schools in 4/7 sub-components in improvements BLEL. o The improvement scores in two phonics sub-components (1 and 3) were significantly better (#1 P<0.01, #3 P<0.05) in Intervention cf. Comparison. • The three phonics sub-components (1-3) each show a small effect size (d = 0.22-0.45) in improvements for Intervention cf. Comparison. • Results across provinces, although not statistically robust, showed large variations: o Comparison schools in Siem Reap and Kampong Thom performed well when compared with TRAC+ Intervention schools. o TRAC+ Intervention schools in Takeo province performed well against Comparison schools. o Further work can be done to determine if the difference in results between provinces depends on other NGO support. • Comparing Grades 1-3, Grade 2 shows the weakest impact of TRAC+ intervention within the three target grades. Gender differences: • Girls out-perform boys in sub-components 1-6 in both TRAC+ Intervention and Comparison schools. • In sub-component 1 (LNR), boys from the TRAC+ Intervention schools appear to be benefitting more than the girls from the TRAC+ intervention. • In sub-component 6 (RCO), boys from the Comparison schools are out-performing boys from the TRAC+ Intervention schools. Grade 2 students in TRAC+ Intervention schools should be in their second year of TRAC+ intervention and would therefore be expected to show the effects from TRAC+ year 1 interventions. However, the re-selection of Comparison schools took place and corrected this meaning that the baselines are approximately equal.
18
Table 8.2.1: TRAC+ Intervention (T) against Comparison (C) schools for EGRA sub-components at Grade 2 EL SubComp
T/C Baseline
Endline Data: T against C SCORES
Raw Score Raw Score (unit) (unit)
T>C
1. LNR
T 18.3 (LPM) 32.8 (LPM) T>C C 18.5 (LPM) 28.8 (LPM) TRUE **
2. FWR
T C T C T C
3. NWR 4. SR
3.2 (WPM) 3.5 (WPM) 2.2 (WPM) 1.9 (WPM) 4.1 (WPM) 3.9 (WPM)
T>C Score
Endline Data: T against C IMPROVEMENT Cohen’s ‘d’ Students score >0 (%) (effect) T > C Raw imp. T>C T>C T>C BL EL Imp. % diff. (unit) Score % diff.
+4.0 (LPM)
+14%
13.6 (WPM) T>C +0.4 13.2 (WPM) TRUE (WPM) 7.8 (WPM) T>C +2.0 5.8 (WPM) TRUE ** (WPM) 18.8 (WPM) T>C -0.5 19.3 (WPM) FALSE (WPM)
+2%
12.5 (LPM) T>C +2.2 (LPM) 10.3 (LPM) TRUE **
10.4 (WPM) T>C 9.7 (WPM) TRUE +35% 5.6 (WPM) T>C 3.7 (WPM) TRUE * -3% 14.7 (WPM) T>C 15.4 (WPM) FALSE
5. ORF
T 5.4 (WPM) 16.8 (WPM) C 4.5 (WPM) 16.4 (WPM)
T>C TRUE
+0.4 (LPM)
+2%
11.4 (WPM) 11.9 (WPM)
6. RCO 18 Adjust
T C
0.23 (CA) 0.18 (CA)
0.90 (CA) 0.86 (CA)
T>C TRUE
+0.04 (CA)
+4%
7. LCO
T C
1.10 (CA) 1.13 (CA)
2.01 (CA) 1.93 (CA)
T>C TRUE
+0.16 (CA)
+8%
T>C FALSE
+0.7 (WPM) +1.7 (WPM) -0.9 (WPM)
+21%
0.45 (SMALL)
87.5% 96.0% 8.5% 88.8% 93.5% 4.7%
+5%
0.22 (SMALL) 0.39 (SMALL) -0.02 (NULL)
31.5% 35.2% 35.8% 29.6% 40.1% 30.2%
+44% -6%
30.8% 20.5% 57.0% 48.5% 64.0% 63.5%
-0.7% -14.7% +21.2% +18.9% +23.9% +33.3%
-0.5 (LPM)
-4%
-0.02 (NULL)
28.4% 59.5% +31.1% 27.9% 58.0% +30.1%
0.67 (CA) 0.68 (CA)
T>C -0.01 (CA) FALSE
-1%
-0.04 (NULL)
16.1% 40.3% +24.2% 15.6% 35.0% +19.4%
0.91 (CA) 0.80 (CA)
T>C TRUE
+14%
0.03 (NULL)
68.9% 96.2% +27.3% 71.5% 94.5% +23.0%
+0.11 (CA)
Levels of significance: (#) = P<0.1 * = P<0.05 ** = P<0.01 *** = P<0.001 Figures highlighted in green do not seem consistent with related data. The reason for this has not been found. Comparing TRAC+ intervention with Comparison schools, positive effects can be seen in the phonics sub-components (1-3) where significant endline results and improvements can be seen in LNR (#1) and NWR (#3); and significant endline results in FWR (#2). The effect of these impacts may be measured as ‘small’ (d-values 0.22-0.45). Comparison schools show slightly better improvements in the sub-components related to reading comprehension; this is an area for TRAC+ to investigate further. Graph 8.2.1: EGRA results of Grade 2 EL improvements
140% 120% 100% 80% 60% 40% 20% 0%
Grade 2: EGRA Endline IMPROVEMENTS T vs C (C normalised at 100% for comparison)
Graph 8.2.1 (left) shows the tabular data in bar chart form where the Comparison school figure has been normalised to 100% across all sub-components so that differences between BL and EL can be compared across sub-components. The raw scores, as in Grade 1, show very low reading attainment in the sampled population. The end of Grade 2 represents 988 literacy teaching hours completed. Word reading (T) range is 13.6-18.8 WPM across sub-components. Word reading (C) range is 13.2-19.3 WPM across sub-components.
LNR
FWR NWR SR ORF RCO LCO TRAC+ Intervention Comparison Although the results show improvements for Grade 2 students sampled in TRAC+ Intervention schools against Comparison, when compared with Grades 1 and 3 the level of improvement is not as high. There are various issues that can be considered: • Implementation of the TRAC+ year 2 interventions took place after the start of the academic year at a time after the students had completed the Grade 1 revision lessons where the TRAC+ remedial work should be the most effective. • TRAC+ is more geared towards decoding words phonetically. This grounding in basic phonics skills should be properly established in Grade 1, although the Grade 2 baseline figures clearly show that is not the case.
18
For comparison, adjusted for five questions on baseline and four on endline. Endline scores are multiplied by 1.25 (5/4).
19
The largest percentage improvement result in Grade 2 is in sub-component 3 (NWR), looking at words that can be decoded phonetically but do not carry any meaning (e.g. fot, nam, teg in English). This is the clearest measure of students’ phonic decoding ability but it does not directly affect reading for meaning. Because the Grade 3 EGRA (comparing Intervention and Comparison schools) results are better than the Grade 2 results, and TRAC+ interventions should become less effective as students progress, Grade 2 results were analysed at provincial level as well. Table 8.2.2 shows the breakdown by provinces, looking purely at improvements from BLEL. In the table, figures highlighted in yellow (e.g. BMC (T) LNR = 12.13) are where TRAC+ Intervention schools show an improvement of >10% over Comparison schools, i.e. yellow is a ‘strong’ positive. The converse, where Comparison schools show an improvement BLEL of >10% over TRAC+ Intervention schools are highlighted in red, i.e. red is a ‘strong’ negative (e.g. BMC (C) LCO = 0.62). Pink highlighter is used for when figures are within 10%, the higher figure being highlighted (pale pink for T>C, dark pink and white text for T
T or C
LNR
FWR
NWR
SR
ORF
RCO
LCO
Total ***
***
T improvement
12.13
8.95
3.77
12.59
11.95
0.43
0.46
C improvement
5.95
6.19
3.74
12.24
8.50
0.43
0.62
* *
T improvement
16.32
13.15
6.44
18.77
16.96
1.05
1.39
**
C improvement
17.72
15.11
3.50
23.09
16.36
1.11
0.94
**
**
T improvement
17.42
10.36
6.51
14.27
14.71
0.75
1.02
***
*
C improvement
10.33
8.99
6.11
14.90
13.33
0.87
1.31
**
*
T improvement
13.49
11.85
6.18
15.65
13.80
0.67
0.71
*
C improvement
18.07
18.95
5.25
24.10
21.66
1.25
0.99
******
T improvement
18.46
11.20
6.71
13.67
14.35
0.72
1.09
C improvement
5.83
5.09
1.90
7.42
7.06
0.23
0.44
T improvement
12.50
10.40
5.60
14.70
11.40
0.67
0.90
******** ***
*
*** C improvement 10.30 9.70 3.70 15.40 11.90 0.68 0.80 The small sample size means that the above figures are not statistically valid. The indication is that Comparison schools in Siem Reap (and to a lesser extent Kampong Thom) had relatively good improvements on the EGRA EL test. Siem Reap has a relatively very high level of NGO support (see section 8.4 page 24). In contrast, Comparison schools in Takeo have relatively low improvements on the EGRA EL test. Takeo has a relatively very small level of NGO support (ibid.). 8.2.1
Gender factors at Grade 2
Graphs 8.2.1.1 to 8.2.1.7 are presented in to compare outcomes of boys and girls sampled in each EGRA sub-component. For each graph, the first two columns (from the left) are the figures for the boys in TRAC+ Intervention and Comparison (i.e. M (T) is Male (TRAC+ Intervention)) and the final two columns are the figures for the girls (F(T) and F(C.)). The blue part of each bar is the BL score. The orange part of each bar is the improvement (IMP) from BLEL. Thus, the total height of each bar is the EL figure. All figures are raw scores and use the units as previous explained for each sub-component (Table 4.1.2, page 10). Graph 8.2.1.1: LNR G2 boys and girls 40 30 20
LNR (IMP)
10
LNR (BL)
0 M M (T) (C.)
F F (T) (C.)
20
Graph 8.2.1.2: FWR G2 boys and girls
Graph 8.2.1.3: NWR G2 boys and girls 10.0
20.0
25.0
8.0
15.0
20.0
6.0
10.0 5.0
FWR (IMP)
4.0
NWR (IMP)
FWR (BL)
2.0
NWR (BL)
M M (T) (C.)
M M (T) (C.)
F F (T) (C.)
Graph 8.2.1.5: ORF G2 boys and girls
1.2
20.0
1.0 ORF (IMP)
10.0
ORF (BL)
5.0 0.0
Graph 8.2.1.7: LCO G2 boys and girls
2.0
0.6
RCO (IMP)
0.4
RCO (BL)
0.2
F F (T) (C.)
F F (T) (C.)
2.5 1.5 1.0
LCO (IMP)
0.5
LCO (BL)
0.0
0.0 M M (T) (C.)
SR (BL)
5.0 M M (T) (C.)
0.8
15.0
SR (IMP)
10.0
F F (T) (C.)
Graph 8.2.1.6: RCO G2 boys and girls
25.0
15.0
0.0
0.0
0.0
Graph 8.2.1.4: SR G2 boys and girls
M M (T) (C.)
F F (T) (C.)
M M (T) (C.)
F F (T) (C.)
The main conclusions from these results are: • • • 8.3
Girls out-performed boys in sub-components 1-6 in both TRAC+ Intervention and Comparison schools. In sub-component 1 (LNR), boys from the TRAC+ Intervention schools appear to be benefitting more than the girls from the TRAC+ intervention. In sub-component 6 (RCO), boys from the Comparison schools are out-performing boys from the TRAC+ Intervention schools; it may be that this is an effect of other NGO interventions (See 8.4, page 24) having (or targeting) a greater effect on boys. GRADE 3
Summary • TRAC+ Intervention schools out-perform Comparison schools in all seven sub-components in EGRA endline (and 6/7 at BL). • TRAC+ Intervention schools out-perform Comparison schools in all seven sub-components in improvements BLEL. o The improvement scores in two phonics sub-components (1 and 3) were significantly better (#1 P<0.01, #3 P<0.001) in Intervention cf. Comparison. • The effect sizes showed small effects in FWR (#2, d = 0.35), ORF (#5, d = 0.26) and RCO (#6, d = 0.24); a medium effect in LNR (#1, d = 0.53); and a large effect in NWR (#3, d = 0.83). • TRAC+ year 3 can investigate whether the above results are directly due to project intervention; length of intervention; sampling bias or other external factors. • Results across provinces, although not statistically robust, showed large variations: o Comparison schools in Siem Reap province performed well when compared with TRAC+ Intervention schools. o In the four other target provinces, TRAC+ Intervention schools performed well when compared with Comparison schools.
•
Gender differences: Girls out-perform boys in sub-components 1-6 in both TRAC+ Intervention and Comparison schools (as with Grade 2); this effect is significant (P<0.05) for BL and EL figures. There are too many variables to compare improvements with any reliability.
21
•
In sub-components 1-6, girls from the TRAC+ Intervention schools appear to be benefitting more than the boys from the TRAC+ intervention.
Grade 3 schools consist only of schools in Group A, i.e. those with a full TRAC+ intervention including mLearning. However, the mLearning App does not have content specific to the Grade 3 Khmer language curriculum; students who still require support from the Aan Khmer App should not have been promoted to Grade 3 as they will not have the basic reading competency required to access the curriculum. Because of changes in the Comparison group and some missing data, the number of schools analysed for Grade 3 is 116 in total, made up of 71 Group A schools and 45 Comparison schools; all schools with incomplete data (e.g. just BL or just EL) were removed from the analysis after data cleaning. Grade 3 students in TRAC+ Intervention schools should be in their third year of TRAC+ intervention and would therefore be expected to show the effects from TRAC+ year 1 interventions. There are two factors that reduce the clarity of this situation: (1) Selection bias: Group A schools were selected at the start of the project as being ‘potentially stronger’ and ‘likely to respond 19 better with TRAC+ interventions’ . Thus the BL figures in Grade 3 show TRAC+ Intervention schools are slightly stronger than Comparison schools. (2) Comparison school re-selection took place between TRAC+ year 1 and year 2. The above points mean that greater emphasis has to be placed on the improvement (BLEL) figures rather than the raw EL scores. In Table 8.3.1 below, the absolute EGRA scores are low (as in Grades 1-2). Later analysis will highlight the concern over the number of struggling students (low ability range) who show a lack of reading competencies to access the correct grade level curriculum (discussion in section 8.5, page 25). Table 8.3.1: TRAC+ Intervention (T) against Comparison (C) schools for EGRA sub-components at Grade 3 EL SubComp.
T/C
1. LNR 2. FWR
Baseline
Endline Data: T against C SCORES
Endline Data: T against C IMPROVEMENT Cohen’s Students score >0 (%) ‘d’ T > C Raw imp. T>C T>C T > C (effect) BL EL Imp. % diff. (unit) Score % diff.
Raw Score (unit)
Raw Score (unit)
T>C
T>C Score
T C
28.7 (LPM) 25.7 (LPM)
47.2 (LPM) 37.0 (LPM)
T>C TRUE (n/a)
+12.2 (LPM)
+33%
T C T C T C T C
10.3 (WPM) 8.1 (WPM) 5.2 (WPM) 4.8 (WPM) 15.6 (WPM) 11.9 (WPM) 19.0 (WPM) 14.6 (WPM)
23.5 (WPM) 17.8 (WPM) 13.8 (WPM) 8.4 (WPM) 31.7 (WPM) 26.7 (WPM) 32.1 (WPM) 24.6 (WPM)
T>C TRUE (n/a) T>C TRUE (n/a) T>C TRUE (n/a) T>C TRUE (n/a)
+5.7 (WPM) +5.6 (WPM) +5.0 (WPM) +7.7 (WPM)
+32% 13.2 (WPM) T>C 9.7 (WPM) TRUE (#) +67% 8.6 (WPM) T>C 3.7 (WPM) TRUE *** +19% 15.8 (WPM) T>C 14.8 (WPM) TRUE +31% 13.2 (WPM) T>C 10.9 (WPM) TRUE
6. RCO 20 Adjust
T C
1.12 (CA) 0.94 (CA)
1.69 (CA) 1.30 (CA)
T>C TRUE (n/a)
+0.39 (CA)
+30%
0.56 (CA) 0.36 (CA)
T>C TRUE
+0.20 (CA)
+56%
0.24 44.3% 57.8% 13.5% (SMALL) 42.6% 54.4% 11.8%
7. LCO
T C
1.70 (CA) 1.66 (CA)
2.10 (CA) 2.04 (CA)
T>C TRUE (n/a)
+0.06 (CA)
+1%
0.43 (CA) 0.35 (CA)
T>C TRUE
+0.08 (CA)
+23%
0.12 87.8% 96.6% 8.8% (NULL) 90.1% 94.4% 4.3%
3.NWR 4. SR 5. ORF
Levels of significance:
(#) = P<0.1
* = P<0.05
18.5 (LPM) 11.3 (LPM)
** = P<0.01
T>C TRUE **
+7.2 (LPM)
+64%
0.53 96.4% 96.9% 0.5% (MED.) 91.4% 95.0% 3.6%
+3.5 +36% 0.35 61.8% 80.3% 18.5% (WPM) (SMALL) 54.3% 78.3% 24.0% +4.9 +132% 0.83 51.4% 71.1% 19.7% (WPM) (LARGE) 48.1% 64.4% 16.3% +1.0 +7% 0.09 56.4% 75.9% 19.5% (WPM) (NULL) 45.7% 72.8% 27.1% +2.3 +21% 0.26 62.9% 80.3% 17.4% (WPM) (SMALL) 58.6% 76.7% 18.1%
*** = P<0.001
Where BL scores are significantly different, tests for significance have only been applied to improvements (BLEL). Graph 8.3.1 shows the tabular data in bar chart form where the Comparison school figure has been normalised to 100% across all sub-components so that differences between baseline and endline can be compared across sub-components. Graph 8.3.1: EGRA results of Grade 3 EL improvement 19 20
Internal WEI communication For comparison, adjusted for five questions on baseline and four on endline. Endline scores are multiplied by 1.25 (5/4).
22
Graph 8.3.1 (left) shows the tabular data in bar chart form where the Comparison school figure has been normalised to 100% across all sub-components so that differences between BL and EL can be compared across subcomponents.
Grade 3: EGRA Endline Improvements T vs C (C normalised at 100% for comparison) 250% 200%
Endline figures are not presented graphically because of the variation in BL figures.
150% 100% 50% 0% LNR
FWR
NWR
SR
TRAC+ Intervention 8.3.1
ORF
RCO
LCO
Further analysis could take place as to whether percentage improvements of student scores are a better measure. Students starting from a higher BL should be expected to achieve a higher absolute improvement.
Comparison
Gender factors at Grade 3
Graphs 8.3.1.1 to 8.3.1.7 are presented in to compare outcomes of boys and girls sampled in each EGRA sub-component. For each graph, the first two columns (from the left) are the figures for the boys in TRAC+ Intervention and Comparison (i.e. M (T) is Male (TRAC+ Intervention)) and the final two columns are the figures for the girls (F(T) and F(C.)). The blue part of each bar is the BL score. The orange part of each bar is the improvement (IMP) from BLEL. Thus, the total height of each bar is the EL figure. All figures are raw scores and use the units as previous explained for each sub-component (Table 4.1.2, page 10). Graph 8.3.1.1: LNR G3 boys and girls 60.0 50.0 40.0 30.0 20.0 10.0 0.0
LNR (IMP) LNR (BL) M M (T) (C.)
F F (T) (C.)
Graph 8.3.1.2: FWR G3 boys and girls 30.0 25.0 20.0 15.0 10.0 5.0 0.0
FWR (IMP) FWR (BL) M M (T) (C.)
F F (T) (C.)
8.3.1.5: ORF G3 boys and girls
Graph 8.3.1.3: NWR G3 boys and girls
Graph 8.3.1.4: SR G3 boys and girls
20.0
50.0
15.0
40.0 NWR (IMP)
10.0 5.0
NWR (BL)
0.0 M M (T) (C.)
F F (T) (C.)
Graph 8.3.1.6: RCO G3 boys and girls
30.0 20.0
SR (IMP)
10.0
SR (BL)
0.0 M M (T) (C.)
F F (T) (C.)
Graph 8.3.1.7: LCO G3 boys and girls
23
40.0 30.0
2.5
2.5
2.0
2.0 1.5
1.5
20.0
ORF (IMP)
10.0
ORF (BL)
0.0 M M (T) (C.)
F F (T) (C.)
1.0
RCO (IMP)
1.0
LCO (IMP)
0.5
RCO (BL)
0.5
LCO (BL)
0.0
0.0 M M (T) (C.)
F F (T) (C.)
M M (T) (C.)
F F (T) (C.)
The main conclusions from these results are: • • •
•
Girls out-perform boys in sub-components 1-6 in both TRAC+ Intervention and Comparison schools (as with Grade 2); this effect is significant (P<0.05) for BL and EL figures. There are too many variables to compare improvements with any reliability. In sub-components 1-6, girls from the TRAC+ Intervention schools appear to be benefitting more than the boys from the TRAC+ intervention (direct contrast with Grade 2 where the same trend is observed with boys improving more than girls). In sub-components 4 (SR) and 6 (RCO), boys from the Comparison schools out-performed boys from the TRAC+ Intervention schools on both EL performance and improvement; it may be that this is an effect of other NGO interventions (See 8.4, page 24) having (or targeting) a greater effect on boys. In sub-component 7 (LCO), boys from the Comparison schools are out-performing boys from the TRAC+ Intervention schools in their improvement BLEL although the difference is small.
8.4
NGO support
If results in Grades 2 and 3 are disaggregated by province, Comparison schools in Siem Reap province show substantially greater progress BLEL than schools in other provinces. This may be an effect from external factors, such as levels of NGO support in the area. Data on levels of NGO support is not accurately kept at any central of provincial level. The most relevant data found comes from the 21 Economic Census of Cambodia 2011 (CEC) provincial level reports . For comparison, the relevant figures that can be considered are: • • •
Population Number of establishments owned/rented by NGOs Number of person engaged belonging to the establishment rented by NGO
The CEC data is given in two formats: • •
As a percentage of the total for Cambodia Ranked by the (then) 24 provinces/municipalities within Cambodia, where #1 is the highest percentage ranking.
The CEC data is displayed in table 8.4.1 below.
21
Ministry of Planning (2013) Economic Census of Cambodia 2011 Provincial Reports (#01 BMC, #6 KTM, #13 PVH, #17 SRP, #21 TKO) National Institute of Statistics supported by JICA
24
Table 8.4.1: Province relative populations with relative NGO provision Province
Population Percentage
Number of establishments owned/rented by NGO
Rank (/24)
Percentage
Number of persons engaged belonging to the establishment rented by NGO
Rank (/24)
Percentage
Rank (/24)
SRP
6.7
6
8.9
3
13.8
2
TKO
6.3
7
0.9
20
0.4
21
BMC
5.1
9
6.1
4
8.6
4
KTM
4.7
10
3.5
7
3.5
7
PVH
1.3
18
0.8
21
0.5
20
The figures from table 8.4.1 above can be used to generate a ratio of NGO staff population to provincial population (e.g. SRP: 13.8/6.7 = 2.05, TKO: 0.4/6.3 = 0.06) and if these are considered as a part of the five target provinces, Siem Reap has a very high relative provision of NGO support (41.7% from the five provinces) and Takeo a very low relative provision of NGO support (1.3%). Although this figure does not account for (1) the type or scale of NGO funding or (2) differences between districts within provinces, it does nevertheless give a rough indication of the differing levels of NGO support for the target provinces and especially the relative 22 lack of NGO presence in Takeo province . Chart 8.4.1 below shows the data in pie chart form. Chart 8.4.1: Proportion of NGO staff adjusted for province populations
8.5
Promotion of struggling students
A barriers facing EGR in Cambodia is the requirement for schools to meet MoEYS-mandated targets for Child Friendly School (CFS) status. CFS can be given as: basic (lowest), intermediate or developed (highest). In order to achieve CFS ‘Developed’ status, primary schools require a promotion rate (pass rate) of 93% or better. For intermediate status, the requirement is for 90% (refer to Annex 4 for more details).
22
A possible reason is the high level of manufacturing industry in Takeo shows a ‘lack of need’ when NGOs evaluate provinces for possible project interventions.
25
23
Based on the small sample used in this survey and a general review of EGRA and student performance within Cambodia, these promotion figures do not reflect what is currently being attained: students are being promoted without sufficient reading skills to access the curriculum in their new grade. A solution to this problem is beyond the scope of TRAC+ project; highlighting the problem (once again) is another small step towards meaningful education quality improvement in Cambodia. For reasons of space, this report will consider the two most basic subcomponents of the EGRA. Data and graphs for all sub-components are available from WEI. Sub-component (1) and (3) will be looked at in detail: Sub-component
Rationale
(1) Letter Name Recognition (LNR)
The majority of the Grade 1 MoEYS textbook is concentrated on teaching consonants and vowels and how these are combined together into syllables and words. Without letter recognition, any ability to decode words or recognise familiar words is almost inaccessible. This sub-component should be the purest test of phonics (word decoding) ability. (Although in the EGRA tool, poor word selection limits the effectiveness of this sub-component.)
(3) Non-Word Reading (NWR)
The distribution of the raw scores of individual students allows an analysis of whether students can actually identify and combine letters with sufficient speed and accuracy to begin to access higher-level reading skills. The curriculum focus on letter recognition and word building means that by the end of Grade 1, a successful student should have a competent grasp of the Khmer alphabet and some ability to put letters together to make syllables and words. Figures for LNR and NWR will be shown by grade. Grade 1 Graph 8.5.1: LNR Grade 1 EL (T vs. C)
Graph 8.5.2: NWR Grade 1 EL (T vs. C) Non-Word Reading: Grade 1 Endline
60.0% 50.0%
Percent of students
Percent of students
Letter Name Recognition: Grade 1 Endline
40.0% 30.0% 20.0% 10.0% 0.0% Comparison
Intervention
LPM
100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% Comparison
Intervention
WPM
At the end of Grade 1, after 494 teaching hours, only 60% of TRAC+ Intervention students and 51% of Comparison students sampled were able to correctly identify ten Khmer characters in a minute. 15% of students in TRAC+ Intervention schools scored zero, 22% of students in Comparison schools scored zero. It does not seem possible that any of the students scoring zero should be able to pass onto Grade 2. This is an issue that requires further investigation. Non-Word Reading remains beyond the capacity of the majority of Grade 1 students.
23
Further info. see e.g.: RTI International (2015) Assessment of Early Grade Reading in the Education Sector in Cambodia Report for USAID
26
Grade 2 Graph 8.5.3: LNR Grade 2 BL and EL Intervention (T)
Graph 8.5.4: LNR Grade 2 BL and EL Comparison (C) Letter Name Recognition: Grade 2 Comparison
45.0% 40.0% 35.0% 30.0% 25.0% 20.0% 15.0% 10.0% 5.0% 0.0%
Percent of students
Percent of students
Letter Name Recognition: Grade 2 Treatment
Endline
Baseline
45.0% 40.0% 35.0% 30.0% 25.0% 20.0% 15.0% 10.0% 5.0% 0.0% Endline
LPM
Baseline
LPM
The LNR baseline at Grade 2 for both TRAC+ Intervention and Comparison schools shows a baseline of 60% of students able to read ten letters in 60 seconds. This correlates reasonably well with the Grade 1 endline figures. Although the 40% of students without this basic competency have the first 31 lessons of the Grade 2 curriculum to revise this basic content, it seems that these students are arriving in Grade 2 without the necessary baseline competency to access the curriculum. The TRAC+ Intervention schools show an improvement over the Comparison schools in terms of reducing the most struggling students by the endline. The number of students in the 0-10 LPM category is 18% (T) compared with 25% (C). If an LNR score of 20LPM is taken as an arbitrary minimum for promotion to Grade 3 (WEI suggests that 30LPM would be more appropriate) there are only 67% of TRAC+ Intervention students and 58% of Comparison students sampled with sufficient basic knowledge of the Khmer alphabet to begin to access the Grade 3 curriculum. These figures require further investigation and confirmation; but, if they are a true reflection of the situation, the data shows insufficient effectiveness of the TRAC+ intervention in addressing the problem of students with serious reading difficulties. Graph 8.5.5: NWR Grade 2 BL and EL Intervention (T)
Graph 8.5.6: NWR Grade 2 BL and EL Comparison (C)
90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
Non-Word Reading: Grade 2 Comparison
Percent of students
Percent of students
Non-Word Reading: Grade 2 TRAC+ Intervention
Endline
Baseline
WPM
90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
Endline
Baseline
WPM
The number of Grade 2 students who could not read five ‘non-words’ in a minute is partly related to the poor quality of the EGRA tool being used. However, TRAC+ Intervention schools show 32% of students sampled were able to decode ten or more non-words
27
in a minute, so this level of phonic decoding of words is quite achievable in this grade. Full analysis of this problem cannot take place until the EGRA tool is revised, but Graphs 8.5.5 and 8.5.6 show that the problem clearly exists. Grade 3 Graph 8.5.7: LNR Grade 3 BL and EL Intervention (T)
Graph 8.5.8: LNR Grade 3 BL and EL Comparison (C) Letter Name Recognition: Grade 3 Comparison
30%
30%
25%
25%
Percent of students
Percent of students
Letter Name Recognition: Grade 3 Treatment
20% 15% 10% 5%
20% 15% 10% 5% 0%
0%
Correct Letters Per Minute Endline
Baseline
Correct Letters Per Minute Endline
Baseline
Both TRAC+ Intervention and Comparison school groups at BL show a similar proportion (57-58%) of sampled students able to read 20 letters in one minute (i.e. one letter every three seconds) when entering Grade 3. This figure correlates with the Grade 2 Comparison EL figures. The 10% of TRAC+ Intervention students and 17% of Comparison students from the sample who cannot read 10 letters correctly in one minute will have no possibility of accessing the Grade 4 curriculum in any subject and are highly unlikely to have been able to access the Grade 3 curriculum either. The TRAC+ Intervention schools show an improvement over the Comparison schools in terms of reducing the most struggling students by the endline. The number of students in the 0-30 LPM category is 31% (T) compared with 37% (C). Thus, if an LNR score of 30LPM is taken as an arbitrary minimum for promotion to Grade 4 (and the author feels that 50LPM would be more appropriate) a promotion rate of 60-70% seems likely.
28
Graph 8.5.7: NWR Grade 3 BL and EL Intervention (T)
Graph 8.5.8: NWR Grade 3 BL and EL Comparison (C) Non-Word Reading: Grade 3 Comparison
80%
80%
70%
70%
Percent of students
Percent of students
Non-Word Reading: Grade 3 TRAC+ Intervention
60% 50% 40% 30% 20% 10%
60% 50% 40% 30% 20% 10% 0%
0%
WPM
WPM Endline
Baseline
Endline
Baseline
Phonetically decoding non-words shows improvement in TRAC+ Intervention against Comparison samples. However, only 63% of TRAC+ Intervention students sampled and 55% of Comparison students sampled are able to decode five words in a minute. As stated before, this is also partly a reflection on the quality of the EGRA tool being administered, but WEI considers that these students do not have the fundamental reading competencies to progress further through the curriculum. The MoEYS presented a long-term aim in 2014 that: “Every child who successfully completes Grade 3 will be able to read
with understanding at a level of 45 – 60 words per minute.” 24 The results shown in this sample suggest that this aim is a long way from being achieved. 9
Recommendations for TRAC+ Year 3 and beyond
The recommendations for improvement within the EGRA test are difficult, as alterations to the testing protocol will invalidate comparing TRAC+ year 2 and year 3 data. However, if an improved EGRA test was available in time to create a TRAC+ year 3 BL figure, this would improve data reliability. TRAC+ does not have the human resources to modify, pilot and validate an improved EGRA test. Basic flaws within the EGRA where correction is required are listed in Table 5.1.1 (page 12). Recommendation 1: EGRA for TRAC+ year 3 to use improved EGRA test IF such a resource is available in time. Comparison schools were not selected with sufficient rigour; however, it is practical to continue using the current group of Comparison schools for the final year of TRAC+. The lack of validity of Comparison school selection will need to be taken into consideration in any TRAC+ evaluation. Recommendation 2A: No change in school selection for TRAC+ year 3. Recommendation 2B: Increase sample size in Comparison schools to enable more robust analysis of results. Recommendation 2C: Comparison school selection issues noted in subsequent EGRA or other performance analysis.
24
Kann Puthy (2014) Variations in early reading assessment results between rural and urban populations presented to UNESCO Institute for Statistics conference ‘What works in oral assessments of early reading?’ 21 July 2014, Montreal, Canada on behalf of MoEYS, Cambodia (http://tinyurl.com/hop9538 retrieved 09-Oct-2016)
29
Although the EGRA assesses all sub-components of reading, the TRAC+ interventions (RRS and the use of LCs) primarily address the most basic reading ability: the ability to decode words using letter-sound association and combining sounds. In effect, the intervention is almost entirely phonics-based. Although improvements in decoding words can lead to increased reading fluency, a total reading approach for children would require further interventions using correctly levelled connected text for comprehension and fluency especially in Grade 2 (from lesson 32 onwards) and as a matter of extreme urgency in Grade 3 where basic phonetic knowledge and the ability to decode words is assumed and required to access the Khmer language curriculum (and indeed other core subjects). Recommendation 3A: Additional ‘connected text’ resources (e.g. mini-libraries) to be provided as RRS interventions. (This can be linked with the RRS interventions in the Reading Benchmark Standards books, i.e. specific books can be linked as remedial interventions for specific reading intervals.) Recommendation 3B: Additional levelled readers to be distributed in multiple sets to allow paired or group reading activities. Students are being promoted with insufficient reading ability to access the curriculum at a new grade; this becomes more noticeable at higher grades and there is pressure on SDs to achieve MoEYS-mandated promotion figures as these are key indicators to achieving higher status as a Child-Friendly School (CFS). Students who lack basic reading skills require increased remedial work: this could mean a return to long vacation remedial classes or greater emphasis on RRS or other similar interventions over the academic year. Recommendation 4: TRAC+ to present relevant EGRA data to MoEYS with priority of stopping promotion of students who do not have the requisite competencies to access the next grade. Some crucial RRS interventions were missed in the early part of the academic year 2015-2016 because of the delay in beginning the implementation of TRAC+ year 2 activities. The start of the year is especially crucial as: • •
Students can fall further behind if problems are not picked up quickly. In Grade 2, the first 31 lessons are revision of Grade 1 and are the foundation for the rest of the teaching in that Grade.
Recommendation 5: TRAC+ activities to start promptly at the beginning of the 2016-17 academic year. All recommendations are made on behalf of WEI as a basis for further discussion with stakeholders in EGR in Cambodia.
30
Annex 1: Sample Population for Testing Table A1.1: Sample population for Grade 1 Endline
Total Population Province
District
TRAC+ Intervention
Sample Population
Comparison
TRAC+ Intervention
Comparison
T
F
T
F
T
F
T
F
Preah Neitr Preah
399
189
149
70
37
22
16
8
Phnom Srok
434
200
158
68
48
24
22
9
Svay Check Total in Banteay Meanchey
434
218
300
146
48
25
29
15
1267
607
607
284
133
71
67
32
638
299
106
53
60
30
10
3
638
299
106
53
60
30
10
3
Kulen
383
179
243
113
59
29
21
12
Rovieng
547
249
143
72
63
28
7
4
Sangkum Thmei
378
170
109
47
51
27
6
3
439
160
269
70
23
12
6
3
1747
758
764
302
196
96
40
22
Chikreng
970
456
239
110
82
42
23
12
Puok Total in Siem Reap
701
339
87
46
53
25
12
6
1671
795
326
156
135
67
35
18
Kirivong
470
249
108
55
59
29
16
8
Borei Chulsar
353
133
109
50
29
12
13
5
477
215
264
121
54
28
19
9
1300
597
481
226
142
69
48
22
6623
3056
2284
1021
666
333
200
97
10.1%
10.9%
8.8%
9.5%
Banteay Meanchey
Kampong Thom
Prasath Ballang Total in Kampong Thom
Preah Vihear
Chey Sen Total in Preah Vihear Siem Reap
Takeo
Samrong Total in Takeo Grand Total
% of eligible students sampled
31
Table A1.2: Sample population for Grade 2 Baseline
Total Population Province
District
TRAC+ Intervention T F
Comparison T
F
Preah Neitr Preah
528
209
151
83
Phnom Srok
436
223
139
387
193
1351
Endline
Sample Population TRAC+ Comparis Intervention on T F T F
Sample Population TRAC+ Comparison Intervention T F T F
23
13
8
5
30
19
9
6
58
42
22
21
8
44
24
21
9
290
149
41
23
28
14
38
21
26
13
625
580
290
106
58
57
27
112
64
56
28
585
234
102
61
58
26
12
7
59
32
9
6
585
234
102
61
58
26
12
7
59
32
9
6
Kulen
294
139
213
106
30
13
8
4
68
32
21
11
Rovieng Sangkum Thmei
427
207
124
62
39
26
9
6
38
23
7
4
405
189
81
43
41
21
6
4
39
19
6
3
Chey Sen Total in Preah Vihear
325
136
167
66
25
14
6
4
24
16
6
3
1451
671
585
277
135
74
29
18
169
90
40
21
Chikreng
795
346
317
147
63
26
28
17
67
33
27
16
496
236
103
49
40
21
20
12
38
17
24
12
1291
582
420
196
103
47
48
29
105
50
51
28
497
221
94
43
42
17
11
5
44
20
14
7
361
153
107
59
27
14
6
3
23
14
14
6
451
218
245
109
43
19
16
6
41
21
16
7
1309
592
446
211
112
50
33
14
108
55
44
20
5987
2704
2133
1035
514
255
179
95
553
291
200
103
8.6%
9.4%
8.4% 9.2%
Banteay Meanchey
Svay Check Total in Banteay Meanchey Prasath Kampong Thom Ballang Total in Kampong Thom
Preah Vihear
Siem Reap
Puok Total in Siem Reap
Takeo
Kirivong Borei Chulsar
Samrong Total in Takeo Grand Total
% of eligible students sampled
9.2%
10.8% 9.4% 10.0%
32
Table A1.3: Sample population for Grade 3 All Grade 3 TRAC+ Intervention schools (Groups A and B) were sampled, even though this TRAC+ intervention is only applied to Group A schools. Total Population Province
Banteay Meanchey
District
TRAC+ Intervention T F
Comparison T
F
197
145
58
393
197
135
Preah Neitr Preah
385
Phnom Srok
Baseline
Endline
Sample Population TRAC+ Compariso Intervention n T F T F
Sample Population TRAC+ Comparison Intervention T F T F
21
10
10
3
28
16
18
9
65
38
18
19
11
38
18
19
11
382
192
260
141
36
19
28
19
37
23
29
19
1160
586
540
264
95
47
57
33
103
57
66
39
550
268
124
50
55
30
12
5
50
28
9
4
550
268
124
50
55
30
12
5
50
28
9
4
Kulen
332
151
151
73
29
18
7
5
62
33
17
12
Rovieng Sangkum Thmei
395
206
110
50
36
24
9
6
38
24
7
5
390
200
80
40
42
24
6
3
41
20
6
3
Chey Sen Total in Preah Vihear
340
142
138
54
22
11
6
3
21
11
5
3
1457
699
479
217
129
77
28
17
162
88
35
23
Chikreng
626
318
229
99
45
26
20
12
53
31
18
12
557
263
96
43
41
16
25
14
45
21
25
14
1183
581
325
142
86
42
45
26
98
52
43
26
Kirivong
518
242
97
52
45
20
13
6
43
20
13
7
Borei Chulsar
364
158
105
47
28
15
6
3
30
14
15
8
524
233
273
119
38
19
19
7
41
19
19
8
1406 5756
633 2767
475 1943
218 891
111 476
54 250
38 180
16 97
114 527
53 278
47 200
23 115
8.3%
9.0% 9.3% 10.9%
Svay Check Total in Banteay Meanchey Prasath Kampong Thom Ballang Total in Kampong Thom
Preah Vihear
Siem Reap
Puok Total in Siem Reap
Takeo
Samrong Total in Takeo Grand Total
% of eligible students sampled
9.2%
10.0% 10.3% 12.9%
33
Annex 2: Schools Table A2.1: School participating in TRAC+ as TRAC+ Intervention or Comparison Province District Banteay Meanchey
Schools (T)
(C)
Preah Neitr Preah
10
4
Phnom Srok
12
6
*
12
6
35
16
16
4
16
3
Kulen
11
3
Rovieng
11
3
Sangkum Thmei
13
2
7
2
42
10
Chikreng
10
5
Puok Total in Siem Reap
11
4
21
8
Kirivong
13
4
Borei Chulsar
5
4
7
3
25
11
138
50
Svay Check Total in Banteay Meanchey Kampong Thom
Preah Vihear
Prasath Ballang Total in Kampong Thom
Chey Sen Total in Preah Vihear Siem Reap
Takeo
Samrong Total in Takeo *
Overall Total One school was removed from Svay Check District in BMC at the end of TRAC+ year 1.
34
Annex 3: School level participants trained in EGRA orientation Table A3.1: Participants in EGRA orientation for the school year 2015-2016 Province BMC
KTM PVH
SRP TKO
Total
District Preah Neit Preah Phnom Srok Svay Chek Prasat Balang Cheysen Kulen Rovieng Sangkumtmey Chikreng Pouk Borei Chulsar Kirivong Samraung
SDs (F) 12(1) 16(1) 18(0) 17(1) 7(1) 9(0) 12(1) 15(2) 8(4) 7(0) 6(0) 13(2) 8(2) 148(15)
Teachers (F) 4(1) 0 1(0) 0 1(0) 0 0 0 6(3) 0 3(1) 1(0) 2(1) 18(6)
LC (F) 0 0 3(0) 0 0 0 0 0 1(0) 0 0 3(2) 5(0) 12(2)
POE/DOE (F) 1(1) 1(0) 1(0) 3(0) 2(0) 2(0) 2(0) 0 1(1) 1(1) 1(0) 1(0) 2(0) 18(3)
WVI (F) 1(0) 0 2(0) 0 1(0) 1(0) 1(0) 0 1(0) 0 0(0) 0 0 7(0)
Total (F) 18(3) 17(1) 25(0) 20(1) 11(1) 12(0) 15(1) 15(2) 17(8) 8(1) 10(1) 18(4) 17(3) 203(26)
POE = Provincial Office of Education DOE = District Office of Education
35
Annex 4: DTMT Guidelines for Basic Education [extract] Relevant text from DTMT Working Guidelines for Basic Education (working document, Oct. 2015) pages 13-15. Additional information added for clarity is in red italics. Identification (of CFS): In principle, CFSs are ranked into three different classifications: Basic Level CFS Refers to any school which has not achieved outcomes related to the six dimensions fully and where students’ annual learning outcomes are still low. Intermediate Level CFS Refers to any school which has achieved good outcomes related to the six dimensions and where students’ annual learning outcomes are average. Developed Level CFS Refers to any school which has well achieved outcomes related to the six dimensions fully and where students’ annual learning outcomes are quite good. Learning Outcome/Quality-based Assessment: In this type of assessment, key emphasis is on Net Admission Rate, Gross Enrolment Rate, Promotion Rate, Exam Passing Rate, Dropout Rate and Completion Rate in the academic year against targets set for each academic year. On this basis, we can rank CFS by qualities as follows: Developed Level CFS in terms of Quality: For any primary school which has achieved: - Net Admission Rate > 91% - Net Enrolment Rate > 93% - Repetition Rate < 5% - Dropout Rate < 2% - Completion Rate > 92% (Repetition Rate and Dropout Rate for Developed CFS status requires a Promotion Rate of >93%)
Intermediate Level CFS in terms of Quality: For any primary school which has achieved: - Net Admission Rate 85-91% - Net Enrolment Rate 90-93% - Repetition Rate 5-6% - Dropout Rate 2-4% - Completion Rate 85-92% (Repetition Rate and Dropout Rate for Intermediate CFS status requires a Promotion Rate of 90-93%)
Basic Level CFS in terms of Quality: - Net Admission Rate < 85% - Net Enrolment Rate < 90% - Repetition Rate > 6% - Dropout Rate > 4% - Completion Rate < 85% (All schools can achieve these figures as they are simply the cut-off before intermediate status can be reached)
36