� សា�ស� គេ្រមាង វធ សរុបេដម ងអំ ណានបែន�មស្រមាប់កមា ុ រ ើ ្បព្រង� ី ី TOTAL READING APPROACH FOR CHILDREN PLUS (TRAC Plus) គាំ្រទថវ ិកេដយ:

្របតិបត�ិេដយ:

Early Grades Reading Analysis (EGRA) Report for TRAC+ Year 2. EGRA Endline Analysis.

Report: September/October 2016 th FINAL DRAFT 3: 06 October 2016

(Cover photo: MoEYS EGRA training, March 2016)

1

Contents: Section 1 List of acronyms___________________________________________________________________________ 2 Executive summary________________________________________________________________________ 2.1 Aim of report______________________________________________________________________ 2.2 Introduction to TRAC+_______________________________________________________________ 2.3 EGRA sample, test and analysis_______________________________________________________ 2.4 Summary of main findings___________________________________________________________ 2.5 Summary of recommendations_______________________________________________________ 3 Introduction to TRAC+ and Aim of report______________________________________________________ 3.1 Introduction to TRAC+______________________________________________________________ 3.2 Aim of report______________________________________________________________________ 4 EGRA testing______________________________________________________________________________ 4.1 EGRA protocol_____________________________________________________________________ 4.2 TRAC+ effect on EGRA_______________________________________________________________ 5 Limitations of the EGRA_____________________________________________________________________ 5.1 Limitations of the EGRA test content___________________________________________________ 5.2 Use of the Tangerine app____________________________________________________________ 5.3 Absent students___________________________________________________________________ 5.4 Inconsistencies in the EGRA administration_____________________________________________ 5.5 Alterations in the EGRA test between baseline and endline________________________________ 6 Sampling and limitations on analysis__________________________________________________________ 7 Methods of analysis________________________________________________________________________ 8 Main findings_____________________________________________________________________________ 8.1 Grade 1___________________________________________________________________________ 8.1.1 Gender factors at Grade 1____________________________________________________________ 8.2 Grade 2___________________________________________________________________________ 8.2.1 Gender factors at Grade 2____________________________________________________________ 8.3 Grade 3___________________________________________________________________________ 8.3.1 Gender factors at Grade 3____________________________________________________________ 8.4 NGO factors_______________________________________________________________________ 8.5 Promotion of struggling students______________________________________________________ 9 Recommendations for TRAC+ year 3 and beyond________________________________________________ 10 Annexes_________________________________________________________________________________ Annex 1 Sample population for testing______________________________________________________ Annex 2 Schools________________________________________________________________________ Annex 3 School level participants trained in EGRA orientation___________________________________ Annex 4 DTMT Guidelines for Basic Education [extract]_________________________________________

Page 3 4 4 4 4 5 8 9 9 9 10 10 11 11 11 12 12 13 13 13 14 15 16 17 18 20 21 23 24 25 29 31 31 34 35 36

2

1.

List of Acronyms

ADP BL BMC C CA CEC CFS EGR EGRA EL F IMP (or Imp) KTM LC LPM M MoEYS PVH RRS SD SRP T TKO ToT TTS TRAC(+) USAID WEI WPM WVI

Area Development Programme (World Vision International administrative area unit) Baseline Banteay Meanchey (Province) Comparison (school without TRAC+ intervention) Correct Answers Cambodia Economic Census Child-Friendly School Early Grades Reading Early Grades Reading Assessment Endline Female (for participant numbers) Improvement (difference between baseline and endline, i.e. EL – BL, or written BLEL) Kampong Thom (Province) Literacy Coach Letters (read) Per Minute Male (for participant numbers) Ministry of Education Youth and Sports Preah Vihear (Province) Rapid Response System School Director Siem Reap (Province) TRAC+ school (Intervention school) Takeo (Province) Training of Trainers Thoun Thean Suksa (Translates as: learning resources – TTS make educational games. Also referred to as ‘learning toolkit’) Total Reading Approach for Children (Plus) United States Agency for International Development World Vision, Inc. (NGO) Words (read) Per Minute World Vision International (NGO) 1

EGRA sub-component abbreviations : LNR Letter Name Recognition FWR Familiar Word Reading NWR Non-Word Reading SR Sentence Reading ORF Oral Reading Fluency RCO Reading Comprehension LCO Listening Comprehension

1

For the purposes of graphical display.

3

2.

Executive Summary

2.1 Aim: •

Aim of Report

2.2

Introduction to TRAC+

The aim of this report is to evaluate the current performance of the Total Reading Approach for Children Plus (TRAC+) project interventions based on data provided by the Early Grades Reading Assessment (EGRA) endline survey performed at the end of the 2015-2016 academic year. Purpose: • The purpose of this report is to identify any strengths and weaknesses of the TRAC+ intervention that are displayed by the EGRA analysis and to use these to improve project outcomes for TRAC+ year 3. Outcomes: • Improved practice for TRAC+ year 3 at strategic (project management) level and at local (school, district, provincial) level. • Improved practice in EGRA for TRAC+ year 3. th • Relevant content disseminated for TRAC+ Mid-Term Evaluation (MTE) meeting planned for 27 September 2016. Target Audience: • TRAC+ collaborative partners: World Education, Inc. (WEI) and World Vision International (WVI). • Summary findings will be presented to the Ministry of Education, Youth and Sports (MoEYS) as appropriate and after agreement between WEI and WVI.

The TRAC+ project is a collaborative work between WEI, WVI and MoEYS. TRAC+ provides support for early grades students (Grades 1-3) in 138 primary schools in five provinces in Cambodia. TRAC+ aims relevant to this report are: Project goal: Improve early grade literacy outcomes Project outcomes: # 1: Schools with TRAC+ interventions report better reading achievement than comparable schools that do not use TRAC+ # 4: Project activities are effectively monitored+ To meet these targets, TRAC+ uses a system of Reading Benchmarks to help teachers evaluate the progress of their students as they work through MoEYS textbooks. Students who are unable to pass a benchmark test are given remedial support in the form of the Rapid Response System (RRS). Teachers are assisted in their testing of students and in the interventions by a TRAC+ appointed Literacy Coach (LC). This report focuses on the results from the EGRA tool used to assess specific reading competencies of students at a baseline (BL, November 2015) and endline (EL, July 2016) of the academic year. EGRA is administered using the specifically designed Tangerine App. TRAC+ is a three-year project, with the final year starting in October 2016 and project completion due in September 2017. The total number of students expected to benefit over the lifetime of the project is 19,822; with a project budget of $2.84m, this works out at a cost per beneficiary of $143.27. 2.3

EGRA sample, test and analysis

TRAC+ has 138 TRAC+ (T) Intervention schools, which are split into Group A (73 schools with support for Grades 1-3 that receive an additional mLearning intervention) and Group B (65 schools with support for Grades 1-2 and no mLearning intervention). These schools are compared with a selection of 50 Comparison (C) schools from within the five target provinces. Group A schools were 2 selected as schools that were ‘likely to respond better with TRAC+ interventions’ which presents difficulties in comparing Group A and Group B school learning outcomes. Comparison schools were selected according to those with similar EGRA BL figures to TRAC+ Intervention schools. The BL comparisons were made using incorrectly calculated EGRA scores and selection process was not rigorous enough to ensure a valid Comparison sample. All TRAC+ Intervention and Comparison schools undertook the EGRA, using sample of four children per Grade 1 2

Internal WEI communication

4

class and three each for other target classes. The total sample size used was 9.5% of the eligible student population (2346 students tested altogether for the endline). The EGRA test has seven sub-components as shown below: Sub-component

Early reading skill

Skill demonstrated by student’s ability to:

Output unit

1. Letter Name Recognition (LNR) 2. Familiar Word Reading (FWR) 3. Non-Word Reading (NWR) 4. Sentence Reading (SR)

Letter recognition

5. Oral Reading Fluency (ORF)

Oral reading fluency

Read a narrative text with accuracy, little effort, and at a sufficient rate

6. Reading Comprehension (RCO)

Reading comprehension

Respond to literal and inferential questions about the text read

Correct answers (CA)

7. Listening Comprehension (LCO)

Comprehension

Respond correctly to literal and inferential questions about a text read to the student

Correct answers (CA)

Alphabetic principle/decoding skills Alphabetic principle/decoding skills Reading fluency

Provide the name of graphemes presented in random order Use letter-sound correspondence to read a list of familiar words Use letter sound correspondence to read a list of non-words Read unrelated sentences fluently

Letters per minute (LPM) Words per minute (WPM) Words per minute (WPM) Words per minute (WPM) Words per minute (WPM)

TRAC+ interventions (the Rapid Response System (RRS)) focus on the first three sub-components of the EGRA test that relate to letter recognition and decoding words. The current version of the EGRA test has a number of problems which may be summarised briefly as: • • • • • •

Almost one-third (7/23) dependent vowels missing from sub-component 1 (LNR). Lack of alignment of content with MoEYS textbooks and omission of high-frequency phonetically decodable words. Inconsistent selection of words on sub-component 3 (NWR). Lack of standardisation between baseline and endline testing content. Sub-component 7 (LCO) contains questions that do not distinguish students who understand from those who do not. Inaccessibility of large parts of the test for the majority of Grade 1 students assessed.

EGRA results are analysed by sub-component and comparisons are made between endline scores (T vs. C) and by improvement (BLEL) made over the course of the school years (T vs. C).Gender differences will also undergo analysis. Analysis is by two-tailed t3 test with an α-value of 0.05. The second stage of analysis will seek to look at effect size using Cohen’s d test where a d-value of more than 0.2 may be considered a small effect; a value of more than 0.5 may be considered a medium effect; and a value of more than 0.8 may be considered a large effect. 2.4

Summary of main findings

Limitations on findings: All findings listed in this report should be considered as indications of the current situation; the level of data integrity available for analysis is insufficiently robust to deliver firm conclusions. Some trends based on these indications may later lead to valid conclusions, but any EGRA taking place in TRAC+ year 3 will need to be delivered with greater rigour. The following concerns should be taken into context with regard to any conclusions reached in this report: • Sampling was not performed according to the EGRA guidelines. • Comparison schools were not matched according to an accepted protocol. • EGRA training was of insufficient standard and duration to permit sufficient rigour in gaining test data. • The MoEYS-sanctioned EGRA tool contains too many basic errors to be considered a reliable indicator for a number of subcomponents. • Allocation of ‘better’ schools to Group A precludes analysis of Group A compared with Group B using the small sample size. Overall: • EGRA testing shows some improvements (BLEL) in TRAC+ Intervention schools compared with Comparison schools. 3 http://www.uccs.edu/lbecker/effect-size.html

5

• •

•

• • •

o Improvements are most marked in EGRA sub-components (1-3) relating to letter recognition and decoding words; this is observable across all three grades sampled and is consistent with the type of intervention applied. Grade 1 Intervention schools show significantly (P<0.001) higher results on all EGRA reading sub-components (1-6) compared with Comparison schools; the effect size is small in all instances (Cohen’s d = 0.28-0.39). Grade 2 Intervention schools show higher results and better improvements on all EGRA phonics sub-components (1-3) compared with Comparison schools; the difference is significant for LNR (#1, P<0.05) and NWR (#3, P<0.01) and the effect size is small in all three instances (d = 0.22-0.45). Grade 3 Intervention schools show higher results and better improvements on all EGRA sub-components (1-7) compared with Comparison schools; the difference in improvement is significant for LNR (#1, P<0.01) and NWR (#3, P<0.001). Effects in improvements are shown as: small in sub-components FWR (#2, d = 0.35), ORF (#5, d = 0.26) and RCO (#6, d = 0.24); medium in LNR (#1, d = 0.53); and large in NWR (#3, d = 0.83). Girls out-performed boys consistently in EGRA reading sub-components (1-6) of the EGRA test, regardless of interventions. The level of reading ability shown by many students is insufficient for a promotion rate in line with MoEYS expectations. The improvements in reading ability shown by TRAC+ interventions to date are relatively small.

Grade 1 (EGRA Endline only): • TRAC+ Intervention schools out-perform Comparison schools in all seven sub-components in EGRA endline. • The improvements in reading sub-components (1-6) were all highly significant (P<0.001) and the effect size was ranked as ‘small’ in all instances (d = 0.28-0.39). • Assessed Grade 1 students were unable to sufficiently access sub-component 6 (RCO) (over 90% of students scored zero). o Sub-components 2-5 were also inaccessible (zero scores) to over two-thirds of the sampled population. • The level of reading ability in Grade 1 students is so low that only letter recognition can be reliably analysed; this is a shortfall relative to the curriculum expectations where students should be able to read compound words at the end of Grade 1. • Girls out-performed boys in reading sub-components (1-6) of the EGRA test, regardless of interventions. Graphical results for Grade 1 (nb. ‘zero’ baseline assumed):

600%

Grade 1: EGRA EL results T vs C (C normalised at 100% for comparison)

Although percentage scores (T vs. C) are high, the absolute scores achieved by sampled students are very low. LNR (T) = 18.8 LPM LNR (C) = 13.3 LPM

500% 400%

Word (and non-word) reading (T) range is 2.7-3.4 WPM across subcomponents. Word (and non-word) reading (C) range is 1.2-1.3 WPM across subcomponents.

300% 200% 100% 0% LNR

FWR NWR SR TRAC+ Intervention

ORF RCO Comparison

LCO

Note that the very high (T) column for RCO is based on a very low absolute score (0.15 correct answers (T), 0.0125 correct answers (C)).

Grade 2: • TRAC+ Intervention schools out-perform Comparison schools in 6/7 sub-components in EGRA endline. • The endline scores in two phonics sub-components (1 and 3) were significantly better (P<0.01) in Intervention cf. Comparison. • TRAC+ Intervention schools out-perform Comparison schools in 4/7 sub-components in improvements BLEL. o The improvement scores in two phonics sub-components (1 and 3) were significantly better (#1 P<0.01, #3 P<0.05) in Intervention cf. Comparison. o The three phonics sub-components (1-3) showed a small effect size (d = 0.22-0.45) in improvements for Intervention cf. Comparison. • Results across provinces, although not statistically robust, showed large variations: o Comparison schools in Siem Reap and Kampong Thom performed well when compared with TRAC+ Intervention schools. o TRAC+ Intervention schools in Takeo province performed well against Comparison schools. o Further work can be done to determine if the difference in results between provinces depends on other NGO support. • Comparing Grades 1-3, Grade 2 shows the weakest impact of TRAC+ intervention within the three target grades. • Girls out-perform boys in sub-components 1-6 in both TRAC+ Intervention and Comparison schools.

6

Graphical results for Grade 2:

140% 120% 100% 80% 60% 40% 20% 0%

Absolute scores achieved by sampled students are low.

Grade 2: EGRA Endline IMPROVEMENTS T vs C (C normalised at 100% for comparison)

LNR

FWR NWR SR TRAC+ Intervention

LNR (T) = 32.8 LPM and LNR (C) = 28.8 LPM Word reading (T) range is 13.6-18.8 WPM across sub-components. Word reading (C) range is 13.2-19.3 WPM across sub-components. NWR (T) = 7.8 WPM and NWR (C) = 5.8 WPM

ORF RCO LCO Comparison

The effects of the late start of TRAC+ interventions in the 2015-2016 academic year would be particularly noticeable in Grade 2, where students had finished the part of the curriculum that was most linked to the TRAC+ interventions by the time TRAC+ year 2 began implementation.

Grade 3: • TRAC+ Intervention schools out-perform Comparison schools in all seven sub-components in EGRA endline (and 6/7 at BL). • TRAC+ Intervention schools out-perform Comparison schools in all seven sub-components in improvements BLEL. o The improvement scores in two phonics sub-components (1 and 3) were significantly better (#1 P<0.01, #3 P<0.001) in Intervention cf. Comparison. • The effect sizes showed small effects in FWR (#2, d = 0.35), ORF (#5, d = 0.26) and RCO (#6, d = 0.24); a medium effect in LNR (#1, d = 0.53); and a large effect in NWR (#3, d = 0.83). • TRAC+ year 3 can investigate whether the above results are directly due to project intervention; length of intervention; sampling bias or other external factors. • Results across provinces, although not statistically robust, showed large variations: o Comparison schools in Siem Reap province performed well when compared with TRAC+ Intervention schools. o In the four other target provinces, TRAC+ Intervention schools performed well when compared with Comparison schools. • Girls out-perform boys in sub-components 1-6 in both TRAC+ Intervention and Comparison schools (as with Grade 2); this effect is significant (P<0.05) for BL and EL figures. There are too many variables to compare improvements with any reliability. Graphical results for Grade 3: Grade 3 improvements are hard to assess because of the sampling of only Group A schools which were selected at the start of TRAC+ as being potentially higher attaining schools. Baseline figures at Grade 3 are higher for TRAC+ Intervention schools than control schools in 6/7 sub-components. Brief analysis suggests that this is because of school selection; group A schools create most of the baseline inequality.

Grade 3: EGRA Endline Improvements T vs C (C normalised at 100% for comparison) 250% 200% 150% 100% 50% 0% LNR

FWR NWR

SR

TRAC+ Intervention

ORF

RCO

LCO

LNR (T) = 47.2 LPM and LNR (C) = 37.0 LPM Word reading (T) range is 23.5-32.1 WPM across subcomponents. Word reading (C) range is 17.8-26.7 WPM across subcomponents. NWR (T) = 13.8 WPM and NWR (C) = 8.4 WPM

Comparison

NGO support

7

The question of how much NGO support effects the results of TRAC+ is beyond the scope of this report. Figures from the 2011 4 Economic Census of Cambodia show that Siem Reap (SRP) Province has 32 times the level of NGO staff per head of population compared with Takeo. (SRP:Takeo (TKO) = 32.44:1, SRP:Banteay Meanchey (BMC) = 1.22:1, SRP:Kampong Thom (KTM) = 2.77:1, SRP:Preah Vihear (PVH) = 5.35:1). Promotion of struggling students Detailed analysis of the EGRA figures compared with MoEYS expectations of 90%+ or 93%+ promotion rates shows that students are being promoted to higher grades without sufficient reading ability to access the next class. Analysis of data includes the findings: • Grade 1: Only 60% of TRAC+ Intervention students and 51% of Comparison students sampled were able to correctly identify ten Khmer characters in a minute. Only 85% of students in TRAC+ Intervention schools and 78% of students in Comparison schools were able to identify a single letter. • Grade 2: Only 67% of TRAC+ Intervention students and 58% of Comparison students sampled were able to correctly read 20 Khmer letters in one minute at the end of Grade 2. • Grade 3: Only 69% of TRAC+ Intervention students and 63% of Comparison students sampled were able to correctly read 30 Khmer letters in one minute at the end of Grade 3. Considering the weakest students, 10% of TRAC+ Intervention students and 17% of Comparison students sampled are unable to correctly read 10 Khmer letters in one minute by end Grade 3. The MoEYS presented a long-term aim in 2014 that: “Every child who successfully completes Grade 3 will be able to read

with understanding at a level of 45 – 60 words per minute.” 5 The results shown in this sample suggest that this aim is a long way from being achieved. 2.5

Summary of recommendations

The current MoEYS-sanctioned EGRA tool being administered has serious weaknesses; however, there is little or no chance of a new tool being created, tested and standardised before TRAC+ year 3 EGRA baseline is required to commence. ● Recommendation 1: EGRA for TRAC+ year 3 to use improved EGRA test IF such a resource is available in time. Comparison schools were not selected with sufficient rigour and the sample size (especially for Comparison schools) did not allow sufficient sampling for a meaningful comparison. Re-selection of Comparison schools is a very time-consuming task and would not be possible before the start of TRAC+ year 3. ● Recommendation 2A: No change in school selection for TRAC+ year 3. ● Recommendation 2B: Increase sample size in Comparison schools. ● Recommendation 2C: Comparison school selection issues noted in subsequent EGRA or other performance analysis. TRAC+ interventions are focussed very directly on phonics/decoding reading skills. Support for higher-level reading skills is not yet a part of the project intervention strategy. ● Recommendation 3A: Additional ‘connected text’ resources (e.g. mini-libraries) to be provided as RRS interventions. ● Recommendation 3B: Additional levelled readers to be distributed in multiple sets to allow paired or group reading activities. Students are being promoted with insufficient reading ability to access the curriculum at a new grade; this becomes more noticeable at higher grades (Grade 2 revises most of Grade 1 in the first 31 lessons) and is partly down to the pressure on Directors to achieve MoEYS-mandated pass rates. ● Recommendation 4: TRAC+ to present relevant EGRA data to MoEYS with priority of stopping promotion of students who do not have the requisite competencies to access the next grade. Some crucial RRS interventions were missed in the early part of the academic year 2015-2016 because of the delay in beginning the implementation of TRAC+ year 2 activities. ● Recommendation 5: TRAC+ activities to start promptly at the beginning of the 2016-17 academic year. All recommendations are made on behalf of WEI as a basis for further discussion with stakeholders in EGR in Cambodia.

4

Ministry of Planning (2013) Economic Census of Cambodia 2011 Provincial Reports (#01 BMC, #6 KTM, #13 PVH, #17 SRP, #21 TKO) National Institute of Statistics supported by JICA 5 Kann Puthy (2014) Variations in early reading assessment results between rural and urban populations presented to UNESCO Institute for Statistics conference ‘What works in oral assessments of early reading?’ 21 July 2014, Montreal, Canada on behalf of MoEYS, Cambodia (http://tinyurl.com/hop9538 retrieved 09-Oct-2016)

8

3.

Introduction to TRAC+ and Aim of Report

3.1

Introduction to TRAC+

6

The Total Reading Approach for Children Plus (TRAC+) project scales up the work from the Total Reading Approach for Children (TRAC) project (2012–2014), which was funded under the All Children Reading Grand Challenge, a consortium of donors including USAID, World Vision International (WVI), and Australian Aid. TRAC+ scales up the TRAC project from the pilot project of eight schools up to 138 schools in five provinces. TRAC+ is a three-year project, with the final year starting in October 2016 and project completion due in September 2017. The total number of students expected to benefit over the lifetime of the project is 19,822; with a project budget of $2.84m, this works out at a cost per beneficiary of $143.27. Under the TRAC+ project, World Education, Inc. (WEI) works closely with the Ministry of Education, Youth and Sports (MoEYS) and in partnership with WVI, the donor partners, to improve reading outcomes of Grade 1, 2 and 3 students in Cambodia using an integrated approach comprised of various interventions, including: • Reading Benchmarks WEI collaborated with MoEYS to create the now endorsed Reading Benchmark Standards for the national curriculum. These include specific tasks and skills that students should master at designated intervals. The benchmarks include regular interval assessments which divide reading skills in specific competencies, enabling teachers to deliver targeted remedial interventions. • Literacy Coaches TRAC+ selects and trains school-based Literacy Coaches (LCs) to help teachers administer the benchmark assessments and the Rapid Response System (RRS) interventions. Literacy Coaches also act as the liaison between schools and parents. • Rapid Response System Literacy Coaches implement the RRS after struggling students are identified by means of the benchmark interval assessments. The RRS includes parental engagement, peer tutoring, a reading toolkit, and (in the 73 Group A schools) an mLearning application. The toolkit and mLearning application are aligned with the MoEYS national curriculum textbooks and the Reading Benchmark system. TRAC+ has the following targets which are related to this report: Project goal: Improve early grade literacy outcomes Project outcomes: # 1: Schools with TRAC+ interventions report better reading achievement than comparable schools that do not use TRAC+ # 4: Project activities are effectively monitored TRAC+ has three potential ways to assess project progress through reading outcomes. These are: (1) Standardised EGRA testing using the Tangerine app or paper substitute. (2) Interval test results using the Reading Benchmarks (internally administered by schools with support from TRAC+). (3) Internal end-of-semester tests from schools. This report focuses on the first of these areas: the EGRA test which is externally administered using a Tablet device fitted with the specifically designed Tangerine app. 3.2 Aim: •

Aim of Report The aim of this report is to evaluate the current performance of the TRAC+ project interventions based on data provided by the EGRA endline survey performed at the end of the 2015-2016 academic year.

Purpose: 6

Adapted from website: http://cambodia.worlded.org/projects/total-reading-approach-for-children/

9

•

The purpose of this report is to identify any strengths and weaknesses of the TRAC+ intervention that are displayed by the EGRA analysis and to use these to improve project outcomes for TRAC+ year 3. Outcomes: • Improved practice for TRAC+ year 3 at strategic (project management) level and at local (school, district, provincial) level. • Improved practice in EGRA for TRAC+ year 3. th • Relevant content disseminated for TRAC+ Mid-Term Evaluation (MTE) meeting on 27 September 2016. Target Audience: • TRAC+ collaborative partners: WEI and WVI. • Summary findings report will be presented to MoEYS as appropriate and after agreement between WEI and WVI. 4. 4.1

EGRA Testing EGRA protocol

The Early Grade Reading Assessment (EGRA) is a tool used to measure students’ progress towards learning to read. It is a test that is administered orally, one student at a time and takes about 15 minutes to administer. Details about the testing protocol can be found in the Khmer language book: េតស�អំណានថា�ក់ដំបូង Early Grade Reading Assessment (EGRA) which details the assessment 7

protocol for administering the test using the Tangerine app. Administration of the EGRA tests was co-ordinated by experienced WEI staff with the support of MoEYS and WVI counterparts. Some school-level counterparts also received orientation training on administration of the EGRA test. A total of 34 assessors administered the EGRA, of whom 21 were WEI staff. Training in EGRA orientation and use of the tangerine app was delivered to the following groups as shown in Table 4.1.1 below. Table 4.1.1: Participants trained for administration of EGRA test Group

#participants

MoEYS (Departments of: Primary Education, Curriculum Development, Teacher Training, Information Technology) School level (SDs, Deputy Directors, Teachers, LCs, officials at District and 8 Provincial level) WVI Trainers WVI Staff

Type of training

Duration

10

Orientation

2 days

203

Orientation

1 day

12 9 23

ToT Capacity Building General Training

3 days 2 days

10

In accordance with international norms (RTI, 2011 ) the Cambodian EGRA test is split into sub-components to identify specific reading and language competencies. These are as shown in Table 4.1.2 below. Table 4.1.2: Sub-components of the EGRA test Sub-component 1. Letter name recognition (LNR) 2. Familiar word reading (FWR) 3. Non-word reading (NWR) 4. Sentence reading (SR) 7 8

Early reading skill

Demonstrated by student’s ability to:

Letter recognition

Provide the name of graphemes presented in random order Alphabetic principle Use letter-sound correspondence to read Decoding skills a list of familiar words Alphabetic principle Use letter sound correspondence to read Decoding skills a list of non-words Reading fluency Read unrelated sentences fluently

# items on test: BL 100

EL 100

50

50

50

50

55

57

Output unit Letters per minute (LPM) Words per minute (WPM) Words per minute (WPM) Words per minute

ករែណនាំEGRAនិងរេបៀបេធ�េតស�EGRAេដយេ្របTangerineស្រមាប់អ�កវយតៃម�

Full list in Annex 3 To avoid double-counting, the total number of WVI staff trained was 28. 10 RTI International and International Rescue Committee (2011) Guidance Notes for Planning and Implementing EGRA page vii 9

10

(WPM) 5. Oral reading fluency (ORF)

Oral reading fluency

Read a narrative text with accuracy, little effort, and at a sufficient rate

82

73

Words per minute (WPM)

6. Reading Comprehension (RCO)

Reading comprehension

Respond to literal and inferential questions about the text read

5

4

Correct answers (CA)

7. Listening Comprehension (LCO)

Comprehension

Respond correctly to literal and inferential questions about a text read to the student

3

3

Correct answers (CA)

Total:

345

11

337

Able students may achieve scores above the EGRA ‘ceiling’ scores. For example, a student reading 100 letters correctly in subcomponent 1 (LNR) in 30 seconds would have a score of 100 * (60/30) = 200LPM. Similarly, a student correctly reading 50 words correctly in sub-component 2 (FWR) in 50 seconds would have a score of 50 * (60/50) = 60WPM. 4.2

TRAC+ effect on EGRA

TRAC+ directly addresses sub-components 1-3. Sub-components 4-6 are indirectly addressed, i.e. the reading skills directly addressed by the RRS should be expected to improve learning outcomes on other sub-components. Sub-component 7 (LCO) might almost be considered to be a ‘Control Item’ within the EGRA as it does not actually require any reading skills. However, in this EGRA the low number of questions (three) and the poor question selection means that this control provides little real data. Table 4.2.1 below shows how directly the TRAC+ RRS addresses the seven EGRA sub-components. Table 4.2.1: Level of TRAC+ interventions relating to specific sub-components of the EGRA Subtasks

Early reading skill

Level of TRAC+ intervention

1. Letter name fluency

Letter recognition

VERY DIRECT

2. Familiar word fluency

Alphabetic principle/decoding skills

DIRECT

3. Non-word reading fluency

Alphabetic principle/decoding skills

VERY DIRECT

4. Sentence reading

Reading fluency

INDIRECT

5. Oral reading fluency

Oral reading fluency

INDIRECT

6. Reading Comprehension

Reading comprehension

VERY INDIRECT

7. Listening Comprehension

Comprehension

NOT ADDRESSED

5.

Limitations of the EGRA

5.1 Limitations of the EGRA test content 12

The first EGRA test in Cambodia was implemented by MoEYS in 2010 . The complexity of the Khmer language and the lack of available standardised data means that all current available versions of Khmer language EGRA tests are limited in their effectiveness. The EGRA version used here was endorsed by MoEYS and the most suitable available at the time of testing but has a number of problems that can be quickly identified and will need to be addressed. Some limitations noted by WEI are specified by EGRA subcomponent as shown in Table 5.1.1 below. The relevant departments within MoEYS are aware of the problems with the current EGRA tools and have commissioned Research Triangle Institute (RTI) to collaborate in improving the EGRA tools. A new EGRA tool is unlikely to be available for administration of a TRAC+ year 3 BL. Table 5.1.1: Some limitations of the EGRA test by sub-component as analysed by WEI 11

The total number of items does not represent a ‘total’ score for a test that could be expressed, for example, as a percentage. This is because there is no clear weighting of sub-components, which are considered individually. 12 Further info. see e.g.: RTI International (2015) Assessment of Early Grade Reading in the Education Sector in Cambodia Report for USAID page 64

11

Subtasks

Issue to be addressed:

1. LNR

The seven Khmer dependent vowels that are placed above a consonant are missing from this part of the test. (Vowels: (◌ិ) (◌ី) (◌ឹ) (◌ឺ) (◌ំ) (◌ំ◌ុ) (◌ា◌ំ)). (7/23 vowels)

2. FWR

The section is not aligned with MoEYS textbooks. The most high-frequency words are not included in this part of the test. Obvious high-frequency words excluded which can be decoded phonetically include: I, is, with, and, this, that, have, which (ខ�ុំ , ជា , ជាមួ យ , និង , េនះ , េនាះ , មាន , ែដល ).

3. NWR

Some words are open to multiple possible pronunciations (e.g. ែខស has two possible readings).

4. SR

No obvious problems with this sub-component.

5. ORF

The first line of Version B (EL) uses a technically correct but relatively uncommon plural for ‘students’ that children in early grades would be unlikely to know, much less read. This could be replaced with a simple alternative without affecting meaning: (សិស្សោនុសិស្ស change to សិស្ស).

6. RCO

Version A (BL) of this sub-component uses more repetitive language than Version B (EL); this would appear to make Version A more accessible to students able to infer word meanings from context.

7. LCO

Only three questions. In Version A (BL), the second question does not discriminate students who understand from those that do not understand; the wording needs to be changed.

5.2

Use of the Tangerine App

Assessors report problems with the Tangerine app, in particular the auto-stop feature showing inconsistencies. Another problem was the removal of one question in sub-component 6 (RCO). Although conversations with the assessors suggest that these issues did not affect the overall EGRA results, these problems should be noted and addressed for future implementation of the EGRA. 5.3

Absent students

The intention of the Year 2 EL was to re-assess the students who undertook the Year 2 BL. In the unavoidable case of student absences, EGRA administrators selected an alternative student at random; however, the author’s opinion (backed-up by questioning at school-level) that it is more likely that the more able students are present at the end of the academic year, meaning any detectable bias from re-selection would be expected to be in favour of higher results in the EGRA EL scores. Full analysis is beyond the scope of this report, but the raw data will remain available. Attrition rates are show in Table 5.3.1 below. Table 5.3.1: Attrition rates for students in G2 and G3 Group

# Student BL tested

Students present BL and EL

Attrition rate

Grade 2 (T)

514

`75%

25%

Grade 2 (C)

179

75%

25%

Grade 3 (T)

476

74%

26%

Grade 3 (C)

180

74%

26%

Attrition rates were the same at for Intervention and Comparison at each grade level. An attrition rate of 25-26% is another factor that creates confusion over comparisons of BL and EL data. 5.4

Inconsistencies in EGRA administration

12

The training for EGRA EL administration was more rigorous than that for the EGRA BL, EL training having been conducted based on previous field experience and with new knowledge about how EGRA is applied. Thus, in the opinion of WEI, it is possible that the results for the EL are more reliable than those for the BL. Where alterations in school selection occurred between BL and EL, leading to incomplete figures, the data from those schools has been excluded from analysis. However, this data will be shared with TRAC+ provincial level staff when appropriate to inform on schools that require priority interventions (for example, some schools with incomplete data had very low BL or EL scores; these schools may need extra monitoring and support from TRAC+ field staff). 5.5

Alterations in the EGRA test between baseline and endline

Between the BL and the EL, a number of test items were altered as shown in Table 4.1.2 (page 10) and these also created alterations in the number of test items available. There is no evidence that the number of test items had any effect on the EGRA results as corrections can be made very simply and tested students were not achieving ceiling scores. Further testing would be required to establish whether the different versions of the tests are directly comparable. 6.

Sampling and limitations

In TRAC+ schools are divided into TRAC+ Intervention (T) schools and Comparison (C) schools with no intervention. TRAC+ Intervention schools are further divided into Group A schools – which receive the additional mLearning (Aan Khmer App) support – and Group B schools which receive all the interventions except for the mLearning. All TRAC+ Intervention schools support Grades 12 and Group A schools additionally support Grade 3 students. Five schools (one TRAC+ Intervention, four Comparison) were re-classified between Years 1 and 2 of the TRAC+ project. The schools analysed for the Year 2 EGRA endline are as shown in Table 6.1 below. Table 6.1: Number of TRAC+ Intervention and Comparison schools analysed Group Group A Group B TRAC+ total TRAC+ Intervention schools Comparison schools Total

Intervention TRAC+ Grades 1-3 and mLearning TRAC+ Grades 1-2 and no mLearning No intervention

Number of Schools 73 65 138 50 188

The sampling procedure uses a randomly selected group of students from Grades 1-3 in all TRAC+ Intervention and Comparison schools. The sampling is shown in Table 6.2 below, as well as the percentage of the total eligible students per grade. Note that the sampling is per class not by grade: i.e. a school with three Grade 1 classes would have 3 * 4 = 12 students sampled. Full sampling details are in Annex 1.

13

Table 6.2: Sample sizes for TRAC+ Intervention and Comparison schools by grade Grade

Grade 1 Grade 2 Grade 3

Baseline No. of TRAC+ Intervention No. of Comparison students sampled students sampled (% population) (% population) 0 0 n/a (assumed zero) n/a (assumed zero) 3 per class 3 per class (8.6%) (8.4%) 3 per class 3 per class (8.3%) (9.3%)

Endline No. of TRAC+ Intervention students sampled (% population) 4 per class (10.1%) 3 per class (9.2%) 3 per class (9.2%)

No. of Comparison students sampled (% population) 4 per class (8.8%) 3 per class (9.4%) 3 per class (10.3%)

The overall sample size is 9.5% of the eligible student population (2346 students tested altogether for the endline). The sampling 13 method was adapted from CheckMarket Sample Calculation which requires a sample size of >7% of the total student population for valid analysis according to the TRAC+ Year 1 Annual report. This figure is incorrect. Comparison schools were matched with TRAC+ Intervention schools according to whether those schools achieved an equivalent comparable baseline score with the TRAC+ Intervention schools. All data analysis in this report will have to take into consideration the fact that Comparison school selection was insufficiently rigorous to allow proper analysis. The sample size was also insufficient to provide a valid Comparison population. Matching took place based on an aggregated total EGRA score without consideration of sub-component weighting or geographical issues (district baselines were not matched). Other variables that might affect school selection but were not considered include: •

•

• •

Size of school (e.g. cluster core school or satellite school): o This is potentially very significant as the level of LC support is consistent at one LC per school, regardless of the number of classes requiring support; this varies between two classes and 13 classes. Level of past or current NGO intervention in EGRA or related field: o The most significant NGO intervention observed was that of the NGO ‘Room to Read’ who provide levelled reading materials training and technical support to their intervention schools. Accessibility of schools: o The term ‘remote school’ has been officially removed from MoEYS technical vocabulary; however, matching schools in terms of their accessibility for support and monitoring is a significant issue. Library facilities: o Includes whether the school has: a library, a separate space for the library, and a full-time librarian.

The analysis will nevertheless look for factors where comparisons can be made – remembering that the primary purpose of the analysis is to look for ways to improve the TRAC+ project in the final year of implementation and to achieve maximum sustainability following the conclusion of the project. The purpose of this report is not to evaluate the performance of individual schools or districts, nor is it a method to identify weak teaching. 7.

Methods of Analysis

The EGRA analysis is a key part of the project goal and the related project outcomes #1 and #4 as shown in 3.1 (page 9). The EGRA results are aggregated by: • raw student scores aggregated for school scores at Grade level (Grades 1,2 and 3) • school scores are aggregated for district / ADP results • district / ADP results are aggregated for provincial results • finally, provincial results are aggregated for a project-level analysis.

13

https://www.checkmarket.com/market-research-resources/sample-size-calculator/

14

This report will focus primarily on project-level analysis. Further analysis at lower levels will be provided to assist field staff in their work, but the sampling techniques used preclude statistically valid conclusions. 14

To decide if TRAC+ project outcome#1 is being met, there are two forms of analysis that may be considered: (1) • (2) •

Endline score comparison Are TRAC+ Intervention schools performing better than Comparison schools on the EGRA EL results? Baseline-Endline improvement comparison Are TRAC+ Intervention schools making larger improvements between BL and EL when compared with Comparison schools?

Differences in TRAC+ year 2 are targeted to show a ‘measureable difference’, but statistical analysis will be used to confirm if TRAC+ Intervention/Comparison differences may be considered significant. This analysis will use a two-tailed t-test with an α-value of 0.05. 15 The second stage of analysis will seek to look at effect size using Cohen’s D test where a d-value of more than 0.2 may be considered a small effect; a value of more than 0.5 may be considered a medium effect; and a value of more than 0.8 may be considered a large effect. The following measures will be taken in an effort to maintain data integrity: (1) Test monitoring by staff from outside WEI. (2) Flagging to check data with: a. Absences. b. Areas where BL > EL. c. Exceptionally high or exceptionally low scores at BL and/or EL. d. Inconsistencies within data sets, e.g. where NWR is strong but LNR is weak. e. Incomplete data (e.g. from changed school selection – schools lacking either BL or EL figures will be removed, but data will be kept). Data from aggregated figures checked in (3) above will be checked against original raw data at school or student level as required. 8.

Main findings

16

Limitations on findings: All findings listed in this report should be considered as indications of the current situation; the level of data integrity available for analysis is insufficiently robust to deliver firm conclusions. Some trends based on these indications may later lead to valid conclusions, but any EGRA taking place in TRAC+ year 3 will need to be delivered with greater rigour. The following concerns should be taken into context with regard to any conclusions reached in this report: • Sampling was not performed according to the EGRA guidelines. • Comparison schools were not matched according to an accepted protocol. • EGRA training was of insufficient standard and duration to permit sufficient rigour in gaining test data. • The EGRA tool was changed between BL and EL without standardisation and testing. • The MoEYS-sanctioned EGRA tool contains too many basic errors to be considered a reliable indicator on many sub-components. • Allocation of ‘better’ schools to Group A precludes analysis of Group A compared with Group B using the small sample size. Overall: • EGRA testing shows some improvements (BLEL) in TRAC+ Intervention schools compared with Comparison schools. o Improvements are most marked in EGRA sub-components (1-3) relating to letter recognition and decoding words; this is observable across all three grades sampled and is consistent with the type of intervention applied. • Grade 1 Intervention schools show significantly (P<0.001) higher results on all EGRA reading sub-components (1-6) compared with Comparison schools; the effect size is small in all instances (Cohen’s d = 0.28-0.39). • Grade 2 Intervention schools show higher results and better improvements on all EGRA phonics sub-components (1-3) compared with Comparison schools; the difference is significant for LNR (#1, P<0.05) and NWR (#3, P<0.01) and the effect size is small in all three instances (d = 0.22-0.45).

14

TRAC+ outcome #1: Schools with TRAC+ interventions report better reading achievement than comparable schools that do not use TRAC+ 15 http://www.uccs.edu/lbecker/effect-size.html 16 In the following data analysis some figures may appear to be inaccurate at the least significant digit. This is due to rounding errors where calculations are made carrying figures forward and then displayed in a readable form (one decimal place for most figures).

15

• Grade 3 Intervention schools show higher results and better improvements on all EGRA sub-components (1-7) compared with Comparison schools; the difference in improvement is significant for LNR (#1, P<0.01) and NWR (#3, P<0.001). Effects in improvements are shown as: small in sub-components FWR (#2, d = 0.35), ORF (#5, d = 0.26) and RCO (#6, d = 0.24); medium in LNR (#1, d = 0.53); and large in NWR (#3, d = 0.83). • Girls out-performed boys consistently in EGRA reading sub-components (1-6) of the EGRA test, regardless of interventions. • The level of reading ability shown by many students is insufficient for a promotion rate in line with MoEYS expectations. • The improvements in reading ability shown by TRAC+ interventions to date are relatively small. Detailed analysis by Grade (1-3) follows. 8.1

GRADE 1

Summary: • TRAC+ Intervention schools out-perform Comparison schools in all seven sub-components in EGRA endline. • The improvements in reading sub-components (1-6) were all highly significant (P<0.001) and the effect size was ranked as ‘small’ in all instances (d = 0.28-0.39). • Assessed Grade 1 students were unable to sufficiently access sub-component 6 (RCO) (over 90% of students scored zero). o Sub-components 2-5 were also inaccessible (zero scores) to over two-thirds of the sampled population. • The level of reading ability in Grade 1 students is so low that only letter recognition can be reliably analysed; this is a shortfall relative to the curriculum expectations where students should be able to read compound words at the end of Grade 1. • Girls out-performed boys in reading sub-components (1-6) of the EGRA test, regardless of interventions. Grade 1 students were assessed only at EL. There is no BL for Grade 1 students: the assumption is that Grade 1 students start their academic education with zero knowledge of reading; as the coverage of pre-schools increases, this assumption may become less valid. In six of the seven EGRA sub-components, TRAC+ Intervention schools perform significantly better than Comparison schools at P<0.05. On the seventh sub-component (LCO) TRAC+ Intervention outperforms Comparison at P<0.1. Table 8.1.1 shows the comparison between TRAC+ Intervention (T) and Comparison (C) schools on the seven EGRA sub-components. Table 8.1.1: TRAC+ Intervention (T) against Comparison (C) schools for EGRA sub-components at Grade 1 Endline SubComponent

T or C

Raw Score (unit)

T>C

T > C Raw Score

T>C% difference

Cohen’s ‘d’ 17 (effect)

% students score >0

1. LNR

T C

18.9 (LPM) 13.3 (LPM)

T>C TRUE ***

+5.6 (LPM)

+42%

0.37 (SMALL)

84.7% 78.0%

2. FWR

T C T C T C T C

3.1 (WPM) 1.3 (WPM)

T>C TRUE *** +1.8 (WPM)

+143%

T>C TRUE *** +1.7 (WPM)

+174%

3.4 (WPM) 1.2 (WPM)

T>C TRUE *** +2.2 (WPM)

+190%

3.2 (WPM) 1.2 (WPM)

T>C TRUE ***

+2.0 (LPM)

+162%

30.8% 20.5% 31.5% 20.0% 28.4% 22.0% 20.1% 11.0%

+10.3%

2.7 (WPM) 1.0 (WPM)

0.35 (SMALL) 0.39 (SMALL) 0.28 (SMALL) 0.29 (SMALL)

0.15 (CA) 0.025 (CA)

T>C TRUE *** +0.125 (CA)

+500%

0.31 (SMALL) T>C TRUE (#) +0.12 (CA) +8% 0.13 (NULL) * = P<0.05 ** = P<0.01 *** = P<0.001

9.0% 1.0% 86% 83%

+8%

3. NWR 4. SR 5. ORF 6. RCO

T C 7. LCO T C Levels of significance:

17

1.64 (CA) 1.52 (CA)

(#) = P<0.1

http://www.uccs.edu/lbecker/effect-size.html

T>C difference score >0 +6.7%

+11.5% +6.4% +10.1%

+3%

16

The results of the EGRA EL analysis in Grade 1 schools show that students in TRAC+ Intervention schools are achieving significantly better results than students in the selected Comparison schools in the reading sub-components (1-6) of the EGRA test. From these figures, testing for effect size shows that for each area, after one year of intervention, the effect is small (d-values from 0.28-0.39). Regardless of the effects of the intervention, the absolute scores are so low (2.7-3.4 WPM (T) and 1.2-1.3 WMP (C)) that the effectiveness of education must be seriously questioned for all schools in the sample. Students are being taught how to read compound words during the later stages of the Grade 1 curriculum and the results show that the vast majority are unable to do this. Graph 8.1.1 shows the tabular data of TRAC+ Intervention school scores against Comparison school scores in bar chart form where the Comparison school figure has been normalised to 100% across all sub-components so the levels of attainment can be observed. Graph 8.1.1: EGRA results of Grade 1 EL Grade 1: EGRA EL results T vs C (C normalised at 100% for comparison)

EGRA EL results T vs C: scores and zeros 100%

500%

80%

400%

60%

300%

Zero

40%

200%

Score LCO (T)

LCO (C)

RCO (C)

ORF (C)

RCO (T)

ORF (T)

SR (T)

LCO

SR (C)

RCO

NWR (T)

FWR NWR SR ORF Treatment Control

NWR (C)

LNR

FWR (C.)

0%

0%

FWR (T)

100%

LNR (T)

20% LNR (C.)

600%

Graph 8.1.2: EGRA results of Grade 1 EL: scores and zeros

The largest percentage improvement is in RCO where all scores are exceptionally low. For that particular sub-component, WEI suggests that the data should not be taken as a significant improvement in reading ability. The improvements in sub-components 1-5 are a positive result for the TRAC+ interventions in Grade 1, but these are still very low figures as shown by number of children failing to register marks on each sub-component and the absolute reading ability of the students tested. Graph 8.1.2 above shows the number of children achieving a positive score. It is only in sub-components 1 (LNR) and 7 (LCO) where a majority (or even over a third) of children can access the task set. However, for all seven sub-components, the TRAC+ Intervention schools show a higher level of attainment than the Comparison schools. 8.1.1

Gender factors at Grade 1

Consideration of gender factors is also possible from the disaggregated data. As show in Table 8.1.1.1 and the associated Graph 8.1.1.1 girls out-perform boys in sub-components 1-6 in both TRAC+ Intervention and Comparison schools. The exception is in subcomponent 7 (LCO) where boys in TRAC+ Intervention schools achieve higher scores than girls. Table 8.1.1.1

Grade 1 Endline gender comparison Category

LNR (LPM)

FWR (WPM)

NWR (WPM)

SR (WPM)

ORF (WPM)

RCO (CA)

LCO (CA)

TRAC+ Intervention Comparison

18.7

2.6

2.4

3.1

2.9

0.14

1.70

Male

11.5

0.6

0.8

0.5

0.8

0.00

0.42

TRAC+ Intervention Comparison

19.1

3.5

2.9

3.7

3.6

0.16

1.33

Female

15.2

2.0

1.2

1.9

1.7

0.05

0.51

Graph 8.1.1.1

Grade 1 Endline gender comparison

17

Normalised score where M(T) = 100

Grade 1 Endline Gender Comparison

8.2

160 140 120 100

M (T)

80

F (T)

60

M (C.)

40

F (C.)

Graph 8.1.1.1 (left) shows the tabular figures in graphical form. The figures are normalised to show the male students TRAC+ Intervention (M (T)) scores at 100% and other figures in comparison to that with the aim of showing the relative ability of girls being higher than boys regardless of intervention.

20 0 LNR

FWR

NWR

SR

ORF

RCO

LCO

GRADE 2

Summary • TRAC+ Intervention schools out-perform Comparison schools in 6/7 sub-components in EGRA endline. • The endline scores in two phonics sub-components (1 and 3) were significantly better (P<0.01) in Intervention cf. Comparison. • TRAC+ Intervention schools out-perform Comparison schools in 4/7 sub-components in improvements BLEL. o The improvement scores in two phonics sub-components (1 and 3) were significantly better (#1 P<0.01, #3 P<0.05) in Intervention cf. Comparison. • The three phonics sub-components (1-3) each show a small effect size (d = 0.22-0.45) in improvements for Intervention cf. Comparison. • Results across provinces, although not statistically robust, showed large variations: o Comparison schools in Siem Reap and Kampong Thom performed well when compared with TRAC+ Intervention schools. o TRAC+ Intervention schools in Takeo province performed well against Comparison schools. o Further work can be done to determine if the difference in results between provinces depends on other NGO support. • Comparing Grades 1-3, Grade 2 shows the weakest impact of TRAC+ intervention within the three target grades. Gender differences: • Girls out-perform boys in sub-components 1-6 in both TRAC+ Intervention and Comparison schools. • In sub-component 1 (LNR), boys from the TRAC+ Intervention schools appear to be benefitting more than the girls from the TRAC+ intervention. • In sub-component 6 (RCO), boys from the Comparison schools are out-performing boys from the TRAC+ Intervention schools. Grade 2 students in TRAC+ Intervention schools should be in their second year of TRAC+ intervention and would therefore be expected to show the effects from TRAC+ year 1 interventions. However, the re-selection of Comparison schools took place and corrected this meaning that the baselines are approximately equal.

18

Table 8.2.1: TRAC+ Intervention (T) against Comparison (C) schools for EGRA sub-components at Grade 2 EL SubComp

T/C Baseline

Endline Data: T against C SCORES

Raw Score Raw Score (unit) (unit)

T>C

1. LNR

T 18.3 (LPM) 32.8 (LPM) T>C C 18.5 (LPM) 28.8 (LPM) TRUE **

2. FWR

T C T C T C

3. NWR 4. SR

3.2 (WPM) 3.5 (WPM) 2.2 (WPM) 1.9 (WPM) 4.1 (WPM) 3.9 (WPM)

T>C Score

Endline Data: T against C IMPROVEMENT Cohen’s ‘d’ Students score >0 (%) (effect) T > C Raw imp. T>C T>C T>C BL EL Imp. % diff. (unit) Score % diff.

+4.0 (LPM)

+14%

13.6 (WPM) T>C +0.4 13.2 (WPM) TRUE (WPM) 7.8 (WPM) T>C +2.0 5.8 (WPM) TRUE ** (WPM) 18.8 (WPM) T>C -0.5 19.3 (WPM) FALSE (WPM)

+2%

12.5 (LPM) T>C +2.2 (LPM) 10.3 (LPM) TRUE **

10.4 (WPM) T>C 9.7 (WPM) TRUE +35% 5.6 (WPM) T>C 3.7 (WPM) TRUE * -3% 14.7 (WPM) T>C 15.4 (WPM) FALSE

5. ORF

T 5.4 (WPM) 16.8 (WPM) C 4.5 (WPM) 16.4 (WPM)

T>C TRUE

+0.4 (LPM)

+2%

11.4 (WPM) 11.9 (WPM)

6. RCO 18 Adjust

T C

0.23 (CA) 0.18 (CA)

0.90 (CA) 0.86 (CA)

T>C TRUE

+0.04 (CA)

+4%

7. LCO

T C

1.10 (CA) 1.13 (CA)

2.01 (CA) 1.93 (CA)

T>C TRUE

+0.16 (CA)

+8%

T>C FALSE

+0.7 (WPM) +1.7 (WPM) -0.9 (WPM)

+21%

0.45 (SMALL)

87.5% 96.0% 8.5% 88.8% 93.5% 4.7%

+5%

0.22 (SMALL) 0.39 (SMALL) -0.02 (NULL)

31.5% 35.2% 35.8% 29.6% 40.1% 30.2%

+44% -6%

30.8% 20.5% 57.0% 48.5% 64.0% 63.5%

-0.7% -14.7% +21.2% +18.9% +23.9% +33.3%

-0.5 (LPM)

-4%

-0.02 (NULL)

28.4% 59.5% +31.1% 27.9% 58.0% +30.1%

0.67 (CA) 0.68 (CA)

T>C -0.01 (CA) FALSE

-1%

-0.04 (NULL)

16.1% 40.3% +24.2% 15.6% 35.0% +19.4%

0.91 (CA) 0.80 (CA)

T>C TRUE

+14%

0.03 (NULL)

68.9% 96.2% +27.3% 71.5% 94.5% +23.0%

+0.11 (CA)

Levels of significance: (#) = P<0.1 * = P<0.05 ** = P<0.01 *** = P<0.001 Figures highlighted in green do not seem consistent with related data. The reason for this has not been found. Comparing TRAC+ intervention with Comparison schools, positive effects can be seen in the phonics sub-components (1-3) where significant endline results and improvements can be seen in LNR (#1) and NWR (#3); and significant endline results in FWR (#2). The effect of these impacts may be measured as ‘small’ (d-values 0.22-0.45). Comparison schools show slightly better improvements in the sub-components related to reading comprehension; this is an area for TRAC+ to investigate further. Graph 8.2.1: EGRA results of Grade 2 EL improvements

140% 120% 100% 80% 60% 40% 20% 0%

Grade 2: EGRA Endline IMPROVEMENTS T vs C (C normalised at 100% for comparison)

Graph 8.2.1 (left) shows the tabular data in bar chart form where the Comparison school figure has been normalised to 100% across all sub-components so that differences between BL and EL can be compared across sub-components. The raw scores, as in Grade 1, show very low reading attainment in the sampled population. The end of Grade 2 represents 988 literacy teaching hours completed. Word reading (T) range is 13.6-18.8 WPM across sub-components. Word reading (C) range is 13.2-19.3 WPM across sub-components.

LNR

FWR NWR SR ORF RCO LCO TRAC+ Intervention Comparison Although the results show improvements for Grade 2 students sampled in TRAC+ Intervention schools against Comparison, when compared with Grades 1 and 3 the level of improvement is not as high. There are various issues that can be considered: • Implementation of the TRAC+ year 2 interventions took place after the start of the academic year at a time after the students had completed the Grade 1 revision lessons where the TRAC+ remedial work should be the most effective. • TRAC+ is more geared towards decoding words phonetically. This grounding in basic phonics skills should be properly established in Grade 1, although the Grade 2 baseline figures clearly show that is not the case.

18

For comparison, adjusted for five questions on baseline and four on endline. Endline scores are multiplied by 1.25 (5/4).

19

The largest percentage improvement result in Grade 2 is in sub-component 3 (NWR), looking at words that can be decoded phonetically but do not carry any meaning (e.g. fot, nam, teg in English). This is the clearest measure of students’ phonic decoding ability but it does not directly affect reading for meaning. Because the Grade 3 EGRA (comparing Intervention and Comparison schools) results are better than the Grade 2 results, and TRAC+ interventions should become less effective as students progress, Grade 2 results were analysed at provincial level as well. Table 8.2.2 shows the breakdown by provinces, looking purely at improvements from BLEL. In the table, figures highlighted in yellow (e.g. BMC (T) LNR = 12.13) are where TRAC+ Intervention schools show an improvement of >10% over Comparison schools, i.e. yellow is a ‘strong’ positive. The converse, where Comparison schools show an improvement BLEL of >10% over TRAC+ Intervention schools are highlighted in red, i.e. red is a ‘strong’ negative (e.g. BMC (C) LCO = 0.62). Pink highlighter is used for when figures are within 10%, the higher figure being highlighted (pale pink for T>C, dark pink and white text for T
T or C

LNR

FWR

NWR

SR

ORF

RCO

LCO

Total ***

***

T improvement

12.13

8.95

3.77

12.59

11.95

0.43

0.46

C improvement

5.95

6.19

3.74

12.24

8.50

0.43

0.62

* *

T improvement

16.32

13.15

6.44

18.77

16.96

1.05

1.39

**

C improvement

17.72

15.11

3.50

23.09

16.36

1.11

0.94

**

**

T improvement

17.42

10.36

6.51

14.27

14.71

0.75

1.02

***

*

C improvement

10.33

8.99

6.11

14.90

13.33

0.87

1.31

**

*

T improvement

13.49

11.85

6.18

15.65

13.80

0.67

0.71

*

C improvement

18.07

18.95

5.25

24.10

21.66

1.25

0.99

******

T improvement

18.46

11.20

6.71

13.67

14.35

0.72

1.09

C improvement

5.83

5.09

1.90

7.42

7.06

0.23

0.44

T improvement

12.50

10.40

5.60

14.70

11.40

0.67

0.90

******** ***

*

*** C improvement 10.30 9.70 3.70 15.40 11.90 0.68 0.80 The small sample size means that the above figures are not statistically valid. The indication is that Comparison schools in Siem Reap (and to a lesser extent Kampong Thom) had relatively good improvements on the EGRA EL test. Siem Reap has a relatively very high level of NGO support (see section 8.4 page 24). In contrast, Comparison schools in Takeo have relatively low improvements on the EGRA EL test. Takeo has a relatively very small level of NGO support (ibid.). 8.2.1

Gender factors at Grade 2

Graphs 8.2.1.1 to 8.2.1.7 are presented in to compare outcomes of boys and girls sampled in each EGRA sub-component. For each graph, the first two columns (from the left) are the figures for the boys in TRAC+ Intervention and Comparison (i.e. M (T) is Male (TRAC+ Intervention)) and the final two columns are the figures for the girls (F(T) and F(C.)). The blue part of each bar is the BL score. The orange part of each bar is the improvement (IMP) from BLEL. Thus, the total height of each bar is the EL figure. All figures are raw scores and use the units as previous explained for each sub-component (Table 4.1.2, page 10). Graph 8.2.1.1: LNR G2 boys and girls 40 30 20

LNR (IMP)

10

LNR (BL)

0 M M (T) (C.)

F F (T) (C.)

20

Graph 8.2.1.2: FWR G2 boys and girls

Graph 8.2.1.3: NWR G2 boys and girls 10.0

20.0

25.0

8.0

15.0

20.0

6.0

10.0 5.0

FWR (IMP)

4.0

NWR (IMP)

FWR (BL)

2.0

NWR (BL)

M M (T) (C.)

M M (T) (C.)

F F (T) (C.)

Graph 8.2.1.5: ORF G2 boys and girls

1.2

20.0

1.0 ORF (IMP)

10.0

ORF (BL)

5.0 0.0

Graph 8.2.1.7: LCO G2 boys and girls

2.0

0.6

RCO (IMP)

0.4

RCO (BL)

0.2

F F (T) (C.)

F F (T) (C.)

2.5 1.5 1.0

LCO (IMP)

0.5

LCO (BL)

0.0

0.0 M M (T) (C.)

SR (BL)

5.0 M M (T) (C.)

0.8

15.0

SR (IMP)

10.0

F F (T) (C.)

Graph 8.2.1.6: RCO G2 boys and girls

25.0

15.0

0.0

0.0

0.0

Graph 8.2.1.4: SR G2 boys and girls

M M (T) (C.)

F F (T) (C.)

M M (T) (C.)

F F (T) (C.)

The main conclusions from these results are: • • • 8.3

Girls out-performed boys in sub-components 1-6 in both TRAC+ Intervention and Comparison schools. In sub-component 1 (LNR), boys from the TRAC+ Intervention schools appear to be benefitting more than the girls from the TRAC+ intervention. In sub-component 6 (RCO), boys from the Comparison schools are out-performing boys from the TRAC+ Intervention schools; it may be that this is an effect of other NGO interventions (See 8.4, page 24) having (or targeting) a greater effect on boys. GRADE 3

Summary • TRAC+ Intervention schools out-perform Comparison schools in all seven sub-components in EGRA endline (and 6/7 at BL). • TRAC+ Intervention schools out-perform Comparison schools in all seven sub-components in improvements BLEL. o The improvement scores in two phonics sub-components (1 and 3) were significantly better (#1 P<0.01, #3 P<0.001) in Intervention cf. Comparison. • The effect sizes showed small effects in FWR (#2, d = 0.35), ORF (#5, d = 0.26) and RCO (#6, d = 0.24); a medium effect in LNR (#1, d = 0.53); and a large effect in NWR (#3, d = 0.83). • TRAC+ year 3 can investigate whether the above results are directly due to project intervention; length of intervention; sampling bias or other external factors. • Results across provinces, although not statistically robust, showed large variations: o Comparison schools in Siem Reap province performed well when compared with TRAC+ Intervention schools. o In the four other target provinces, TRAC+ Intervention schools performed well when compared with Comparison schools.

•

Gender differences: Girls out-perform boys in sub-components 1-6 in both TRAC+ Intervention and Comparison schools (as with Grade 2); this effect is significant (P<0.05) for BL and EL figures. There are too many variables to compare improvements with any reliability.

21

•

In sub-components 1-6, girls from the TRAC+ Intervention schools appear to be benefitting more than the boys from the TRAC+ intervention.

Grade 3 schools consist only of schools in Group A, i.e. those with a full TRAC+ intervention including mLearning. However, the mLearning App does not have content specific to the Grade 3 Khmer language curriculum; students who still require support from the Aan Khmer App should not have been promoted to Grade 3 as they will not have the basic reading competency required to access the curriculum. Because of changes in the Comparison group and some missing data, the number of schools analysed for Grade 3 is 116 in total, made up of 71 Group A schools and 45 Comparison schools; all schools with incomplete data (e.g. just BL or just EL) were removed from the analysis after data cleaning. Grade 3 students in TRAC+ Intervention schools should be in their third year of TRAC+ intervention and would therefore be expected to show the effects from TRAC+ year 1 interventions. There are two factors that reduce the clarity of this situation: (1) Selection bias: Group A schools were selected at the start of the project as being ‘potentially stronger’ and ‘likely to respond 19 better with TRAC+ interventions’ . Thus the BL figures in Grade 3 show TRAC+ Intervention schools are slightly stronger than Comparison schools. (2) Comparison school re-selection took place between TRAC+ year 1 and year 2. The above points mean that greater emphasis has to be placed on the improvement (BLEL) figures rather than the raw EL scores. In Table 8.3.1 below, the absolute EGRA scores are low (as in Grades 1-2). Later analysis will highlight the concern over the number of struggling students (low ability range) who show a lack of reading competencies to access the correct grade level curriculum (discussion in section 8.5, page 25). Table 8.3.1: TRAC+ Intervention (T) against Comparison (C) schools for EGRA sub-components at Grade 3 EL SubComp.

T/C

1. LNR 2. FWR

Baseline

Endline Data: T against C SCORES

Endline Data: T against C IMPROVEMENT Cohen’s Students score >0 (%) ‘d’ T > C Raw imp. T>C T>C T > C (effect) BL EL Imp. % diff. (unit) Score % diff.

Raw Score (unit)

Raw Score (unit)

T>C

T>C Score

T C

28.7 (LPM) 25.7 (LPM)

47.2 (LPM) 37.0 (LPM)

T>C TRUE (n/a)

+12.2 (LPM)

+33%

T C T C T C T C

10.3 (WPM) 8.1 (WPM) 5.2 (WPM) 4.8 (WPM) 15.6 (WPM) 11.9 (WPM) 19.0 (WPM) 14.6 (WPM)

23.5 (WPM) 17.8 (WPM) 13.8 (WPM) 8.4 (WPM) 31.7 (WPM) 26.7 (WPM) 32.1 (WPM) 24.6 (WPM)

T>C TRUE (n/a) T>C TRUE (n/a) T>C TRUE (n/a) T>C TRUE (n/a)

+5.7 (WPM) +5.6 (WPM) +5.0 (WPM) +7.7 (WPM)

+32% 13.2 (WPM) T>C 9.7 (WPM) TRUE (#) +67% 8.6 (WPM) T>C 3.7 (WPM) TRUE *** +19% 15.8 (WPM) T>C 14.8 (WPM) TRUE +31% 13.2 (WPM) T>C 10.9 (WPM) TRUE

6. RCO 20 Adjust

T C

1.12 (CA) 0.94 (CA)

1.69 (CA) 1.30 (CA)

T>C TRUE (n/a)

+0.39 (CA)

+30%

0.56 (CA) 0.36 (CA)

T>C TRUE

+0.20 (CA)

+56%

0.24 44.3% 57.8% 13.5% (SMALL) 42.6% 54.4% 11.8%

7. LCO

T C

1.70 (CA) 1.66 (CA)

2.10 (CA) 2.04 (CA)

T>C TRUE (n/a)

+0.06 (CA)

+1%

0.43 (CA) 0.35 (CA)

T>C TRUE

+0.08 (CA)

+23%

0.12 87.8% 96.6% 8.8% (NULL) 90.1% 94.4% 4.3%

3.NWR 4. SR 5. ORF

Levels of significance:

(#) = P<0.1

* = P<0.05

18.5 (LPM) 11.3 (LPM)

** = P<0.01

T>C TRUE **

+7.2 (LPM)

+64%

0.53 96.4% 96.9% 0.5% (MED.) 91.4% 95.0% 3.6%

+3.5 +36% 0.35 61.8% 80.3% 18.5% (WPM) (SMALL) 54.3% 78.3% 24.0% +4.9 +132% 0.83 51.4% 71.1% 19.7% (WPM) (LARGE) 48.1% 64.4% 16.3% +1.0 +7% 0.09 56.4% 75.9% 19.5% (WPM) (NULL) 45.7% 72.8% 27.1% +2.3 +21% 0.26 62.9% 80.3% 17.4% (WPM) (SMALL) 58.6% 76.7% 18.1%

*** = P<0.001

Where BL scores are significantly different, tests for significance have only been applied to improvements (BLEL). Graph 8.3.1 shows the tabular data in bar chart form where the Comparison school figure has been normalised to 100% across all sub-components so that differences between baseline and endline can be compared across sub-components. Graph 8.3.1: EGRA results of Grade 3 EL improvement 19 20

Internal WEI communication For comparison, adjusted for five questions on baseline and four on endline. Endline scores are multiplied by 1.25 (5/4).

22

Graph 8.3.1 (left) shows the tabular data in bar chart form where the Comparison school figure has been normalised to 100% across all sub-components so that differences between BL and EL can be compared across subcomponents.

Grade 3: EGRA Endline Improvements T vs C (C normalised at 100% for comparison) 250% 200%

Endline figures are not presented graphically because of the variation in BL figures.

150% 100% 50% 0% LNR

FWR

NWR

SR

TRAC+ Intervention 8.3.1

ORF

RCO

LCO

Further analysis could take place as to whether percentage improvements of student scores are a better measure. Students starting from a higher BL should be expected to achieve a higher absolute improvement.

Comparison

Gender factors at Grade 3

Graphs 8.3.1.1 to 8.3.1.7 are presented in to compare outcomes of boys and girls sampled in each EGRA sub-component. For each graph, the first two columns (from the left) are the figures for the boys in TRAC+ Intervention and Comparison (i.e. M (T) is Male (TRAC+ Intervention)) and the final two columns are the figures for the girls (F(T) and F(C.)). The blue part of each bar is the BL score. The orange part of each bar is the improvement (IMP) from BLEL. Thus, the total height of each bar is the EL figure. All figures are raw scores and use the units as previous explained for each sub-component (Table 4.1.2, page 10). Graph 8.3.1.1: LNR G3 boys and girls 60.0 50.0 40.0 30.0 20.0 10.0 0.0

LNR (IMP) LNR (BL) M M (T) (C.)

F F (T) (C.)

Graph 8.3.1.2: FWR G3 boys and girls 30.0 25.0 20.0 15.0 10.0 5.0 0.0

FWR (IMP) FWR (BL) M M (T) (C.)

F F (T) (C.)

8.3.1.5: ORF G3 boys and girls

Graph 8.3.1.3: NWR G3 boys and girls

Graph 8.3.1.4: SR G3 boys and girls

20.0

50.0

15.0

40.0 NWR (IMP)

10.0 5.0

NWR (BL)

0.0 M M (T) (C.)

F F (T) (C.)

Graph 8.3.1.6: RCO G3 boys and girls

30.0 20.0

SR (IMP)

10.0

SR (BL)

0.0 M M (T) (C.)

F F (T) (C.)

Graph 8.3.1.7: LCO G3 boys and girls

23

40.0 30.0

2.5

2.5

2.0

2.0 1.5

1.5

20.0

ORF (IMP)

10.0

ORF (BL)

0.0 M M (T) (C.)

F F (T) (C.)

1.0

RCO (IMP)

1.0

LCO (IMP)

0.5

RCO (BL)

0.5

LCO (BL)

0.0

0.0 M M (T) (C.)

F F (T) (C.)

M M (T) (C.)

F F (T) (C.)

The main conclusions from these results are: • • •

•

Girls out-perform boys in sub-components 1-6 in both TRAC+ Intervention and Comparison schools (as with Grade 2); this effect is significant (P<0.05) for BL and EL figures. There are too many variables to compare improvements with any reliability. In sub-components 1-6, girls from the TRAC+ Intervention schools appear to be benefitting more than the boys from the TRAC+ intervention (direct contrast with Grade 2 where the same trend is observed with boys improving more than girls). In sub-components 4 (SR) and 6 (RCO), boys from the Comparison schools out-performed boys from the TRAC+ Intervention schools on both EL performance and improvement; it may be that this is an effect of other NGO interventions (See 8.4, page 24) having (or targeting) a greater effect on boys. In sub-component 7 (LCO), boys from the Comparison schools are out-performing boys from the TRAC+ Intervention schools in their improvement BLEL although the difference is small.

8.4

NGO support

If results in Grades 2 and 3 are disaggregated by province, Comparison schools in Siem Reap province show substantially greater progress BLEL than schools in other provinces. This may be an effect from external factors, such as levels of NGO support in the area. Data on levels of NGO support is not accurately kept at any central of provincial level. The most relevant data found comes from the 21 Economic Census of Cambodia 2011 (CEC) provincial level reports . For comparison, the relevant figures that can be considered are: • • •

Population Number of establishments owned/rented by NGOs Number of person engaged belonging to the establishment rented by NGO

The CEC data is given in two formats: • •

As a percentage of the total for Cambodia Ranked by the (then) 24 provinces/municipalities within Cambodia, where #1 is the highest percentage ranking.

The CEC data is displayed in table 8.4.1 below.

21

Ministry of Planning (2013) Economic Census of Cambodia 2011 Provincial Reports (#01 BMC, #6 KTM, #13 PVH, #17 SRP, #21 TKO) National Institute of Statistics supported by JICA

24

Table 8.4.1: Province relative populations with relative NGO provision Province

Population Percentage

Number of establishments owned/rented by NGO

Rank (/24)

Percentage

Number of persons engaged belonging to the establishment rented by NGO

Rank (/24)

Percentage

Rank (/24)

SRP

6.7

6

8.9

3

13.8

2

TKO

6.3

7

0.9

20

0.4

21

BMC

5.1

9

6.1

4

8.6

4

KTM

4.7

10

3.5

7

3.5

7

PVH

1.3

18

0.8

21

0.5

20

The figures from table 8.4.1 above can be used to generate a ratio of NGO staff population to provincial population (e.g. SRP: 13.8/6.7 = 2.05, TKO: 0.4/6.3 = 0.06) and if these are considered as a part of the five target provinces, Siem Reap has a very high relative provision of NGO support (41.7% from the five provinces) and Takeo a very low relative provision of NGO support (1.3%). Although this figure does not account for (1) the type or scale of NGO funding or (2) differences between districts within provinces, it does nevertheless give a rough indication of the differing levels of NGO support for the target provinces and especially the relative 22 lack of NGO presence in Takeo province . Chart 8.4.1 below shows the data in pie chart form. Chart 8.4.1: Proportion of NGO staff adjusted for province populations

8.5

Promotion of struggling students

A barriers facing EGR in Cambodia is the requirement for schools to meet MoEYS-mandated targets for Child Friendly School (CFS) status. CFS can be given as: basic (lowest), intermediate or developed (highest). In order to achieve CFS ‘Developed’ status, primary schools require a promotion rate (pass rate) of 93% or better. For intermediate status, the requirement is for 90% (refer to Annex 4 for more details).

22

A possible reason is the high level of manufacturing industry in Takeo shows a ‘lack of need’ when NGOs evaluate provinces for possible project interventions.

25

23

Based on the small sample used in this survey and a general review of EGRA and student performance within Cambodia, these promotion figures do not reflect what is currently being attained: students are being promoted without sufficient reading skills to access the curriculum in their new grade. A solution to this problem is beyond the scope of TRAC+ project; highlighting the problem (once again) is another small step towards meaningful education quality improvement in Cambodia. For reasons of space, this report will consider the two most basic subcomponents of the EGRA. Data and graphs for all sub-components are available from WEI. Sub-component (1) and (3) will be looked at in detail: Sub-component

Rationale

(1) Letter Name Recognition (LNR)

The majority of the Grade 1 MoEYS textbook is concentrated on teaching consonants and vowels and how these are combined together into syllables and words. Without letter recognition, any ability to decode words or recognise familiar words is almost inaccessible. This sub-component should be the purest test of phonics (word decoding) ability. (Although in the EGRA tool, poor word selection limits the effectiveness of this sub-component.)

(3) Non-Word Reading (NWR)

The distribution of the raw scores of individual students allows an analysis of whether students can actually identify and combine letters with sufficient speed and accuracy to begin to access higher-level reading skills. The curriculum focus on letter recognition and word building means that by the end of Grade 1, a successful student should have a competent grasp of the Khmer alphabet and some ability to put letters together to make syllables and words. Figures for LNR and NWR will be shown by grade. Grade 1 Graph 8.5.1: LNR Grade 1 EL (T vs. C)

Graph 8.5.2: NWR Grade 1 EL (T vs. C) Non-Word Reading: Grade 1 Endline

60.0% 50.0%

Percent of students

Percent of students

Letter Name Recognition: Grade 1 Endline

40.0% 30.0% 20.0% 10.0% 0.0% Comparison

Intervention

LPM

100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% Comparison

Intervention

WPM

At the end of Grade 1, after 494 teaching hours, only 60% of TRAC+ Intervention students and 51% of Comparison students sampled were able to correctly identify ten Khmer characters in a minute. 15% of students in TRAC+ Intervention schools scored zero, 22% of students in Comparison schools scored zero. It does not seem possible that any of the students scoring zero should be able to pass onto Grade 2. This is an issue that requires further investigation. Non-Word Reading remains beyond the capacity of the majority of Grade 1 students.

23

Further info. see e.g.: RTI International (2015) Assessment of Early Grade Reading in the Education Sector in Cambodia Report for USAID

26

Grade 2 Graph 8.5.3: LNR Grade 2 BL and EL Intervention (T)

Graph 8.5.4: LNR Grade 2 BL and EL Comparison (C) Letter Name Recognition: Grade 2 Comparison

45.0% 40.0% 35.0% 30.0% 25.0% 20.0% 15.0% 10.0% 5.0% 0.0%

Percent of students

Percent of students

Letter Name Recognition: Grade 2 Treatment

Endline

Baseline

45.0% 40.0% 35.0% 30.0% 25.0% 20.0% 15.0% 10.0% 5.0% 0.0% Endline

LPM

Baseline

LPM

The LNR baseline at Grade 2 for both TRAC+ Intervention and Comparison schools shows a baseline of 60% of students able to read ten letters in 60 seconds. This correlates reasonably well with the Grade 1 endline figures. Although the 40% of students without this basic competency have the first 31 lessons of the Grade 2 curriculum to revise this basic content, it seems that these students are arriving in Grade 2 without the necessary baseline competency to access the curriculum. The TRAC+ Intervention schools show an improvement over the Comparison schools in terms of reducing the most struggling students by the endline. The number of students in the 0-10 LPM category is 18% (T) compared with 25% (C). If an LNR score of 20LPM is taken as an arbitrary minimum for promotion to Grade 3 (WEI suggests that 30LPM would be more appropriate) there are only 67% of TRAC+ Intervention students and 58% of Comparison students sampled with sufficient basic knowledge of the Khmer alphabet to begin to access the Grade 3 curriculum. These figures require further investigation and confirmation; but, if they are a true reflection of the situation, the data shows insufficient effectiveness of the TRAC+ intervention in addressing the problem of students with serious reading difficulties. Graph 8.5.5: NWR Grade 2 BL and EL Intervention (T)

Graph 8.5.6: NWR Grade 2 BL and EL Comparison (C)

90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

Non-Word Reading: Grade 2 Comparison

Percent of students

Percent of students

Non-Word Reading: Grade 2 TRAC+ Intervention

Endline

Baseline

WPM

90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

Endline

Baseline

WPM

The number of Grade 2 students who could not read five ‘non-words’ in a minute is partly related to the poor quality of the EGRA tool being used. However, TRAC+ Intervention schools show 32% of students sampled were able to decode ten or more non-words

27

in a minute, so this level of phonic decoding of words is quite achievable in this grade. Full analysis of this problem cannot take place until the EGRA tool is revised, but Graphs 8.5.5 and 8.5.6 show that the problem clearly exists. Grade 3 Graph 8.5.7: LNR Grade 3 BL and EL Intervention (T)

Graph 8.5.8: LNR Grade 3 BL and EL Comparison (C) Letter Name Recognition: Grade 3 Comparison

30%

30%

25%

25%

Percent of students

Percent of students

Letter Name Recognition: Grade 3 Treatment

20% 15% 10% 5%

20% 15% 10% 5% 0%

0%

Correct Letters Per Minute Endline

Baseline

Correct Letters Per Minute Endline

Baseline

Both TRAC+ Intervention and Comparison school groups at BL show a similar proportion (57-58%) of sampled students able to read 20 letters in one minute (i.e. one letter every three seconds) when entering Grade 3. This figure correlates with the Grade 2 Comparison EL figures. The 10% of TRAC+ Intervention students and 17% of Comparison students from the sample who cannot read 10 letters correctly in one minute will have no possibility of accessing the Grade 4 curriculum in any subject and are highly unlikely to have been able to access the Grade 3 curriculum either. The TRAC+ Intervention schools show an improvement over the Comparison schools in terms of reducing the most struggling students by the endline. The number of students in the 0-30 LPM category is 31% (T) compared with 37% (C). Thus, if an LNR score of 30LPM is taken as an arbitrary minimum for promotion to Grade 4 (and the author feels that 50LPM would be more appropriate) a promotion rate of 60-70% seems likely.

28

Graph 8.5.7: NWR Grade 3 BL and EL Intervention (T)

Graph 8.5.8: NWR Grade 3 BL and EL Comparison (C) Non-Word Reading: Grade 3 Comparison

80%

80%

70%

70%

Percent of students

Percent of students

Non-Word Reading: Grade 3 TRAC+ Intervention

60% 50% 40% 30% 20% 10%

60% 50% 40% 30% 20% 10% 0%

0%

WPM

WPM Endline

Baseline

Endline

Baseline

Phonetically decoding non-words shows improvement in TRAC+ Intervention against Comparison samples. However, only 63% of TRAC+ Intervention students sampled and 55% of Comparison students sampled are able to decode five words in a minute. As stated before, this is also partly a reflection on the quality of the EGRA tool being administered, but WEI considers that these students do not have the fundamental reading competencies to progress further through the curriculum. The MoEYS presented a long-term aim in 2014 that: “Every child who successfully completes Grade 3 will be able to read

with understanding at a level of 45 – 60 words per minute.” 24 The results shown in this sample suggest that this aim is a long way from being achieved. 9

Recommendations for TRAC+ Year 3 and beyond

The recommendations for improvement within the EGRA test are difficult, as alterations to the testing protocol will invalidate comparing TRAC+ year 2 and year 3 data. However, if an improved EGRA test was available in time to create a TRAC+ year 3 BL figure, this would improve data reliability. TRAC+ does not have the human resources to modify, pilot and validate an improved EGRA test. Basic flaws within the EGRA where correction is required are listed in Table 5.1.1 (page 12). Recommendation 1: EGRA for TRAC+ year 3 to use improved EGRA test IF such a resource is available in time. Comparison schools were not selected with sufficient rigour; however, it is practical to continue using the current group of Comparison schools for the final year of TRAC+. The lack of validity of Comparison school selection will need to be taken into consideration in any TRAC+ evaluation. Recommendation 2A: No change in school selection for TRAC+ year 3. Recommendation 2B: Increase sample size in Comparison schools to enable more robust analysis of results. Recommendation 2C: Comparison school selection issues noted in subsequent EGRA or other performance analysis.

24

Kann Puthy (2014) Variations in early reading assessment results between rural and urban populations presented to UNESCO Institute for Statistics conference ‘What works in oral assessments of early reading?’ 21 July 2014, Montreal, Canada on behalf of MoEYS, Cambodia (http://tinyurl.com/hop9538 retrieved 09-Oct-2016)

29

Although the EGRA assesses all sub-components of reading, the TRAC+ interventions (RRS and the use of LCs) primarily address the most basic reading ability: the ability to decode words using letter-sound association and combining sounds. In effect, the intervention is almost entirely phonics-based. Although improvements in decoding words can lead to increased reading fluency, a total reading approach for children would require further interventions using correctly levelled connected text for comprehension and fluency especially in Grade 2 (from lesson 32 onwards) and as a matter of extreme urgency in Grade 3 where basic phonetic knowledge and the ability to decode words is assumed and required to access the Khmer language curriculum (and indeed other core subjects). Recommendation 3A: Additional ‘connected text’ resources (e.g. mini-libraries) to be provided as RRS interventions. (This can be linked with the RRS interventions in the Reading Benchmark Standards books, i.e. specific books can be linked as remedial interventions for specific reading intervals.) Recommendation 3B: Additional levelled readers to be distributed in multiple sets to allow paired or group reading activities. Students are being promoted with insufficient reading ability to access the curriculum at a new grade; this becomes more noticeable at higher grades and there is pressure on SDs to achieve MoEYS-mandated promotion figures as these are key indicators to achieving higher status as a Child-Friendly School (CFS). Students who lack basic reading skills require increased remedial work: this could mean a return to long vacation remedial classes or greater emphasis on RRS or other similar interventions over the academic year. Recommendation 4: TRAC+ to present relevant EGRA data to MoEYS with priority of stopping promotion of students who do not have the requisite competencies to access the next grade. Some crucial RRS interventions were missed in the early part of the academic year 2015-2016 because of the delay in beginning the implementation of TRAC+ year 2 activities. The start of the year is especially crucial as: • •

Students can fall further behind if problems are not picked up quickly. In Grade 2, the first 31 lessons are revision of Grade 1 and are the foundation for the rest of the teaching in that Grade.

Recommendation 5: TRAC+ activities to start promptly at the beginning of the 2016-17 academic year. All recommendations are made on behalf of WEI as a basis for further discussion with stakeholders in EGR in Cambodia.

30

Annex 1: Sample Population for Testing Table A1.1: Sample population for Grade 1 Endline

Total Population Province

District

TRAC+ Intervention

Sample Population

Comparison

TRAC+ Intervention

Comparison

T

F

T

F

T

F

T

F

Preah Neitr Preah

399

189

149

70

37

22

16

8

Phnom Srok

434

200

158

68

48

24

22

9

Svay Check Total in Banteay Meanchey

434

218

300

146

48

25

29

15

1267

607

607

284

133

71

67

32

638

299

106

53

60

30

10

3

638

299

106

53

60

30

10

3

Kulen

383

179

243

113

59

29

21

12

Rovieng

547

249

143

72

63

28

7

4

Sangkum Thmei

378

170

109

47

51

27

6

3

439

160

269

70

23

12

6

3

1747

758

764

302

196

96

40

22

Chikreng

970

456

239

110

82

42

23

12

Puok Total in Siem Reap

701

339

87

46

53

25

12

6

1671

795

326

156

135

67

35

18

Kirivong

470

249

108

55

59

29

16

8

Borei Chulsar

353

133

109

50

29

12

13

5

477

215

264

121

54

28

19

9

1300

597

481

226

142

69

48

22

6623

3056

2284

1021

666

333

200

97

10.1%

10.9%

8.8%

9.5%

Banteay Meanchey

Kampong Thom

Prasath Ballang Total in Kampong Thom

Preah Vihear

Chey Sen Total in Preah Vihear Siem Reap

Takeo

Samrong Total in Takeo Grand Total

% of eligible students sampled

31

Table A1.2: Sample population for Grade 2 Baseline

Total Population Province

District

TRAC+ Intervention T F

Comparison T

F

Preah Neitr Preah

528

209

151

83

Phnom Srok

436

223

139

387

193

1351

Endline

Sample Population TRAC+ Comparis Intervention on T F T F

Sample Population TRAC+ Comparison Intervention T F T F

23

13

8

5

30

19

9

6

58

42

22

21

8

44

24

21

9

290

149

41

23

28

14

38

21

26

13

625

580

290

106

58

57

27

112

64

56

28

585

234

102

61

58

26

12

7

59

32

9

6

585

234

102

61

58

26

12

7

59

32

9

6

Kulen

294

139

213

106

30

13

8

4

68

32

21

11

Rovieng Sangkum Thmei

427

207

124

62

39

26

9

6

38

23

7

4

405

189

81

43

41

21

6

4

39

19

6

3

Chey Sen Total in Preah Vihear

325

136

167

66

25

14

6

4

24

16

6

3

1451

671

585

277

135

74

29

18

169

90

40

21

Chikreng

795

346

317

147

63

26

28

17

67

33

27

16

496

236

103

49

40

21

20

12

38

17

24

12

1291

582

420

196

103

47

48

29

105

50

51

28

497

221

94

43

42

17

11

5

44

20

14

7

361

153

107

59

27

14

6

3

23

14

14

6

451

218

245

109

43

19

16

6

41

21

16

7

1309

592

446

211

112

50

33

14

108

55

44

20

5987

2704

2133

1035

514

255

179

95

553

291

200

103

8.6%

9.4%

8.4% 9.2%

Banteay Meanchey

Svay Check Total in Banteay Meanchey Prasath Kampong Thom Ballang Total in Kampong Thom

Preah Vihear

Siem Reap

Puok Total in Siem Reap

Takeo

Kirivong Borei Chulsar

Samrong Total in Takeo Grand Total

% of eligible students sampled

9.2%

10.8% 9.4% 10.0%

32

Table A1.3: Sample population for Grade 3 All Grade 3 TRAC+ Intervention schools (Groups A and B) were sampled, even though this TRAC+ intervention is only applied to Group A schools. Total Population Province

Banteay Meanchey

District

TRAC+ Intervention T F

Comparison T

F

197

145

58

393

197

135

Preah Neitr Preah

385

Phnom Srok

Baseline

Endline

Sample Population TRAC+ Compariso Intervention n T F T F

Sample Population TRAC+ Comparison Intervention T F T F

21

10

10

3

28

16

18

9

65

38

18

19

11

38

18

19

11

382

192

260

141

36

19

28

19

37

23

29

19

1160

586

540

264

95

47

57

33

103

57

66

39

550

268

124

50

55

30

12

5

50

28

9

4

550

268

124

50

55

30

12

5

50

28

9

4

Kulen

332

151

151

73

29

18

7

5

62

33

17

12

Rovieng Sangkum Thmei

395

206

110

50

36

24

9

6

38

24

7

5

390

200

80

40

42

24

6

3

41

20

6

3

Chey Sen Total in Preah Vihear

340

142

138

54

22

11

6

3

21

11

5

3

1457

699

479

217

129

77

28

17

162

88

35

23

Chikreng

626

318

229

99

45

26

20

12

53

31

18

12

557

263

96

43

41

16

25

14

45

21

25

14

1183

581

325

142

86

42

45

26

98

52

43

26

Kirivong

518

242

97

52

45

20

13

6

43

20

13

7

Borei Chulsar

364

158

105

47

28

15

6

3

30

14

15

8

524

233

273

119

38

19

19

7

41

19

19

8

1406 5756

633 2767

475 1943

218 891

111 476

54 250

38 180

16 97

114 527

53 278

47 200

23 115

8.3%

9.0% 9.3% 10.9%

Svay Check Total in Banteay Meanchey Prasath Kampong Thom Ballang Total in Kampong Thom

Preah Vihear

Siem Reap

Puok Total in Siem Reap

Takeo

Samrong Total in Takeo Grand Total

% of eligible students sampled

9.2%

10.0% 10.3% 12.9%

33

Annex 2: Schools Table A2.1: School participating in TRAC+ as TRAC+ Intervention or Comparison Province District Banteay Meanchey

Schools (T)

(C)

Preah Neitr Preah

10

4

Phnom Srok

12

6

*

12

6

35

16

16

4

16

3

Kulen

11

3

Rovieng

11

3

Sangkum Thmei

13

2

7

2

42

10

Chikreng

10

5

Puok Total in Siem Reap

11

4

21

8

Kirivong

13

4

Borei Chulsar

5

4

7

3

25

11

138

50

Svay Check Total in Banteay Meanchey Kampong Thom

Preah Vihear

Prasath Ballang Total in Kampong Thom

Chey Sen Total in Preah Vihear Siem Reap

Takeo

Samrong Total in Takeo *

Overall Total One school was removed from Svay Check District in BMC at the end of TRAC+ year 1.

34

Annex 3: School level participants trained in EGRA orientation Table A3.1: Participants in EGRA orientation for the school year 2015-2016 Province BMC

KTM PVH

SRP TKO

Total

District Preah Neit Preah Phnom Srok Svay Chek Prasat Balang Cheysen Kulen Rovieng Sangkumtmey Chikreng Pouk Borei Chulsar Kirivong Samraung

SDs (F) 12(1) 16(1) 18(0) 17(1) 7(1) 9(0) 12(1) 15(2) 8(4) 7(0) 6(0) 13(2) 8(2) 148(15)

Teachers (F) 4(1) 0 1(0) 0 1(0) 0 0 0 6(3) 0 3(1) 1(0) 2(1) 18(6)

LC (F) 0 0 3(0) 0 0 0 0 0 1(0) 0 0 3(2) 5(0) 12(2)

POE/DOE (F) 1(1) 1(0) 1(0) 3(0) 2(0) 2(0) 2(0) 0 1(1) 1(1) 1(0) 1(0) 2(0) 18(3)

WVI (F) 1(0) 0 2(0) 0 1(0) 1(0) 1(0) 0 1(0) 0 0(0) 0 0 7(0)

Total (F) 18(3) 17(1) 25(0) 20(1) 11(1) 12(0) 15(1) 15(2) 17(8) 8(1) 10(1) 18(4) 17(3) 203(26)

POE = Provincial Office of Education DOE = District Office of Education

35

Annex 4: DTMT Guidelines for Basic Education [extract] Relevant text from DTMT Working Guidelines for Basic Education (working document, Oct. 2015) pages 13-15. Additional information added for clarity is in red italics. Identification (of CFS): In principle, CFSs are ranked into three different classifications: Basic Level CFS Refers to any school which has not achieved outcomes related to the six dimensions fully and where students’ annual learning outcomes are still low. Intermediate Level CFS Refers to any school which has achieved good outcomes related to the six dimensions and where students’ annual learning outcomes are average. Developed Level CFS Refers to any school which has well achieved outcomes related to the six dimensions fully and where students’ annual learning outcomes are quite good. Learning Outcome/Quality-based Assessment: In this type of assessment, key emphasis is on Net Admission Rate, Gross Enrolment Rate, Promotion Rate, Exam Passing Rate, Dropout Rate and Completion Rate in the academic year against targets set for each academic year. On this basis, we can rank CFS by qualities as follows: Developed Level CFS in terms of Quality: For any primary school which has achieved: - Net Admission Rate > 91% - Net Enrolment Rate > 93% - Repetition Rate < 5% - Dropout Rate < 2% - Completion Rate > 92% (Repetition Rate and Dropout Rate for Developed CFS status requires a Promotion Rate of >93%)

Intermediate Level CFS in terms of Quality: For any primary school which has achieved: - Net Admission Rate 85-91% - Net Enrolment Rate 90-93% - Repetition Rate 5-6% - Dropout Rate 2-4% - Completion Rate 85-92% (Repetition Rate and Dropout Rate for Intermediate CFS status requires a Promotion Rate of 90-93%)

Basic Level CFS in terms of Quality: - Net Admission Rate < 85% - Net Enrolment Rate < 90% - Repetition Rate > 6% - Dropout Rate > 4% - Completion Rate < 85% (All schools can achieve these figures as they are simply the cut-off before intermediate status can be reached)

36