PEDIATRICS/ORIGINAL RESEARCH

Validation of the Ottawa Knee Rule in Children: A Multicenter Study

Blake Bulloch, MD Gina Neto, MD Amy Plint, MD Rodrick Lim, MD Per Lidman, MD Martin Reed, MD Cheri Nijssen-Jordan, MD Milton Tenenbein, MD Terry P. Klassen, MD A Pediatric Emergency Researchers of Canada Study From the Department of Emergency Medicine, Children’s Hospital, Winnipeg, Manitoba (Bulloch, Tenenbein); the Division of Emergency Medicine, Children’s Hospital of Eastern Ontario, Ottawa, Ontario (Neto, Plint); the Department of Emergency Medicine, Children’s Hospital of Western Ontario, London, Ontario (Lim); the Department of Emergency Medicine, Stollery Children’s Hospital, Edmonton, Alberta (Lidman, Klassen); the Department of Pediatric Radiology, Children’s Hospital, Winnipeg, Manitoba (Reed); and the Department of Emergency Medicine, Alberta Children’s Hospital, Calgary, Alberta (Nijssen-Jordan), Canada.

Study objective: The main objective of this study was to determine the sensitivity and specificity of the Ottawa Knee Rules when they were applied to children. The secondary objective was to determine post hoc whether use of the rules would reduce the number of knee radiographs ordered. Methods: This prospective, multicenter validation study included children aged 2 to 16 years who presented to the emergency department with a knee injury sustained in the preceding 7 days. Children were assessed for the variables comprising the Ottawa Knee Rules, and physicians ordered radiographs at their discretion. A positive outcome was defined as any fracture. A negative outcome was defined as children who did not have a fracture on radiograph or, if no radiograph was obtained, were asymptomatic after 14 days. Results: A total of 750 children were enrolled. The mean age was 11.8±3.1 years, and 443 (58.7%) were male patients. Seventy children had fractures. Radiography was performed for 670 children, whereas 80 children had only a structured telephone interview. The Ottawa Knee Rules were 100% sensitive (95% confidence interval [CI] 94.9% to 100%), with a specificity of 42.8% (95% CI 39.1% to 46.5%). Only 460 children would have required a radiograph if radiographs had been performed according to the Ottawa Knee Rules, which would have resulted in an absolute reduction of 209 (31.2%) radiographs. Conclusion: The Ottawa Knee Rules are valid in children and have the potential to decrease the use of radiography in children with knee injuries. [Ann Emerg Med. 2003;42:48-55.]

Copyright © 2003 by the American College of Emergency Physicians. 0196-0644/2003/$30.00 + 0 doi:10.1067/mem.2003.196

4 8

ANNALS OF EMERGENCY MEDICINE

42:1

JULY 2003

OTTAWA KNEE RULE VALIDATION IN CHILDREN Bulloch et al

INTRODUCTION

Of all children presenting to the emergency department (ED), between 10% and 20% have a chief complaint involving an extremity injury.1,2 More than 90% of children presenting with extremity injuries undergo radiographic studies.1-4 Approximately 8% of these injuries involve the knee.2 Clinical decision rules have been developed for adults and have been shown to be sensitive in the detection of fractures. The implementation of these rules has reduced the number of radiographs ordered and may decrease health care costs and waiting times in the ED.5-9 Studies by Stiell et al8,9 in the adult population have found that the Ottawa Knee Rules are 100% sensitive for detecting clinically significant fractures in adults with knee injuries. The Ottawa Knee Rules state that knee roentgenography is required only for patients who have acute knee injury and at least 1 of the following findings related to age, tenderness, or function (Figure): (1) aged 55 years or older, (2) tenderness at head of fibula, (3) isolated tenderness of patella (no bone tenderness of knee other than patella), (4) inability to flex to 90

Figure.

The Ottawa Knee Rules. Reprinted with permission from Stiell IG, Wells GA, Hoag RH, et al. Implementation of the Ottawa Knee Rule for the use of radiography in acute knee injuries. JAMA. 1997;278:2075-2079. Copyrighted 1997, American Medical Association.10

A knee x-ray series is only required for knee injury patients with any of these findings: 1. age 55 years or older or 2. isolated tenderness of patella* or 3. tenderness at head of fibula or 4. inability to flex to 90° or 5. inability to bear weight both immediately and in the emergency department (4 steps)** *No bone tenderness of knee other than patella. ** Unable to transfer weight twice onto each lower

JULY 2003

42:1

degrees, and (5) inability to bear weight immediately and in the ED (defined as an inability to transfer weight twice onto each lower limb, regardless of limping).8 Clinically significant fractures were defined as any bone fragment at least 5 mm in breadth or any avulsion fracture regardless of size, if associated with complete disruption of tendons or ligaments. Applying these rules resulted in a 28% reduction in the number of radiographs ordered.10 As pointed out by Plint et al,7 extremity injuries in children have several complicating factors. Children have open epiphyseal plates (growth plates) and may sustain Salter-Harris fractures. Assessment of pain and weight bearing may also be difficult in a young or frightened child. However, the authors were able to apply the Ottawa Ankle Rules in children in a prospective study and found them to be 100% sensitive.7 Only 1 small previous study has been published that prospectively applied the Ottawa Knee Rules to children with knee injuries.11 However, this study enrolled only 234 patients and identified only 13 fractures, which resulted in wide confidence intervals (CIs). The main objective of this study was to determine the sensitivity and specificity of the Ottawa Knee Rules when they were applied to children aged 2 to 16 years. The rule was modified to include bone fragments of any size as being a significant fracture because there is no evidence in the literature to suggest otherwise in this population. A secondary objective was to determine post hoc whether use of the rules would reduce the number of knee radiographs ordered for children presenting with knee injuries to the ED. M AT E R I A L S A N D M E T H O D S

Patella Head of fibula

limb regardless of limping.

ANNALS OF EMERGENCY MEDICINE

This prospective, multicenter validation study was conducted between January 1999 and May 2002 at 5 urban academic pediatric EDs with a combined census of approximately 195,000 patient visits yearly. All children who were between 2 and 16 years of age and presented with acute injuries to the knee were eligible for the study. The lower age limit of 2 years was based on the article validating the Ottawa Ankle Rule,7 in which

4 9

OTTAWA KNEE RULE VALIDATION IN CHILDREN Bulloch et al

2 years was chosen as a cutoff age that maximized enrollment but minimized difficulties with cooperation. Children were eligible for enrollment 24 hours a day. As in the studies by Stiell et al,8-10 the knee was defined broadly as the patella, the head and neck of the fibula, the proximal 8 cm of the tibia, and the distal 8 cm of the femur. This study was approved by the institutional review board at each of the participating centers. Informed written consent was obtained from all parents or legal guardians. Inclusion criteria consisted of children who were aged 2 to 16 years, had sustained a knee injury in the preceding 7 days, and had evidence of bony injury to the knee on physical examination. Children were excluded if they had isolated injuries to the skin (abrasion or laceration), were referred from outside the hospital with a diagnosed fracture, were returning for reassessment of the same knee injury, had an altered level of consciousness (eg, intoxication) or multiple distracting injuries, had a history of metabolic bone disease (eg, osteogenesis imperfecta, rickets), or had an underlying disease with sensory abnormalities (eg, meningomyelocele). Emergency faculty physicians or emergency medicine fellows oriented and trained in the application of the Ottawa Knee Rules assessed all children, and radiographs were ordered at the physicians’ or fellows’ discretion. Training was conducted by each site’s principal investigator and consisted of either individual sessions or group lectures. These sessions were used to tell physicians how to perform and interpret the components of the rules. Physicians were asked to record their examination results on a standardized data collection sheet before reviewing the radiographs. Children were assessed for the variables composing the Ottawa Knee Rules: tenderness at the head of the fibula, isolated tenderness of the patella, inability to flex 90 degrees, and inability to bear weight immediately and in the ED (4 steps). Other information collected included age, sex, mechanism of injury, and time from injury. When possible, a second emergency physician or fellow independently examined the child to test the interobserver reliability of the rule when it was applied to children.

5 0

The radiographs were reviewed by the staff physician for immediate treatment and were interpreted by a radiologist within 48 hours. The radiologist was blinded to the Ottawa Knee Rules results. To assess interobserver reliability for radiograph interpretation, 1 pediatric radiologist at the coordinating center reviewed 20% of all radiographs while blinded to the interpretation of the initial radiologist. If there was disagreement between the first and second radiologist, a third radiologist, also blinded to the results of the Ottawa Knee Rules, assessed the radiograph. In cases in which telephone follow-up was achieved if the child was asymptomatic, the radiograph results were considered negative. If no follow-up was obtained, the result was based on the third radiologist’s interpretation. A positive outcome was defined as any fracture, regardless of size. The radiologist made the final determination of fracture, blinded to information on the data collection sheet. Patients diagnosed clinically as having Salter-Harris type I fractures (a normal radiograph result but tenderness at the growth plate) were noted and followed up clinically. A negative outcome was defined as children who did not have a fracture on radiograph or, if no radiograph was obtained, were asymptomatic after 14 days. Because not all fractures may be readily identified on initial radiographs, attempts were made to call all children with normal radiographs to determine whether they had remained asymptomatic.12 Any child who was not asymptomatic on followup was instructed to return to the ED for reassessment. The telephone interview included the following questions that could be answered yes or no to ensure that no missed injury was later identified: (1) “Is the pain better?” (2) “Is the ability to walk better?” (3) “Does your child walk without a limp or pain?” (4) “Does your child require assistance to walk (crutches, cast, splint)?” (5) “Has your child returned to normal physical activities?” and (6) “Has an x-ray of the knee been done anywhere else since your visit?” These questions were scripted and have been successfully applied in previous studies.5-10 Nurse research assistants made all telephone follow-up calls and were blinded to the results of the Ottawa Knee Rule.

ANNALS OF EMERGENCY MEDICINE

42:1

JULY 2003

OTTAWA KNEE RULE VALIDATION IN CHILDREN Bulloch et al

A flow chart was used to track the numbers of patients presenting with knee injuries during the study period. Sensitivity, specificity, positive predictive value, negative predictive value, and likelihood ratios for the Ottawa Knee Rules decision rule were derived with 95% CIs. The interobserver agreement for radiograph interpretation and the reliability of application of the Ottawa Knee Rules were measured using the κ coefficient, the potential agreement beyond pure chance. The software used for statistical analysis was SAS (version 8.2, SAS Institute, Inc., Cary, NC) with 95% CIs calculated using Confidence Interval Analysis software (version 1.1, BMJ Publishing Group, London, United Kingdom). We expected the Ottawa Knee Rule to attain a 99% sensitivity, with a CI from 97% to 100%. A sample size of 96 children with positive radiographic findings was required to ensure 2% accuracy around the sensitivity estimate, with a power of 80% and false-positive rate of 5%. Because previous studies in the adult population reported that 6% to 11% of patients presenting with acute blunt injuries to the knee had positive radiograph results, it was determined that between 873 and 1,600 children would need to be enrolled. However, because of funding constrictions, enrollment was stopped after 750, causing wider CIs.

R E S U LT S

During the study period from January 1, 1999, to May 31, 2002, a total of 1,139 children were eligible for enrollment. Of these, 92 met exclusion criteria and 297 were missed, leaving a total of 750 (66%) children that were enrolled at the 5 centers (Table 1). The overall mean age in the study group was 11.8±3.1 years, and 443 (58.7%) were male patients, which was similar to those who were not enrolled, in which the average age was 12.0±2.9 years and male patients accounted for 52.9% (n=206). The number of children enrolled at each center varied because of the length of time each center was involved (Table 1). There were 45 children between 2 and 5 years of age, 375 between 6 and 12 years, and 330 between 13 and 16 years. Seventy children had fractures, 2 had Salter-Harris type I fractures diagnosed clinically on initial examination, and 680 children were diagnosed with soft-tissue injuries, defined as sprains, strains, or contusions. Of the 2 children with Salter-Harris type I fractures, 1 was subsequently confirmed by magnetic resonance imaging and the other on orthopedic follow-up examination.

Table 1.

Characteristics of the patient population. Center Characteristic Months patients enrolled, No. Eligible children, No. Patients enrolled, No. Patients eligible but not enrolled, No. Age, mean±SD, y Male sex, No. Outcome assessment, No. (%) Radiograph only Telephone follow-up only Radiograph and telephone follow-up Mechanism of injury, No. (%) Twist Direct trauma Fall Other

JULY 2003

42:1

1

2

3

4

5

Total (%)

41 483 373 110 11.4±3.1 216

29 511 300 211 12.1±3.2 172

9 64 41 23 12.3±2.6 26

8 47 32 15 12.5±2.7 23

4 39 9 30 9.8±3.6 5

91 1,139 750 (66) 389 (34) 11.8±3.1 443 (58.7)

78 (20.9) 27 (7.2) 268 (71.9)

30 (10) 38 (12.6) 228 (75.5)

31 (75.6) 10 (24.4) 0

0 3 (9.4) 29 (90.6)

0 2 (22.2) 7 (77.8)

138 (18.4) 80 (10.6) 532 (70.7)

61 (16.4) 153 (41.2) 154 (41.5) 3 (0.8)

73 (24.5) 81 (27.2) 128 (42.9) 16 (5.4)

6 (18.2) 12 (36.4) 13 (39.4) 2 (6.1)

6 (20.7) 11 (37.9) 10 (34.5) 2 (6.9)

1 (14.3) 2 (28.6) 4 (57.1) 0

147 (19.6) 259 (34.5) 309 (41.2) 35 (4.7)

ANNALS OF EMERGENCY MEDICINE

5 1

OTTAWA KNEE RULE VALIDATION IN CHILDREN Bulloch et al

The types of fractures and their frequencies are noted in Table 2 according to age. To assess interobserver reliability, 1 pediatric radiologist reviewed 28% (n=191) of all radiographs while blinded to the interpretation of the initial radiologist. The κ value was high (0.75; 95% CI 0.62 to 0.87). Six hundred seventy children underwent radiography. Eighty children received a structured telephone interview only as a substitute for radiograph (a proxy outcome). A total of 532 children had both a radiograph and a telephone interview. On telephone follow-up, 3 children were still symptomatic and were asked to return to the ED. None of these children were found to have fractures on follow-up assessment.

Table 2.

Types of fractures identified according to different age groups. Age, y Fracture Type Patellar Tibial spine Proximal tibia Tibial plateau Tibial tuberosity Distal femur SH type I distal femoral epiphysis* SH type I Fibular head Total

Overall, the Ottawa Knee Rules were 100% sensitive (95% CI 94.9% to 100%) with a specificity of 42.8% (95% CI 39.1% to 46.5%), as illustrated in Table 3. To assess interobserver reliability in the application of the knee rules by physicians, 48 children were examined by 2 physicians. The κ value was 0.85 (95% CI 0.69 to 1.0) overall, but ranged from 0.62 in the youngest age group to 1.0 in the oldest (Table 4). The secondary objective of this study was to determine post hoc whether use of the rules would reduce the number of knee radiographs ordered for children presenting with knee injuries to the ED. There were a total of 670 radiographs performed (89% of all children enrolled), but only 460 would have required a radiograph if radiographs had been performed according to the Ottawa Knee Rules, which would have resulted in an absolute reduction of 209 (31.2%) radiographs (95% CI 27.7% to 34.7%; Table 5). DISCUSSION

2–5

6–12

13–16

0 0 4 1 0 0 0 0 0 5

11 8 4 0 1 4 0 2 1 31

8 11 2 5 4 1 3 0 0 34

SH, Salter-Harris. * With widening of epiphyses on radiograph.

To our knowledge, no other study has prospectively examined such a large number of children with knee injuries to determine the sensitivity and specificity of a clinical decision rule. To avoid changing physician management, children did not need to have a radiograph to be enrolled in the study. It was thought that all children who were asymptomatic on follow-up at 14 days postinjury did not experience a fracture. As with adults, the Ottawa Knee Rules are 100% sensitive in identifying fractures of the knee. Unlike the adult rule,

Table 3.

Performance of the Ottawa Knee Rules in the identification of knee fractures in children according to age. Performance Criteria, % (95% CI) Age Group 2–5 y (n=45) 6–12 y (n=375) 13–16 y (n=330) Overall (n=750)

5 2

Sensitivity

Specificity

Positive Predictive Value

Negative Predictive Value

Likelihood Ratio

100 (47.8–100) 100 (88.8–100) 100 (89.7–100) 100 (94.9–100)

60 (43.3–75.1) 47.1 (41.8–52.4) 35.5 (30.0–40.9) 42.8 (39.1–46.5)

23.8 (8.2–47.2) 14.6 (9.8–19.3) 10.7 (6.6–14.7) 15.3 (12.0–18.5)

100 (85.8–100) 100 (97.7–100) 100 (96.5–100) 100 (98.7–100)

2.5 1.9 1.6 1.7

ANNALS OF EMERGENCY MEDICINE

42:1

JULY 2003

OTTAWA KNEE RULE VALIDATION IN CHILDREN Bulloch et al

in which only fracture fragments larger than 5 mm were considered significant, we considered all fractures to be significant, regardless of size. This decision was based on a lack of evidence suggesting that fracture fragments smaller than 5 mm are insignificant in this population. The average age of children in this study was 11.8 years, with a range of 2 to 16 years. This range, combined with the small number of children enrolled in the 2- to 5-year age group, may limit the clinical utility of the Ottawa Knee Rules in younger children. However, because of the enrollment size of 750 children, we believe that this study is applicable to the average child who presents to the ED with knee injuries. The secondary objective, to determine post hoc whether use of the rules would reduce the number of knee radiographs, was calculated according to all the children enrolled in the study, another consideration in support of not obtaining a radiograph in all children, because it would change standard practice and falsely increase the number of radiographs that could have been omitted. We calculated that 209 radiographs could have been eliminated (ie, all children who had negative rules and a normal radiograph result). Using the Ottawa Knee Rules would have resulted in an overall 31.2% reduction in radiograph use in our study. However, until an implementation trial is performed, the true reduction in radiography cannot be determined. We believe that the use of clinical decision rules, especially when related to extremity injury, is impor-

tant. Although radiographs may be relatively low cost, they are a high-volume test. Additionally, implementing these rules may decrease time spent in the ED. Although this measure was not used in this study, Stiell et al10 found that adults who underwent radiography spent an average of 118.8 minutes in the ED versus 85.7 minutes if they did not. Previous studies have examined the use of several knee rules. 11,13-20 In the study by Seaberg et al, 13 a comparison between the Ottawa Knee Rules and the Pittsburgh rules was made. A total of 934 patients were enrolled and ranged in age from 6 to 96 years. Contusions, sprains, and strains accounted for 77% (732) of the total diagnoses, and the researchers found that 11% (103) had fractures. The Ottawa Knee Rules were applied to only 750 patients because the patients had to be 18 years of age or older for the rule to be applicable. The sensitivity of the Ottawa Knee Rules was 97% (95% CI 90% to 99%), with a specificity of 27% (95% CI 23% to 30%). Three patients with fractures were misclassified with the Ottawa Knee Rules. The Pittsburgh rule was applied to children in the study, which revealed 10 fractures in 55 children (18.2%). The Pittsburgh rules were 100% sensitive (95% CI 66% to 100%) and 41% specific (95% CI 27% to 57%). Because of the wide 95% CIs, the researchers concluded that the rules should not be used in children. With their results, a likelihood ratio of 1.7 can be calculated, which is identical to ours.

Table 4.

Table 5.

Interobserver reliability in the application of the Ottawa Knee Rules according to age.

Baseline and potential reductions in radiographs that could be achieved at each center by using the Ottawa Knee Rules. Center

κ* (95% CI)

Age Group

No. of Radiographs 2–5 y (n=5) 6–12 y (n=22) 13–16 y (n=21) Overall (n=48)

0.62 (0–1.0) 0.81 (0.55–1.0) 1.0 (1.0–1.0) 0.85 (0.69-–1.0)

κ reliability statistic is a measure of repeatability of a diagnosis or a prediction. A κ below 0.4 is considered poor, a κ of 0.7 and up is excellent, and one between 0.4 and 0.7 is fair to good. The CI is the usual range of what the κ will be if repeated 100 times with the same sample sizes.24 *The

JULY 2003

42:1

ANNALS OF EMERGENCY MEDICINE

1

Baseline 346 radiographs Radiographs if 235 Ottawa Knee Rules used, No. Absolute reduc- 111 (32.1) tion, No. (%)

2

3

4

5

Total

257

31

29

7

670

167

28

24

6

460

89 (34.8) 3 (9.7) 5 (17.2) 1 (14.3) 209 (31.2)

5 3

OTTAWA KNEE RULE VALIDATION IN CHILDREN Bulloch et al

A comparison of the Ottawa Knee Rules with the Pittsburgh rules would have been helpful, given our large sample size. However, the definition of inability to walk is different for both rules.13 The Pittsburgh rule requires 4 full steps on the toe pads and heel pad of each foot. The Ottawa rule states that any foot transfer is adequate, and, as such, severe limping is considered ability to bear weight. It was believed that the study would be complicated if 2 definitions were used. The Ottawa Knee Rules have been shown in other studies to have good interobserver agreement, with κ values ranging from 0.77 to 0.91.8-10 The Pittsburgh rules’ definition of ability to bear weight has never been examined for interobserver reliability.13 In our study, a total of 48 children were examined independently by 2 physicians to determine interobserver agreement. The κ value was 0.85 (95% CI 0.69 to 1.0), which is similar to that observed in the adult population. The κ reliability statistic is a measure of repeatability of a diagnosis or a prediction; a κ value below 0.4 is considered poor, a κ of 0.7 and higher is excellent, and one between 0.4 and 0.7 is fair to good. Unfortunately, few children in the youngest age group of 2 to 5 years were interobserver cases, which limits the rules’ ability to reliably predict the absence of a fracture in this age group. Clinical prediction rules often do not perform well when tested in a population other than that in which it was derived.21,22 However, several investigators have performed validation studies of the Ottawa Knee Rules in different adult populations and found them to be reliable. Khine et al11 enrolled 234 children in a prospective study examining the use of the Ottawa Knee Rules in children and found them to have a sensitivity of 92% (95% CI 64% to 99%). In their study, 1 of 13 fractures was missed by the rule, a nondisplaced fracture of the proximal diaphysis of the tibia. Our study is the largest prospective study in children that has assessed this rule, and it performed well with narrow CIs. Strict methodologic standards for the development and testing of clinical decision rules have been established.22,23 In our study, the outcome is biologic and clearly defined as a fracture of any size, which can be reproduced in other settings. The outcome measure was

5 4

assessed blindly. The predictor variables were standardized, have been shown to have good interobserver agreement, and were performed before knowledge of the outcome. This study is limited because not all children with isolated knee injuries were enrolled, which introduces the possibility of selection bias. A total of 34% (389/1,139) of eligible children were not enrolled. These children tended to have more minor injuries, with only 16 (4%) having experienced fractures. A review of the medical records indicated that each of these 16 children with fractures had documentation to suggest that a radiograph would have been appropriate according to application of the knee rule. The study would also have been strengthened if there had been more interobserver cases. Although this was a multicenter study, the majority of patients were enrolled at 2 of the 5 institutions, which, combined with the fact that all of the hospitals were academic, limits its external validity. Additionally, 80 children had a telephone interview as a surrogate of outcome. Because the radiograph is the criterion standard when the sensitivity and specificity analysis is considered, telephone follow-up may not accurately capture all missed fractures. The Ottawa Knee Rules are valid in children who are older than 5 years and have knee injuries. These rules have been prospectively shown to be sensitive in the identification of fractures and have the potential to decrease the use of radiography in children with knee injuries. We thank Mary Cheang, PhD, for her statistical consultation. Author contributions: BB and TPK conceived the study and designed the initial version of the trial. BB obtained research funding. A meeting was held in Ottawa and attended by BB, GN, AP, PR, MR, CN-J, and TPK, at which time the study design was finalized. RL and MT reviewed the final design before implementation. BB, GN, RL, PL, and CN-J supervised the conduct of the trial and data collection. BB drafted the manuscript, and all authors contributed substantially to its revisions. BB takes responsibility of the paper as a whole. Received for publication October 16, 2002. Revision received February 18, 2003. Accepted for publication March 10, 2003.

ANNALS OF EMERGENCY MEDICINE

42:1

JULY 2003

OTTAWA KNEE RULE VALIDATION IN CHILDREN Bulloch et al

Presented as a platform presentation at the Pediatric Academic Society annual meeting, Seattle, WA, May 2003.

19. Ketelslegers E, Collard X, Vande Berg B, et al. Validation of the Ottawa Knee Rules in an emergency teaching center. Eur Radiol. 2002;12:1218-1220.

This study was funded by a grant from the Children’s Hospital Foundation of Manitoba, Inc.

20. Emparanza JI, Aginaga JR, for the Estudio Multicentro en Urgencies de Osakidefza: Reglas de Ottawa (EMUORO) Group. Validation of the Ottawa knee rules. Ann Emerg Med. 2001;38:364-368.

Reprints not available from the authors.

21. Charlson ME, Ales KL, Simon R, et al. Why predictive indexes perform less well in validation studies. Arch Intern Med. 1987;147:2155-2161.

Address for correspondence: Blake Bulloch, MD, Department of Emergency Medicine, Phoenix Children’s Hospital, 1919 East Thomas Road, Phoenix, AZ 85016; 602-546-1910, fax 602-546-1907; E-mail [email protected].

22. Stiell IG, Wells GA. Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med. 1999;33:437-447. 23. Wasson JH, Sox HC, Neff RK, et al. Clinical prediction rules: application and methodological standards. N Engl J Med. 1985;313:793-799. 24. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-174.

REFERENCES 1. Gallagher SS, Finison K, Guyer B, et al. The incidence of injury among 87,000 Massachusetts children and adolescents: results of the 1980-81 statewide childhood injury prevention program surveillance system. Am J Pub Health. 1984;74:1340-1347. 2. McConnochie KM, Roghmann KJ, Pasternack J, et al. Prediction rules for selective radiographic assessment of extremity injuries in children and adolescents. Pediatrics. 1990;86:45-57. 3. Gratton MC, Salomone JA, Watson WA, et al. Clinically significant radiograph misinterpretations at an emergency medicine residency program. Ann Emerg Med. 1990;19:497-502. 4. Klassen TP, Ropp LJ, Sutcliffe T, et al. A randomized controlled clinical trial of radiograph ordering for extremity trauma in a pediatric emergency department. Ann Emerg Med. 1993;18:1524-1529. 5. Stiell IG, Greenberg GH, McKnight RD, et al. Decision rules for the use of radiography in acute ankle injuries, refinement and prospective validation. JAMA. 1993;269:11271132. 6. Stiell IG, McKnight RD, Greenberg GH, et al. Implementation of the Ottawa ankle rules. JAMA. 1994;271:827-832. 7. Plint AC, Bulloch B, Osmond MH, et al. Validation of the Ottawa Ankle Rules in children with ankle injuries. Acad Emerg Med. 1999;6:1005-1009. 8. Stiell IG, Greenberg GH, Wells GA, et al. Derivation of a decision rule for the use of radiography in acute knee injuries. Ann Emerg Med. 1995;26:405-413. 9. Stiell IG, Greenberg GH, Wells GA, et al. Prospective validation of a decision rule for the use of radiography in acute knee injuries. JAMA. 1996;275:611-615. 10. Stiell IG, Wells GA, Hoag RH, et al. Implementation of the Ottawa Knee Rule for the use of radiography in acute knee injuries. JAMA. 1997;278:2075-2079. 11. Khine H, Dorfman DH, Avner JR. Applicability of Ottawa Knee Rule for knee injury in children. Pediatr Emerg Care. 2001;17:401-404. 12. Naranja Jr RJ, Gregg JR, Dormans JP, et al. Pediatric fracture without radiographic abnormality. Clin Orthop. 1997;342:141-146. 13. Seaberg DC, Yealy DM, Lukens T, et al. Multicenter comparison of two clinical decision rules for the use of radiography in acute, high-risk knee injuries. Ann Emerg Med. 1998;32:8-13. 14. Bauer SJ, Hollander JE, Fuchs SH, et al. A clinical decision rule in the evaluation of acute knee injuries. J Emerg Med. 1995;13:611-615. 15. Weber JE, Jackson RE, Peacock WF, et al. Clinical decision rules discriminate between fractures and nonfractures in acute isolated knee trauma. Ann Emerg Med. 1995;26:429-433. 16. Richman PB, McCuskey CF, Nashed A, et al. Performance of two clinical decision rules for knee radiography. J Emerg Med. 1997;15:459-463. 17. Tigges S, Pitts S, Mukundan S Jr, et al. External validation of the Ottawa Knee Rules in an urban trauma center in the United States. Am J Roentgenol. 1999;172:10691071. 18. Cohen DM, Jagger JW, Kean JR, et al. Clinical criteria for using radiography for children with acute knee injuries. Pediatr Emerg Care. 1998;14:185-187.

JULY 2003

42:1

ANNALS OF EMERGENCY MEDICINE

5 5