Psychology in the Schools, Vol. 40(5), 2003 Published online in Wiley InterScience (www.interscience.wiley.com).

© 2003 Wiley Periodicals, Inc. DOI: 10.1002/pits.10108

RURAL UNDERPINNINGS FOR RESILIENCY AND LINKAGES (RURAL): A SAFE SCHOOLS/HEALTHY STUDENTS PROJECT LESLIE Z. PAIGE

Hays Unified School District 489, Hays, Kansas STEPHEN N. KITZIS AND JOYCE WOLFE

Fort Hays State University, Hays, Kansas Rural Underpinnings for Resiliency and Linkages (RURAL) is an example of a Safe Schools/ Healthy Students (SS/HS) project. RURAL provides services to children and families in a rural Kansas county. The county’s rates of juvenile alcohol and illicit drug use are higher than state averages. The RURAL project was designed to enhance existing partnerships among three school districts, the local community mental health center, law enforcement, the regional prevention center, and Fort Hays State University. A significant number of evidence-based strategies were implemented in the schools and community in a short period of time. Using a public health approach to increase school safety and promote healthy behaviors, RURAL introduced strategies designed to provide universal prevention for the school population, early intervention for at-risk children and families, and intensive services for those with the greatest needs. Services were provided to students (preschool–young adults) and their families. School psychologists held key roles in the development, implementation, and management of the project. Preliminary evaluation findings are presented, and the importance of local process and outcome evaluation is discussed. © 2003 Wiley Periodicals, Inc.

In 1869, famous lawmen and gunfighters of the American West walked the streets of Hays City, KS, home to some of the most unsavory saloons and bordellos of the day. Gunfights and murders were common. Later residents were sturdy pioneer immigrants who endured the dust bowl; today, their descendants struggle to cope with the declining farm economy and the current recession. Hays is a small rural community of nearly 20,000 located in Ellis County (total population 28,000), and is the major shopping, medical, and educational center of northwest Kansas, far from metropolitan areas. Like many isolated rural communities, the schools provide most of the services and available recreation for children and youth. The community’s heritage has left a legacy of ambivalent attitudes toward alcohol and drug use and strong opinions about the right to have firearms. Rates of underage alcohol use are higher than state averages, and marijuana and methamphetamines are growing concerns (Connect Kansas, 2001; Kids Count, 2001). Alcohol, drugs, and guns are readily available to students (Connect Kansas, 2001). Local Need for Substance Abuse and Violence Prevention Strategies RURAL serves 5,500 students in Ellis County. Most attend school in the Hays Unified School District, which has 3,630 students. Ninety-one percent of children living in Ellis County are non-Hispanic White, 13.7% live in poverty, and 27% are economically disadvantaged students (Census, 2000; Kansas State Department of Education, 2002). The graduation rate in the school district fell from 97% in 1997 to 90.7% in 2001 (Kansas State Department of Education, 2002).

This project developed under a grant from the Departments of Education, Justice, and Health and Human Services (CFDA 84.184L Coordinated Grant to Local Educational Agencies for Safe and Drug-Free Learning Environments; Safe Schools/Healthy Students Initiative). This article represents the views of the authors. The contents do not necessarily represent the federal departments’ policy, and endorsement by the federal government should not be assumed. It does not necessarily reflect those of the Hays Unified School District nor Fort Hays State University. Correspondence to: Leslie Z. Paige, USD 489, 323 West 12th Street, Hays, KS 67601. E-mail: lpaige@ hays489.k12.ks.us

531

532

Paige, Kitzis, and Wolfe

Other risk factors in the community include an increased number of births to single teens, out-ofhome placements of children, and increased rates of confirmed child abuse (Kids Count, 2001). Juvenile court filings have increased significantly with approximately 29% of 2001 arrests for driving under the influence in the county involving minors ages 14–21. The Regional Prevention Center and the Ellis County Community Partnership (involving schools, social service, mental health, law enforcement, and other organizations) have sponsored annual Communities that Care surveys (Connect Kansas, 2001) since 1995. This survey of grades 6, 8, 10, and 12 has shown alarming increases since 1995 in the number of youth who use drugs. The rates of drug and alcohol among county youth are higher than the state averages, with marijuana and methamphetamine use becoming increasingly common (Connect Kansas, 2001). Related to increased substance use, the Sheriff’s Department indicates that gang membership is growing. Other risk factors in Hays are the availability and permissive attitude toward the use of alcohol and illegal drugs, decreased parent involvement (Connect Kansas, 2001), and increasing rates of crime. Easy access to illicit drugs is of increasing concern. In addition to being located on a major interstate route for drug trafficking, Kansas currently ranks second in the nation (following California) in illicit methamphetamine labs, and fifth nationally in drug trafficking seizures (Legislative Division of the Post Audit, 2001). The Hays Drug Enforcement Unit seized 11 illicit labs in 2001. In examining these needs, the RURAL project is an attempt to address gaps in services for at-risk and high-risk children and families, as well as emphasize prevention of those factors that could increase risk. RURAL Framework The Development of the RURAL Partnerships and Coalitions Ellis County has enjoyed a long history of collaborations between agencies. The school district’s school psychologists and the High Plains Mental Health Center clinical staff have shared information and collaborated for many years regarding students (when releases of information are available). Prior to seeking the SS/HS federal grant, the school district, mental health center, courts, Social and Rehabilitation Services, and other agencies pooled funding and staff to support an alternative school/partial hospitalization program. The community also has several multidisciplinary teams and coalitions in place that address child abuse, substance abuse, child protection, early childhood, and other issues. The existing coalitions first came together because of shared goals addressing substance abuse and juvenile crime. They achieved substantial success in the creation of an alternative school, and brought about public policy changes and increased treatment and referral options. Building upon these previous successes, RURAL was a “next step.” The RURAL steering committee blended with the Ellis County Community Partnership (ECCP) because of the similarity of the missions and membership. RURAL brought additional members to what is now a much larger group as a result of the additional funding and new leadership that made it possible to address community problems with renewed energy. The RURAL/ECCP committee mapped all community services for children and youth, and has helped to identify and prioritize service gaps and needs in the community. Task groups have developed strategic plans to resolve community problems, and have had success in implementing the plans. RURAL has broadened coalition memberships, encouraged the focusing of limited resources, and generated sufficient support so that activities may be accomplished. One good example of this impact was the RURAL facilitation of a meeting with the heads of all the juvenile justice and law enforcement agencies related to the processing of juvenile offenders. The discussion of the need to share data and for a unifying means to collect data across the different agencies led to the

Rural Underpinnings for Resiliency and Linkages

533

assignment of a juvenile justice liaison between the RURAL program and these agencies to develop a unifying juvenile justice information form and database. Prior to this there was no way for agencies and schools to track services and outcomes for juveniles, and “turf issues” frequently resulted in schools and agencies not knowing the status of juvenile offenders. With the Hays Unified School District acting as lead agency, the partnering agencies support the project by providing input into the project’s development and implementation, as well as substantial in-kind and funding support for implementation. Because of the history of successful partnerships, problems of trust and communication have been minimal. Partnering agencies include three public school districts; High Plains Mental Health Center; Ellis County Law Enforcement (Ellis County Sheriff, Hays Police Department, and County Attorney); Juvenile Justice; the Ellis County Community Partnership (ECCP); the Regional Prevention Center; Fort Hays State University; the Docking Institute; and others. Coalition members include three parochial schools, many youth-serving organizations, the faith community, the local hospital, United Way of Ellis County, and nonprofit agencies. Partnering agencies communicate frequently, both informally and formally. The coalition members participate in quarterly steering committee meetings, which typically involve 35– 40 partner or coalition representatives. They provide input for planning, decision making, resource sharing, community goal setting, and evaluation. The RURAL Public Health Framework RURAL is based on the public health model, with project functions divided into prevention, intervention, treatment, community outreach, and evaluation. The public health approach encourages alliances between disciplines, professions, organizations, and community stakeholders to prioritize health concerns and encourages changes in personal practices. As described in the 2001 Surgeon General’s Report on Youth Violence (U.S. Department of Health and Human Services, 2001), the public health approach encompasses more than the medical model, which is limited to the diagnosis and treatment of specific illnesses in individuals. Instead, the public health model is a goal-oriented, practical, and community-based approach to promoting and maintaining health. It identifies problems and develops solutions for population groups, using surveillance processes to gather data establishing the nature, incidence, trends, and prevalence of the problem. After risk and protective factors are determined, interventions are designed to be effective and generalizable, and education efforts are coordinated in order to reach the general public (U.S. Department of Health and Human Services, 2001). Using this model, RURAL developed a framework of comprehensive strategies to better meet the needs of children and their families. Service gaps and coordination problems were addressed by bringing together stakeholders and prioritizing community needs and challenges based on the risk factors discussed above. Strategies were selected based on the evidence of their effectiveness for addressing relevant risk and protective factors. Prioritizing needs and strategies has helped to focus limited resources and enhance collaborative efforts. Resources were allocated as recommended by Dwyer and Osher (2000), with prevention strategies universally available, intervention strategies focused on at-risk populations, and treatment services are intended for the small proportion of the population who are at high risk. This structure of prevention, early intervention, and treatment is illustrated in Table 1, and described in greater detail below. Strategy Selection The selection of strategies was determined not only by the risk factors and needs of the community, but also available resources. For example, because Ellis County has very strong early

534

Paige, Kitzis, and Wolfe Table 1 Major RURAL Components and Component Categories Component Categories

Components

Prevention programs

Prevention Team Support Parent Involvement Plan Summer School Program YouthFriends School-Based Mentoring After-School Tutors and Prevention Program SRO expanded role Evidence-based drug/violence programs Second Step Violence Prevention Life Skills Training Bullying Prevention Program Peer Mediation Training Developmental Assets Crisis Response Planning Security Systems Equipment

Intervention services

Early Intervention Team Individual/Family Social Work Services Case Management School-based group interventions Parenting classes Student and parent support groups

Treatment services

FFT program Learning Center of Ellis County Drop-Out Prevention Drop-Out Recovery English Language Classes

Staff development

Preservice Educational Activities Inservice Educational Activities

School policies

Safe Schools Policy Work Group Building Cadres Student Advisory Councils Project Steering Committee

Community awareness

Community Service Coordination Community Outreach Resource Library Clearing House Project Dissemination Project Evaluation

childhood services such as Healthy Start, Head Start, Early Head Start, and Parents as Teachers, adding another program was not necessary. Instead, it was decided to enhance these programs by providing additional staff training and social work services. Aside from being required by the grant, selecting research-based programs was important because the outcomes would be more predictable, cost effective in the broad sense, and appropriate

Rural Underpinnings for Resiliency and Linkages

535

for the community. Prevention programs were selected based on the quality of their research base, appropriateness for the population, and recognition by agencies, such as the Center for the Study of Prevention of Violence (Blueprints for Violence Prevention, n.d.). Prevention strategies included Life Skills Training (Botvin, 1996), Second Step Violence Prevention (Committee for Children, 1997), Bullying Prevention (Olweus, 1993), and Peer Mediation Training (Schrumpf, Crawford, & Bodine, 1997). Other strategies were selected based on what the literature indicated was needed to build resilience and protective factors. YouthFriends (YouthFriends, 1999), a school-based mentoring program, was selected based upon the established benefits of mentoring, and supported by research conducted by the University of Missouri, Kansas City (Portwood, 2000). Other strategies included crisis response planning, after-school program support, tutoring services, and youth advisory group development. Social work support was selected because of the gap in school–home linkages and the need for early interventions for families struggling with parenting, accessing resources, and other issues, with the intent of preventing more significant problems. These services targeted very young children and their families, as well as school-age populations. Social work services included individual, family, and group interventions. It should be noted that, prior to SS/HS funding, there were no school-based social work services in the county. In collaboration with mental health center administration, Functional Family Therapy (Alexander et al., 1998) was selected as a treatment service, based on the growing population of juvenile offenders and the need for a more effective approach to dysfunctional families. Although not intended at the time of proposal development, this decision served to streamline a cumbersome consent process and resulted in other substantial changes at the mental health center. The other treatment program provides dropout prevention and recovery services. In a small community, it is relatively simple to observe the difficulties these individuals face, including underemployment, substance abuse, crime, and other maladaptive behaviors. Helping to keep students in school, as well as helping those who have dropped out to become more educated and more employable was a long-held district value that was previously not possible without the SS/HS funding. Project Components Prevention Component. As recommended by Dwyer and Osher (2000), most resources were allocated to prevention activities. These prevention activities were school-based, and targeted preschool through middle school students because of greater potential for preventing risk behaviors. Rather than attempting to mandate top-down change, RURAL allowed individual schools to choose their preferred level of participation. It was believed that the quality of the programs would be evident once teachers and support staff observed their effectiveness. As a result, the schools would “own” the programs, resulting in improved implementation. This decision was made during the development of the grant proposal because of the nature of the site-based management in the Hays schools, and sensitivity to the smaller school districts’ culture of independence. During proposal development and after the grant was awarded, the project director met with superintendents and principals to reconfirm their commitment and to encourage participation. Although there was some initial hesitancy on the part of a few schools, all schools are implementing some of the prevention strategies, and several are implementing all prevention strategies in all classrooms. The role of the Prevention Team (comprised of school psychologists and a school social worker) was to train and provide technical assistance to school staff to maintain and expand the prevention programs. The Prevention Team utilized a strategic change process emphasizing awareness, training, support, and sustainability. Teachers could request varying levels of support, including modeling, hand-outs, corrective feedback, or consultation. Schools or teachers were able to

536

Paige, Kitzis, and Wolfe

request additional materials such as posters, videos, customized handouts, and the like to support the new strategies. The Prevention Team was responsible for providing ongoing support and securing needed resources. Each school was asked to designate a safety cadre (typically the school psychologist, principal, school counselor, and one or more key teachers). The cadre’s responsibility was to (a) assess the safety needs of their school through safety audits provided by the School Resource Officer; (b) hold debriefings of crisis drills; and (c) assess concerns such as communication gaps between the playground and the office, locations prone to discipline issues such as student bathrooms, and the like. Student climate data provided by the local evaluator was also used to develop plans to address concerns such as bullying, social isolation, and fairness in enforcing rules. Some schools desired to implement character education programs, and the Prevention Team provided technical assistance and support in selecting strategies with evidence of effectiveness. Each cadre was provided with potential strategies selected by RURAL and could determine which strategies to adopt. Additionally, each cadre was told at the outset that they would be able to access training and resources for their building over the life of the grant, but that when the federal funding ceased it would be their responsibility to sustain the activities. Cadre members were provided with nominal honoraria and social recognition. Requests for resources and support from RURAL needed to be linked to each school’s cadre plan. Cadre plans were individualized for each building, with some focusing on the needs of the school population (e.g., non-English-speaking families, high incidence of students with emotional or behavioral issues), while others focused on a particular strategy (e.g., Bullying Prevention, Second Step). Some schools changed their cadre plans each year, making revisions in reaction to changing needs or a change in perception of what was needed. Intervention Component. The Intervention Team was comprised of social work staff assigned to specific schools or programs. The school districts had never before received school-based social work services. Typically, school social work services in Kansas are restricted to children receiving special education services because of state funding requirements. With SS/HS funding, RURAL was able to develop a social work program that could be accessed by any child or family. An entirely new program was developed and refined. In 2001–2002, 164 students and families accepted services, perhaps because the services were voluntary, associated with the schools, convenient for the family, and free. Services included individual and/or family social work services, school-based psychoeducational groups, family/parent support groups, targeted school-based groups, parenting education, and consultative and staff development for early childhood programs. Needs assessments were conducted at each school to determine needed school-based services. Individual or family services were provided in the home or school setting, during the school day or in the evening, utilizing family-driven solution-focused plans. Typical issues included basic parenting strategies, supervision, boundaries, school attendance, discipline, and resource acquisition. Family issues such as depression, loss, financial stress, divorce, or mental illness were common. Case management and referrals to other services were also available. Parents or teachers referred children to the school psychologist or counselor prior to referral to RURAL. This was done to ensure that there was a demonstrated need for services and that the referral was not an attempt to bypass more appropriate referrals to community mental health or special education teams. Unless a parent initiated the referral, the school was required to contact the parent prior to the social worker contact. This was due to the community’s suspicious perception of social workers, who were associated with child protective services. Having a school employee contact the family first served to decrease barriers to accepting services.

Rural Underpinnings for Resiliency and Linkages

537

Psychoeducational and targeted school-based groups were frequently co-led with the building’s school psychologist or counselor. Topics included dating, healthy relationships, and depression, and targeted students participated in anger management or social skills groups. Brief intervention groups for high school students who had violated school substance abuse policies were also provided. Because school policies require an intervention following violation, students could choose to pay for services from a local provider or attend the free intervention group at the school. Thirty-one students participated during the 2001–2002 school year. The parenting support and education groups were very popular, as evidenced by full, well-attended classes, waiting lists, and needing to add extra sessions—52 adults participated. Treatment Component. In partnership with High Plains Mental Health Center, RURAL provided support for Functional Family Therapy (FFT) training for Center clinicians. School psychologists and counselors were invited to participate in the introductory training. SS/HS funding was used also to support FFT site certification and supervision training for the clinical lead. During the life of the grant, FFT services were free for over 40 qualifying families. More than 450 individuals enrolled at the Learning Center between June 2000 and June 2002. It provides dropout prevention and recovery services, English as a Second Language instruction, and Adult Basic Education. The curriculum is individualized, self-paced, and uses computerassisted instruction, videos, CD-ROMs, and print materials such as books, magazines, and newspapers. Childcare services and evening hours are used by parenting or working students. Individuals who have dropped out of school enroll in the dropout recovery program. High school diplomas are awarded by one of five high schools once all credits have been earned. In 2 years, over 25 individuals have earned their high school diplomas. This is significant because many students enroll with very few high school credits and may only be able to attend for a few hours each week. Dropout prevention services have benefited more than 200 students. Students may earn high school credits for failed classes after school or during the summer. Several students attend the Learning Center full-time as an alternative educational setting. In addition to serving high-risk students, adult English language classes are offered and have grown from seven adults in 2000–2001 to 80 in 2001–2002. Parents attend evening classes located in a neighborhood school, with free babysitting provided to encourage attendance. Community Outreach. RURAL has been very active in the community. The Prevention and Intervention Teams provided programs to the University, home–school association meetings, and community groups. RURAL staff make many media contacts, including TV, radio, and newspaper. The “15⫹ Take Time to Listen, Take Time to Talk” public service announcements have local contact information inserted and air several times each day. The RURAL Resource Library is located at the school district offices, and more than 500 resources are available to anyone in the community. The list of resources is available on the project Web site and in hard copy. The district’s school psychologists, teachers, and counselors use many of the materials. Parents and community agencies frequently check out resources for a semester, or as needed. These resources include reference books, government publications, games, videos, program guides, parenting programs, and counseling materials. Topics include crisis response, parenting, child development, divorce, death, alcohol and other drugs, specific disabilities such as attention deficit disorder, home visits, cultural competence, tolerance, school security, violence prevention, conflict management, peer mediation, bullying, and others. A yearlong communications campaign, developed by a task group of the RURAL/ECCP committee, with assistance from the national SS/HS Communications Team, is popular. The If Children Are the Future, Parents Hold the Key Initiative is attempting to address substance abuse

538

Paige, Kitzis, and Wolfe

and domestic violence issues by increasing awareness of good parenting skills and the availability of community resources. Another collaborative endeavor is the Safe Instead of Sorry Conference, now held on an annual basis. The conference has featured nationally known speakers, including Kevin Dwyer (school violence prevention), Michael Haines (substance abuse prevention), and James Vollbracht (developmental assets). Other activities have included presentations by local experts on related topics and “village meetings.” The conferences have been well attended, and have received local and foundation funding in addition to grant support. Other community outreach activities have included local and national dissemination of RURAL strategies. Articles in national newsletters such as the Communiqué (Paige, 2000a, 2001a) and the Challenge (“Safe Schools/Healthy Students Initiative,” 2001) have described RURAL. There have been presentations at conferences, including the National Association of School Psychologists (Paige, Francis, Schiada, 2002; Paige, Hodgdon, Douglas, & O’Day, 2001), state school psychology associations conferences (Paige & Kitson, 1999, 2000, 2001), the National Conference on Advancing School-Based Mental Health Programs (Paige, 2002b), and the Safe Schools/ Healthy Students National Conferences (Cohen & Paige, 2002; Paige, 2000b, 2002a). RURAL was also featured at the Surgeon General’s Community Forum on Youth Violence (Paige, 2001b) and at the Centers for Disease Control SafeUSA Leadership Conference (Paige, 2001c). The Role of School Psychologists School psychologists played a key role in the development, implementation, and evaluation of the project. RURAL was written by two school psychologists, who later became the Project Director and Prevention Team Coordinator. Other members of the Prevention Team included school psychologists because of the need for systems change and consultation skills. School psychologists are key leaders at the building level, frequently driving the school cadre activities, and helping to shape the strategic plans for crisis response, bullying prevention, and other project goals. The influence of school psychology on RURAL is critical. Hays school psychologists have a history of being proactive leaders within the district, and had previously obtained and managed other grants. These previous experiences with systems change grants and program development were beneficial in developing the SS/HS proposal and in managing a very ambitious project. The local school psychologists championed program fidelity and data collection. Understanding research and data-based decision making facilitated program development and evaluation activities. Skills needed for consultation, group process and systems change were valuable, as was knowledge of the schools and their organizational opportunities and threats. The SS/HS initiative provided a unique opportunity to use school psychologists’ skills and experiences to address school and community needs in a proactive manner. RURAL Local Evaluation All SS/HS sites are required to include an evaluation component to monitor progress towards project goals and objectives. Although RURAL participated in national evaluation activities such as cost benefit analyses, school staff surveys, focus groups, partnership surveys, and more detailed evaluation activities related to its status as one of 15 “Sentinel Sites,” this article will focus upon local evaluation activities. Evaluation Planning and Process Evaluation Development The evaluating agency was the Docking Institute of Public Affairs, a research subsidiary of Fort Hays State University. The Docking Institute became involved with the RURAL project after

Rural Underpinnings for Resiliency and Linkages

539

the proposal was almost completely written. This situation is not uncommon and does not necessarily decrease evaluation quality, but it does require that evaluators take adequate steps to become familiar with project components and their evaluation needs. Two factors made the development of the RURAL evaluation plan complicated. One was that RURAL is composed of a complex set of components that mesh new and existing services. The other was that the goals and objectives were not initially well delineated. The following steps were taken to develop the evaluation plan: 1.

2.

3.

4.

Goals and objectives were delineated. The original RURAL goals and objectives freely intermixed implementation details and broadly overarching target goals. Before the evaluation could begin, the evaluators needed to transform the goals and objectives into a set of straightforward, tangible items. Local evaluators summarized RURAL goals and objectives as: (a) providing designated school services, (b) providing designated social services, (c) providing designated mental health services, (d) decreasing substance abuse, violence, and crime, (e) improving school safety, (f ) adopting school safety policies, and (g) evaluating RURAL’s activities. Major RURAL components and component categories were identified. Some components were specific programs that functioned independently, while others were ongoing activities or committees. The evaluators created an evaluation plan that tracked specific information for major RURAL components. The major RURAL components were grouped into five broader categories of prevention programs, intervention/treatment services, staff development, school policies, and community awareness, as shown in Table 1. The selection of tracking procedures occurred. After the components were identified, the tracking procedures were developed. The measures shown in Table 2 were designed for semiannual collection. However, as the federal initiative proceeded, it became apparent that there was going to be duplication in some of these measures due to activities of RURAL project staff and the national evaluator. Consequently, the local evaluation focused on the unduplicated set of tracking procedures (items 5 through 9 in Table 2) mainly consisting of satisfaction surveys and service utilization tracking measures. Evaluation questions were developed by component categories. Although each component category had somewhat different assessment needs, as much commonality as possible was maintained across the different categories. In general, each new program or service was assessed from the points of view of those receiving or providing services. In addition, the views of parents and the wider community were obtained. The questions addressed for each component category, and the groups surveyed, are indicated in Table 3. Where appropriate, these tracked items were supplemented with in-depth case studies.

Table 2 RURAL Tracking Procedures and Their Sources Source RURAL (district)

Partner provider

Docking institute

Measures Tracked 1. Scheduled implementation date(s) in proposal plan 2. Actual implementation date (or dates if phased) 3. Description of services provided and changes 4. Service costs 5. Client utilization rates 6. Specific staff-assessed client outcome measures 7. Specific client/parent/guardian-assessed outcome measures 8. Client/parent/guardian surveys on services provided 9. Staff surveys on services provided

540

Paige, Kitzis, and Wolfe Table 3 Evaluation Component Categories and Questions 1. Prevention programs How well attended was the program? (staff) Does the program appear to be effective? (staff, students) 2. Intervention services How well utilized was the service? (staff) Does the service appear to be effective? (staff, clients, parents) 3. Staff development How well attended was the program? (instructor) Does the program appear to be effective? (instructor, attendees) 4. School policies How well known are the policies? (staff, students, parents, community) Do the policies appear to be effective? (staff, students) 5. Community awareness How well known are the services and service providers? (staff, students, parents, community) Have attitudes concerning the targeted issues changed? (staff, students, parents, community) Have behaviors changed? (staff, students, parents, community) Do the services appear to be effective? (staff, students, parents, community)

5.

6.

Surveys were developed to address evaluation questions. The list of surveys used for RURAL is shown in Table 4. These included process evaluation-related surveys measuring satisfaction, and surveys for outcome or attitudinal measurement. Surveys were developed in-house, specifically with the RURAL project’s needs in mind. All surveys included open-ended as well as multiple-choice questions. For comparison between different service components and respondent types, question wording was kept as similar as possible across surveys. Confidential process interviews were planned. A final element of the process evaluation plan was confidential interviews with RURAL service providers and the RURAL Prevention Team. These interviews served two purposes: to identify specific case studies to complement the more global outcome measures and to augment information from the process evaluation surveys.

Table 4 Local Evaluation Surveys Instruments Survey Name Presentation evaluation Presenter evaluation Client satisfaction survey Staff satisfaction survey Staff interaction survey School climate survey Community climate survey Rural Web survey (Planned September 2002)

Survey Description Inservice and presentation attendees assess presentations Presenters assess audience characteristics Clients/adult students assess satisfaction with programs RURAL staff assess working climate School personnel/community members assess interactions with RURAL staff Parents, teachers, staff, and students assess school climate Parents, teachers, staff, students, and community assess community Parents, teachers, staff (maybe students) will provide comments regarding RURAL-sponsored programs and activities

Rural Underpinnings for Resiliency and Linkages

541

School and Community Climate Surveys Climate Survey Development. Many of the RURAL goals were related to creating changes in school and community climate regarding violence, drugs, safety, and the utilization of mental health and social services. Therefore, evaluation efforts were directed towards monitoring attitude changes. Two surveys were developed for this purpose. A school climate survey included questions about school activity or affiliation; rule adhesion or school strictness; bullying; violence; awareness of alcohol, drugs, or weapons being brought to school; isolation and emotional support; educational support; parent and community involvement; teacher/administration/principal relations; and building conditions. A community climate survey focused on awareness and willingness to utilize new RURAL services; school safety issues; and drug and alcohol use and violence issues in the community. Prior to the RURAL grant, each school site used a unique school climate survey. The questions were all different, and they were administered differently for each building. RURAL needed a time-efficient survey appropriate for use across different respondents (parents, teachers, staff, and students) and grade levels (grades 4–12). There were several barriers to finding an existing instrument to serve the evaluation’s purposes. Research into existing school climate instruments such as Baldwin and Coney’s School Effectiveness Questionnaire (1993); Dodson’s School Improvement Program Needs Assessment Questionnaires (1987); and the National Association of Secondary School Principals (NASSP) School Climate Survey (n.d.) did not result in the location of a survey with all the desired topical elements, with the ability for use across different respondents, or with the desired length. (RURAL staff and evaluation team members wanted to keep the surveying time at about 10–15 minutes). Cost was also a concern. Commercially available instruments are cost-prohibitive. It was decided that creating a school climate survey to fit all of the desired survey characteristics was the right approach. The research into other survey instruments was helpful in developing questions with age-appropriate question wording and in selecting question topics for the RURAL school climate survey. Much effort was made to develop question wording that would be appropriate across the entire age range of respondents and that would increase the likelihood of respondents providing truthful answers. Questions on socially sensitive issues also had to be carefully phrased. The RURAL evaluation uses climate data for both process and outcome evaluation purposes. As a process evaluation indicator, the data were provided to RURAL project staff and used to assist individual school needs assessments. Reports were provided to every school building so that problem areas could be addressed. As an outcome measure, climate data will be used to make determinations about whether bullying, alcohol and drug use, school violence, student alienation, and other indicators have changed over the course of the grant period. Climate Administration. Gaining the privilege of intruding into any school system, especially a small one, is a serious responsibility. Gaining the trust of school personnel was an important step in planning the school-based survey activities. Local evaluators discussed the survey process with school administrators in all of the school buildings. In nearly all of the schools, the local evaluators were invited to staff meetings to discuss the purpose of the surveys and to discuss administration procedures. Relationships developed with school principals and other school staff provided a foundation for future evaluation discussions. Evaluators wanted to keep as low a profile as possible in the schools. It was thought that the standard pretest–posttest methodology would be too intrusive and would not be sensitive enough to measure small changes in climate. Instead, a continuous random sampling schedule was used. At the beginning of each school year, the school district provided evaluators with a list of classrooms

542

Paige, Kitzis, and Wolfe

from each school. The frequency of surveying in any given building depended upon the number of classrooms. Larger schools (the high school and the middle schools) were not necessarily sampled every week, but were sampled based on a weekly schedule that was created. The elementary schools, which were much smaller, were sampled once per semester or per year. (During survey year 1, there was a baseline sampling of all buildings during the fall, with continuous sampling during the spring semester. Survey year 2 had continuous sampling the entire year.) To even further lessen the impact on students and to minimize surveying time, students were not asked to complete both the school and community climate surveys in one sitting. Rather, they completed either one or the other (only adults were asked to complete both surveys). This was accomplished by surveying in classroom pairs. Whenever a school was surveyed, each of the climate surveys was administered in at least one classroom. Teachers and staff were sampled on a continuous random sampling schedule parallel with the student sampling schedule. Teacher and staff lists for each building were provided to evaluators, and a weekly schedule was created to disseminate surveys on a regular basis over the course of a school year. Parents from all schools were sampled during events such as parent/teacher conferences or school performances. Impact Outcome Evaluation The goals and objectives of the RURAL program and the mandates of the federal initiative provided the framework for the selection of outcome measures. Education indicators included attendance and dropout rates, suspension and expulsion rates, informal disciplinary reports, and academic test scores. Social indicators included child abuse/neglect reports and births to unwed teens. Criminal justice indicators included juvenile court filings, criminal court filings, and alcohol/ drug usage rates. For RURAL, these data sources were primarily archival. One drawback of using archival data is the lag time that often is associated with them. Some of RURAL’s archival data sources were as much as 1 to 3 years behind. Developing the capacity to gather local data sources for this project would have been ideal, but unfortunately, was not feasible to put into place in time to gather data for this grant. For example, the evaluators felt it was important to collect informal disciplinary reports to track minor infractions such as fights, vandalism, and detentions. In the communities covered by the grant, there were relatively few suspensions or expulsions as reported to the Kansas Department of Education. However, the data for informal sanctions were not collected in a standardized manner across the different schools. Each school had unique definitions for their sanctions. Complicating matters even further, these data were not tracked in a systematic way over several years, reducing the amount of available data that RURAL needed to establish trends. Now that this issue has been identified, future efforts will address this problem by assisting the schools to create a unified data system to track informal infractions. Another area in which local data would have been useful is juvenile justice. There are five agencies within the county that provide services to juvenile offenders. The agencies collect data in paper format, not electronically. The forms are all very different and there is very little collaboration among agencies. Part of the RURAL grant efforts aimed to create a single data entry form that could be tabulated electronically and compared across agencies. This process has now been put into place, but the time it has taken to collect and enter the data was too long to provide time-relevant data to the RURAL project evaluation. Case Study Interviews It was apparent from the start that outcome data collection efforts alone would not be able to capture the types of changes expected to occur with clients of RURAL’s intervention and treatment

Rural Underpinnings for Resiliency and Linkages

543

programs (Functional Family Therapy, Early Intervention Services, and the Learning Center). The changes in individuals and families resulting from these programs would not be expected to markedly influence global outcome measures. This points to the importance of parallel structure between services and evaluation. Just as many SS/HS sites used a multilevel prevention/intervention model (e.g., Dwyer, Osher, & Wagner, 1998), the evaluation needs to be flexible enough to capture changes at all levels. Treatment and intervention programs may affect a relatively small number of individuals and families in comparison to prevention programs, but can be expected to produce a more immediate impact. Case studies were an important evaluation element, but it was important to conduct them so that the anonymity of clients in a small town would be maintained. To achieve this, service providers were interviewed rather than their clients. A protocol was developed and provided in advance to all interview respondents. About half of each interview was concerned with process evaluation issues described earlier. The other half was devoted to encouraging the service providers to describe what was considered to be the best examples of success and failure stories among their clients. Among these stories, there were two that included children who shared characteristics of previous school shooters including feelings of and/or being alienated, suicidal thoughts, and playing with bombs or other weapons. For the moment, these cases have apparently happy or at least uneventful endings. RURAL appears to have provided timely and appropriate support to the right families. Only case studies can provide insight into the impact of RURAL on rare but extreme school violence. The Success of Research-Based Programs The final element of the RURAL evaluation was to gather information about the implementation and impacts of its evidence-based prevention programs (Second Step, Life Skills Training, and Bullying Prevention). Given that RURAL was a community-wide effort, it was inappropriate to implement a design using comparison or control groups. Assessing changes in outcomes measures in a pretest–postest program implementation was the best possible option in assessing changes, using the program participants as their own controls or as “reflexive controls” (Rossi, Freeman, & Lipsey, 1996). If the programs have the impact on students that is expected, one would expect reductions in indicators of risk factors and increases in indicators of protective factors. Two challenges to the measurement of program outcomes should be mentioned. Often, the first year or two of program implementation is spent developing ways to apply program principles to curriculum and daily activities (Frey, Hirschstein, & Guzzo, 2000). In addition, many programs implement booster sessions 1 and 2 years following the initial year of implementation to solidify program content (e.g., Life Skills Training). Given that the SS/HS funding was for 3 years and RURAL prevention programs have been administered only over years 2 and 3 (with the first year devoted to hiring, training, and preparation), demonstrating visible effects related to the prevention programs will require additional follow-up assessments. Another challenge to measuring program outcomes is program fidelity. The RURAL project staff made a decision early on in the project to not administer programs with a top-down approach. It was felt that such an approach for the various districts would be counterproductive. Instead, a bottom-up approach was used whereby school sites could participate in programs as much or as little as they wished, believing that once teachers and staff were able to determine that the programs were successful, low implementing buildings would increase implementation over time. This has proven to be the case. Participation has steadily increased and will soon reach 100% at most schools. This approach is highly desirable from an administration perspective, but is not ideal for a highly controlled evaluation.

544

Paige, Kitzis, and Wolfe Sustainability

One of the challenges facing RURAL has been the change in the state and national financial picture. In 1999, it was decided that either the school district or other agencies would help support key programs once the federal funding ceased. The current fiscal crisis in Kansas and the rest of the nation, plus the impact of 9/11 were not predictable. The last year of SS/HS funding has therefore been focused on reevaluating earlier plans and solving the problem of continuing successful programs in the current fiscal realities. Because the primary focus of RURAL is on systems change and prevention, sustainability issues have focused on strategies devised by the Intervention Team. Mental health services and interventions are costly because they require salaries and funding to support staff. At the time of this article, other sources of funding this critical component of RURAL are still being explored. However, the other components have been easier to sustain for several reasons. The Prevention Team’s efforts were focused primarily on increasing the capacity of teachers and support staff to implement research-based strategies. Although 1 more year would have probably ensured nearly complete success, the Prevention Team’s efforts have certainly made a substantial difference in the Ellis County schools. One reason for using evidence-based strategies is their predictability. Therefore, it is important for the schools to continue using the strategies with fidelity in order to see the expected behavioral change in time. Another key sustainability factor is the school cadres, which have had 3 years of support, encouragement, and technical assistance. They will be key in maintaining the prevention programs, monitoring fidelity, and training new hires. Building liaisons will support key strategies such as Second Step and Life Skills Training. The Learning Center of Ellis County was provided with critical start-up support. Students who are enrolled in the center as dropouts generate part of the district’s state funding. Affordable class fees are charged for students who have not dropped out but need to recover credits. Fees are waived for disadvantaged students. Other operating costs have been covered by the district’s general fund. Grants will be written to expand services. Now that key staff are trained, and a certified supervisor is on-site, it is expected that the mental health center will continue to utilize Functional Family Therapy with their clients. As mentioned earlier, this strategy sparked significant changes in service delivery to the 20 counties in northwest Kansas served by High Plains Mental Health Center. Resources are being explored to help defray costs for families who do not have Medicaid or other mental health coverage. Other agencies have adopted programs, such as the Single Parent Support Group, the Parenting Education Programs, the community conference and parenting campaign, and other activities. The YouthFriends and after-school support programs will need to be funded by grants, which are being aggressively pursued. The resource library will continue to be available to the community, and many staff members from other agencies participated in RURAL training activities. The Steering Committee will remain active, and the Community Mobilization Committee and other groups are carrying on issues relevant to RURAL. Generalization to Other Communities RURAL could be replicated in other communities for several reasons. It demonstrates a way to use the public health model to address school safety and healthy behaviors in a comprehensive manner. The emphasis on prevention and staff development is also important. The strategies used to enhance and sustain school and community change have seemed to work. This model has been very effective in a rural community.

Rural Underpinnings for Resiliency and Linkages

545

Lessons Learned Evaluation Lessons. The following suggestions are based on the experiences and interactions between RURAL evaluation team members and RURAL project staff. 1.

Evaluators should be involved in many aspects of the program and not just those that provide data or where there is personal interest. Being genuine participants allows evaluators to be more than passive data collectors. It was helpful to have more than one person on the evaluation team so that responsibilities could be divided. 2. Evaluators should maintain independence when developing and conducting a rigorous goal analysis and evaluation plan. For example, there was initial disagreement about the administration of the school and community climate surveys. After discussing the pros and cons of a traditional pretest–posttest administration and the proposed Continuous Process Analysis, both sides were able to come to agreement about which approach to use. Being assertive, yet somewhat flexible is a good practice. 3. Evaluation methodologies should be designed to accommodate growth and change. Everyone involved in the evaluation process, including program staff should be aware of this requirement. Evaluation planners need to anticipate program changes. 4. Do not underestimate the importance of process evaluation. Learning about the problems and difficulties with program implementation from an early stage is important so that information can be fed back into the project to amend or replace ineffective practices. It is more valuable to find out what does not work early on so that it can be changed. Process information was provided back to the RURAL project staff. In retrospect, more efforts in process evaluation and in developing tracking mechanisms would have been beneficial. General Lessons Learned. The most positive lesson is that substantial funding in a short amount of time can be a very powerful change agent. Although it is possible that some of these programs could have been introduced to the community over time, the impact would not have been comprehensive. RURAL was able to make a dramatic impact because so much could be accomplished in a short amount of time. Another lesson was the benefit of funding being tied to research-based programs and clearly defined purposes. This clarified expectations and helped team members to emphasize issues such as strategy selection and treatment fidelity. The hardest lesson learned was that sustainability plans need to be developed and implemented from the initiation of the project. SS/HS sites need to focus on sustaining strategies and essential functions, not the “ Project.” From day 1, SS/HS sites need to plan for the time when federal funding ceases and should emphasize sustainable strategies. Goals and activities need to be realistic and attainable. Projects should target outcomes that can be achieved in a short period of time. When developing goals and objectives, remember that 3 years of SS/HS funding is a relatively short time to accomplish measurable systems change as a result of new programs. Start-up may be slow due to the mismatch between the federal funding cycle and the school year. Implementation can begin rapidly if the project plan is well organized and considers how to recruit, hire, and train staff after the school year has begun. Support from partners and the district stakeholders is essential. One reason for RURAL’s rapid start-up is that key gate keepers and stakeholders did not need to be “sold” on the value of the Initiative, because they had been asked for input when the grant was written. Would we do this again? Yes! References Alexander, J., Barton, C., Gordon, D., Grotpeter, J., Hansson, K., Harrison, R., & et al. (1998). Blueprints for Violence Prevention, Book Three: Functional Family Therapy. Boulder, CO: Center for the Study and Prevention of Violence. Baldwin, L., & Coney, F. (1993). School Effectiveness Questionnaire. Eric Test Collection (TC 019644).

546

Paige, Kitzis, and Wolfe

Blueprints for Violence Prevention. (n.d.). University of Colorado, Center for the Study and Prevention of Violence (CSPV). Retrieved August 30, 2002, from http://www.Colorado.EDU/cspv/blueprints/about/main.htm Botvin, G.J. (1996). Life skills training. Princeton, NJ: Princeton Health Press. Census 2000. Suitland, MD: U.S. Census Bureau. Cohen, L., & Paige, L.Z. (2002, May). Developing and maintaining collaborations. Paper presented at the Safe Schools/ Healthy Students National Conference, Tyson’s Corner, VA. Committee for Children. (1997). Second Step: A violence prevention curriculum. Web site: http://www.cfchildren.org Connect Kansas. (2001). Ellis County Communities that Care survey results. Retrieved August 39, 2002, from http://216.49.238.48/ck /portal.php Dodson, A.G. (1987). School improvement program needs assessment questionnaires (elementary, middle, and secondary schools versions). ETS Tests in microfiche collection: 015982, 015983, 015984. Dwyer, K., & Osher, D. (2000). Safeguarding our children: An action guide. Washington, DC: U.S. Departments of Education and Justice, American Institutes for Research. Dwyer, K., Osher, D., & Wagner, C. (1998). Early warning, timely response: A guide to safe schools. Washington, DC: U.S. Department of Education. Frey, K.S., Hirschstein, M.S., & Guzzo, B.A. (2000). Second Step: Preventing aggression by promoting social competence. Journal of Emotional and Behavioral Disorders, 8, 102–112. Kansas State Department of Education. (2002). Ellis County statistics. Retrieved August 30, 2002, from http://www.ksde.org Kids Count. (2001). Ellis County report. Retrieved August 30, 2002, from http://www.socwel.ukans.edu/⬃kidcount/ Legislative Division of the Post Audit. (2001, July). Performance audit report. Methamphetamine labs: Reviewing Kansas’ enforcement efforts. A Report to the Legislative Post Audit Committee, State of Kansas: Author. Olweus, D. (1993). Bullying at school: What we know and what we can do. Oxford, UK: Blackwell. Paige, L.Z. (2000a, June). R.U.R.A.L.: A Safe Schools/Healthy Students initiative. The Communique, 2(8), 26. Paige, L.Z. (2000b, December). Consensus building—Involving all stakeholders: Site presentation. Presentation at the Safe Schools/Healthy Students National Conference, Washington, DC. Paige, L.Z. (2001a, March). Writing a Safe Schools/Healthy Students grant. The Communiqué, 29(6), 28. Paige, L.Z. (2001b, November). Rural underpinnings for resiliency and linkages: Safe Schools/Healthy students. Youth Violence and Public Health: A Community Forum Sponsored by the U.S. Department of Health and Human Services, Office of the Surgeon General, U.S. Public Health Service Regions V and VII, University of Illinois at Chicago, School of Public Health. Paige, L.Z. (2001c, December). School safety is everybody’s business. Presentation at SafeUSA Leadership Conference to Reduce Violence and Injury in America, U.S. Department of Health and Human Services, Centers for Disease Control, Atlanta, GA. Paige, L.Z. (2002a, May). The value of communications. Presentation at the Safe Schools/Healthy Students National Conference, Tyson’s Corner, VA. Paige, L.Z. (2002b, September). School safety is everybody’s business: A collaborative public health approach to safe schools and healthy students. Presentation at the National Conference on Advancing School-Based Mental Health Programs, preconference workshop, Philadelphia, PA. Paige, L.Z., & Kitson, J. (1999, November). Rural Underpinnings for Resiliency and Linkages: The Kansas Safe Schools/ Healthy Students initiative project. Presentation at the Kansas Association of School Psychologists Annual Conference, Hutchinson, KS. Paige, L.Z., & Kitson, J. (2000, October). Collaboration of schools, families, and community agencies: A model for meeting the educational and health needs of children and youth, Presentation at the Kansas Association of School Psychologists National Conference, Topeka, KS. Paige, L.Z., & Kitson, J. (2001, March). School safety is everybody’s business; Safe Schools/Healthy Students and school psychologists. Presentation at the Illinois School Psychologists’ Association Annual Conference, Chicago, IL. Paige, L.Z., Francis, K., & Schiada, G. (2002, February). Safe Schools/Healthy Students grant sites discuss sustainable programs. Presentation at the National Association of School Psychologists National Conference Special Session, Chicago, IL. Paige, L.Z., Hodgdon, J., Douglas, L., & O’Day, J. (2001, April). Safe Schools/Healthy Students Initiative: Schools and communities working together. Presentation at the National Association of School Psychologists National Convention, Washington, DC. Portwood, S.S. (2000). Youth friends evaluation. Retrieved August 30, 2002, from http://www.youthfriends.org/about/ evaluation/index.html Rossi, P.H., Freeman, H.E., & Lipsey, M.W. (1996). Evaluation: A systematic approach (6th ed.). Thousand Oaks, CA: Sage. Safe Schools/Healthy Students Initiative supports broad-based prevention efforts. (2001). The Challenge, 10(1), 1–2.

Rural Underpinnings for Resiliency and Linkages

547

Schrumpf, F., Crawford, D.K., & Bodine, R.J. (1997). Peer mediation: Conflict resolution in schools (Rev. ed.). Champaign, IL: Research Press. U.S. Department of Health and Human Services. (2001). Youth violence: A report of the Surgeon General. Rockville, MD: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Injury Prevention and Control; Substance Abuse and Mental Health Services Administration, Center for Mental Health Services; and National Institutes of Health, National Institute of Mental Health. YouthFriends. (1999). YouthFriends operations manual. Kansas City, MO: YouthFriends.