Roberto Carlos Flores EDIT 5584 Program and Product Evaluation Evaluation Plan Draft Evaluation of Students Learning Outcomes for Virginia Tech's Faculty-‐led Study Abroad Programs Introduction The evaluation plan presented in this document outlines the process to evaluate the learning outcomes produced by undergraduate students at Virginia Tech after participating in the Faculty-‐led study abroad programs. This plan comprises general information about the program, reasons of conducting this evaluation, stakeholders, and intended actions that the Global Education Office at Virginia Tech may take at the end of this work. At the same time, this evaluation plan describes the criteria of selecting the sampling, data collection methods and instruments to be implemented during the evaluation process. At the end, this document presents the evaluations’ requirements and limitations, as well as the timeline and budget. This evaluation plan was designed by Roberto Carlos Flores, VT’s faculty-‐led programs project evaluator. Also, it has been prepared for the Global Education Office as a primary client in collaboration with Dr. Selma Haghamed, Faculty Support Associate. Background The Global Education Office (GEO) strives to strengthen Virginia Tech's mission of global engagement, discovery, and service. Nevertheless, this office reports directly to the Vice President for the Outreach and International Affairs and its objective is intimately aligned to it. The purpose of the Outreach and International Affairs is to share the best of Virginia Tech by working step by step with the communities around the Commonwealth and throughout the world developing, transforming and engaging the world while inventing the future.
Among the different programs that the GEO offers to students, there is one called Faculty-‐led study abroad programs. They are usually designed by VT faculty members and sponsored by a specific VT department. What the students should know, do or appreciate as a result of participating in these programs is currently determined by each course/instructor. They design and lead this experience in order to bring global engagement, discovery, and service by connecting faculties and students to their peers abroad; inspiring and contributing to the faculty’s creation and promotion of cross-‐cultural learning experiences; and supporting the development of VT’s students into globally competent citizens. These programs are guided by the Global Education Office and their range is from one week to a full semester. Although the majority occur over the winter session (2 weeks) and summer (4-‐5 weeks). Students at VT from all the backgrounds and disciplines can apply online to these programs and need to meet specific requirements. For instance, they must have a minimum 2.5 GPA. Nevertheless, some of these programs demand a higher GPA. Any exception is at the discretion of the faculty program leader. When students are selected to participate in this program, they need to go through a pre-‐decision and post-‐decision stages. Once this is completed, they need to complete completely accepted they would have the opportunity to explore, learn and engage with other cultures from this experience. With 50+ programs spanning all colleges and about 78% of overall student mobility, the faculty-‐ led context makes up a key program category at Virginia Tech and according to the Global Education Office, they merit specific evaluation. Purpose The purpose of this project is to determine the common student learning outcomes, aligned with the Global Education Office and Outreach and International Affair’s objectives, as a result of participating in the faculty-‐led study abroad programs at Virginia Tech.
Stakeholders/Audiences Primary: -‐ Global Education Office. Main client and organization that will have access to the final results of this evaluation project. This includes the following persons: • Selma Haghamed, Faculty Support Associate • Rachel Fitzgerald, Assistant Director for Program Management • Theresa Johansson, Director of Global Education • Betty Watts, Coordinator of Global Initiatives Secondary: -‐ Outreach and International Affairs who also may be interested in knowing the projects’ findings. Tertiary -‐ Students who would like to be part of the faculty-‐led programs. -‐ Faculty professors since they are the ones in charge of designing the programs since students’ learning outcomes are determined by the programs’ objectives. Decisions: The results of this evaluation will be used by the Global Education Office in order to: -‐ Determine the students’ learning outcomes for Virginia Tech faculty-‐led programs in consultation with the Outreach and International Affairs’ mission. -‐
Break the resulting learning outcomes into key performance indicators for continuous quality performance of the programs.
-‐
Support the development of specialized evaluation instruments that will measure the identified student learning outcomes.
Sample: Throughout the year, there are more than 50 faculty led programs that are implemented in around 20 countries throughout the year. In order to narrow this down, two important factors were considered in the selection of the evaluation’s sample: the period of time in which the programs took place and the geographical location based on this time. The period of time was summer 2016 which was the closest period of the implementation for this evaluation and the region was Latin America because of the diversity of programs. At the end, the sample group was reduced to three and the sample population to 60 students (around 20 per group), 2 faculties, three professors in charge of these programs and 3 program’s designers. Methods and Instrumentation The information to be collected for this evaluation project comprises qualitative and qualitative data which means that the methodology involves a Mixed method design. The two instruments that will help the evaluator gather this data will come from: -‐ Electronic Survey. The evaluator will create a survey using Qualtrics and this one will be sent to the sample group through an email. The Global Education Office will help the evaluator distribute this survey using its institutional email account. -‐ Interviews. With the help of the GOE, the evaluator will identify the designers of the three faculty-‐led programs as well as the three professors in charge of developing the programs with the students in order to conduct an interview.
Data Decision Matrix KEY EVALUATION QUESTIONS
MOST EFFICIENT AND EFFECTIVE DATA SOURCES AND COLLECTION METHODS TO ANSWER THIS QUESTION In what ways do students’ learning outcomes Questionnaire to sample groups. Items to be are aligned to GEO and outreach and included: International Affairs’ goals? -‐ Multiple choice questions -‐ Rating scale To what extend is the program facilitating assessment to students?
What extend is the relationship between the pre orientation programs and learning outcomes?
-‐ Open-‐ended questions Interview faculty-‐led programs Designers. Questions related to the design, program goals, assessment rubrics, professors’ role, etc. Interview Faculty-‐led professors or mentors about previous programs and their experiences coaching students.
Logistics This evaluation project will be carried out by Roberto Carlos Flores. As a project evaluator, he will be responsible to perform the following actions: -‐ Design the evaluation plan. -‐ Create and administer the different data collection methods. -‐ Analyze the information gathered from the surveys and interviews. -‐ Provide the findings of this evaluation to the main stakeholder Timeline
Milestone
Target Completion Date
Complete the Evaluation Plan
October 20
Distribute surveys to the sample groups
November 1
Conduct interviews to designers and professors
November 10
Data Analysis
November 21
Write final report and present it to GEO
December 1
Budget The evaluation of this program will not require any economical expenses Limitations -‐ One of the biggest limitations consider in this project will be the rate of students’ responses to the survey from the sample group. -‐ Faculty-‐led programs designers and professors may have limited time to help the evaluator conduct the interviews.