Roberto  Carlos  Flores     EDIT  5584  Program  and  Product  Evaluation     Evaluation  Plan  Draft     Evaluation  of  Students  Learning  Outcomes  for  Virginia  Tech's  Faculty-­‐led  Study  Abroad   Programs     Introduction   The  evaluation  plan  presented  in  this  document  outlines  the  process  to  evaluate  the  learning   outcomes  produced  by  undergraduate  students  at  Virginia  Tech  after  participating  in  the   Faculty-­‐led  study  abroad  programs.  This  plan  comprises  general  information  about  the   program,  reasons  of  conducting  this  evaluation,  stakeholders,  and  intended  actions  that  the   Global  Education  Office  at  Virginia  Tech  may  take  at  the  end  of  this  work.  At  the  same  time,  this   evaluation  plan  describes  the  criteria  of  selecting  the  sampling,  data  collection  methods  and   instruments  to  be  implemented  during  the  evaluation  process.  At  the  end,  this  document   presents  the  evaluations’  requirements  and  limitations,  as  well  as  the  timeline  and  budget.       This  evaluation  plan  was  designed  by  Roberto  Carlos  Flores,  VT’s  faculty-­‐led  programs  project   evaluator.  Also,  it  has  been  prepared  for  the  Global  Education  Office  as  a  primary  client  in   collaboration  with  Dr.  Selma  Haghamed,  Faculty  Support  Associate.     Background     The  Global  Education  Office  (GEO)  strives  to  strengthen  Virginia  Tech's  mission  of  global   engagement,  discovery,  and  service.  Nevertheless,  this  office  reports  directly  to  the  Vice   President  for  the  Outreach  and  International  Affairs  and  its  objective  is  intimately  aligned  to  it.   The  purpose  of  the  Outreach  and  International  Affairs  is  to  share  the  best  of  Virginia  Tech  by   working  step  by  step  with  the  communities  around  the  Commonwealth  and  throughout  the   world  developing,  transforming  and  engaging  the  world  while  inventing  the  future.    

Among  the  different  programs  that  the  GEO  offers  to  students,  there  is  one  called  Faculty-­‐led   study  abroad  programs.  They  are  usually  designed  by  VT  faculty  members  and  sponsored  by  a   specific  VT  department.  What  the  students  should  know,  do  or  appreciate  as  a  result  of   participating  in  these  programs  is  currently  determined  by  each  course/instructor.  They  design   and  lead  this  experience  in  order  to  bring  global  engagement,  discovery,  and  service  by   connecting  faculties  and  students  to  their  peers  abroad;  inspiring  and  contributing  to  the   faculty’s  creation  and  promotion  of  cross-­‐cultural  learning  experiences;  and  supporting  the   development  of  VT’s  students  into  globally  competent  citizens.   These  programs  are  guided  by  the  Global  Education  Office  and  their  range  is  from  one  week  to   a  full  semester.  Although  the  majority  occur  over  the  winter  session  (2  weeks)  and  summer  (4-­‐5   weeks).  Students  at  VT  from  all  the  backgrounds  and  disciplines  can  apply  online  to  these   programs  and  need  to  meet  specific  requirements.  For  instance,  they  must  have  a  minimum  2.5   GPA.  Nevertheless,  some  of  these  programs  demand  a  higher  GPA.  Any  exception  is  at  the   discretion  of  the  faculty  program  leader.  When  students  are  selected  to  participate  in  this   program,  they  need  to  go  through  a  pre-­‐decision  and  post-­‐decision  stages.  Once  this  is   completed,  they  need  to  complete  completely  accepted  they  would  have  the  opportunity  to   explore,  learn  and  engage  with  other  cultures  from  this  experience.   With  50+  programs  spanning  all  colleges  and  about  78%  of  overall  student  mobility,  the  faculty-­‐ led  context  makes  up  a  key  program  category  at  Virginia  Tech  and  according  to  the  Global   Education  Office,  they  merit  specific  evaluation.   Purpose   The  purpose  of  this  project  is  to  determine  the  common  student  learning  outcomes,  aligned   with  the  Global  Education  Office  and  Outreach  and  International  Affair’s  objectives,  as  a  result   of  participating  in  the  faculty-­‐led  study  abroad  programs  at  Virginia  Tech.      

Stakeholders/Audiences   Primary:     -­‐   Global  Education  Office.  Main  client  and  organization  that  will  have  access  to  the  final   results  of  this  evaluation  project.  This  includes  the  following  persons:   •   Selma  Haghamed,  Faculty  Support  Associate   •   Rachel  Fitzgerald,  Assistant  Director  for  Program  Management   •   Theresa  Johansson,  Director  of  Global  Education   •   Betty  Watts,  Coordinator  of  Global  Initiatives     Secondary:     -­‐   Outreach  and  International  Affairs  who  also  may  be  interested  in  knowing  the  projects’   findings.     Tertiary       -­‐   Students  who  would  like  to  be  part  of  the  faculty-­‐led  programs.   -­‐   Faculty  professors  since  they  are  the  ones  in  charge  of  designing  the  programs  since   students’  learning  outcomes  are  determined  by  the  programs’  objectives.     Decisions:   The  results  of  this  evaluation  will  be  used  by  the  Global  Education  Office  in  order  to:   -­‐   Determine  the  students’  learning  outcomes  for  Virginia  Tech  faculty-­‐led  programs  in   consultation  with  the  Outreach  and  International  Affairs’  mission.       -­‐  

Break  the  resulting  learning  outcomes  into  key  performance  indicators  for  continuous   quality  performance  of  the  programs.  

-­‐  

Support  the  development  of  specialized  evaluation  instruments  that  will  measure  the   identified  student  learning  outcomes.  

Sample:   Throughout  the  year,  there  are  more  than  50  faculty  led  programs  that  are  implemented  in   around  20  countries  throughout  the  year.  In  order  to  narrow  this  down,  two  important  factors   were  considered  in  the  selection  of  the  evaluation’s  sample:  the  period  of  time  in  which  the   programs  took  place  and  the  geographical  location  based  on  this  time.  The  period  of  time  was   summer  2016  which  was  the  closest  period  of  the  implementation  for  this  evaluation  and  the   region  was  Latin  America  because  of  the  diversity  of  programs.  At  the  end,  the  sample  group   was  reduced  to  three  and  the  sample  population  to  60  students  (around  20  per  group),  2   faculties,  three  professors  in  charge  of  these  programs  and  3  program’s  designers.       Methods  and  Instrumentation   The  information  to  be  collected  for  this  evaluation  project  comprises  qualitative  and  qualitative   data  which  means  that  the  methodology  involves  a  Mixed  method  design.  The  two  instruments   that  will  help  the  evaluator  gather  this  data  will  come  from:     -­‐   Electronic  Survey.  The  evaluator  will  create  a  survey  using  Qualtrics  and  this  one  will  be   sent  to  the  sample  group  through  an  email.  The  Global  Education  Office  will  help  the   evaluator  distribute  this  survey  using  its  institutional  email  account.     -­‐   Interviews.  With  the  help  of  the  GOE,  the  evaluator  will  identify  the  designers  of  the   three  faculty-­‐led  programs  as  well  as  the  three  professors  in  charge  of  developing  the   programs  with  the  students  in  order  to  conduct  an  interview.                    

Data  Decision  Matrix   KEY  EVALUATION  QUESTIONS  

MOST  EFFICIENT  AND  EFFECTIVE  DATA   SOURCES  AND  COLLECTION  METHODS  TO   ANSWER  THIS  QUESTION   In  what  ways  do  students’  learning  outcomes   Questionnaire  to  sample  groups.  Items  to  be   are  aligned  to  GEO  and  outreach  and   included:   International  Affairs’  goals?   -­‐              Multiple  choice  questions   -­‐              Rating  scale     To  what  extend  is  the  program  facilitating   assessment  to  students?  

What  extend  is  the  relationship  between  the   pre  orientation  programs  and  learning   outcomes?  

-­‐              Open-­‐ended  questions   Interview  faculty-­‐led  programs  Designers.   Questions  related  to  the  design,  program   goals,  assessment  rubrics,  professors’  role,   etc.   Interview  Faculty-­‐led  professors  or  mentors   about  previous  programs  and  their   experiences  coaching  students.    

  Logistics   This  evaluation  project  will  be  carried  out  by  Roberto  Carlos  Flores.  As  a  project  evaluator,  he   will  be  responsible  to  perform  the  following  actions:   -­‐   Design  the  evaluation  plan.   -­‐   Create  and  administer  the  different  data  collection  methods.   -­‐   Analyze  the  information  gathered  from  the  surveys  and  interviews.   -­‐   Provide  the  findings  of  this  evaluation  to  the  main  stakeholder   Timeline  

 

Milestone  

Target  Completion  Date  

Complete  the  Evaluation  Plan    

October  20  

Distribute  surveys  to  the  sample  groups    

November  1  

Conduct  interviews  to  designers  and  professors  

November  10  

Data  Analysis    

November  21  

Write  final  report  and  present  it  to  GEO  

December  1  

Budget   The  evaluation  of  this  program  will  not  require  any  economical  expenses     Limitations   -­‐   One  of  the  biggest  limitations  consider  in  this  project  will  be  the  rate  of  students’   responses  to  the  survey  from  the  sample  group.     -­‐   Faculty-­‐led  programs  designers  and  professors  may  have  limited  time  to  help  the   evaluator  conduct  the  interviews.