Abstract:  Introduction  to  Cognitive  Science  is  a  course  that  tests  students  in  a  way   that  requires  a  great  deal  of  conceptual  and  procedural  knowledge  about  the  subject   matter.  This  project  used  various  teaching  methods  in  class  sections  to  test  which   ones  were  most  effective  in  obtaining  the  highest  level  of  scores  on  exam  questions   that  were  explicitly  taught  in  sections  and  why  that  method  was  most  effective.   Teaching  students  through  student  led  and  performed  skits  of  the  conceptual  and   procedural  knowledge  was  the  most  effective  in  performing  well  on  the  exam  and   retaining  the  information.       Striking  a  balance  between  learning  introductory  course  materials  and  course  study   methods     Michelle  D.  Greenwood   Fifth  Year  Ph.D.  Candidate   School  of  Social  Sciences,  Humanities,  and  Arts   University  of  California,  Merced   Council  of  Graduate  Schools  Undergraduate  Outcomes  Assessment:  Pedagogy  and   Program  Planning  Project   Introduction  to  Cognitive  Science         The  course  Introduction  to  Cognitive  Science,  an  interdisciplinary  study  of   the  mind,  is  a  course  that  exposes  students  to  the  discipline  of  cognitive  science  and   provides  a  fundamental  foundation  for  undergraduates  who  have  chosen  cognitive   science  as  their  major.  It  also  serves  as  a  general  education  requirement  for  other   disciplines  alternatively.    The  major  is  intended  to  develop  individuals  with   fundamental  skills  that  are  also  optimally  marketable  in  both  industry  and  academia.   The  interdisciplinary  nature  of  cognitive  science  allows  students  to  experience   several  different  subjects  and  how  they  intersect.    Due  to  the  multiple  themes  and   the  lower  division  level  of  the  course  it  follows  that  students  who  join  this  course   get  a  broader  scope  of  content  rather  than  a  depth  that  comes  from  the  upper   division  courses  within  the  major.       The  intended  learning  outcomes  for  the  course  are  that  the  students  will  be   come  familiar  with  the  theoretical  frameworks  of  philosophy  of  mind,  cognitive   neuroscience,  cognitive  psychology,  theoretical  linguistics,  and  artificial  intelligence.   They  will  become  familiar  with  the  different  methods  of  data  collection  and  analysis   that  accompanies  those  aforementioned  frameworks  and  acquire  basic  knowledge   of  formal  logic,  experimental  design,  statistics,  linguistic  formalism,  computing   theory,  and  the  brain.  More  specifically,  the  student  learning  outcomes  for  the   required  discussion  sections  are  to  gauge  attendance  and  participation  within  the   course  while  ensuring  students  are  doing  the  assigned  readings  for  the  course   supplementing  the  lectures  and  their  understanding  of  main  lecture  content.  The   main  way  of  evaluating  progress  in  this  course  is  through  two  midterms  and  a  final.      

The  University  of  California,  Merced  program  learning  outcomes  for  the   course  Introduction  to  Cognitive  Science  are  all  meant  to  be  at  an  introductory  level.i   The  course  is  structured  such  that  students  attend  two,  one  hour  and  fifteen  minute   lectures  weekly  and  an  additional  fifty  minute  discussion  section  weekly.  The   lecture  is  a  large  classroom  meant  to  deliver  the  majority  of  the  course  content.  The   sections  are  meant  to  supplement  the  course  content  by  ensuring  the  students  are   fulfilling  the  reading  requirements,  participating  through  discussion,  and  clarifying   any  unclear  lecture  material  in  a  small  classroom  format.       The  assessment  I  conducted  in  my  Fall  2014  discussion  section  has  focused   on  the  effectiveness  of  the  discussion  sections  and  how  they  relate  to  the  course   learning  outcomes  and  also  the  program  learning  outcomes  for  cognitive  science.  In   developing  weekly  lesson  plans  for  the  discussion  sections  I  used  many  of  the   hallmarks  of  learner-­‐centered  teaching  (Huba,  2000)  and  the  principles  outlined  in   “Learning  in  Groups”  (Davis,  1993),  often  referred  to  as  active  learning.  I  also  tried   to  incorporate  technology  into  these  active  learning  lessons  using  the  tools  students   already  had  at  their  fingertips  and  as  suggested  by  “New  Media  Technologies  and   the  Scholarship  of  Teaching  and  Learning  (Wesch,  2009),”  with  timely  feedback   while  students  were  presenting  (Hattie  &  Timperley,  2007).  The  lesson  plans   consisted  of  a  variety  of  different  techniques  to  encourage  students  to  delve  into  the   readings,  draw  out  main  themes,  and  apply  them  to  the  lecture  material  they  would   be  receiving  in  class.  The  discussion  sections  occurred  earlier  in  the  day  before  all   lectures  of  the  week.  Lessons  were  planned  so  that  they  would  be  reviewing   readings  for  the  coming  week  to  accompany  coming  lectures.  The  goal  was  to  help   students  digest  the  readings  before  the  lectures  giving  them  a  foundation  to  build   upon  before  they  attended  lectures.       These  different  active  learning  lessons  allowed  me  to  experiment  with   multiple  methods  for  helping  students  solidify  the  concepts  in  cognitive  science  and   to  build  on  a  fundamental  knowledge  base  needed  to  do  well  on  their  exams.  While   exams  are  not  the  only  way  of  ascertaining  whether  they  have  comprehended  the   course  material  it  is  the  main  convention  used  within  the  course  lecture  to   determine  if  they  have  grasped  the  underlying  information.  The  exams  consisted   mainly  of  multiple-­‐choice  questions  and  several  fill  in  the  blank  questions.  To  assess   how  effective  my  discussion  sections  were  I  chose  several  multiple-­‐choice  questions   and  a  few  fill  in  the  blank  questions  from  the  exams  that  covered  the  material  I   stressed  in  my  sections.           In  the  needs  assessment  surveyii  conducted  at  the  beginning  of  the  semester   students  were  asked  about  many  things  used  to  help  in  shaping  the  active  learning   discussion  lesson  plans.  Two  responses  I  would  point  out  from  that  survey  is  that   students  self-­‐reported  their  primary  learning  style  was  visual,  67%  and  51%  said   their  secondary  learning  style  was  kinesthetic.  The  other  finding  was  that  54%  of   the  students  reported  that  English  was  their  second  language.  This  finding  becomes   especially  significant  when  interpreting  the  other  findings  from  the  exams.  The   needs  assessment  suggested  that  various  forms  of  technology,  social  media,  and  

other  forms  of  learning  instruction  would  be  appropriate  given  the  learning  styles  of   these  students.       After  seven  weeks  of  discussion  sections,  the  students  were  required  to  take   a  mid-­‐term  exam,  and  they  also  filled  out  a  surveyiii  about  the  discussion  sections  in   order  to  gauge  improvements  and  any  adjustments  that  needed  to  be  made.   Students  self-­‐reported  that  they  understood  the  readings  better  (84%)  and  the   lectures  better  (73%)  because  of  their  attendance  in  sections.  Because  the  midterm   survey  question  was  vague  their  better  understanding  of  the  readings  and  lectures   is  vague  and  could  refer  to  than  the  beginning  of  the  course  or  relative  to  their   classmates  in  other  sections  or  both.  One  thing  I  was  particularly  interested  in   finding  out  was  whether  certain  “active  learning”  activities  were  more  effective  in   helping  students  retain  the  information  over  other  activities.  In  their  midterm   assessment  I  discovered  the  information  they  remembered  the  most  was  the   Chinese  Room  experiment,  the  various  neurological  methods  for  evaluating   different  brain  activity  patterns,  and  the  different  types  of  neural  networks  (see   figures  1  and  2).  Their  favorite  activities  were  skits  created  and  performed  by  the   students,  a  video  clip  from  the  Big  Bang  Theory,  and  review  of  the  course  material   (see  figures  1  and  2).  For  the  content  students  remembered  the  most  there  is  some   overlap  of  those  also  being  their  favorite  activities  as  well.  One  of  the  most   interesting  details  about  the  content  they  remembered  was  that  it  involved  skits.  In   order  for  them  to  perform  the  skit  they  needed  to  learn  the  material  well  enough  to   teach  it  to  someone  else.  I  think  that  embodying  the  information  also  helped   reinforce  the  learning  in  turn  helping  them  to  remember  it.  The  average  scores  of   the  exams  for  all  the  classes  were  as  follows:  midterm  one,  60%;  midterm  two,  75%;   and  the  final,  70%iv.       I  chose  eleven  questions  from  the  first  and  second  midterm  exams  and  eight   from  the  final  to  get  a  more  accurate  assessment  of  the  effectiveness  of  my  sectionsv.   The  questions  were  chosen  based  on  the  relevance  of  the  test  question  compared  to   the  material  specifically  taught  in  my  sections.  The  question  they  performed  the  best   on  from  the  first  midterm  (82%  correct)  was  one  that  referred  to  neural  networks.   This  particular  question  was  also  the  one  that  the  students  performed  skits  about   and  reported  remembering  on  the  midterm  survey  as  well  as  liking  more  than  other   activities.  If  the  students  truly  are  visual  and  kinesthetic  learners  this  correlates   nicely  with  the  midterm  survey  results  and  the  course  outcome.  Out  of  the  eleven   exam  questions  sampled  from  the  first  midterm  based  on  what  was  taught  in   sections,  students  received  a  correct  answer  over  60%  of  the  time  for  seven  of  them,   which  is  over  60%  of  the  questions  sampled.  For  the  second  midterm  the  question   they  performed  the  best  (93%  correct)  on  was  regarding  how  neurons  work   together  to  create  a  mental  representation.  Out  of  the  eleven  exam  questions   sampled  from  the  second  midterm  based  on  what  was  taught  in  sections,  students   received  a  correct  answer  over  70%  of  the  time  for  eight  of  them,  which  is  over  70%   of  the  questions  sampled.  As  for  the  final,  six  out  of  the  eight  questioned  sampled   students  in  my  sections  scored  better  than  70%  on  them.  One  question  100%  of  the   students  answered  correctly.    

  I  also  analyzed  the  first  and  second  midterms  and  the  final  exam  questions   based  on  the  “The  Taxonomy  for  Learning,  Teaching,  and  Assessing”  (Bloom  &   Krathwohl,  1956;  http://www.celt.iastate.edu/pdfs-­‐ docs/teaching/RevisedBloomsHandout.pdf).  For  the  first  midterm  83%  of  the   questions  were  conceptual  and  procedural  questions  requiring  a  higher  order  of   cognitive  processing.  If  you  compare  this  to  the  second  midterm  which  only  had   70%  conceptual  and  procedural  questions  while  the  final  only  had  58%  of  those   type  of  questions  you  could  imagine  why  the  students  did  not  perform  as  well  on  the   first  midterm  in  an  introductory  level  course.  According  to  Bloom  &  Krathwohl’s   model  this  also  requires  teaching  “active  learning”  lessons  that  help  prepare   students  to  know  more  than  just  facts  but  also  prepare  them  to  apply  the  knowledge   conceptually  and  procedurally.  Having  students  work  in  groups,  prepare  skits,  and   present  the  material  to  others  help  them  remember,  understand,  apply,  analyze,   evaluate,  and  create  information  and  perform  better  on  exams.  This  higher  level  of   cognitive  processing  will  also  help  them  in  their  other  courses  within  the  major.  It   also  achieves  the  program  course  outcomes  for  this  introductory  course  within  the   major.  This  aligns  nicely  with  the  program  review  conducted  by  outside   consultantsvi  who  recognize  the  strength  of  the  conceptual  and  procedural  elements   of  the  major  along  with  a  consideration  of  “offering  more  ‘hands-­‐on’  courses  with  a   technical  component  such  as  data  science,  speech  recognition,  interface  design,  and   other  aspects  of  human  computer  interaction.”  While  this  might  be  difficult  to   conduct  in  a  large  introductory  course  this  class  serves  as  an  excellent  foundation   for  that  type  of  interaction  in  upper  division  courses  for  the  major.       In  conclusion,  I  have  learned  that  providing  students  with  opportunities  to   embody  the  material  and  then  teach  it  to  others  is  an  effective  way  for  students  to   learn.  As  a  teacher,  I  recognize  the  depth  of  understanding  you  have  about  the   material  you  are  teaching  can  influence  the  way  you  teach  the  material.  This  has   changed  the  way  I  think  about  teaching  and  has  given  me  ideas  about  ways  I  will   change  my  teaching  style  in  the  future.  Often  has  been  the  case  that  my  strategy  for   lesson  planning  has  been  to  do  the  bulk  of  the  work  for  the  students  in  order  to   provide  them  with  valuable  content.  Continuing  forward,  I  think  the  focus  should  be   on  providing  an  atmosphere  and  a  structure  that  encourages  the  students  to  do  the   bulk  of  the  work  themselves.  When  students  embody  the  material  it  helps  them  to   grasp  the  concept,  reinforce  their  learning  and  promotes  better  retention  of  the   material  as  they  then  teach  it  to  fellow  classmates.                      

                    References:     Bloom, B. S. (1964). David R. Krath wohl, Taxonomy of Educational Objectives: Cognitive Domain(New York: David McKay, 1956). Davis, J. R. (1993). Better Teaching, More Learning: Strategies for Success in Postsecondary Settings. American Council on Education Series on Higher Education. Oryx Press, 4041 North Central at Indian School Road, Phoenix, AZ 85012-3397.. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of educational research, 77(1), 81-112. Huba, M. E., & Freed, J. E. (2000). Learner centered assessment on college campuses: Shifting the focus from teaching to learning. Community College Journal of Research and Practice, 24(9), 759-766. Wesch, M. (2009). From knowledgeable to knowledge-able: Learning in new media environments. Academic Commons, 7.

                       

Active  Learning   Associated  with   concepts  students   remembered  the   most  and  was  their   favorite  activity  

Neural   Networks  

Chinese   Room   Experiment  

• Remembered  Most  (43%)   • Favorite  Activity:  Skits   (30%)   • Favorite  Activity:  Review   of  course  material  (24%)    

Neurological   Methods  

• Remebered  Most  (41%)   • Favorite  Acitvity:  Review   of  course  material  (24%)  

• Remembered  Most  (33%)   • Favorite  Activity:  Skits   (30%)   • Favorite  Acitivity:  Review   of  Course  Material  (24%)    

  Figure  1:  Free  response  from  survey  shows  the  intersection  of  students’  favorite  activities  that  were   used  for  learning  the  concept  with  the  concepts  that  students  remembered  the  most.    

    LESSON  CONTENT   REMEMBERED   MOST   Chinese  Room   Experiment  (43%)   Neurological   Methods  (41%)   Neural  Networks   (33%)  

FAVORITE   ACTIVITY   Skits  (30%)   Review  of  Course   Material  (24%)   Video  Clip  (Big   Bang  Theory)   (27%)  

Figure  2:  The  percentages  of  student  reported  of  the  lesson  content  they  remembered  and  their  favorite   activities.    

               

                                                                                                                i  Cognitive  Science  Program  Learning  Outcomes   Explain  and  apply  knowledge  of  landmark  findings  and  theories  in  cognitive  science.   Design,  interpret,  and  evaluate  simple  behavioral  and  neuroscientific  experiments.   Interpret  and  appreciate  formal  and  computational  approaches  in  cognitive  science.   Argue  for  or  against  theoretical  positions  in  cognitive  science.   Use  a  cognitive  science  education  outside  of  the  undergraduate  classroom,  particularly  in  the  

 

service  of  careers.  

ii  Needs  Assessment    

  What’s  your  major?   What  year  are  you?   Do  you  have  a  smart  phone?   If  you  have  a  smart  phone  do  you  have  a  plan  that  allows  you  to  text  enough  that   you  can  text  in  class  many  times  throughout  the  semester?     If  you  do  not  have  a  smart  phone  please  answer  N/A.   Do  you  have  a  laptop  or  portable  computer  device  that  you  can  bring  to  class  with   you?   How  well  do  you  typically  do  on  tests?  Please  explain  your  answer...  Why  do  you  do   well  or  not  do  well?   Do  you  have  a  Facebook  account?   What  are  your  top  5  favorite  TV  shows?   What  are  your  top  5  favorite  musicians/bands/CDs(mp3s)?   Please  upload  a  picture  of  yourself   Is  English  your  second  language?   What  grade  do  you  hope  to  earn  in  this  course?     Which  of  the  following  objectives  will  help  you  succeed  in  this  course?  Prioritize  this   list  with  1,  2,  &  3;  with  1  being  most  beneficial  and  3  least     Visit  TA  or  professor  at  office  hours     Attend  all  lectures     Complete  the  assigned  text  readings     Read  the  assigned  text  before  sections     Take  note  on  readings     Participate  actively  (e.g.  ask  questions)  in  class  and  in  sections     Take  notes  in  classes/section     Join  a  study  group     Other   Which  learning  types  do  you  think  most  suit  your  learning  style?  Prioritize  this  list   with  1  being  most  identified  and  3  being  least.       Auditory  (e.g.  listening  to  a  lecture)     Visual  (e.g.  movies,  graphs,  and  through  note-­‐taking)     Kinesthetic  (e.g.  activities  and  problem  sets)    

                                                                                                                                                                                                                                                                                                                                          If  you  could  get  help  with  one  thing  in  lecture  what  would  it  be?   Comments?  Feel  free  to  vent,  encourage,  tell  a  joke,  or  explain  something  (please   give  links  to  pictures  or  videos  as  necessary).       iii  Mid-­‐term  Survey   What  are  some  “real  life”  daily  experiences  that  you  think  about  differently  now   because  of  this  class?     Do  you  think  you  understand  the  readings  better  as  a  result  of  attending  sections?     Do  you  think  you  understand  the  lecture  material  better  as  a  result  of  attending   sections?     Which  activity  that  we’ve  done  in  section  helped  you  remember  course  material  the   most?         What  exactly  do  you  remember  about  the  course  material  because  of  that  activity?       What  has  been  your  favorite  activity  we’ve  done  in  sections  so  far?     Find  your  match  of  either  vocabulary  word  or  the  definition     Example  of  Chinese  (Spanish)  Room  Experiment     Big  Bang  Theory  TV  Clip   Drawn  a  picture  of  a  probabilistic  state  change  over  time   Skits  of  Neural  Networks   Paragraph  summary  of  the  reading   Reviewed  the  sections  and  the  lectures       What  are  the  top  three  things  Michelle  Greenwood  could  change  to  help  you  learn   Cognitive  Science  better?  (Please  keep  in  mind  that  I  can  only  fit  so  much  into  a  50   minute  period  but  this  helps  me  prioritize)     iv  Attendance  to  sections:  While  there  were  some  who  self-­‐reported  understanding   the  material  better  and  feeling  like  they  did  better  because  they  attended  sections   there  were  some  who  did  well  on  the  exams  and  did  not  attend  sections  regularly  or   often.  The  converse  is  also  true  in  that  others  attended  all  the  sections  and   participated  regularly  and  still  had  a  low  exam  score.  It  is  difficult  to  tell  how  much   attendance  and  participation  in  sections  helped  with  scores.  However,  there  is  one   piece  of  evidence  that  seems  to  corroborate  this  line  of  reasoning.  For  the  first   midterm  the  average  score  for  the  all  the  sections  was  a  60  and  the  average  for  my   sections  was  68.  It  was  during  the  sections  for  the  material  up  through  the  first   midterm  that  a  large  portion  of  the  active  learning  lessons  was  implemented.       v  Exam  Questions  

                                                                                                                                                                                                                                                                                                                                          Due  to  the  re-­‐use  of  exam  questions  and  the  public  nature  of  this  paper  I’m  not   including  any  of  the  actual  test  questions  in  this  paper.  However,  here  are  some   review  questions  that  will  give  you  an  example  of  the  type  of  questions  that  would   be  similar  to  those  that  would  appear  on  the  test.   COGS 1 Prep Quiz for MidTerm Exam #1 This 15-question quiz will not be graded. It is purely for study practice purposes. 1. The field of cognitive science is best be described as: a. a philosophical study of the mind based on introspection b. a neural network approach to brain science c. a collaborative, interdisciplinary study of the mind d. the convergence of philosophy and psychology 2. The phrase that characterizes Gestalt Psychology (as well as emergent properties in dynamical systems) is: a. “Neurons that fire together wire together.” b. “Beware the combinatorial explosion.” c. “The whole is equal to sum of its parts.” d. “The whole is more than the sum of its parts.” 3. What does it mean to operationalize your dependent variable? a. It means to identify a quantifiable experimental manipulation that you will impose on your subjects in an experiment. b. It means to program a computer so that it will carry on a natural conversation. c. It means to convert your independent variable into a dependent variable. d. It means to identify a quantified experimental measure that you will record from your subjects and treat as an index of some perceptual or cognitive process. 4. Cognitive psychology began largely as a response to: a. Gestalt Psychology b. Behaviorism c. Philosophy d. Freudian Psychoanalysis 5. These four major theoretical movements in experimental psychology over the past century follow what order in time (starting with the oldest): a. Introspectionism, Connectionism, Behaviorism, Information-Processing. b. Behaviorism, Introspectionism, Connectionism, Information-Processing. c. Information-Processing, Introspectionism, Behaviorism, Connectionism. d. Introspectionism, Behaviorism, Information-Processing, Connectionism. 6. When an action potential from one neuron influences the activity of another neuron, the electrochemical signal typically travels: a. from the axon to the synaptic cleft to the dendrite to the cell body. b. from the synaptic cleft to the axon to the cell body to the dendrite. c. from the axon to the dendrite to the synaptic cleft to the cell body. d. from the axon to the cell body to the dendrite to the synaptic cleft. 7. The grammar of a language is made up of rules that govern its: a. structure, function, and plasticity b. phonology, homology, structure, and gestalts c. phonology, morphology, syntax, and semantics d. sound system, written system, and auditory system

                                                                                                                                                                                                                                                                                                                                          8. What kind of learning system searches the solution space of a problem by trying many different, initially random, combinations of features in a species, and combinations of features that do well enough at solving the problem get their features recombined with one another (while other combinations are discarded) to make a new set of feature combinations that the next generation of the species will use to try to solve the problem? a. neural network b. connectionist model c. genetic algorithm d. ACT* 9. Which of the following is the second stage in Cognitive Psychology's idealized description of human information processing? a. Recognition b. Detection c. Response Selection d. Reasoning 10. In traditional AI approaches, a tree-search algorithm that looked first at a long sequence of potential moves and countermoves, before considering the range of possible first moves, would be called what kind of search? a. breadth-first b. depth-first c. NP-complete d. parallel 11. When multiple units in a system cooperate to encode the meaning of an external stimulus, this is called ______________________________________________ 12. When a single unit in a system encodes (all by itself) the meaning of an external stimulus, it is called: _________________________________________________ 13. What is the class of philosophical theories of the mind in which one postulates both a physical brain that influences behavior to some degree and a non-physical mind that also influences behavior (by modulating how the brain functions)? 14. The smallest unit of sound in a spoken language, which typically carries no meaning on its own, is called a: _________________________ 15. In Linguistics, a sentence can be diagrammed as a hierarchical structure where the sentence node branches down into phrase nodes which then branch down into the words that make those phrases. This downward-branching tree is called a:     vi  Report  of  External  Review  Committee  on  the  Undergraduate  Cognitive  Sciences  (CIS)   Program  at  UC  Merced   Submitted  May  31,  2013   External  Review  Committee  Members:  Toby  Mintz  (University  of  Southern  California),  Seana   Coulson  (University  of  California,  San  Diego)  and  Jeff  Gilger  (University  of  California,  Merced)   Dates  of  Review:  May  1  &  2   We  are  pleased  to  provide  our  assessment  of  UC  Merced's  Cognitive  Science  Program,  managed  

                                                                                                                                                                                                                                                                                                                                          by  the  faculty  of  the  Cognitive  and  Information  Sciences  (CIS)  Bylaw-­‐55  Unit.  The  undergraduate   degree  program  is  Cognitive  Science,  although  we  often  refer  to  the  bylaw  unit,  CIS,  in  this   document.   We  would  like  to  thank  Executive  Vice  Chancellor  and  Provost  Tom  Peterson;  Vice  Provost  for   Undergraduate  Education  Jack  Vevea;  Vice  Chancellor  for  Student  Affairs  Jane  Lawrence;  Mark   Aldenderfer,  SSHA  Dean;  Professor  Cristián  Ricci,  UGC  Chair  and  PRC  Chair;  Professor  Michael   Spivey,  CIS  Program  FAO,  as  well  as  their  colleagues  in  the  CIS  Program,  for  their  cordial   welcome  and  for  a  highly  informative  and  enjoyable  visit  to  UC  Merced  on  May  1-­‐2,  2013.  We   are  especially  grateful  to  Ms.Fatima  Paul  for  her  expert  handling  of  the  site  visit.  We  would  also   like  to  express  our  admiration  for  the  enthusiastic  undergraduates  with  whom  we  spoke.  We   trust  that  our  observations,  inferences,  and  recommendations  reflect  our  respect  and  high   regard  for  the  CIS  Program,  the  achievements  of  its  faculty  and  students  to  date,  and  their   aspirations  for  the  future  of  the  Program.     Overview  of  Process   In  advance  of  the  site  visit,  the  Review  Committee  (RC)  was  provided  with  (i)  a  tentative   schedule  for  the  visit;  (ii)  the  CIS  self-­‐study  document;  (iii)  a  link  to  the  UCM  General  Catalog;   and  (iv)  a  link  to  documentation  regarding  guidelines  and  questions  for  reviewers.  The  Review   Committee  (RC)  examined  the  CIS  self-­‐review  and  accompanying  documents,  and  met  with  CIS   faculty  and  students,  as  well  as  UC  Merced  administrators.  The  on-­‐campus  visit  by  the  RC   occurred  on  May  1st  and  2nd  (see  attached  agenda).  The  review  protocol  and  materials  were   fairly  standard  with  the  exception  of  the  lack  of  confidential  faculty  surveys  and  teaching   evaluations  by  students.  However,  by  the  end  of  the  site  visit,  the  RC  felt  that  the  surveys  and   evaluations  were  not  needed,  and  that  an  adequate  picture  of  the  program  was  presented  via   the  combination  of  the  CIS  self-­‐report  and  the  faculty/student  interviews.     Abstract   The  CIS  program  is  relatively  small  in  terms  of  faculty  and  student  base.  This  is  in  part  due  to  its   fairly  recent  establishment  on  a  new  UC  campus  as  a  major  in  2007.  CIS  is  a  highly   interdisciplinary  program,  with  roughly  10  key  faculty  spread  across  multiple  home  bylaw  units.   The  number  of  current  majors  is  approximately  79.  CIS  has  four  main  areas  of  research  focus:   computation,  reasoning,  perception,  and  language.  While  this  Committee  was  tasked  with  a   review  of  the  undergraduate  program,  it  is  noteworthy  that  CIS  has  a  healthy  graduate  and   research  agenda  that  clearly  impacts  the  undergraduate  experience  for  majors  in  very  positive   ways.  In  fact,  it  is  the  integration  of  research-­‐related  experiences  at  the  undergraduate  level   that  distinguishes  the  CIS  program  from  many  similar  programs  at  other  universities.  The   philosophy  of  CIS  is  to  maintain  controlled  undergraduate  growth,  focusing  mainly  on  student   quality  rather  than  quantity,  and  to  develop  ways  to  best  insure  successful  educational   experiences  and  career  trajectories.  CIS  has  a  well-­‐conceived  strategic  plan  that  integrates   faculty  hires,  curriculum,  and  undergraduate  training  with  the  interdisciplinary  nature  of  the   field  and  with  an  eye  toward  the  future  landscape  of  science  and  technology  from  both  applied   and  theoretical  perspectives.     Evaluation   According  to  the  UC  Merced  Undergraduate  Program  Review  Policy  and  Procedures,  systematic   and  regular  review  of  undergraduate  academic  programs  is  intended  to  ensure  that  students  are   learning  what  is  intended,  that  educational  efforts  are  appropriate  to  a  diverse  student  body,  

                                                                                                                                                                                                                                                                                                                                          and  that  the  benefits  of  scholarly  inquiry  will  inform  educational  processes  and  outcomes.  Thus,   a  program  review  should  be  formative  (in  that  it  shapes  the  actions  of  a  program  in  its  ongoing   development)  and  summative  (in  that  it  identifies  particular  issues  and  problems  that  may  need   to  be  addressed  and  identifies  actions  required  to  address  such  issues  and  problems).   Towards  these  objectives,  this  report  briefly  summarizes  key  issues  coming  out  of  the   Committee  review,  and  it  includes  RC  comments  on  various  aspects  of  the  CIS  program.   Strengths  and  weaknesses  are  highlighted,  and  where  appropriate,  recommendations  follow.     Level  of  the  Faculty,  Unit,  School  and  University   The  success  of  any  program  depends  on  the  hierarchical  and  integrative  interactions  among   faculty,  the  bylaw  unit,  the  school  housing  the  unit,  and  the  overall  university  infrastructure.  At   the  faculty  and  unit  level,  CIS  has  done  a  good  job  at  maintaining  a  coherent  collection  of   affiliates  across  multiple  disciplines,  including  cognitive  and  computer  scientists,  psychologists,   economists,  and  philosophers.  Senior  faculty  features  internationally  recognized  scholars,  and   junior  faculty  includes  a  number  of  rising  stars.  Although  the  program  is  relatively  small  (10   faculty),  CIS  has  done  much  to  promote  their  unique  brand  of  cognitive  science  with  focus  on  a   technology-­‐cognition  interface.  For  example,  the  2013  meeting  of  the  Cognitive  Science  Society   in  Berlin  will  feature  over  a  dozen  papers  whose  first  author  is  affiliated  with  UC  Merced  –  more   than  any  other  single  institution.  The  program  attracts  excellent  graduate  students  from  all  over   the  world,  and  is  already  recognized  as  one  of  the  leading  cognitive  science  programs  in  the   nation.   CIS  has  thought  carefully  about  the  direction  of  the  program  and  made  strategic  choices  as  to   the  hires  they  have  made.  Rather  than  focusing  exclusively  on  the  undergraduate  curriculum,   CIS  has  hired  programmatically  for  their  research  identity,  promoting  their  national  and   international  profile.  Their  stated  goal  was  to  hire  superstars  with  just  the  right  amount  of   overlap  to  promote  synergistic  research.  Their  approach  to  focus  on  a  technology-­‐cognition   interface  makes  them  unique,  attractive  to  students,  and  potentially  able  to  fill  a  growing   demand  in  industry  for  computer  literate  individuals  with  an  understanding  of  human  behavior.   Faculty  quality  and  breadth  of  coverage  is  indeed  adequate  for  a  strong  undergraduate  program.   Relative  to  most  other  cognitive  science  programs,  CIS  has  a  dearth  of  faculty  covering  the   biological  underpinnings  of  cognition,  and  undergraduates  expressed  a  desire  for  the   development  of  this  side  of  the  curriculum.  The  addition  of  recent  hires  should,  however,  off-­‐set   this  shortcoming.  Given  limitations  on  lab  space,  it  might  be  more  prudent  to  focus  future  hires   on  existing  strengths  of  the  program.  Along  those  lines,  future  hires  in  computational  linguistics,   cognitive  engineering,  and  applied  cognitive  science  could  offer  courses  that  teach  students   different  ways  to  understand  large  data  sets  that  should  aid  their  marketability  in  the   technology  sector.  The  faculty  we  spoke  to  report  good  levels  of  satisfaction  and  a  collegial   working  environment.  While  each  faculty  has  his/her  own  unique  area  of  expertise,  they   contribute  to  the  curriculum  in  specialized  yet  integrated  ways.  Thus,  students  receive  a  broad   education  with  enough  exposure  to  different  disciplines  such  that  personal  interests  can  be   fostered  and  explored,  say  in  cognitive  science  as  it  related  to  the  philosophy  of  mind,   technology  or  computer  applications,  linguistics,  and  so  on.  The  number  of  breadth  of  other   faculty  involved  in  research  and  exposure  to  undergraduates  is  relatively  large,  and  spans   anthropology,  biology,  business  and  others.   CIS  has  consciously  planned  to  grow  its  undergraduate  major  slowly,  adding  courses  as  they  get   new  faculty  and  students.  This  has  allowed  for  a  more  “intimate”  environment  for  students  and   an  ability  to  maintain  quality.  While  CIS  plans  to  continue  to  grow  its  faculty  and  student  base,  

                                                                                                                                                                                                                                                                                                                                          they  are  doing  so  with  clear  objectives  and  with  the  aim  of  making  opportunities  for   undergraduate  research  experiences.  The  RC  endorses  CIS  long-­‐range  growth  plans  that  would   maintain  the  current  student:  faculty  ratio.  Advising  staff  view  CIS  as  one  of  the  more  engaged   faculty  on  campus  in  terms  of  supporting  undergraduates,  especially  in  participation  in  research.   Faculty  is  very  responsive  to  concerns  of  the  advising  staff,  and  receptive  to  suggestions  for   improvement.  For  example,  the  introductory  cognitive  science  course  is  very  popular,  and  when   CIS  majors  were  not  able  to  enroll,  faculty  adjusted  prerequisites  and  increased  enrollment   numbers.  Undergraduate  involvement  in  research  is  seen  as  tightly  linked  to  CIS  goals  for   student  learning,  for  career  development  in  academia,  and  even  for  industry.   Faculty  are  doing  an  excellent  job  of  intellectual  development  of  the  students  and  providing  an   enriching  experience.  The  program  is  one  of  the  best  in  the  university  in  terms  of  having  ladder   rank  faculty  teaching  the  majority  of  their  courses.  The  number  of  courses  taught  by  lecturers   (~33%)  is  small  compared  to  many  other  programs  at  UC  Merced.  CIS  faculty  has  been  very   involved  in  monitoring  their  lecturers  and  making  sure  that  they  are  doing  a  good  job  of   achieving  learning  objectives.  One  lecturer  was  even  reassigned  because  his  teaching  did  not   align  with  the  learning  outcomes  desired  by  the  faculty.  In  general,  lecturers  are  considered   valued  members  of  the  CIS  program  by  faculty  and  students  alike.   Students  in  particular  drew  no  distinction  between  ladder  rank  faculty  and  lecturers  and   expressed  positive  evaluations  of  CIS  lecturers.  One  faculty  expressed  a  desire  to  offer  lecturers   security  of  employment.  The  graduate  (N  =  2)  and  undergraduate  students  (N  >  9)  we  spoke   with  all  expressed  their  appreciation  of  the  faculty  and  held  an  admiration  for  their  teaching   ability  as  well  as  their  passion  for  the  subject  area.  The  undergraduates  described  a  faculty  who   worked  well  with  students,  provided  opportunities  for  research  and  interaction,  and  who   seemed  to  really  care  about  their  success  and  development.  The  RC  was  given  some  student   evaluations  of  teaching  after  the  campus  visit.  A  review  of  these  evaluations  indicates  that  most   all  CIS  courses  are  well  received.  The  undergraduates  we  interviewed  also  spoke  very  highly  of   the  teachers  and  teaching  methods  affiliated  with  CIS.     At  the  level  of  the  school  (Social  Sciences,  Humanities  and  Arts)  and  university,  CIS  is  well   regarded  and  considered  one  of  the  best  programs  on  campus.  It  is  has  been  aligned  well  with   the  initial  principles  and  strategic  plans  of  UC  Merced  since  its  inception,  and  is  a  model  for   programs  with  an  interdisciplinary  agenda.  Although  the  program  is  relatively  small,  it  has  an   actively  growing  graduate  degree.  Faculty  service  demands  are  high  compared  to  most  other   universities,  but  this  is  common  given  the  newness  of  UC  Merced,  and  does  not  appear  to  be   adversely  affecting  work  in  the  domains  of  research  and  teaching.   Support  for  CIS  seems  strong  at  the  school  and  university  level.  The  environment  of  the  campus   is  conducive  to  cross-­‐disciplinary  work  and  innovation  without  concern  for  disciplinary   boundaries.  Instructional  support  in  terms  of  TAs,  computer  labs,  and  technical  support  were  all   cited  as  being  strong.  During  the  site  visit,  the  RC  commonly  heard  some  concerns  about  the   future  growth  of  the  campus  and  concomitant  complications  of  space,  adding  faculty  lines,  staff   support,  and  shifting  to  a  departmental  status  (as  opposed  to  remaining  a  bylaw  unit).  While  CIS   is  content  to  travel  along  its  current  path,  these  are  issues  that  will  need  to  be  addressed  at   some  future  point  (see  recommendations  below).     Level  of  the  Students  and  Curriculum   At  the  level  of  students,  the  number  of  undergraduate  minors  and  majors  has  grown  since  2007   to  2011:  5  to  38  minors  and  31  to  120  majors.  Some  of  the  faculty  of  CIS  have  percentage  

                                                                                                                                                                                                                                                                                                                                          appointments  in  units  outside  of  the  CIS  bylaw  unit,  but  roughly  10  faculty  are  identified  in  the   CIS  self-­‐study  document.  Thus,  the  2011  student-­‐tenure  track  faculty  ratio  for  the  120  majors   reported  in  the  self-­‐study  is  approximately  12:1.   The  RC  saw  only  9  or  so  undergraduate  students,  all  of  whom  were  very  much  involved  in  CIS   beyond  simple  coursework.  These  were  students  who  engaged  in  extracurricular  activities   attached  to  CIS  and  typically  worked  in  the  labs  of  CIS  faculty.  These  students  reported  highly   positive  experiences  with  the  program,  interactions  with  graduate  students  and  professors.   Many  of  the  students  also  planned  to  attend  graduate  school.  Their  comments  along  with  those   of  the  staff  and  professors  we  met  with,  suggest  an  active  and  intimate  group/major  that  allows   for  interactions  and  lab  experiences  far  beyond  that  common  to  other  national  programs  in   cognitive  science.  This  is  a  hallmark  characteristic  of  CIS  and  it  stands  out  from  other  programs   on  campus.  This  collaborative  and  intimate  atmosphere  for  students  is  made  possible  by  the   faculty  along  with  the  small  size  of  the  major.  According  to  the  CIS  self  review  and  our   conversation  with  faculty  and  staff,  CIS  has  strategically  planned  to  limit  their  growth  over  time   with  one  of  the  objectives  being  just  the  type  of  undergraduate  experience  highlighted  above.   Had  simple  growth  over  quality  or  experiential  opportunities  been  emphasized,  CIS  would  not   look  the  way  it  looks  today.  Undergraduate  evaluations  of  professors  and  courses  were  provided   to  the  RC  after  the  site  visit.  A  review  of  these  forms  indicates  that  students  have  a  generally   positive  attitude  and  opinion  of  the  teachers  and  classes.  The  self  review  also  reports  an  internal   program  evaluation  of  student  performance  and  success,  including  indices  such  as  graduate   school  applications,  GPA,  exit  exam  data,  job  acquisition,  etc.,  using  both  quantitative  and   qualitative  assessments.  It  is  clear  that  CIS  has  taken  program  evaluation  seriously,  including   tests  of  reliability  of  qualitative  measures.  CIS  has  also  attempted  to  align  their  assessments   with  an  explicit  curriculum  map.   The  statistics  provided  in  the  self  review  on  retention  rates,  GPAs,  demographics,  etc.,  are  in  line   with  the  general  trends  at  UC  Merced.  It  appears  that  CIS  admits  roughly  80%  of  their  applicants,   and  some  20%  of  these  actually  enroll.  The  trend  in  CIS  has  been  for  decreasing  numbers  of   Hispanic  students  with  increasing  numbers  of  those  self  identified  as  Asian,  and  very  high  rate  of   1st  generation  college  freshmen.  The  graduating  classes  have  been  relatively  small  due  to  the   newness  and  select  nature  of  the  program.  Thus,  data  on  graduates  is  understandably  limited   due  to  sample  size  although  signs  of  successful  career  achievement  is  still  evident.     At  the  level  of  the  curriculum  CIS  has  designed  a  unique  BA  and  BS  program.  During  our   interviews  with  undergraduates  it  became  apparent  that  the  introductory  cognitive  science   course  was  a  particular  strength  of  the  program.  It  is  a  popular  course  with  large  enrollments   and  seems  to  be  serving  as  a  sort  of  gateway  to  discovery  of  the  CIS  major  as  an  option.  The  CIS   unit  also  has  a  number  of  events,  such  as  lecture  series  that  students  can  participate  in  outside   of  class.   The  CIS  program  emphasizes  technology,  computer  applications,  and  modeling  in  the  context  of   various  specialized  areas  of  study  such  as  linguistics,  decision  making  and  perception.  Compared   to  many  other  programs  where  cognitive  science  has  a  more  neuroscientific  focus,  CIS  does  not   carry  a  heavy  biology  requirement  or  option.  While  research  experiences  (e.g.,  in  a  lab)  are   emphasized  they  are  not  part  of  the  required  curriculum.   The  BA/BS  differentiation  allows  some  students  to  take  a  heavier  load  of  computer  science  and   math  courses.  This  is  thought  to  be  helpful  as  students  pursue  graduate  degrees  or  employment   in  the  private  sector.  However,  small  numbers  of  students  surveyed  in  the  CIS  evaluation   prohibit  drawing  strong  conclusions  as  to  the  ultimate  value  of  having  two  degrees.  Nonetheless,  

                                                                                                                                                                                                                                                                                                                                          approximately  half  of  the  majors  do  opt  for  the  BS  program  and  it  assumed  that  these  additional   classes  allow  them  to  pursue  their  interests  and  develop  strengths.  The  staff  and  faculty   describe  a  well  functioning  articulation  and  advising  system  that  is  responsive  to  student  needs.   The  advisors  maintain  good  communication  with  the  CIS  program.   Cognitive  science  has  been  very  effective  in  setting  up  assessment  tools  for  how  they  are   achieving  their  learning  objectives.  Students  seem  to  be  very  well  informed  about  what  faculty   want  them  to  learn,  and  are  performing  well  in  terms  of  student  benchmarks.     Recommendations   The  tenor  of  our  review  thus  far  has  been  quite  positive.  Indeed,  we  were  very  impressed  with   the  CIS  program.  What  follows  are  some  recommendations  that  might  be  considered  as  the   program  and  university  move  forward  in  the  years  to  come.   Level  of  the  Faculty,  Unit,  School  and  University   1.  Better  alumni  relations  and  mechanisms  for  tracking  would  help  CIS.  Currently  much  of  this  is   done  at  the  CIS  unit  level.  This  is  a  burden  for  the  faculty  who  do  not  have  the  resources  or   training  in  the  necessary  methods.  It  is  advisable  that  offices  elsewhere  on  campus  take  on  more   of  this  burden  in  a  coordinated  fashion,  and  share  organized  data  with  CIS  in  accordance  with   the  indices  CIS  deems  important.   2.  While  CIS  plans  for  modest  growth  of  the  major,  the  RC  often  heard  from  a  number  of  sources   about  current  and  future  complication  arising  from  space  needs,  especially  the  limited  number   of  large  classrooms.  In  addition,  there  is  a  serious  shortage  of  lab  space  looming  for  new  faculty   hires,  and  this  is  a  major  problem  for  recruitment  and  retention  of  staff.  The  RC  recommends   prompt  administrative  action  and  planning  on  how  to  deal  with  this  as  the  CIS  faculty  (and  the   campus  in  general)  seeks  to  grow  and  maintain  quality  instructors/researchers  who  can  provide   the  unique  student  research  experiences  currently  emphasized  by  CIS.   3.  CIS  has  no  staff  support  beyond  that  available  in  the  dean’s  office.  This  is  virtually  unheard  of   in  similar  departments  or  units  at  other  universities.  In  the  near  future,  a  staffing  formula  should   be  developed  where  much  of  the  day-­‐to-­‐day  operations  of  the  unit  are  handled  by  staff  more   closely  associated  and  devoted  to  the  unit.  This  will  allow  for  staff  to  develop  the  needed   expertise  to  more  efficiently  serve  CIS  and  help  alleviate  some  of  the  large  service  burdens  of   CIS  faculty.   4.  One  of  the  strengths  of  the  CIS  undergraduate  program  is  the  diversity  of  the  students.   Reinstating  the  McNair  scholars  program  would  be  an  ideal  way  to  advance  the  involvement  of   undergraduates  from  traditionally  underrepresented  groups  in  research.  Given  the  CIS  faculty’s   experience  with  and  dedication  to  undergraduate  research  mentorship,  the  McNair  program   and  CIS  would  be  natural  partners.   5.  Many  universities  with  strong  cognitive  science  majors  have  mechanisms  to  award  modest   stipends  in  support  of  student  research  over  the  summer.  Initiating  a  student  grant  program   would  recognize  and  reward  students  who  take  the  initiative  to  become  involved  with  cognitive   science  research  outside  of  the  classroom.  But  more  importantly,  it  would  enable  students  to   gain  valuable  research  experience  when  they  might  not  otherwise  be  able  to,  due  to  financial   constraints.  Beyond  the  clear  educational  value,  such  an  experience  would  also  be  an  asset  to   students  in  pursuing  academic  or  professional  careers.     Level  of  the  Students  and  Curriculum   1.  The  CIS  has  an  internal  program  evaluation  method  and  plan.  While  it  has  strengths,  we   assume  that  it  is  still  evolving.  As  the  program  aspires  to  be  nationally  recognized,  we  suggest  

                                                                                                                                                                                                                                                                                                                                          that  it  identify  several  high,  medium  and  low  peer  programs  and  include  comparative  data  in   their  evaluations.  This  can  be  used  for  both  undergraduate  and  graduate  program  concerns.   Common  benchmarks  of  success  can  be  so  identified  and  included  as  part  of  the  measurement   of  objectives  in  future  strategic  plans.   2.  Current  assessment  of  students’  computational  skills  is  inadequate.  Rather  than  relying  on  a   survey  of  students,  these  assessments  should  be  derived  from  student  performance  in   computational  courses  such  as  Professor  Spivey’s  course  on  complex  adaptive  systems,   Professor  Noelle’s  course  on  computational  cognitive  neuroscience,  and  Professor  Yoshimi’s   course  on  neural  networks.   3.  As  faculty  hires  allow,  consider  offering  more  “hands-­‐on”  courses  with  a  technical  component   such  as  data  science,  speech  recognition,  interface  design,  and  other  aspects  of  human   computer  interaction.   4.  Given  the  goals  CIS  has  with  regard  to  their  undergraduate  community  and  experiential   offerings,  the  unit  may  want  to  consider  formalizing  some  sort  of  “club”  for  students,  perhaps   akin  to  Psi  Chi.  This  may  help  CIS  in  its  laudable  undergraduate  endeavors,  and  it  could  be   supervised  by  a  graduate  student.   5.  Similarly,  there  are  consortiums  of  various  disciplines  across  CA  campuses  (e.g.,  Stanford,   UCs)  in  areas  like  developmental  psychology,  neuroscience/brain  science,  autism  research,  etc.   CIS  at  UC  Merced  may  find  a  place  there  as  well  or,  if  need  be,  help  develop  a  cognitive  science   consortium,  say,  across  UC  campuses.  This  would  provide  for  additional  collaborations  and   perhaps  student  experience  opportunities.   6.  CIS  should  be  part  of  the  discussion  at  the  university  level  as  it  considers  an  Honors   College/Program.  CIS  students  and  the  CIS  philosophy  seem  well  suited  to  this  endeavor.   7.  CIS  should  continue  to  explore  formalized  mechanisms  for  student  internships.  Given   Merced’s  location  and  the  CIS  focus  on  technology,  many  such  opportunities  may  exist  within  a   2-­‐3  hours  driving  radius.     Conclusion   The  opinion  of  this  RC  is  that  the  CIS  undergraduate  program  is  of  high  quality  and  moving  in  the   right  direction.  If  followed,  the  recommendations  above  may  facilitate  the  further  positive   development  of  this  program  as  it  assumes  its  place  as  one  of  the  best  in  the  nation.  This  report   is  respectively  submitted  by  the  review  committee,  June  1,  2013:     Toby  Mintz  (University  of  Southern  California)     Seana  Coulson  (University  of  California,  San  Diego)     Jeffrey  W.  Gilger  (University  of  California,  Merced)