Learning & Teaching Colloquium 2010
Focusing on Assessment Don’t forget to write down any concerns, questions and problems with assessment on the flipcharts
Please choose a Graduate Attribute you would like to discuss and go to the appropriate table Table 1: the skills involved in scholarly enquiry Table 2: an in-depth engagement with the relevant disciplinary knowledge in its interdisciplinary context Table 3: the capacity for analytical and critical thinking and for creative problem solving Table 4: the ability to engage in independent and reflective learning Table 5: the capacity for enterprise, initiative and creativity Table 6: an appreciation of, and respect for, diversity Table 7: a capacity to contribute to, and work within, the international community Table 8: the skills required for collaborative and multidisciplinary work Table 9: an appreciation of, and a responsiveness to, change Table 10: a respect for ethical practice and social responsibility
Learning & Teaching Colloquium 2010 Focusing on Assessment
Welcome ANN WILSON Project Manager, Learning & Teaching
Learning & Teaching Colloquium 2010 Focusing on Assessment
SPOTLIGHT ON FEEDBACK Shirley Scott FASS Director of Learning & Teaching
What do students understand by `feedback’? A clear majority think in broad terms: being able to gauge at each stage of a course how they are going in terms of the knowledge, understanding, and skills that will determine their result in the course Æ No nasty surprises
What do students want from feedback?: To use the feedback from one item of assessment to improve performance both in the remainder of that course and in future courses Æ Implications for course designers and program evaluations
Are Students Receiving the Feedback They Want? • Students reported a varied feedback experience ÆImplications for role of course coordinators and, potentially, for quality assurance
Learning & Teaching Colloquium 2010 Focusing on Assessment
Introduction Professor STEPHEN MARSHALL Director, Learning & Teaching
How important is Assessment? What are some of the problems and challenges associated with assessment? Pivot or Postscript!
248 X46 --------1448 992 ---------11368 --------
What grade out of 10 would you give a student for this? What feedback would you give to justify the grade?
9 A simple carrying error, otherwise it’s correct.
8 Simple error deduct 2 7 Three process & layout, errors deduct 1 mark each. 6 Should have known the process 5 Knows the process, but can’t reward carelessness 4 Three errors deduct 2 marks each. 3 Three processes correct 1 mark each 2 Careless work but partly correct 1 For trying 0 Wrong! No marks!
Quality of other student’s work, Assessors’ personality Assessors’ experiences and beliefs Knowledge of the student and classroom events Surface features Introductory paragraph, Presentation factors
Is a very human and fallible activity! Is a high stakes activity! Is ethically, technically and educationally complex
Assessment Colloquium: Orrell May 2010
20
(A)
It is a major driver in student learning
(B)
Focus of quality assurance
Teachers and students are concerned with (A)
Governments, Institutions, Administrators are concerned with (B)
For Student Learning Types of assessment used Frequency Feedback Quality Hidden curriculum & markers Emphasis on summative purposes
For Quality Assurance Grading is variable Plagiarism minimisation Standards Risk minimisation Moderation and objectivity Measurement functions
Due to: Lack of adequate feedback to warn students of potential failure Arbitrary number of events for skill development What was assessed was not taught (inconsistency) Tests inadequately devised (validity) Tests inadequately assessed (reliability) Tests (Content/process) unfairly discriminated against a group (consistency) Due process not followed (McManus, 1998 in Fry, Ketteridge and Marshall 1999, p.51)
Informs & Shapes Learning Tells students what you think is important › Relevance › Relatedness
Shapes students’ learning engagement and activities Defines students to themselves as learners › Effort › Ability › Persistence
Definitional Formative Summative
Assessment Colloquium: Orrell May 2010
25
1. Assessment 2. The
Purposes (Why assess?)
focus of Assessment (What is assessed?)
3. The
process and tasks of assessment (How to assess?)
4. How
to interpret assessment products? (How to grade?)
5. How
to respond to students, systems and society about the outcomes? (How to report (account and certify) and provide feedback?)
Traditions
Conventions
Wisdom of experience
Contemporary trends
Expedience
Scholarship
Academics design tasks, award grades and provide feedback that, They feel comfortable with, They believe the student will feel is fair, Can with stand robust scrutiny by colleagues. They make decisions that lead to preferred and expected outcomes using their experience to predict the best and worst outcomes!
9
Appropriate tasks that encourage learning
9
Clear descriptions of grading criteria & grade related performance standards
9
Timely feedback that linked to the assessment criteria
9
Decision-making guided by posted criteria and standards when grading
(adapted from Carroll 2004)
The
Course What kind of graduates or Graduate Capabilities does the course aspire to produce? What role or Learning Outcomes does this unit have in the overall course? The Task What learning outcomes does this task equip students to meet?
Knowledgeable disciplinary specialists? Ethical, efficient & trust worthy professionals? Collaborative, reliable colleagues and team players? Community & corporate citizens? Effective problem solvers? Creative & critical reasoning thinkers? Effective verbal & written communicators?
Culturally literate? Technologically literate?
Skills
Knowledge
Dispositions
Learning products & performance
Performance Context
Students require risk and ambiguity to learn Students require challenge to engage in their learning Students need support and feedback in their learning Students who are meta-cognitive are more likely to succeed
is knowing what you know and don’t know, how you learn and how to go about finding out what you don’t know. Its awareness of your own approach to learning and the current status of your learning.
Too Much assessment is taken up with surface learning and trying to ‘spin out a grade’. Students, academics and administrators require induction to assessment.
Assessment Colloquium: Orrell May 2010
35
Ideographic: “How does the performance or product measure against this student’s earlier performances or products?”
Pre-determined behavioural norms:
“How does the student performance or learning product
compare against established norms for this particular level of students?”
Pre-established criteria:
“Did the student performance or learning product demonstrate or address the criteria for which the task was established?”
Pre- determined standards:
“What level of performance and understanding does the student demonstrate in this task?
Cognitive
Ethico-moral Reasoning
Extended Abstract
Limited Commitment
Relational
Commitment
Relativism
Multiplicity
Black & White thinking
Multi-structural Uni-structural
Level
Biggs
Perry
Qualities
1.Developing
Uni-structural
Dualistic
Limited knowledge
2.Functional
Multi-structural Multiplistic
3.Proficient
Relational
Relational
4.Advanced
Extended Abstract
Limited Metacognitive commitment & abstract
Knowledge no structure
Analysis & Commitment application
Attributes •indicators
Unacceptable Acceptable
Limited and inaccurate knowledge of •Recognition of CCs and use of terminology or concepts translation into own words. •Use of terminology Knowledge of CCs
•Translation of terminology
CCs are known, but limited capacity to to translate into own language and use them to describe ideas
Proficient
Advanced
Can analyse and apply core concepts to novel contexts Can use and translate the terminology into own language
Knows, understands, analyses, applies & critiques CCs. Creative use of CCs to generate new learning
Attributes •indicators
Unacceptable
Acceptable Proficient
Advanced
Critical Thinking
Repeats the thinking of authorities
Black and Uses evidence white thinking to form validated by opinion authority. Recognises Fixed multiplicity of opinions that opinion and are not factors that supported by may have evidence contributed to the diversity of opinion
Forms creative and novel opinions based on evidence & principles.
•Using evidence •Forming opinions •Recognising multiplicity & diversity
eg books, teachers, peers etc,
Anecdotal thinking
Synthesis complex information Can evaluate and adjust opinions in the light of new evidence & hypothetico deductive reasoning
Hoop jumping activities and ritualized education Disaggregation of really useful learning activities from merely useful learning Surface and achievement oriented approaches to learning at the expense of deep learning and higher order thinking
Assessment Colloquium: Orrell May 2010
42
Core to the activities of learning and teaching Is the process by which students learn Graduate attribute development is integrated with disciplinary learning Minimizes opportunities for plagiarism and cheating
Assessment Colloquium: Orrell May 2010
43
Forensic Science (lab bench work skills) › students collect samples and test them
Health Science (critical consumers of research for evidence based practice) › Students search out research article,
summarise and apply to practice
Business (technical computations) › Case analysis including computations and
report
Assessment Colloquium: Orrell May 2010
44
Authentic Tasks: Assessment as Learning Eg. Library search (critical selection & team activity)
Forensic role play (investigation/problem
solving)
Observation (noticing)
Case presentations (theory/practice
application)
Debates and position statement (evidence-
Mastery learning (Skill development)
based, opinion formation)
Clinical and work place placements Field work Community analysis and report preparation Survey construction Archive research, analysis and report Team work group projects Creative outputs and performances Debates and opinion formation
Assessment Colloquium: Orrell May 2010
46
End of semester exams End of semester essays
There is no chance for the student to use the information to make improvements or changes. Postscript assessment serves administrative accountability purposes to produce a grade Assessment Colloquium: Orrell May 2010
47
NO! Mid term exams can be used to review and re- set learning and teaching agenda Essays can be part of a sequential process of development of ideas and dialogue between students and student and teacher.
Assessment Colloquium: Orrell May 2010
48
Tutorial participation Weekly tests and exercises Mid semester exams End of semester exam
Group work/essay/project
Tutorial participation/presentation 2000 essay or 2x 2000 essay End of semester exam
Example 1
Example 2
Assessment Colloquium: Orrell May 2010
49
Tutorial participation Worked examples Weekly tests and exercises On-line tests Mid semester exams Authentic activity End of semester exam Why?
Group work/essay/project
Tutorial participation/presentation Debate 2000 essay or 2x 2000 essay Authentic activity End of semester exam Why?
Example 2
Example 1
Assessment Colloquium: Orrell May 2010
50
ACADEMIC FREEDOM!
Learning & Teaching Colloquium 2010 Focusing on Assessment
Table Activity Part 1:
Discussion of Assessment of Graduate Attributes
Please choose an attribute you would like to discuss and go to the appropriate table Table 1: the skills involved in scholarly enquiry Table 2: an in-depth engagement with the relevant disciplinary knowledge in its interdisciplinary context Table 3: the capacity for analytical and critical thinking and for creative problem solving Table 4: the ability to engage in independent and reflective learning Table 5: the capacity for enterprise, initiative and creativity Table 6: an appreciation of, and respect for, diversity Table 7: a capacity to contribute to, and work within, the international community Table 8: the skills required for collaborative and multidisciplinary work Table 9: an appreciation of, and a responsiveness to, change Table 10: a respect for ethical practice and social responsibility
University College Dublin: Matching Learning Outcomes to Assessment Types Types of Learning: Learning outcomes Thinking critically and making judgments Solving problems/developing plans
What is required from students? Development of arguments, reflection, judgment, evaluation Identify problems, define problems, analyze data, review, design experiments, plan, apply information
Performing procedures and demonstrating techniques
Take readings, use equipment, follow laboratory procedures, follow protocols, carry out instructions
Demonstrating knowledge and understanding
Recall, describe, report, identify, recognise, recount, relate, etc.
Managing/developing yourself
Work co-operatively and, independently, be selfdirected, manage time, manage tasks
Designing, creating, performing
Design, create, perform, produce, etc.
Assessing and managing information
Information search and retrieval, investigate, interpret, review information
Communicating
Written, oral, visual and technical skills
Examples of Assessment Essay, Report, Book review Problem scenario, Group Work, Work-based problem, Analyze a case, Conference paper (or notes for a conference paper plus annotated bibliography) Demonstration, Role Play, Make a video (write script and produce/make a video), Produce a poster, Lab report Written examination, Oral examination, MCQs, Essays, Reports, Short answer questions, Mini tests Learning journal, Portfolio, Learning Contracts, Self-evaluation, Group projects, Peer assessment Design project, Portfolio, Presentation, Performance Annotated bibliographies, Use of bibliographic software, Library research assignment, Data based project Written presentation, Oral presentation, Discussions /Debates/ role plays, Group work
www.ucd.ie/teaching/resources/assessment/howdoyouassessstudentlearning/
Learning & Teaching Colloquium 2010 Focusing on Assessment
Online Formative Feedback Assessments: Benefits for Learning Gary Velan (
[email protected]) School of Medical Sciences, Faculty of Medicine, The University of New South Wales, Sydney 2052 Australia
Rationale for Formative Assessments • Assessment has powerful effects on what is learned, i.e. “the de facto curriculum” (Ramsden, 1992), as well as students’ approaches to learning (Entwistle, 1987) • Formative assessments that provide timely, relevant and supportive feedback (not just grades) contribute to improved learning outcomes (Gipps, 2005) • If assessment is intended to foster better learning outcomes, formative assessment could be considered the most important assessment practice (Black & Wiliam, 1998) • Most higher education programs fail to provide adequate feedback to students on their learning (Gibbs & Simpson, 2004)
Rationale for Online Formative Assessments •
Paper-based formative assessments have a number of limitations: – – –
•
invigilation may be required individualised feedback is time-consuming might not be feasible with large class sizes
Web-based formative assessments offer clear advantages for staff and students: – – – –
flexibility in timing and location of assessment feedback can contain links to learning resources opportunity for repetition and interactivity support equity and inclusiveness (Velan, Kumar, Dziegielewski, Wakefield, 2002)
UNSW Medicine Case Study: Online Formative Feedback Assessments • Integrated online formative assessments were developed in the biomedical sciences with automated individualised feedback • Embedded in each of the sequential 8-week courses in Phase 1 of the Medicine program • Questions based on familiar clinical scenarios, providing an authentic context for learning • Emphasis on curriculum goals of integration between biomedical sciences, as well as integration of biomedical with clinical, social and behavioural sciences (Velan, Jones , McNeil, Kumar 2008)
Process • The software tool used was Questionmark Perception™ (Questionmark, UK), “best of breed” for online assessment: – integrates with LMS – wizards for a wide variety of question types, e.g. drag-and-drop – capacity to add high-quality graphics, audio or other multimedia to questions and/or feedback – enables adaptive testing – powerful, customisable reporting capabilities
• Mix of short answer questions and objective items used, similar to end of course (EOC) summative examinations • Students provided with individualised automated feedback • Students able to repeat each assessment as desired • Continual improvement cycle: item analysis; student input
Measures of Impact (1) • Over two consecutive years, in all courses: – EOC marks for those students who completed the online formative assessment were significantly higher than for those who did not – The highest online assessment score for each student was significantly correlated with their EOC mark
Measures of Impact (2) • Online formative feedback assessments have high rates of participation (>75%) and repetition (>40%) • In online evaluation surveys (2009 figures shown), students agreed that these assessments: – – – – – –
were challenging (94%) assisted learning in biomedical sciences (92%) provided useful feedback on learning (90%) helped guide study (91%) were enjoyable (65%) were valuable overall (96%)
• Statistically significant improvements in student evaluations
Student Evaluation (3) • Open-ended comments by students: – “Provides an opportunity to correct misconceptions and actually learn/relearn concepts.” – “Integration of a variety of concepts learnt throughout the course”. – “Gives good insight into students’ current performance and areas of potential improvement which can be addressed for the actual exam.” – “Only real form of feedback through out the whole course.” – “Feedback is tailored to which response you gave (if it was wrong), and given in detail. Labelling of diagrams (with pins etc) is fun.” – “I love the formative assessments. They are the best part of the course and a really good indicator of how I am going to go in the EOC exam.”
Discussion • The data indicate that these assessments: – had a measurable beneficial effect on student learning – were perceived positively (and increasingly so) by students – provoked further thought and study by students
• Compared with previously published studies, our online formative feedback assessments differed as follows: – – – – –
systematic approach to development and continual improvement integrated across disciplines broader in scope available for a longer period of time in each course embedded throughout a program of study
• Implications for course and program design
Acknowledgements • Prof Adrian Lee • Prof Patrick McNeil • Prof Philip Jones • Prof Rakesh Kumar • Dr Alex Yueping Wang
References •
Black P, Wiliam D. Assessment and classroom learning. Assessment Educ 1998;5:7-74.
•
Entwistle N. Styles of Learning and Teaching. 2nd ed. Chichester: Wiley; 1987.
•
Gibbs G, Simpson C. Conditions under which assessment supports students’ learning. Learn Teach Higher Educ 2004;1:3-31.
•
Gipps CV. What is the role for ICT-based assessment in universities? Stud High Educ 2005;30:171-180.
•
Ramsden P. Learning to Teach in Higher Education. London: Routledge; 1992.
•
Velan GM, Kumar RK, Dziegielewski M, Wakefield D. Web-based assessments in pathology with QuestionMark Perception. Pathology 2002;34:282-284.
•
Velan GM, Jones P, McNeil HP, Kumar RK . Integrated online formative assessments in the biomedical sciences for medical students: benefits for learning. BMC Med Ed 2008; 8:52. http://www.biomedcentral.com/1472-6920/8/52
Learning & Teaching Colloquium 2010 Focusing on Assessment
Assessing through Role Play Fuel for the Future
Elizabeth Rosser UNSW Foundation Year
[email protected]
Blended online & F2F Role Play 4 week blended activity – Wiki – Face‐to‐face team meetings & emergency summit – Reflection
Contributes 25% to total course assessment: http://fuelfuture.unsw.wikispaces.net/
– Team – Indiv Debriefing Report
10% 15%
Role play assessment principles ¾Structure tasks to align with desired learning outcomes ¾Provide clear expectations to guide students ¾Make tasks as authentic as possible ¾Sequence tasks to advance purposes & processes of role play ¾Avoid over‐assessing
Learning & Teaching Colloquium 2010 Focusing on Assessment
Table Activity Part 2: Discussion & Convergence What are the resources, policies and other elements needed to support assessment of attributes?
Learning & Teaching Colloquium 2010 Focusing on Assessment
Wrap-Up & Close Conjoint Associate Professor JAN ORRELL, Education Consultant
Thank you Assessment resources: www.learningandteaching.unsw.edu.2010colloquium
