University of Alberta

Course Guide for SPH 631 Winter 631 (Campus)

Health Program Evaluation Thursdays 0900-1150 hr ECHA- 1 163

Instructor:

Dr. Zubia Mumtaz Phone: 780 492 7709 e-mail: [email protected] Office: 3309 ECHA

Course description

This course introduces the concept of program evaluation, its importance and use as a management, planning and policy formulation tool. It provides students with a theoretical and practical understanding of the different types of evaluations needs assessment, program theory, process evaluation, monitoring of outputs and outcomes, impact assessment, and cost analysis. A wide range of quantitative and qualitative methods including experimental and quasi-experimental study designs will be presented. Discussions will centre on the social and political dimensions of program evaluation with examples from industrialized and low- and middle-income countries (LMIC’s).

Objectives

Upon completion of this course, students will be able to: 1. Have an understanding of the importance and use of health program evaluations. 2. Describe the key aspects and types of program evaluation. 3. Understand the principles underlying the selection of appropriate strategies, methods and data sources necessary for evaluating a health program. 4. Be able to carry out a program evaluation designed to guide management, planning and policy decisions.

Competencies

SPH 631 2014 Course Syllabus

 Describe key concepts in monitoring and evaluation  Discuss and differentiate program evaluation, policy evaluation, evaluation research  Explain/create a program evaluation plan  Initiate and guide evaluation activities, in response to specific community, health program and management needs  Demonstrate ability to match evaluation questions with appropriate methodologies and approaches  Describe/explain the range of program evaluation methods that generate knowledge to inform program policy development, along with strengths and limitations and appropriate use of each.  Identify major sources of existing health data (e.g., local, provincial/state, national and international sources of health data).  Assess relevance and quality of evidence from a range of sources and apply to a specific public health, or management issue 2

University of Alberta, School of Public Health Page

 Identify and communicate potential and /or common pitfalls in using various research methods in program evaluation

Library

The University of Alberta library system’s website www.library.ualberta.ca details the range of services offered to students on and off campus. If you need further information or assistance, contact the Library's Electronic Reference Desk at www.library.ualberta.ca/ereference/index.cfm or call 1-800-2070172.

SPH 631 2014 Course Syllabus

3

University of Alberta, School of Public Health Page

Course Evaluation Mid-Point Course Evaluation:

Final Course Evaluation:

As is the case with all SPH instructors, we are interested in improving the course. We need to hear from you in order to do that well, so will gather feedback and recommendations from you at the mid-point of the course, by asking a student to administer and deliver an anonymous course evaluation.

Following completion of the course, you will receive a standardized summative evaluation. The standard University of Alberta course evaluation procedure will be followed. This will involve a standard questionnaire with anonymous responses returned to the Student Services Coordinator.

Student Evaluation Assignment Mid-term Examination End of term examination A Program Evaluation Proposal Classroom Participation

Percentage 20% 20%

Date 9.00 am, 12th February 2014. 9.00 am, 9th April, 2014

50%

4.00 pm, 9th April, 2014,

10%

Whole term

Evaluation of Course Program Evaluation Proposal Work This assignment aims to develop your skills in program evaluation, specifically how to develop an evaluation proposal. It is designed to test your understanding of issues related to evaluation questions, types of evaluation, study design, stakeholders and to encourage you to think about problems arising in a program evaluation in a practical setting. This will be a group project. You will work in teams of four or five students. You will select a health care program, or a health promotion program, either a domestic program or an international program and develop a proposal for its evaluation. The assignment is divided into four parts that should parallel your understanding of ideas and issues as presented and discussed in class 1. 2. 3. 4.

Part Part Part Part

I: Define the problem and describe the program II: Decide the type of evaluation III: Methods used IV: How will the evaluation results be used?

The project mark is a group mark. Each team member will present a SPH 631 2014 Course Syllabus

4

University of Alberta, School of Public Health Page

part of the proposal (15%) and the each group will submit one writeup of the proposal (35%). The distribution of the presentation marks are as follows: out of a total of 15%, 5% will be a group mark and 10% will be based on individual presentation skills as per criteria laid out in the presentation score sheet attached. The distribution of the written proposal are as follows: the total of 35% will be distributed in four parts: Part 1- 10%, Part 2 - 10%, Part 3 - 10% and Part 4 - 5%. Class room participation Students will be expected to participate in classroom discussions supplemented by background or assigned readings. Some classroom participation may be done individually or in small groups. Students will be expected to apply the relevant reading material to the topic and questions at hand. Quality is preferred over the quantity of contribution, and participation should facilitate group, as well as individual learning. Participation grades will be assigned based on an assessment of your preparation and contributions, as well as attendance in class. 10/10 •Attends regularly •Comes prepared, having read the assigned readings. •Comes with highlighted readings or written responses to the discussion questions to be shared with classmates but not handed in to the instructors. •May have done additional reading on the subject (optional readings or identified own sources of information), or makes an effort to relate the required reading to previous learning or work experience. •Contributes relevant comments, or asks questions during class. •Does not dominate whole class or small group discussions. Allows others to participate in the discussions. •Volunteers to act as spokesperson for small group feedback.

Grading

See ‘Assignments and Marking Criteria’.

University of Alberta Calendar Section 23.4 Regulations and Information for Students Evaluation Procedures and Grading System The University of Alberta Grading System The University of Alberta uses a letter grading system with a four-point scale of numerical equivalents for calculating grade point averages. Grades reflect judgments of student achievement made by instructors. These judgments are SPH 631 2014 Course Syllabus

5

University of Alberta, School of Public Health Page

based on a combination of absolute achievement and relative performance in a class. Some instructors assign grades as intervals during the course and others assign marks (e.g. percentages) throughout the term and then assign a letter grade at the end. Instructors must adapt their approaches to reflect the letter grading system. Grade distribution should reflect those shown in this document. (EXEC 03 FEB 2003) Descriptor Excellent

Good Satisfactory Failure

Academic Integrity

SPH 631 2014 Course Syllabus

Grading in Graduate Courses Letter Grade Grade A+ A AB+ B BC+ C CD+ D F

Point Value 4.0 4.0 3.7 3.3 3.0 2.7 2.3 2.0 1.7 1.3 1.0 0.0

Plagiarism is a serious offence. The University of Alberta is committed to the highest standards of academic integrity and honesty. Students are expected to be familiar with these standards regarding academic honesty and to uphold the policies of the University in this respect. Students are particularly urged to familiarize themselves with the provisions of the Code of Student Behaviour (online at www.ualberta.ca/CodeofStudentBehaviour) and avoid any behaviour which could potentially result in suspicions of cheating, plagiarism, misrepresentation of facts and/or participation in an offence. Academic dishonesty is a serious offence and can result in suspension or expulsion from the University. University of Alberta policy about course outline can be found in Section 23.4(2) of the University Calendar. (GFC 29 SEP 2003)

6

University of Alberta, School of Public Health Page

Schedule at a Glance Date/2014 Jan 8 Jan 15 Jan 22 Jan 29 Feb 5 Feb 12 Feb 19 Feb 26 Mar 2 Mar 12 Mar 19 Mar 26 Apr 2 Apr 9

Topic Foundations of Health Program Evaluation Types of Program Evaluation Assessing the need for a Program Program Theory Process Evaluation Midterm Exam Reading week Impact Evaluation Detecting, interpreting and analyzing program effects Economic Evaluation of Health programs Policy implications of Evaluation findings Making a difference: Ethics in Program evaluation Group Presentation of Program Evaluation proposals End of term exam and Final Program Evaluation proposals due – at 4.00 pm

SPH 631 2014 Course Syllabus

Instructor Zubia Mumtaz Zubia Mumtaz Zubia Mumtaz Zubia Mumtaz Zubia Mumtaz Zubia Mumtaz Zubia Mumtaz Zubia Mumtaz Arto Ohinmaa Zubia Mumtaz Bob McKim Zubia Mumtaz Zubia Mumtaz

7

University of Alberta, School of Public Health Page

DETAILED COURSE OUTLINE BY WEEK WEEK 1 (January 8) – Introduction to PHS 631

Instructor: Zubia Mumtaz

Learning Objectives Overall Course objectives 1. Have an understanding of the importance and use of health program evaluations. 2. Describe the key aspects and types of program evaluation. 3. Understand the principles underlying the selection of appropriate strategies, methods and data sources necessary for evaluating a health program. Required Readings 1. Rossi, Peter, Lipsey, M. W. and Freeman, H. E. (2007) Evaluation: A systematic approach (8th edition). Newbury Park, CA: SAGE Publications, Chapter 1. 2. David Ogilvie, Steven Cummins, Mark Petticrew, Martin White, Andy Jones, and Kathryn Wheeler (2011). Assessing the Evaluability of complex Public Health interventions: five questions for Researchers, funders, and policymakers. The Milbank Quarterly, vol. 89, no. 2, 2011 (pp. 206–225). External Speaker: Courtney Petruik and Maki Ueyama, Evaluation Services, Alberta Health Services.

WEEK 2 (January 15) Types of Program Evaluation

Instructor: Zubia Mumtaz

Learning Objectives 1. To become familiar with a. Types of stakeholders; identification of stakeholder need. b. Nature of the evaluator-stakeholder relationship c. Politics of evaluation. 2. Understand types of program evaluation - needs assessment, program theory, assessment of program implementation, outcomes and impact, interpreting and analyzing program effects, measuring efficiency. 3. Identifying issues when formulating evaluation questions Required Readings 1. Abma, T.A. (2000). Stakeholder conflict: A case study. Evaluation and Program Planning. Vol 23: 199-210.

SPH 631 2014 Course Syllabus

8

University of Alberta, School of Public Health Page

2. Vivianne E. Baur, Arnold H.G. Van Elteren, Christi J. Nierse, Tineke Abma (2010) Dealing with Distrust and Power Dynamics: Asymmetric Relations among Stakeholders in Responsive Evaluation 16(3) 233–248. 3. Dennis M. Gorman and Eugenia Conde (2007) Conflict of interest in the evaluation and dissemination of “model” school-based drug and violence prevention programs. Evaluation and Program Planning 30: 422-429. 4. Vivianne E. Baur, Tineke A. Abma, Guy A.M. Widdershoven (2010) Participation of marginalized groups in evaluation: Mission impossible? Evaluation and Program Planning 33: 238–245. 5. John M. Bryson, Michael Quinn Patton, Ruth A. Bowman (2010) Working with evaluation stakeholders: A rationale, step-wise approach and toolkit Evaluation and Program Planning 34 (2011) 1–12. Movie: The Silence Within Week 3 (January 22) Assessing the need for a program

Instructor: Zubia Mumtaz

Learning Objectives   

Gain a better understanding of the ways of defining the problem Become familiar with different sources of data Learn to validate and prioritize identified needs

External speaker: Birgitta Larsson, Principal, BIM Larsson& Associates Required Reading 1. Lockwood A, Marshall M (1999) Can a standardized needs assessment be used to improve the care of people with severe mental disorders? A pilot study of 'needs feedback' Journal of Advanced Nursing 30 (6): 1408-1415. 2. Muela S., Ribera J. and Tanner M (1998) Fake malaria and hidden parasites-the ambiguity of malaria. Anthropology and Medicine 5(1)43-61.

Week 4 (January 29) Program theory

Instructor: Zubia Mumtaz

Learning Objectives 1. Be able to express program theory- what it is and what it does. 2. Develop and use logic models 3. Assess program theory in relation to needs, its logic and plausibility.

SPH 631 2014 Course Syllabus

9

University of Alberta, School of Public Health Page

Required Reading 1. Chris L. S. Coryn, Lindsay A. Noakes, Carl D. Westine and Daniela C. Schröter (2011) A Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009. American Journal of Evaluation 2011 32: 199. 2. Harting J and Van Assema P (2010) Exploring the conceptualization of program theories in Dutch community programs: a multiple case study. Health Promotion International, Vol. 26 No. 1 Doi:10.1093/heapro/daq045 3. Gugiu C.P and Rodriguez_Campos L (2007) Semi structured interview protocol for constructing logic models. Evaluation and Program Planning 30: 339-350. 4. Kaplan S. and Garrett K. (2005). The use of logic models by community-based initiatives. Evaluation and Program Planning 28:167-172. External Speaker: Renee Martin and Vanessa Gibbons Reid, Evaluation Services, Alberta Health Services. Week 5 (February 5) Process evaluation

Instructor: Zubia Mumtaz

Learning Objectives 1. Understand the principles of process evaluation 2. Understand the different methods and data sources that enable a process evaluation.

Required Reading 1. Andre Maiorana, Susan Kegeles, Percy Fernandez, Ximena Salazar, Carlos Cáceres, Clara Sandoval, Ana María Rosasco and Thomas Coates (2007). Implementation and evaluation of an HIV/STD intervention in Peru. Evaluation and Program Planning 30: 82-93 2. Ruth P. Saunders, Dianne Ward, Gwen M. Felton, Marsha Dowda and Russell R. Pate (2006) Examining the link between program implementation and behavior outcomes in the lifestyle education for activity program (LEAP). Evaluation and Program Planning 29: 352-364 External Presenter: Shelby Corley, Evaluation Services, Alberta Health Services Week 6 – February 12 Mid term exam Week 7 – February 19 Reading week

Week 8 (Feb 26) Impact evaluation

SPH 631 2014 Course Syllabus

Instructor: Zubia Mumtaz

10

University of Alberta, School of Public Health Page

Learning Objectives 1. Understand key concepts in impact assessment 2. Appreciate the importance, but the simultaneous limitations of randomized field experiments in assessing program impacts. Required Reading 1. Bond L, Patton G, Glover S, Carlin John, Butker H, Thomas L and Bowes G (2004). The Gatehouse Project: Can a multilevel school intervention affect emotional well-being and health risk behaviors? Journal of Community Epidemiology 58:997-1003. 2. Patton G., Glover S., Bond L., Butler H., Godfrey C., Di Pietro G. and Bowes G. (2000) The Gatehouse Project: a systematic approach to mental health promotion in secondary schools. Australia and New Zealand Journal of Psychiatry 34:586-593. 3. Victora, CG, J-P Habicht & J Bryce (2004) Evidence-based public health: moving beyond randomized trials. American Journal of Public Health 94(3): 400-405. 4. Smith, GCS & JP Pell (2003) Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomized control trails. BMJ 327:1459-61. Exercise: El Arifeen S, Blum LS, Hoque DME, et al. (2004). Integrated management of Childhood illness (IMCI) in Bangladesh: early findings from a cluster-randomised study. LANCET 364: (9445): 1595-1602

Week 9 (March 5) Detecting, interpreting and analyzing program effects Instructor: Zubia Mumtaz Learning Objectives   

Become familiar with quasi-experimental impact assessment methods Gain an understanding of the limitations of the quasi-experimental study designs Develop a critical understating of the bases and errors when detecting, interpreting and analyzing program effects

Required Reading 1. Tones K (1997). Beyond the randomized controlled trial: a case for ‘judicial review’. Health Education Research, 12:1-4. 2. Habicht, J-P, CG Victora & JP Vaughan (1999) Evaluation designs for adequacy, plausibility and probability of public health program performance and impact. International Journal of Epidemiology 28(10-18). 3. Audrey E Pettifor, Catherine MacPhail, Stefano Bertozzi and Helen V Rees (2007) Challenge of evaluating a national HIV prevention program: the case of loveLife, South Africa. Sexually Transmitted Infections 2007;83:i70-i74. Published online 7th February 2007 at http://sti.bmj.com.login.ezproxy.library.ualberta.ca/cgi/content/full/83/suppl_1/i70 Debate: Randomized controlled trails vs. Quasi-experimental impact assessment methods. SPH 631 2014 Course Syllabus

11

University of Alberta, School of Public Health Page

Week 10 (March 12) Economic evaluation of health programs Instructor: Dr Arto Ohinmaa) Learning Objectives   

Become familiar with the key concepts in efficiency analysis Be able to conducting a program cost-benefit analysis Appreciate the limitations of conducting cost-effective analysis in health program evaluations

Required Reading 1. Wasem J. (2007) Evaluation of prevention: A challenge for economists. International Journal of Public Health 52:337-338 2. Palmer, S., Byford S., and Raftery J (1999) Economic Notes: Types of economic evaluation. British Medical Journal 318:1349 3.Schwappach D. (2007) The economic evaluation of prevention - Lets talk about values and the case of discounting. International Journal of Public Health 52:335-336 4. Shiell A. (2007) In search of social value. International Journal of Public Health 52:333-334 5. Raftery J (2000) Economics notes: costing in economic evaluation. British Medical Journal. 320:1597.

Week 11 (March 19) Policy implications

Instructor: Zubia Mumtaz

Learning Objectives  

Understand the importance of dissemination and diffusion activities Develop a critical understanding of the issues of policy significance, o Of political time and evaluation time o Of policy space

Required Reading 1. Petticrew, M. et al (2004) Evidence for public health policy on inequalities: 1: The reality according to policy makers. Journal of Epidemiology and Community Health, Oct 2004; 58: 811 - 816. 2. Whitehead, M et al (2004) Evidence for public health policy on inequalities: Assembling the evidence jigsaw. Journal of Epidemiology and Community Health, Oct 2004; 58: 817 - 821.

SPH 631 2014 Course Syllabus

12

University of Alberta, School of Public Health Page

External Speaker: Bob McKim, Consultant

Week 12 (March 26) Making a difference: ethics in program evaluation Instructor: Bob McKim Learning Objectives   

Gain an understanding of the importance of ethics in program evaluation Develop a critical understanding of value judgments Be able to apply the principle of ‘doing ethics’: Trustworthiness, justice, moral pluralism and cultural dialogue.

Required Reading 1. Newman D. and Brown R (1996) Applied ethics for program evaluation. Sage Publication Thousand Oaks , London New Delhi. Chapter 1. 2. Thomas A. Schwandt (2007) Expanding the conversation on evaluation ethics. Evaluation and Program Planning 30: 400-403

SPH 631 2014 Course Syllabus

13

University of Alberta, School of Public Health Page

SPH 631 Health Program Evaluation Mumtaz W2015.pdf

There was a problem previewing this document. Retrying... Download. Connect more apps... Try one of the apps below to open or edit this item. SPH 631 Health ...

530KB Sizes 4 Downloads 169 Views

Recommend Documents

SPH 631 Health Program Evaluation (W2017).pdf
www.library.ualberta.ca/ereference/index.cfm or call 1-800-207-. 0172. Page 3 of 14. SPH 631 Health Program Evaluation (W2017).pdf. SPH 631 Health ...

SPH 671 Economic Evaluation of Health Care Nguyen W2015.pdf ...
Whoops! There was a problem loading more pages. Retrying... SPH 671 Economic Evaluation of Health Care Nguyen W2015.pdf. SPH 671 Economic ...

SPH 504 Health Promotion Planning and Evaluation McCaffrey ...
Laurie McCaffrey, M.Sc. Phone: (780) ... Patton, M.Q. (2012). Essentials of ... SPH 504 Health Promotion Planning and Evaluation McCaffrey F2015.pdf. SPH 504 ...

SPH 504 Health Promotion Planning & Evaluation Farish W2016.pdf ...
Page 3 of 20. SPH 504 Health Promotion Planning & Evaluation Farish W2016.pdf. SPH 504 Health Promotion Planning & Evaluation Farish W2016.pdf. Open.

PDF Health Program Planning And Evaluation: A ...
and practical tools and concepts, this outstanding ... community health with updated examples and references throughout. New ... and monitoring • A discussion ...