Implementing Continuous Temperature Monitoring: Trips, Tips, and Lessons Learned from Coast to Coast

Centers for Disease Control and Prevention National Center for Immunization and Respiratory Diseases

Overview of PPHF 2012 Cooperative Agreement to Improve Vaccine Storage & Handling at the Provider and Awardee Level Lessons Learned and Learning Lessons John Stevenson CDC/NCIRD/Immunization Services Division National Immunization Conference September 29, 2014

Centers for Disease Control and Prevention National Center for Immunization and Respiratory Diseases

22 AWARDEES, ~ $13M

PROJECT PERIOD: 10/1/2012 - 9/31/2014 MOST AWARDEES HAVE RECEIVED AN EXTENSION THROUGH 9/30/2015

Objectives of Cooperative Agreement To improve vaccine storage and handling at provider and grantee level by improving  monitoring of refrigerator and freezer temperatures  management approaches  vaccine storage and handling infrastructure

Recommendations on temperature monitoring 

Exploit new technology for temperature monitoring  From • Twice daily measurement of temperature using liquid filled thermometers

 To: • • • •

Continuous temperature monitoring Digital data loggers (DDLs) Probe in glycol-filled vial Optional alarm features: o o o

Light Sound Real time alerts via text, phone, email

Mid-course progress and findings 

 





Most awardees have added to their temperature logs fields to record daily min/max temperatures Most awardees have completed a pilot test using DDLs Several awardees have rolled out DDLs to almost all VFC providers Significant lessons learned since issuance of CDC interim guidance Differences found between loggers  IT requirements, effort required in setup, training and support  Three loggers are most popular with awardees  One logger has clear advantages ease of use

Mid-course progress and findings 

Monitoring data from DDLs is a significant challenge for awardees  Data management, analytic support needed  Need to develop efficient, sustainable system  Some promising development integrating into IIS



Several awardees are adding to their IIS, capability to document daily temperatures, ideally to include min/max

Late Course Lessons Learned 

Our traditional evaluation approach may be too limiting



Develop a model program:       

linear theory of change Create logic model Develop pilot program Conduct formative evaluation Formalize the program Conduct summative evaluation to confirm best practices Take to the program to scale

Late Course Lessons Learned 







We can benefit by applying concepts from developmental evaluation Identify proven principles for adapting strategies to local context, complex systems, and changing environment Support and document development of new strategies and changes to systems Focus on  What is getting developed and  What are the implications of what gets developed

Developmental Evaluation Defined 



Purpose: Developmental evaluation (DE) informs and supports innovative and adaptive development in complex dynamic environments. DE brings to innovation and adaptation the processes of  asking evaluative questions,  applying evaluation logic,  and gathering and reporting evaluative data

to support project, program, product, and/or organizational development with timely feedback

Developmental Evaluation Designed for Situations Involving      

High innovation Development High uncertainty Dynamic Emergent Systems Change

Principles      



Provide direction but not detailed prescription Inform choices at forks in the road Context and situation sensitive Require judgment Are the only way to navigate in complex dynamic systems Point to outcomes and impacts Can be evaluated for both process (implementation) -- and results

Key observations  

Continuous temperature monitoring is vital Traditional twice daily checks of current temperature are not sufficient  for monitoring excursions,  understanding storage unit performance,  supporting key decisions at provider level

Identifying Principles to Apply Moving Forward 





Exploit each temperature excursion as opportunity to learn and to improve storage and handling system Focus on how providers are using the data from continuous temperature monitoring Categorize providers: properly using continuous temperature monitoring. Yes/No  Documenting daily min/max  Taking appropriate action

Identifying Principles to Apply Moving Forward 

Categorize storage units:   

 

 

Good – minimize awardee effort Bad – target intervention, follow-up Ugly - replace

Build, discard (e.g., multiple surveys), re-try approach Describe the risk of temperature excursions among VFC providers Document what you know and what you don’t know Work to fill in gaps in what you don’t know

If you are feeling uncertain, you are not alone 

“No battle plan ever survives contact with the enemy.” Field Marshall Helmuth Carl Bernard von Moltke



“Everyone has a plan…until he gets hit.” Former World Heavyweight boxing champion, Mike Tyson



 

Tom Peters (1996) Liberation Management : “READY. FIRE. AIM.”

Let’s use concepts and methods from developmental evaluation to 



Informs and support innovative and adaptive development in complex dynamic environments. Brings to innovation and adaptation the processes of  asking evaluative questions,  applying evaluation logic,  and gathering and reporting evaluative data

to support project, program, product, and/or organizational development with timely feedback

Acknowledgements CDC PPHF Storage and Handling Team Kayla Calhoun Samantha Dolan Marie Hicks Maureen Kolasa Ebony Wardlaw

Questions and Discussion