National Institute for Health Research

NIHR CLAHRC for South Yorkshire

Challenges of measuring CLAHRC-SY

Steven Ariss Evaluation team lead Jo Cooke Programme manager © Copyright, NIHR CLAHRC for South Yorkshire, 2010

NIHR CLAHRC for South Yorkshire

Realistic Evaluation

National Institute for Health Research

(Pawson & Tilley, 1997)

• Developed in the tradition of Popper & Campbell: taking account of context • Experimental/quasi experimental: ‘Does it work?’ • Realistic evaluation: ‘What works, for whom, in what circumstances?’ • Sensitive to the context • Outcomes are influenced by more than a specific intervention © Copyright, NIHR CLAHRC for South Yorkshire, 2010

NIHR CLAHRC for South Yorkshire

Social Reform

National Institute for Health Research

• Totalitarian utopianism (holist) • • • •

Large scale Potential for big mistakes Insensitive to individual interests Limited recognition of errors and unintended effects

• Piecemeal social engineering (realist) • • • •

Small scale (pilots) Cautious/Gradual reform Sensitive to individual interests Self critical

© Copyright, NIHR CLAHRC for South Yorkshire, 2010

(e.g. Freeman, 1975)

National Institute for Health Research

sio n

St ro ke Di ab ete s

Knowledge into action

Intelligent commissioning

Achieving translation © Copyright, NIHR CLAHRC for South Yorkshire, 2010

Inequalities in health

cs

User-centred Healthcare Design

ti ne

Self-management and self-care of long term conditions

Ge

De pr es

f no tio es ca gi pli olo Ap hn tec

re ca T) nd aC h a (T alt es he gi le- olo Te chn Te

C co hro n d ni itio c CO PD ns

Ob es ity

NIHR CLAHRC for South Yorkshire

11 CLAHRC-SY Themes

Evaluation Indicators

National Institute for Health Research

NIHR CLAHRC for South Yorkshire

Example from Co-Production, Networks and Collaboration Theme: Context: (e.g. Principles, Standards, Environment)

Standard #2: coproduction in projects from beginning to the end will influence knowledge use and impact

Mechanisms Outcomes / Processes

Impact

Collaborative priority setting:

Collaborative Project Activity:

Who were the partners and how were they involved in developing research ideas?

Who had active involvement in projects?

How does the level of collaborative involvement link to intended / actual impact?

TIME © Copyright, NIHR CLAHRC for South Yorkshire, 2010

NIHR CLAHRC for South Yorkshire

Interpreting the Evidence

National Institute for Health Research

• Taking account of the social context • “Misleading assumptions can…be corrected by the use of several sources of information or multiple indicators” (Mulkay, 1991) • Information is produced in specific social contexts for particular purposes • “need…to assess, or interpret these kinds of information quite differently” (Mulkay, 1991) © Copyright, NIHR CLAHRC for South Yorkshire, 2010

NIHR CLAHRC for South Yorkshire

Process Utopia? (Campbell & Russo, 1999)

• Cautious, self-critical social reform • Sensitive to individual interests • Aware of complex influences over outcomes

© Copyright, NIHR CLAHRC for South Yorkshire, 2010

National Institute for Health Research

NIHR CLAHRC for South Yorkshire

Role of the Evaluation

National Institute for Health Research

• Interpreting evidence of knowledge use in complex social environments • ‘Critical friend’ • Feedback • Utilization Focused Evaluation (M. Patton, 2008)

© Copyright, NIHR CLAHRC for South Yorkshire, 2010

NIHR CLAHRC for South Yorkshire

National Institute for Health Research

NIHR CLAHRC for South Yorkshire

National Institute for Health Research

NIHR CLAHRC for South Yorkshire

Challenges of Diversity

National Institute for Health Research

• Diverse collaboration of projects and themes • Made us ask questions • What is the bed-rock of the collaboration? • What is the glue that keeps us together? • What makes it more than the sum of parts?

© Copyright, NIHR CLAHRC for South Yorkshire, 2010

NIHR CLAHRC for South Yorkshire

Meeting the challenge of diversity

National Institute for Health Research

• Self management of long term conditions • Map impacts of CLAHRC projects – Patients and people – services

• Development of principles about the way we work • Co-production • Research Capacity Building • Addressing inequalities

• Cross theme working © Copyright, NIHR CLAHRC for South Yorkshire, 2010

NIHR CLAHRC for South Yorkshire

Challenge of organisational turbulence and change

National Institute for Health Research

NIHR CLAHRC for South Yorkshire

Meeting the challenge

National Institute for Health Research

• Need to capture change in real time to illustrate ability ( or not) to be responsive • Tools need to make transparent change over time • Data collection methods • Person Gantt • Match funding-over match and under-match and why this might be • Demonstrate responsiveness through qualitative work

© Copyright, NIHR CLAHRC for South Yorkshire, 2010

NIHR CLAHRC for South Yorkshire

The challenge of capturing what bridges the researchpractice gap

National Institute for Health Research

NIHR CLAHRC for South Yorkshire

Some surprises

National Institute for Health Research

• Wide range of project types: planned and response mode • Nature of knowledge: • policy, practice, service user, research, organisational

• Iterations of knowledge mapping framework • Moved to a less prescriptive data collection tool that capture knowledge use for all types of project

© Copyright, NIHR CLAHRC for South Yorkshire, 2010

NIHR CLAHRC for South Yorkshire

Meeting the challenges

National Institute for Health Research