Introduction

Consensus Crash Course

Optimistically Terminating Consensus designing reusable low-latency agreement protocols

Piotr Zieli´ nski Cavendish Laboratory University of Cambridge United Kingdom

6 July 2006

Piotr Zieli´ nski

Optimistically Terminating Consensus

Unifying framework

Introduction

Consensus Crash Course

Unifying framework

Agreement problems

Agreement on whether a transaction succeded or not (Atomic Commit) which client’s request arrived first (State Machine Replication) which server is the master (Leader Election) agreement problems are common but difficult because of failures Piotr Zieli´ nski

Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying framework

Agreement problems A A

B

C

B C

System assumptions message passing: communication by messages process failures: servers can crash

A

message loss: messages can get lost asynchrony: no time bounds for messages, no clocks Piotr Zieli´ nski

Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying framework

Consensus A B C

1 2

1

Consensus

1

2

1

propose

decide

Consensus Processes propose values and make decisions validity: decision is one of the proposals agreement: all decisions are the same termination: all correct processes decide

Piotr Zieli´ nski

Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying framework

Consensus Valid outcomes A B C

1 2

1 Cons

2

1 1

A B C

1 2 2

2 Cons

2 2

A B C

1 2 2

2 Cons

Consensus Processes propose values and make decisions validity: decision is one of the proposals agreement: all decisions are the same termination: all correct processes decide

Piotr Zieli´ nski

Optimistically Terminating Consensus

2

Introduction

Consensus Crash Course

Unifying framework

Consensus Valid outcomes A B C

1 2

1 Cons

2

1 1

A B C

1 2

2 Cons

2

2

2

1

1

A B C

1 2 2

2 Cons

2

Invalid outcomes A B C

1 2 2

3 Cons

3 3

A B C

2 2

Piotr Zieli´ nski

Cons

1 2

A B C

1 2

1 Cons

2

Optimistically Terminating Consensus

1

Introduction

Consensus Crash Course

Unifying framework

Consensus Valid outcomes A B C

1 2

1 Cons

2

1 1

A B C

1 2

2 Cons

2

2

2

1

2

A B C

1 2 2

2 Cons

2

Fault-tolerance A B C

1 2

2 Cons

2

A B C

A

2

Cons

2 B

2

Piotr Zieli´ nski

C

1 2

2 Cons

2

Optimistically Terminating Consensus

2

Introduction

Consensus Crash Course

Unifying framework

Consensus: current state A B C

1 2

1 Cons

2

1 1

A B C

1 2

2 Cons

2

2 2

A B C

1 2 2

2 Cons

Current state dozens of protocols in existence slight changes in assumptions require new protocols malicious participants 10+ pages of correctness proofs! highly non-trivial to design

Conclusion wasted effort → unified approach necessary Piotr Zieli´ nski

Optimistically Terminating Consensus

2

Introduction

Consensus Crash Course

Unifying framework

Consensus Crash Course

Consensus Crash Course

Piotr Zieli´ nski

Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying framework

Democracy vs. Dictatorship Democracy: majority wins A B C

1

1

1

1

2

1

A B C

1 1

A

1

2

B C

1

1

1 2

decision depends on all inputs: not recoverable with any failure Dictatorship: leader decides A B C

1

1

1

1

2

1

A B C

1

1

1 2

A B

1

C

1

1

1 2

decision depends on one input: not recoverable when leader fails Piotr Zieli´ nski

Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying framework

Two-step approach A B C

1

1

1

2

1

1

1

Algorithm 1

broadcast the message from the leader A

2

decide when received the same (1) from a majority ( A B ) assume a majority of processes are correct

majority contains a correct process → recovery always possible

Piotr Zieli´ nski

Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying framework

Two-step approach A B C

1

1

2

A B

1

C

no decision

1

1

2

1

1

A

1 1

B C

possibly 1

1

1

2

1

1

1 decision 1

Algorithm 1

broadcast the message from the leader A

2

decide when received the same (1) from a majority ( A B ) assume a majority of processes are correct

majority contains a correct process → recovery always possible

Piotr Zieli´ nski

Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying framework

Everything together

A B C

1

1

2

1

2 2

2

2

2

2

2

1

2 round 1

2

round 2

2

round 3

Complete Consensus algorithm each round as before: leader proposes, decision by majority if not successful, new round with new leader do not propose values conflicting with previous decisions Piotr Zieli´ nski

Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying Framework

Unifying Framework

Piotr Zieli´ nski

Optimistically Terminating Consensus

Unifying framework

Introduction

Consensus Crash Course

Unifying framework

Optimistically Terminating Consensus (OTC)

A B C

1

1

1

2

1

1

1

A B

1

1

1

2

1

1

1

C

2

Single round of Consensus second step isolated into OTC

2

propose the value 1 from the leader decide if receive the same from ≥ 2 processes 2 is a parameter

Piotr Zieli´ nski

Optimistically Terminating Consensus

OTC

Introduction

Consensus Crash Course

Unifying framework

Optimistically Terminating Consensus (OTC)

A B C

1

1

1

2

2 2

2

2

2

2

OTC as a black box one OTC per one Consensus round decision if all correct processes propose the same the decision is recoverably unique Piotr Zieli´ nski

2

2

2

2

1 2

2

Optimistically Terminating Consensus

2

Introduction

Consensus Crash Course

Unifying framework

Optimistically Terminating Consensus (OTC)

A B C

1

1

2

2

2

2 2

OTC as a black box one OTC per one Consensus round decision if all correct processes propose the same the decision is recoverably unique Piotr Zieli´ nski

2

2

2

2

1 2

2

Optimistically Terminating Consensus

2

Introduction

Consensus Crash Course

Malicious participants

Opportunities for cheating leader can send different proposals processes can modify forwarded messages recovery phase full algorithms very complicated Piotr Zieli´ nski

Optimistically Terminating Consensus

Unifying framework

Introduction

Consensus Crash Course

Unifying framework

Malicious participants

A B C

1

1

1

1

1

1

2

1

1

D 3

Opportunities for cheating leader can send different proposals processes can modify forwarded messages recovery phase full algorithms very complicated Piotr Zieli´ nski

Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying framework

Malicious participants

A B C D

1

1

1

1

1

1

2

1

1

3

A B C D

2

2

2

1

2

2

2

2

2

3

3

decision 1

A B C

?

?

?

1

1

1

2

2

2

3

D 3

decision 2

3

what happened?

Who is cheating? maybe A broadcast 1 to B , and 2 to C ? maybe B really received 2, and A is just slow? maybe C really received 1, and A is just slow? impossible to determine Piotr Zieli´ nski

Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying framework

Malicious participants A B C

1

1

1

1

1

1

1

1

2

1

1

1

D

Malicious participants three steps necessary [Castro and Liskov, 1999] 2nd and 3rd steps are OTCs no need to look inside OTCs to prove correctness (blackbox) composition of two OTCs is also an OTC Piotr Zieli´ nski

Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying framework

Malicious participants A B C

1

1

1

1

1

1

1

1

2

1

1

1

D 3

3

Malicious participants three steps necessary [Castro and Liskov, 1999] 2nd and 3rd steps are OTCs no need to look inside OTCs to prove correctness (blackbox) composition of two OTCs is also an OTC Piotr Zieli´ nski

Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying framework

Malicious participants A B C

1

1

1

1

1

1

1

1

2

1

1

1

D 3

3

Malicious participants three steps necessary [Castro and Liskov, 1999] 2nd and 3rd steps are OTCs no need to look inside OTCs to prove correctness (blackbox) composition of two OTCs is also an OTC Piotr Zieli´ nski

Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying framework

Malicious participants A B C

1

1

1

1

1

1

1

1

2

1

1

1

D 3

3

Malicious participants three steps necessary [Castro and Liskov, 1999] 2nd and 3rd steps are OTCs no need to look inside OTCs to prove correctness (blackbox) composition of two OTCs is also an OTC Piotr Zieli´ nski

Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying framework

Malicious participants

A B C D

1

1

1

SLOW

2

1

2

1

1

2

1

2 3

1 1

3

3

3

1

1

1

1

1

1

1

1

1

2

2

2

3

Malicious participants three steps necessary [Castro and Liskov, 1999] 2nd and 3rd steps are OTCs no need to look inside OTCs to prove correctness (blackbox) composition of two OTCs is also an OTC Piotr Zieli´ nski

Optimistically Terminating Consensus

3

Introduction

Consensus Crash Course

Unifying framework

Reconstructed algorithms Algorithm

Steps

Round 1

Processes

Chandra and Toueg [1996]

2

2

n > 2f

Lamport and Massa [2004]

2

2

n>f

Brasileiro et al. [2001]

1

3

n > 3f

cheap one-step (new)

1

3

n > 2f

cheap one-step Byzantine (new)

1

4

n > 3f

Martin and Alvisi [2004]

2

5

n > 5f

Castro and Liskov [1999]

3

3 3

n > 3f

2/3

3 3

n > 3f

3 3 3

n > 3f

Dutta et al. [2004] multi-step Byzantine (new) n: number of processes Piotr Zieli´ nski

2/3/4

f : number of failures Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying framework

OTC Summary Simple implementation broadcast and wait for a given number of replies simple to extend to custom failure models automatic verification and discovery possible [Zieli´ nski, 2006]

Reconstructs all known Consensus protocols even with malicious participants no overhead in latency or processes lower bounds attained

Modularity precisely defined interface (blackbox) dramatically reduces development time and proofs

Applicable to similar problems non-blocking Atomic Commit in 2 communication steps

Piotr Zieli´ nski

Optimistically Terminating Consensus

Introduction

Consensus Crash Course

Unifying framework

References Francisco Brasileiro, Fab´ıola Greve, Achour Most´efaoui, and Michel Raynal. Consensus in one communication step. Lecture Notes in Computer Science, 2127:42–50, 2001. Miguel Castro and Barbara Liskov. Practical Byzantine fault tolerance. In Proceedings of the Third Symposium on Operating Systems Design and Implementation, pages 173–186, New Orleans, Louisiana, February 1999. USENIX Association. Tushar Deepak Chandra and Sam Toueg. Unreliable failure detectors for reliable distributed systems. Journal of the ACM, 43(2):225–267, 1996. Partha Dutta, Rachid Guerraoui, and Marko Vukolic. Asynchronous Byzantine Consensus: Complexity, resilience and authentication. Technical Report 200479, EPFL, September 2004. Leslie Lamport and Mike Massa. Cheap Paxos. In Proceedings of 2004 International Conference on Dependable Systems and Networks, pages 307–314, Florence, Italy, June 2004. Jean-Philippe Martin and Lorenzo Alvisi. Fast Byzantine Paxos. Technical Report TR-04-07, University of Texas at Austin, Department of Computer Science., 2004. Piotr Zieli´ nski. MinimizingPiotr latency protocols. PhD thesis, Zieli´ nski of agreement Optimistically Terminating Consensus