Optimistic Generic Broadcast Piotr Zieli´ nski Computer Laboratory University of Cambridge

September 28, 2005

Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Replication example Our goal: minimizing latency Generic Broadcast

Hotel booking system

Protocol 1

client → server: “book room 5”

2

server → client: “room booked”

client

book room 5

client

room booked

server server

Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Replication example Our goal: minimizing latency Generic Broadcast

Fault tolerance by replication Problem a single server crash blocks the entire system

A

A

A

A

B

B

C

C

Solution introduce many servers system still usable despite some servers being down

Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Replication example Our goal: minimizing latency Generic Broadcast

Consistency problems Problem messages might reach the replicas in different orders A and B book the room to

client , replica C to client . results: unpredictable

A B C

Solution ensure that replicas receive requests in the same order by using Atomic Broadcast to disseminate requests

A B C

Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Replication example Our goal: minimizing latency Generic Broadcast

Atomic Broadcast atomic broadcast

Atomic Broadcast clients atomically broadcast messages, such as and replicas atomically deliver them

A

replicas atomically deliver all messages in the same order

B

Atomic Broadcast

C

fault-tolerant

Goal: Minimizing latency in common scenarios

Piotr Zieli´ nski

Optimistic Generic Broadcast

atomic delivery

Introduction Optimistic Generic Broadcast Conclusion

Replication example Our goal: minimizing latency Generic Broadcast

System assumptions

Messages message passing: communication by messages reliable channels: no message loss between correct processes asynchrony: no time bounds for messages, no clocks Processes crash-stop model: only crash failures, no malicious processes Ω leader elector: leader eventually correct and fixed n > 3f: less than a third of the servers can crash Consensus implementable in one step

Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Replication example Our goal: minimizing latency Generic Broadcast

Goal: minimizing latency

Goal Atomic Broadcast with minimum latency if the leader is correct and does not change.

Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Replication example Our goal: minimizing latency Generic Broadcast

Generic Broadcast read x write to x r w r w r r w w

A B

Atomic Broadcast

r r w w r r w w

C

Observations ordering all messages is expensive (Atomic Broadcast) not all messages have to be ordered (Generic Broadcast) r r w w = r r w w 6= r r w w Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Replication example Our goal: minimizing latency Generic Broadcast

Generic Broadcast read x write to x r w r w r r w w

A B

Generic Broadcast

r r w w r r w w

C

Observations ordering all messages is expensive (Atomic Broadcast) not all messages have to be ordered (Generic Broadcast) r r w w = r r w w 6= r r w w Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Generic Broadcast r w

w r

Meta-solution 1

Define the conflict relation“ ”. Only conflicting messages must be delivered in the same order.

2

Determine the partial order “ of conflicting messages.

3

Deliver messages in any total order consistent with “ ”.

r w

w r

r

r w w

r

r w w Piotr Zieli´ nski

Optimistic Generic Broadcast

”

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1 order 2:

order 1:

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1 order 2:

order 1:

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1 order 2:

order 1:

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1 order 2:

order 1:

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1 order 2:

order 1:

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1 order 2:

order 1: m2

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1 order 2:

order 1: m2 ,m3

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1 order 2:

order 1: m2 ,m3

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1 order 2:

order 1: m2 ,m3

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1 order 2:

order 1: m2 ,m3

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1 order 2:

order 1: m2 ,m3 ,m1

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1 order 2:

order 1: m2 ,m3 ,m1 ,m4

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1 order 2:

order 1: m2 ,m3 ,m1 ,m4

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5

order 2:

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

order 2:

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

order 2:

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

order 2:

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

order 2:

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

order 2:

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

order 2:

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

order 2: m1

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

order 2: m1

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

order 2: m1

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

order 2: m1 ,m2

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

order 2: m1 ,m2

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

m1

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

m1

order 2: m1 ,m2 ,m3

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

m1

order 2: m1 ,m2 ,m3

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

m1

order 2: m1 ,m2 ,m3 ,m4

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

m1

order 2: m1 ,m2 ,m3 ,m4

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

m1

order 2: m1 ,m2 ,m3 ,m4 ,m5

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

m1

order 2: m1 ,m2 ,m3 ,m4 ,m5 ,m6

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m4

m4

m5

m3

m6

m2

m3

m6

m2

m1

order 1: m2 ,m3 ,m1 ,m4 ,m5 ,m6

m1

order 2: m1 ,m2 ,m3 ,m4 ,m5 ,m6

Delivery rule Deliver a message when all undelivered conflicting messages are its successors Piotr Zieli´ nski

m5

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Problems m1

m2 1

m2 m3

use a separate Consensus instance for each pair of messages, all executed in parallel 2

Cycles if no failures, the leader dictates the order, no cycles if failures occur, a cycle-resolution algorithm used

m1 3

m1

Different processes perceive different orders

m2 m3 m4 m5 m6

The graph contains all possible messages infinitely many parallel instances of Consensus most of them identical, only finitely many different implementable with finite resources

Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Agreement on message order: two messages Problem messages

and

m1

conflict

m2

different processes see different orders Solution Consensus to decide on the order each replica proposes the first message received

A

if decision

, then

→

B

if decision

, then

→

C

Piotr Zieli´ nski

Optimistic Generic Broadcast

?

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Agreement on message order: many messages

m1 m2

m3 m4

m1

m2

m3

m4

m1

no conflict

m1 ↔ m2

m1 ↔ m3

m1 ↔ m4

m2

m1 ↔ m2

no conflict

no conflict

m2 ↔ m4

m3

m1 ↔ m3

no conflict

no conflict

m3 ↔ m4

m4

m1 ↔ m4

m2 ↔ m4

m3 ↔ m4

no conflict

Comments many parallel Consensus instances mi ↔ mj one instance for each pair of conflicting messages mi and mj message pairs are unordered: mi ↔ mj ≡ mj ↔ mi Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Agreement on message order: many messages

m1 m2

m3 m4

m1

m2

m3

m4

m1

no conflict

m1

m1

m1

m2

m1

no conflict

no conflict

m2

m3

m1

no conflict

no conflict

?

m4

m1

m2

?

no conflict

Comments many parallel Consensus instances mi ↔ mj one instance for each pair of conflicting messages mi and mj message pairs are unordered: mi ↔ mj ≡ mj ↔ mi Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Infinitely many instances at the same time Example. All processes receive only m1 , m2 , m3 , in this order

m1

m2

m3

m4 m5 m6 m7 m8

m1 m2 m3 m4 m5 m6 .. .

m1

m2

m3

m4

m5

m6

···

— m1 m1 m1 m1 m1 .. .

m1 — m2 m2 m2 m2 .. .

m1 m2 — m3 m3 m3 .. .

m1 m2 m3 — ? ? .. .

m1 m2 m3 ? — ? .. .

m1 m2 m3 ? ? — .. .

··· ··· ··· ··· ··· ··· .. .

infinitely many instances mi ↔ mj

identical instances share state

finitely many different instances

finite resources sufficient

Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Infinitely many instances at the same time Example. All processes receive only m1 , m2 , m3 , in this order

m1

m2

m3

m4 m5 m6 m7 m8

m1 m2 m3 m4 m5 m6 .. .

m1

m2

m3

m4

m5

m6

···

— m1 m1 m1 m1 m1 .. .

m1 — m2 m2 m2 m2 .. .

m1 m2 — m3 m3 m3 .. .

m1 m2 m3 — ? ? .. .

m1 m2 m3 ? — ? .. .

m1 m2 m3 ? ? — .. .

··· ··· ··· ··· ··· ··· .. .

infinitely many instances mi ↔ mj

identical instances share state

finitely many different instances

finite resources sufficient

Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Latency 1

1

1

A B

2

A

?

B

C

?

C

Different orders

Same order conflicting messages received in the same order

conflicting messages received in different orders

same Consensus proposals

different Consensus proposals

delivery in 1 + 1 = 2 steps

delivery in 1 + 2 = 3 steps

Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Related work

Generic Broadcast

same order

Chandra and Toueg [1996] Pedone and Schiper [1998] Pedone and Schiper [1999] Aguilera et al. [2000] This work

3 2 4 4 2

steps steps steps steps steps

no conflicts 3 4 2 2 2

steps steps steps steps steps

The latency of our algorithm is provably optimal.

Piotr Zieli´ nski

Optimistic Generic Broadcast

other 3 4 4 4 3

steps steps steps steps steps

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Cycles Observation If the leader does not change, cycles do not appear

m1

m3

m5

m2

m4

m6

Proof by contradiction 1

Consensus instances adopt leader’s proposal, so

2

the leader proposed m3 → m4 → m5 → m3 , so

3

the leader received m3 before m4 before m5 before m3 . Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Cycles Observation If cycles appear, they must be resolved

m1

m3

m5

m2

m4

m6

Cycle resolution messages in cycles and their successors are blocked (grey) break cycles by delivering the first message (m3 ) use Atomic Broadcast to agree on a total order on messages Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

m1 , m2 , m3 , m4 , m5 , m6 Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

m1 , m2 , m3 , m4 , m5 , m6 Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

m1 , m2 , m3 , m4 , m5 , m6 Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

m1 , m2 , m3 , m4 , m5 , m6 Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

m1 , m2 , m3 , m4 , m5 , m6 Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

m1 , m2 , m3 , m4 , m5 , m6 Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

m1 , m2 , m3 , m4 , m5 , m6 Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

m1 , m2 , m3 , m4 , m5 , m6 Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Example m1

m3

m5

m1

m3

m5

m2

m4

m6

m2

m4

m6

m1 , m2 , m3 , m4 , m5 , m6

m3 , m1 , m4 , m5 , m6 , m2

Delivery rule Deliver a message when all undelivered conflicting messages 1

succeed it in the partial order, or

2

succeed it in the total order and are blocked Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Problem 1: Agreement on message order Problem 2: Infinitely many instances of Consensus Problem 3: Cycle resolution

Problems m1

m2 1

m2 m3

use a separate Consensus instance for each pair of messages, all executed in parallel 2

Cycles if no failures, the leader dictates the order, no cycles if failures occur, a cycle-resolution algorithm used

m1 3

m1

Different processes perceive different orders

m2 m3 m4 m5 m6

The graph contains all possible messages infinitely many parallel instances of Consensus most of them identical, only finitely many different implementable with finite resources

Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Frequently asked questions Summary of the talk

Frequently Asked Questions Is it possible to . . . 1

deliver some messages faster than in two steps? No, this means delivery before feedback from others.

2

drop the requirement n > 3f ? No, 2-step delivery requires n > 3f . [Pedone and Schiper, 2004]

3

not use the oracle for 2-step deliveries (thriftiness)? No, at least not with Consensus-based implementations. [Guerraoui and Raynal, 2003]

4

deliver all messages in two steps in all runs? Yes, but only in closed groups, with no failures, and perfectly synchronized clocks. Otherwise, no. [Zieli´ nski, 2005]

Piotr Zieli´ nski

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Frequently asked questions Summary of the talk

Summary of the talk m4

Optimistic Generic Broadcast

m5

m3

based on agreed partial order delivery in 2 or 3 steps

m6

latency provably optimal m2

m1

m2 m1

m3

m2

subsumes Generic Broadcast and Optimistic Atomic Broadcast

m1

m2 m3 m4 m5 m6

Contributions

m1

Piotr Zieli´ nski

1

1-2-step Consensus

2

cycle resolution

3

infinitely many instances

Optimistic Generic Broadcast

Introduction Optimistic Generic Broadcast Conclusion

Frequently asked questions Summary of the talk

References Marcos Kawazoe Aguilera, Carole Delporte-Gallet, Hugues Fauconnier, and Sam Toueg. Thrifty Generic Broadcast. Lecture Notes in Computer Science, 1914:268–282, 2000. Tushar Deepak Chandra and Sam Toueg. Unreliable failure detectors for reliable distributed systems. Journal of the ACM, 43(2):225–267, 1996. Rachid Guerraoui and Michel Raynal. The information structure of indulgent Consensus. Technical Report PI-1531, IRISA, April, 2003. Fernando Pedone and Andr´e Schiper. On the inherent cost of Generic Broadcast. Technical Report IC/2004/46, Swiss Federal Institute of Technology (EPFL), May 2004. Fernando Pedone and Andr´e Schiper. Optimistic Atomic Broadcast. In Proceedings of the 12th International Symposium on Distributed Computing, September 1998. Fernando Pedone and Andr´e Schiper. Generic Broadcast. In Proceedings of the 13th International Symposium on Distributed Computing, 1999. Piotr Zieli´ nski

Optimistic Generic Broadcast