A Performance study on Operator-based stream processing systems

Miyuru Dayarathna, Souhei Takeno, Toyotaro Suzumura Department of Computer Science Tokyo Institute of Technology Japan

Stream Computing Systems 

Insights from data in motion ◦ It is impossible to store data on disk ◦ The volume of the data is very large



Process data on-the-fly in-memory Route keyless input events

OP 1

Join the serve and click events

OP 2

BotFilter

OP 3

Compute the correct click throughput rate

OP 4

Streaming Click-Through Rate Computation

2

Essence of our Performance Study System S (IBM) and S4 (Yahoo)  Four benchmarks (60 application Scenarios)  Five metrics 

3

12

140

Thousands

Thousands

Results - Throughput Throughput observed for four applications on S4

Throughput observed for five applications on System S CDR

120

10

VWAP

6 CDR Optimized

4

VWAP Twitter

2

Micro-benchmark

Throughput (Tuples\s)

Throughput (Events\s)

100 8

Micro-benchmark CDR Optimized

80 Twitter

60 40 20

CDR

0

0 0

2

4

6

8

Number of Nodes

(c)

10

12

14

0

2

4

6

8

10

12

14

Number of Nodes

(d)

4

Essence of our Performance Study System S (IBM) and S4 (Yahoo)  Four benchmarks (60 application Scenarios)  Five metrics  Conclusions on Stream Processing system architectures 

5