INTRACOM TELECOM - GitHub

Viewer
Transcript

OpenDaylight Performance Stress Test Report v1.2: Lithium vs Beryllium

Intracom Telecom SDN/NFV Lab

www.intracom-telecom.com | intracom-telecom-sdn.github.com | [email protected]

Intracom Telecom | SDN/NFV Lab G 2016

OpenDaylight Performance Stress Tests Report v1.2: Lithium vs Beryllium Executive Summary In this report we investigate several performance and scalability aspects of the OpenDaylight Beryllium controller and compare them against the Lithium release. Beryllium outperforms Lithium in idle switch scalability tests both with Multinet (realistic OF1.3) and MT–Cbench (artificial OF1.0) emulators. A single Beryllium instance can boot ”Linear” and ”Disconnected” idle Multinet topologies of up to 6400 switches, as opposed to 4800 with Lithium. In general, Beryllium manages to successfully boot topologies that Lithium fails to, or it can boot them faster. In MT–Cbench tests the superiority of Beryllium is even more evident, being able to successfully discover topologies that expose themselves to the controller at a faster rate as compared to Lithium. With regard to controller stability, both releases exhibit similarly stable behavior. In addition, Multinet–based tests reveal traffic optimizations in Beryllium, witnessed by the reduced volume related to MULTIPART messages. Finally, in NorthBound performance tests Beryllium seems to underperform in accepting ﬂow installation requests via RESTCONF, but as far as the end–to–end ﬂow installation is concerned it is the clear winner. In many extreme cases of large topologies (e.g. 5000 switches) or large ﬂow counts (1M), Beryllium manages to successfully install ﬂows while Lithium fails.

Contents 1

Introduction

3

2

NSTAT Toolkit

4

3

Experimental setup

4

4

Switch scalability stress tests 4.1 Idle Multinet switches . . . . . . . . . . . . . . 4.1.1 Test conﬁguration . . . . . . . . . . . 4.1.2 Results . . . . . . . . . . . . . . . . . 4.2 Idle MT–Cbench switches . . . . . . . . . . . . 4.2.1 Test conﬁguration . . . . . . . . . . . 4.2.2 Results . . . . . . . . . . . . . . . . . 4.3 Active Multinet switches . . . . . . . . . . . . . 4.3.1 Test conﬁguration . . . . . . . . . . . 4.3.2 Results . . . . . . . . . . . . . . . . . 4.4 Active MT–Cbench switches . . . . . . . . . . . 4.4.1 Test conﬁguration, ”RPC” mode . . . . 4.4.2 Test conﬁguration, ”DataStore” mode 4.4.3 Results . . . . . . . . . . . . . . . . . 4.5 Conclusions . . . . . . . . . . . . . . . . . . . . 4.5.1 Idle scalability tests . . . . . . . . . . 4.5.2 Active scalability . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

5 6 7 7 7 8 8 8 8 9 9 9 9 9 9 9 10

Stability tests 5.1 Idle Multinet switches . . . . . . . . . . . . . . 5.1.1 Test conﬁguration . . . . . . . . . . . 5.1.2 Results . . . . . . . . . . . . . . . . . 5.2 Active MT–Cbench switches . . . . . . . . . . . 5.2.1 Test conﬁguration, ”DataStore” mode 5.2.2 Results . . . . . . . . . . . . . . . . . 5.2.3 Test conﬁguration, ”RPC” mode . . . . 5.2.4 Results . . . . . . . . . . . . . . . . . 5.3 Conclusions . . . . . . . . . . . . . . . . . . . . 5.3.1 Idle stability tests . . . . . . . . . . . 5.3.2 Active stability tests . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

10 10 11 11 11 13 13 13 13 13 13 13

6

Flow scalability tests 6.1 Test conﬁguration . . . . . . . . . . . . . . . . . . . . . . . 6.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Add controller time/rate . . . . . . . . . . . . . 6.2.2 End–to–end flows installation controller time/rate 6.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . .

13 15 16 16 16 16

7

Contributors

16

References

5

OpenDaylight Performance Stress Tests Report v1.2: Lithium vs Beryllium

16

1. Introduction In this report we investigate several performance and scalability aspects of the OpenDaylight Beryllium controller and compare them against the Lithium release. The investigation focuses on both the SouthBound (SB) and NorthBound (NB) controller’s interface and targets the following objectives: • • • •

controller throughput, switch scalability, controller stability (sustained throughput), flow scalability and flow provisioning time

For our evaluation we have used NSTAT [1], an open source environment written in Python for easily writing SDN controller stress tests and executing them in a fully automated and end– to–end manner. NSTAT’s architecture is exhibited in Fig.1 and its key components are the • SDN controller, • SouthBound OpenFlow traffic generator, • NorthBound flow generator (RESTCONF traffic). The SDN controller used in this series of experiments is OpenDaylight. We perform and demonstrate results from the Lithium SR3 and Beryllium RC2 releases. Date: May 20, 2016 G p.3/16

conﬁg traffic

NB generator

NSTAT orchestration

reporting

test dimensioning

sampling

monitoring

In our stress tests we have experimented with emulated switches operating in two modes, idle and active mode: switches in idle mode do not initiate any traffic to the controller, but rather respond to messages sent by it. Switches in active mode consistently initiate traffic to the controller, in the form of PACKET_IN messages. In most stress tests, MT–Cbench and Multinet switches operate both in active and idle modes.

controller

2. NSTAT Toolkit OF traﬃc

lifecycle management

are being connected to the SDN controller, one can easily test various switch group size/delay combinations and discover conﬁgurations that yield optimal boot–up times for a large– scale topology.

SB generator

Fig. 1: NSTAT architecture

For SouthBound (SB) traffic generation we have used both MT– Cbench [2] and Multinet [3]. For NorthBound (NB) traffic generation NSTAT uses custom scripts [4], originally developed by the OpenDaylight community [5], for creating and writing flows to the controller conﬁguration data store. MT–Cbench is a direct extension of the Cbench [6] emulator which uses threading to generate OpenFlow traffic from multiple streams in parallel. The motivation behind this extension was to be able to boot–up and operate network topologies much larger than those with the original Cbench, by gradually adding switches in groups. We note here that, as in the original Cbench, the MT–Cbench switches implement only a minimal subset of the OF1.0 protocol, and therefore the results presented for that generator are expected to vary in real–world deployments. This gap is largely ﬁlled using Multinet [7] as an additional SB traffic generator, as it uses OVS virtual switches that accurately emulate the OF1.3 protocol. The goal of Multinet is to provide a fast, controlled and resource eﬃcient way to boot large– scale SDN topologies. This is achieved by booting in parallel multiple isolated Mininet topologies, each launched on a separate virtual machine, and all connected to the same controller1 . Finally, by providing control on the way that switches

The general architecture of NSTAT is depicted in Fig.1. The NSTAT node lies at the heart of the environment and controls all others. The test nodes --controller node, SB node(s), NB node-- are mapped on one or more physical or virtual interconnected machines. Unless otherwise stated, in all our experiments every node is mapped on a separate virtual machine. NSTAT is responsible for automating and sequencing every step of the testing procedure, including building and deploying components on their corresponding nodes, initiating and scaling SB and NB traffic, monitoring controller operation in terms of correctness and performance, and producing performance reports along with system and application health state and proﬁling statistics. Each of these steps is highly conﬁgurable using a rich and simple JSON–based conﬁguration system. Each stress test scenario features a rich set of parameters that determine the interaction of the SB and NB components with the controller, and subsequently trigger varying controller behavior. Such parameters are for example the number of OpenFlow switches, the way they connect to the controller, the number of NB application instances, the rate of NB/SB traffic etc., and in a sense these deﬁne the ”dimensions” of a stress test scenario. One of NSTAT’s key features is that it allows the user to specify multiple values for one or more such dimensions at the same time, and the tool itself takes care to repeat the test over all their combinations in a single session. This kind of exhaustively exploring the multi–dimensional experimental space offers a comprehensive overview of the controller’s behavior on a wide range of conditions, making it possible to easily discover scaling trends, performance bounds and pathological cases.

3. Experimental setup Details of the experimental setup are presented in Table 1.

1 As an example, Multinet can create more than 3000 Openvswitch OF1.3

switches over moderate virtual resources (10 vCPUs, <40GB memory) in less than 10 minutes. OpenDaylight Performance Stress Tests Report v1.2: Lithium vs Beryllium

As mentioned in section 1, in our stress tests we have experimented with emulated switches operating in idle and active Date: May 20, 2016 G p.4/16

SDN controller

SDN controller

∗ TS s∗ ES LY QU EP ic E R f f R _ T_ tra ART .3 AR /1 TIP TIP 1.0 MUL UL F , O s, M Ys ST PL UE RE EQ O_ R H _ EC HO EC

Mininet switch 1

Mininet switch 2

∗ ffic ∗∗ c 1. raﬃ t F 0 O 1. OF ra 0t

Mininet switch 3

...

* the vast majority of packets

MT–Cbench switch 1

MT–Cbench switch 2

MT–Cbench switch 3

...

*, ** a few messages during initialization, idle during main execution.

(a) Multinet switches sit idle during main operation. ECHO and MULTIPART mes- (b) MT–Cbench switches sit idle during main operation, without sending or resages dominate the traffic being exchanged between the controller and the ceiving any kind of messages. switches. Fig. 2: Representation of a switch scalability test with idle Multinet and MT-Cbench switches. Table 1: Stress tests experimental setup. Host operating system Hypervisor Guest operating system Server platform Processor model Total system CPUs CPUs configuration Main memory SDN Controllers under test OpenDaylight conﬁguration Controller JVM options Multinet OVS version

Centos 7, kernel 3.10.0 QEMU v.2.0.0 Ubuntu 14.04 Dell R720 Intel Xeon CPU E5–2640 v2 @ 2.00GHz 32 2 sockets × 8 cores/socket × 2 HW-threads/core @ 2.00GHz 256 GB, 1.6 GHz RDIMM OpenDaylight Beryllium (RC2), OpenDaylight Lithium (SR3) OF plugin implementation: ”Helium design” -Xmx8G, -Xms8G, -XX:+UseG1GC, -XX:MaxPermSize=8G 2.0.2

mode using both MT–Cbench and Multinet for emulating the network topology. In MT–Cbench tests the controller is conﬁgured to start with the ”drop–test” feature installed. The emulated switches are arranged in a disconnected topology, meaning they do not have any interconnection between them. As we have already mentioned, this feature, along with the limited protocol support, constitute MT–Cbench a special–purpose OpenFlow generator and not a full–fledged, realistic OpenFlow switch emulator. Two different conﬁgurations of the controller are evaluated with MT–Cbench active switches: in ”RPC” mode, the controller is conﬁgured to directly reply to PACKET_IN’s sent by the switches with a predeﬁned message at the OpenFlow plugin level. In ”DataStore” mode, the controller additionally performs updates in its data store. In all cases, MT– Cbench is conﬁgured to operate in ”Latency” mode, meaning that each switch sends a PACKET_IN message only after it has received a reply for the previous one. In all stress tests the ”Helium–design” implementation of the OpenFlow plugin was used. In future versions of this report

OpenDaylight Performance Stress Tests Report v1.2: Lithium vs Beryllium

we will evaluate the new implementation, codenamed ”Lithium–design”.

4. Switch scalability stress tests Switch scalability tests aim at exploring the maximum number of switches the controller can sustain, when switches are being gradually added either in idle or active mode. Apart from the maximum number of switches the controller can successfully see, a few more additional metrics are reported: • in idle mode tests, NSTAT also reports the topology boot– up time for diﬀerent policies of connecting switches to the controller. From these results we can deduce how to optimally boot–up a certain–sized topology and connect it to the controller, so that the latter can successfully discover it at the minimum time. • in active mode tests, NSTAT also reports the controller throughput, i.e. the rate at which the controller replies back to the switch–initiated messages. Therefore, in this case, we also get an overview of how controller throughput scales as the topology size scales. Date: May 20, 2016 G p.5/16

858

Beryllium (RC2)

6400

bootup time [s]

Lithium (SR3)

1578

4800

1036 520

3200

122 8000 9600

103

number of network switches [N]

4800

3200

347

1600

−1

Beryllium (RC2) Lithium (SR3)

bootup time [s]

2348

1600 6400 8000 9600

−1 103

104

(a) Group size: 1, topology type: Linear

number of network switches [N]

104

(b) Group size: 5, topology type: Linear

Fig. 3: Switch scalability stress test results for idle Multinet switches. Network topology size from 1600→9600 switches. Topology type: Linear. Boot–up time is forced to -1 when switch discovery fails.

1457

Beryllium (RC2) Lithium (SR3)

1540

4800

1027 519

3200 1600

Lithium (SR3)

6400

720

4800 3200

321

−1

103

Beryllium (RC2)

6400

bootup time [s]

bootup time [s]

2220

8000 9600

number of network switches [N]

104

(a) Group size: 1, topology type: Disconnected.

121 −1 103

1600 8000 9600 number of network switches [N]

104

(b) Group size: 5, topology type: Disconnected.

Fig. 4: Switch scalability stress test results for idle Multinet switches. Network size scales from 1600→9600 switches. Topology type: Disconnected. Boot–up time is forced to -1 when switch discovery fails.

4.1 Idle Multinet switches This is a switch scalability test with idle switches emulated using Multinet, Fig.2(a). The objective of this test is twofold: • to ﬁnd the largest number of idle switches the controller can accept and maintain. • to ﬁnd the combination of boot–up–related conﬁguration options that leads to the fastest successful network boot–up. Speciﬁcally, these options are the group size at which switches are being connected, and the delay between each group. We consider a boot–up as successful when all network switches have become visible in the operational data store of the conOpenDaylight Performance Stress Tests Report v1.2: Lithium vs Beryllium

troller. If not all switches have been reflected in the data store within a certain deadline after the last update, we declare the boot–up as failed. NSTAT reports the number of switches ﬁnally discovered by the controller, and the discovery time. During main test execution Multinet switches respond to ECHO and MULTIPART messages sent by the controller at regular intervals. These types of messages dominate the total traffic volume during execution. We evaluate two diﬀerent topology types, ”Linear” and ”Disconnected”. In order to push the controller performance to its limits, the controller node is executed on bare metal. Multinet is executed within a set of virtual machines, see master/worker functionality of Multinet, [8].

Date: May 20, 2016 G p.6/16

group delay: 0.5sec

group delay: 1sec

Beryllium (RC2)

Beryllium (RC2)

Lithium (SR3)

4000 3000

32

bootup time [s]

bootup time [s]

41

2500

27

1500

16 800

10 5 32 −1

50

400

100

3500 1000 2000 5000

200

101 91 81 71 61 51 41 31 21 17 8 42

Lithium (SR3)

5000 4500 4000 3500 3000 2500 2000

50

100

200

400

1500 1000 800

102 103 number of network switches [N]

102 103 number of network switches [N]

(a) group delay: 0.5s.

(b) group delay: 1s.

Fig. 5: Switch scalability stress test results with idle MT–Cbench switches. Topology size scales from 50 → 5000 switches. group delay: 8sec

group delay: 16sec

Beryllium (RC2)

5000

Lithium (SR3)

bootup time [s]

3500

561

3000

481

2500

401

2000

321

1500

241

1000 800 50

100

200

400

102 103 number of network switches [N] (a) group delay: 8s.

5000

Lithium (SR3)

4500

1441

4000

641

161 129 64 32 16 8

1601

4500

721

bootup time [s]

801

Beryllium (RC2)

4000

1281

3500

1121

3000

961

2500

801

2000

641

1500

481 321 257 128 64 32 16

1000 800 50

400 100 200 102 103 number of network switches [N] (b) group delay: 16s.

Fig. 6: Switch scalability stress test results with idle MT–Cbench switches. Topology size scales from 50 → 5000 switches.

4.1.1 Test conﬁguration

• topology types: ”Linear”, ”Disconnected” • topology size per worker node: 100, 200, 300, 400, 500, 600 • number of worker nodes: 16 • group size: 1, 5 • group delay: 5000ms • persistence: enabled In this case, the total topology size is equal to 1600, 3200, 4800, 6400, 8000, 9600. 4.1.2 Results

Results from this series of tests are presented in Fig.3(a), 3(b) for a ”Linear” topology and Fig.4(a), 4(b) for a ”Disconnected” OpenDaylight Performance Stress Tests Report v1.2: Lithium vs Beryllium

topology respectively. 4.2 Idle MT–Cbench switches This is a switch scalability test with idle switches emulated using MT–Cbench, Fig.2(b). As in the Multinet case, the goal is to explore the maximum number of idle switches the controller can sustain, and how a certain–sized topology should be optimally connected to it. In contrast to idle Multinet switches that exchange ECHO and MULTIPART messages with the controller, MT–Cbench switches typically sit idle during main operation, without sending or receiving any kind of messages. Due to this fact, the ability of the controller to accept and maintain MT–Cbench switches is expected to be much larger than the case of using realistic OpenFlow switches. Date: May 20, 2016 G p.7/16

SDN controller

SDN controller

s∗ ∗ fic _IN Ds rt af KET MO 1.0 AC W_ OF ial P FLO c tiﬁ ial ar tiﬁc ar

Ys PL fic RE raf HO_ Ts t ES .3 /1 , EC PLYs QU E 1.0 s∗ RE _ OF T_In T_R O E PAR CH STS I CK ,E E PA ULT s ∗ QU M OD _RE _M ART OW IP FL ULT M

Mininet switch 1

Mininet switch 2

Mininet switch 3

MT-Cbench switch 1

...

MT-Cbench switch 2

MT-Cbench switch 3

...

* the vast majority of packets

* the vast majority of packets

(a) The switches send OF1.3 PACKET_IN messages to the controller, which (b) The switches send artiﬁcial OF1.0 PACKET_IN messages to the controller, replies with OF1.3 FLOW_MODs. which replies with also artiﬁcial OF1.0 FLOW_MOD messages. Fig. 7: Representation of switch scalability stress test with active (a) Multinet and (b) MT–Cbench switches.

35 Beryllium (RC2)

In this case switch topology size is equal to: 50, 100, 200, 300, 400, 800, 1000, 1500 2000, 2500, 3000, 3500, 4000, 4500, 5000.

4800|32.19

Lithium (SR3) 30 3200|27.15

[Mbytes/s]

25

192|23.20

400|23.48

4.2.2 Results

1600|24.75

800|23.93

Results of this test are presented in Fig.5(a), 5(b), 6(a), 6(b).

20

4.3 Active Multinet switches

15

This is a switch scalability test with switches in active mode emulated using Multinet, Fig.7(a).

4800|13.02

192|9.98

400|10.02

1600|10.36

800|10.11

3200|11.09

10

5 2 10

10

3

number of network switches [N]

10

4

Fig. 8: Switch scalability stress test results with active Multinet switches. Comparison analysis of controller throughput variation Mbytes/s vs number of network switches N. Topology size scales from 192 → 4800 switches.

In order to push the controller performance to its limits, all test nodes (controller, MT–Cbench) were executed on bare metal. To isolate the nodes from each other, the CPU shares feature of NSTAT was used, [9].

Its target is to explore the maximum number of switches the controller can sustain while they consistently initiate traffic to it (active), and how the controller servicing throughput scales as more switches are being added. The switches start sending messages after the entire topology has been connected to the controller. They trigger OF1.3 PACKET_IN messages for a certain interval and at a conﬁgurable rate, and the controller replies with OF1.3 FLOW_MODs. These message types dominate the traﬃc exchanged between the switches and the controller. Our target metric is the throughput of the controller’s outgoing OpenFlow traffic. NSTAT uses the oftraf [10] tool to measure it. In order to push the controller performance to its limits, the controller is executed on the bare metal and Multinet on a set of interconnected virtual machines.

4.2.1 Test conﬁguration

• controller: ”RPC” mode • generator: MT–Cbench, latency mode • number of MT–Cbench threads: 1, 2, 4, 8, 16, 20, 30, 40, 50, 60, 70 , 80, 90, 100 threads. • topology size per MT–Cbench thread: 50 switches • group delay: 500, 1000, 2000, 4000, 8000, 16000. • persistence: enabled OpenDaylight Performance Stress Tests Report v1.2: Lithium vs Beryllium

4.3.1 Test conﬁguration

• controller: with l2switch plugin installed, conﬁgured to respond with mac–to–mac FLOW_MODs to PACKET_IN messages with ARP payload [11] • topology size per worker node: 12, 25, 50, 100, 200, 300 • number of worker nodes: 16 • group size: 1 • group delay: 3000ms Date: May 20, 2016 G p.8/16

300

105000 100

Beryllium (RC2)

200

267.8|5000

Lithium (SR3)

100000

253.9|5000

400

95000

800 1000

90000

85000

4000 2000 3000

5000

50

80000

75000

throughput [responses/s]

throughput [responses/s]

250 223.5|4000

203.25|4000

200 174.65|3000

161.9|3000 150 124.725|2000 113.925|2000 100

Beryllium (RC2) [mean] 74.45|1000 66.65|800 67.72|1000 46.95|400 57.27|800 39.85|200

Lithium SR3 [mean]

70000

Beryllium (RC2) [min] Lithium SR3 [min]

65000

50

Beryllium (RC2) [max]

19.475|50 11.075|50

Lithium SR3 [max] 60000 10

2

10

3

0

15.735|100

10

number of network switches [N]

36.75|400

15.325|100 19.075|200 2

10

3

number of network switches [N]

(a) OpenDaylight running in RPC mode.

(b) OpenDaylight running in DS mode.

Fig. 9: Switch scalability stress test results with active MT–Cbench switches. Comparison analysis of controller throughput variation [responses/s] vs number of network switches N with OpenDaylight running both on RPC and DataStore mode.

• • • • •

topology type: ”Linear” hosts per switch: 2 traffic generation interval: 60000ms PACKET_IN transmission delay: 500ms persistence: disabled

Switch topology size scales as follows: 192, 400, 800, 1600, 3200, 4800. 4.3.2 Results

Results of this test are presented in Fig.8. 4.4 Active MT–Cbench switches This test is equivalent to the active Multinet switch scalability test described in section 4.3 The switches start sending messages after the entire topology has been connected to the controller. MT–Cbench switches send artiﬁcial OF1.0 PACKET_IN messages to the controller, which replies with also artiﬁcial OF1.0 FLOW_MOD messages; these are the dominant message types during execution. In order to push the controller performance to its limits, all test nodes (controller, MT–Cbench) were executed on bare metal. To isolate the nodes from each other, the CPU shares feature [9] of NSTAT was used. 4.4.1 Test conﬁguration, ”RPC” mode

• controller: ”RPC” mode • generator: MT–Cbench, latency mode • number of MT–Cbench threads: 1, 2, 4, 8, 16, 20, 40, 60, 80, 100 threads, • topology size per MT–Cbench thread: 50 switches OpenDaylight Performance Stress Tests Report v1.2: Lithium vs Beryllium

• group delay: 15s • trafficgeneration interval: 20s • persistence: enabled In this case, the total topology size is equal to 50, 100, 200, 400, 800, 1000, 2000, 3000, 4000, 5000. 4.4.2 Test conﬁguration, ”DataStore” mode

• controller: ”DataStore” mode • generator: MT–Cbench, latency mode, • number of MT–Cbench threads: 1, 2, 4, 8, 16, 20, 40, 60, 80, 100 threads, • topology size per MT–Cbench thread: 50 switches • group delay: 15s • trafficgeneration interval: 20s • persistence: enabled In this case, the total topology size is equal to 50, 100, 200, 300, 400, 800, 1000, 2000, 3000, 4000, 5000. 4.4.3 Results

Results for test conﬁgurations deﬁned in sections 4.4.1, 4.4.2 are presented in Figs.9(a), 9(b) respectively. 4.5 Conclusions 4.5.1 Idle scalability tests

In idle Multinet switch scalability tests we can see that Beryllium controller has a signiﬁcantly improved performance as it manages to handle larger switch topologies than Lithium. Beryllium can successfully boot ”Linear” topologies of up to 6400 switches in groups of 1, and 4800 switches in groups Date: May 20, 2016 G p.9/16

SDN controller

SDN controller

* Ys PL

c* _RE rt aﬃ ART P * 1.3 LTI s TS .0/ MU S T ES E 1 U OF LYs, QU EQ RE T_R EP _ R _ HO AR HO EC LTIP EC U M

Mininet switch 1

Mininet switch 2

s∗ fic IN s∗ raf ET_ t OD .0 CK M 1 _ OF al PA OW L ci lF tiﬁ cia ar ﬁ i t ar

Mininet switch 3

* the vast majority of packets (a) Representation of switch stability stress test with idle Multinet switches.

MT–Cbench switch 1

MT-Cbench switch 2

MT–Cbench switch 3

* the vast majority of packets (b) Representation of switch stability stress test with active MT–Cbench switches.

Fig. 10: Representation of switch stability stress test with idle Multinet (a) and active MT–Cbench switches (b). The controller accepts a standard rate of incoming traffic and its response throughput is being sampled periodically.

of 5, while Lithium can boot such topologies for up to 4800 switches regardless of the group size, Fig.3. For ”Disconnected” topologies Beryllium manages to boot topologies of 6400 switches for both combinations of group size, Fig.4. Thinking of the best boot–up times for a certain sized topology and for both topology types, we conclude that Beryllium can either successfully boot a topology that Lithium cannot, or it can boot it faster. We note here that the boot–up times demonstrated in our tests are not necessarily the best possible that can be achieved for a certain topology size /type; testing with larger group sizes and/or smaller group delays could reveal further enhancements. In idle MT–Cbench switch scalability tests the superiority of Beryllium is even clearer. As we can see, Beryllium is much more successful in discovering switches that expose themse– lves to the controller at a faster rate (lower group delay, Figs.5(a) and 5(b)). This performance diﬀerence disappears as the switches are being added less aggressively (Figs.6(a) and 6(b)). The Beryllium release can successfully discover almost the max number of switches even with a group delay of 0.5 seconds. In contrast, Lithium can achieve the same numbers with a delay of at least 8 seconds. Beryllium can generally boot large topologies much faster than Lithium.

4.5.2 Active scalability

In Multinet–based tests Lithium exhibits signiﬁcantly higher throughput levels than Beryllium for every switch count. Nevertheless, this does not necessarily imply inferior performance, since, as we will show in section 5.3.1, traffic optimizations might have led to reduced volume overall. In any case, this scenario needs further analysis to better understand the diffeOpenDaylight Performance Stress Tests Report v1.2: Lithium vs Beryllium

rence, and will be a matter of future research. In MT–Cbench based tests the performance gap between the two releases tends to close. In ”RPC” mode the throughput is almost equivalent both for Lithium and Beryllium. In ”DataStore” mode there is also equivalence, with a slight superiority of Lithium. In this case the throughput values are much lower, but this is expected as the controller data store is being intensively hit with the persistence module enabled. This adds to the critical path of the packet processing pipeline.

5. Stability tests Stability tests explore how controller throughput behaves in a large time window with a ﬁxed topology connected to it, Fig.10(a), 10(b). The goal is to detect performance fluctuations over time. The controller accepts a standard rate of incoming traffic and its response throughput is being sampled periodically. NSTAT reports these samples in a time series. 5.1 Idle Multinet switches The purpose of this test is to investigate the stability of the controller to serve standard traffic requirements of a large scale Multinet topology of idle switches. During main test execution Multinet switches respond to ECHO and MULTIPART messages sent by the controller at regular intervals. These types of messages dominate the total traﬃc volume during execution. NSTAT uses the oftraf [10] to measure the outgoing traﬃc of the controller. The metrics presented for this case are the Date: May 20, 2016 G p.10/16

controller throughput [Beryllium (RC2)]

30

30

25

25

20 15 10

20 15 10

5

5

0

0

1000

2000 3000 4000 5000 repeat number [N] system total used memory Gbytes, [Beryllium (RC2)]

250

250

200

200

[Gbytes]

[Gbytes]

0

150

100

50

50

0

1000

2000 3000 repeat number [N]

4000

5000

0

1000

2000 3000 4000 5000 repeat number [N] system total used memory Gbytes, [Lithium (RC2)]

0

1000

150

100

0

controller throughput [Lithium (SR3)

35

throughput [responses/sec]

throughput [responses/sec]

35

0

2000 3000 repeat number [N]

4000

5000

Fig. 11: Controller stability stress test with idle MT–Cbench switches. Throughput and system total used memory comparison analysis between OpenDaylight Lithium (SR3) and Beryllium (RC2) versions. OpenDaylight running in ”DataStore” mode.

OpenFlow packets and bytes collected by NSTAT every second, Fig.13.

In this case switch topology size is equal to 3200.

In order to push the controller performance to its limits, the controller is executed on the bare metal and Multinet on a set of interconnected virtual machines

5.1.2 Results

The results of this test are presented in Fig.13 5.2 Active MT–Cbench switches

5.1.1 Test conﬁguration

• • • • • • • • • •

topology size per worker node: 200 number of worker nodes: 16 group size: 1 group delay: 2000ms topology type: ”Linear” hosts per switch: 1 period between samples: 10s number of samples: 4320 persistence: enabled total running time: 12h

OpenDaylight Performance Stress Tests Report v1.2: Lithium vs Beryllium

In this series of experiments NSTAT uses a ﬁxed topology of active MT–Cbench switches to generate traffic for a large time window. The switches send artiﬁcial OF1.0 PACKET_IN messages at a ﬁxed rate to the controller, which replies with also artiﬁcial OF1.0 FLOW_MOD messages; these message types dominate the traffic exchanged between the switches and the controller. We evaluate the controller both in ”RPC” and ”DataStore” modes. In order to push the controller performance to its limits, all test nodes (controller, MT–Cbench) were executed on bare metal. To isolate the nodes from each other, the CPU shares feature Date: May 20, 2016 G p.11/16

controller throughput [Beryllium (RC2)]

100000

95000

90000

85000

80000 0

100000

95000

90000

85000

1000

2000 3000 repeat number [N]

4000

5000

80000 0

250

250

200

200

150

100

50

50

2000 3000 repeat number [N]

4000

2000 3000 repeat number [N]

4000

5000

150

100

1000

1000

system total used memory Gbytes, [Lithium (RC2)]

[Gbytes]

[Gbytes]

system total used memory Gbytes, [Beryllium (RC2)]

0 0

controller throughput [Lithium (SR3)

105000

throughput [responses/sec]

throughput [responses/sec]

105000

0 0

5000

1000

2000 3000 repeat number [N]

4000

5000

Fig. 12: Controller stability stress test with idle MT–Cbench switches. Throughput and system total used memory comparison analysis between OpenDaylight Lithium (SR3) and Beryllium (RC2) versions. OpenDaylight running in ”RPC” mode.

Beryllium (RC2) Lithium SR3

Beryllium (RC2) Lithium SR3

2500 150 2000 [kbytes/s]

outgoing packets per [s]

outgoing kbytes per [s] Vs repeat number

200

outgoing packets per [s] Vs repeat number

3000

1500

100

1000 50 500 00

1000

2000 3000 repeat number [N]

4000

5000

0 0

1000

2000 3000 repeat number [N]

4000

5000

(a) Comparison analysis for outgoing packets/s between OpenDaylight (b) Comparison analysis for outgoing kbytes/s between OpenDaylight Lithium (SR3) and Beryllium (RC2) releases. Lithium (SR3) and Beryllium (RC2) releases. Fig. 13: Controller 12–hour stability stress test with idle Multinet switches.

OpenDaylight Performance Stress Tests Report v1.2: Lithium vs Beryllium

Date: May 20, 2016 G p.12/16

In this case, the total topology size is equal to 500 switches. NB app 1

NB app 2

con

ﬁg

NB app 3

...

The results of this test are presented in Fig.12.

tra

ffic

(RE

ST

)

5.3 Conclusions 5.3.1 Idle stability tests

SDN controller

From Fig.13 it is apparent that both releases exhibit stable behavior in a time window of 12 hours.

fic raf

s∗ LY EP

t ∗ Ts 1.3 RT_R UES .1 0/ LTIPA _RE*Q U OF LYs, M , ECHUOESTS s Q OD_RE _MART OWLTIP L F U M

P RE

O_

H EC

5.2.4 Results

Mininet switch 1

Mininet switch 2

Mininet switch 3

* the vast majority of packets

A ﬁrst observation is that Beryllium has lower idle traffic overhead compared to Lithium. To investigate the exact reasons for this discrepancy we analyzed the traffic during a sample 1–min stability test with a Linear–switch Multinet topology. Both controllers were conﬁgured to request statistics every 5000 ms. The traffic breakdown for both cases is depicted in Table 2. It is made clear that the reduced traffic in Beryllium Table 2: OpenFlow traffic breakdown for a sample 1–min stability test with a 3200–switch Multinet topology.

Fig. 14: Representation of ﬂow scalability stress test. An increasing number of NB clients (NB appj , j=1,2,. . .) install flows on the switches of an underlying OpenFlow switch topology.

Lithium (SR3)

Beryllium (RC2)

Packets

Bytes

Packets

Bytes

6558 4723 139528

45906 612772 8029791

6832 4957 4787

47824 643468 3402334

5037 9135 80040

643848 63945 4265880

5307 8901 4797

678138 62307 132531

Incoming traffic OFPT_ECHO_REQUEST

of NSTAT was used [9].

OFPT_PACKET_IN OFPT_MULTIPART_REPLY

5.2.1 Test conﬁguration, ”DataStore” mode

• • • • • • • • •

controller: ”DataStore” mode generator: MT–Cbench, latency mode number of MT-–Cbench threads: 10 topology size per MT–Cbench thread: 50 switches group delay: 8s number of samples: 4320 period between samples: 10s persistence: enabled total running time: 12h

In this case, the total topology size is equal to 500 switches. 5.2.2 Results

The results of this test are presented in Fig.11. 5.2.3 Test conﬁguration, ”RPC” mode

• • • • • • • • •

controller: ”RPC” mode generator: MT–Cbench, latency mode number of MT–Cbench threads: 10 topology size per MT–Cbench thread: 50 switches group delay: 8s number of samples: 4320, period between samples: 10s persistence: enabled total running time: 12h

OpenDaylight Performance Stress Tests Report v1.2: Lithium vs Beryllium

Outgoing traffic OFPT_PACKET_OUT OFPT_ECHO_REPLY OFPT_MULTIPART_REQUEST

is attributed to the reduction of MULTIPART messages, both incoming and outgoing, possibly as a result of optimizations implemented in Beryllium in this area. 5.3.2 Active stability tests

As we can see in Figs.11, 12, both controllers exhibit in general stable performance, at almost the same throughput levels. Lithium performs slightly better than Beryllium in ”RPC” mode. In all four cases, a transient performance degradation is observed for a relatively large period. After analyzing system metrics we concluded that this behavior could be related to an increase in the total used memory of our system (see Figs.11, 12) as a result of possible memory leak.

6. Flow scalability tests With flow scalability stress tests we try to investigate both capacity and timing aspects of flows installation via the controller NB (RESTCONF) interface, Fig.14. This test uses the NorthBound ﬂow generator [4] to create ﬂows in a scalable and conﬁgurable manner (number of ﬂows, Date: May 20, 2016 G p.13/16

number of flows: N=10000

20

number of flows: N=10000

1800

Beryllium (RC2) Lithium (SR3)

add controller rate [Flows/s]

1600

add controller time [s]

4950|15.35

15 4950|14.45

10

30|9.54 15|8.92

60|9.01

525|9.11

1050|9.71 1050|8.45

15|7.87

30|7.72

525|8.28

60|7.75

1400 15|1271.17

30|1295.37 60|1289.55 525|1207.38

1200 15|1121.41 1000

5

60|1110.01

1050|1058.17 525|1097.42 1050|1029.91

30|1048.13

800

4950|692.25

600

4950|651.58

400 200

Beryllium (RC2) Lithium (SR3)

0 101

0 101

104

102 103 number of network switches [N]

102 103 number of network switches [N]

(a) Flow operations: 104 .

104

(b) Flow operations: 104 .

Fig. 15: Flow scalability stress test result. Comparison performance for add controller time/rate vs number of switches for various numbers of flow operations N. Add controller time is forced to -1 when test fails. Add controller time/rate vs number of switches for N=104 flow operations. number of flows: N=100000

120 Lithium (SR3)

15|1616.96

4950|111.23

1600 add controller rate [Flows/s]

add controller time [s]

100 1050|81.47

80 60

525|75.45 15|69.46 30|71.34 60|70.81

1050|75.50 525|68.48

15|61.84

30|61.65

number of flows: N=100000

1800

4950|113.78

Beryllium (RC2)

60|62.43

40

30|1621.99 60|1601.91 525|1460.27

1400 15|1439.71

1050|1324.46

30|1401.7360|1412.21

525|1325.40

1200

1050|1227.38

1000

4950 | 899.02 4950|878.55

800 600 400

20 200

Beryllium (RC2) Lithium (SR3)

0 101

102 103 number of network switches [N]

104

(a) Flow operations: 105 .

0 101

102 103 number of network switches [N]

104

(b) Flow operations: 105 .

Fig. 16: Flow scalability stress test result. Comparison performance for add controller time/rate vs number of switches for various numbers of flow operations N. Add controller time is forced to -1 when test fails. Add controller time/rate vs number of switches for N=105 flow operations.

delay between ﬂows, number of ﬂow writer threads). The ﬂows are being written to the controller conﬁguration data store via its NB interface, and then forwarded to an underlying Multinet topology as ﬂow modiﬁcations, where they are distributed into switches in a balanced fashion. The test veriﬁes the success or failure of ﬂow operations via the controller’s operational data store. An experiment is considered successful when all ﬂows have been installed on switches and have been reﬂected in the operational data store of the controller. If not all of them have become visible in the data store within a certain deadline after the last update, the experiment is considered failed. Intuitively, this test emulates a scenario where multiple NB apOpenDaylight Performance Stress Tests Report v1.2: Lithium vs Beryllium

plications, each controlling a subset of the topology2 , send simultaneously ﬂows to their switches via the controller’s NB interface at a conﬁgurable rate. The metrics measured in this test are: • Add controller time (tadd ): the time needed for all requests to be sent and their response to be received (as in [5]). • Add controller rate (radd ): radd = N / tadd , where N the aggregate number of ﬂows to be installed by worker threads. • End–to–end installation time (te2e ): the time from the 2 subsets are non-overlapping and equally sized

Date: May 20, 2016 G p.14/16

number of flows: N=1000000 Beryllium (RC2)

4950|2333.10

Lithium (SR3)

add controller rate [Flows/s]

add controller time [s]

1500 4950|1179.97

1000

1050|843.74 525|775.11 60|692.08

30|688.36

15|1631.1630|1620.94 60|1587.51

1600

2000

15|695.87

number of flows: N=1000000

1800

1050|821.27

1400

15|1437.05

30|1452.7360|1444.92

525|1374.15 1050|1217.63 525|1290.14

1200

1050|1185.19

1000 800

4950|847.48

600

525|727.72

500 15|613.06

30|616.93

60|629.92

400

4950|428.61

200

Beryllium (RC2) Lithium (SR3)

0 101

0 101

104

102 103 number of network switches [N]

(a) Add controller time vs number of switches for N=106 flow operations.

104

102 103 number of network switches [N]

(b) Add controller rate vs number of switches for N=106 flow operations.

Fig. 17: Flow scalability stress test result. Comparison performance for add controller time/rate vs number of switches for various numbers of flow operations N. Add controller time is forced to -1 when test fails. Add controller time/rate vs number of switches for N=106 flow operations. number of flows: N=1000

800

number of flows: N=10000

800 end–to–end installation rate [Flows/s]

end–to–end installation rate [Flows/s]

Beryllium (RC2) Lithium (SR3)

700 600 500 400 300 200 15|158.84 30|143.88

100 15|111.09 30|139.75

60|153.74

525|21.02

0 101

Lithium (SR3) 15|617.12

600

525|16.80

1050|8.62 1050|8.62

60|611.79 30|611.44

500 15|483.38

60|457.90

400 300 525|192.86

200 525|120.67

100

60|97.19

Beryllium (RC2)

30|718.00

700

1050|77.56 1050|55.25

4950|-1.00

0

4950|-1.00

102 103 number of network switches [N]

104

4950|2.5 4950|-1

101

(a) Flow operations: 103 .

102 103 number of network switches [N]

104

(b) Flow operations: 104 .

Fig. 18: Flow scalability stress test result. Comparison performance for end–to–end installation rate vs number of switches for various numbers of flow operations N. End–to–end installation rate is forced to -1 when test fails.

ﬁrst ﬂow installation request until all ﬂows have been installed and become visible in the operational data store. • End-to-end installation rate (re2e ): re2e = N / te2e In this test, Multinet switches operate in idle mode, without initiating any traffic apart from the MULTIPART and ECHO messages with which they reply to controller’s requests at regular intervals.

• • • • • • • • •

number of worker nodes: 15 group size: 1 group delay: 3000ms topology type: ”Linear” hosts per switch: 1 total flows to be added: 1K, 10K, 100K, 1M flow creation delay: 0ms flow worker threads: 5 persistence: disabled

6.1 Test conﬁguration For both Lithium and Beryllium we used the following setup

In this case switch topology size is equal to: 15, 30, 60, 525, 1050, 4950.

• topology size per worker node: 1, 2, 4, 35, 70, 330. OpenDaylight Performance Stress Tests Report v1.2: Lithium vs Beryllium

Date: May 20, 2016 G p.15/16

number of flows: N=100000

800

number of flows: N=1000000

800 Beryllium (RC2)

end–to–end installation rate [Flows/s]

end–to–end installation rate [Flows/s]

Beryllium (RC2) Lithium (SR3)

700 60|641.27

525|662.48 525|548.07

600 500

60|516.09 1050|415.67

400

30|366.71

300

30|262.69

200

1050|411.62

15|147.72

Lithium (SR3)

700 600 500

525|391.94

400

1050|384.22

300 200 100

100 4950|28.09

0

15|-1

101

0

4950|-1

102 103 number of network switches [N]

104

15|-1

30|-1

60|-1

15|-1

30|-1

60|-1

101

(a) Flow operations: 105 .

4950|-1 525|-1

1050|-1

102 103 number of network switches [N]

4950|-1

104

(b) Flow operations: 106 .

Fig. 19: Flow scalability stress test result. Comparison performance for end–to–end installation rate vs number of switches for various numbers of flow operations N. End–to–end installation rate is forced to -1 when test fails.

6.2 Results

References

6.2.1 Add controller time/rate

[1]

“NSTAT: Network Stress Test Automation Toolkit.” https:// github.com/intracom-telecom-sdn/nstat.

[2]

“MT–Cbench: A multithreaded version of the Cbench OpenFlow traffic generator.” https://github.com/ intracom-telecom-sdn/mtcbench.

[3]

“Multinet: Large–scale SDN emulator based on Mininet.” https://github.com/intracom-telecom-sdn/multinet.

[4]

“NSTAT: NorthBound Flow Generator.” https://github. com/intracom-telecom-sdn/nstat/wiki/NorthBoundflow-generator.

[5]

“OpenDaylight Performance: A practical, empirical guide. End–to–End Scenarios for Common Usage in Large Carrier, Enterprise, and Research Networks.” https://www.opendaylight.org/sites/www.opendaylight. org/files/odl_wp_perftechreport_031516a.pdf.

[6]

“Cbench: OpenFlow traffic generator.” https://github. com/andi-bigswitch/oflops/tree/master/cbench.

[7]

“Mininet. An instant virtual network on your Laptop.” http: //mininet.org/.

[8]

“Multinet Features.” https://github.com/intracomtelecom-sdn/multinet#features.

[9]

“Capping controller and emulator CPU resources in collocated tests.” https://github.com/intracom-telecomsdn/nstat/wiki/Capping-controller-and-emulator-CPUresources-in-collocated-tests.

[10]

“OFTraf: pcap–based, RESTful OpenFlow trafﬁc monitor.” https://github.com/intracom-telecom-sdn/oftraf.

[11]

“Generate PACKET_IN events with ARP payload.” https://github.com/intracom-telecom-sdn/multinet# generate-packet_in-events-with-arp-payload.

The results of this test are presented in Figs.15, 16, 17. 6.2.2 End–to–end flows installation controller time/rate

The results of this experiment are presented in Figs.18, 19.

6.3 Conclusions Regarding NB performance, as it is expressed in terms of add controller time/ rate, Lithium performs generally better than Beryllium, but this trend tends to diminish and ﬁnally gets reversed for large topologies. Notably, Be outperforms Li by a large factor in the 1M flows/5K switches case. However, regarding end–to–end performance, Beryllium is the clear winner, outperforming Lithium in all cases of topology sizes and flow counts. In many extreme cases where Lithium fails, Beryllium manages to successfully install flows. This happens for example in the case for 1M flows, or in some 5K switch topologies. This witnesses improvements in the Beryllium release relating to flow installation and discovery.

7. Contributors • Nikos Anastopoulos • Panagiotis Georgiopoulos • Konstantinos Papadopoulos

OpenDaylight Performance Stress Tests Report v1.2: Lithium vs Beryllium

Date: May 20, 2016 G p.16/16

INTRACOM TELECOM: SDN/NFV Lab: OpenDaylight ... - GitHub