OpenDaylight Performance Stress Test Report v1.3: Beryllium vs Boron
Intracom Telecom SDN/NFV Lab
www.intracom-telecom.com | intracom-telecom-sdn.github.com |
[email protected]
Intracom Telecom | SDN/NFV Lab G 2017
OpenDaylight Performance Stress Tests Report v1.3: Beryllium vs Boron Executive Summary In this report we present comparative stress test results performed on OpenDaylight Beryllium SR0 controller against OpenDaylight Boron SR0. For these tests the NSTAT (Network Stress–Test Automation Toolkit) [1] testing platform and its external testing components (Multinet [2] , OFTraf [3], MT–Cbench [4], nstat-nb-emulator [5]) have been used. The test cases presented in this report are identical to those presented in our previous performance report v1.2, so the interested reader is referred to [6] for comprehensive descriptions. As opposed to v1.2, in this report all test components have been executed within Docker containers [7] instead of KVM instances.
Contents
Switch scalability stress tests 3.1 Idle Multinet switches . . . . . . . . . . . . . . 3.1.1 Test configuration . . . . . . . . . . . 3.1.2 Results . . . . . . . . . . . . . . . . . 3.2 Idle MT–Cbench switches . . . . . . . . . . . . 3.2.1 Test configuration . . . . . . . . . . . 3.2.2 Results . . . . . . . . . . . . . . . . . 3.3 Active Multinet switches . . . . . . . . . . . . . 3.3.1 Test configuration . . . . . . . . . . . 3.3.2 Results . . . . . . . . . . . . . . . . . 3.4 Active MT–Cbench switches . . . . . . . . . . . 3.4.1 Test configuration, ”RPC” mode . . . . 3.4.2 Test configuration, ”DataStore” mode 3.4.3 Results . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . .
4 4 4 4 5 5 5 5 5 6 6 6 6 7
. . . . . . . .
7 7 7 8 8 9 9 9 10
4
Stability tests 4.1 Idle Multinet switches . . . . . . . . . . . . . . 4.1.1 Test configuration . . . . . . . . . . . 4.1.2 Results . . . . . . . . . . . . . . . . . 4.2 Active MT–Cbench switches . . . . . . . . . . . 4.2.1 Test configuration, ”DataStore” mode 4.2.2 Results . . . . . . . . . . . . . . . . . 4.2.3 Test configuration, ”RPC” mode . . . . 4.2.4 Results . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . .
. . . . . . . . . . . . .
. . . . . . . .
. . . . . . . . . . . . .
. . . . . . . .
. . . . . . . . . . . . .
. . . . . . . .
. . . . . . . . . . . . .
. . . . . . . .
5
Flow scalability tests 5.1 Test configuration . . . . . . . . . . . . . . . . . . . . . . . 5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 Add controller time/rate . . . . . . . . . . . . . 5.2.2 End–to–end flows installation controller time/rate
10 11 12 12 12
6
Contributors
12
References
12
1. NSTAT Toolkit The NSTAT toolkit follows a modular and distributed architecture. With the term modular we mean that the core application works as an orchestrator that coordinates the testing OpenDaylight Performance Stress Tests Report v1.3: Beryllium vs Boron
con
fig
traf fic
co nfi
gt
raf
fic
NSTAT node lifecycle management
SCGEN–NB node N
orchestration
reporting
test dimensioning
sampling & profiling
monitoring
controller node
OF
SCGEN-SB node 1
ffic
tra
SCGEN-SB node 2
...
OF traffic
3
c
4
ffi
Experimental setup
...
SCGEN–NB node 2
tra
2
SCGEN–NB node 1
OF
3
fic
NSTAT Toolkit
config tr af
1
SCGEN-SB node N
Fig. 1: NSTAT architecture
process. It controls the lifecycle of different testing components and also the coordination and lifecycle management of the testing subject, the OpenDaylight controller. With the term distributed, we mean that each component, controlled by NSTAT, can either run on the same node or on different nodes, which can be either physical or virtual machines. In the latest implementation of NSTAT we introduced the use of Docker containers [7]. With the use of containers1 we can isolate the separate components and their use of resources. In older versions of NSTAT we were using CPU affinity [9] to achieve this resource isolation. In Fig.1 NSTAT toolkit architecture is depicted. 1 The provisioning of docker containers along with their interconnection, is achieved with the use of 1) the docker–compose provisioning tool and 2) the pre–built docker images which are present at docker hub [8]. All containers were running on the same physical server.
Date: 7th Feb G p.3/14
SDN controller
SDN controller
∗ TS s∗ ES LY QU EP ic E R f f R _ T_ tra ART .3 AR /1 TIP TIP 1.0 MUL UL F , O s, M Ys ST PL UE RE EQ O_ R H _ EC HO EC
Mininet switch 1
Mininet switch 2
∗ ffic ∗∗ c 1. raffi t F 0 O 1. OF ra 0t
Mininet switch 3
...
* the vast majority of packets
MT–Cbench switch 1
MT–Cbench switch 2
MT–Cbench switch 3
...
*, ** a few messages during initialization, idle during main execution.
(a) Multinet switches sit idle during main operation. ECHO and MULTIPART mes- (b) MT–Cbench switches sit idle during main operation, without sending or resages dominate the traffic being exchanged between the controller and the swit- ceiving any kind of messages. ches. Fig. 2: Representation of a switch scalability test with idle Multinet and MT-Cbench switches. Table 1: Stress tests experimental setup. Host operating system Nodes type Container OS Physical Server platform Processor model Total system CPUs CPUs configuration Main memory SDN Controllers under test Controller JVM options Multinet OVS version
Centos 7, kernel 3.10.0 Docker containers (Docker version 1.12.1, build 23cf638) Ubuntu 14.04 Dell R720 Intel Xeon CPU E5–2640 v2 @ 2.00GHz 32 2 sockets × 8 cores/socket × 2 HW-threads/core @ 2.00GHz 256 GB, 1.6 GHz RDIMM OpenDaylight Beryllium (SR1), OpenDaylight Boron (SR0) -Xmx8G, -Xms8G, -XX:+UseG1GC, -XX:MaxPermSize=8G 2.3.0
2. Experimental setup Details of the experimental setup are presented in Table 1.
3. Switch scalability stress tests In switch scalability tests we test controller towards different scales of OpenFlow switches networks. In order to create these networks we use either MT–Cbench [4] or Multinet [2]. MT– Cbench generates OpenFlow traffic emulating a “fake” OpenFlow v1.0 switch topology. Multinet utilizes Mininet [10] and OpenVSwitch v2.3.0 [11] to emulate distributed OpenFlow v1.3 switch topologies. In our stress tests we have experimented with topology switches operating in two modes, idle and active mode: switches in idle mode do not initiate any traffic to the controller, but rather respond to messages sent by it. Switches in active mode consistently initiate traffic to the controller, in the form of PACKET_IN messages. In most stress tests, MT–Cbench and Multinet switches operate both in active and idle modes. For more details regarding the tests setup, the reade should refer to the NSTAT: OpenDaylight Performance Stress Test ReOpenDaylight Performance Stress Tests Report v1.3: Beryllium vs Boron
port v1.2, [6] 3.1 Idle Multinet switches 3.1.1 Test configuration
• topology types: ”Linear”, ”Disconnected” • topology size per worker node: 100, 200, 300, 400, 500, 600 • number of worker nodes: 16 • group size: 1, 5 • group delay: 5000ms • persistence: enabled In this case, the total topology size is equal to 1600, 3200, 4800, 6400, 8000, 9600. 3.1.2 Results
Results from this series of tests are presented in Fig.3(a), 3(b) for a ”Linear” topology and Fig.4(a), 4(b) for a ”Disconnected” topology respectively.
Date: 7th Feb G p.4/14
Beryllium Boron
2756
Beryllium
8000
Boron
6400
1563
bootup time [s]
bootup time [s]
2163
4800
1030
3200
511
6400
1363 4800
702
1600
−1
3200
293 113 −1
8000 9600
1600
9600
number of network switches [N]
number of network switches [N] (a) Group size: 1, topology type: Linear
(b) Group size: 5, topology type: Linear
Fig. 3: Switch scalability stress test results for idle Multinet switches. Network topology size from 1600→9600 switches. Topology type: Linear. Boot–up time is forced to -1 when switch discovery fails. 1357
Beryllium
6400
4800
1545 3200
1024
509
Beryllium Boron
bootu time [s]
bootup time [s]
2150
6400
Boron
1600
672
4800
286 80009600
−1
3200
1600
111
80009600
−1 number of network switches [N] (a) Group size: 1, topology type: Disconnected.
number of network switches [N] (b) Group size: 5, topology type: Disconnected.
Fig. 4: Switch scalability stress test results for idle Multinet switches. Network size scales from 1600→9600 switches. Topology type: Disconnected. Boot–up time is forced to -1 when switch discovery fails.
3.2 Idle MT–Cbench switches
3.2.2 Results
3.2.1 Test configuration
Results of this test are presented in Fig.5(a), 5(b), 6(a), 6(b).
• controller: ”RPC” mode • generator: MT–Cbench, latency mode • number of MT–Cbench threads: 1, 2, 4, 8, 16, 20, 30, 40, 50, 60, 70 , 80, 90, 100 threads. • topology size per MT–Cbench thread: 50 switches • group delay: 500, 1000, 2000, 4000, 8000, 16000. • persistence: enabled
In this case switch topology size is equal to: 50, 100, 200, 300, 400, 800, 1000, 1500 2000, 2500, 3000, 3500, 4000, 4500, 5000. OpenDaylight Performance Stress Tests Report v1.3: Beryllium vs Boron
3.3 Active Multinet switches 3.3.1 Test configuration
• controller: with l2switch plugin installed, configured to respond with mac–to–mac FLOW_MODs to PACKET_IN messages with ARP payload [12] • topology size per worker node: 12, 25, 50, 100, 200, 300 • number of worker nodes: 16 • group size: 1 • group delay: 3000ms • topology type: ”Linear” Date: 7th Feb G p.5/14
inter thread creation delay: 1sec
inter thread creation delay: 0.5sec Beryllium
Beryllium
Boron
Boron
51
5000
46 41
4000
35
3500
30 26
3000 2500
21
4000
71
3500
61
3000
51
2500 2000 1500
21 16 9 42 1
1000 800 50
81
31
1500
4 2
4500
41
2000
15 11 9
5000
91
4500 bootup time [s]
bootup time [s]
102
200 400 100 102 103 number of network switches [N]
1000 800 50
(a) group delay: 0.5s.
100 200 400 102 103 number of network switches [N] (b) group delay: 1s.
Fig. 5: Switch scalability stress test results with idle MT–Cbench switches. Topology size scales from 50 → 5000 switches. inter thread creation delay: 16sec
inter thread creation delay: 8sec Beryllium
Beryllium Boron
5000
bootup time [s]
721
4000
561
3500
481
3000
401
2500
321
2000
241
1500 1000 800 50
400 100 200 2 10 103 number of network switches [N] (a) group delay: 8s.
5000
Boron
1441
4500
641
161 129 64 32 16 8
1601
bootup time [s]
801
4500
1281
4000
1121
3500
961
3000 2500
801
2000
641
1500
481 321 257 129 64 32 16
1000 800 50
400 100 200 102 103 number of network switches [N] (b) group delay: 16s.
Fig. 6: Switch scalability stress test results with idle MT–Cbench switches. Topology size scales from 50 → 5000 switches.
• • • •
hosts per switch: 2 traffic generation interval: 60000ms PACKET_IN transmission delay: 500ms persistence: disabled
Switch topology size scales as follows: 192, 400, 800, 1600, 3200, 4800. 3.3.2 Results
• generator: MT–Cbench, latency mode • number of MT–Cbench threads: 1, 2, 4, 8, 16, 20, 40, 60, 80, 100 threads, • topology size per MT–Cbench thread: 50 switches • group delay: 15s • traffic generation interval: 20s • persistence: enabled In this case, the total topology size is equal to 50, 100, 200, 400, 800, 1000, 2000, 3000, 4000, 5000.
Results of this test are presented in Fig.8. 3.4.2 Test configuration, ”DataStore” mode
3.4 Active MT–Cbench switches 3.4.1 Test configuration, ”RPC” mode
• controller: ”RPC” mode OpenDaylight Performance Stress Tests Report v1.3: Beryllium vs Boron
• controller: ”DataStore” mode • generator: MT–Cbench, latency mode, • number of MT–Cbench threads: 1, 2, 4, 8, 16, 20, 40, 60, 80, 100 threads, Date: 7th Feb G p.6/14
SDN controller
SDN controller
s∗ ∗ fic _IN Ds rt af KET MO 1.0 AC W_ OF ial P FLO c tifi ial ar tific ar
Ys PL fic RE raf HO_ Ts t ES .3 /1 , EC PLYs QU E 1.0 s∗ RE _ OF T_In T_R O E PAR CH STS I CK ,E E PA ULT s∗ QU M OD _RE _M ART OW IP FL ULT M
Mininet switch 1
Mininet switch 2
Mininet switch 3
...
MT-Cbench switch 1
MT-Cbench switch 2
MT-Cbench switch 3
...
* the vast majority of packets
* the vast majority of packets
(a) The switches send OF1.3 PACKET_IN messages to the controller, which (b) The switches send artificial OF1.0 PACKET_IN messages to the controller, replies with OF1.3 FLOW_MODs. which replies with also artificial OF1.0 FLOW_MOD messages. Fig. 7: Representation of switch scalability stress test with active (a) Multinet and (b) MT–Cbench switches.
8
4. Stability tests
Beryllium Boron
7
4800|7.31
Stability tests explore how controller throughput behaves in a large time window with a fixed topology connected to it, Fig.10(a), 10(b). The goal is to detect performance fluctuations over time.
[Mbytes/s]
6
5
The controller accepts a standard rate of incoming traffic and its response throughput is being sampled periodically. NSTAT reports these samples in a time series.
4
3 3200|2.21 2 4800|1.68
4.1 Idle Multinet switches
1600|0.79
1 192|0.12 192|0.08
0
400|0.19 400|0.14
800|0.34
1600|0.79
3200|0.96
800|0.32
number of network switches [N]
Fig. 8: Switch scalability stress test results with active Multinet switches. Comparison analysis of controller throughput variation Mbytes/s vs number of network switches N. Topology size scales from 192 → 4800 switches.
• • • •
topology size per MT–Cbench thread: 50 switches group delay: 15s traffic generation interval: 20s persistence: enabled
In this case, the total topology size is equal to 50, 100, 200, 300, 400, 800, 1000, 2000, 3000, 4000, 5000.
The purpose of this test is to investigate the stability of the controller to serve standard traffic requirements of a large scale Multinet topology of idle switches. During main test execution Multinet switches respond to ECHO and MULTIPART messages sent by the controller at regular intervals. These types of messages dominate the total traffic volume during execution. NSTAT uses the oftraf [3] to measure the outgoing traffic of the controller. The metrics presented for this case are the OpenFlow packets and bytes collected by NSTAT every second, Fig.15. In order to push the controller performance to its limits, the controller is executed on the bare metal and Multinet on a set of interconnected virtual machines 4.1.1 Test configuration
3.4.3 Results
Results for test configurations defined in sections 3.4.1, 3.4.2 are presented in Figs.9(a), 9(b) respectively.
OpenDaylight Performance Stress Tests Report v1.3: Beryllium vs Boron
• • • • • •
topology size per worker node: 200 number of worker nodes: 16 group size: 1 group delay: 2000ms topology type: ”Linear” hosts per switch: 1 Date: 7th Feb G p.7/14
300
120000
throughput [responses/s]
110000
Beryllium Boron
109236.57 106481.125
100000 91807.225 86160.75
90000
89492.7 83618.975 79954.35 77676.57
80000 76331.55
72263.425 67044.675 68305.975 61657.6 65230.85
212.725 | 4000
200
165.05 | 3000
150
125.375 | 5000 114.775 | 2000 100
106.225 | 4000
74327.55
68647.42 67052.9
70000
258.95 | 5000
250
throughput [responses/s]
Beryllium (mean) Beryllium (min) Beryllium (max) Boron (mean) Boron (min) Boron (max)
60000
64.9 | 1000 50
59155.84
50
50000
100 10 2
200
400
8001000 10 3
55723.6 2000 300040005000
14.8 | 50
14.2 | 100
0 10
35.05 | 400 25.87 | 200
63 | 3000
55.55 | 800
19.35 | 1000 26.6 | 2000 15.075 | 20014.77 | 400 14.37 | 800
2
10
3
number of network switches [N]
number of network switches [N]
(a) OpenDaylight running in RPC mode.
17.4 | 50
20.07 | 100
(b) OpenDaylight running in DS mode.
Fig. 9: Switch scalability stress test results with active MT–Cbench switches. Comparison analysis of controller throughput variation [responses/s] vs number of network switches N with OpenDaylight running both on RPC and DataStore mode.
SDN controller
SDN controller
* Ys PL c* _RE ffi T tra AR .1 3 LTIP S* / Ts ST ES UE 1.0 , MU F U Q O LYs EQ RE P _R RT_ RE O _ H A HO EC LTIP EC U M
Mininet switch 1
Mininet switch 2
s∗ fic IN s∗ rt af ET_ OD .0 CK M 1 _ OF al PA OW ci FL fi l i t cia ar tifi ar
Mininet switch 3
* the vast majority of packets (a) Representation of switch stability stress test with idle Multinet switches.
MT–Cbench switch 1
MT-Cbench switch 2
MT–Cbench switch 3
* the vast majority of packets (b) Representation of switch stability stress test with active MT–Cbench switches.
Fig. 10: Representation of switch stability stress test with idle Multinet (a) and active MT–Cbench switches (b). The controller accepts a standard rate of incoming traffic and its response throughput is being sampled periodically.
• • • •
period between samples: 10s number of samples: 4320 persistence: enabled total running time: 12h
In this case switch topology size is equal to 3200.
4.1.2 Results
The results of this test are presented in Fig.15
OpenDaylight Performance Stress Tests Report v1.3: Beryllium vs Boron
4.2 Active MT–Cbench switches In this series of experiments NSTAT uses a fixed topology of active MT–Cbench switches to generate traffic for a large time window. The switches send artificial OF1.0 PACKET_IN messages at a fixed rate to the controller, which replies with also artificial OF1.0 FLOW_MOD messages; these message types dominate the traffic exchanged between the switches and the controller. We evaluate the controller both in ”RPC” and ”DataStore” modes. In order to push the controller performance to its limits, all test nodes (controller, MT–Cbench) were executed on bare metal. To isolate the nodes from each other, the CPU shares feature
Date: 7th Feb G p.8/14
controller throughput [Boron]
controller throughput [Beryllium]
35
35
30
throughput [responses/sec]
throughput [responses/sec]
30
25
20
15
10
20
10
15
5
5
0
25
0 0
1000
3000
2000
repeat number [N]
4000
5000
0
1000
3000
2000
4000
5000
repeat number [N]
(a)
(b)
Fig. 11: Controller stability stress test with idle MT–Cbench switches. Throughput comparison analysis between OpenDaylight Beryllium and Boron versions. OpenDaylight running in ”DS” mode. used memory Gbytes, [Boron]
250
250
200
200
[Gbytes]
[Gbytes]
used memory Gbytes, [Beryllium]
150
100
150
100
50
0
50
0
1000
3000
2000
4000
5000
0
0
1000
3000
2000
4000
5000
repeat number [N]
repeat number [N] (a)
(b)
Fig. 12: Controller stability stress test with idle MT–Cbench switches. Throughput comparison analysis between OpenDaylight Beryllium and Boron versions. OpenDaylight running in ”DS” mode.
of NSTAT was used.
In this case, the total topology size is equal to 500 switches. 4.2.2 Results
4.2.1 Test configuration, ”DataStore” mode
• • • • • • • • •
controller: ”DataStore” mode generator: MT–Cbench, latency mode number of MT-–Cbench threads: 10 topology size per MT–Cbench thread: 50 switches group delay: 8s number of samples: 4320 period between samples: 10s persistence: enabled total running time: 12h
OpenDaylight Performance Stress Tests Report v1.3: Beryllium vs Boron
The results of this test are presented in Fig.11, 12. 4.2.3 Test configuration, ”RPC” mode
• • • • • •
controller: ”RPC” mode generator: MT–Cbench, latency mode number of MT–Cbench threads: 10 topology size per MT–Cbench thread: 50 switches group delay: 8s number of samples: 4320, Date: 7th Feb G p.9/14
controller throughput [Beryllium]
110000
100000
90000
80000
110000
100000
90000
80000
70000
70000
60000
60000
50000 0
1000
2000
3000
4000
repeat number [N]
controller throughput [Boron]
120000
throughput [responses/sec]
throughput [responses/sec]
120000
50000 0
5000
1000
2000
3000
repeat number [N]
(a)
4000
5000
(b)
Fig. 13: Controller stability stress test with idle MT–Cbench switches. Throughput comparison analysis between OpenDaylight Beryllium and Boron versions. OpenDaylight running in ”RPC” mode. used memory Gbytes, [Boron]
used memory Gbytes, [Beryllium] 250
250
200
[Gbytes]
[Gbytes]
200
150
150
100
100
50
50
0
0 0
1000
2000
3000
4000
5000
repeat number [N] (a)
0
1000
2000
3000
4000
5000
repeat number [N] (b)
Fig. 14: Controller stability stress test with idle MT–Cbench switches. Throughput comparison analysis between OpenDaylight Beryllium and Boron versions. OpenDaylight running in ”RPC” mode.
• period between samples: 10s • persistence: enabled • total running time: 12h In this case, the total topology size is equal to 500 switches. 4.2.4 Results
The results of this test are presented in Fig.13, 14.
5. Flow scalability tests With flow scalability stress tests we try to investigate both capacity and timing aspects of flows installation via the controlOpenDaylight Performance Stress Tests Report v1.3: Beryllium vs Boron
ler NB (RESTCONF) interface, Fig.16. This test uses the NorthBound flow emulator [5] to create flows in a scalable and configurable manner (number of flows, delay between flows, number of flow writer threads). The flows are being written to the controller configuration data store via its NB interface, and then forwarded to an underlying Multinet topology as flow modifications, where they are distributed into switches in a balanced fashion. The test verifies the success or failure of flow operations via the controller’s operational data store. An experiment is considered successful when all flows have been installed on switches and have been reflected in the operational data store of the controller. If not all of them have become visible in the
Date: 7th Feb G p.10/14
outgoing kbytes [s] Vs repeat number
outgoing packets per [s] Vs repeat number
300
4000
Beryllium Boron
Beryllium Boron 250
3000 200
[kbytes/s]
outgoing packets per [s]
3500
2500
2000
1500
150
100
1000 50 500
0
0 0
1000
2000
3000
repeat number [N]
4000
5000
0
1000
2000
3000
4000
5000
repeat number [N]
(a) Comparison analysis for outgoing packets/s between OpenDaylight Beryl- (b) Comparison analysis for outgoing kbytes/s between OpenDaylight Belium and Boron releases. ryllium and Boron releases. Fig. 15: Controller 12–hour stability stress test with idle Multinet switches.
The metrics measured in this test are: NB app 1
NB app 2
con
fig
tra
ffic
(RE
ST
NB app 3
...
)
SDN controller
ffic s∗ tra PLY ∗ .1 3 RT_RE UESTs / A Q 1.0 LTIP _RE*
OF
U O TS ,M CH ES Ys s, EEQU PL D E R O_ _R _MART HO OW IP EC FL ULT M
Mininet switch 1
Mininet switch 2
Mininet switch 3
• Add controller time (tadd ): the time needed for all requests to be sent and their response to be received (as in [9]). • Add controller rate (radd ): radd = N / tadd , where N the aggregate number of flows to be installed by worker threads. • End–to–end installation time (te2e ): the time from the first flow installation request until all flows have been installed and become visible in the operational data store. • End-to-end installation rate (re2e ): re2e = N / te2e In this test, Multinet switches operate in idle mode, without initiating any traffic apart from the MULTIPART and ECHO messages with which they reply to controller’s requests at regular intervals. 5.1 Test configuration
* the vast majority of packets Fig. 16: Representation of flow scalability stress test. An increasing number of NB clients (NB appj , j=1,2,. . .) install flows on the switches of an underlying OpenFlow switch topology.
data store within a certain deadline after the last update, the experiment is considered failed. Intuitively, this test emulates a scenario where multiple NB applications, each controlling a subset of the topology2 , send simultaneously flows to their switches via the controller’s NB interface at a configurable rate. 2 subsets are non-overlapping and equally sized
OpenDaylight Performance Stress Tests Report v1.3: Beryllium vs Boron
For both Lithium and Beryllium we used the following setup • • • • • • • • • •
topology size per worker node: 1, 2, 4, 35, 70, 330. number of worker nodes: 15 group size: 1 group delay: 3000ms topology type: ”Linear” hosts per switch: 1 total flows to be added: 1K, 10K, 100K, 1M flow creation delay: 0ms flow worker threads: 5 persistence: disabled
In this case switch topology size is equal to: 15, 30, 60, 525, 1050, 4950. Date: 7th Feb G p.11/14
number of flows: N = 10000
30
number of flows: N = 10000
2000
Beryllium
Beryllium
Boron
Boron
25 add controller rate [Flows/s]
add controller time [s]
1200 | 21.45
20
4950 | 19.28 1050 | 19.46
15 525 | 12.18
10
15 | 9.02
30 | 8.47
60 | 8.61
15 | 8.24 30 | 8.15
60 | 8.09
1050 | 11.36 525 | 9.75
1500 30 | 1227.62 60 | 1236.00 15 | 1213.06
1000
30 | 1180.27 60 | 1161.44 15 | 1109.18
525 | 1026.16 1050 | 880.34 525 | 820.98 1050 | 513.76
500
1200 | 466.24
5 0 1 10
102 103 number of network switches [N]
0 1 10
104
4950 | 518.80
102 103 number of network switches [N]
(a) Flow operations: 104 .
104
(b) Flow operations: 104 .
Fig. 17: Flow scalability stress test result. Comparison performance for add controller time/rate vs number of switches for various numbers of flow operations N. Add controller time is forced to -1 when test fails. Add controller time/rate vs number of switches for N=104 flow operations. number of flows: N = 100000 Beryllium
Beryllium
Boron
Boron 1200 | 121.54
120 100
add controller rate [Flows/s]
add controller time [s]
1050 | 117.72 525 | 96.78
80 15 | 67.14
60
number of flows: N = 100000
2000
4950 | 143.58
140
15 | 65.10
30 | 68.35 60 | 68.25
1050 | 81.20 525 | 73.92
30 | 66.01 60 | 67.60
40
15 | 1536.22 30 | 1514.84 60 | 1479.19
1500
15 | 1489.40 30 | 1463.10 60 | 1465.25
525 | 1352.82 1050 | 1231.59 525 | 1033.29
1000 1050 | 849.47 1200 | 822.75 4950 | 696.45
500
20 0 101
102 103 number of network switches [N]
0 1 10
104
(a) Flow operations: 105 .
102 103 number of network switches [N]
104
(b) Flow operations: 105 .
Fig. 18: Flow scalability stress test result. Comparison performance for add controller time/rate vs number of switches for various numbers of flow operations N. Add controller time is forced to -1 when test fails. Add controller time/rate vs number of switches for N=105 flow operations.
5.2 Results
• Konstantinos Papadopoulos • Thomas Sounapoglou
5.2.1 Add controller time/rate
The results of this test are presented in Figs.17, 18, 19. 5.2.2 End–to–end flows installation controller time/rate
References
The results of this experiment are presented in Figs.20, 21.
[1]
“NSTAT: Network Stress Test Automation Toolkit.” https:// github.com/intracom-telecom-sdn/nstat.
[2]
“Multinet: Large–scale SDN emulator based on Mininet.” https://github.com/intracom-telecom-sdn/multinet.
[3]
“OFTraf: pcap–based, RESTful OpenFlow traffic monitor.” https://github.com/intracom-telecom-sdn/oftraf.
6. Contributors • Nikos Anastopoulos • Panagiotis Georgiopoulos OpenDaylight Performance Stress Tests Report v1.3: Beryllium vs Boron
Date: 7th Feb G p.12/14
number of flows: N = 1000000
number of flows: N = 1000000 2000
2000 Beryllium
Beryllium
Boron
Boron 1200 | 1611.04 1050 | 1575.76
30 | 1268.08 60 | 1172.52
525 | 1236.87
15 | 1128.84
1000
1500
add controller rate [Flows/s]
add controller time [s]
1500
4950 | 1205.91
1050 | 1116.14 30 | 1021.58
15 | 1013.41
60 | 963.22
500
0 101
60 | 1038.18
1000
525 | 1024.03
102 103 number of network switches [N]
525 | 976.53 1050 | 895.95 4950 | 829.25
15 | 885.86 30 | 788.60
60 | 832.86
525 | 808.49 1200 | 620.72 1050 | 834.61
500
0 101
104
(a) Add controller time vs number of switches for N=106 flow operations.
15 | 986.76 30 | 978.87
102 103 number of network switches [N]
104
(b) Add controller rate vs number of switches for N=106 flow operations.
Fig. 19: Flow scalability stress test result. Comparison performance for add controller time/rate vs number of switches for various numbers of flow operations N. Add controller time is forced to -1 when test fails. Add controller time/rate vs number of switches for N=106 flow operations. number of flows: N = 10000
number of flows: N = 1000
300
350 Beryllium
Beryllium
Boron
flows controller installation rate [Flows/s]
flows controller installation rate [Flows/s]
Boron
250 200 150 1050 | 103.07
100 50 525 | 45.91 1200 | 24.62 15 | 5.12
0 10115 | 4.72
30 | 6.78
60 | 6.14
60 | 4.88 30 | 4.36
1050 | 21.53 525 | 10.57
102
103
number of network switches [N]
250 200 150 1050 | 111.95
100 525 | 54.60
50
4950 | -1
104
(a) Flow operations: 103 .
1200 | 83.77
15 | 15.78
30 | 14.26
0 15 | 12.04 101
30 | 12.13
60 | 15.15 60 | 12.77
1050 | 35.26 525 | 20.91
4950 | -1
102 of network switches 103[N] number
104
(b) Flow operations: 104 .
Fig. 20: Flow scalability stress test result. Comparison performance for add controller time/rate vs number of switches for various numbers of flow operations N. Add controller time is forced to -1 when test fails. Add controller time/rate vs number of switches for N=104 flow operations.
[4]
“MT–Cbench: A multithreaded version of the Cbench OpenFlow traffic generator.” https://github.com/ intracom-telecom-sdn/mtcbench.
[5]
“NSTAT: NorthBound Flow Emulator.” https://github.com/ intracom-telecom-sdn/nstat-nb-emulator.
[6]
“NSTAT: Performance Stress Tests Report v1.2: ”Beryllium Vs Lithium SR3”.” https://raw.githubusercontent. com/wiki/intracom-telecom-sdn/nstat/files/ODL_ performance_report_v1.2.pdf.
[7]
“Docker docker.
containers.”
https://www.docker.com/what-
OpenDaylight Performance Stress Tests Report v1.3: Beryllium vs Boron
[8]
“Docker–hub Intracom–repository.” https://hub.docker. com/u/intracom/.
[9]
“OpenDaylight Performance: A practical, empirical guide. End–to–End Scenarios for Common Usage in Large Carrier, Enterprise, and Research Networks.” https://www.opendaylight.org/sites/www.opendaylight. org/files/odl_wp_perftechreport_031516a.pdf.
[10]
“Mininet. An instant virtual network on your Laptop.” http: //mininet.org/.
[11]
“OpenVSwitch: http://openvswitch.org/.” //openvswitch.org/.
http:
Date: 7th Feb G p.13/14
number of flows: N = 100000
300
number of flows: N = 1000000 3000
Beryllium
Beryllium Boron
flows controller installation rate [Flows/s]
flows controller installation rate [Flows/s]
Boron
250
2500
1050|209.06
30|207.6
200 1200|153.58
525|146.34
150 60|127.52
525|118.87
1000
15|76.48 30|79.44
60|80.82
50 0 15|-1 101
4950|-1
102
1050|1883.71
1000
1050|147.27
100
525|2063.32
1500
103
number of network switches [N]
500 15|-1
104
(a) Flow operations: 105 .
0 1 10 15|-1
30|-1 30|-1
525|-1 1050|-1
60|-1 60|-1
102
103
4950|-1 1050|-1
number of network switches [N]
104
(b) Flow operations: 106 .
Fig. 21: Flow scalability stress test result. Comparison performance for add controller time/rate vs number of switches for various numbers of flow operations N. Add controller time is forced to -1 when test fails. Add controller time/rate vs number of switches for N=106 flow operations.
[12]
“Generate PACKET_IN events with ARP payload.” https://github.com/intracom-telecom-sdn/multinet# generate-packet_in-events-with-arp-payload.
OpenDaylight Performance Stress Tests Report v1.3: Beryllium vs Boron
Date: 7th Feb G p.14/14