SDN Benchmark Testing Plan
Page 1
HUAWEI TECHNOLOGIES CO., LTD. HUAWEI TECHNOLOGIES CO., LTD.
www.huawei.com
Index 1. Overview 1. 4 reports from industry 2. Huawei Internal Testing Results 1. OpenDaylight Test Reproduce 2. IXIA 3. SoupUI 4. Spirent 3. Testing cases proposal from Huawei 4. SDN Controller Test Setup Category 5. SDN Controller Test Case Category 6. Performance Test List 7. Current Test Difficulty
Page 2
Overview – ONF OpenFlow Controller Benchmarking (Nov 2015) 1. 2.
OpenFlow Controller Channel Capacity Measure Packet Out Transmission Efficiency of Auxiliary Channel 3. Measure Master-Slave Recovery Time 4. Measure the Role Intimation Time for a Controller Cluster At Startup 5. Measure the Role Intimation Time for a Controller Cluster After Master Failover 6. Flow Setup Rate: Proactive Mode 7. Flow Setup Rate: Reactive Mode 8. Flow Setup Rate with Auxiliary Channel: Reactive mode 9. End-to-End Flow Setup Time 10. Controller Message Processing 11. Datapath Failure Recovery Time 12. Datapath Switchover Time
ONF Nov 2015
ODL Mar 2016
2016
IETF July 2016
SDNCTC July 2016
Page 3
Overview - OpenDaylight Performance: A Practical, Empirical Guide (May, 2016) OpenFlow: 1. Northbound (REST) Benchmark 2. Southbound (Plugin) Benchmark NETCONF: 1. Southbound Scale 2. Northbound Scale 3. Southbound Performance 4. End-to-end Performance OVSDB BGP: 1. BGP Data Ingestion 2. BGP Data Advertisement 3. BGP App peer populated By RESTCONF 4. BGP APP peer populated By JAVA PCEP
ONF Nov 2015
ODL Mar 2016
2016
IETF July 2016
SDNCTC July 2016
Page 4
ODL Report V2.0 Planning OpenFlow: 1. Northbound REST Test 2. Network Messages Test 3. Scale: 1000 or similar switches doing other tests BGP: 1. Performance with multi Devices
Cluster Test: Pick some tests to include in Clustered version (No HA or Fail-over scenarios this cycle) 1. OpenFlow Flow Programming 2. NetConf 3. Possible NetVirt Other point of the output report 1. All test would be automate 2. Graphs and report for JVM usage (CPU, RAM usage)
Netconf: 1. Performance with multi Devices Netvirt(VTN) 1. Full Stack Scale 2. Full Stack Performance 3. Full Stack Network Failure
ONF Nov 2015
ODL Mar 2016
2016
IETF July 2016
SDNCTC July 2016
ODL v2.0 July 2016
Page 5
Overview - IETF: SDN Controller Performance
(July 2016)
Performance
Scalability
1.
Network Topology Discovery Time
1.
Control Session Capacity
2.
Asynchronous Message Processing Time
2.
Network Discovery Size
3.
Asynchronous Message Processing Rate
3.
Forwarding Table Capacity
4.
Reactive Path Provisioning Time
Security
5.
Proactive Path Provisioning Time
1.
Exception Handling
6.
Reactive Path Provisioning Rate
2.
Denial of Service Handling
7.
Proactive Path Provisioning Rate
Reliability
8.
Network Topology Change Detection Time
1.
Controller Failover Time
2.
Network Re-Provisioning Time
ONF Nov 2015
ODL Mar 2016
2016
IETF July 2016
SDNCTC July 2016
Page 6
Overview – SDNCTC China 1. 2. 3. 4. 5. 6.
(July 2016)
Control Channel Capacity Test Topology Discovery Time Test Packet_out Sending Rate Test Flow MOD Sending Rate Test End-to-end Link Establishment Time Test Role Election Time Test
ONF Nov 2015
ODL Mar 2016
2016
IETF July 2016
SDNCTC July 2016
Page 7
Observation of Current Performance Testing Reports 1. Performance test should test through Southbound protocol and Northbound API, under a same circumstance and related to real-world business. 2. Most of the test reports now are focus on OpenFlow controller‟s benchmarking, instead of actually performance. 3. A Controller‟s performance shouldn‟t be only define by its performance with OpenFlow protocol. 4. Clustering performance should be stand alone as performance test case. Also comparing single-node and clustering performance will be a good comparison.
Scalability
Performance Reliability
Security Maintainability Page 8
SDN Controller Test Setup Category Currently the all test design can be categorize as two part. Application
Application to Controller or Software Testing: Mainly by pressuring through controller‟s northbound access to test a controller‟s performance. Example: TestON, ODL Test Tool
Controller
Possible test area: Data Store, HA, Clustering, API Responsive
Application to Controller or Network Testing: By simulating virtual device under southbound to test a controller‟s performance. Device Example: IXIA, Spirent Possible test area: Operation Time, Rate, Stability and Scalability Page 9
Current Work Achivement 1. Duplicated OpenDaylight‟s Openflow Northbound and Southbound test case. And confirmed that ONOS has better performance in single node‟s environment. (ONOS 1.5.1 v.s. ODL Beryllium Boron plugin) 2. Controller‟s Scalability and Capacity in 1, 3, 5, 7 node‟s environment, using IXIA and Spirent as device emulator. 3. Using SoapUI send out RESTful request to test controller„s VTN performance.
Page 10
OpenDaylight OpenFlow Northbound Test Procedure Test Tool Initial status check (REST)
Add flows (REST)
Remove flows (REST)
1. 2.
detect device ids detect flows on each devices (require to detect device ids again)
3.
5. 1 delete flows to remove 5. 2 delete all flows at one time (remove controller time)
Generate random flows based on deivice_ids Prepare flows information in queue for multi-thread posting
Confirm flows (REST) recursively
Confirm flows (REST) recursively
4. check flows on each devices (require to detect device ids again) (add confirm time)
6.
post flows to add (add controller time)
Prepare flows to add • •
check flows on each devices (require to detect device ids again) (remove confirm time)
Controller
OVS Mininet (15, 31, 63) REST API: 1. get devices ids:.../restconf/operational/opendaylight-inventory:nodes 2. get flows on each devices: .../restconf/operational/opendaylight-inventory:nodes/node/{device_id}/table/0 3. post flows to add: .../restconf/config/opendaylight-inventory:nodes/node/{device_id}/table/0 + json: {"flow": flow_list} 4. get Confirm flows: .../restconf/operational/opendaylight-inventory:nodes/node/{device_id}/table/0 5. 5. 1 delete flows to remove: .../restconf/config/opendaylight-inventory:nodes/node/{device_id}/table/0/flow/{flow_id} 5. 2 delete all flows(bulk_delete): .../restconf/config/opendaylight-inventory:nodes 6. get Confirm flows: .../restconf/operational/opendaylight-inventory:nodes/node/{device_id}/table/0 Script command:
./ovs-scripts/get-total-found.sh
Check OVS status recursively ./ovs-scripts/get-total-found.sh check status from first rest call responded till ovs get 100,000 flows (add/remove switch time)
OpenDaylight OpenFlow Northbound benchmark testing result 1 Flow Remove Controller Time + Remove Confirm Time
1 Flow Add Controller Time + Add Confirm Time 160
200
140 120
150
100 80
100
60 40
50
20 0
time(s)/switch
0
15
31
63 ODL
127
255
511
1023
time(s)/switch
15
31
ONOS
63 ODL
200 Flows Add Controller + Add Confirm Time
127
255
511
1023
ONOS
200 Flows Remove Controller + Remove Confirm
120
100
100
80
80
60
60 40
40
20
20 0
time(s)/switch
0
15
31
63 ODL
127
255
ONOS
511
1023
time(s)/switch
15
31
63 ODL
127
255
511
1023
ONOS Page 12
OpenDaylight CPU and Memory usage 1 flow / REST call with 31 switches (OVS 2.4.0) while true; do (echo "%CPU %MEM ARGS $(date)" && ps -e -o pcpu,pmem,args --sort=pcpu | cut -d" " -f1-5 | tail) >> ps.log; sleep 5; done
ONOS & ODL CPU Java Usage
ONOS & ODL Memory Java Usage
1000
10
900
9
800
8
700
7
600
6
500
5
400
4
300
3
200
2
100
1
0
ONOS
%/ Time
1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105
0
1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105
%/ Time
ODL
24 cores, 64 GB RAM; y-axis unit in %
ONOS
ODL
Page 13
SoapUI test Use SoapUI to test a controller’s VTN performance by sending RESTful request and obeserve the time delay between send and respond. 1. Single network creation time 2. Single network modification time 3. Single network query time 4. Single network deletion time 5. and more...
Page 14
Spirent and IXIA test Using Spirent and IXIA as device emulator for several protocol. Mainly test on a controller‟s capacity, in single node and cluster environment. 1. 2. 3. 4. 5.
BASIC – Linear Topology, TCP Connection, No hosts SECURE – Same as above but with TLS Connections Complex – Grid Topology v.s. Linear Users - Adding Hosts per Switch Load – “Packet-Ins” to Controller
Protocols now available for testing: 1. OpenFlow 2. BGP-LS 3. PCEP 4. OVSDB 5. IS-IS 6. OSPF
Page 15
SDN Controller Test Category Northbound
Operation
REST API
CLI
WAN L2/L3 VPN, Optical, SDN IP/BGP ...
GUI
Intent Interface
DCN VTN, SFC, L2 Forwarding
Yang Shell
RPC
SDN/NFV Proxy ARP, Flow Analyzer, Load Balancer, Control Plane Manager
Scalability
HighAvailability
Performance
Data Store
Clustering
NerConf
OVSDB
BGP
P4
SNMP
OSPF
PCECC
IS-IS
Consistency
Abstraction
Security
Reliability
OpenFlow
PCEP
Controller
Southbound (Protocol)
Page 16
Current testing situation and Future test planning 1. Impact from OpenDaylight‟s previous performance testing report. Because of ODL‟s report in March falsely test with ONOS 1.5.0 rc2 and compared the data with a stable version of OpenDaylight, which gave people an impression that ONOS‟s performance is worse than OpenDaylight. 2. Cperf: link Cperf is an OPNFV Project opened for SDN Controller‟s performance test. Current most members in the project are from ODL. Currently their work is to integrate ODL testing with OPNFV‟s installer. 3. Community support 1. Different institution have their own perspective towards how to test a SDN controller. Making the testing result looks different . We need a community work together to find a command way for SDN controller testing. 2. No much institution have knowledge on all SDN controller‟s operation. Community will be a good option to share a standard operation procedure. 3. The option of test tool is limited. Community can help to gathering resource and support for various testing cases’ demand. 4. SDN is evolving, standard and test methodology can be changed during daily developing and practicing, there is no the one way to test and benchmark SDN controller all way long. By communicating in one community, and build up a mutual-agreed test operation guideline and test methodology, can providing users a clear view of SDN controller’s performance and potential.
4. Full test case list page 18 to 20. Page 17
Benchmark Planning: Performance Performance Southbound Flow Injection Performance Protocols NETCONF/OVSDB/SNMP Configuration Injection BGPLS/PCEP Topology Discovery BGP/ISIS/OSPF Route Detection Operation
WAN DCN Cloud VPN Other RESTful API
Topology
Physical Network Topo Discovery Time Physical Network Topo Change Detection Time Logical Network Topo Discovery Time Logical Network Topo Change Detection Time
Page 18
Benchmark Planning: Scalability Scalability
Cluster Size
Cluster Size supporting
Network Topology Size
Physical Topo Size Logical Topo Size
Southbound session Size
Configuration Broadcast OpenFlow, NETCONF, OVSDB, SNMP Topology Discovery: BGPLS, PCEP Route Detection: BGP, IS-IS, OSPF
Routing Table Size
PCEP LS OpenFlow BGP IS-IS OSPF
Operation Size
DCN WAN Cloud VPN Other operation scalability performance Page 19
Benchmark Planning: Reliability, Security and Maintainability Reliability
Topo Breakdown
OpenFlow PCEP
Security
Cluster Breakdown
Cluster Recovery Time
Exception
Exception Handling
Attack defence
Defense for DOS attack
Page 20
