RDMA in the Cloud: Enabling high-bandwidth, low-latency communication in virtual environments for HPC

Josh Simons VMware Office of the CTO © 2014 VMware Inc. All rights reserved.

Server Virtualization Virtual Machine (VM)

Traditional Architecture

Virtual Architecture

2

Secure Private Cloud for HPC Research Group 1

Research Group m

Users

IT Hybrid/Public Clouds

VMware vRealize Automation User Portals

Blueprints

Security VMware vRA API

Research Cluster 1

Research Cluster n

NSX

Programmatic Control and Integrations

VMware vCenter Server

VMware vCenter Server

VMware vCenter Server

VMware vSphere

VMware vSphere

VMware vSphere

Task Parallel Performance

4

Testbed Configuration • Hardware – Four two-socket HP DL380 G8 servers (3.3 GHz E5-2667v2 CPUs; 128 GB) – Dual-ported Mellanox FDR / 10 Gb RoCE adaptor – Mellanox 12-port FDR switch

• Software – ESXi 5.5u1 hypervisor

– RHEL 6.5 (native and guest) – MLNX OFED 2.2.1

BioPerf Benchmark Suite Native to Virtual Ratios (Higher is Better) 1.2

1 CLUSTALW GLIMMER GRAPPA HMMER PHYLIP PREDATOR TCOFFEE BLAST FASTA

0.8 0.6 0.4 0.2 0

ESXi5.5u1

6

BLAST Native to Virtual Ratios (Higher is Better) 1 0.8

OMP_NUM_THREADS=1 OMP_NUM_THREADS=4

0.6

OMP_NUM_THREADS=8 OMP_NUM_THREADS=16

0.4 0.2 0

ESXi5.5u1 7

RDMA Performance

8

Kernel Bypass Model application

user

user

application

rdma

kernel

sockets tcp/ip driver

guest kernel

rdma sockets tcp/ip driver vmkernel hardware PHYSICAL

hardware VIRTUAL 9

FDR InfiniBand Read Latency ib_read_lat / passthrough

Half Round trip Latency (µs)

2048 1024 512 256 128 64 32 16

3 2.5 2 1.5

Native

1

0.5

8

ESXi 5.5

4 2 1 0.5

Message Sizes (Bytes) 10

HPC Challenge Benchmark (HPCC) Native to Virtual Ratios (Higher is Better)

1.0

1.0

0.8

0.8 n4np4 n4np8

0.6

n4np4 0.6

n4np16 0.4

n4np32 n4np64

0.2

0.4

n4np8 n4np16 n4np32

0.2

n4np64

0.0

0.0 N5000

N10000

N20000

High Performance LINPACK

11

NAS Parallel Benchmarks (NPB) Native to Virtual Ratios (Higher is Better)

NAS Parallel Benchmarks (Class C) 1.2 1 0.8

n4np4 n4np8 n4np16 n4np32 n4np64

0.6 0.4 0.2

0 IS

EP

CG

MG

LU

12

NAMD Native to Virtual Ratios (Higher is Better)

1 0.8

n4np4 n4np8 n4np16 n4np32 n4np64

0.6 0.4 0.2 0

Apoa1

f1atpase

13

NWCHEM Native to Virtual Ratios (Higher is Better)

1.2 1 n4np4 n4np8 n4np16 n4np32 n4np64

0.8 0.6 0.4 0.2 0

H2O7 MP2

14

Source: NWChem Performance Benchmark and Profiling, HPC Advisory Council 15

FDR InfiniBand Read Latency: Future ib_read_lat / passthrough / polling completions

2048

512

3

256

2.5

128 64 32 16 8

2 1.5 1

Native

0.5 2 4 8 16 32 64 128 256 512 1024

Half Round trip Latency (µs)

1024

Prototype

4 2 1 0.5

Message Sizes (Bytes) 16

RDMA Storage Performance

17

Remote Storage Access Path

app

device driver

PCI device

app

app

OS storage server

HW

switch 18

Passthrough Mode Limitation app

Guest OS driver

PCI device

Guest OS

Guest OS

ₓ ₓ

storage server

hardware

switch 19

Single-Root I/O Virtualization (SR-IOV)

Guest OS VF driver

PF driver

PCI device

Guest OS VF driver

Guest OS VF driver

vmkernel

storage server

hardware

switch 20

FDR InfiniBand Read Latency ib_read_lat / passthrough, SR-IOV

2048

Half Round trip Latency (µs)

1024

3

512

2.5

256

2

128

1.5

64

1

32

Native

0.5

16

ESXi 5.5

8 4 2

ESXi 5.5 with SR-IOV

1 0.5

Message Sizes (Bytes)

21

IOR Bandwidth Performance 3VM x 4core versus bare-metal Linux 12core 4000 3500

BW [MB/sec]

3000

Two-socket (8-core) IVB 64 GB memory MLX ConnectX-3 FDR IB 256 GB IOR dataset CentOS 6.4 Lustre 2.6

2500

VM write

2000

VM read

Bare Metal write

1500

Bare Metal read

1000 500

0

1

2

3

6

12

Data provided by Sorin Faibish, EMC Office of the CTO

No. of Procs

22

Summary • Virtualized HPC performance for throughput applications generally very

close to bare-metal (well under 5% overhead)

• Passthrough RDMA can deliver close to native performance for some

MPI benchmarks and applications

– Will  continue  to  improve  as  latency  overheads  are  reduced…or  eliminated – Higher-scale testing

• SR-IOV can enable access to RDMA-connect parallel file systems from

virtual environments with good performance

23

Thank You Josh Simons [email protected]

RDMA in the Cloud - VMware's Office of the CTO

RDMA Storage Performance. 17. Page 18. Remote Storage Access Path. 18. OS. HW. PCI device device driver app app app switch storage server ...
Download PDF
1MB Sizes 1 Downloads 144 Views
Report

Recommend Documents

'Cloud' Hanging Over the Adoption of Cloud Computing in Australian ...
Dec 11, 2016 - of what the term cloud computing means and its benefits; the 23% of .... all wireless and wired systems that permit users in sharing resources.
'Cloud' Hanging Over the Adoption of Cloud Computing in Australian ...
Dec 11, 2016 - In Australia, cloud computing is increasingly becoming important especially with the new accessibility provided by the development of the ...
The Ik in the Office
one that holds lessons for us all: The Ik were unable to cope .... elementary schools, high schools, colleges ...... grancy in the Central Florida orange groves, Life-.
Rethink Technology In The Age Of The Cloud ... Services
browser has become a central access point for communication and collaboration in the cloud. ... comprise 26% of today's information workers and are likely to be a high growth segment as .... Senior Market Impact Consultant. Contributing ...
Rethink Technology In The Age Of The Cloud ... Services
with a standard-issue laptop or workstation. For example, the ... frontline workers, to warehouse/logistics teams, to digital business professionals. ... can use their universal account information to log into any device. ITDMs appreciate the ...
In continuation of this Office Proceedings in the ...
17 Medak. 46. 518. 518. 18 Nizamabad. 36. 544. 533. 19 Adilabad. 52. 711. 711. 20 Karimnagar. 57. 0. 510. 21 Warangal. 51. 331. 331. 22 Khammam. 46. 215. 215. 23 Nalgonda. 59. 182. 182. Total. 1128. 8728. 8248. Sd/-. Dr. Y. Ali Akbar Basha ASPD II. F
Grant of Compensatory Time-Off (CTO) or Service Credit in the ...
Grant of Compensatory Time-Off (CTO) or Service Cre ... Mass Training Workshop on Common-General Topics.pdf. Open. Extract. Open with. Sign In. Main menu. Displaying Grant of Compensatory Time-Off (CTO) or Service Credit in the Engagement to the Seni
pdf-1424\in-the-land-of-the-long-white-cloud-in-the ...
... the apps below to open or edit this item. pdf-1424\in-the-land-of-the-long-white-cloud-in-the-land-of-the-long-white-cloud-saga-book-1-by-sarah-lark.pdf.
Cto._Andalucia_Infantil.pdf
Semifinal 1 28/05/2016 18:10 Viento: +0.0. 1 145 Pablo Villena Martin. Cueva de Nerja-UMA. 13/03/2003. MA7405. 5 10.12 Q. 2 54 (t) Oscar Benitez Escribano.
cto. benjamin.pdf
Retrying... Download. Connect more apps... Try one of the apps below to open or edit this item. cto. benjamin.pdf. cto. benjamin.pdf. Open. Extract. Open with.
CTO Sessions.pdf
Page 1 of 4. TCEA www.tcea.org. TCEA 2018 Convention & Exposition Sessions. for CTOs/Technology Directors. Monday and Tuesday. 8:00 – 5:00 Transformational Learning Academy Hilton. Leaders must create a culture of change for all to embrace digital.
RO8_OM_s2017_028 - Designation as Office-In-Charge of the ...
RO8_OM_s2017_028 - Designation as Office-In-Charge of the CLMD on June 13-16, 2017.pdf. RO8_OM_s2017_028 - Designation as Office-In-Charge of the ...
Planning for eDiscovery in the Cloud - Media13
Intel IT is implementing our technology roadmap for using hybrid cloud ... practices that create business value and ... roadmap for the use of hybrid clouds—a.
Planning for eDiscovery in the Cloud - Media13
files one at a time within a one- to two-week .... SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY ... laptops, desktop PCs, or in the cloud.
Google Apps: Energy Efficiency in the Cloud
cloud is also remarkably efficient.1 Last year we released a paper on the energy ... actual savings achieved by the U.S. General Services Administration (GSA), ... Increases energy 2–3% from use of Google servers and more network traffic.
OFFICE OF THE DIRECTOR OF GOVERNMENT ... - gunturbadi
Tick ( ) against Subjects for which Photostat copy is required. (Specify ... Signature of the Headmaster with Office Seal. (Instructions/Guidelines – see overleaf). /05/2014. Affix Latest Passport size Photo of the candidate duly attested ... 7) En
Automate Configuration of Application Networking Services in the Cloud
Page 1 ... and monitoring of applications and services in the cloud is not easy. ... networking services management solution for F5 BIG-IP®–enabled cloud ...
Securing elasticity in the cloud
code or system changes could result in business-impacting (or .... client interface such as a Web browser. (for example ... of file-based encryption within a vir-.
THE CITY OF MOBILE, ALABAMA OFFICE OF THE MAYOR
Office of the Secretary of the Navy. 1000 Navy Pentago. Room 4D652. Washington ... BOX 1827 . MOBILE, ALABAMA 36633—1827 . PHONE (251) 208—7395 ...