G-lambda: An Interface for Bandwidth Reservation from Applications and Middleware Tomohiro Kudoh1, Michiaki Hayashi2, Akira Hirano3, Shuichi Okamoto4, Atsuko Takefusa1, Takahiro Miyamoto2, Yukio Tsukishima3, Tomohiro Otani4, Hidemoto Nakada1, Hideaki Tanaka2, Atsushi Taniguchi3, Yasunori Sameshima4, 1. National Institute of Advanced Industrial Science and Technology (AIST) 2. KDDI R&D Laboratories, 3. NTT Network Innovation Laboratories, 4. National Institute of Information and Communications Technology (NICT)

Page 1

Outline • • • • •

In-advance bandwidth reservation and Grid G- lambda project: GNS-WSI and architecture Experiment on a single-domain network Architecture for multi-domain network Experiment on a multi-domain network

Page 2

In-advance bandwidth reservation and Grid • Grid provides a single system image to users by virtualization of service infrastructure such as computing, data and network resources from multiple domains. • Users do not care about actual resources they are using. Grid middleware (such as planner, broker and scheduler) coordinates resources and provides virtual infrastructure. Software catalogs

user

Grid Grid Middleware Middleware virtualizes virtualizes resources resources

Computers Sensor nets

Data archives Page 3

Network service for Grid • To realize such virtual infrastructure for Grid, resource management is one of the key issues. • Grid middleware should allocate appropriate resources, including network resources, according to user’s request. • Network resource manager should provide resource management service to Grid middleware.

Network Service • A standard open interface between Grid middleware and network resource manager is required, but not yet established.

Page 4

Timing of resource provisioning •

On demand provisioning – Common in traffic engineering of network – End time of resource provisioning is undetermined

•

Batch scheduling – Most of the schedulers for computing resources use the batch model • Make a queue of jobs with priority, and execute jobs in the order • Good for resources managed by a single scheduler

•

In-advance reservation/scheduling – For resources provided by multiple providers, advance reservation is suitable • Each provider can control its own resources with a reservation table – Existing protocols such as GMPLS does not support advance reservation. • The routing function of GMPLS assumes on-demand provisioning of paths – Not very common for computing resources too. Page 5

Design of the network service interface • Web Services – Grid is being built based on Web Services technology – Network service should be provided as a “Web Services”.

• SLA support – Bandwidth, latency etc.

• In-advance reservation – Reserve bandwidth

Page 6

What is “Web Services”?

• Application components which can be accessed thorough open standard web protocols (XML, SOAP, etc.). • Web Services interface enables interaction between application components – Very high level interoperability among the components.

• A standard Web Services based open interface between Grid middleware and network resource manager is required Web Services Interface

Grid MW

Grid MW Grid MW

Network Service Interface Network Resource Manager Web Services

Page 7

Outline • • • • •

In-advance bandwidth reservation and Grid G- lambda project: GNS-WSI and architecture Experiment on a single-domain network Architecture for multi-domain network Experiment on a multi-domain network

Page 8

G- lambda project overview • Joint project of KDDI R&D labs., NTT, NICT and AIST. • G-lambda project has been started in December 2004. • The goal of this project is to establish a standard web services interface (GNS-WSI) between Grid resource manager and network resource manager provided by network operators.

Page 9

The G-lambda Team

• • • • • • • • • •

Tomohiro Kudoh Hidemoto Nakada Atsuko Takefusa Yoshio Tanaka Fumihiro Okazaki Satoshi Sekiguchi Hiroshi Takemiya Motohiko Matsuda Seiya Yanagita Katsuhiko Okubo

• • •

Shuichi Okamoto Tomohiro Otani Yasunori Sameshima

• • • • • •

Masatoshi Suzuki Hideaki Tanaka Tomohiro Otani MunefumiTsurusawa Michiaki Hayashi Takahiro Miyamoto

• • • • • • • •

Akira Hirano Yasunori Sameshima Wataru Imajuku Takuya Ohara Yukio Tsukishima Atsushi Taniguchi Masahiko Jinno Yoshihiro Takigawa

Page 10

System architecture User

Resource Coordinator CRM: Computing Resource Manager

GNS-WSI NRM

NRM: Network Resource Manager

CRM

CRM CRM CRM

Page 11

Two application components interact through GNS-WSI • Global Resource Coordinator – According to users’ request, reserves computing and network resources (lambda paths) in advance

• Network Resource Manager – Responses to the requests from GRS through GNS-WSI – Manages reservation database – Hides detailed implementation. Provide required bandwidth between end points. (Path virtualization) – When the reserved time arrives, activate paths

Page 12

GNS-WSI (Grid Network Service / Web Services Interface) • • • • •

Grid Network Service-Web Services Interface Interface to realize advance reservation of bandwidth Based on the Web Services interface technology Can be used for inter-domain coordination Polling-based operations – – – –

Advance reservation of a path between end points Modification of reservation (i.e. reservation time or duration) Query of reservation status Cancellation of reservation

• GNS-WSI2 – WSRF(Web Services Resource Framework) based interface • GT4 (Globus Toolkit 4) Java WS Core http://www.globus.org/toolkit/ – 2-phase commit Page 13

An example XML exchanged through GNS-WSI Booking request (netResourceReservation) Site IDs, Reservation Time, Bandwidth(, availability, latency) (Example: Site A-B, 15:00-18:00, 1Gbps) Reference of the requested reservation Get command status

Client (E.g. GRS, Resource Coordinator)

"Prepared"

NRM

Commit the requested reservation Get reservation status "Reserved"

Page 14

Service Parameters Parameter

Usage

Value

Remarks

Site ID (APoint, ZPoint)

ID to specify A and Z points

String

Name or ID of sites

bandwidth

Bandwidth of the resource

Positive integer (kbit/s)

latency

Latency between end points

Positive integer (msec)

availability

Network protection of network resource

Integer (-232 ∼ 232-1)

0 = Un-protected 1 = Protected

Reservation time (startTime, endTime)

Start time and end time of the reservation

xsd:dateTime

YYYY-MMDDTHH:MM:SSZ

localUsername

user name of certificate

String

GT4 GSI

reservationStatus

status of reservation

String

Created/Reserved/Activa ted/Released/Error

commandStatus

status of each command

String

Initial/Prepared/Committ ed/Aborted

resourceStatus

status of network resource

String

Available / NotAvailable

Page 15

Outline • • • • •

In-advance bandwidth reservation and Grid G- lambda project: GNS-WSI and architecture Experiment on a single-domain network Architecture for multi-domain network Experiment on a multi-domain network

Page 16

Experiment on a single-domain network User Program Grid portal ASP

Global Resource Coordinator

CRM: Computing Resource Manager

GNS-WSI NRM

AIST developed WSRF based Grid scheduler

NRM: Network Resource Manager

CRM

KDDI R&D labs. and NTT developed NRMs

CRM

CRM CRM

Cluster computer

JGN2 GMPLS network operated by NICT was used for experiments

AIST developed Computing Resource Manager Page 17

5

GUI

① User requests service via GUI, specifying the required number of computers and the network bandwidth needed

0.5

Gb

ps

Overview of Demonstration at iGrid2005

1

10 1Gbps

WSRF Grid Resource Scheduler (GRS)

GT4

GNS-WSI

Web Services I/F Computing Resource Manager (CRM)

GMPLS

Network Resource Management System (NRM)

JGN II Kanazawa

JGN II Fukuoka

KDDI Labs. Kamifukuoka

AIST Tsukuba

GMPLS Router Optical Cross-Connect Cluster Gigabit Ether ( X n streams)

JGN ⅡOsaka Research Center

JGN Ⅱ

②The computing resources and GMPLS network resources are reserved as the result of interworking between the GRS and NRM using GNS-WSI (Grid Network Service / Web Services Interface) ③ A molecular dynamics simulation is executed using the reserved computers and lambda paths. Ninf-G2 and Globus Toolkit 2 (GT2) are used at each cluster.

AIST Akihabara

Page 18

Grid Resource Scheduler (GRS) • A Grid scheduler developed by AIST – Implemented using GT4 (Globus Toolkit 4)

• According to users’ request, reserves computing and network resources (lambda paths) in advance – Accepts requests which specify required # of clusters, # of CPUs at each clusters, and the bandwidth between clusters. – GRS selects appropriate clusters by interworking between the NRM and multiple CRMs (Computing Resource Manager) Globus Toolkit 4 (GT4) • Globus Toolkit (GT) is one of most popular open source software toolkit for Grid. • GT supports functions including communication, user authentication, resource management. • Globus Toolkit 4 (GT4) is the latest version which uses Web Services technology Page 19

Network Resource Management System (NRM) • NRM developed by KDDI R&D Labs. was used. • Response to the requests from GRS through GNS-WSI • Hide detailed path implementation. Provide a path between end points. (Path virtualization) • Schedule and manage lambda paths. When the reserved time arrives, activate paths using GMPLS protocol.

Page 20

KAN

TKB KMF 180Miles

410Miles

40Miles

150Miles

AKB 250Miles FUK

OSA

Page 21

Demo Environment JGNⅡ Fukuoka

JGNⅡ Kanazawa

4

KAN

FUK

KDDI Kamifukuoka

16

4

KMF TKB 2

3

2 2

32 3

4 2

2 OSA

AKB

2

16

12 JGN Ⅱ Osaka

GMPLS Router Optical Cross-Connect

AIST Tsukuba

2

JGN Ⅱ

n

2

AIST Akiba

Cluster with n processors Lambda path (GbE)

Clusters distributed over six locations in Japan are connected over GMPLS network test-bed deployed by JGN II Page 22

Overview of the Demo Application •

A molecular dynamics simulation implemented with a Grid Middleware called Ninf-G2, that is developed by AIST, Japan – Ninf-G2 conforms the GridRPC API, a Global Grid Forum standard programming API for Grid – Uses Globus Toolkit 2 for job invocation and communication

• Simulation Scenario – Silicon and water reaction under stress

Global Grid Forum： A standardization body for grid related technologies Globus Toolkit： Infra-ware for the Grid Page 23

Demonstration

Page 24

Outline • • • • •

In-advance bandwidth reservation and Grid G- lambda project: GNS-WSI and architecture Experiment on a single-domain network Architecture for multi-domain network Experiment on a multi-domain network

Page 25

Architecture for multi-domain network • Multi-domain – Network consists of multiple domains – Each domain is managed by its own NRM

• Bandwidth between end points in different domains – Coordination among domains is required – Three models of coordination

Page 26

Three models of inter-domain coordination (1) NW Control Plane Layer inter-working (ex. GMPLS E-NNI) User Program

Global RC1 Resource Coordinator L. Local Resource Manager NRM1 Layer NW Control Plane Layer

User Program

User Program

User Program

RC2

RC3

RC4

NRM2

NRM3

NW data Plane Layer Domain 1

Domain 2

Domain 3

Page 27

Three models of inter-domain coordination (2) Local Resource Manager Layer inter-working User Program

Global RC1 Resource Coordinator L. Local Resource Manager NRM1 Layer NW Control Plane Layer

User Program

User Program

User Program

RC2

RC3

RC4

NRM2

NRM3

NW data Plane Layer Domain 1

Domain 2

Domain 3

Page 28

Three models of inter-domain coordination (3) Global Resource Coordinator Layer inter-working User Program

Global RC1 Resource Coordinator L. Local Resource Manager NRM1 Layer NW Control Plane Layer

User Program

User Program

User Program

RC2

RC3

RC4

NRM2

NRM3

NW data Plane Layer Domain 1

Domain 2

Domain 3

Page 29

Pros and Cons of the three models 1. NW Control Plane Layer inter-working (ex. GMPLS ENNI) – Pros: User do not have to care about “multiple domains” – Cons: GMPLS is an on-demand protocol and can not support advance reservation – Cons: Very close relationship between domains is required. May not be always possible for commercial service.

2. Resource Manager Layer inter-working – Pros: User do not have to care about “multiple domains”. – Cons: Requested NRM may make a reservation which is advantageous for the domain

3. Global Resource Coordinator Layer inter-working – Pros: User can control combination of domains – Pros: No under-layer interaction is required – Cons: User must have knowledge of inter-domain connection WE EMPLOYED THIS MODEL FOR INTER-DOMAIN CONNECTION Page 30

Outline • • • • •

In-advance bandwidth reservation and Grid G- lambda project: GNS-WSI and architecture Experiment on a single-domain network Architecture for multi-domain network Experiment on a multi-domain network

Page 31

Experiment on a multi-domain network (GLIF 2006) • G-lambda & Enlightened collaboration • “Automated” interoperability between network and computing resources in two countries’ grid computing research testbeds is shown – the first such experiment of this scale between two countries

• Integrated computing and communication technology – Automated simultaneous in-advance reservation of network bandwidth between the US and Japan, and computing resources in the US and Japan – World’s first inter-domain coordination of resource mangers for inadvance reservation • Resource managers have different I/F and are independently developed

Page 32

EnLIGHTened Project Overview •

•

•

Established in 2005, is a NSF seed-funded collaborative interdisciplinary research initiative that seeks to research the integration of optical control planes with Grid middleware under both highly dynamic and advanced reservation application requests. Team: MCNC, LSU, NCSU, RENCI, Cisco, AT&T, Calient The focuses are on research and integration of cross-layer (applications, Grid resource co-scheduling, and optical network control plane) and interactions between Management , Control plane and Grid middleware. The goal of the Enlightened research project is establishing dynamic, adaptive, coordinated, and optimized use of networks connecting geographically distributed high-end computing and scientific instrumentation resources for faster problem resolution.

Page 33

The Enlightened Team • • • • • • • • • • • • • •

Yufeng Xin Steve Thorpe Gigi Karmous-Edwards John Moore Carla Hunt Lina Battestilli Andrew Mabe Trevyn Leighton Ray Suitte Shane Rockriver Bonnie Hurst Avery Smith Syam Sundar Phil Misenheimer

• • •

Jon Maclaren Andrei Hutanu Lonnie Leger

• •

• •

Savera Tanwir Harry Perros

Olivier Jerphagnon John Bowers

Page 34

Application Layer

GL Application

Global Resource Coordinator Layer Local Resource Manager Layer

App. GUI

GL Application

HARC Acceptor

GRS

RM1

App. Launcher

RM2

GRC

RM3

NW Control Plane Layer

NW data Plane Layer

Domain 1

Domain 2

Domain 3

Page 35

Application Layer

GL Application

Global Resource Coordinator Layer Wrappers

App. GUI

RM1

App. Launcher HARC Acceptor

GRS

GNS-WSI

Local Resource Manager Layer

GL Application

EL→GL EL→GL wrapper GNS-WSI wrapper

RM2

GRC

HARC RM I/F

RM3

NW Control Plane Layer

NW data Plane Layer

Domain 1

Domain 2

Domain 3

Page 36

G-lambda/Enlightened middleware coordination diagram

KDDI NRM

CRM

CRM

Cluster Cluster

GNS-WSI GL→EL GL→EL CRM GNS-WSI wrapper wrapper

NTT NRM

JAPAN

GL Grid Resource Scheduler

US Application

EL App. Launcher

EL→GL wrapper

US

Japan Application

EL Grid Resource Coordinator HARC Acceptor

EL NRM

CRM

CRM

CRM

CRM

CRM

Cluster

Cluster

Cluster

Cluster Cluster

CRM Cluster

Request Network bandwidth and Computers

CRM

CRM

Cluster Cluster

Application (Visualization) Request Network bandwidth and Computers

Reservation From xx:xx to yy:yy KDDI NRM

US

JAPAN

Application (MPI)

Reservation From xx:xx to yy:yy EL NRM

NTT NRM

CRM

CRM

CRM

CRM

CRM

Cluster

Cluster

Cluster

Cluster Cluster

CRM Cluster Page 38

Resource map CH1 (SL)

KMF FUK

VC1 (NCSU)

RA1 (MCNC)

BT1 (LSU) Pelican

Back up

TKB 6509

(UR1) 0.11a.6

(UO2) (UO1)

4G

KAN

X1N 5G 5G

X2N 2G

(UO3)

X1U

(UR2)

US

10.16a.2

(UR3)

X1S

X2

LA Foundry

X2S AKB

OSA

Japan South

BT2 (LSU) Santaka

0.11a.2

(UO4)

X1

Japan North

0.11a.7

BT3 (LSU) Viz Machine Client

LA1 (Caltech)

NR3 KHN Page 39

Demo overview 1. G-lambda makes a reservation. Reservation status will be shown. 2. Enlightened makes a reservation. Reservation status will be shown. 3. When the reserved time arrives, applications start running. 4. Activated paths and computing resources will be shown on MonALISA and RNDS. - Enlightened Viz client will show a blackhole.

5. (optional) reserve one more job from G-lambda

Page 40

Enlightened: Visualization of remote data •Data generated by remote simulation •Here : a black hole simulation •Need to explore and visualize the dataset •Enhanced Amira visualization system to take advantage of optical networks

Page 41

Enlightened: Distributed data server •Data available at multiple sites •Distribution can be beneficial (parallelism, caching options, executing simple operations) •A distributed data server (using the optical networks) can be faster than the local disk

Page 42

G-lambda: QM/MD simulation • Surveying a chemical reaction path by Nudged Elastic Band method – calculating system configurations during the reaction in parallel

End point

Start point

DEMO

Thank you

G- lambda project

http://www.g-lambda.net/

Page 44