A Sufficiency-Based Resource Reallocation Mechanism for Internet-Scale Computer Resource Sharing Masahiro Kawato, Fumio Machida, and Yoshiharu Maeno Internet Systems Research Laboratories, NEC Corporation {m-kawato@ap, h-machida@ab, y-maeno@aj}.jp.nec.com

Abstract In systems from which multiple resource providers dynamically provide computer resources over the Internet, a decentralized resource allocation mechanism is essential. In such an environment, resource providers and resource users have different requirements. Satisfying the requirements of both is difficult because there is no centralized resource broker. To resolve this issue, we propose a sufficiencybased, on-demand resource reallocation mechanism that permits reallocation of resources when a current resource user has a higher utility level than that of a resource requestor. We have implemented this mechanism and used it to allocate resources in a simulated environment. Our experimental results have shown that sufficient resources were provided to 48 percent more users by applying our mechanism as compared with a similar mechanism without reallocation.

1. Introduction Many internet-based systems, such as WWW and video streaming, require computer resources in proportion to accesses from end users. Since access frequency dynamically changes over time, fixed server allocation for applications is not cost-effective. One solution is to reconfigure resources for different applications based on a predefined schedule. Unfortunately, this solution does not work if access patterns cannot be predicted in advance. An alternative solution is on-demand server provisioning, that is, to dynamically allocate computer resources when demand for resources increases. As shown in Figure 1, on-demand server provisioning can be generalized as a utility Grid. In the utility Grid model, virtual organizations (VOs) [5] are dynamically composed from resources provided by a number of resource providers (RPs).

Proceedings of the 2005 Symposium on Applications and the Internet (SAINT’06)

0-7695-2262-9/06 $20.00 © 2006 IEEE

A Virtual Organization (VO)

VO Agent VO Agent

Agreement Agreement

The Internet

Resource Allocation

RP Agent RP Agent

Computer Resources (e.g. web servers, streaming servers)

RP Agent

A Resource Provider (RP)

Figure 1: Model of global utility Grid. The RPs and VOs in a utility Grid have different requirements for resource allocation. For RPs, high resource utilization is desirable. In other words, it is better if the number of free resources is smaller. On the other hand, for VOs, it is desirable to be able to constantly acquire sufficient resources for their applications To satisfy both these requirements is not easy because installing a centralized resource broker would create a performance bottleneck. To resolve this issue, we propose a sufficiency-based resource reallocation mechanism. In this mechanism, an RP tries to allocate its resource to a VO immediately after the VO requests for a resource. If the RP cannot provide the resource, it tries to reallocate a resource previously allocated to another VO. Reallocation succeeds only if the current resource user has a higher utility level than that of the resource requestor. The remainder of this paper is organized as follows. In Section 2, we will formalize the assumptions for underlying systems and state our goals. Then in Section 3 we will propose a resource reallocation mechanism to achieve our goals. In Section 4, we will present the design of our prototype system. Section 5 will describe the results of a performance evaluation of this prototype. Finally, we will discuss the relationship

of this paper to other work in Section 6, and we will conclude discussion of the present work in Section 7.

2. Assumptions and Definitions In this section, we will list some assumptions and definitions that will be used in the later sections.

2.1 Assumptions about Underlying Systems We assume a computer system in which resources are managed in a decentralized form. More specifically, we assume the following items. • A resource provided by an RP is allocated to a VO only if an agreement on resource provision has been established between the RP and the VO. • There is no centralized resource broker that mediates agreements between RPs and VOs. • Only the provider and the current user of the resource know the latest allocation state of a resource. The other entities can only refer to the allocation state with some time lag. The third assumption is derived from the first assumption and communication latency in the Internet.

2.2 Quantification of a System State In this paper, we formalize our performance criteria as Mean Resource Utilization (MRU) and Mean Sufficiency Rate (MSR). Here we define MRU and MSR. We quantify Resource Utilization (RU) as the ratio of busy resources (i.e., those allocated to some VO) to all the resources. We define MRU as the mean value of RU in a certain period. A traditional approach to quantifying the VOs’ utility is defining utility functions for the VOs and evaluating Pareto efficiency [8]. This approach is not suitable to our work without modification, because one allocation state might be more desirable than another, even when both states are Pareto optimal. For example, assume a system that consists of two resources and two VOs. In one state, one VO has two resources and the other has no resource. In another state, each VO has a resource. Assuming utility functions of the VOs are proportional to the number of resources, both the states are Pareto optimal. However, to consider the latter state better is more natural, since both the VOs can work in the latter state, while the second VO cannot work in the former state. Our solution to this issue is to define two levels of utility: minimum sufficiency and target sufficiency. A

Proceedings of the 2005 Symposium on Applications and the Internet (SAINT’06)

0-7695-2262-9/06 $20.00 © 2006 IEEE

VO with minimum sufficiency has the minimum amount of resources required to perform its applications. A VO with a target sufficiency has enough resources to satisfy its service level objective. Based on the definition of minimum sufficiency, we define Sufficiency Rate (SR) as the ratio of the VOs with minimum sufficiency to all the VOs. Additionally, we define MSR as the mean value of SR in a certain period. More formally, we define the performance criteria as follows. Definitions: • Resource state: a resource is free if it is not allocated to any VO; otherwise, the resource is busy. • Resource utilization: The number of busy resources at a time t . RU (t ) = The total number of resources

• Allocated resource amount: ARA[VOi ] is the number of resources allocated to VOi. • Resource demand: RD[VOi ] is the number of resources required for VOi to achieve the minimum sufficiency. • Target resource amount: TRA[VOi ] is the number of resources required for VOi to achieve the target sufficiency. • Satisfaction: VOi is satisfied if ARA[VOi ] ≥ RD[VOi ] . • Sufficiency rate: The number of satisfied VOs at a time t SR(t ) = . The total number of VOs • Statistics on system state:

MRU = MSR =

1 N 1 N

N

¦ RU (t ) (Mean Resource Utilization), k

k =1

N

¦ SR(t

k)

(Mean Sufficiency Rate).

k =1

3. Sufficiency-Based Resource Reallocation Mechanism In this section, we will describe the new decentralized resource reallocation mechanism that we developed to maximize the MRU and MSR introduced in the previous section.

Requestor

Coordinator

Current User

(1) Request for resource acquisition (2) Selection of resource providers (3) Request for resource provision

(4) Workflow generation and resource reservation (5) Request for resource release

(6) Decision making

(7) Response to release request (8) Response to provision request (9) Selection of a workflow (10) Request for workflow execution

(13) Response to execution request

(11) Workflow execution

(12) Resource release report

Figure 2: Sequence chart of the sufficiency-based resource reallocation mechanism. Our sufficiency-based resource reallocation mechanism consists of a negotiation protocol between agents, and a set of decision-making algorithms for those agents. The negotiation protocol is an enhancement to Contract Net Protocol [7]. In our protocol, there are three roles: requestor, coordinator, and current user. The requestor is a VO agent who sends a request to the coordinator for an additional resource. The coordinator is an RP agent who tries to allocate one of the RP’s resources to the requestor. If the resource selected by the coordinator is busy, the VO agent who has the resource is called the current user. In such a case, the coordinator proposes reallocation of the resource. If both the requestor and the current user agree to the proposal, the resource is reallocated to the requestor. Some steps in the negotiation protocol require decision-making rules. For example, the requestor requires a decision-making rule to select RPs to which requests are sent. We have designed a set of heuristics for decision-making. Section 3.1 will describe the entire sequence of the negotiation protocol. After that, we will discuss the decision-making rules in the mechanism.

3.1 Resource Reallocation Sequence Figure 2 shows a typical sequence of the sufficiency-based resource reallocation mechanism. This sequence assumes that a resource allocated to the current user is being reallocated to the requestor. This sequence consists of the following steps: (1) The requestor receives a request for acquiring a resource (resource acquisition request). This request

Proceedings of the 2005 Symposium on Applications and the Internet (SAINT’06)

0-7695-2262-9/06 $20.00 © 2006 IEEE

may be submitted by a human operator, or by an autonomic control mechanism. Here, we assume that one resource is requested, although more than one resource may be requested. (2) The requestor selects one or more RPs from which it tries to acquire a resource. Figure 2 shows only one RP as the coordinator, although the requestor may select more than one RP. Here an RP selection rule, to be discussed in Section 3.2, is applied. (3) The requestor sends the coordinator a request to provide an additional resource (resource provision request). (4) The coordinator selects a resource and generates a workflow to allocate the resource to the requestor. After that, the coordinator reserves the resource for that workflow. The resource selection rule applied here will be discussed in Section 3.3. (5) The coordinator sends the current user a proposal to release the selected resource (resource release proposal). (6) The current user decides whether to agree to the resource reallocation. We assume the current user decides to agree to the proposal. The decision-making rule applied here will be described in section 3.4. (7) The current user sends the coordinator a response to the resource release proposal. (8) The coordinator sends the requestor a response to the resource provision request. This response includes a workflow for reconfiguration of the resource selected in step (4). (9) The requestor decides whether to execute the workflow proposed by the coordinator. If the requestor has received workflows from multiple RPs, the requestor selects one of those workflows. The

workflow selection rule applied here will be discussed in section 3.5. (10) The requestor sends the coordinator a request to execute the workflow generated in (4) (workflow execution request). (11) The coordinator executes the workflow, and then releases the reservation made in (4). (12) The coordinator reports completion of the resource reallocation to the current user. (13) The coordinator reports completion of the workflow execution to the requestor as a response to the workflow execution request. The steps (5) through (7) and (12) above appear only when reallocation is applied. Otherwise, the coordinator sends a response to a resource provision request immediately after step (4).

3.2 RP Selection Rule of the Requestor A requirement for the RP selection rule (the step (2) in Figure 2) is that of selecting RPs that are most likely to provide a resource to the requestor. Our strategy is to give higher priority to selecting the RPs that have more free resources. Definitions and notations: • An allocatable resource is a free resource that satisfies a requestor’s requirements. • A reallocatable resource is a busy resource that satisfies a requestor’s requirements and that is not allocated to the requestor. • N alloc [ p, q] denotes the number of resources that are provided by an RP p and that are allocatable to the requestor q. • N realloc[ p, q] denotes the number of resources that are provided by an RP p and that are reallocatable to the requestor q. • N [q ] denotes the number of resources that the requestor q ties to acquire. RP selection rule: The requestor selects NumRP RPs according to the following preferences (NumRP is a predefined positive integer constant). (1) RPs that satisfy N alloc [ p, q] ≥ cfree N [q ] (cfree is a predefined positive constant) (2) RPs that satisfy N alloc [ p, q] ≥ 1 and N alloc [ p, q] + N realloc [ p, q] ≥ N [q] (3) RPs that satisfy N alloc [ p, q] + N realloc [ p, q] ≥ 1

Proceedings of the 2005 Symposium on Applications and the Internet (SAINT’06)

0-7695-2262-9/06 $20.00 © 2006 IEEE

According to the assumption given in Section 2.1, the requestor cannot directly obtain the values of N alloc [ p, q] and N realloc [ p, q] . The rule above assumes a requestor maintains a cache of the resource properties required to obtain these values.

3.3 Resource Selection Rule of the Coordinator In the step (4) of Figure 2, if the coordinator has at least one allocatable resource, the coordinator should select an allocatable one because, from the point of view of resource utilization, this choice is better than a reallocatable resource. Otherwise, the coordinator has to select a reallocatable resource. Regarding VOs’ utility, the current user of the reallocation target should have a higher utility level than that of the requestor. Based on the discussion above, we have designed a resource selection rule as follows: Notations: • user(r) denotes the current user of a resource r. The notations introduced in Section 3.2 also appear here. Resource selection rule: The coordinator selects the requested number of resources owned by the coordinator according to the following preference: (1) An allocatable resource (2) A reallocatable resource r that satisfies ARA[user(r)] >TRA[user(r)], under the condition ARA[q] < TRA[q] (3) A reallocatable resource r that satisfies ARA[user(r)] >RD[user(r)], under the condition ARA[q] < RD[q] According to the assumption presented in Section 2.1, the coordinator does not necessarily know the latest allocation state of all the resources allocated to user(r). Thus the coordinator needs to cache the allocation state of the resources allocated to user(r).

3.4 Decision-Making Rule for the Current User The current user, denoted as u, accepts a resource release proposal if ARA[u] > RD[u]. This decision does not depend on a cached resource allocation state, because we have assumed that a VO maintains the latest allocation state of its resources.

3.5 Workflow Selection Rule for the Requestor

If the requestor receives a positive response from just one RP in step (8), the requestor sends a workflow execution request to the RP. Otherwise, the requestor has to select a workflow to be executed. Regarding resource utilization, to select a workflow without reallocation is better than reallocating one. For this reason, the requestor gives higher preference to selecting a workflow without reallocation rather than selecting one requiring reallocation.

4. Prototype Implementation We have implemented our sufficiency-based resource allocation mechanism as a stand-alone Java program. Scheduler

VO Agent

VO Agent

VO Agent

Resource State DB

RP Agent

RP Agent

: Request,

RP Agent

The VO Agent is implemented with a simple message queuing system. Each VO Agent maintains a message queue in which incoming requests and responses are stored. Then, the agent fetches the stored messages one by one. In the current implementation, incoming messages are fetched on a first-in-first-out basis. The RP selection rule introduced in Section 3.2 requires two parameters: NumRP and cfree. The values of these parameters are specified by a configuration file.

4.2 RP Agent The RP Agent shares a common message queuing system common with the VO Agent. Similarly to the VO Agent, incoming requests are processed one by one on a first-in first-out basis. The resource selection rule described in Section 3.3 uses two parameters: RD and TRA. The values of these parameters are set by the Scheduler as will be described in Section 4.4.

: Response

Figure 3: Block diagram of our prototype implementation. As illustrated in Figure 3, the simulator consists of four types of modules, VO Agent, RP Agent, Resource State Database (DB), and Scheduler. The VO Agents and RP Agents in Figure 3 correspond respectively to VO agents and RP agents in Figure 1. The Resource State DB is an on-memory database that stores the allocation states of all resources. A single instance of Resource State DB is shared by all the instances of VO Agent, RP Agent and Client. The Scheduler is a module that issues requests and updates parameters based on a predefined schedule. This design is not strictly compliant with the assumptions described in section 2.1, since the latest allocation state of all the resources is shared by all the VOs and RPs. However, that does not seriously affect the simulation result because each VO Agent and RP Agent only refers to information of resources belonging to the VO and RP. All the instances of the modules run on a singe Java virtual machine; the modules communicate via method invocation. In the remainder of this section, we will describe the modules in detail.

Proceedings of the 2005 Symposium on Applications and the Internet (SAINT’06)

0-7695-2262-9/06 $20.00 © 2006 IEEE

4.1 VO Agent

4.3 Resource State Database The Resource State DB maintains properties of all the resources in a system. Each data item in the DB has four fields. • The ResourceID identifies a resource, • the OwnerID identifies the RP, • the UserID identifies the VO that is currently using the resource, and • the ReservationUserID identifies the VO that currently reserves the resource. The contents of the Resource State DB are obtained and stored from the configuration file at an initialization process. The values of UserID and ReservationUserID may be updated by RP Agents during execution.

4.4 Scheduler The Scheduler performs two types of tasks, issuing requests and updating parameters. Regarding parameter updates, the RD and TRA introduced in Section 2.2 are subject to parameter updates by the Scheduler. The value of RD is assigned so as to form a sine curve pattern. Parameters of a sine curve (i.e., mean value, amplitude, and cycle time) are specified for each VO in the configuration file.

A value of TRA is calculated by the following expression.

VO1

VO2

VO3

VO4

VO5

total

250

TRA[VOi] = cTRA * RD[VOi]. 200

Resource Demand

In addition to RD and TRA, the Scheduler automatically updates two parameters. Resource Amount To Acquire: RATA[VOi] and Resource Amount To Release: RATR [VOi]. These parameters are referred by the request issuing mechanism of the Scheduler. The values of these variables are assigned as follows:

150

100

50

RATA[VOi] = cRATA * RD[VOi], RATR[VOi] = cRATR * RD[VOi].

0 0

• If ARA[VOi] < RATA[VOi], the instance issues a resource acquisition request to the VOAgent instance of VOi. • If ARA[VOi] > RATR[VOi], the instance issues a resource release request. This request causes VOi to release one of resources allocated to VOi.

5. Performance Evaluation We have evaluated our resource reallocation mechanism with the simulator described in the previous section.

5.1 Configuration of the Simulation The simulated system consists of five VOs and five RPs. All the RPs have the same number of resources. All the resources are free at the initial state. As described in Section 4.3, each VO has its own demand pattern. Figure 4 shows the demand patterns used in the evaluation. We have moved the total number of resources from 50 to 500. These resources are equally divided among the RPs. For example, each RP has 10 resources when the total number of resources is 50.

200

400

600

800

1000

Elapsed Time [sec.]

Figure 4: Resource demand pattern for simulation.

To evaluate the effectiveness of our reallocation mechanism, we have performed the simulation with two algorithm variations. One is the full version described in Section 3. The other is the same as the full version except that reallocation is never applied. We will call the two variations with-realloc and withoutrealloc. In addition, we have assigned the following values to the parameters. • • • • •

NumRP = 2 (see Section 3.2), cfree = 2.0 (see Section 3.2), cTRA = 1.2 (see Section 3.3), cRATA = 1.2 (see Section 4.4), cRATR = 1.4 (see Section 4.4).

We think that they are typical values in real environments.

5.2 Results MRU [w/realloc]

MSR [w/realloc]

MRU [w/o realloc]

MSR [w/o realloc]

1.2 1 0.8

MSR/MRU

The values of the coefficients cTRA, cRATA, and cRATR are specified in the configuration file. Regarding request issuing, the Scheduler issues requests according to the following rules:

0.6 0.4 0.2 0 50

100

150

200

250

300

350

400

Total Number of Resources

Figure 5: Simulation results.

Proceedings of the 2005 Symposium on Applications and the Internet (SAINT’06)

0-7695-2262-9/06 $20.00 © 2006 IEEE

450

500

Figure 5 shows the values of MRU and MSR for different numbers of resources. Regarding MRU, withrealloc obtained better results than without-realloc when the number of resources was 200 or higher. Regarding MSR, with-realloc was better than without-realloc when the number of resources was 150 or larger. Especially, with-realloc obtained better MRU than that of without-realloc by 48 percent when the number of resources was 250 or larger. This result means sufficient resources were distributed to VOs by reallocation when the alternative mechanism without reallocation could not appropriately allocate resources. That demonstrates the effectiveness of our resource reallocation mechanism.

6. Related Work There are a number of works on resource allocation for multi-organizational computing resource sharing. Unity [2] is a multi-agent system in which agents are autonomously interacts to allocate resources. The motivation of this work is similar to ours. The main difference from our work is that decision-making is based on utility functions. The Condor Project [6] has developed a “Matchmaking” mechanism to match resource providers and resource requestors that advertise their own capacities and requirements. This work focuses on allocating resources that match requestors’ requirements, without considering resource utilization and allocation fairness. The GRid Architecture for Computational Economy (GRACE) [3] proposes a market-based resource trading model. This work is complementary to ours, for our mechanism does not include a market model. The Grid Resource Allocation Agreement Protocol Working Group (GRAAP-WG) in the Global Grid Forum is standardizing WS-Agreement [4], an agreement language for negotiating resource allocation. We are considering incorporating WSAgreement into our project as a negotiation protocol.

7. Conclusion We have designed and implemented an on-demand resource allocation mechanism with sufficiency-based reallocation. We have compared two variations of our mechanism, with reallocation and without reallocation. The result showed that the variation with reallocation achieved better resource utilization and utility. This result demonstrates the effectiveness of our approach. We are planning to pursue three issues as future work. The first is to develop a method of obtaining mechanism parameters for a particular system

Proceedings of the 2005 Symposium on Applications and the Internet (SAINT’06)

0-7695-2262-9/06 $20.00 © 2006 IEEE

configuration. The second is to apply our mechanism to real computing systems. We are planning to incorporate the resource allocation mechanism into Polimatica [1], our utility Grid testbed. The third issue is to enhance our mechanism to be adaptable to more complex models, such as cost-based agreement and advance resource reservation.

References [1] Yoshiharu Maeno, Masahiro Kawato, Shoji Nisimura, Fumio Machida and Tsunehiko Kamachi, “Polimatica: Abstraction for Customizable Private Virtual Organizations in Global Grids,” Proceedings of the IEEE International Conference on Web Services, pp. 674-681, San Diego, July 2004. [2] Gerald Tesauro, David M. Chess, William E. Walsh, Rajarshi Das, Alla Segal, Ian Whalley, Jeffrey O. Kephart, and Steve R. White, “A Multi-Agent Systems Approach to Autonomic Computing,” Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems volume 1, pp. 464-471, 2004. [3] Rajkumar Buyya, David Abramson, and Jonathan Giddy, “An Economy Driven Resource Management Architecture for Global Computational Power Grids,” The 2000 International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA), June 2000. [4] Alain Andrieux, Karl Czajkowski, Asit Dan, Kate Keahey, Heiko Ludwig, Jim Pruyne, John Rofrano, Steve Tuecke, and Ming Xu, “Web Services Agreement Specification (WS-Agreement),” Global Grid Forum draft document, 2005. https://forge.gridforum.org/projects/graap-wg [5] Ian Foster, Carl Kesselman, and Steve Tuecke, “The Anatomy of the Grid: Enabling Scalable Virtual Organizations,” International Journal of Supercomputer Applications, Volume 15, Issue 3, 2001. [6] Rajesh Raman, Miron Livny, and Marvin Solomon, “Policy Driven Heterogeneous Resource Co-Allocation with Gangmatching,” Proceeding of the 12th IEEE International Symposium on High-Performance Distributed Computing (HPDC-12), pp.80-89, 2003. [7] Reid G. Smith, “The Contract Net Protocol: High-Level Communication and Control in a Distributed Problem Solver,” IEEE Transactions on Computers, 29(12),11041113, 1980. [8] Rajkumar Buyya, David Abramson, Jonathan Giddy, and Heinz Stockinger, “Economic Models for Resource Management and Scheduling in Grid Computing,” The Journal of Concurrency and Computation: Practice and Experience (CCPE), Volume 14, Issue 13-15, pp. 1507-1542, Wiley Press, 2002.