CloudBridge: On Integrated Hardware-Software Consolidation Aritra Sen Ankit Garg Akshat Verma Tapan Nayak IBM Research - India I.I.T Delhi IBM Research - India IBM Research - India

ABSTRACT Data center consolidation has emerged as an important tool to improve the hardware utilization of data centers and reduce delivery costs. Consolidation has traditionally used virtualization to consolidate multiple workloads as different virtual machines running on a shared physical server. Consolidation leads to reduction in the hardware and the facilities cost (e.g., space and energy) but does not reduce software maintenance cost, which is often proportional to the number of instances of the software. The ever-increasing proportion of software cost (labour and license) in data center operations make software consolidation an important tool in data centers. In this work, we investigate the problem of data center consolidation with the goal of minimizing the total costs in running a data center. We build on existing work on virtualization-driven hardware consolidation and software consolidation to design CloudBridge that reduces total data center cost. CloudBridge uses an algorithm for finding the optimal software consolidation level for workloads consolidated on a physical server. Further, it intelligently optimizes the tradeoff between hardware and software consolidation to identify suitable workloads that should be co-located on a shared physical server. We present both theoretical and experimental evidence that establishes the effectiveness of CloudBridge.

1.

INTRODUCTION

The ever-increasing costs of owning and managing data centers have forced enterprises to explore innovative ways to reduce data center costs. Virtualization-driven consolidation and cloud computing have emerged as some of the most promising technologies to reduce the costs of running a data center. Virtualization allows enterprises to run multiple applications on a shared physical server, leading to a significant reduction in the number of physical servers deployed in the data center. Virtualization has led to the emergence of clouds as compute platform for the future. Clouds use virtualization to consolidate applications on fewer physical servers and use on demand resource allocation to reduce the physical footprint of the data center. In a typical data center operation, the costs are often broken into facilities cost (power, space, cooling costs), hardware costs (server, storage and network), server labour costs, software cost (license cost of system, middleware and application software), software labour costs, and implementation costs (planning, project, deployment). Fig. 1 provides a typical breakup of the various data center costs (Source: [16]).

Facilities (26) Hardware (5)

Software Labor (20)

Server Labor (17) License (9) App Implementation (23)

Figure 1: Data Center Cost Components

The left part of the pie chart consists of cost components that are dictated by the physical footprint of the data center and constitutes 48% of the total cost. Virtualization-driven hardware consolidation reduces the overall server footprint directly reducing hardware costs and indirectly reducing the facilities and hardware labour costs [22]. Hence, a lot of research has focussed on the problem of reducing the physical footprint of the data center [9, 10, 15, 20, 23, 25]. A close look at the cost breakup reveals that more than 50% of data center cost is incurred in software licenses, maintenance and deployment. Software licensing models typically fall into one of the following three categories (i) usage-based cost (e.g., Oracle Enterprise DB, IBM DB2) (ii) cost per physical core (VMWare ESX, IBM DB2 [3]), and (iii) cost per instance (e.g., Windows XP [1]). The license cost for the first model depends on the number of users or throughput served by the software and would not be impacted by hardware or software consolidation. In the second model, software costs are determined by the number of cores, or a normalized processing capacity (e.g., Processor Value Unit [8]) associated with the cores, used to run the software. Software with this kind of license model may benefit due to hardware consolidation since the number of physical cores is reduced. The cost of software following the third licensing model can be reduced by software consolidation but not by hardware consolidation. Software labour costs for each software are determined primarily by the number of software instances deployed in the data centers and constitutes about 20% of data center cost [16]. Software consolidation or reducing the number of instances of a software in a data center directly reduces the software labour cost. Further, the application implementa-

tion costs are impacted significantly by the number of application instances that need to be configured. Hence, software consolidation significantly reduces the application implementation costs. With increasing penetration of virtualizationdriven hardware consolidation and the profusion of software images with the emergence of cloud, the labour cost of maintaining the configuration of a large number of software instances in a data center is only going to increase. In this paper, we use the term hardware-related cost to capture the components of data center cost that are determined by physical footprint and software-related cost to capture the components of data center cost that are determined by the number of software instances deployed in the data center. Software consolidation fundamentally differs from hardware consolidation in 2 key aspects. Firstly, hardware consolidation is performed at only 1 level, which is at the level of virtual machine images. On the other hand, a typical application consists of multiple layers of software (operating system, middleware, application) and software consolidation can be performed at one or more of these layers. The associated software labour savings would vary depending on the software layer at which the consolidation is done. Secondly, virtualization enables a low-cost mechanism to create virtual server images that can be migrated on target physical server. Hence, hardware consolidation can be performed at very low price-points. On the other hand, consolidating 2 workloads at a specific software level l requires the migration of all software stack above l to the other instance. Hence, there may be a sizeable migration cost of software consolidation, which again varies based on level of software consolidation.

1.1 Motivation CRM_Web1

CRM_Web2

WPS 6.1

WPS 6.1

WAS

WAS

Aix 6.1

Aix 6.1

IBM APV

IBM APV

SS: 1 Aix, APV, WAS,WPS MC: 2 CRM Migration HC: 1 server

the portal servers, we save on the software-related costs for WPS and all layers below it (OS, WAS and WPS savings). However, we need to migrate the custom CRM application from one WPS instance to another and incur the migration cost for CRM. In both these cases, we consolidate 2 servers to 1 server and save the hardware-related cost for 1 server. It is easy to observe that the optimal consolidation strategy would vary based on the cost of migration and the softwarerelated costs. Further, the software consolidation strategy should place those workloads on a server that provide more potential to reduce software-related costs. Finally, as we show later, the best software consolidation strategy may not be ideal for hardware consolidation. This large search space makes finding an integrated hardware-software consolidation strategy, which minimizes total data center costs, a very challenging problem.

1.2 Contribution In this work, we present CloudBridge, an integrated softwarehardware consolidation strategy for data centers undergoing transformation. CloudBridge models each workload undergoing transformation using both its software stack as well as its utilization profile, allowing it to optimize the tradeoff between software and hardware cost savings. The utilization profile of workloads is used to find complementary workloads and reduce hardware-related costs, whereas similarity in software stack is used for software consolidation and reduce software-related costs. We also present BP 2, an algorithm to identify the optimal consolidation level for workloads placed on each server. We present a comprehensive experimental study to establish the effectiveness of CloudBridge to reduce the hardware and software costs for a cloud-based data center. We show that CloudBridge can reduce the total data center cost by an additional 20% over and above any savings due to baseline virtualization driven server consolidation.

2. MODEL AND PRELIMINARIES 2.1 Terminology

SS: 1 Aix, 1APV MC: 2 WAS Migration HC: 1 server

Figure 2: Software Consolidation Example We motivate the problem of finding an optimal software consolidation level with the following simple example. Consider a clustered web component of CRM application running on a portal server. The portal server is hosted on a web application server running on an Aix 6.1 operating system, which is managed by IBM Advanced Power Virtualization (APV) hypervisor. We can consolidate the two workloads at any of the 5 levels shown in Fig. 2. If we consolidate at the operating system level, we save the software-related cost for 1 OS and 1 hypervisor instance. The software-related savings (SS) would include labour cost, implementation cost and any potential license cost saved in managing one less instance. However, we need to migrate 2 WAS instances and incur the migration cost. Similarly, if we consolidate

We consider the migration of N workloads Wi hosted on standalone servers with operating system, one or more middleware stack, and an application instance to a cloud based data center. We indicate each layer of the application stack by a level l and denote L as the total number of levels. Additionally we have the zeroth level reserved for hypervisor.Typically, level l = 1 will be the OS and level L will be the application. Each workload can be migrated either at an image level (the operating system image is captured and restored on a target hypervisor), any middleware level (the application/higher-level middleware is replicated on a target middleware/operating system), or application level (workload is redirected to an instance of the same application on the target server). The cost of migration varies for each software type and is denoted by M Cg for software type Tg . To elaborate further, a websphere application server (W AS) may have inbuilt support to migrate hosted applications across servers whereas an apache-tomcat server may lack the support. Hence, the migration cost for W AS may be less than the migration cost for apache-tomcat. Similarly, the migration cost would vary across different levels (migrating virtual machines would usually incur less labour cost than application migration).

Workload Hardware-related Cost Hardware Savings

Wi HC HS

Num Workloads Software Type Consolidation Level

N Tg l

Target Server Migration Cost Num Levels

Tj M Cg L

Server Capacity Software-related Cost Software Savings

Cj SC SS

Table 1: Terminology Migration to cloud based data center leads to savings for the enterprise. The savings are in the form of reduced data center footprint (number of servers and accompanying power, space savings), reduction in software license and labour cost. The savings due to physical reduction in the number of servers can be modeled as reduced hardwarerelated cost HC. In our work, HC also includes any software license cost that is based on per-processor core software license model. We capture any costs that are related to the number of software instances as software-related cost SC. Finally, the consolidation requires migration of workloads at any one of the application stacks requiring both labour and tool support. We capture the total cost of consolidation (labour as well as tools) using the migration cost parameter M C. Consolidating k workloads at level l − 1 removes k − 1 software stacks each of height l − 1 and incurs migration cost of the k workloads at level l. We use hypervisor as the lowest level and the highest level L is a dummy level used to capture the migration cost induced by consolidating workloads at the application layer. The notations used in this work are summarized in Table. 1. Definition 1. Hardware-related Cost: Hardware-related costs are all data center costs that are determined by the hardware footprint of the data center. It includes server purchase and labour costs, power, cooling and space costs, as well as the license costs for software that are charged based on per-core usage. Definition 2. Software-related Cost: Software-related costs are determined by the number of software instances in a data center. It includes software labour and application implementation costs as well as the license costs for software that are charged based on per-instance usage.

2.2 Problem Formulation The goal of data center consolidation is to reduce the total cost of running the data center. The total cost of the data center is defined as sum of the hardware-related cost HC, software-related cost SC and the migration cost M C. The above problem can be captured as finding a mapping xi,j from workload Wi to server Tj , the consolidation level(s) yi,l for the workload, and the set of applications Si,l it is consolidated with. We also need to ensure that the performance of the workloads does not change due to consolidation (or no workload faces any resource constraints). The performance constraint for each workload hosted on a shared server Tj is captured by the following equation. X xi,j cti ≤ Cj , ∀t, j (1) i

cti

where denotes the resource required by the workload at time t and Cj denotes the capacity of the server. The above equation needs to be replicated for each resource used by the workloads (e.g., CPU, memory).

2.3 Modeling Migration Cost

Migration of workloads for consolidation at any layer requires one or more of tool support, labour, and downtime. In the case of enterprise applications, the cost of migrating an application to a consolidated environment may be significant. Usually if an application is migrated at a virtual machine layer, automated Physical to Virtual (P2V) tools can be used that recreate the application stack as a virtual machine on the target server. Hence, the migration can be done fairly quickly with minimal labour cost and downtime. The complexity of migration typically increases as we move up the application stack. Popular middleware usually provide tools for migrating middleware configuration and installed applications to a target server. The tooling cost and complexity increases with customization. Further, migration may introduce faults in the application requiring remediation, which is also included in the overall migration cost. The overall migration costs for migrating custom applications may be in the order of $10K or more. Hence, migration cost (M C) is a very important parameter in the overall consolidation strategy. Given lm as the license cost at level m, the total savings by consolidating k standalone workloads running on identical standalone servers T to a single server Tj at a common level l is given by the following equation. (k − 1)(HC(Tj ) +

l X

lm ) − k(M Cl+1 )

(2)

m=1

2.4 Solution Idea Our solution for finding an integrated software-hardware consolidation strategy leverages existing solutions for hardware and software consolidation. One of the best performing hardware consolidation strategy is the P CP algorithm [22]. P CP uses workload analysis to cluster workloads based on when they peak. By carefully placing workloads from all clusters on each server, P CP ensures that every server hosts workloads with complementary peak-trough patterns. The multiplexing of server resources in time between complementary workloads is the key intuition behind the P CP algorithm. Similarly, software consolidation strategies tend to group workloads with same software together and consolidate them. Hence, hardware and software consolidation are both based on grouping strategies, even though the grouping criterion is different. We present a 2-level algorithm to solve the twin problems of (a) finding the best placement of workloads on a server and (b) finding the best software consolidation level for the workloads placed on each server. Our top-level algorithm is also based on a grouping strategy of workloads. However, our algorithm groups workloads based on both complementarity in workload patterns as well as similarity in software stacks to obtain an initial solution. In order to deal with the scenario where the two objectives lead to different grouping patterns, we propose a locally optimal swapping scheme to refine our current solutions. Once our algorithm identifies the appropriate placement of workloads to servers,

WAS

TPC-W Web1 Rubis Web1

WAS

TPC-W DB1 DB2

DB2

WAS WINDOWS

TPC-W Web 3 Rubis Web 3

WINDOWS

TPC-W DB 2

Rubis DB1

TPC-C 1

TPC-W Web 2 Rubis Web 2

WINDOWS

DB2

LINUX

POSTMARK

TPC-W DB 3

Rubis DB 2

DB2 Rubis DB 3

CLUSTER 2

SERVER1 (Proportional Allocation)

TPC-C 2

DB2

LINUX

SERVER2 (Proportional Allocation) CLUSTER 3 CLUSTER 1

SWAP Candidate

Figure 3: Sorting based on application stack to capture affinity in PCP we use the low-level Recursive Software Consolidation algorithm BP 2 that finds the optimal consolidation level for the workloads placed on each server. We describe the toplevel algorithm to map workloads to servers in Section 3 and the per-server low-level software consolidation algorithm to identify the appropriate consolidation levels in Section 4.

3.

CLOUDBRIDGE METHODOLOGY

The goal of CloudBridge is to reduce the total hardwarerelated and software-related costs, while reducing the cost of migration. We now present an algorithm that identifies the placement of workloads to physical servers and uses the perserver BP 2 algorithm to identify the optimal consolidation level for each placed workload. Hardware-related costs are minimized by clustering workloads based on their peaks and ensuring that each server is allocated workloads from each cluster in a proportionate manner [22]. On the other hand, a placement that minimizes software-related costs would place workloads with similar software stack on the same physical server. This would give the low-level algorithm BP 2 more opportunities for software consolidation and reduce software-related costs. An algorithm to minimize the total cost would aim to place workloads on the same server that optimizes the tradeoff between hardware and softwarerelated costs. Our overall strategy for integrated consolidation consists of (i) introducing software affinity in PCP virtual machine consolidation algorithm (ii) model hardware-related savings as multiplexing savings (iii) use a locally optimal search that refines the output from modified PCP to achieve the tradeoff and (iv) use BP 2 on each server locally.

3.1 Introducing Affinity in PCP We first investigate the PCP algorithm to minimize hardwarerelated cost and enhance it to be software-aware. PCP creates clusters of workloads based on when they peak together. On each target server, it allocates workloads from all clusters in a proportional manner to ensure that every server has complementary workloads [22]. The workloads in each cluster are sorted by workload size to minimize fragmentation. The above sorting ensures that the first selected server have large-sized workloads. We use the above insight to modify the PCP algorithm and sort workloads in each cluster by their application stack. Since all clusters are sorted using the same ordering of software stacks and the placement on a server from each cluster is made using this order, it increases the likelihood that all workloads in a server have similar soft-

ware stack. Hence, the refinement makes PCP application aware and leads to reduced software-related cost. Consider the example in Fig. 3 with 3 workload clusters that can be placed on 2 target servers. In all the clusters, we place Windows workloads before Linux. Further, within windows workloads, we place web servers before database servers. As a result, all the web servers are hosted on the first target server and can be consolidated together. It is easy to see that a software stack oblivious ordering may not lead to a placement that groups applications with similar software stack on the same server. The exact changes in the PCP algorithm are listed below. 1. Cluster workloads based on the their peak clustering as done by PCP 2. Sort all workloads by Operating System 3. Within each Operating System, sort all workloads by middleware stack(s) 4. Within each middleware stack(s), sort all workloads by application type 5. For each target server, identify workloads from each cluster in a proportional manner as done by PCP

3.2 Modeling Multiplexing Savings The enhancement to PCP algorithm implicitly groups workloads with similar software stack on the same server. However, there may be cases where workloads with the same stack may be correlated and peak at the same time. Hence, in order to reduce the hardware-related costs, we would need to place them on different servers. On the other hand, placing them on the same server may help in reducing softwarerelated costs. The optimal algorithm in such a case would optimize the tradeoff between hardware-related and softwarerelated costs. The first step to optimize this tradeoff is to characterize the hardware-related cost in a way that we can compare it directly with software-related costs. Hence, we next model the hardware-related savings due to multiplexing workloads with complementary peaks on a server. Virtual machine consolidation algorithms usually size workloads based on the peak resource usage Cmax of the workload [20]. Sizing based on peak workload intensity often leads to relatively low utilization. The primary goal of a workloadaware hardware consolidation algorithm like PCP is to increase the average utilization of each server. The algorithm identifies workloads with complementary resource usage and

s

1 L

s

k L

BASE CASE WebApp1

WebApp2

WebApp1

WebApp2

WebApp1

WebApp2

WebApp1

WebApp2

WPS 6.1

WPS 6.1

WPS 6.5

WPS 6.5

JetSpeed2.1

JetSpeed 2.1

JetSpeed2.1

JetSpeed2.2

WAS

WAS

WAS

Apache

Apache

Apache

WAS

s

k

Apache

s

INDUCTION

l

k l

k

s l-1 Aix 6.1

Aix 6.1

Aix 6.1

Aix 6.1

Aix 6.1

Aix 6.1

Aix 6.1

Aix 6.1

IBM APV

IBM APV

IBM APV

IBM APV

IBM APV

IBM APV

IBM APV

IBM APV

Figure 4: Application Consolidation Example uses time-based multiplexing between them to improve the average utilization. Hence, it allows workloads to be sized based on their typical resource usage CB as opposed to peak resource usage Cmax . The savings using workload-aware multiplexing can be characterized by the difference between the body of the workload capacity distribution CB and the peak (i.e., Cmax − CB ). In order to get a dollar value for the savings, we multiply the resource savings by the hardware cost per unit capacity for the data center.

3.3 Local Optimal Search CloudBridge uses the placement given by software-aware PCP and refines the solution by swapping workloads that minimize the sum of hardware-related multiplexing cost and net software-related cost. At each step in the algorithm, we pick the swap candidates that lead to the largest decrease in the total cost of the data center. A possible swap candidate is shown in Fig. 3. The software-aware PCP algorithm places two instances of T P C-W application running on DB2 on two different servers. Swapping T P C-W DB2 with Rubis DB1 may help us consolidate the two T P C-W instances. However, it may lead to increased hardware cost. We go ahead with the swap if the decrease in software cost outweights any increase in hardware cost. Hence, the search terminates with a local optimal solution. The local search works in the following manner. We take the solution for the first server returned by software-aware PCP and label the selected workloads as S1 . The remaining workloads are labeled as S2 . Our goal is to find the optimal set of workloads from S1 that can be swapped with workloads from S2 to minimize the total cost. In order to achieve this, we pick the first k software-stacks in S1 and label it as S10 . We next find workloads from S2 with one of the k software stacks as possible swap candidates S20 . We next add the workload from S20 that leads to the largest total savings (hardware-related and software-related). We use the model for hardware-related savings outlined in the previous section. We vary k from 1 to the number of distinct software stack in S1 and run BP 2 to obtain the total savings for each possible placement. We finally pick the solution S1∗ with the highest savings for the server. We iterate the above procedure on the remaining servers by taking the workloads

S1∗ already placed and terminate when all workloads are placed.

4. RECURSIVE SOFTWARE CONSOLIDATION - BP2 ALGORITHM We present a recursive algorithm to find the optimal consolidation for a set of applications hosted on a shared server (proof in appendix). Let Pl be a partitioning set that separates applications into sets skl with similarity upto level l. To elaborate further, all elements in the set skl have identical first l workload levels/stacks. Further, any pair of elements from two distinct sets (e.g., an element from sk1 and anl other from sk2 l ) differ in at least one level at l or below (i.e. levels1, 2...l). To take an example, all workloads running identical workload stack (from 0 to L) are partitioned as PL and are part of a set skL . Similarly, the partition P0 leads to a single set s10 , which includes all workloads placed on the server. Fig. 4 shows an example consolidation scenario where each set of applications with identical stacks are shown between vertical lines (skL ). All applications are running on Aix 6.1 operating systems and run web applications using a portal server (Websphere Portal Server or JetSpeed) running on either a web application server (Websphere Application Server) or HTTP server (Apache). We use OP TL−1 (s) to denote the optimal solution for any set of applications s using any consolidation upto level L − 1. Our algorithm recursively uses the optimal solution on partitions at level l (OP TL−1 (skl )) to create the optimal solution on partitions at level l − 1 (OP TL−1 (skl−1 )). In Fig. 4, we show the induction step where the optimal solution for applications running W AS and the optimal solution for applications running Apache (skl−1 ) are used to find the optimal solution for all applications using Aix 6.1 (skl−1 ). The inductive algorithm works in the following manner. Base Case: OP TL−1 (skL ), ∀skL ∈ PL In the base case, we need to compute the optimal solution for each set skL separately. For each set skL and level l, we compute the total cost of running the data center under the assumption that all applications in skL will be consolidated at level l. We compute the optimal level for the set as the con-

solidation level lk∗ that leads to the least total cost. The final solution returned by the base case are the consolidation levels {l1∗ , . . . , lk∗ , . . . } respectively for the sets {s1L , . . . , skL , . . . }. We will later show (Lemma 2) that this solution is optimal for the individual sets. Inductive Step: Given OP TL−1 (skl ), ∀skl ∈ Pl , we compute OP TL−1 (skl−1 ), ∀skl−1 ∈ Pl−1 . In this step, we need to compute the optimal for each of the sets skl−1 . We first identify the elements of Pl , whose union leads to skl−1 . We define these sets at the partition level Pl as {p1k , . . . , prk , . . . }. The algorithm is based on the insight that any consolidation that are cost-effective in set prk are also cost-effective in the larger set skl−1 (formally stated as Lemma 3). However, since the set skl−1 has more candidates for consolidation, we may be able to save additional software cost that may make some more consolidations cost-effective. Our inductive algorithm has two sub-algorithms for the ’Leader’ scenario and the ’Leaderless’ scenario. If any of the sets prk has a consolidation at a level l − 1 or higher in OP TL−1 (prk ), we call this set a leader set. If at least one of the set prk is a leader, we have a ’Leader’ scenario. Otherwise, we have a ’Leaderless’ scenario. Since a leader set is able to consolidate at a level higher than the similarity level l (by itself), a leader set can derive no advantages from any other sets. On the other hand, the presence of a leader allows non-leader sets to leverage the leader’s consolidation and save software costs upto the similarity level l. The algorithm for the ’Leader’ scenario consists of the following steps. 1. Base Leveling: Base-leveling does not change the consolidation levels for the component sets but only combines existing consolidations over the larger set. Since all workloads have common stack upto level l − 1 in skl−1 , the goal of base-leveling is to use a common software instance across the different constituent sets prk for consolidations in OP TL−1 (prk ). We observe that for each non-leader set prk , OP TL−1 (prk ) will have consolidation at a common level blkr ≤ l − 1 (proved as Lemma 2). Base-leveling implies that all non-leaders prk can reuse the software instances used by one of the leaders upto their consolidation level in OP TL−1 (prk ) Pblrk i.e. (blkr ) and save software cost j=1 lj , where lj is the software cost at level j. Similarly, if there are multiple leaders, the leaders can also reuse the software of only 1 leader upto the similarity level l − 1. 2. Leader Piggybacking: The piggybacking step is performed only for the non-leader sets. In this step, the non-leaders use the existing consolidation of the leader to reduce their consolidation cost. Hence, they estimate if they can piggyback on the license used by the leader upto application level l−1 to find any additional consolidation candidates. One may note that since the different sets only have levels upto l − 1 common, piggybacking can only use the license of the leader till level l − 1. The piggybacking step formally consists of finding a piggyback level plkr (l − 1 ≥ plkr ≥ blkr ) that Pplrk maximizes the term f (plkr ) = j=bl r +1 lj −M Cplr +1 + k k M Cblrk +1 . 3. Promotion: The promotion step attempts to find any consolidation candidates that may have become

feasible due to the piggybacking step. Since the different sets prk do not have any common software beyond level l − 1, the promotion step is perfomed by each non-leader set separately. In this step, we explore cost-effective consolidation opportunities beyond the level l − 1 by framing a consolidation sub-problem from levels l − 1 to level L. Formally, we run the algorithm on a sub-problem with L − l + 2 levels for each prk , where the migration cost of the lowest level is 0. Further, migration cost at other levels is reduced by Pl−1 r j=plr +1 lj + M Cpl +1 . If the solution returned has consolidation at level 0, it implies that the application will continue to get consolidated at the piggyback level plr . However, if the solution returns a consolidation level lp > 0, then the application is migrated at the level l − 1 + lp for the set prk . LEADER

p k1

NON-LEADER

App1

App2 App1

App3

6

BI1

WPS

WPS

WPS

5

SPSS

l

WPS

bl

WAS 7.1

l-1

pl

2 bl

App1

App2 App1

App3

WPS

WPS

WPS

WPS

BI1

BI1

BI2

BI2

SPSS SPSS SPSS

WAS6 WAS6 WAS6 WAS6

2

HTTP HTTP HTTP HTTP

2

Windows

1

KVM

l-1

4 3

HTTP Windows

l

1

BI1

2

pk

KVM

BI2

SPSS

BI2

PROMOTION

WAS6

WAS 7.1 HTTP Windows

PIGGYBACKING BASE LEVELING

KVM

Figure 5: Example Algorithm Execution for ’Leader’ Scenario We illustrate the ’Leader’ scenario with an example in Fig. 5. We consider two sets that have common software till level l − 1 and are being merged. The first set is a ’Leader’ set and has a consolidation at level l. The base consolidation level for the two sets are at level 4 and 2 respectively. Hence, the Base Leveling step consolidated the two sets at level 2. The Leader set takes no further part in the consolidation. In the Leader Piggybacking step, we consider the cost of piggybacking on the leader’s license and consolidate at level 3. In this example, the cost savings are positive, and the non-leader set uses ’HTTP’ server license of the leader for consolidation. Finally, the promotion step is executed on p2k set, which leads to a consolidation at the SP SS software level. The algorithm for the ’Leaderless’ scenario consists of the following steps. 1. Base Leveling:This step happens identical to the ’Leader’ scenario with one exception. Since each set prk is a nonleader, all non-leaders use the common software for the r non-leader pr∗ k with the highest consolidation level blk and save software costs. The set pr∗ does not get any k additional benefit due to base-leveling.

2. Leaderless Piggybacking: The leaderless piggybacking step is similar to ’Leader piggybacking’. However, in this case, there is no leader to piggyback upon. Instead, we estimate if the larger set allows us to consolidate at a level higher than blkr . Initially we consolidate all sets to a level blk = maxr blkr . Then, we find a consolidation level plk (l − 2 ≥ plk ≥ blk ) that maximizes P k the function f (plk ) = (t − 1) pl j=blk +1 −tM Cplk +1 + tM Cblk +1 , where t is the number of applications in the set skl−1 . 3. Promotion: The promotion step again attempts to find any consolidation candidates that may have become feasible due to the piggybacking step. Here we compute a solution corresponding to each set prk assuming that this set is now equipped to play the role of a leader. For this we define the notion of a best solution for each set prk .For each set prk , we first compute a level plkr (l − 1 ≥ plkr ≥ plr ) that maximises Pplrk f (plkr ) = j=pl+1 lj − M Cplrk +1 + M Cpl+1 . Now as in the promotion step of Leader scenario, for each set plkr , we run the algorithm on a sub-problem with L − l + 2 levels. Again the migration cost at the lowest level will be zero and higher migration costs will be reduced by P l−1 r j=plr +1 lj + M Cplk +1 . If the solution returned has k consolidation at level 0, it implies that the application will continue to get consolidated at the level plkr . However, if the solution returns a consolidation level lp > 0, then the application is migrated at the level l − 1 + lp for the set prk . Now for each prk , we define a feasible solution as follows : if for some set pjk 6= prk , best solution for pjk contains consolidation at level ≥ l − 1, then consolidate all sets according to their best solutions. Otherwise all other sets will have their plkj valr ues equal, say plbest (which is evident from the nature of our uplifting). Then run the algorithm on another r subproblem with L − plbest + 1 levels, with migration cost at the lowest level as 0 and other migration costs reduced by M Cplrbest +1 . Now choose the best out of these scenarios : feasible solution for each prk and all the applications conslidated at level pl (which we call the base consolidation). We have shown that the above algorithm finds the optimal software consolidation level for all workloads on the server. For readability reasons, we defer the proof to the end of this paper.

5.

EXPERIMENTAL EVALUATION

We conducted a large number of experiments to evaluate CloudBridge under a wide variety of settings. In this section, we report some of our key findings.

5.1 Experimental Setup We implemented CloudBridge in the Anonymized Datacenter Consolidation Planning tool that has been used and validated in earlier studies [Anonymized]. CloudBridge was integrated as a new consolidation planning algorithm in the tool and the user could select any of the algorithms for consolidation planning. We extended the tools to also take the application stack information (operating system, middleware stacks, application) for each workload. Further, we ex-

tended its knowledge base with the migration and softwarerelated costs for each workload stack. In order to compare our algorithm over existing consolidation strategies, we also implemented the following competing algorithms and compare the cost of our solution with them. • Baseline Virtualization: We implement a simple server consolidation algorithm that attempts to pack the workloads on a minimum number of servers. The technique uses First Fit Decreasing (FFD) packing to minimize fragmentation and is one of the most popular techniques for consolidation [20, 21]. • Peak Clustering based Placement (PCP): The PCP algorithm is the best known workload-aware virtual machine consolidation scheme. We use the existing implementation of PCP described in [22]. • Middleware Consolidation (MwC): The Middleware Consolidation strategy captures a natural way to integrate hardware and software consolidation. In this scheme, we pick a middleware layer l for consolidation and colocate all applications having the middleware l and consolidate them. We report the savings for middleware layer l that achieves the lowest data center cost across all levels 1 to L. • PCP + BP2 : In this algorithm, we use PCP to obtain the placement of workloads to servers. We then use the low-level BP2 algorithm to obtain the optimal consolidation level for each server. A natural differentation between M wC and P CP + BP 2 is that M wC gives priority to reducing software costs first and co-locates workloads based on software stack. Within this solution, it performs the best possible hardware consolidation. P CP + BP 2, on the other hand, first co-locates workloads to achieve the best hardware consolidation and then performs the best software consolidation within this solution. CloudBridge does not prioritize hardware or software consolidation but instead aims to optimize the tradeoff between both the costs. In order to conduct a realistic study, we used workload traces and application stack information from the production data center of a Fortune 500 company. The workload traces captured the 5-minute average of CPU, memory, disk and network resources used by each virtual machine over a period of 2 months. The application software stack consisted of operating system (Aix), databases (IBM DB2), web middleware (Websphere Application Server, IBM HTTP Server, Websphere Portal Server, LDAP), and custom applications. For privacy reasons, we are unable to report the 1-to-1 mapping between the software stack and the workload. We used CloudBridge to come up with a consolidation plan for migration to a private cloud infrastructure with mid-range servers having 16 cores each. We used migration, hardware and software cost based on information available online (Data compiled, October 2010) for our experimental runs (Table. 2). We used the above parameters to create a baseline consolidation scenario and conducted our experimental evaluation. In order to understand the relative performance of various algorithms under a wide variety of settings, we used a multiplier for the cost parameters and target server models. This allowed us to capture the behavior of the algorithms for different software stacks and hardware platforms. The baseline

SC($) 1000 1000 1000 13000 20000 52000 12400 29000 20500

M C($) 5 100 100 1000 1000 5000 1000 5000 1000

Table 2: Annualized software-related and migration costs Parameter Baseline Range N 16 N.A L 6 N.A M C Multiplier 1 0.5 - 5 SC Multiplier 1 0.5 - 2 HC Multiplier 1 0.5 - 2 ServerCores 16 8 - 32

above the baseline consolidation but has higher hardwarerelated cost than P CP . P CP + BP 2(CloudBridge) algorithm is able to further reduce the software-related cost by finding the optimal software consolidation strategy on each server. CloudBridge outperforms all the algorithms by first placing applications in a manner that optimizes the tradeoff between compatibility in workload pattern and similarity in software stack and then finding the optimal software consolidation for the workloads on each server. We observe that CloudBridge can achieve upto 20% additional savings over and above any savings due to baseline consolidation using virutalization. 400 Migration Cost Software Cost Hardware Cost

350 Cost(x1000$ per year)

Software OS Java Tools HTTP Server Application Server Database Server Messaging Server Directory Server Portal Server Access Server

300 250 200 150 100 50 0

Table 3: Experimental Parameters

Baseline . . . .

PCP . .

. MwC . . . Target Server

PCP+BP2 . .

Cloudbridge . .

(a) values for our experimental parameters and their ranges are listed in Table. 3. We now report the comparative performance of all the algorithms for the default setting.

5.2.1 Comparative Evaluation

SoftStack−1 SoftStack−2 SoftStack−3 SoftStack−4 SoftStack−5 SoftStack−6

CloudBridge PCP+BP2

7

Nubmer of Workloads

5.2 Experimental Results

8

6

MwC

5 4 3 2 1

700 Hardware Cost Software Cost Migration Cost

Cost(x1000$ per year)

600 500

0

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

Server Id

(b)

400

Figure 7: (a) Comparative Migration, Software and Hardware Costs on Each Server after Consolidation. (b) Distribution of Workloads on Servers under different Algorithms

300 200 100 0 Original

Baseline

PCP Target Server

MwC

PCP+BP2

Figure 6: Comparative Migration, Software and Hardware Costs for each Algorithm after Consolidation We first study the relative performance of various algorithms with respect to the overall data center cost (Figure. 6). The most notable observation is that all the schemes are able to significantly reduce hardware-related costs. This increases the overall weightage of software-related costs in total data center costs, highlighting the importance of software consolidation. The baseline virtualization strategy and P CP do not consolidate at an application level and hence have 0 application migration cost. They also have the highest software-related cost as the number of software instances do not reduce due to consolidation. P CP outperforms baseline virtualization by reducing the hardware-related cost using workload-aware consolidation. We observe that M wC saves some license costs by consolidating software over and

We next understand the reasons behind the relative performance of the different algorithms. Fig. 7(a) shows the servers selected for consolidation using the competing strategies as well as the hardware-related, software-related, and migration cost on each target server. We observe that a workload-aware strategy like P CP is able to consolidate the workloads on fewer servers than simple first fit consolidation. This allows P CP and algorithms that leverage workloadawareness to use fewer servers and reduce the hardware cost. However, vanilla P CP incurrs high software cost on each of the target servers. P CP + BP 2 attempts software consolidation but is unable to reduce software costs significantly. We conjecture that since P CP is unaware of the software stack, BP 2 may not find many opportunities for software consolidation. Fig. 7(b) confirms our conjecture. We observe that Vanilla P CP places software stacks randomly on the target servers. For example, Server 1 has 1 instance each of 5 different software stacks. On the other hand, CloudBridge has only 1 software stack on server 3. Similarly, M wC also places applications based on software stack

Additional Savings Over Virtualization (%)

We next vary the different cost parameters from their baseline values and study their impact on the performance of the competing methodologies. PCP MwC PCP+BP2 Cloudbridge

20

15

10

5

0 0.4

0.6

0.8

1 1.2 1.4 1.6 Software Cost Multiplier (x)

1.8

Additional Savings Over Virtualization (%)

(a)

10

5

0 1

1.5

2 2.5 3 3.5 Migration Cost Multiplier (x)

4

PCP MwC PCP+BP2 Cloudbridge

15

10

5

0 0.5

1

1.5

2 2.5 3 3.5 Hardware Cost Multiplier (x)

4

4.5

5

(a) 25

20

PCP MwC PCP+BP2 Cloudbridge

15

10

5

0 10

15

20 Server Cores

25

30

Figure 9: Effect of change in (a) Hardware-related Cost and (b) Server Size

PCP MwC PCP+BP2 Cloudbridge

0.5

20

(b)

2

20

15

Savings Over First Fit Consolidation (%)

5.2.2 Impact of Various Consolidation Parameters

Additional Savings Over Virtualization (%)

and has no more than 2 software stack on each server. The awareness of software stack allows both M wC and CloudBridge to find more opportunities for software consolidation. The baseline scenario illustrates the tradeoff achieved by CloudBridge in finding a consolidation solution that optimizes all three components of cost, namely hardware, software and migration cost.

4.5

5

(b) Figure 8: Effect of change in (a) Software-related Cost and (b) Migration Cost In Fig. 8(a), we vary the software-related costs for all the workloads under study. Increase in software costs help all the methodologies that are aware of the software-stack to achieve additional savings over baseline virtualization. On the other hand, P CP achieves smaller savings with increase in license cost. We note that P CP only saves a constant hardware cost. As the software costs increase, the total cost of the data center increases. Hence, even though the magnitude of savings for P CP stays constant, the percentage savings on the total data center costs is reduced. We also observe that CloudBridge outperforms its closest competitor P CP + BP 2 more with increase in software cost. This is because CloudBridge adapts to the increase in software cost by giving higher weightage to software stack similarity (as opposed to workload profile complementarity). The above experiment establishes CloudBridge’s ability to adapt to changes in software cost and optimize the tradeoff between software and hardware cost. We next study the impact of an increase in migration cost (Fig. 8(b)). An increase in software migration cost leads to lower savings due to software consolidation. Hence, we observe that all software-aware strategies converge as the migration cost is increased by a factor of 5. In this case, software consolidation is avoided by all schemes and they

achieve the same savings as Vanilla P CP . Similarly, an increase in hardware cost (Fig. 9(a)) makes software consolidation less profitable. Hence, the relative savings by softwareaware strategies is reduced. However, in all these scenarios, CloudBridge is the best performing strategy and is able to adapt to variation in the cost parameters by switching from aggresive software consolidation to aggresive hardware consolidation. In our last experiment, we study the performance of the algorithms with change in the server size. A large server allows more workloads to be consolidated together. Hence, for small server sizes, none of the techniques are able to consolidate many software instances (Fig. 9(b)). Hence, the relative improvement of software-aware strategies over P CP is small. On the other hand, for large server sizes, all softwareaware schemes get more opportunities for software consolidation since there are more workloads on each server. Hence, workload-aware strategies lead to significant cost savings. It is interesting to note that the relative performance of M wC and P CP + BP 2 typically moves in opposite directions as we vary an operating parameter. For example, as the hardware-related costs are increased, M wC is unable to achieve significant savings over baseline virtualization, whereas P CP + BP 2 is able to perform better. Similarly, as the number of cores increase, M wC approaches the performance of CloudBridge whereas P CP + BP 2 does not perform as well. The observation highlights the importance of a flexible strategy like CloudBridge that can adapt to various operating conditions and focus on one or more of the 3 cost components, based on their relative weightage. Our experimental study thus comprehensively establishes the ability of CloudBridge to adapt to various operating conditions and achieve the optimal tradeoff between software and hardware costs. CloudBridge outperforms all the competing algorithms over the entire range of operating conditions leadings to a net data center savings of over 20% over

baseline virtualization.

6.

RELATED WORK AND CONCLUSION

Data center consolidation has been an active area of research over the years. The related work in this area can broadly be classified into Static Consolidation: The advent of virtualization also brought into focus the challenges associated in data center consolidation. There are many tools available from various vendors that provide recommendations for consolidating a data center using virtualization [4, 5, 6, 17, 19]. Most of these tools use monitored data to come up with a resource demand for the workloads and use bin-packing heuristics or Integer Linear Programming-based techniques to find a good consolidation solution. Korupolu al. [14] also investigate coupled placement on servers as well as storage to take into account both server resources and I/O bandwidth requirements. Verma et al. [22] extend consolidation solutions from a fixed-VM size strategy to a workload-aware strategy, where workloads with complementary peak-trough behaviour are consolidated together. Similarly, software consolidation is applied in industry to reduce license cost. Database consolidation has been a popular technique to reduce the number of database instances and reduce costs [11]. Similar strategies have been proposed to reduce license cost for popular middleware like web servers as well. However, to the best of our knowledge, there has been no work that tries to reduce both hardware, operational, and software costs together. Dynamic Virtual Machine Consolidation: Dynamic consolidation in a virtualized data center has also been an active research area [10, 9, 15, 20, 23, 25]. Dynamic consolidation differs from static consolidation by working on a smaller time granularity and continuously optimizing the virtualized data center. In [15], the authors advocate presenting guest virtual machines with a set of soft power states such that application-specific power requirements can be integrated as inputs to the system policies, without application specificity at the virtualization-level. In an earlier work, we have proposed dynamic consolidation mechanisms to minimize power [20] and optimize performance subject to a power budget [23]. In [9], the authors optimize multiple objectives namely performance, power and adaptation costs. Workload Characterization: Data center consolidation needs to profile workloads and ensure that the performance of the applications does not suffer due to consolidation. Workload profiling to minimize resource bottleneck and identify good candidates for static [22, 7] and dynamic consolidation has been studied before[2]. Similarly, researchers have worked on modeling the power drawn by servers in a data center [12, 18]. Wood et al. profile the resource usage of workloads in a virtualized environment and incorporate virtualization overheads due to I/O processing [24]. The impact of shared caches and provisioning the memory resources has been studied and modeled in [21, 13]. The workload characterization work is complimentary to our work, which leverages workload profiles to come up with integrated software and hardware consolidation plan for a data center. In this work, we have integrated software and hardware consolidation strategies to design CloudBridge, a unified hardware-software consolidation methodology. We present the first formal algorithm BP 2 to identify the appropriate software consolidation for workloads placed on each server. BP 2 uses the steps of base-leveling, piggybacking and pro-

motion to identify the optimal software consolidation level for each workload. CloudBridge uses BP 2 and awareness of both utilization profiles and software stack to achieve the right tradeoff between software and hardware cost, reducing data center costs by upto 20% in many realistic scenarios.

7. REFERENCES [1] Microsoft Windows XP Professional END-USER LICENSE AGREEMENT. http://www.microsoft.com/windowsxp/eula/pro.mspx. [2] N. Bobroff, A. Kochut, and K. Beaty. Dynamic placement of virtual machines for managing sla violations. In IEEE Conf. Integrated Network Management, 2007. [3] Rajendra Chaudhary. The multi-core licensing debate. In Express Computer Online. http://www.expresscomp uteronline.com/20100712/expressintelligententerprise 03.shtml, 2010. [4] Server Consolidation and Virtualization Analysis by CiRBA. http://www.cirba.com/. [5] VMWare Guided Consolidation. http://www.vmware.com/products/vi/vc/features.html. [6] Virtual Iron: True Server Virtualization for Everyone. http://www.virtualiron.com/. [7] D. Gmach, J. Rolia, L. Cherkasova, and A. Kemper. Workload analysis and demand prediction of enterprise data center applications. In IISWC, 2007. [8] IBM. Processor value unit [pvu] licensing for distributed software. In http://www-01.ibm.com/ software/lotus/passportadvantage/pvu licensing for customers.html, 2010. [9] G. Jung, M. Hiltunen, K. Joshi, R.Schlichting, and Calton Pu. Mistral: Dynamically managing power, performance, and adaptation cost in cloud infrastructures. In Proc. ICDCS, 2010. [10] G. Jung, K. Joshi, M. Hiltunen, R.Schlichting, and Calton Pu. A cost-sensitive adaptation engine for server consolidation of multitier applications. In Proc. Middleware, 2009. [11] J. P. Kirby and E. Klein. Database consolidation: Reducing cost and complexity. In AMR Research Report, http://download.microsoft.com/download/ 1/5/6/15667d1c-3963-4b62-b11c-7bb204248831/ amr consolidation.pdf, 2004. [12] R. Koller, A. Verma, and A. Neogi. Wattapp: An application aware power meter for shared data centers. In ICAC, 2010. [13] R. Koller, A. Verma, and R. Rangaswami. Generalized erss tree model: Revisiting working sets. In IFIP Performance, 2010. [14] M. Korupolu, A. Singh, and B. Bamba. Coupled placement in modern data centers. In IEEE IPDPS, 2009. [15] Ripal Nathuji and Karsten Schwan. Virtualpower: coordinated power management in virtualized enterprise systems. In Proc. ACM SOSP, 2007. [16] R. O’Hara, P. Ross, and R. Wright. Driving cost out of it. In Microsoft System Center. https://www.microsoftsalesuniversity.com/infrastructure/ resources/content/Presentation Driving Cost Out of IT.pptx, 2009.

[17] VMWareCapacity Planner. http://www.vmware.com/products/capacity-planner/. [18] S. Rivoire, P. Ranganathan, and C. Kozyrakis. A comparison of high-level full-system power models. In HotPower, 2008. [19] HP Capacity Advisor Consolidation software. http://h18013.www1.hp.com/products/servers/ management/capad/index.html. [20] A. Verma, P. Ahuja, and A. Neogi. pmapper: Power and migration cost aware application placement in virtualized systems. In Proc. Middleware, 2008. [21] A. Verma, P. Ahuja, and A. Neogi. Power-aware dynamic placement of hpc applications. In ACM ICS, 2008. [22] A. Verma, G. Dasgupta, T. Nayak, P. De, and R. Kothari. Server workload analysis for power minimization using consolidation. In Proc. Usenix ATC, 2009. [23] A. Verma, P. De, V. Mann, T. Nayak, A. Purohit, G. Dasgupta, and R. Kothari. Brownmap: Enforcing power budget in shared data centers. In Proc. Middleware, 2010. [24] T. Wood, L. Cherkasova, K. Ozonat, and P. Shenoy. Profiling and modeling resource usage of virtualized applications. In Proc. Middleware, 2008. [25] Qian Zhu, Jiedan Zhu, and Gagan Agrawal. Power-aware consolidation of scientific workflows in virtualized environments. In Proc. SC, 2010.

8.

APPENDIX:OPTIMALITY PROOF OF BP2

Let us denote the solution computed by our algorithm for a set skl by ALGL−1 (skl ). In a sequence of lemmas, we will prove that ALGL−1 (skl ) is infact the optimal consolidation pattern for the set skl . We first state a lemma to prove the claims made in the Piggybacking steps of Leader and Leaderless scenarios. Lemma 1. Let S be a set of applications with similarity level r. If ALGL−1 (S) contains no application consolidated at any level ≥ r, then it contains consolidation at a common level r 0 < r. Proof. Proof is by reverse induction on r. Clearly the claim is true for r ≥ L, due to the nature of our algorithm for this case. Now say that we have a set of applications S with similarty level r ≤ L − 1, and that ALGL−1 (S) contains no consolidation at any level ≥ r. Let S1 , S2 , ..., Sm be the subsets according to similarities upto level r + 1. Then clearly none of ALGL−1 (Si ) contains consolidation at any level ≥ r, since we do not degrade the consolidation levels at any step. Now since in ALGL−1 (S), there is no consolidation at any level ≥ r, hence only the piggybacking step is done at this stage (i.e. the promotion step chose the base consolidation as the best one). Thus the consolidation will be at a common level r 0 < r. (Note that by induction hypothesis each Si will contain consolidation at a common level ≤ r, and hence this will be the Leaderless scenario). We next prove a lemma that implies the optimality of the base-case, i.e.,ALGL−1 (skL ). Lemma 2. Let OP TL−1 (S) be an optimal conslidation pattern for a set of applications, S. If an app I is consolidated at level r and an app I 0 is consolidated at level r 0 , with r 0 < r,

and if I and I 0 have the same stack upto level r + 1, then there exists and optimal placement OP TL0 (S), such that I and I 0 are consolidated at level r in OP TL0 (S). Proof. Since OP TL−1 (S) is an optimal consolidation pattern, I and I 0 will contain the same stack upto level r 0 . Denote by P L the consolidation pattern obtained by consolidating I 0 at level r. Since there is already a stack upto level r in use, we have that Savings(I 0 ) − Savings(OP TL−1 (S)) = P r 0 0 r 0 +1 lj − M Cr+1 + M Cr +1 . Denote by P L the consolidation pattern obtained by deconsolidating I from level r to P level r 0 . Then Savings(P L0 ) − Savings(OP TL−1 (S)) ≥ − rr0 +1 lj + M Cr+1 − M Cr0 +1 , since we are using atmost one extra licence for the levels r 0 + 1 to r (which might not be the case if I is the lone consolidated application at some levels). Thus we get that Savings(P L) + Savings(P L0 ) ≥ 2Savings(OP TL−1 (S)). Since OP TL−1 (S) is an optimal consolidation pattern, Savings(P L) = Savings(OP TL−1 (S)), and hence P L is the required consolidation pattern. Let P lOP TL−1 (S) (I) denote the consolidation level of I in OP TL−1 (S). We next show the the consolidation in a superset is always at same or higher level than its subset. Lemma 3. Let OP TL−1 (S) be an optimal conslidation pattern for a set of applications, S. Then for any set of applications S 0 ⊇ S, there exists an optimal consolidation pattern for S 0 , OP TL−1 (S 0 ), such that P lOP TL−1 (S 0 ) (I) ≥ P lOP TL−1 (S) (I) for all I ∈ S. Proof. Let OP T (S 0 ) be an optimal consolidation pattern for S 0 , for which there is a set of applications T ⊆ S, such that P lOP TL−1 (S) (I) > P lOP T (S 0 ) (I) for all I ∈ T . Denote by P L the consolidation pattern obtained from OP T (S 0 ) by consolidating the applications in T to their consolidation levels in OP TL−1 (S). Denote by P L0 the consolidation pattern obtained from OP TL−1 (S) by deconsolidating the applications in T to their levels in OP T (S 0 ). We will prove that Savings(P L) − Savings(OP T (S 0 )) ≥ Savings(OP TL−1 (S)) − Savings(P L0 ), and thus P L would be the required optimal placement. The migration cost componenets are the same in Savings(P L)− Savings(OP T (S 0 )) and Savings(OP TL−1 (S))−Savings(P L0 ), and thus we just need to compare the licence components. Now Savings(OP TL−1 (S)) − Savings(P L0 ) can be written P Pr(I) as I∈T l(I), where l(I) = j=r 0 (I)+1 l(j, I)s(j, I). Here I is deconsolidated from level r(I) to r 0 (I), l(j, I) is the cost of j th stack component of I. Let Ij be the set of apps with whom I shared the licence of the j th stack component in OP TL−1 (S). s(j, I) = 1, if P lOP T (S 0 ) (I 0 ) ≥ j for some I 0 ∈ Ij ; (|Ij | − 1)/|Ij |, if P lOP T (S 0 ) (I 0 ) < j for all I 0 ∈ Ij ; 0 if |Ij | = 0. Savings(OP TL−1 (S)) − Savings(P L0) can be written in the same form, but instead of s(I, j) use t(I, j). Now if s(I, j) = 1, then t(I, j) also is 1, since we’ll save the j th licence cost, as there exists an application consolidated at a level higher in OP T (S 0 ) by the defintion of s(I, j). If s(I, j) = (|Ij | − 1)/|Ij |, then in P L, we’ll definitely save |Ij | − 1 licences and those savings can be distributed among the |Ij | apps, hence t(I, j) ≥ (|Ij | − 1)/|Ij |. Thus t(I, j) ≥ s(I, j) for all I ∈ T and j ∈ {r 0 (I) + 1, ...., r(I)}, and hence Savings(P L)−Savings(OP T (S 0 )) ≥ Savings(OP TL−1 (S)) − Savings(P L0 ). Definition 3. We call a set S with similarity level r + 1, a leader, if there exists an optimal consolidation pattern

OP TL−1 (S) for S, with some application consolidated at level ≥ r, and we call this OP TL−1 (S), a leader-witness consolidation pattern for Si . Lemma 4. Let S be a set of applications with similarity level r. Let S1 , S2 ,.....,Sm be the subsets according to similarities upto level r+1. Then there exists an optimal consolidation pattern for S, OP TL−1 (S), such that P lOP TL−1 (S) (I) = P lOP TL−1 (Si ) (I) for all I ∈ Si , and for all leader Si , where OP TL−1 (Si ) is a leader-witness consolidation pattern for Si . Proof. Suppose we are given {OP TL−1 (Si ) : 1 ≤ i ≤ m}. From the previous lemma, we have that there exists an optimal consolidation pattern for S, OP T (S) such that for all i, P lOP T (S)(I) ≥ P lOP TL−1 (Si ) (I) for all I ∈ Si . Now for OP T (S), let that for some i, OP TL−1 (Si ) is a leader-witness consolidation pattern and that the consolidation pattern of Si in OP T (S) is P L0 (Si ) 6= OP TL−1 (Si ). Note that P L0 (S) must also be a leader-witness consolidation patter of Si . Denote by P L, the consolidation pattern obtained from OP T (S) by replacing P L0 (Si ) by OP TL−1 (Si ). Clearly Savings(P L)−Savings(OP T (S)) = Savings(OP TL−1 (S))− Savings(P L0 (Si )) (migration costs are varying for Si only, and for the licence costs, the licence savings for other sets are the same for both P L and OP T (S)), and hence Savings(P L) ≥ Savings(OP T (S)). Continuing in this way we get the required consolidation pattern for S. Theorem 1. For all sets S, and for all k, ALGk (S) is an optimal consolidation pattern for S, if consolidation is allowed only upto level k. Proof. Proof is by double induction, a reverse induction on the similarity level of S and induction on k. Base cases k = 1, and r = k + 1 are easy to verify. Now for k ≥ 2 and r ≤ k, let that for all k0 < k, ALGk0 (S) is the optimal consolidation pattern for for all sets S, if consolidation is allowed till level k0 , and for all r 0 ≥ r + 1, ALGk (S) is an optimal consolidation pattern for all sets S with similarity level r 0 , if consolidation is allowed only upto level k. Now let S be a set with similarity level r. We prove that ALGk (S) is an optimal consolidation pattern for S, if consolidation is allowed only upto level k. Let S1 , S2 , ..., Sm be the subsets according to similarities at level r + 1. We prove the Leader and Leaderless scenarios separately. Leader Scenario: By Induction Hypothesis, ALGk (Si ) is an optimal consolidation pattern for Si . Hence from Lemma 3.3 and Lemma 3.4, we obtain that there exists an optimal placement for S, OP T (S) such that P lOP T (S) (I) ≥ P lALGk (Si ) (I) for all Si , and all I ∈ Si . Also for leader sets Si , P lOP T (S)(I) = P lALGk (Si ) (I) for all I ∈ Si . Clearly OP T (S) will have to perform the base levelling step. Now consider a non-leader set Si , by Lemma 3.1, ALGk (Si ) will contain all apps consolidated at a level blki . Now let the level returned by Piggybacking step be pli . Then we get that there exists an optimal placment OP T , such that P lOP T (I) ≥ Ppli pli for all I ∈ Si , since by the maxima of l − j=blik +1 j Ppli M Cpli +1 + M Cbli +1 , we get that j=pl+1 lj − M Cpli +1 + k

M Cpl+1 ≥ 0 for all blki ≤ pl ≤ pli . From the maxima we also get that no application will be consolidated at a level pl s.t pli + 1 ≤ pl ≤ r. Now consider the consolidation pattern of Si in OP T , say OP T (Si ). Also let ALG0k−pli (Si ) denote the solution returned by algorithm run on the modified Si . Then it is easy to see that Savings(OP T (Si )) −

P i Savings(ALG0k−pli (Si )) = |Si | pl j=1 lj − |Si |M Cpli +1 (We account all the common licences to the leader-set). By Induction Hypothesis, ALG0k−pli (Si ) is the optimal consolidation pattern for each non-leader Si , and hence OP T is the optimal consolidation pattern for S. Leaderless Scenario: As in the previous case, there exists an optimal placement for S, OP T (S) such that P lOP T (S) (I) ≥ P lALGk (Si ) (I) for all Si , and all I ∈ Si . By Lemma 1, ALGk (Si ) will consist of consolidation at level blki . Let blk = maxi blki . Now clearly P lOP T (S)(I) ≥ blk for all I ∈ S, since Pbli∗ k if blki∗ = blk , then if (|Si∗ | − 1) j=bl i +1 lj − |Si∗ |M Cbli∗ +1 + k k Pbli∗ k |Si∗ |M Cbli +1 ≥ 0, then j=bl i +1 lj −M Cbli∗ +1 +M Cbli +1 ≥ k

k

k

k

0. If the Piggybacking step returns pl, then by a similar argument P lOP T (S) (I) ≥ pl for all I ∈ S. Now in OP T (S), denote the consolidation pattern of Si by OP T (Si ). Now two cases arise : one if for no i, does OP T (Si ) contain a consolidation at level ≥ r, and the other one in which for some i, OP T (Si ) contains a consolidation at level ≥ r. In the former case, clearly OP T (S) is the base consolidation(as defined in the Promotion Step of Leaderless scenario), due to the choice of pl, and hence is equal to ALGk (S). In the latter case, we claim that OP T (S) must be equal to the feasible solution corresponding to Si , and hence must be equal to ALGk (S). For j 6= i, let plkj be the level returned by the Promotion Step in the first phase. Then clearly by the choice of plkj , P lOP T (S) (I) ≥ plkj for all I ∈ Sj . Also by an argument similar to the one in the Leader scenario case, OP T (S) must contain the consolidation as in ALG0k−plj (Si ). Also if k

for some j, the best solution for Sj contains an application consolidated at level ≥ r, then OP T (S) will contain the best solution for Si . Else if for no j 6= i, the best solution for Sj contains an application consolidated at level ≥ r, then OP T (S) will contain all apps in Sj , j 6= i consolidated at a level plbest , and hence for all I ∈ Si , P lOP T (S) (I) ≥ plbest , and hence OP T (S) will contain ALG0k−plbest (Si ), and thus OP T (S) is the feasible solution corresponding to Si .