DRESS Codes for the Storage Cloud: Simple Randomized Constructions Sameer Pawar, Nima Noorshams, Salim El Rouayheb, Kannan Ramchandran Dept. of Electrical Engineering and Computer Sciences University of California, Berkeley {spawar, nshams, salim, kannanr}@eecs.berkeley.edu Abstract—We introduce Distributed Replication-based Simple Storage (DRESS) codes as new efficient codes for distributed data storage. DRESS codes are characterized by an uncoded repair process which has minimum bandwidth, Disk I/O and computational overheads when repairing from failures. A file is first encoded using a Maximum Distance Separable (MDS) code, then multiple replicas of the coded packets are placed on different nodes in the system. When the outer code is chosen to be a low-complexity erasure code, such as a Fountain or Raptor code, DRESS codes will have a linear decoding complexity at the user side. We present two randomized code constructions based on bins-and-balls and expander graphs and analytically demonstrate their good performance.

I. I NTRODUCTION Distributed storage systems (DSS), commonly known as “cloud storage”, use a large number of individually unreliable storage nodes to store user data reliably for a long duration. Applications using this principle are now of increasing popularity and include data centers [1], [2] and peer-to-peer (p2p) systems [3]. At such massive scale, node failure is a frequent event and occurs for various reasons such as hardware failure and peer churning. To guarantee reliability, the data is stored redundantly in the system. Moreover, the system is periodically repaired by replacing the failed nodes with new ones. Erasure codes [4], [5], [6] can be used here to guarantee the reliability of the data. However, they suffer from a high repair cost. When a new node contacts other nodes to recover a coded chunk, lost due to failure, it has to read, download and decode the whole file. Recently, Dimakis et al. introduced in [7] a new class of codes, called regenerating codes, characterized by a minimum repair bandwidth overhead. The main idea is to let nodes contacted for repair forward random linear combinations of their packets to the new-comer. Subsequent works [8], [9], [10] focused on deterministic code constructions under various system constraints. While previous constructions of regenerating codes solved the repair bandwidth problem, they generally incur a high computational and Disk I/O overhead. They require an active node to read all its stored data in order to send a linear combination of them to a new node replacing a failed one. This can cause excessive energy consumption and delay especially in scenarios where the disk read/write bandwidth is the bottleneck [11]. Moreover, the associated decoder implements a matrix inversion which can be computationally expensive

1

failure 2

File X1,...,X6

v1

X1 X2 X3

v2

X3 X4 X5

v3

X1 X5 X6

v4

X1 X4 X7

6 7

3

4

5

(b)

X1,...,X7

MDS

X7=X1+...+X6

v5

X2 X5 X7

v6

X3 X6 X7

v7

X2 X4 X6

X1 X2

new node

X1 X2 X3

X3

User

Fractional Repetition

(a)

Fig. 1. (a) An example of a DRESS code with repetition factor ρ = 3 for a DSS of parameters (n, k, d) = (7, 3, 3), where n is the total number of nodes, k is the number of nodes contacted by the user, and d is the number of packets per node. A file of size 6 packets is first encoded using an MDS code that appends to it a parity check packet. Then, an inner code, referred to as Fractional Repetition (FR) code, places 3 replicas of each coded packet in the system. Any user contacting 3 nodes will observe 6 distinct packets and can recover any missing packet by a simple xor operation. Notice also the uncoded repair process illustrated when node v1 has failed. The system is then repaired with minimum reads, minimum download bandwidth and with no processing involved. (b) The Projective plane of order 2, also known as the Fano plane. The pattern of the FR code is derived by associating points in the Fano plane to coded packets and lines (including the circle) to storage nodes (see [13] for more details).

for large file sizes. For instance, practical implementations of regenerating codes were investigated in [12] where a high encoding/decoding delay equal to 1 hour per stored GB was reported. In this paper, we study the design of regenerating codes that minimize bandwidth, Disk I/O and computation overheads of the repair process. This is achieved by imposing the uncoded repair property [13] on the code as a design requirement. With uncoded repair a survivor node reads the exact amount of data he needs to send to a new node and forwards it without any processing. We call these codes Distributed REplication-based Simple Storage (DRESS) codes. Fig. 1(a) depicts an example of a DRESS code used in a DSS consisting of n = 7 nodes each having a storage capacity

equal to 3 packets. A user should be able to recover his stored file by contacting any k = 3 nodes in the system. When the repair bandwidth is to be minimized, it can be shown that the maximum file size that can be delivered to the user is equal to 6 packets [15]. The DRESS code first encodes the file using a (7, 6) MDS code that simply adds to it a parity check packet. We say that the DRESS code has a replication factor ρ = 3. Each of the coded packets is then replicated on 3 nodes in the system following a specific pattern derived from the Fano plane ( see Fig. 1(b)). It can be checked that any user contacting 3 nodes will observe at least 6 distinct packets and therefore can recover the missing packet of the file by a simple xor operation (no need to invert a matrix). Since each coded packet is replicated 3 times, the system can efficiently recover from up to 2 simultaneous failures by contacting the nodes having the surviving copies. Thus, the amount of data read and downloaded during repair is minimum and equal to the lost data. The outer component of the DRESS code can be chosen to be a low-complexity erasure code such as a Fountain code [14] to achieve a linear decoding complexity at the user side. The inner code, which was introduced in [13] and was termed Fractional Repetition (FR) code, ensures that replicas are spread uniformly across the system to guarantee the delivery of the file to any user. Deterministic constructions of FR codes for general repetition factor were proposed in [13] based on projective spaces and Steiner systems. Our work is motivated by earlier code constructions of Rashmi et al. [9] for a replication factor equal to 2 and where all surviving nodes have to be contacted for repair. We present in this paper two randomized constructions for DRESS codes. In these constructions, the inner FR code randomly places the coded packet replicas on the nodes based on a bins-and-balls approach in the first method, and based on expander graphs in the second. We show then that any user contacting a minimum number of nodes in the system will be able to recover his file with a high probability. This probabilistic approach permits more flexibility in terms of possible system parameters (number of nodes, replication factor, storage,...) compared to the previous combinatorial constructions in [13] which are worst-case driven and where in many cases only existence guarantees are available. The rest of paper is organized as follows: In Section II we briefly provide the background on the distributed storage systems. In Section III we propose two randomized constructions and provide simulation results. Section IV provides summary of our main results and Section V contains the theoretical analysis of our constructed codes. We conclude the paper in Section VI and discuss some interesting open problems. II. BACKGROUND AND M ODEL A distributed storage system DSS is defined by the triplet (n, k, d), where n is the total number of storage nodes, v1 , . . . , vn , in the system, k < n is the number of nodes contacted by the user to retrieve his stored file, and d ≥ k is

the repair degree that specifies the number of nodes contacted by a replacement node during repair. We consider distributed storage systems operating in the minimum-bandwidth regime (MBR) on the storage/bandwidth tradeoff curve described in [15]. Our focus on this regime is motivated by the asymmetrical cost of resources in practical systems where bandwidth is more expensive than storage. In this case, the repair bandwidth of the system, i.e., the total amount of data downloaded by a replacement node is minimized. As a result, the new node needs to download only the amount of data he will store, but no more. For load-balancing requirements, we assume a symmetric model for repair where the replacement node downloads and stores an equal amount of data, referred to as a packet, from each of the d nodes it contacts. Therefore, in the minimum bandwidth regime, d also represents the node storage capacity expressed in packets. Under this model, the storage capacity CM BR in packets of the DSS, representing the information-theoretic limit on the maximum file size that can be delivered to any user contacting k out of the n nodes, was shown in [15] to be � � k CM BR (n, k, d) = kd − . (1) 2

This expression assumes a functional repair model where the only constraint on the data regenerated (stored) at the new node is maintaining the MDS property of the system. This allows the regenerated data to be different from the lost data as long as it is “functionally” equivalent. However, a more stringent form of repair, known as exact repair, that is capable of reproducing an exact copy of the lost data, may be required for many system considerations such as maintaining a systematic form of the data and reducing protocol overhead. In this case, regenerating codes are referred to as being exact. Recently, Rashmi et al. showed the interesting result that, in the minimum-bandwidth regime, there is no loss in the DSS capacity incurred by requiring exact repair and constructed exact MBR codes that achieve the capacity CM BR of (1). Consider a file F of size R packets that is to be distributively stored on a DSS of parameters (n, k, d) using a DRESS code of replication factor ρ. First, F is encoded using an outer (θ, R) MDS code, where θ = nd ρ . The coded packets are then replicated in the system, each ρ times, following the pattern indicated by an inner Fractional Repetition code. For completion, we recall here the formal definition of an FR code as introduced in [13]. Definition 1 (Fractional Repetition Codes): A Fractional Repetition (FR) code C, with repetition degree ρ, for an (n, k, d) DSS, such that θ := nd ρ is an integer, is a collection of n subsets of size d, C = {V1 , V2 , . . . , Vn }, of a set Ω = {1, . . . , θ}, such that each element of Ω belongs to ρ sets in the collection. In this definition, each set Vi contains the indices of the packets that are stored on node vi , i = 1, . . . , n, and which are output by the MDS code. For example, following this definition, the FR code of the example of Fig. 1 can be written

III. C ONSTRUCTIONS AND S IMULATIONS We now describe our two randomized DRESS code constructions which are based on two approaches: bins-and-balls and expander graphs. A. Bins-and-Balls Construction The n storage nodes correspond to n bins and the θ = nd/ρ coded packets at the output of the MDS code correspond to θ distinct colors. We assume that we have ρ balls of each color; that is ρθ = nd balls in total. The code is obtained by throwing each ball in one of the n bins picked independently and uniformly at random. In this setting a user will contact k bins and counts the number of distinct colors he can observe; this corresponds to the file size ( number of distinct coded packets) that he can recover. This file size will be relatively small if the user happens to contact bins containing few number of balls, or if the contacted bins have a large number of colors in common. For the system parameters of interest we show that these two events are rare and the number of colors observed by the user is highly concentrated around its mean. B. Expander Graph Construction The bins-and-balls construction may lead to certain nodes storing less packets then others which may disadvantage the users contacting these nodes. In this construction, we try to remedy this problem by guaranteeing a better load balancing among the nodes. The code construction is based on a random bipartite graph. The storage nodes will correspond to the left vertex set L, |L| = n of the bipartite graph, and the coded packets will correspond to the right one R, |R| = θ. To construct the graph, we start with d half edges extended from each of vertex in L and ρ half edges extended from each vertex in R. These half edges are then matched to each other using a random permutation block of size nd × ρθ1 . The code is then obtained by storing on each node the packets that are adjacent to it in the obtained bipartite graph. 1 This random procedure can also be interpreted in terms of the bins-andballs language as follows: Start with a pool of ρθ balls, where there are ρ balls of each of the θ distinct colors. Then, fill the bins consecutively with d balls each chosen uniformly at random from the pool of balls without replacement.

Expander graph

% of users

as C = {V1 , . . . , V7 } with V1 = {1, 2, 3}, V2 = {3, 4, 5}, V3 = {1, 5, 6}, V4 = {1, 4, 7} V5 = {2, 5, 7}, V6 = {3, 6, 7}, V7 = {2, 4, 6}, where Ω = {1, . . . , 7}. Since MDS codes exist for all feasible parameters provided that the packets are taken from an alphabet of large enough size, the challenging part of the DRESS code construction is designing the Fractional Repetition code in order to maximize the size of the file F that can be delivered to any user. A good design rule for the FR code which guarantees that all users observe at least CM BR packets is to make sure that nodes have at most one packet in common. Combinatorial code constructions based on this rule were studied in [13]. We are interested here in randomized code construction. Therefore, we relax the value of the parameters d and ρ from deterministic to average requirements.

Bins and balls

File size

Fig. 2. Average performance of DRESS codes with repetition factor ρ = 13 obtained by the two randomized constructions: bins-and-balls and expander graphs. The DSS have parameters (n, k, d) = (169, 13, 13). For each 4 construction, a code is generated 10 times, � and� in each generation 10 users are picked randomly among the possible 169 users. The graph depicts two 13 histograms representing the percentage of users observing a certain file size. The deterministic bound of (1) gives a file size equal to 91 packets. With the proposed randomized constructions the users will see with high probability an average file size equal to 110 packets.

C. Simulations To motivate our proposed constructions, we provide simulation results on the performance of DRESS codes for an example of a repetition factor ρ = 13 and for a DSS with (n, k, d) = (169, 13, 13). Fig. 2 shows the two obtained histograms, for the bins-and-balls and the expander graph constructions. The histogram gives for each file size the percentage of users that can decode it. Few interesting points to note in Fig. 2 are: 1) In the deterministic construction of [15], all the users will observe the same file size CMBR = 91 given by (1). 2) Almost all the users (more than 99% in both constructions) observe a file size larger than 91 packets. 3) For the proposed randomized constructions, the observed file size is concentrated around its mean equal to 110 > 91, with a better concentration for the expander graph construction. The simulations results indicate the advantages of abandoning a worst-case design of the system in favor of a probabilistic one that allows a small percentage of users who cannot decode the file. This way we can easily construct codes for distributed storage that have efficient repair capabilities. IV. M AIN R ESULTS We state now our main analytical results concerning the two proposed randomized constructions. Let I, I ⊂ {1, . . . , n}, be a set of size k containing the indices of the k different storage nodes contacted by a user. Denote by FI the file size that this user can recover. That is the total number of packets of distinct indices stored on the nodes indexed by I. The first result roughly says that for the bins-and-balls approach any user will observe the average file size with a

high probability. Theorem 2: Consider a DRESS code of repetition factor ρ obtained from the randomized bins-and-balls construction described in Sec. III-A and used for an (n, k, d) DSS. Then we have: a) The average file size observed by a user is � � �ρ � k Fave : = E[FI ] = 1 − 1 − θ. n b) Furthermore, FI is concentrated around its mean. More precisely, for any fixed � > 0, we have � � � �k� k P sup |FI − Fave | ≤ B ≥ 1 − 2 , en |I|=k � where B = 2θ (1 + �) k log(ne/k). For the expander graph construction, the next result gives a guarantee on the minimum file size that can be observed by the worst-case user. Theorem 3: Consider a DRESS code of repetition factor ρ obtained from the randomized expander graph construction described in Sec. III-B and used for an (n, k, d) DSS. Moreover, define the monotonically decreasing function � � eθ f (�) := �d log(1 − �) − log . kd

Then, with arbitrarily large probability p, 0 < p < 1, the file size observed by a user is lower bounded by min FI ≥ (1 − �(p)) k d,

|I|=k

� where �(p) = f −1 log

(1−p)1/k e1+d ρ

− log

θ kd

�

(2)

∈ (0, 1).

V. T HEORETICAL A NALYSIS

We now present the theoretical proofs of our main results. A. Bins-and-Balls Construction We proceed to prove the concentration result of Th. 2 which, at a high level, says that with high probability any user contacting k nodes will observe the average file size. Part a) Let I be the set containing the indices of the nodes contacted by a user (I ⊂ {1, . . . , n}, |I| = k). Without loss of generality assume that I = {1, 2, · · · , k} and let Ai (i = 1, 2, · · · , θ) denote the event that there is at least one ball of color i in the first k bins. In other words, the corresponding user I observes a packet of the i-th color. Then obviously we have FI =

θ � i=1

I {Ai } ,

where I {·} denotes the indicator function. Since balls are distributed randomly and uniformly, the probability that no ball falls into the first k bins is (1 − nk )ρ . Therefore, we have � � �ρ � k P(Ai ) = 1 − 1 − , n

hence, Fave =

� � �ρ � k P(Ai ) = θ 1 − 1 − . n i=1

θ �

Part b) In order to bound the concentration probability, we use the Azuma-Hoeffding inequality. Since FI is the sum of some independent bounded random variables, this inequality guarantees that for and fixed B > 0, we have � � B2 P (|FI − Fave | > B) ≤ 2 exp − . (3) 2θ Now for every subset I ⊂ {1, 2, · · · , n} of size |I| = k define the “good” events, AI , as follows AI : = {|FI − Fave | ≤ B} .

In other words, the “good” event happens if the number of colors observed by the user is close enough to the average. Then by using the union bound and the fact that AI ’s are identically distributed, we obtain P (∩I AI ) = 1 − P (∪I AcI ) � ≥ 1− P (AcI )

�I � n = 1− P (|FI − Fave | > B) . k

Substituting the inequality (3) into the above expression yields � � � � n B2 P (∩I AI ) ≥ 1 − 2 exp − . (4) k 2θ � � � �k Next, using the fact�that �nk ≤ ne , we upper bound the k � n� B2 expression k exp − 2θ as follows � � � � � � n B2 B2 ne exp − ≤ exp − + k log . (5) k 2θ 2θ k � Now setting the parameter B = 2θ (1 + �) k log(ne/k), inequality (5) becomes � � � � � �k� n B2 k exp − ≤ . (6) k 2θ ne

Finally, substituting the inequality (6) into (4) yields the claim � � � �k� k P sup |FI − Fave | ≤ B = P (∩I AI ) ≥ 1 − 2 . ne |I|=k Corollary 4: Assume that ρ → ∞ satisfying kρ = cn for some constant c > 0 and also θ = dn for constant d > 0. Then we have Fθave → 1 − e−c and also Bθ → 0 as ρ → ∞. Proof: This is an immediate consequence of Th. 2. The first part can be seen from the basic fact � �ρ � �ρ k c 1− = 1− → e−c as ρ → ∞, n ρ whereas the second part would follow form � ρe B 2(1 + �)c log c = → 0 as ρ → ∞. θ d ρ

Corollary 4 basically says that if the number of contacted bins times the repetition factor is comparable to the total number of bins (more precisely of the same order), then we have high concentration around the mean and every user observes roughly Fave balls which is (1 − e−c )θ. B. Expander Graph Construction Let G = (L, R, E) be a bi-regular unbalanced bipartite graph, where L = {1, 2, · · · , n} and R = {1, 2, · · · , θ} denote the sets of left and right vertices corresponding to the set of storage nodes and the coded packets respectively, and (i, j) ∈ E for some i ∈ L and j ∈ R if and only if packet j is stored at storage node i. For any subset I ⊂ L define Γ(I) = {j ∈ R | ∃ i ∈ I s.t. (i, j) ∈ E} to be the neighborhood set of I. We will show that almost all random graphs G result in a DRESS code that satisfies the lower bound (2) on the worstcase file size. For arbitrary sets I ⊂ L and T ⊂ R of cardinalities |I| = k and |T | = t = (1 − �)kd, for some � ∈ (0, 1), define the “Bad” events AI,T , as follows AI,T : = {Γ(I) ⊆ T } , that is, in the randomly generated graph G all the edges from set I go to set T . Then, the largest value of t, or correspondingly of (1 − �), for which � I {AI,T } = 0, I,T

provides a lower bound on the file size that can be delivered to the user using the DRESS code obtained from G. It is clear that for fixed sets I and T , � �kd t P (I {AI,T } = 1)) = . θ Applying the union bound over all possible values of I and T , we obtain   � � P I {AI,T } > 0 ≤ P (I {AI,T } = 1) I,T

I,T

= ≤

� �� � � �kd n θ t k t θ � �(kd−t) � � t n k θ

k � n�

Using the identities ρθ = nd, and t = (1 − �)kd we simplify the above expression � �k(�d−1) kd ρk (1 − �)�kd ek(1+(1−�)d) ≤ 1 − p, θ hence � � eθ (1 − p)1/k θ �d log(1 − �) − log − log + log ≤ 0. 1+d kd e ρ kd (7) Recall the definition of the function f (�), and define β := 1/k θ log (1−p) − log kd , then equation (7) can be written as e1+d ρ f (�) − β ≤ 0. It is easy to verify that f � (�) < 0, ∀ � ∈ (0, 1), i.e. f (·) is a monotonically decreasing function of � for � ∈ (0, 1). Furthermore, we know that f (0) > β and f (1) = −∞. Hence, there exist a unique �∗ for which f (�∗ ) = β. Thus, a randomly generated graph G, with high probability, results in a DRESS code with capacity FI > (1 − �∗ )kd, where �∗ = f −1 (β). VI. C ONCLUSION In this paper we proposed a new efficient class of codes for distributed storage that we call DRESS codes. These codes can simultaneously minimize bandwidth, disk I/O and computational overheads of system repair. DRESS codes enjoy these benefits due to their uncoded repair property resulting from their simple separation architecture that consists of an outer low-complexity erasure code and an inner repetition code. We proposed two randomized constructions for DRESS codes based on bins-and-balls and expander graphs approaches For both randomized constructions, we provided theoretical analysis in terms of concentration results and lower bounds. One of the most desirable features for cloud storage to have is to be able to easily grow or shrink the system to accommodate the fluctuations in the demand of the file or data. Randomized constructions proposed in this paper are the initial step towards achieving this goal. They also indicate the advantages, in terms of encoding, decoding and repair complexity, of abandoning a worst-case system design in favor of a probabilistic one. Many related questions remain open: Can we tighten the bounds here using bounding techniques more powerful than the union bound? Can load balancing techniques, such as “power-of-2” be used here to construct codes with better performance? R EFERENCES

e

k+t

,

k where the inequality follows from�the fact that k ≤ ( ne k ) . �� To satisfy P I,T I {AI,T } > 0 ≤ (1 − p), it is sufficient to have � �(kd−t) � � t n k k+t e ≤ 1 − p. θ k

[1] S. Ghemawat, H. Gobioff, and S.-T. Leung, “The google file system,” in 19th ACM Symposium on Operating Systems Principles, 2003. [2] G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels in Amazon.com, 2007. [3] S. Rhea, C. Wells, P. Eaton, D. Geels, B. Zhao, H. Weatherspoon, and J. Kubiatowicz, “Maintenance-free global data storage,” IEEE Internet Computing, pp. 40–49, 2001. [4] S. Rhea, P. Eaton, D. Geels, H. Weatherspoon, B. Zhao, and J. Kubiatowicz, “Pond: the oceanstore prototype,” in USENIX File and Storage Technologies (FAST), 2003.

[5] R. Bhagwan, K. Tati, Y.-C. Cheng, S. Savage, and G. M. Voelker, “Total recall: System support for automated availability management,” in Proc. NSDI, 2004. [6] F. Dabek, J. Li, E. Sit, J. Robertson, M. Kaashoek, and R. Morris, “Designing a DHT for low latency and high throughput,” in Proc. NSDI, 2004. [7] A. G. Dimakis, P. B. Godfrey, M. J. Wainwright, and K. Ramchandran, “Network coding for distributed storage systems,” in INFOCOM’07, 2007. [8] C. Suh and K. Ramchandran, “Exact regeneration codes for distributed storage repair using interference alignment,” in Proc. IEEE Intl Symp. on Information Theory (ISIT), 2010. [9] K. Rashmi, N. B. Shah, P. V. Kumar, and K. Ramchandran, “Explicit construction of optimal exact regenerating codes for distributed storage,” in Allerton Conference on Control, Computing, and Communication, 2009. [10] K. V. Rashmi, N. B. Shah, and P. V. Kumar, “Optimal exact-regenerating codes for distributed storage at the msr and mbr points via a productmatrix construction,” in arxiv:1005.4178, 2010. [11] V. Venkatesan, “Fast rebuilds in distributed storage systems using network coding,” tech. rep., IBM Research GmbH, Zurich Research Laboratory, 2009. [12] A. Duminuco and E. Biersack, “A practical study of regenerating codes for peer-to-peer backup systems,” in Distributed Computing Systems (ICDCS ’09), 2009. [13] S. E. Rouayheb and K. Ramchandran, “Fractional repetition codes for repair in distributed storage systems,” in Proc. Annual Allerton Conf. on Commun., Control, and Computing, 2010. [14] M. Luby, “Lt codes,” in Proc. IEEE Symposium on the Foundations of Computer Science (FOCS), 2002. [15] A. Dimakis, P. Godfrey, Y. Wu, M. Wainright, and K. Ramchandran, “Network coding for distributed storage systems,” IEEE Trans. Inform. Theory, vol. 56, pp. 4539–4551, Sep. 2010.