Building Native Erasure Coding Support in HDFS +

+

+

+

+

+

Zhe Zhang , Kai Zheng , Bo Li , Andrew Wang , Vinayakumar B , Uma Gangumalla , +

+

+

+

+

Todd Lipcon , Yi Liu , Weihua Jiang , Aaron Myers & Silvius Rus +Cloudera, *Intel,

[email protected], [email protected]

Problem Statement

Unique Research Challenges Reduce NameNode overhead

Benefits of triplication − Fault tolerance − Better locality − Load balancing

− Hierarchical block naming protocol − Fixed placement groups − Peer monitoring and recovery in a group

200% overhead Secondary replicas rarely accessed

BlockManager

Erasure coding?

blocksMap

− Same or better fault tolerance − < 50% overhead in a typical setup

blockGroupsMap blockGroup 0

blk_1 0x00 0x00

blk_1 0x00 0x08

DN 0

Data Layouts

128~256 M

DataNode5

…

block 5

block 1

block 0

0~128 M

DataNode1

DN 8

block ID

…

Contiguous DataNode0

Faster codec calculation

flag

DataNode6

DataNode8

Preserve data locality

…

640~768 M

data blocks

index in group

block group ID

− Hybrid storage forms for individual files INodeFile

parity blocks

block block

Good compatibility with locality-sensitive applications Poor handling of small files

blockGroup blockGroup

block runtime choice

DataNode1

0~1M 6~7M … …

1~2M 7~8M … …

…

block 5

DataNode0 block 1

block 0

Striping DataNode5 5~6M 11~12M … …

DataNode6

DataNode8

Preliminary Results

… File categorization

data blocks

parity blocks

Improved I/O performance with high speed networking Heavier memory and CPU overhead on NameNode

− − − −

Storage usage simulation

Assuming (6,3) coding schema Small files: < 1 block, Medium files: 1~6 blocks Large files: > 6 blocks (1 group)

Cluster A Profile Replication Ceph (before firefly) Lustre

HDFS

96.29%

Erasure Coding Ceph (optional w/ firefly) QFS Facebook f4 Azure

Memory usage calculation

− Contiguous skips a file if parity data is larger than secondary replicas

Cluster B Profile

file count

86.59%

− Each block uses ~78 bytes − Each additional replica location uses ~16 bytes

Cluster C Profile 99.64%

file count

file count

76.05%

Striping

Contiguous

space usage

64.61%

HDFS-EC aims to enable all 4 forms to support heterogenous workloads

space usage

space usage

36.03%

40.08%

23.89%

26.06%

20.75%

HDFS-EC Architecture

1.86% 9.33%

small

Storage Saving

ECManager BlockGroup

ECSchema DataNode ECWorker

DataNode

50.00%

ECSchema

ECWorker

striping

large

medium

small

large

small

Top 2% files occupy ~40% space

Memory Overhead 400%

0.36%

2.03%

1.85%

Top 2% files occupy ~65% space

NameNode

BlockGroup

medium

11.38%

Storage Saving 50.00%

striping

medium

large

Dominated by small files

Memory Overhead

striping

0.00% 3.20%

350.00%

striping

Storage Saving

Memory Overhead

48.00% striping

540.00% striping

Client DataNode

ECWorker

ECClient

contiguous

…

34.00%

contiguous

DataNode

27.00%

ECWorker

BlockGroup: data and parity blocks in an erasure coding group ECSchema: e.g., 6 data + 3 parity blocks, with Reed-Solomon ECManager: group allocation, placement, monitoring ECWorker/ECClient: codec calculation and striped read/write logics

striping w/ hierarchical block naming 44%

contig. 3%

striping w/ hierarchical block naming

striping w/ hierarchical block naming contig.

31.00% 8.00%

86.00% 0.02%

contig. 0.00%

Problem Statement Data Layouts Unique Research ... - GitHub

Cluster C Profile. HDFS-EC Architecture. NameNode. ECManager. DataNode. ECWorker. Client. ECClient. BlockGroup. ECSchema. BlockGroup. ECSchema. DataNode. DataNode. DataNode … ECWorker. ECWorker. ECWorker. BlockGroup: data and parity blocks in an erasure coding group. ECSchema: e.g., 6 data + 3 ...
Download PDF
2MB Sizes 0 Downloads 395 Views
Report

Recommend Documents

Research Data Management Training - GitHub
Overview. Research Data management Training Working Group: Approach and. Methodology ... CC Australia ported licence) licence. ... http://www.griffith.edu.au/__data/assets/pdf_file/0009/528993/Best_Practice_Guidelines.pdf. University of ...
Statement of Accomplishment - GitHub
You have successfully completed the online Machine Learn- ... not confer a Stanford grade; it does not confer Stanford credit; it does not confer a Stanford ...
Funded Research Projects in Data Science - GitHub
logs, social media posts and blogs, satellites ... Due to the proliferation of social media, sensors, and the Internet of. Things .... “troll” on an Internet Web site simply ..... levels of “bad” cholesterol that sometimes kills adults in the
Sample Statement of Work - GitHub
CONFIDENTIAL: The contents of this document are confidential and are intended exclusively for the designated recipients. The contents of this page is defined ...
Research Statement
Jun 1, 2017 - Moreover, it encourages me to investigate alternative .... how we can develop a quantum annealing algorithm to compute the expected energy.
Research Statement -
Nov 2, 2012 - First, I find that search frictions generate a counter-cyclical interest rate spread by varying bank loans in both extensive and intensive margins, which amplifies ... mechanism reduces intertemporal substitution towards savings.
Research Statement
Nov 7, 2016 - (2006) argue that, first, public health infrastructure and, later, medical innovations made large contributions to the mortality ... In particular, I draw on transcriptions of hand-collected archival material, complete-count census reco
Research statement
Nov 29, 2016 - The energy of φ ∈ Ham is. E(φ) := inf{. ∫ 1 .... alternative: 1. b1(L;Z) is ... point of L, whose energy is smaller than the Hofer distance. When the ...
Research Statement
Nov 2, 2012 - In my research, I aim to understand the linkage between real and finan- ... In my job market paper, titled “Search Frictions, Bank Leverage, and ...
research statement
Fractal geometry is the study, within geometric measure theory, of sets and .... game, and the target set S is said to be (β, c, H)-potential winning if Alice has a ...
research statement
forward and automatically learn from these data sets answers to sociological ... have significant implications for sociologists, information analysts as well as online ..... Towards Better and Faster Topic Models: There is still room for improvement 
Research Statement
a ten-year book series of design recommendations for ITS [19]. ... tionships may be persistent (e.g., in-person social networks) or temporary (e.g., students ...
Research Statement
Symbolic Logic, 63(4):1404–1412, 1998. [3] Arthur W. Apter and Joel David Hamkins. Universal indestructibility. Kobe J. Math., 16(2):119–130, 1999. [4] Arthur ...
STRUCTURE and Problem #2 - GitHub
Feb 7, 2017 - Uses multi-locus genotype data to investigate population ... the data betwee successive K values ... For this project, analyzing Fst outlier loci.
Research Statement Background
infinite descending chains and incompatible elements in the consistency hierarchy, but it is a surprising empirical fact that all natural extensions of ZFC are well-ordered. Any cardinal whose existence can not be proved in ZFC is considered a large
Statement of Research
are the major tools used in this methodology. ... to develop useful and powerful tools for optimal decision making. Network ... Automation Conference, 2009.
Problem Tutorial: “Apples” - GitHub
careful when finding x, cause the multiplication might not fit in the limits of long long. Also don't the forget the case when there's no answer. Page 1 of 1.
Online Course Statement of Accomplishment - GitHub
Sep 20, 2013 - HAS SUCCESSFULLY COMPLETED A FREE ONLINE OFFERING OF ... used by companies and organizations to drive a culture of innovation.
Statement of Accomplishment SERKAN SERTTOP - GitHub
Nov 15, 2013 - Markets Hypothesis, the role of information in pricing, historical ... A GEORGIA INSTITUTE OF TECHNOLOGY DEGREE; AND IT DOES NOT.
Statement of Accomplishment SERKAN SERTTOP - GitHub
Mar 21, 2014 - This integrative course covers randomness (its definition, use and ... PLEASE NOTE: THE ONLINE OFFERING OF THIS CLASS DOES NOT REFLECT ... A NATIONAL UNIVERSITY OF SINGAPORE DEGREE; AND IT DOES.
Statement of Accomplishment SHICHAO YUAN - GitHub
Dec 1, 2013 - This advanced undergraduate programming course covers the ... PROFESSOR OF COMPUTER SCIENCE. ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE. DISCLAIMER : THIS ONLINE OFFERING DOES NOT REFLECT THE ... A ECOLE POLYTECHNIQUE FÉDÉRALE DE LAU
Online Course Statement of Accomplishment WALTER ... - GitHub
Jan 7, 2014 - PROFESSOR OF COMPUTER SCIENCE, ... GRADE, COURSE CREDIT OR DEGREE, AND IT DOES NOT VERIFY THE IDENTITY OF.
Online Course Statement of Accomplishment - GitHub
Aug 7, 2013 - challenges and opportunities in the global education space, and ... Chief Technology Officer and Assistant Dean, School of Education, Stanford ...
Statement of Accomplishment WALTER SCHULZE - GitHub
HAS SUCCESSFULLY COMPLETED BERKLEE COLLEGE OF MUSIC'S MOOC: Introduction to Music Production. The recipient of this statement spent at least 28 hours studying microphones, mixing boards, DAWs, audio effects, synthesizers, and other essential music pr