Building  an  Impenetrable  ZooKeeper   Kathleen  Ting,  [email protected],  @kate_Fng  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

  How  to  Kill  ZooKeeper     with  8  MisconfiguraFons    

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

2  

Who  Am  I?     Kathleen  Ting   –  Apache  ZooKeeper  Subject  MaRer  Expert   –  Apache  Sqoop  CommiRer,  PMC  member   –  Support  Manager,  Cloudera               Apache  ZooKeeper,  ZooKeeper,  Apache,  and  the  Apache  ZooKeeper  project  logo  are  trademarks  of  The   Apache  SoXware  FoundaFon.  

  Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

3  

What  is  ZooKeeper?   •  Coordinator  of  distributed  applica:ons  

•  Small  clusters  reliably  serve  many  coordina:on  needs  

•  Canary  in  the  Hadoop  coal  mine  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

4  

Why  is  ZooKeeper  Important?     •  High  Availability   –  Replicate  to  withstand  machine  failures     •  Distributed  Coordina:on   –  One  consistent  framework  to  rule  coordinaFon  across  all  systems   –  Observe  every  operaFon  by  every  client  in  exactly  the  same  order  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

5  

Who  Uses  ZooKeeper?   •  •  •  •  •  •  •  • 

HBase   MapReduce  (YARN)   HDFS  (High  Availability)   Solr   Kada   S4   Accumulo   Numerous  custom  soluFons:   hRps://cwiki.apache.org/confluence/display/ZOOKEEPER/poweredby  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

6  

Who  Doesn’t  Depend  on  ZooKeeper?   MR  

App  

HBase  

ZooKeeper  

HDFS  

JVM  /  Linux  

Disk/Network Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

7  

What  are  MisconfiguraFons?   •  Any  diagnosFc  Fcket  requiring  a  change  to  ZooKeeper  (HBase,  Hadoop..)   or  to  OS  config  files   •  Comprise  44%  of  Fckets   •  e.g.  resource-­‐allocaFon:  memory,  file-­‐handles,  disk-­‐space  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

8  

Ticket  Breakdown  by  Type   8%  

Misconfig  

4%  

Bug    

10%   44%  

App  

JVM/Linux   34%   Disk/NW  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

9  

Ticket  Breakdown  by  Component   3%   3%  

ZooKeeper  

10%  

HBase   7%  

43%  

Pig   Flume  

34%  

HDFS   System  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

10  

Analysis  of  a  Year’s  ZooKeeper  Tickets   •  Typically,  ZK  is  straight-­‐forward  to  set  up  and  operate   •  Issues  tend  to  be  client  issues  rather  than  ZK  issues   •  Our  examples  tend  to  be  HBase  and  Hadoop  centric   –  But  soluFons  are  applicable  to  other  systems  using  ZK  for  coordinaFon  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

11  

3  ZooKeeper  Ensemble  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

12  

Common  Issues   •  Connec:on  Mismanagement  

•  Time  Mismanagement  

•  Disk  Mismanagement  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

13  

Common  Issues   •  ConnecFon  Mismanagement  

•  Time  Mismanagement  

•  Disk  Mismanagement  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

14  

1.  Too  Many  ConnecFons   WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from /xx.x.xx.xxx - max is 60 !

How  can  it  be  resolved?   •  Running  out  of  ZK  connecFons?   –  Set  maxClientCnxns=200  in  zoo.cfg   •  HBase  client  leaking  connecFons?   –  Manually  close  connecFons   –  Fixed  in  HBASE-­‐3777,  HBASE-­‐4773,  and  HBASE-­‐5466  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

15  

2.  ConnecFon  Closes  Prematurely   ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately.!

  How  can  it  be  resolved?   •  If  hbase.cluster.distributed  =  true  in  hbase-­‐site,  then  in  zoo.cfg,  quorum   can’t  be  set  to  localhost     •  Bring  up  an  interface  with  the  same  IP  address  from  the  downed  ZK   without  any  service  running  on  port  2181  so  the  client  can  fail  over  to  the   next  ZK  server  from  the  quorum   •  In  hbase-­‐site,  set  hbase.zookeeper.recoverable.waipme=30000ms   –  Provides  enough  Fme  for  HBase  client  to  try  another  ZK  server   –  Fixed  in  HBASE-­‐3065     Strange  Loop  2012.  9/24/12.  Copyright  2012.     16   Cloudera  Inc.  All  rights  reserved.  

3.  Pig  Hangs  ConnecFng  to  HBase   WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectionException: Connection refused!

  What  causes  this?     •  LocaFon  of  ZK  quorum  is  not  known  to  Pig  (default  127.0.0.1:2181  fails)   How  can  it  be  resolved?   •  Use  Pig  10,  which  includes  PIG-­‐2115   •  If  there  is  overlap  between  TaskTrackers  and  ZK  quorum  nodes   –  Set  hbase.zookeeper.quorum  to  final  in  hbase-­‐site.xml     –  Otherwise,  add  "hbase.zookeeper.quorum=hadoophbasemaster.lan: 2181"  to  "pig.properFes”  (fixed  in  PIG-­‐2821)   Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

17  

Common  Issues   •  ConnecFon  Mismanagement  

•  Time  Mismanagement  

•  Disk  Mismanagement  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

18  

4.  Client  Session  Timed  Out   INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session , timeout of 40000ms exceeded!

  How  can  it  be  resolved?   •  ZK  and  HBase  need  same  session  Fmeout  values:   –  zoo.cfg:  maxSessionTimeout=180000   –  hbase-­‐site.xml:  zookeeper.session.Fmeout=180000   •  Don’t  co-­‐locate  ZK  with  IO-­‐intense  DataNode  or  RegionServer   •  Make  sure  your  session  Fmeout  is  sufficiently  long   •  Specify  right  amount  of  heap  and  tune  GC  flags   –  Turn  on  Parallel/CMS/Incremental  GC  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

19  

5.  Clients  Lose  ConnecFons   WARN org.apache.zookeeper.ClientCnxn - Session for server , unexpected error, closing socket connection and attempting reconnect! java.io.IOException: Broken pipe!

  Don’t  use  SSD  drive  for  ZK  transac:on  log   •  ZK  opFmized  for  mechanical  spindles  and  for  sequenFal  IO   •  SSD  provides  liRle  benefit  and  suffers  from  high  latency  spikes   –  hRp://storagemojo.com/2012/06/07/the-­‐ssd-­‐write-­‐cliff-­‐in-­‐real-­‐life/   –  SSD  pre-­‐allocates  disk  extents  to  avoid  directory  updates  but  that   doubles  the  load  on  the  SSD   –  SSD  disk  stops  for  40  sec  (which  is  greater  than  session  Fmeout)  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

20  

Common  Issues   •  ConnecFon  Mismanagement  

•  Time  Mismanagement  

•  Disk  Mismanagement  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

21  

6.  Unable  to  Load  Database  –   Unable  to  Run  Quorum  Server   FATAL Unable to load database on disk ! java.io.IOException: Failed to process transaction type: 2 error: KeeperErrorCode = NoNode for at org.apache.zookeeper.server.persistence.FileTxnSnapLog.res tore(FileTxnSnapLog.java:152)!

  How  can  it  be  resolved?     •  Archive  and  wipe  /var/zookeeper/version-­‐2  if  other  two  ZK  servers  are   running      

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

22  

7.  Unable  to  Load  Database  –   Unreasonable  Length  ExcepFon   FATAL Unable to load database on disk ! java.io.IOException: Unreasonable length = 1048583 ! at org.apache.jute.BinaryInputArchive.readBuffer (BinaryInputArchive.java:100) !

  How  can  it  be  resolved?     •  Server  allows  a  client  to  set  data  larger  than  the  server  can  read  from  disk   •  If  a  znode  is  not  readable,  increase  jute.maxbuffer   –  Look  for  "Packet  len    is  out  of  range"  in  the  client  log   –  Increase  it  by  20%     –  Set  in  JVMFLAGS="-­‐Djute.maxbuffer=yy"  bin/zkCli.sh   •  Fixed  in  ZOOKEEPER-­‐1513   Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

23  

8.  Failure  to  Follow  Leader   WARN org.apache.zookeeper.server.quorum.Learner: Exception when following the leader java.net.SocketTimeoutException: Read timed out !

  What  causes  this?     •  Disk  IO  contenFon,  Network  Issues   •  ZK  snapshot  is  too  large  (lots  of  ZK  nodes)   How  can  it  be  resolved?   •  Reduce  IO  contenFon  by  pupng  dataDir  on  dedicated  spindle   •  Increase  initLimit  on  all  ZK  servers  and  restart,  see  ZOOKEEPER-­‐1521   •  Monitor  network  (e.g.  ifconfig)   Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

24  

OpFmal  Ensemble  Size   #  of  ZK  Servers   Purpose  

 

1    

CoordinaFon  

3  

Reliability  for  producFon  environment  

5  

Permits  taking  one  server  down  for  maintenance  

Why not run 11 ZK servers?

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

25  

Trust  But  Verify   •  zk-­‐smoketest   –  hRps://github.com/phunt/zk-­‐smoketest   –  Verify  new,  updated,  &  exisFng  installaFons   –  IdenFfy  latency  issues   •  zk-­‐top   –  hRps://github.com/phunt/zktop   –  Unix  “top”  like  uFlity  for  ZK   •  4  leXer  words/JMX  (e.g.  ruok,  srvr)   –  hRp://zookeeper.apache.org/doc/current/ zookeeperAdmin.html#sc_zkCommands   –  Use  "stat"  to  get  an  idea  what  your  request  latency  looks  like   Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

26  

Best  PracFces   DOs   •  Separate  spindles  for  dataDir  &  dataLogDir     –  Avoids  compeFFon  between  logging  and  snapshots   –  Improves  throughput  and  latency   •  Allocate  3  or  5  ZK  servers     •  Tune  Garbage  CollecFon   •  Run  zkCleanup.sh  script  via  cron   DON’Ts   •  Don’t  co-­‐locate  ZK  with  I/O  intense  DataNode  or  RegionServer   –  ZK  is  latency  sensiFve   •  Don’t  use  SSD  drive  for  ZK  transacFon  log     Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

27  

Configure  ZooKeeper  Correctly..     ..and  it’ll  be  as  impenetrable  as  a   distributed  system  allows.       QuesFons?  

Strange  Loop  2012.  9/24/12.  Copyright  2012.     Cloudera  Inc.  All  rights  reserved.  

28  

Building an Impenetrable ZooKeeper - GitHub

Sep 24, 2012 - One consistent framework to rule coordinawon across all systems. – Observe every operawon ... HBase. App. MR. Disk/Network ... without any service running on port 2181 so the client can fail over to the next ZK server from ...
Download PDF
617KB Sizes 2 Downloads 397 Views
Report

Recommend Documents

Overview Building - GitHub
Using the external or internal host, after loading the RTE,. $ ncpBootMem -a ... ACP2=> tftp 4010000 . ACP2=> ssp w 0 ...
Building Blocks Design - GitHub
daily-ipad-app-blocksworld-hd-lets-you-build-and-play-with-3d-b/. [4] Maister ... zombies-run-naomi-alderman-app. [6] Ohan ... columbia.edu/~ohan/oda08.pdf.
Building patient-level predictive models - GitHub
Jul 19, 2017 - 3. 3.2 Preparing the cohort and outcome of interest . .... We will call this the cohort of interest or cohort for short. .... in a way that ensures R does not run out of memory, even when the data are large. We can get some .... plotDe
Building an Endowment
past directors and students, etc., the next step is to develop a website with all of ... tion debates or tournaments to be hosted), information for alumni (for .... visual aids, or financial support for ten students creating visual aids, or boxes to
Building ARM Trusted Firmware for Axxia - GitHub
For example, after installing the Yocto tools, set up the environment as follows. ... make PLAT=axxia USE_COHERENT_MEM=0 CRASH_REPORTING=1 bl31 or.
Building a Better Board Game - GitHub
Building a Better Board Game. Ryan Calme ... might garner from members of the site. ... A Pearson correlation value of 0.8 or greater indicates a reasonably good model fit. ..... 2 http://www.stat.berkeley.edu/~ledell/docs/dlab_ensembles.pdf ...
An Introduction to BigQuery - GitHub
The ISB-CGC platform includes an interactive Web App, over a Petabyte of TCGA data in Google Genomics and Cloud Storage, and tutorials and code ...
An Educated Guess (PDF) - GitHub
“I had some ideas for an email client so I built one today” ... up our species is to take the best and to spread it around to everybody, so that ... Today we're good ...
Delivering an Olympic Games - GitHub
Nov 26, 2013 - More than 900 servers, 1,000 network devices, ... 3.2.1 Java Scaffolding . ..... provided cluster services that were used during the disaster ...
Building an Endowment
some cases, it might be appropriate to offer a student academic credit in the way of an Independent Study for completing a team history. While websites are as ...
[M457.Ebook] Download PDF ZooKeeper: Distributed ...
Coordination in soft file system. So easy and fast! Reading the soft file publication ZooKeeper: Distributed. Process Coordination will certainly offer you simple way to check out. It could likewise be faster because you can read your publication Zoo
pdf-19134\lifting-the-impenetrable-veil-from-yellow ...
Microbiology, Immunology and Pathology, College of Veterinary Medicine and Biomedical. Sciences, Colorado State University, Fort Collins, and a member of the Columbia University,. Center for Infection and Immunity, Mailman School of Public Health. He
Develop for an international audience - GitHub
About me. • Developer and QA at www.transifex.com ... A Django based startup. • It's like a Github ... msginit -i app.pot -o locale/en/LC_MESSAGES/en.po -l en.
Emscripten: An LLVM-to-JavaScript Compiler - GitHub
May 14, 2013 - Emscripten, or (2) Compile a language's entire runtime into ...... html.) • Poppler and FreeType: Poppler12 is an open source. PDF rendering ...
Heterogeneous Agent Macroeconomics: An Example and an ... - GitHub
Keynesian multipliers should be big in liquidity trap. Crude Keynesianism: .... Calibrate income process to match macro and micro data. 2. Calibrate other ...
Choosing an Appropriate Performance Measure - GitHub
We compare the performance of the classifier (here, we use a support vector machine) ... Meeting Planner. Washington, DC: Society for Neuroscience, 2011.
An Automated Interaction Application on Twitter - GitHub
select the responses which are best matches to the user input ..... the last response when the bot talked about free ... User> go and take control the website that I.
Emscripten: An LLVM-to-JavaScript Compiler - GitHub
Apr 6, 2011 - written in languages other than JavaScript on the web: (1). Compile code ... pile that into JavaScript using Emscripten, or (2) Compile a ... detail the methods used in Emscripten to deal with those ..... All the tests were run on a Len
Building Single Page Applications using Web API and ... - GitHub
This book is the pdf version of the online post in chsakell's Blog and ..... For our application there will be only the Admin role (employees) but we will discuss later the scalability options we have in ...... not only to be authenticated but also b
Reimagining IT for an omnichannel world - GitHub
Figure 1: IT budgets are growing rapidly, and most of the expansion is in new projects. 25%. CAGR. Ratio. 2.4x. Indexed IT hours by project type. Note: Run category includes IT services, maintenance, admin, PMO admin and training time; grow category
Mariokart An Autonomous Go-Kart - GitHub
Mar 9, 2011 - Make a robust platform for future projects ... Nice hardware platform for future years. - Project almost stuck to time ... Automation Software. 45d.