Review Board 1.7.22


HBASE-4377 [hbck] Offline rebuild .META. from fs data only

Review Request #2126 - Created Sept. 29, 2011 and submitted

Jonathan Hsieh
trunk
HBASE-4377
Reviewers
hbase
apurtell, stack
hbase-git
commit fbf82c17be6b3ecca5a981f5270cf93aac26e479
Author: Jonathan Hsieh <jon@cloudera.com>
Date:   Wed Sep 28 10:18:11 2011 -0700

    HBASE-4377 [hbck] Offline rebuild .META. from fs data only
    

This patch rebuilds a new .META. table by reading all the .regioninfo files in the hbase main directory.  It depends on the yet to be committed HBASE-4515 (either my verison or Gary's version), HBASE-4509, and HBASE-4506.  

Some follow on work includes backporting to 0.90, auto-patching true holes, and adding documentation.
An earlier version of this code (backported to 0.90) was used to diagnose and repair a cluster that had 2700 inconsistencies due to failed splits (the cluster was underprovisioned memory-wise, and on restart, the some regions would start splitting and then die due to oome's).  This was not actually used on a live cluster -- it was used to reconstruct a .META. from .regioninfo's laid out in hbase's directory structure.

Note also that this is not an automatic fix -- whenever any problems are found, this bails out but dumps info on holes, suggests some fixes, and displays sets of overlapping regions.  It is up to the user to merge regions, to create .regioninfo files to plug hole, and to do any potential data loosing operations.

The tests demonstrate current expected behavior -- rebuild meta if things line up, and fail without making modifications if holes or overlaps exist.
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
Revision b9c850d New Change
[20] 367 lines
[+20] [+] private static void bootstrap(final Path rd, final Configuration c)
368
      LOG.error("bootstrap", e);
368
      LOG.error("bootstrap", e);
369
      throw e;
369
      throw e;
370
    }
370
    }
371
  }
371
  }
372

    
   
372

   
373
  private static void setInfoFamilyCachingForRoot(final boolean b) {
373
  /**

    
   
374
   * Enable in memory caching for root

    
   
375
   * 

    
   
376
   * @param b enable in memory caching?

    
   
377
   */

    
   
378
  public static void setInfoFamilyCachingForRoot(final boolean b) {
374
    for (HColumnDescriptor hcd:
379
    for (HColumnDescriptor hcd:
375
        HTableDescriptor.ROOT_TABLEDESC.families.values()) {
380
        HTableDescriptor.ROOT_TABLEDESC.families.values()) {
376
       if (Bytes.equals(hcd.getName(), HConstants.CATALOG_FAMILY)) {
381
       if (Bytes.equals(hcd.getName(), HConstants.CATALOG_FAMILY)) {
377
         hcd.setBlockCacheEnabled(b);
382
         hcd.setBlockCacheEnabled(b);
378
         hcd.setInMemory(b);
383
         hcd.setInMemory(b);
379
     }
384
     }
380
    }
385
    }
381
  }
386
  }
382

    
   
387

   
383
  private static void setInfoFamilyCachingForMeta(final boolean b) {
388
  /**

    
   
389
   * Enable in memory caching for meta

    
   
390
   * 

    
   
391
   * @param b enable in memory caching?

    
   
392
   */

    
   
393
  public static void setInfoFamilyCachingForMeta(final boolean b) {
384
    for (HColumnDescriptor hcd:
394
    for (HColumnDescriptor hcd:
385
        HTableDescriptor.META_TABLEDESC.families.values()) {
395
        HTableDescriptor.META_TABLEDESC.families.values()) {
386
      if (Bytes.equals(hcd.getName(), HConstants.CATALOG_FAMILY)) {
396
      if (Bytes.equals(hcd.getName(), HConstants.CATALOG_FAMILY)) {
387
        hcd.setBlockCacheEnabled(b);
397
        hcd.setBlockCacheEnabled(b);
388
        hcd.setInMemory(b);
398
        hcd.setInMemory(b);
[+20] [20] 107 lines
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
Revision 8465724 New Change
 
src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java
New File
 
src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
Revision fae0881 New Change
 
src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java
New File
 
src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuild.java
New File
 
  1. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java: Loading...
  2. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java: Loading...
  3. src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java: Loading...
  4. src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java: Loading...
  5. src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java: Loading...
  6. src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuild.java: Loading...