Review Board 1.7.22


HBASE-5547: Don't delete HFiles when in backup mode.

Review Request #4633 - Created April 4, 2012 and submitted

Jesse Yates
HBASE-5547
Reviewers
hbase
lhofhansl, stack
hbase-git
Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

Implementation wise: 
    Updated the HMasterInterface to pass the calls onto the zookeeper.
    Added a zk listener to handle updates from the master to the RS to backup.
    Added a utility for removing files and finding archive directories
    Added tests for the regionserver and catalogjanitor approaches.
    Added creation of manager in regionserver.
Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

Diff revision 2

This is not the most recent revision of the diff. The latest diff is revision 15. See what's changed.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

  1. src/main/java/org/apache/hadoop/hbase/HConstants.java: Loading...
  2. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java: Loading...
  3. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java: Loading...
  4. src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java: Loading...
  5. src/main/java/org/apache/hadoop/hbase/master/HMaster.java: Loading...
  6. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java: Loading...
  7. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java: Loading...
  8. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java: Loading...
  9. src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java: Loading...
  10. src/main/java/org/apache/hadoop/hbase/regionserver/Store.java: Loading...
  11. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java: Loading...
  12. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java: Loading...
  13. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java: Loading...
  14. src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java: Loading...
  15. src/main/resources/hbase-default.xml: Loading...
  16. src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java: Loading...
  17. src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java: Loading...
  18. src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java: Loading...
  19. src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java: Loading...
  20. src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java: Loading...
src/main/java/org/apache/hadoop/hbase/HConstants.java
Revision a4b989e New Change
[20] 651 lines
[+20]
652
    "hbase.regionserver.wal.enablecompression";
652
    "hbase.regionserver.wal.enablecompression";
653

    
   
653

   
654
/** Region in Transition metrics threshold time */
654
  /** Region in Transition metrics threshold time */
655
  public static final String METRICS_RIT_STUCK_WARNING_THRESHOLD="hbase.metrics.rit.stuck.warning.threshold";
655
  public static final String METRICS_RIT_STUCK_WARNING_THRESHOLD="hbase.metrics.rit.stuck.warning.threshold";
656

    
   
656

   

    
   
657
  /** Configuration key for the directory to backup HFiles for a table */

    
   
658
  public static final String HFILE_ARCHIVE_DIRECTORY = "hbase.table.archive.directory";

    
   
659
  public static final String DEFAULT_HFILE_ARCHIVE_DIRECTORY = ".archive";

    
   
660
  public static final String HFILE_ARCHIVE_ZNODE_PARENT = "hfilearchive";

    
   
661

   
657
  private HConstants() {
662
  private HConstants() {
658
    // Can't be instantiated with this ctor.
663
    // Can't be instantiated with this ctor.
659
  }
664
  }
660
}
665
}
src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
Revision 16e4017 New Change
 
src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
New File
 
src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
Revision 79d5fdd New Change
 
src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Revision fb21bdd New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java
New File
 
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Revision c3df319 New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Revision 8a61f7d New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
Revision 6884d53 New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
Revision 509a467 New Change
 
src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
New File
 
src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
New File
 
src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java
New File
 
src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
Revision 4fc105f New Change
 
src/main/resources/hbase-default.xml
Revision 44ee689 New Change
 
src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
Revision 41616c8 New Change
 
src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
Revision b4dcb83 New Change
 
src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java
New File
 
src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java
New File
 
src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
Revision 7d02759 New Change
 
  1. src/main/java/org/apache/hadoop/hbase/HConstants.java: Loading...
  2. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java: Loading...
  3. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java: Loading...
  4. src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java: Loading...
  5. src/main/java/org/apache/hadoop/hbase/master/HMaster.java: Loading...
  6. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java: Loading...
  7. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java: Loading...
  8. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java: Loading...
  9. src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java: Loading...
  10. src/main/java/org/apache/hadoop/hbase/regionserver/Store.java: Loading...
  11. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java: Loading...
  12. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java: Loading...
  13. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java: Loading...
  14. src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java: Loading...
  15. src/main/resources/hbase-default.xml: Loading...
  16. src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java: Loading...
  17. src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java: Loading...
  18. src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java: Loading...
  19. src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java: Loading...
  20. src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java: Loading...