Review Board 1.7.22


HBASE-5547: Don't delete HFiles when in backup mode.

Review Request #4633 - Created April 4, 2012 and submitted

Jesse Yates
HBASE-5547
Reviewers
hbase
lhofhansl, stack
hbase-git
Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

Implementation wise: 
    Updated the HMasterInterface to pass the calls onto the zookeeper.
    Added a zk listener to handle updates from the master to the RS to backup.
    Added a utility for removing files and finding archive directories
    Added tests for the regionserver and catalogjanitor approaches.
    Added creation of manager in regionserver.
Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

Diff revision 6

This is not the most recent revision of the diff. The latest diff is revision 15. See what's changed.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

  1. hbase-server/src/main/java/org/apache/hadoop/hbase/HConstants.java: Loading...
  2. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java: Loading...
  3. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java: Loading...
  4. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java: Loading...
  5. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java: Loading...
  6. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/TableHFileArchiveTracker.java: Loading...
  7. hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java: Loading...
  8. hbase-server/src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java: Loading...
  9. hbase-server/src/main/java/org/apache/hadoop/hbase/io/Reference.java: Loading...
  10. hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java: Loading...
  11. hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java: Loading...
  12. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java: Loading...
  13. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java: Loading...
  14. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java: Loading...
  15. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java: Loading...
  16. hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java: Loading...
  17. hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java: Loading...
  18. hbase-server/src/main/resources/hbase-default.xml: Loading...
  19. hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java: Loading...
  20. hbase-server/src/test/java/org/apache/hadoop/hbase/backup/SimpleHFileArchiveTableMonitor.java: Loading...
  21. hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchivingCleanup.java: Loading...
  22. hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestRegionDisposer.java: Loading...
  23. hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java: Loading...
  24. hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java: Loading...
  25. hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java: Loading...
  26. hbase-server/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java: Loading...
  27. hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java: Loading...
hbase-server/src/main/java/org/apache/hadoop/hbase/HConstants.java
Revision 5f14680 New Change
[20] 670 lines
[+20]
671
   * The byte array represents for NO_NEXT_INDEXED_KEY;
671
   * The byte array represents for NO_NEXT_INDEXED_KEY;
672
   * The actual value is irrelevant because this is always compared by reference.
672
   * The actual value is irrelevant because this is always compared by reference.
673
   */
673
   */
674
  public static final byte [] NO_NEXT_INDEXED_KEY = Bytes.toBytes("NO_NEXT_INDEXED_KEY");
674
  public static final byte [] NO_NEXT_INDEXED_KEY = Bytes.toBytes("NO_NEXT_INDEXED_KEY");
675

    
   
675

   

    
   
676
  /** Configuration key for the directory to backup HFiles for a table */

    
   
677
  public static final String HFILE_ARCHIVE_DIRECTORY = "hbase.table.archive.directory";

    
   
678
  public static final String HFILE_ARCHIVE_ZNODE_PARENT = "hfilearchive";

    
   
679

   
676
  private HConstants() {
680
  private HConstants() {
677
    // Can't be instantiated with this ctor.
681
    // Can't be instantiated with this ctor.
678
  }
682
  }
679
}
683
}
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/TableHFileArchiveTracker.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
Revision 9bf2396 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/Reference.java
Revision 5804a84 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
Revision e751f65 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Revision a3af820 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Revision 86f8e77 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Revision 89e3d25 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
Revision 6884d53 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
Revision 7157c04 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
Revision 33bc1d0 New Change
 
hbase-server/src/main/resources/hbase-default.xml
Revision 370b06f New Change
 
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
Revision be932d7 New Change
 
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/SimpleHFileArchiveTableMonitor.java
New File
 
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchivingCleanup.java
New File
 
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestRegionDisposer.java
New File
 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
Revision 04ed1a3 New Change
 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
Revision 5303608 New Change
 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java
New File
 
hbase-server/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java
New File
 
hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
Revision 3f61cfb New Change
 
  1. hbase-server/src/main/java/org/apache/hadoop/hbase/HConstants.java: Loading...
  2. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java: Loading...
  3. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java: Loading...
  4. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java: Loading...
  5. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java: Loading...
  6. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/TableHFileArchiveTracker.java: Loading...
  7. hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java: Loading...
  8. hbase-server/src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java: Loading...
  9. hbase-server/src/main/java/org/apache/hadoop/hbase/io/Reference.java: Loading...
  10. hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java: Loading...
  11. hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java: Loading...
  12. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java: Loading...
  13. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java: Loading...
  14. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java: Loading...
  15. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java: Loading...
  16. hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java: Loading...
  17. hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java: Loading...
  18. hbase-server/src/main/resources/hbase-default.xml: Loading...
  19. hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java: Loading...
  20. hbase-server/src/test/java/org/apache/hadoop/hbase/backup/SimpleHFileArchiveTableMonitor.java: Loading...
  21. hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchivingCleanup.java: Loading...
  22. hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestRegionDisposer.java: Loading...
  23. hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java: Loading...
  24. hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java: Loading...
  25. hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java: Loading...
  26. hbase-server/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java: Loading...
  27. hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java: Loading...