Review Board 1.7.22


HBASE-5547: Don't delete HFiles when in backup mode.

Review Request #4633 - Created April 4, 2012 and submitted

Jesse Yates
HBASE-5547
Reviewers
hbase
lhofhansl, stack
hbase-git
Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

Implementation wise: 
    Updated the HMasterInterface to pass the calls onto the zookeeper.
    Added a zk listener to handle updates from the master to the RS to backup.
    Added a utility for removing files and finding archive directories
    Added tests for the regionserver and catalogjanitor approaches.
    Added creation of manager in regionserver.
Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

Diff revision 11

This is not the most recent revision of the diff. The latest diff is revision 15. See what's changed.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

  1. hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java: Loading...
  2. hbase-server/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java: Loading...
  3. hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java: Loading...
  4. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java: Loading...
  5. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java: Loading...
  6. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/ZKTableArchiveClient.java: Loading...
  7. hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java: Loading...
  8. hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java: Loading...
  9. hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java: Loading...
  10. hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java: Loading...
  11. hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java: Loading...
  12. hbase-server/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java: Loading...
  13. hbase-server/src/main/java/org/apache/hadoop/hbase/master/backup/HFileArchiveTableMonitor.java: Loading...
  14. hbase-server/src/main/java/org/apache/hadoop/hbase/master/backup/TableHFileArchiveTracker.java: Loading...
  15. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/ArchivingHFileCleaner.java: Loading...
  16. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java: Loading...
  17. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java: Loading...
  18. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java: Loading...
  19. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java: Loading...
  20. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java: Loading...
This diff has been split across 2 pages: 1 2 >
hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
Revision 82cf976 New Change
[20] 680 lines
[+20]
681
   */
681
   */
682
  public static final byte [] NO_NEXT_INDEXED_KEY = Bytes.toBytes("NO_NEXT_INDEXED_KEY");
682
  public static final byte [] NO_NEXT_INDEXED_KEY = Bytes.toBytes("NO_NEXT_INDEXED_KEY");
683
  /** delimiter used between portions of a region name */
683
  /** delimiter used between portions of a region name */
684
  public static final int DELIMITER = ',';
684
  public static final int DELIMITER = ',';
685

    
   
685

   

    
   
686
  /** Configuration key for the directory to backup HFiles for a table */

    
   
687
  public static final String HFILE_ARCHIVE_DIRECTORY = "hbase.table.archive.directory";

    
   
688

   
686
  private HConstants() {
689
  private HConstants() {
687
    // Can't be instantiated with this ctor.
690
    // Can't be instantiated with this ctor.
688
  }
691
  }
689
}
692
}
hbase-server/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java
Revision 58a0407 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/ZKTableArchiveClient.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
Revision 8eed2fa New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
Revision 1492548 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Revision af6caba New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java
Revision 1be3cb6 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java
Revision ffb3ef5 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java
Revision f29b0cf New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/backup/HFileArchiveTableMonitor.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/backup/TableHFileArchiveTracker.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/ArchivingHFileCleaner.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
New File
 
  1. hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java: Loading...
  2. hbase-server/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java: Loading...
  3. hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java: Loading...
  4. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java: Loading...
  5. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java: Loading...
  6. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/ZKTableArchiveClient.java: Loading...
  7. hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java: Loading...
  8. hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java: Loading...
  9. hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java: Loading...
  10. hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java: Loading...
  11. hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java: Loading...
  12. hbase-server/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java: Loading...
  13. hbase-server/src/main/java/org/apache/hadoop/hbase/master/backup/HFileArchiveTableMonitor.java: Loading...
  14. hbase-server/src/main/java/org/apache/hadoop/hbase/master/backup/TableHFileArchiveTracker.java: Loading...
  15. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/ArchivingHFileCleaner.java: Loading...
  16. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java: Loading...
  17. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java: Loading...
  18. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java: Loading...
  19. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java: Loading...
  20. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java: Loading...
This diff has been split across 2 pages: 1 2 >