Review Board 1.7.22


HBASE-5547: Don't delete HFiles when in backup mode.

Review Request #4633 - Created April 4, 2012 and submitted

Jesse Yates
HBASE-5547
Reviewers
hbase
lhofhansl, stack
hbase-git
Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

Implementation wise: 
    Updated the HMasterInterface to pass the calls onto the zookeeper.
    Added a zk listener to handle updates from the master to the RS to backup.
    Added a utility for removing files and finding archive directories
    Added tests for the regionserver and catalogjanitor approaches.
    Added creation of manager in regionserver.
Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

Diff revision 15 (Latest)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

  1. hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java: Loading...
  2. hbase-server/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java: Loading...
  3. hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java: Loading...
  4. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java: Loading...
  5. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java: Loading...
  6. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveTableMonitor.java: Loading...
  7. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java: Loading...
  8. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/TableHFileArchiveTracker.java: Loading...
  9. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/ZKTableArchiveClient.java: Loading...
  10. hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java: Loading...
  11. hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java: Loading...
  12. hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java: Loading...
  13. hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java: Loading...
  14. hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java: Loading...
  15. hbase-server/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java: Loading...
  16. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java: Loading...
  17. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java: Loading...
  18. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java: Loading...
  19. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java: Loading...
  20. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java: Loading...
This diff has been split across 2 pages: 1 2 >
hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
Revision e0e3e6c New Change
[20] 686 lines
[+20]
687
   */
687
   */
688
  public static final byte [] NO_NEXT_INDEXED_KEY = Bytes.toBytes("NO_NEXT_INDEXED_KEY");
688
  public static final byte [] NO_NEXT_INDEXED_KEY = Bytes.toBytes("NO_NEXT_INDEXED_KEY");
689
  /** delimiter used between portions of a region name */
689
  /** delimiter used between portions of a region name */
690
  public static final int DELIMITER = ',';
690
  public static final int DELIMITER = ',';
691

    
   
691

   

    
   
692
  /** Configuration key for the directory to backup HFiles for a table */

    
   
693
  public static final String HFILE_ARCHIVE_DIRECTORY = "hbase.table.archive.directory";

    
   
694

   
692
  private HConstants() {
695
  private HConstants() {
693
    // Can't be instantiated with this ctor.
696
    // Can't be instantiated with this ctor.
694
  }
697
  }
695
}
698
}
hbase-server/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java
Revision 58a0407 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveTableMonitor.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/TableHFileArchiveTracker.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/ZKTableArchiveClient.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
Revision 1492548 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Revision a97a433 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java
Revision 1be3cb6 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java
Revision ffb3ef5 New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
Revision ca964ad New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java
Revision f29b0cf New Change
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java
New File
 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
New File
 
  1. hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java: Loading...
  2. hbase-server/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java: Loading...
  3. hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java: Loading...
  4. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java: Loading...
  5. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java: Loading...
  6. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveTableMonitor.java: Loading...
  7. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java: Loading...
  8. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/TableHFileArchiveTracker.java: Loading...
  9. hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/ZKTableArchiveClient.java: Loading...
  10. hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java: Loading...
  11. hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java: Loading...
  12. hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java: Loading...
  13. hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java: Loading...
  14. hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java: Loading...
  15. hbase-server/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java: Loading...
  16. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java: Loading...
  17. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java: Loading...
  18. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java: Loading...
  19. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java: Loading...
  20. hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java: Loading...
This diff has been split across 2 pages: 1 2 >