Review Board 1.7.22


HBASE-5616 Make compaction code standalone

Review Request #4469 - Created March 23, 2012 and updated

Michael Stack
hbase-5616
Reviewers
hbase
hbase-git
Introduces a standalone CompactionTool under src/test. Can call it from its main
and have it run compactions on arbitrary storefiles.

Compaction code is moved out of Store into a new Compactor class.

CompactionTool needs a Store and a mocked up Region to run because
compacting uses a StoreScanner (A StoreScanner needs a Store. A
Store needs an HRegion). Rather than expect to be passed a coherent
HRegion pointer, instead, we fake up one using Mockito.

I tried to break out of HRegion a "Region" Interface. This Region Interface
would have a basic subset of HRegion functionality and we'd pass this
instead of HRegion to SplitTransaction, Store, CompactionRequest, etc.
but the change would be massive. Everything expects to be able to do
anything on an HRegion. This is work we need to do but I"m not doing
it as part of this patch.

M src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
Use new CompactionTool instead of asking Store directly to compact.
M src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
Formatting.
M src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
Format the log message so properly spaced.
A src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
New compactiontool. Runs all compactions.
M src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Make getOpenAndCloseThreadPool static.
M src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
Move compaction code out to CompactionTool.
Refactor so can override ttl and Store home directory so Store
is more mockable; can now stand up a Store on its own w/o real
HRegion context.
M src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
Formatting.

 
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
Revision b52e5d3 New Change
[20] 625 lines
[+20] [+] public void run() {
626
  public void logStats() {
626
  public void logStats() {
627
    if (!LOG.isDebugEnabled()) return;
627
    if (!LOG.isDebugEnabled()) return;
628
    // Log size
628
    // Log size
629
    long totalSize = heapSize();
629
    long totalSize = heapSize();
630
    long freeSize = maxSize - totalSize;
630
    long freeSize = maxSize - totalSize;
631
    LruBlockCache.LOG.debug("LRU Stats: " +
631
    LruBlockCache.LOG.debug("Stats: " +
632
        "total=" + StringUtils.byteDesc(totalSize) + ", " +
632
        "total=" + StringUtils.byteDesc(totalSize) + ", " +
633
        "free=" + StringUtils.byteDesc(freeSize) + ", " +
633
        "free=" + StringUtils.byteDesc(freeSize) + ", " +
634
        "max=" + StringUtils.byteDesc(this.maxSize) + ", " +
634
        "max=" + StringUtils.byteDesc(this.maxSize) + ", " +
635
        "blocks=" + size() +", " +
635
        "blocks=" + size() +", " +
636
        "accesses=" + stats.getRequestCount() + ", " +
636
        "accesses=" + stats.getRequestCount() + ", " +
637
        "hits=" + stats.getHitCount() + ", " +
637
        "hits=" + stats.getHitCount() + ", " +
638
        "hitRatio=" +
638
        "hitRatio=" +
639
          (stats.getHitCount() == 0 ? "0" : (StringUtils.formatPercent(stats.getHitRatio(), 2)+ ", ")) +
639
          (stats.getHitCount() == 0 ? "0" : (StringUtils.formatPercent(stats.getHitRatio(), 2)+ ", ")) + ", " +
640
        "cachingAccesses=" + stats.getRequestCachingCount() + ", " +
640
        "cachingAccesses=" + stats.getRequestCachingCount() + ", " +
641
        "cachingHits=" + stats.getHitCachingCount() + ", " +
641
        "cachingHits=" + stats.getHitCachingCount() + ", " +
642
        "cachingHitsRatio=" +
642
        "cachingHitsRatio=" +
643
          (stats.getHitCachingCount() == 0 ? "0" : (StringUtils.formatPercent(stats.getHitCachingRatio(), 2)+ ", ")) +
643
          (stats.getHitCachingCount() == 0 ? "0" : (StringUtils.formatPercent(stats.getHitCachingRatio(), 2)+ ", ")) + ", " +
644
        "evictions=" + stats.getEvictionCount() + ", " +
644
        "evictions=" + stats.getEvictionCount() + ", " +
645
        "evicted=" + stats.getEvictedCount() + ", " +
645
        "evicted=" + stats.getEvictedCount() + ", " +
646
        "evictedPerRun=" + stats.evictedPerEviction());
646
        "evictedPerRun=" + stats.evictedPerEviction());
647
  }
647
  }
648

    
   
648

   
[+20] [20] 123 lines
src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java
New File
 
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Revision 4efdc6b New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
Revision dcede5a New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java
Revision 553eee0 New Change
 
src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
Revision d2329e1 New Change
 
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
Revision 4bfd42f New Change
 
src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
New File
 
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
Revision 91ac652 New Change
 
  1. src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java: Loading...
  2. src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java: Loading...
  3. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java: Loading...
  4. src/main/java/org/apache/hadoop/hbase/regionserver/Store.java: Loading...
  5. src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java: Loading...
  6. src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java: Loading...
  7. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java: Loading...
  8. src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java: Loading...
  9. src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java: Loading...