Review Board 1.7.22


[HBASE-4465] Lazy-seek optimization for StoreFile scanners

Review Request #2180 - Created Oct. 4, 2011 and updated

Mikhail Bautin
HBASE-4465
Reviewers
hbase
hbase-git
Previously, if we had several StoreFiles for a column family in a region, we would seek in each of them and only then merge the results, even though the row/column we are looking for might only be in the most recent (and the smallest) file. Now we prioritize our reads from those files so that we check the most recent file first. This is done by doing a "lazy seek" which pretends that the next value in the StoreFile is (seekRow, seekColumn, lastTimestampInStoreFile), which is earlier in the KV order than anything that might actually occur in the file. So if we don't find the result in earlier files, that fake KV will bubble up to the top of the KV heap and a real seek will be done. This is expected to significantly reduce the amount of disk IO (as of 09/22/2011 we are doing dark launch testing and measurement).

This is joint work with Liyin Tang – huge thanks to him for many helpful discussions on this and the idea of putting fake KVs with the highest timestamp of the StoreFile in the scanner priority queue.
All unit tests should be passing now. Will rebase and re-run again just in case.
src/main/java/org/apache/hadoop/hbase/KeyValue.java
Revision aa34006 New Change
[20] 1754 lines
[+20] [+] public KeyValue createLastOnRowCol() {
1755
        bytes, getQualifierOffset(), getQualifierLength(),
1755
        bytes, getQualifierOffset(), getQualifierLength(),
1756
        HConstants.OLDEST_TIMESTAMP, Type.Minimum, null, 0, 0);
1756
        HConstants.OLDEST_TIMESTAMP, Type.Minimum, null, 0, 0);
1757
  }
1757
  }
1758

    
   
1758

   
1759
  /**
1759
  /**

    
   
1760
   * Creates the first KV with the row/family/qualifier of this KV and the

    
   
1761
   * given timestamp. Uses the "maximum" KV type that guarantees that the new

    
   
1762
   * KV is the lowest possible for this combination of row, family, qualifier,

    
   
1763
   * and timestamp. This KV's own timestamp is ignored. While this function

    
   
1764
   * copies the value from this KV, it is normally used on key-only KVs.

    
   
1765
   */

    
   
1766
  public KeyValue createFirstOnRowColTS(long ts) {

    
   
1767
    return new KeyValue(

    
   
1768
        bytes, getRowOffset(), getRowLength(),

    
   
1769
        bytes, getFamilyOffset(), getFamilyLength(),

    
   
1770
        bytes, getQualifierOffset(), getQualifierLength(),

    
   
1771
        ts, Type.Maximum, bytes, getValueOffset(), getValueLength());

    
   
1772
  }

    
   
1773

   

    
   
1774
  /**
1760
   * @param b
1775
   * @param b
1761
   * @return A KeyValue made of a byte array that holds the key-only part.
1776
   * @return A KeyValue made of a byte array that holds the key-only part.
1762
   * Needed to convert hfile index members to KeyValues.
1777
   * Needed to convert hfile index members to KeyValues.
1763
   */
1778
   */
1764
  public static KeyValue createKeyValueFromKey(final byte [] b) {
1779
  public static KeyValue createKeyValueFromKey(final byte [] b) {
[+20] [20] 277 lines
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java
Revision 94ddce7 New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/ColumnCount.java
Revision 1be0280 New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
Revision b8d33e8 New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
Revision fbcd276 New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
Revision 035f765 New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java
New File
 
src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
Revision dad278a New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
Revision abb5931 New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
Revision 31bfea7 New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
Revision 64a6e3e New Change
 
src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java
Revision 8ad5aab New Change
 
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
Revision 9d2b2a7 New Change
 
  1. src/main/java/org/apache/hadoop/hbase/KeyValue.java: Loading...
  2. src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java: Loading...
  3. src/main/java/org/apache/hadoop/hbase/regionserver/ColumnCount.java: Loading...
  4. src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java: Loading...
  5. src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java: Loading...
  6. src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java: Loading...
  7. src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java: Loading...
  8. src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java: Loading...
  9. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java: Loading...
  10. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java: Loading...
  11. src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java: Loading...
  12. src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java: Loading...
  13. src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java: Loading...