Review Board 1.7.22


HIVE-2036: Update bitmap indexes for automatic usage

Review Request #857 - Created June 6, 2011 and updated

Syed Albiz
trunk
HIVE-2036
Reviewers
hive
jvs
hive-git
Add support for generating index queries to support automatic usage of bitmap indexes. This required changing the interface to the IndexHandlers to support accepting queries on multiple indexes. The compact indexes were modified to use this new interface as well, although no functional changes were made to how they work. Only supports AND predicates right now, but it should be possibly to extend the BitmapQuery interface defined in this patch to easily support OR predicates as well. Currently benchmarking these changes on a test cluster.
Passes unit tests, additional testcase to test automatic bitmap indexing index_bitmap_auto.q was also added to the TestCliDriver suite. Currently benchmarking changes on a test cluster.
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
Revision 4fba845 New Change
[20] 476 lines
[+20] [+] public int execute(DriverContext driverContext) {
477
    if (work.getInputformat() != null) {
477
    if (work.getInputformat() != null) {
478
      HiveConf.setVar(conf, HiveConf.ConfVars.HIVEINPUTFORMAT, work.getInputformat());
478
      HiveConf.setVar(conf, HiveConf.ConfVars.HIVEINPUTFORMAT, work.getInputformat());
479
    }
479
    }
480
    if (work.getIndexIntermediateFile() != null) {
480
    if (work.getIndexIntermediateFile() != null) {
481
      conf.set("hive.index.compact.file", work.getIndexIntermediateFile());
481
      conf.set("hive.index.compact.file", work.getIndexIntermediateFile());

    
   
482
      conf.set("hive.index.blockfilter.file", work.getIndexIntermediateFile());
482
    }
483
    }
483
  }
484
  }
484

    
   
485

   
485
  public boolean mapStarted() {
486
  public boolean mapStarted() {
486
    return this.jobExecHelper.mapStarted();
487
    return this.jobExecHelper.mapStarted();
[+20] [20] 453 lines
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java
Revision e5ee183 New Change
 
ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java
Revision af9d7b1 New Change
 
ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapInnerQuery.java
New File
 
ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapOuterQuery.java
New File
 
ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapQuery.java
New File
 
ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/HiveBitmapIndexInputFormat.java
New File
 
ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
Revision 56e7609 New Change
 
ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
Revision d64e88b New Change
 
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
Revision 268560d New Change
 
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java
Revision 0873e1a New Change
 
ql/src/test/queries/clientpositive/index_bitmap3.q
Revision 508eb94 New Change
 
ql/src/test/queries/clientpositive/index_bitmap_auto.q
New File
 
ql/src/test/results/clientpositive/index_bitmap_auto.q.out
New File
 
  1. ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java: Loading...
  2. ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java: Loading...
  3. ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java: Loading...
  4. ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapInnerQuery.java: Loading...
  5. ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapOuterQuery.java: Loading...
  6. ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapQuery.java: Loading...
  7. ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/HiveBitmapIndexInputFormat.java: Loading...
  8. ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java: Loading...
  9. ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java: Loading...
  10. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java: Loading...
  11. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java: Loading...
  12. ql/src/test/queries/clientpositive/index_bitmap3.q: Loading...
  13. ql/src/test/queries/clientpositive/index_bitmap_auto.q: Loading...
  14. ql/src/test/results/clientpositive/index_bitmap_auto.q.out: Loading...