Review Board 1.7.22


HIVE-5095: Hive needs new operator walker for parallelization/optimization for tez

Review Request #13787 - Created Aug. 23, 2013 and updated

Vikram Dixit Kumaraswamy
tez
HIVE-5095
Reviewers
hive
haglein
hive-git
For tez to compute the number of reducers, we should be walking the operator tree in a topological fashion so that the reducers down the tree get the estimate from all parents. However, the current walkers in hive only walk the operator tree in a depth-first fashion. We need to add a new walker for the topological walk. Also, since information about the parent operators needs to be propagated on a per parent basis, we need to retain some context across operators to be passed to the child which the walker will co-ordinate.

 
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Revision 7408a5a New Change
[20] 185 lines
[+20] [+] public class HiveConf extends Configuration {
186
    STREAMREPORTERENABLED("stream.stderr.reporter.enabled", true),
186
    STREAMREPORTERENABLED("stream.stderr.reporter.enabled", true),
187
    COMPRESSRESULT("hive.exec.compress.output", false),
187
    COMPRESSRESULT("hive.exec.compress.output", false),
188
    COMPRESSINTERMEDIATE("hive.exec.compress.intermediate", false),
188
    COMPRESSINTERMEDIATE("hive.exec.compress.intermediate", false),
189
    COMPRESSINTERMEDIATECODEC("hive.intermediate.compression.codec", ""),
189
    COMPRESSINTERMEDIATECODEC("hive.intermediate.compression.codec", ""),
190
    COMPRESSINTERMEDIATETYPE("hive.intermediate.compression.type", ""),
190
    COMPRESSINTERMEDIATETYPE("hive.intermediate.compression.type", ""),
191
    BYTESPERREDUCER("hive.exec.reducers.bytes.per.reducer", (long) (1000 * 1000 * 1000)),
191
    BYTESPERREDUCER("hive.exec.reducers.bytes.per.reducer", (long) (1 * 1 * 1000)),
192
    MAXREDUCERS("hive.exec.reducers.max", 999),
192
    MAXREDUCERS("hive.exec.reducers.max", 999),
193
    PREEXECHOOKS("hive.exec.pre.hooks", ""),
193
    PREEXECHOOKS("hive.exec.pre.hooks", ""),
194
    POSTEXECHOOKS("hive.exec.post.hooks", ""),
194
    POSTEXECHOOKS("hive.exec.post.hooks", ""),
195
    ONFAILUREHOOKS("hive.exec.failure.hooks", ""),
195
    ONFAILUREHOOKS("hive.exec.failure.hooks", ""),
196
    CLIENTSTATSPUBLISHERS("hive.client.stats.publishers", ""),
196
    CLIENTSTATSPUBLISHERS("hive.client.stats.publishers", ""),
[+20] [20] 1058 lines
ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
Revision ca48f5e New Change
 
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
Revision 6a538e8 New Change
 
ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java
Revision 6ee13ec New Change
 
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
Revision f3c34d1 New Change
 
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java
Revision 49a0ee3 New Change
 
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
Revision 7433ddc New Change
 
ql/src/java/org/apache/hadoop/hive/ql/optimizer/OpProcessor.java
New File
 
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/OpProcContext.java
New File
 
ql/src/java/org/apache/hadoop/hive/ql/parse/GenOpGraphWalker.java
New File
 
ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java
Revision 827637a New Change
 
ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java
Revision ff8b17b New Change
 
ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java
Revision c1c1da5 New Change
 
ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java
Revision 248eb03 New Change
 
ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
Revision 5abedfe New Change
 
ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
Revision 5fd8d828 New Change
 
  1. common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: Loading...
  2. ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java: Loading...
  3. ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java: Loading...
  4. ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java: Loading...
  5. ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java: Loading...
  6. ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapRedTask.java: Loading...
  7. ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java: Loading...
  8. ql/src/java/org/apache/hadoop/hive/ql/optimizer/OpProcessor.java: Loading...
  9. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/OpProcContext.java: Loading...
  10. ql/src/java/org/apache/hadoop/hive/ql/parse/GenOpGraphWalker.java: Loading...
  11. ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java: Loading...
  12. ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java: Loading...
  13. ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java: Loading...
  14. ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java: Loading...
  15. ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java: Loading...
  16. ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java: Loading...