Review Board 1.7.22


HIVE-4032:Inserting data into Hive table from a query, when the query is a partitioned table and select * ,will generate a SemanticException

Review Request #10234 - Created April 2, 2013 and updated

fangkun cao
hive-trunk
HIVE-4032
Reviewers
hive
hive
See: https://issues.apache.org/jira/browse/HIVE-4032
Before fix :
hive (default)> explain                                                                                         
              > insert into table zr_test partition (dt='20130217') select * from zr_test_1 where dt='20130217';
FAILED: SemanticException [Error 10044]: Line 2:18 Cannot insert into target table because column number/types are different ''20130217'': Table insclause-0 has 2 columns, but query has 3 columns.

After fix:
hive (default)> explain
              > insert into table zr_test partition (dt='20130217') select * from zr_test_1 where dt='20130217';
[2013-04-02 12:33:12]: Compile Start 
[2013-04-02 12:33:15]: Compile End
[2013-04-02 12:33:15]: OK
ABSTRACT SYNTAX TREE:
  (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME zr_test_1))) (TOK_INSERT (TOK_INSERT_INTO (TOK_TAB (TOK_TABNAME zr_test) (TOK_PARTSPEC (TOK_PARTVAL dt '20130217')))) (TOK_SELECT (TOK_SELEXPR TOK_ALLCOLREF)) (TOK_WHERE (= (TOK_TABLE_OR_COL dt) '20130217'))))

STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-5 depends on stages: Stage-1 , consists of Stage-4, Stage-3
  Stage-4
  Stage-0 depends on stages: Stage-4, Stage-3
  Stage-2 depends on stages: Stage-0
  Stage-3

STAGE PLANS:
  Stage: Stage-1
    Map Reduce
      Alias -> Map Operator Tree:
        zr_test_1 
          TableScan
            alias: zr_test_1
            Filter Operator
              predicate:
                  expr: (dt = '20130217')
                  type: boolean
              Select Operator
                expressions:
                      expr: key
                      type: string
                      expr: value
                      type: string
                      expr: dt
                      type: string
                outputColumnNames: _col0, _col1, _col2
                File Output Operator
                  compressed: false
                  GlobalTableId: 1
                  table:
                      input format: org.apache.hadoop.mapred.TextInputFormat
                      output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                      serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                      name: default.zr_test

  Stage: Stage-5
    Conditional Operator

  Stage: Stage-4
    Move Operator
      files:
          hdfs directory: true
          destination: hdfs://hd17-vm5:9101/user/zongren/hive-scratchdir/hive_2013-04-02_12-33-12_425_5386104740729125/-ext-10000

  Stage: Stage-0
    Move Operator
      tables:
          partition:
            dt 20130217
          replace: false
          table:
              input format: org.apache.hadoop.mapred.TextInputFormat
              output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
              name: default.zr_test

  Stage: Stage-2
    Stats-Aggr Operator

  Stage: Stage-3
    Map Reduce
      Alias -> Map Operator Tree:
        hdfs://hd17-vm5:9101/user/zongren/hive-scratchdir/hive_2013-04-02_12-33-12_425_5386104740729125/-ext-10002 
            File Output Operator
              compressed: false
              GlobalTableId: 0
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                  name: default.zr_test


[2013-04-02 12:33:15]: Time taken: 3.001 seconds
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
Revision 1463367 New Change
[20] 5201 lines
[+20] [+] private Operator genFileSinkPlan(String dest, QB qb, Operator input)
5202
    // Check column number
5202
    // Check column number
5203
    List<? extends StructField> tableFields = oi.getAllStructFieldRefs();
5203
    List<? extends StructField> tableFields = oi.getAllStructFieldRefs();
5204
    boolean dynPart = HiveConf.getBoolVar(conf, HiveConf.ConfVars.DYNAMICPARTITIONING);
5204
    boolean dynPart = HiveConf.getBoolVar(conf, HiveConf.ConfVars.DYNAMICPARTITIONING);
5205
    ArrayList<ColumnInfo> rowFields = opParseCtx.get(input).getRowResolver()
5205
    ArrayList<ColumnInfo> rowFields = opParseCtx.get(input).getRowResolver()
5206
        .getColumnInfos();
5206
        .getColumnInfos();
5207
    int inColumnCnt = rowFields.size();
5207
    int inColumnCnt = 0;

    
   
5208
    for (ColumnInfo rowField : rowFields) {

    
   
5209
      if (!rowField.getIsVirtualCol()) {

    
   
5210
        inColumnCnt += 1;

    
   
5211
      }

    
   
5212
    }
5208
    int outColumnCnt = tableFields.size();
5213
    int outColumnCnt = tableFields.size();
5209
    if (dynPart && dpCtx != null) {
5214
    if (dynPart && dpCtx != null) {
5210
      outColumnCnt += dpCtx.getNumDPCols();
5215
      outColumnCnt += dpCtx.getNumDPCols();
5211
    }
5216
    }
5212

    
   
5217

   
[+20] [20] 4463 lines
  1. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java: Loading...