Review Board 1.7.22


Hive-3159 Update AvroSerde to determine schema of new tables

Review Request #11925 - Created June 18, 2013 and updated

Mohammad Islam
trunk
HIVE-3159
Reviewers
hive
ashutoshc, jghoman
hive-git
Problem:
Hive doesn't support to create a Avro-based table using HQL create table command. It currently requires to specify Avro schema literal or schema file name.
For multiple cases, it is very inconvenient for user.
Some of the un-supported use cases:
1. Create table ... <Avro-SERDE etc.> as SELECT ... from <NON-AVRO FILE>
2. Create table ... <Avro-SERDE etc.> as SELECT from <AVRO TABLE>
3. Create  table  without specifying Avro schema.

Wrote a new java Test class for a new Java class. Added a new test case into existing java test class. In addition, there are 4 .q file for testing multiple use-cases.
ql/src/test/queries/clientpositive/avro_create_as_select.q
New File

    
   
1
-- verify 1) can create table using create table as select from non-avro table 2) LOAD avro data into new table and read data from the new table

    
   
2
CREATE TABLE doctors(number int, first_name STRING, last_name STRING) STORED AS TEXTFILE;

    
   
3
DESCRIBE doctors;

    
   
4

   

    
   
5
CREATE TABLE copy_doctors ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' as SELECT * FROM doctors;

    
   
6
DESCRIBE copy_doctors;

    
   
7

   

    
   
8
LOAD DATA LOCAL INPATH '../data/files/doctors.avro' INTO TABLE copy_doctors;

    
   
9

   

    
   
10
SELECT * FROM copy_doctors ORDER BY number;

    
   
11

   

    
   
12

   

    
   
13

   
ql/src/test/queries/clientpositive/avro_create_as_select2.q
New File
 
ql/src/test/queries/clientpositive/avro_no_schema_test.q
New File
 
ql/src/test/queries/clientpositive/avro_without_schema.q
New File
 
ql/src/test/results/clientpositive/avro_create_as_select.q.out
New File
 
ql/src/test/results/clientpositive/avro_create_as_select2.q.out
New File
 
ql/src/test/results/clientpositive/avro_no_schema_test.q.out
New File
 
ql/src/test/results/clientpositive/avro_without_schema.q.out
New File
 
serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java
Revision 13848b6 New Change
 
serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java
New File
 
serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerdeUtils.java
Revision 010f614 New Change
 
serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java
New File
 
  1. ql/src/test/queries/clientpositive/avro_create_as_select.q: Loading...
  2. ql/src/test/queries/clientpositive/avro_create_as_select2.q: Loading...
  3. ql/src/test/queries/clientpositive/avro_no_schema_test.q: Loading...
  4. ql/src/test/queries/clientpositive/avro_without_schema.q: Loading...
  5. ql/src/test/results/clientpositive/avro_create_as_select.q.out: Loading...
  6. ql/src/test/results/clientpositive/avro_create_as_select2.q.out: Loading...
  7. ql/src/test/results/clientpositive/avro_no_schema_test.q.out: Loading...
  8. ql/src/test/results/clientpositive/avro_without_schema.q.out: Loading...
  9. serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java: Loading...
  10. serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java: Loading...
  11. serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerdeUtils.java: Loading...
  12. serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java: Loading...