Review Board 1.7.22


HIVE-3528

Review Request #7431 - Created Oct. 3, 2012 and submitted

Sean Busbey
Reviewers
hive
jghoman
hive
Changes AvroSerDe to properly give the non-null schema to serialization routines when using Nullable complex types. Properly restores the enum-ness of Avro Enums prior to serialization.
Adds tests that check each of the Avro types that Serialization needs to use a user-provided schema, both as top level fields and as nested members of a complex type.

Adds a client positive test that reads in a CSV table with NULLs, copies that data into an Avro backed table, then reads the data out of the table.
/trunk/data/files/csv.txt
New File

    
   
1
why hello there,42,3,100,1412341,true,42.43,85.23423424,alpha:beta:gamma,Earth#42:Control#86:Bob#31,17:true:Abe Linkedin,BLUE,72,0:1:2:3:4:5,50:51:53

    
   
2
another record,98,4,101,9999999,false,99.89,0.00000009,beta,Earth#101,1134:false:wazzup,RED,\N,6:7:8:9:10,54:55:56

    
   
3
third record,45,5,102,999999999,true,89.99,0.00000000000009,alpha:gamma,Earth#237:Bob#723,102:false:BNL,GREEN,\N,11:12:13,57:58:59

    
   
4
\N,42,3,100,1412341,true,42.43,85.23423424,alpha:beta:gamma,Earth#42:Control#86:Bob#31,17:true:Abe Linkedin,BLUE,72,0:1:2:3:4:5,50:51:53

    
   
5
string,\N,3,100,1412341,true,42.43,85.23423424,alpha:beta:gamma,Earth#42:Control#86:Bob#31,17:true:Abe Linkedin,BLUE,72,0:1:2:3:4:5,50:51:53

    
   
6
string,42,\N,100,1412341,true,42.43,85.23423424,alpha:beta:gamma,Earth#42:Control#86:Bob#31,17:true:Abe Linkedin,BLUE,72,0:1:2:3:4:5,50:51:53

    
   
7
string,42,3,\N,1412341,true,42.43,85.23423424,alpha:beta:gamma,Earth#42:Control#86:Bob#31,17:true:Abe Linkedin,BLUE,72,0:1:2:3:4:5,50:51:53

    
   
8
string,42,3,100,\N,true,42.43,85.23423424,alpha:beta:gamma,Earth#42:Control#86:Bob#31,17:true:Abe Linkedin,BLUE,72,0:1:2:3:4:5,50:51:53

    
   
9
string,42,3,100,1412341,\N,42.43,85.23423424,alpha:beta:gamma,Earth#42:Control#86:Bob#31,17:true:Abe Linkedin,BLUE,72,0:1:2:3:4:5,50:51:53

    
   
10
string,42,3,100,1412341,true,\N,85.23423424,alpha:beta:gamma,Earth#42:Control#86:Bob#31,17:true:Abe Linkedin,BLUE,72,0:1:2:3:4:5,50:51:53

    
   
11
string,42,3,100,1412341,true,42.43,\N,alpha:beta:gamma,Earth#42:Control#86:Bob#31,17:true:Abe Linkedin,BLUE,72,0:1:2:3:4:5,50:51:53

    
   
12
string,42,3,100,1412341,true,42.43,85.23423424,\N,Earth#42:Control#86:Bob#31,17:true:Abe Linkedin,BLUE,72,0:1:2:3:4:5,50:51:53

    
   
13
string,42,3,100,1412341,true,42.43,85.23423424,alpha:beta:gamma,\N,17:true:Abe Linkedin,BLUE,72,0:1:2:3:4:5,50:51:53

    
   
14
string,42,3,100,1412341,true,42.43,85.23423424,alpha:beta:gamma,Earth#42:Control#86:Bob#31,\N,BLUE,72,0:1:2:3:4:5,50:51:53

    
   
15
string,42,3,100,1412341,true,42.43,85.23423424,alpha:beta:gamma,Earth#42:Control#86:Bob#31,17:true:Abe Linkedin,\N,72,0:1:2:3:4:5,50:51:53

    
   
16
string,42,3,100,1412341,true,42.43,85.23423424,alpha:beta:gamma,Earth#42:Control#86:Bob#31,17:true:Abe Linkedin,BLUE,\N,0:1:2:3:4:5,50:51:53

    
   
17
string,42,3,100,1412341,true,42.43,85.23423424,alpha:beta:gamma,Earth#42:Control#86:Bob#31,17:true:Abe Linkedin,BLUE,72,\N,50:51:53

    
   
18
string,42,3,100,1412341,true,42.43,85.23423424,alpha:beta:gamma,Earth#42:Control#86:Bob#31,17:true:Abe Linkedin,BLUE,72,0:1:2:3:4:5,\N
/trunk/ql/src/test/queries/clientpositive/avro_nullable_fields.q
New File
 
/trunk/ql/src/test/results/clientpositive/avro_nullable_fields.q.out
New File
 
/trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java
Revision 1426606 New Change
 
/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroDeserializer.java
Revision 1426606 New Change
 
/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroObjectInspectorGenerator.java
Revision 1426606 New Change
 
/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerializer.java
Revision 1426606 New Change
 
  1. /trunk/data/files/csv.txt: Loading...
  2. /trunk/ql/src/test/queries/clientpositive/avro_nullable_fields.q: Loading...
  3. /trunk/ql/src/test/results/clientpositive/avro_nullable_fields.q.out: Loading...
  4. /trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java: Loading...
  5. /trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroDeserializer.java: Loading...
  6. /trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroObjectInspectorGenerator.java: Loading...
  7. /trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerializer.java: Loading...