Review Board 1.7.22


SQOOP-428: Support compression for Avro import

Review Request #3600 - Created Jan. 24, 2012 and updated

Lars Francke
SQOOP-428
Reviewers
Sqoop
sqoop-trunk
This basically only ports all the code from Avro's (1.5.4) AvroOutputFormat to the new MR API.

I've changed the test to extract the common functionality into a helper method because they are the same apart from the two command line arguments.

I could have deleted AvroJob completely but as I was told last time that binary compatibility needs to be maintained I left it in. It's not needed anymore as all necessary functionality can be gotten from Avro's own version of that file as far as I can tell. So if it's okay to delete that redundant file (two actually, cloudera and apache package) let me know and I'll provide a new patch.
All tests pass for hadoopversion=20 but TestColumnTypes fails for me on 23. I can't see how that's related though.
src/java/com/cloudera/sqoop/io/CodecMap.java
Revision ffe949b New Change
[20] 67 lines
[+20] [+] public static CompressionCodec getCodec(String codecName,
68
   * Return the set of available codec names.
68
   * Return the set of available codec names.
69
   */
69
   */
70
  public static Set<String> getCodecNames() {
70
  public static Set<String> getCodecNames() {
71
    return org.apache.sqoop.io.CodecMap.getCodecNames();
71
    return org.apache.sqoop.io.CodecMap.getCodecNames();
72
  }
72
  }

    
   
73

   

    
   
74
  /**

    
   
75
   * Return the short name of the codec.

    
   
76
   * See {@link org.apache.sqoop.io.CodecMap#getCodecShortNameByName(String,

    
   
77
   * Configuration)}.

    
   
78
   */

    
   
79
  public static String getCodecShortNameByName(String codecName,

    
   
80
    Configuration conf) throws UnsupportedCodecException {

    
   
81
    return org.apache.sqoop.io.CodecMap

    
   
82
      .getCodecShortNameByName(codecName, conf);

    
   
83
  }
73
}
84
}
src/java/org/apache/sqoop/io/CodecMap.java
Revision 5b67206 New Change
 
src/java/org/apache/sqoop/mapreduce/AvroJob.java
Revision a57aaf1 New Change
 
src/java/org/apache/sqoop/mapreduce/AvroOutputFormat.java
Revision 96befd7 New Change
 
src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
Revision ed6954a New Change
 
src/test/com/cloudera/sqoop/TestAvroImport.java
Revision 06834fc New Change
 
src/test/com/cloudera/sqoop/io/TestCodecMap.java
Revision f2f4039 New Change
 
  1. src/java/com/cloudera/sqoop/io/CodecMap.java: Loading...
  2. src/java/org/apache/sqoop/io/CodecMap.java: Loading...
  3. src/java/org/apache/sqoop/mapreduce/AvroJob.java: Loading...
  4. src/java/org/apache/sqoop/mapreduce/AvroOutputFormat.java: Loading...
  5. src/java/org/apache/sqoop/mapreduce/ImportJobBase.java: Loading...
  6. src/test/com/cloudera/sqoop/TestAvroImport.java: Loading...
  7. src/test/com/cloudera/sqoop/io/TestCodecMap.java: Loading...