Review Board 1.7.22


SQOOP-1071 Avro import is failing when columns are starting with underscore

Review Request #11748 - Created June 8, 2013 and submitted

Jarek Cecho
SQOOP-1071
Reviewers
Sqoop
sqoop-trunk
The issue is hiding in fact that ClassWriter will use toJavaIdentifier() method to generate attribute name, however AvroSchemaGenerator is using only toIdentifier(). I've changed the AvroSchemaGenerator to generate the proper schema in the same way as is doing the ClassWriter. This is backward incompatible change, however considering that Sqoop was not able to import anything without this, I would assume that consequences will be minimal if there will be any.
I've added test that will fail without the fix and refactored a bit BaseSqoopTestCase to allow override column names. All tests seems to be passing.
src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java
Revision 864fa0d83c981043e3ff75854d8575ab13e00296 New Change
[20] 57 lines
[+20] [+] public Schema generate() throws IOException {
58
    Map<String, Integer> columnTypes = classWriter.getColumnTypes();
58
    Map<String, Integer> columnTypes = classWriter.getColumnTypes();
59
    String[] columnNames = classWriter.getColumnNames(columnTypes);
59
    String[] columnNames = classWriter.getColumnNames(columnTypes);
60

    
   
60

   
61
    List<Field> fields = new ArrayList<Field>();
61
    List<Field> fields = new ArrayList<Field>();
62
    for (String columnName : columnNames) {
62
    for (String columnName : columnNames) {
63
      String cleanedCol = ClassWriter.toIdentifier(columnName);
63
      String cleanedCol = ClassWriter.toJavaIdentifier(columnName);
64
      int sqlType = columnTypes.get(columnName);
64
      int sqlType = columnTypes.get(columnName);
65
      Schema avroSchema = toAvroSchema(sqlType, columnName);
65
      Schema avroSchema = toAvroSchema(sqlType, columnName);
66
      Field field = new Field(cleanedCol, avroSchema, null, null);
66
      Field field = new Field(cleanedCol, avroSchema, null, null);
67
      field.addProp("columnName", columnName);
67
      field.addProp("columnName", columnName);
68
      field.addProp("sqlType", Integer.toString(sqlType));
68
      field.addProp("sqlType", Integer.toString(sqlType));
[+20] [20] 58 lines
src/test/com/cloudera/sqoop/TestAvroImport.java
Revision 32d1bbea7459ce6f28253d74571e786749030cf0 New Change
 
src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java
Revision cf41b9691fa3c18c7caac99f27ebf327687a0649 New Change
 
  1. src/java/org/apache/sqoop/orm/AvroSchemaGenerator.java: Loading...
  2. src/test/com/cloudera/sqoop/TestAvroImport.java: Loading...
  3. src/test/com/cloudera/sqoop/testutil/BaseSqoopTestCase.java: Loading...