Review Board 1.7.22


SQOOP-788 Sqoop2: Import sometimes duplicate some data

Review Request #8677 - Created Dec. 18, 2012 and submitted

Jarek Cecho
SQOOP-788
Reviewers
Sqoop
sqoop-sqoop2
I've submitted patch with very simple but not optimal solution. I'm recreating the object each time so that it can't happen that we're changing object that is being referenced from somewhere else. This is just temporary solution that will be further improved in SQOOP-791.
Unit tests are passing and I've also tested it on real cluster.
connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportExtractor.java
Revision b856ce6040572125bcec926de0db37ab679d7463 New Change
[20] 56 lines
[+20] [+] public void run(ImmutableContext context, Object connectionC, Object jobC, Partition partition, DataWriter writer) {
57
    ResultSet resultSet = executor.executeQuery(query);
57
    ResultSet resultSet = executor.executeQuery(query);
58

    
   
58

   
59
    try {
59
    try {
60
      ResultSetMetaData metaData = resultSet.getMetaData();
60
      ResultSetMetaData metaData = resultSet.getMetaData();
61
      int column = metaData.getColumnCount();
61
      int column = metaData.getColumnCount();
62
      Object[] array = new Object[column];

   
63
      while (resultSet.next()) {
62
      while (resultSet.next()) {

    
   
63
        Object[] array = new Object[column];
64
        for (int i = 0; i< column; i++) {
64
        for (int i = 0; i< column; i++) {
65
          array[i] = resultSet.getObject(i+1);
65
          array[i] = resultSet.getObject(i+1);
66
        }
66
        }
67
        writer.writeArrayRecord(array);
67
        writer.writeArrayRecord(array);
68
        rowsRead++;
68
        rowsRead++;
[+20] [20] 16 lines
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceExportExtractor.java
Revision 16afcdb6acac6d50bcc04a029fbf487a1b902204 New Change
 
execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextExportExtractor.java
Revision 805514065e47401ee947f9a4abb0606a4a5d70e3 New Change
 
  1. connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcImportExtractor.java: Loading...
  2. execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceExportExtractor.java: Loading...
  3. execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextExportExtractor.java: Loading...