Review Board 1.7.22


SQOOP-314 Basic export hangs when target database does not support INSERT syntax with multiple rows of values

Review Request #1585 - Created Aug. 18, 2011 and submitted

Bilung Lee
SQOOP-314
Reviewers
Sqoop
aprabhakar, jmhsieh
sqoop-trunk
Basic export job will hang when the target database does not support insert syntax with multiple rows of values, such as INSERT INTO tbl (col1, col2) VALUES(11, 12),(21, 22),(23, 24)

This is because, in close(), AsyncSqlRecordWriter will still wait for AsyncSqlExecThread to finish even when an SQLException is thrown underneath.

The fix is to introduce a new "--batch" option to execute underlying statements in batch mode instead of a single multirow insert statement.

 
src/docs/man/sqoop-export.txt
Revision 6090ea1 New Change
[20] 52 lines
[+20]
53
  The table in which data will be staged before being inserted in destination table.
53
  The table in which data will be staged before being inserted in destination table.
54

    
   
54

   
55
--clear-staging-table::
55
--clear-staging-table::
56
  Will result in deletion of any data that exists in the staging table.
56
  Will result in deletion of any data that exists in the staging table.
57

    
   
57

   

    
   
58
--batch::

    
   
59
  Use batch mode for underlying statement execution. This is useful, for example,

    
   
60
  for those databases that do not support multirow insert in a single statement yet.

    
   
61

   
58
include::input-args.txt[]
62
include::input-args.txt[]
59

    
   
63

   
60
include::output-args.txt[]
64
include::output-args.txt[]
61

    
   
65

   
62
include::codegen-args.txt[]
66
include::codegen-args.txt[]
[+20] [20] 37 lines
src/docs/user/export.txt
Revision 4f87886 New Change
 
src/java/com/cloudera/sqoop/SqoopOptions.java
Revision d760d39 New Change
 
src/java/com/cloudera/sqoop/manager/OracleManager.java
Revision 6a55312 New Change
 
src/java/com/cloudera/sqoop/manager/SQLServerManager.java
Revision e1ce2af New Change
 
src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java
Revision 193cf41 New Change
 
src/java/com/cloudera/sqoop/mapreduce/ExportBatchOutputFormat.java
New File
 
src/java/com/cloudera/sqoop/mapreduce/ExportJobBase.java
Revision 9799e37 New Change
 
src/java/com/cloudera/sqoop/mapreduce/ExportOutputFormat.java
Revision d2a6cf6 New Change
 
src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java
Revision 8f629f1 New Change
 
src/java/com/cloudera/sqoop/tool/ExportTool.java
Revision b4b1091 New Change
 
src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java
Revision 8687b0c New Change
 
  1. src/docs/man/sqoop-export.txt: Loading...
  2. src/docs/user/export.txt: Loading...
  3. src/java/com/cloudera/sqoop/SqoopOptions.java: Loading...
  4. src/java/com/cloudera/sqoop/manager/OracleManager.java: Loading...
  5. src/java/com/cloudera/sqoop/manager/SQLServerManager.java: Loading...
  6. src/java/com/cloudera/sqoop/mapreduce/AsyncSqlRecordWriter.java: Loading...
  7. src/java/com/cloudera/sqoop/mapreduce/ExportBatchOutputFormat.java: Loading...
  8. src/java/com/cloudera/sqoop/mapreduce/ExportJobBase.java: Loading...
  9. src/java/com/cloudera/sqoop/mapreduce/ExportOutputFormat.java: Loading...
  10. src/java/com/cloudera/sqoop/tool/BaseSqoopTool.java: Loading...
  11. src/java/com/cloudera/sqoop/tool/ExportTool.java: Loading...
  12. src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java: Loading...