Review Board 1.7.22


SQOOP-846 Provide Netezza connector

Review Request #9543 - Created Feb. 21, 2013 and updated

Venkat Ranganathan
SQOOP-846
Reviewers
Sqoop
jarcec
sqoop-trunk
This addresses SQOOP-846 (provide a Netezza connector)
Ran all sqoop tests.   Ran Netezza manual tests against Netezza VMs version 6 and 7

Diff revision 8

This is not the most recent revision of the diff. The latest diff is revision 9. See what's changed.

1 2 3 4 5 6 7 8 9
1 2 3 4 5 6 7 8 9

  1. src/docs/user/connectors.txt: Loading...
  2. src/java/org/apache/sqoop/lib/DelimiterSet.java: Loading...
  3. src/java/org/apache/sqoop/manager/DefaultManagerFactory.java: Loading...
  4. src/java/org/apache/sqoop/manager/DirectNetezzaManager.java: Loading...
  5. src/java/org/apache/sqoop/manager/MySQLUtils.java: Loading...
  6. src/java/org/apache/sqoop/manager/NetezzaManager.java: Loading...
  7. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaDBDataSliceSplitter.java: Loading...
  8. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java: Loading...
  9. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableImportMapper.java: Loading...
  10. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableRecordExportMapper.java: Loading...
  11. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableTextExportMapper.java: Loading...
  12. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java: Loading...
  13. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaDataDrivenDBInputFormat.java: Loading...
  14. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableExportJob.java: Loading...
  15. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableImportJob.java: Loading...
  16. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputFormat.java: Loading...
  17. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputSplit.java: Loading...
  18. src/test/com/cloudera/sqoop/manager/DirectNetezzaExportManualTest.java: Loading...
  19. src/test/com/cloudera/sqoop/manager/NetezzaImportManualTest.java: Loading...
  20. src/test/com/cloudera/sqoop/manager/NetezzaTestUtils.java: Loading...
src/docs/user/connectors.txt
Revision 7dd2a2e New Change
[20] 240 lines
[+20]
241
Staging tables are automatically dropped if tasks successfully complete
241
Staging tables are automatically dropped if tasks successfully complete
242
or map tasks fail.
242
or map tasks fail.
243
When reduce task fails,
243
When reduce task fails,
244
staging table for the task are left for manual retry and
244
staging table for the task are left for manual retry and
245
users must take care of it.
245
users must take care of it.

    
   
246
----

    
   
247

   

    
   
248

   

    
   
249
Netezza Connector

    
   
250
~~~~~~~~~~~~~~~~~

    
   
251

   

    
   
252
Extra arguments

    
   
253
^^^^^^^^^^^^^^^

    
   
254

   

    
   
255
List of all extra arguments supported by Netezza Connector is shown below:

    
   
256

   

    
   
257
.Supported PostgreSQL extra arguments:

    
   
258
[grid="all"]

    
   
259
`----------------------------------------`---------------------------------------

    
   
260
Argument                                 Description

    
   
261
---------------------------------------------------------------------------------

    
   
262
+partitioned-access+                  Whether each mapper acts on a subset\

    
   
263
                                      of data slices of a table or all\

    
   
264
                                      Default is "false" for standard mode\

    
   
265
                                      and "true" for direct mode.

    
   
266
+max-errors+                          Applicable only in direct mode.\

    
   
267
                                      This option specifies the error threshold\

    
   
268
                                      per mapper while transferring data. If\

    
   
269
                                      the number of errors encountered exceed\

    
   
270
                                      this threshold then the job will fail.

    
   
271
                                      Default value is 1.

    
   
272
+log-dir+                             Applicable only in direct mode.\

    
   
273
                                      Specifies the directory where Netezza\

    
   
274
                                      external table operation logs are stored.\

    
   
275
                                      Default value is /tmp.

    
   
276
---------------------------------------------------------------------------------

    
   
277

   

    
   
278
----

    
   
279

   

    
   
280
Direct Mode

    
   
281
^^^^^^^^^^^^

    
   
282
Netezza connector supports an optimized data transfer facility using the

    
   
283
Netezza external tables feature.  Each map tasks of Netezza connector's import

    
   
284
job will work on a subset of the Netezza partitions and transparently create

    
   
285
and use an external table to transport data.  Similarly, export jobs will use

    
   
286
the external table to push data fast onto the NZ system.   Direct mode does

    
   
287
not support staging tables, upsert options etc. 

    
   
288
.

    
   
289
----

    
   
290

   

    
   
291
Here is an example of complete command line for import using the Netezza

    
   
292
external table feature

    
   
293

   

    
   
294
sqoop import --direct --connect jdbc:netezza://nzhost:5480/sqoop \

    
   
295
  --table <nztable> --username <nzuser> --password <nzpass> \

    
   
296
  --target-dir <export-dir>

    
   
297
  

    
   
298
Here is an example of complete command line for export with <tab> as the

    
   
299
delimiter

    
   
300

   

    
   
301
sqoop export --direct --connect jdbc:netezza://nzhost:5480/sqoop \

    
   
302
  --table <nztable> --username <nzuser> --password <nzpass> \

    
   
303
  --export-dir <export-dir> --input-fields-terminated-by "\t"
src/java/org/apache/sqoop/lib/DelimiterSet.java
Revision 4e9bcab New Change
 
src/java/org/apache/sqoop/manager/DefaultManagerFactory.java
Revision 54eb258 New Change
 
src/java/org/apache/sqoop/manager/DirectNetezzaManager.java
New File
 
src/java/org/apache/sqoop/manager/MySQLUtils.java
Revision ef18818 New Change
 
src/java/org/apache/sqoop/manager/NetezzaManager.java
New File
 
src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaDBDataSliceSplitter.java
New File
 
src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java
New File
 
src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableImportMapper.java
New File
 
src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableRecordExportMapper.java
New File
 
src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableTextExportMapper.java
New File
 
src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java
New File
 
src/java/org/apache/sqoop/mapreduce/netezza/NetezzaDataDrivenDBInputFormat.java
New File
 
src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableExportJob.java
New File
 
src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableImportJob.java
New File
 
src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputFormat.java
New File
 
src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputSplit.java
New File
 
src/test/com/cloudera/sqoop/manager/DirectNetezzaExportManualTest.java
New File
 
src/test/com/cloudera/sqoop/manager/NetezzaImportManualTest.java
New File
 
src/test/com/cloudera/sqoop/manager/NetezzaTestUtils.java
New File
 
  1. src/docs/user/connectors.txt: Loading...
  2. src/java/org/apache/sqoop/lib/DelimiterSet.java: Loading...
  3. src/java/org/apache/sqoop/manager/DefaultManagerFactory.java: Loading...
  4. src/java/org/apache/sqoop/manager/DirectNetezzaManager.java: Loading...
  5. src/java/org/apache/sqoop/manager/MySQLUtils.java: Loading...
  6. src/java/org/apache/sqoop/manager/NetezzaManager.java: Loading...
  7. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaDBDataSliceSplitter.java: Loading...
  8. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java: Loading...
  9. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableImportMapper.java: Loading...
  10. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableRecordExportMapper.java: Loading...
  11. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableTextExportMapper.java: Loading...
  12. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java: Loading...
  13. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaDataDrivenDBInputFormat.java: Loading...
  14. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableExportJob.java: Loading...
  15. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableImportJob.java: Loading...
  16. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputFormat.java: Loading...
  17. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputSplit.java: Loading...
  18. src/test/com/cloudera/sqoop/manager/DirectNetezzaExportManualTest.java: Loading...
  19. src/test/com/cloudera/sqoop/manager/NetezzaImportManualTest.java: Loading...
  20. src/test/com/cloudera/sqoop/manager/NetezzaTestUtils.java: Loading...