Review Board 1.7.22


SQOOP-846 Provide Netezza connector

Review Request #9543 - Created Feb. 21, 2013 and updated

Venkat Ranganathan
SQOOP-846
Reviewers
Sqoop
jarcec
sqoop-trunk
This addresses SQOOP-846 (provide a Netezza connector)
Ran all sqoop tests.   Ran Netezza manual tests against Netezza VMs version 6 and 7

Diff revision 9 (Latest)

1 2 3 4 5 6 7 8 9
1 2 3 4 5 6 7 8 9

  1. src/docs/user/connectors.txt: Loading...
  2. src/java/org/apache/sqoop/lib/DelimiterSet.java: Loading...
  3. src/java/org/apache/sqoop/manager/DefaultManagerFactory.java: Loading...
  4. src/java/org/apache/sqoop/manager/DirectNetezzaManager.java: Loading...
  5. src/java/org/apache/sqoop/manager/MySQLUtils.java: Loading...
  6. src/java/org/apache/sqoop/manager/NetezzaManager.java: Loading...
  7. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaDBDataSliceSplitter.java: Loading...
  8. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java: Loading...
  9. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableImportMapper.java: Loading...
  10. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableRecordExportMapper.java: Loading...
  11. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableTextExportMapper.java: Loading...
  12. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java: Loading...
  13. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaDataDrivenDBInputFormat.java: Loading...
  14. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableExportJob.java: Loading...
  15. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableImportJob.java: Loading...
  16. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputFormat.java: Loading...
  17. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputSplit.java: Loading...
  18. src/test/com/cloudera/sqoop/manager/DirectNetezzaExportManualTest.java: Loading...
  19. src/test/com/cloudera/sqoop/manager/NetezzaImportManualTest.java: Loading...
  20. src/test/com/cloudera/sqoop/manager/NetezzaTestUtils.java: Loading...
src/docs/user/connectors.txt
Revision 7dd2a2e New Change
[20] 240 lines
[+20]
241
Staging tables are automatically dropped if tasks successfully complete
241
Staging tables are automatically dropped if tasks successfully complete
242
or map tasks fail.
242
or map tasks fail.
243
When reduce task fails,
243
When reduce task fails,
244
staging table for the task are left for manual retry and
244
staging table for the task are left for manual retry and
245
users must take care of it.
245
users must take care of it.

    
   
246

   

    
   
247
Netezza Connector

    
   
248
~~~~~~~~~~~~~~~~~

    
   
249

   

    
   
250
Extra arguments

    
   
251
^^^^^^^^^^^^^^^

    
   
252

   

    
   
253
List of all extra arguments supported by Netezza Connector is shown below:

    
   
254

   

    
   
255
.Supported Netezza extra arguments:

    
   
256
[grid="all"]

    
   
257
`-------------------------------------`----------------------------------------

    
   
258
Argument                              Description

    
   
259
-------------------------------------------------------------------------------

    
   
260
+--partitioned-access+                Whether each mapper acts on a subset\

    
   
261
                                      of data slices of a table or all\

    
   
262
                                      Default is "false" for standard mode\

    
   
263
                                      and "true" for direct mode.

    
   
264
+--max-errors+                        Applicable only in direct mode.\

    
   
265
                                      This option specifies the error threshold\

    
   
266
                                      per mapper while transferring data. If\

    
   
267
                                      the number of errors encountered exceed\

    
   
268
                                      this threshold then the job will fail.

    
   
269
                                      Default value is 1.

    
   
270
+--log-dir+                           Applicable only in direct mode.\

    
   
271
                                      Specifies the directory where Netezza\

    
   
272
                                      external table operation logs are stored.\

    
   
273
                                      Default value is /tmp.

    
   
274
--------------------------------------------------------------------------------

    
   
275

   

    
   
276

   

    
   
277
Direct Mode

    
   
278
^^^^^^^^^^^

    
   
279
Netezza connector supports an optimized data transfer facility using the

    
   
280
Netezza external tables feature.  Each map tasks of Netezza connector's import

    
   
281
job will work on a subset of the Netezza partitions and transparently create

    
   
282
and use an external table to transport data.  Similarly, export jobs will use

    
   
283
the external table to push data fast onto the NZ system.   Direct mode does

    
   
284
not support staging tables, upsert options etc.

    
   
285

   

    
   
286
Here is an example of complete command line for import using the Netezza

    
   
287
external table feature.

    
   
288

   

    
   
289
----

    
   
290
$ sqoop import \

    
   
291
    --direct \

    
   
292
    --connect jdbc:netezza://nzhost:5480/sqoop \

    
   
293
    --table nztable \

    
   
294
    --username nzuser \

    
   
295
    --password nzpass \

    
   
296
    --target-dir hdfsdir

    
   
297

   

    
   
298
----

    
   
299

   

    
   
300
Here is an example of complete command line for export with tab as the field

    
   
301
terminator character.

    
   
302

   

    
   
303
----

    
   
304
$ sqoop export \

    
   
305
    --direct \

    
   
306
    --connect jdbc:netezza://nzhost:5480/sqoop \

    
   
307
    --table nztable \

    
   
308
    --username nzuser \

    
   
309
    --password nzpass \

    
   
310
    --export-dir hdfsdir \

    
   
311
    --input-fields-terminated-by "\t"

    
   
312
----
src/java/org/apache/sqoop/lib/DelimiterSet.java
Revision 4e9bcab New Change
 
src/java/org/apache/sqoop/manager/DefaultManagerFactory.java
Revision 54eb258 New Change
 
src/java/org/apache/sqoop/manager/DirectNetezzaManager.java
New File
 
src/java/org/apache/sqoop/manager/MySQLUtils.java
Revision ef18818 New Change
 
src/java/org/apache/sqoop/manager/NetezzaManager.java
New File
 
src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaDBDataSliceSplitter.java
New File
 
src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java
New File
 
src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableImportMapper.java
New File
 
src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableRecordExportMapper.java
New File
 
src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableTextExportMapper.java
New File
 
src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java
New File
 
src/java/org/apache/sqoop/mapreduce/netezza/NetezzaDataDrivenDBInputFormat.java
New File
 
src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableExportJob.java
New File
 
src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableImportJob.java
New File
 
src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputFormat.java
New File
 
src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputSplit.java
New File
 
src/test/com/cloudera/sqoop/manager/DirectNetezzaExportManualTest.java
New File
 
src/test/com/cloudera/sqoop/manager/NetezzaImportManualTest.java
New File
 
src/test/com/cloudera/sqoop/manager/NetezzaTestUtils.java
New File
 
  1. src/docs/user/connectors.txt: Loading...
  2. src/java/org/apache/sqoop/lib/DelimiterSet.java: Loading...
  3. src/java/org/apache/sqoop/manager/DefaultManagerFactory.java: Loading...
  4. src/java/org/apache/sqoop/manager/DirectNetezzaManager.java: Loading...
  5. src/java/org/apache/sqoop/manager/MySQLUtils.java: Loading...
  6. src/java/org/apache/sqoop/manager/NetezzaManager.java: Loading...
  7. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaDBDataSliceSplitter.java: Loading...
  8. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableExportMapper.java: Loading...
  9. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableImportMapper.java: Loading...
  10. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableRecordExportMapper.java: Loading...
  11. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableTextExportMapper.java: Loading...
  12. src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaJDBCStatementRunner.java: Loading...
  13. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaDataDrivenDBInputFormat.java: Loading...
  14. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableExportJob.java: Loading...
  15. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableImportJob.java: Loading...
  16. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputFormat.java: Loading...
  17. src/java/org/apache/sqoop/mapreduce/netezza/NetezzaExternalTableInputSplit.java: Loading...
  18. src/test/com/cloudera/sqoop/manager/DirectNetezzaExportManualTest.java: Loading...
  19. src/test/com/cloudera/sqoop/manager/NetezzaImportManualTest.java: Loading...
  20. src/test/com/cloudera/sqoop/manager/NetezzaTestUtils.java: Loading...