Review Board 1.7.22


SQOOP-621: Requesting support for upsert export with MySQL

Review Request #7588 - Created Oct. 15, 2012 and submitted

Jarek Cecho
SQOOP-621
Reviewers
Sqoop
sqoop-trunk
I've implemented upsert functionality using MySQL clause INSERT INTO ... ON DUPLICATE KEY UPDATE. This clause have slightly different purpose than Oracle's MERGE statement and therefore the functionality is slightly different. I've provided warning message notifying user that column names specified in --update-key parameter are not going to be used.
I've added new unit tests plus live testing.
src/docs/user/connectors.txt
Revision 930a4996e47a18a763bfbed59656cf8ea5cff05e New Change
[20] 18 lines
[+20]
19

    
   
19

   
20

    
   
20

   
21
Notes for specific connectors
21
Notes for specific connectors
22
-----------------------------
22
-----------------------------
23

    
   
23

   

    
   
24
MySQL JDBC Connector

    
   
25
~~~~~~~~~~~~~~~~~~~~

    
   
26

   

    
   
27
This section contains information specific to MySQL JDBC Connector.

    
   
28

   

    
   
29
Upsert functionality

    
   
30
^^^^^^^^^^^^^^^^^^^^

    
   
31

   

    
   
32
MySQL JDBC Connector is supporting upsert functionality using argument

    
   
33
+\--update-mode allowinsert+. To achieve that Sqoop is using MySQL clause INSERT INTO

    
   
34
... ON DUPLICATE KEY UPDATE. This clause do not allow user to specify which columns

    
   
35
should be used to distinct whether we should update existing row or add new row. Instead

    
   
36
this clause relies on table's unique keys (primary key belongs to this set). MySQL

    
   
37
will try to insert new row and if the insertion fails with duplicate unique key error

    
   
38
it will update appropriate row instead. As a result, Sqoop is ignoring values specified

    
   
39
in parameter +\--update-key+, however user needs to specify at least one valid column

    
   
40
to turn on update mode itself.

    
   
41

   
24
PostgreSQL Connector
42
PostgreSQL Connector
25
~~~~~~~~~~~~~~~~~~~~~
43
~~~~~~~~~~~~~~~~~~~~~
26

    
   
44

   
27
Extra arguments
45
Extra arguments
28
^^^^^^^^^^^^^^^
46
^^^^^^^^^^^^^^^
[+20] [20] 142 lines
src/java/org/apache/sqoop/manager/DirectMySQLManager.java
Revision 2e8d63e5b45f0b74b0ccce9e9bff4a1f798bb6a8 New Change
 
src/java/org/apache/sqoop/manager/MySQLManager.java
Revision a817aa41fee3385b6e8796cadd4c09319b0b6e68 New Change
 
src/java/org/apache/sqoop/mapreduce/JdbcUpdateExportJob.java
Revision c8e17c236f272387fd14ef6d222cc0edb5fe59ab New Change
 
src/java/org/apache/sqoop/mapreduce/mysql/MySQLUpsertOutputFormat.java
New File
 
src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java
Revision f00cac4eb7c0600dc567717eff391909a831c6fb New Change
 
src/test/com/cloudera/sqoop/manager/ManualMySQLTests.java
New File
 
  1. src/docs/user/connectors.txt: Loading...
  2. src/java/org/apache/sqoop/manager/DirectMySQLManager.java: Loading...
  3. src/java/org/apache/sqoop/manager/MySQLManager.java: Loading...
  4. src/java/org/apache/sqoop/mapreduce/JdbcUpdateExportJob.java: Loading...
  5. src/java/org/apache/sqoop/mapreduce/mysql/MySQLUpsertOutputFormat.java: Loading...
  6. src/test/com/cloudera/sqoop/manager/JdbcMySQLExportTest.java: Loading...
  7. src/test/com/cloudera/sqoop/manager/ManualMySQLTests.java: Loading...