Review Board 1.7.22


SQOOP-601: Support custom schemas in PostgreSQL Connector

Review Request #7055 - Created Sept. 12, 2012 and submitted

Jarek Cecho
SQOOP-601
Reviewers
Sqoop
sqoop-trunk
This patch is adding support for custom schemas into PostgreSQL manager. Changes for import job were very simple and rather straightforward. On the other side, changes for export job were little tricky, but I believe that I've succeeded without breaking backward compatibility.
I've added a lot of third party tests (e.g. they won't be run automatically during "ant test") to properly test various combinations of parameters. You can invoke those tests issuing one of following commands:

ant test -Dtestcase=PostgresqlImportTest
ant test -Dtestcase=PostgresqlExportTest

I've tested compatibility by running ant test -Dhadoopversion={20,100,23,200}
src/docs/user/connectors.txt
Revision a93f14e New Change
[20] 18 lines
[+20]
19

    
   
19

   
20

    
   
20

   
21
Notes for specific connectors
21
Notes for specific connectors
22
-----------------------------
22
-----------------------------
23

    
   
23

   

    
   
24
PostgreSQL Connector

    
   
25
~~~~~~~~~~~~~~~~~~~~~

    
   
26

   

    
   
27
Extra arguments

    
   
28
^^^^^^^^^^^^^^^

    
   
29

   

    
   
30
List of all extra arguments supported by PostgreSQL Connector is shown on table

    
   
31
below:

    
   
32

   

    
   
33
.Supported PostgreSQL extra arguments:

    
   
34
[grid="all"]

    
   
35
`----------------------------------------`---------------------------------------

    
   
36
Argument                                 Description

    
   
37
---------------------------------------------------------------------------------

    
   
38
+\--schema <name>+                       Scheme name that sqoop should use. \

    
   
39
                                         Default is "public".

    
   
40
---------------------------------------------------------------------------------

    
   
41

   

    
   
42
Schema support

    
   
43
^^^^^^^^^^^^^^

    
   
44

   

    
   
45
If you need to work with table that is located in schema other than default one,

    
   
46
you need to specify extra argument +\--schema+. Custom schemas are supported for

    
   
47
both import and export job (optional staging table however must be present in the

    
   
48
same schema as target table). Example invocation:

    
   
49

   

    
   
50
----

    
   
51
$ sqoop import ... --table custom_table -- --schema custom_schema

    
   
52
----

    
   
53

   

    
   
54

   

    
   
55

   
24
pg_bulkload connector
56
pg_bulkload connector
25
~~~~~~~~~~~~~~~~~~~~~
57
~~~~~~~~~~~~~~~~~~~~~
26

    
   
58

   
27
Purpose
59
Purpose
28
^^^^^^^
60
^^^^^^^
[+20] [20] 110 lines
src/java/com/cloudera/sqoop/manager/PostgresqlManager.java
Revision 16adeb2 New Change
 
src/java/org/apache/sqoop/manager/CatalogQueryManager.java
Revision 5f2f89f New Change
 
src/java/org/apache/sqoop/manager/DirectPostgresqlManager.java
Revision a557aa1 New Change
 
src/java/org/apache/sqoop/manager/PGBulkloadManager.java
Revision 92174f8 New Change
 
src/java/org/apache/sqoop/manager/PostgresqlManager.java
Revision d18321c New Change
 
src/java/org/apache/sqoop/manager/SqlManager.java
Revision ea961cd New Change
 
src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java
Revision b574f82 New Change
 
src/java/org/apache/sqoop/mapreduce/JdbcUpdateExportJob.java
Revision 7be5ed9 New Change
 
src/java/org/apache/sqoop/mapreduce/JdbcUpsertExportJob.java
Revision f299f98 New Change
 
src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java
Revision 38e9fb9 New Change
 
src/test/com/cloudera/sqoop/ThirdPartyTests.java
Revision eeab7f3 New Change
 
src/test/com/cloudera/sqoop/manager/PostgresqlExportTest.java
New File
 
src/test/com/cloudera/sqoop/manager/PostgresqlImportTest.java
New File
 
src/test/com/cloudera/sqoop/manager/PostgresqlTest.java
Revision 0dfd1fc New Change
 
  1. src/docs/user/connectors.txt: Loading...
  2. src/java/com/cloudera/sqoop/manager/PostgresqlManager.java: Loading...
  3. src/java/org/apache/sqoop/manager/CatalogQueryManager.java: Loading...
  4. src/java/org/apache/sqoop/manager/DirectPostgresqlManager.java: Loading...
  5. src/java/org/apache/sqoop/manager/PGBulkloadManager.java: Loading...
  6. src/java/org/apache/sqoop/manager/PostgresqlManager.java: Loading...
  7. src/java/org/apache/sqoop/manager/SqlManager.java: Loading...
  8. src/java/org/apache/sqoop/mapreduce/JdbcExportJob.java: Loading...
  9. src/java/org/apache/sqoop/mapreduce/JdbcUpdateExportJob.java: Loading...
  10. src/java/org/apache/sqoop/mapreduce/JdbcUpsertExportJob.java: Loading...
  11. src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java: Loading...
  12. src/test/com/cloudera/sqoop/ThirdPartyTests.java: Loading...
  13. src/test/com/cloudera/sqoop/manager/PostgresqlExportTest.java: Loading...
  14. src/test/com/cloudera/sqoop/manager/PostgresqlImportTest.java: Loading...
  15. src/test/com/cloudera/sqoop/manager/PostgresqlTest.java: Loading...