Review Board 1.7.22


SQOOP-1220 Sqoop2: Improve default catalina.properties file to work out of the box on both Hadoop 1 and 2

Review Request #14795 - Created Oct. 21, 2013 and updated

Jarek Cecho
SQOOP-1220
Reviewers
Sqoop
sqoop-sqoop2
I've tweak the configuration in a way that it works out of the box on both Hadoop 1 and Hadoop 2 (BigTop based installations). There is one caveat though, on Hadoop 1.0 when stopping server, following warning will be printed out:

Oct 21, 2013 1:06:07 PM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/usr/lib/hadoop-hdfs], exists: [false], isDirectory: [false], canRead: [false]
Oct 21, 2013 1:06:07 PM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/usr/lib/hadoop-hdfs/lib], exists: [false], isDirectory: [false], canRead: [false]
Oct 21, 2013 1:06:07 PM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/usr/lib/hadoop-mapreduce], exists: [false], isDirectory: [false], canRead: [false]
Oct 21, 2013 1:06:07 PM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/usr/lib/hadoop-mapreduce/lib], exists: [false], isDirectory: [false], canRead: [false]
Oct 21, 2013 1:06:07 PM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/usr/lib/hadoop-yarn], exists: [false], isDirectory: [false], canRead: [false]
Oct 21, 2013 1:06:07 PM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/usr/lib/hadoop-yarn/lib], exists: [false], isDirectory: [false], canRead: [false]

The warnings are harmless and are printed out only while shutting the server down, so hopefully they won't be confusing users who are running default configuration on Hadoop 1.
Tried the default config file on both Hadoop 1 (BigTop 0.3) and Hadoop 2 (BigTop 0.6).

Diff revision 1 (Latest)

  1. dist/src/main/server/conf/catalina.properties: Loading...
dist/src/main/server/conf/catalina.properties
Revision 5a47295d6e1bfa2114f86d620bb1452bc576df71 New Change
[20] 41 lines
[+20]
42
# Examples:
42
# Examples:
43
#     "foo": Add this folder as a class repository
43
#     "foo": Add this folder as a class repository
44
#     "foo/*.jar": Add all the JARs of the specified folder as class 
44
#     "foo/*.jar": Add all the JARs of the specified folder as class 
45
#                  repositories
45
#                  repositories
46
#     "foo/bar.jar": Add bar.jar as a class repository
46
#     "foo/bar.jar": Add bar.jar as a class repository
47
common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar
47
#

    
   
48
# Sqoop Comment:

    
   
49
# You need set all Hadoop dependencies in this property. Please check out

    
   
50
# Sqoop installation guide for more details.

    
   
51
#

    
   
52
# Hadoop 1 example (based on default BigTop locations):

    
   
53
#common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar

    
   
54
#

    
   
55
# Hadoop 2 example (based on default BigTop locations):

    
   
56
#common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar,/usr/lib/hadoop/lib/*.jar,/usr/lib/hadoop-hdfs/*.jar,/usr/lib/hadoop-hdfs/lib/*.jar,/usr/lib/hadoop-mapreduce/*.jar,/usr/lib/hadoop-mapreduce/lib/*.jar,/usr/lib/hadoop-yarn/*.jar,/usr/lib/hadoop-yarn/lib/*.jar

    
   
57
#

    
   
58
common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar,/usr/lib/hadoop/lib/*.jar,/usr/lib/hadoop-hdfs/*.jar,/usr/lib/hadoop-hdfs/lib/*.jar,/usr/lib/hadoop-mapreduce/*.jar,/usr/lib/hadoop-mapreduce/lib/*.jar,/usr/lib/hadoop-yarn/*.jar,/usr/lib/hadoop-yarn/lib/*.jar
48

    
   
59

   
49
#
60
#
50
# List of comma-separated paths defining the contents of the "server" 
61
# List of comma-separated paths defining the contents of the "server" 
51
# classloader. Prefixes should be used to define what is the repository type.
62
# classloader. Prefixes should be used to define what is the repository type.
52
# Path may be relative to the CATALINA_HOME or CATALINA_BASE path or absolute.
63
# Path may be relative to the CATALINA_HOME or CATALINA_BASE path or absolute.
[+20] [20] 29 lines
  1. dist/src/main/server/conf/catalina.properties: Loading...