Review Board 1.7.22


SQOOP-656: End to end submission engine

Review Request #7783 - Created Oct. 30, 2012 and submitted

Jarek Cecho
SQOOP-656
Reviewers
Sqoop
sqoop-sqoop2
This is sort of mega patch that is implementing first (not exactly stable and finished) version of submission engine. I've got like dozen other notes what to improve and implement, but the patch is extreme in size already. So I would like to propose getting it in, to give others chance to see what I'm working on at the moment and also retrieve their feedback. In meantime I'll continue my effort to make it very stable and very "nice" :-) My justification for doing that is that since entire sqoop 2 is alpha, it doesn't hurt committing alpha submission engine :-)

Submission engine is asynchronous pluggable interface for submitting mapreduce jobs. I've provided default mapreduce implementation (implementation that directly talks to JobTracker).

Procedure to get it working on dev box:
* Configure your hadoop cluster as usual
* Put config files to /etc/hadoop/conf or reconfigure sqoop to use different directory
* Put all required hadoop jars to tomcat lib folder (issue with resolving jar dependency is still open, so this is like temporal hack)
* Use client command submission to launch actual mapreduce jobs
I've tested it on real cluster and I was able to move first rows! Otherwise my patch is commenting a lot of tests to disable them for the time being.
Review request changed
Updated (Nov. 7, 2012, 6:22 p.m.)
  • changed from pending to submitted