PIG-3502 Initial implementation of TezLauncher
Review Request #14601 - Created Oct. 11, 2013 and submitted
Initial implementation of TezLauncher. The patch includes the following changes: - Refactors Launcher and MapReduceLauncher. Basically, I moved what's MR-specific to MapReduceLauncher and kept what's common in Launcher. - Removes kill() method from ExecutionEngine interface. It was used merely by Launcher, and it didn't seem useful. - Implements the DAG submission of TezLauncher. Basically, I overrode the submit() method in TezJob (extension of ControlledJob) that calls DAGClient.submit(). - Adds a skeleton TezStats class that extends PigStats. This class needs to be implemented to monitor job status after submission. - Adds a unit test called TestTezLauncher. For now, all it does is to compile queries and submit a DAG. More meaningful unit tests will be added after TezStats is implemented. - Adds a TezLocalExecType. To start up a YARN mini cluster, I needed non-local Tez mode. So I created "tez_local" and "tez" modes. - Whitespace and indentation fixes.
TestTezLauncher is added.
|We'll also need to set some environment variables (I think CLASSPATH at a minimal). Check out MRHelpers||Mark Wagner||Oct. 12, 2013, 5:30 a.m.||Open|
|This returns a configuration w/o defaults, but the TezJobControlCompiler turns it into a TezConfiguration, so it needs the tez-site items ...||Mark Wagner||Oct. 12, 2013, 5:30 a.m.||Open|
Review request changed
Updated (Oct. 12, 2013, 1:19 a.m.)
Looking at the mini cluster log, I found that DAG was failing with NPE because local resources were not set for each vertex. I updated DAG construction code to set local resources in vertex. Now DAG is successfully launched by AM, but it fails due to YARN-1229 in hadoop 2.1.1-beta. The test case doesn't fail because it doesn't check the job status for now. I will follow up with Tez devs about this YARN AM failure. Btw, there was an incorrect statement in my description: >> Removes kill() method from ExecutionEngine interface. It was used merely by Launcher, and it didn't seem useful. kill() was used by ExecutionEngineKiller, but since I moved it to Launcher and renamed it as HangingJobKiller, kill() is no longer needed in ExecutionEngine interface.
Posted (Oct. 12, 2013, 5:30 a.m.)
I played around with this this afternoon and noticed a few things. I haven't gone through all of it yet.
We'll also need to set some environment variables (I think CLASSPATH at a minimal). Check out MRHelpers
This returns a configuration w/o defaults, but the TezJobControlCompiler turns it into a TezConfiguration, so it needs the tez-site items as well. I added a toConfiguration(Props, bool) that optionally includes defaults.
I'm also getting error messages when I close the session about killing the ExecutionEngine. I haven't looked into that, though.
Review request changed
Updated (Oct. 14, 2013, 6:55 p.m.)
- changed from pending to submitted