Review Board 1.7.22


disable optimizations via pig properties

Review Request #11032 - Created May 9, 2013 and updated

Travis Crawford
PIG-3317
Reviewers
pig
billgraham, julien, pengfeng
pig-git
Update pig to allow disabling optimizations via pig properties. Currently optimizations must be disabled via command-line options. Pig properties can be set in pig.properties, "set" commands in scripts themselves, and command-line -D options.

The use-case is, for scripts that require certain optimizations to be disabled, allowing the script itself to disable the optimization. Currently whatever runs the script needs to specially handle disabling the optimization for that specific query.
Manually tested on a fully-distributed cluster.

THIS FAILS:
PIG_CONF_DIR=/etc/pig/conf ./bin/pig -c query.pig

THIS WORKS:
PIG_CONF_DIR=/etc/pig/conf ./bin/pig -Dpig.optimizer.rules.disabled=ColumnMapKeyPrune -c query.pig

Notice how "-Dpig.optimizer.rules.disabled=ColumnMapKeyPrune" specifies a pig property, which could be in pig.properties, or the script itself.


Failure message:

Pig Stack Trace
---------------
ERROR 2229: Couldn't find matching uid -1 for project (Name: Project Type: bytearray Uid: 97550 Input: 0 Column: 1)

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1067: Unable to explain alias null
	at org.apache.pig.PigServer.explain(PigServer.java:1057)
	at org.apache.pig.tools.grunt.GruntParser.explainCurrentBatch(GruntParser.java:419)
	at org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:351)
	at org.apache.pig.tools.grunt.Grunt.checkScript(Grunt.java:98)
	at org.apache.pig.Main.run(Main.java:607)
	at org.apache.pig.Main.main(Main.java:152)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
	at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
	at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:281)
	at org.apache.pig.PigServer.compilePp(PigServer.java:1380)
	at org.apache.pig.PigServer.explain(PigServer.java:1042)
	... 10 more
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2229: Couldn't find matching uid -1 for project (Name: Project Type: bytearray Uid: 97550 Input: 0 Column: 1)
	at org.apache.pig.newplan.logical.optimizer.ProjectionPatcher$ProjectionRewriter.visit(ProjectionPatcher.java:91)
	at org.apache.pig.newplan.logical.expression.ProjectExpression.accept(ProjectExpression.java:207)
	at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
	at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
	at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
	at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
	at org.apache.pig.newplan.logical.relational.LOInnerLoad.accept(LOInnerLoad.java:128)
	at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
	at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:124)
	at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:76)
	at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
	at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
	at org.apache.pig.newplan.logical.optimizer.ProjectionPatcher.transformed(ProjectionPatcher.java:48)
	at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:113)
	... 13 more
Total:
1
Open:
0
Resolved:
1
Dropped:
0
Status:
From:
Review request changed
Updated (May 16, 2013, 12:07 a.m.)
Minor change.