Review Board 1.7.22


disable optimizations via pig properties

Review Request #11032 - Created May 9, 2013 and updated

Travis Crawford
PIG-3317
Reviewers
pig
billgraham, julien, pengfeng
pig-git
Update pig to allow disabling optimizations via pig properties. Currently optimizations must be disabled via command-line options. Pig properties can be set in pig.properties, "set" commands in scripts themselves, and command-line -D options.

The use-case is, for scripts that require certain optimizations to be disabled, allowing the script itself to disable the optimization. Currently whatever runs the script needs to specially handle disabling the optimization for that specific query.
Manually tested on a fully-distributed cluster.

THIS FAILS:
PIG_CONF_DIR=/etc/pig/conf ./bin/pig -c query.pig

THIS WORKS:
PIG_CONF_DIR=/etc/pig/conf ./bin/pig -Dpig.optimizer.rules.disabled=ColumnMapKeyPrune -c query.pig

Notice how "-Dpig.optimizer.rules.disabled=ColumnMapKeyPrune" specifies a pig property, which could be in pig.properties, or the script itself.


Failure message:

Pig Stack Trace
---------------
ERROR 2229: Couldn't find matching uid -1 for project (Name: Project Type: bytearray Uid: 97550 Input: 0 Column: 1)

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1067: Unable to explain alias null
	at org.apache.pig.PigServer.explain(PigServer.java:1057)
	at org.apache.pig.tools.grunt.GruntParser.explainCurrentBatch(GruntParser.java:419)
	at org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:351)
	at org.apache.pig.tools.grunt.Grunt.checkScript(Grunt.java:98)
	at org.apache.pig.Main.run(Main.java:607)
	at org.apache.pig.Main.main(Main.java:152)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
	at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
	at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:281)
	at org.apache.pig.PigServer.compilePp(PigServer.java:1380)
	at org.apache.pig.PigServer.explain(PigServer.java:1042)
	... 10 more
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2229: Couldn't find matching uid -1 for project (Name: Project Type: bytearray Uid: 97550 Input: 0 Column: 1)
	at org.apache.pig.newplan.logical.optimizer.ProjectionPatcher$ProjectionRewriter.visit(ProjectionPatcher.java:91)
	at org.apache.pig.newplan.logical.expression.ProjectExpression.accept(ProjectExpression.java:207)
	at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
	at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
	at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
	at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
	at org.apache.pig.newplan.logical.relational.LOInnerLoad.accept(LOInnerLoad.java:128)
	at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
	at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:124)
	at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:76)
	at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
	at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
	at org.apache.pig.newplan.logical.optimizer.ProjectionPatcher.transformed(ProjectionPatcher.java:48)
	at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:113)
	... 13 more
src/docs/src/documentation/content/xdocs/perf.xml
Revision 108ae7e New Change
[20] 474 lines
[+20]
475
<!-- ==================================================================== -->
475
<!-- ==================================================================== -->
476
 <!-- OPTIMIZATION RULES -->
476
 <!-- OPTIMIZATION RULES -->
477
<section id="optimization-rules">
477
<section id="optimization-rules">
478
<title>Optimization Rules</title>
478
<title>Optimization Rules</title>
479

    
   
479

   
480
<p>Pig supports various optimization rules. By default optimization, and all optimization rules, are turned on. 
480
<p>Pig supports various optimization rules, all of which are enabled by default.
481
To turn off optimiztion, use:</p>
481
To disable all or specific optimizations, use one of the following. Please note

    
   
482
some optimization rules are mandatory and cannot be disabled.</p>
482

    
   
483

   
483
<source>
484
    <ul>
484
pig -optimizer_off [opt_rule | all ]
485
        <li>The <code>pig.optimizer.rules.disabled</code> pig property, which accepts a
485
</source>
486
            comma-separated list of optimization rules to disable; the <code>all</code>
486

    
   
487
            keyword disables all non-mandatory optimizations. Setting this property
487
<p>Note that some rules are mandatory and cannot be turned off.</p>
488
            as a <a href="cmds.html#set"><code>set</code></a> command takes precedence over setting

    
   
489
            as a command-line <code>-D</code> property.

    
   
490
            (e.g.: <code>set pig.optimizer.rules.disabled 'ColumnMapKeyPrune';</code>)</li>

    
   
491
        <li>The <code>-t, -optimizer_off</code> command-line options. Rules disabled via

    
   
492
            the command-line are always disabled.

    
   
493
            (e.g.: <code>pig -optimizer_off [opt_rule | all]</code>)</li>

    
   
494
    </ul>
488

    
   
495

   
489
<!-- +++++++++++++++++++++++++++++++ -->
496
<!-- +++++++++++++++++++++++++++++++ -->
490
<section id="FilterLogicExpressionSimplifier">
497
<section id="FilterLogicExpressionSimplifier">
491
<title>FilterLogicExpressionSimplifier</title>
498
<title>FilterLogicExpressionSimplifier</title>
492
<p>This rule simplifies the expression in filter statement.</p>
499
<p>This rule simplifies the expression in filter statement.</p>
[+20] [20] 740 lines
src/org/apache/pig/Main.java
Revision f97ed9f New Change
 
src/org/apache/pig/PigConstants.java
Revision ea77e97 New Change
 
src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
Revision 4dab4e8 New Change
 
src/org/apache/pig/newplan/logical/optimizer/LogicalPlanOptimizer.java
Revision d26f381 New Change
 
test/org/apache/pig/test/TestEvalPipeline2.java
Revision 39cf807 New Change
 
  1. src/docs/src/documentation/content/xdocs/perf.xml: Loading...
  2. src/org/apache/pig/Main.java: Loading...
  3. src/org/apache/pig/PigConstants.java: Loading...
  4. src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java: Loading...
  5. src/org/apache/pig/newplan/logical/optimizer/LogicalPlanOptimizer.java: Loading...
  6. test/org/apache/pig/test/TestEvalPipeline2.java: Loading...