Review Board 1.7.22


disable optimizations via pig properties

Review Request #11032 - Created May 9, 2013 and updated

Travis Crawford
PIG-3317
Reviewers
pig
billgraham, julien, pengfeng
pig-git
Update pig to allow disabling optimizations via pig properties. Currently optimizations must be disabled via command-line options. Pig properties can be set in pig.properties, "set" commands in scripts themselves, and command-line -D options.

The use-case is, for scripts that require certain optimizations to be disabled, allowing the script itself to disable the optimization. Currently whatever runs the script needs to specially handle disabling the optimization for that specific query.
Manually tested on a fully-distributed cluster.

THIS FAILS:
PIG_CONF_DIR=/etc/pig/conf ./bin/pig -c query.pig

THIS WORKS:
PIG_CONF_DIR=/etc/pig/conf ./bin/pig -Dpig.optimizer.rules.disabled=ColumnMapKeyPrune -c query.pig

Notice how "-Dpig.optimizer.rules.disabled=ColumnMapKeyPrune" specifies a pig property, which could be in pig.properties, or the script itself.


Failure message:

Pig Stack Trace
---------------
ERROR 2229: Couldn't find matching uid -1 for project (Name: Project Type: bytearray Uid: 97550 Input: 0 Column: 1)

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1067: Unable to explain alias null
	at org.apache.pig.PigServer.explain(PigServer.java:1057)
	at org.apache.pig.tools.grunt.GruntParser.explainCurrentBatch(GruntParser.java:419)
	at org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:351)
	at org.apache.pig.tools.grunt.Grunt.checkScript(Grunt.java:98)
	at org.apache.pig.Main.run(Main.java:607)
	at org.apache.pig.Main.main(Main.java:152)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
	at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
	at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:281)
	at org.apache.pig.PigServer.compilePp(PigServer.java:1380)
	at org.apache.pig.PigServer.explain(PigServer.java:1042)
	... 10 more
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2229: Couldn't find matching uid -1 for project (Name: Project Type: bytearray Uid: 97550 Input: 0 Column: 1)
	at org.apache.pig.newplan.logical.optimizer.ProjectionPatcher$ProjectionRewriter.visit(ProjectionPatcher.java:91)
	at org.apache.pig.newplan.logical.expression.ProjectExpression.accept(ProjectExpression.java:207)
	at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
	at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
	at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
	at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:142)
	at org.apache.pig.newplan.logical.relational.LOInnerLoad.accept(LOInnerLoad.java:128)
	at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
	at org.apache.pig.newplan.logical.optimizer.AllExpressionVisitor.visit(AllExpressionVisitor.java:124)
	at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:76)
	at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
	at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
	at org.apache.pig.newplan.logical.optimizer.ProjectionPatcher.transformed(ProjectionPatcher.java:48)
	at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:113)
	... 13 more
src/docs/src/documentation/content/xdocs/perf.xml
Revision 108ae7e New Change
[20] 474 lines
[+20]
475
<!-- ==================================================================== -->
475
<!-- ==================================================================== -->
476
 <!-- OPTIMIZATION RULES -->
476
 <!-- OPTIMIZATION RULES -->
477
<section id="optimization-rules">
477
<section id="optimization-rules">
478
<title>Optimization Rules</title>
478
<title>Optimization Rules</title>
479

    
   
479

   
480
<p>Pig supports various optimization rules. By default optimization, and all optimization rules, are turned on. 
480
<p>Pig supports various optimization rules, all of which are enabled by default.
481
To turn off optimiztion, use:</p>
481
To disable all or specific optimizations, use one or more of the following methods.

    
   
482
Note some optimization rules are mandatory and cannot be disabled.</p>
482

    
   
483

   
483
<source>
484
    <ul>
484
pig -optimizer_off [opt_rule | all ]
485
        <li>The <code>pig.optimizer.rules.disabled</code>
485
</source>
486
            <a href="start.html#properties">pig property</a>, which accepts a

    
   
487
            comma-separated list of optimization rules to disable; the <code>all</code>

    
   
488
            keyword disables all non-mandatory optimizations.

    
   
489
            (e.g.: <code>set pig.optimizer.rules.disabled 'ColumnMapKeyPrune';</code>)</li>

    
   
490
        <li>The <code>-t, -optimizer_off</code> command-line options.

    
   
491
            (e.g.: <code>pig -optimizer_off [opt_rule | all]</code>)</li>

    
   
492
    </ul>

    
   
493

   

    
   
494
<p><code>FilterLogicExpressionSimplifier</code> is an exception to the above.

    
   
495
The rule is disabled by default, and enabled by setting the

    
   
496
<code>pig.exec.filterLogicExpressionSimplifier</code> pig property to true.</p>
486

    
   
497

   
487
<p>Note that some rules are mandatory and cannot be turned off.</p>

   
488

    
   
498

   
489
<!-- +++++++++++++++++++++++++++++++ -->
499
<!-- +++++++++++++++++++++++++++++++ -->
490
<section id="FilterLogicExpressionSimplifier">
500
<section id="FilterLogicExpressionSimplifier">
491
<title>FilterLogicExpressionSimplifier</title>
501
<title>FilterLogicExpressionSimplifier</title>
492
<p>This rule simplifies the expression in filter statement.</p>
502
<p>This rule simplifies the expression in filter statement.</p>
[+20] [20] 740 lines
src/org/apache/pig/Main.java
Revision f97ed9f New Change
 
src/org/apache/pig/PigConstants.java
Revision ea77e97 New Change
 
src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
Revision 4dab4e8 New Change
 
src/org/apache/pig/impl/PigImplConstants.java
New File
 
src/org/apache/pig/newplan/logical/optimizer/LogicalPlanOptimizer.java
Revision d26f381 New Change
 
test/org/apache/pig/test/TestEvalPipeline2.java
Revision 39cf807 New Change
 
  1. src/docs/src/documentation/content/xdocs/perf.xml: Loading...
  2. src/org/apache/pig/Main.java: Loading...
  3. src/org/apache/pig/PigConstants.java: Loading...
  4. src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java: Loading...
  5. src/org/apache/pig/impl/PigImplConstants.java: Loading...
  6. src/org/apache/pig/newplan/logical/optimizer/LogicalPlanOptimizer.java: Loading...
  7. test/org/apache/pig/test/TestEvalPipeline2.java: Loading...