Review Board 1.7.22


PIG-3047 Check the size of a relation before adding it to distributed cache in Replicated join

Review Request #14964 - Created Oct. 26, 2013 and updated

Aniket Mokashi
PIG-3047
Reviewers
pig
cheolsoo, daijy, dvryaboy, julien
pig
-Check the size of a relation before adding it to distributed cache in Replicated join - 1G by default

 
trunk/conf/pig.properties
Revision 1536246 New Change
[20] 81 lines
[+20]
82

    
   
82

   
83
#Use this option only when your Pig job will otherwise die because of
83
#Use this option only when your Pig job will otherwise die because of
84
#using more counters than hadoop configured limit
84
#using more counters than hadoop configured limit
85
#pig.disable.counter=true
85
#pig.disable.counter=true
86

    
   
86

   
87
# Use this option to turn on UDF timers. This will cause two 
87
# By default, pig will allow 1GB of data to be replicated using

    
   
88
# the distributed cache when doing fragment-replicated join.

    
   
89
# pig.join.replicated.max.bytes=1000000000

    
   
90

   

    
   
91
# Use this option to turn on UDF timers. This will cause two
88
# counters to be tracked for every UDF and LoadFunc in your script:
92
# counters to be tracked for every UDF and LoadFunc in your script:
89
# approx_microsecs measures approximate time spent inside a UDF
93
# approx_microsecs measures approximate time spent inside a UDF
90
# approx_invocations reports the approximate number of times the UDF was invoked
94
# approx_invocations reports the approximate number of times the UDF was invoked
91
# pig.udf.profile=false
95
# pig.udf.profile=false
92

    
   
96

   
[+20] [20] 143 lines
trunk/src/docs/src/documentation/content/xdocs/perf.xml
Revision 1536246 New Change
 
trunk/src/org/apache/pig/PigConfiguration.java
Revision 1536246 New Change
 
trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/InputSizeReducerEstimator.java
Revision 1536246 New Change
 
trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
Revision 1536246 New Change
 
trunk/src/org/apache/pig/backend/hadoop/executionengine/util/MapRedUtil.java
Revision 1536246 New Change
 
trunk/src/org/apache/pig/impl/util/Utils.java
Revision 1536246 New Change
 
trunk/test/org/apache/pig/test/PigStorageWithStatistics.java
Revision 1536246 New Change
 
trunk/test/org/apache/pig/test/TestFRJoin2.java
Revision 1536246 New Change
 
  1. trunk/conf/pig.properties: Loading...
  2. trunk/src/docs/src/documentation/content/xdocs/perf.xml: Loading...
  3. trunk/src/org/apache/pig/PigConfiguration.java: Loading...
  4. trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/InputSizeReducerEstimator.java: Loading...
  5. trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java: Loading...
  6. trunk/src/org/apache/pig/backend/hadoop/executionengine/util/MapRedUtil.java: Loading...
  7. trunk/src/org/apache/pig/impl/util/Utils.java: Loading...
  8. trunk/test/org/apache/pig/test/PigStorageWithStatistics.java: Loading...
  9. trunk/test/org/apache/pig/test/TestFRJoin2.java: Loading...