Review Board 1.7.22


PIG-2924 PigStats should not be assuming all Storage classes to be file-based storage

Review Request #8122 - Created Nov. 19, 2012 and submitted

Cheolsoo Park
PIG-2924
Reviewers
pig
billgraham, sms
pig-git
Please see:
https://issues.apache.org/jira/browse/PIG-2924
ant test -Dhadoopversion=20/23
conf/pig.properties
Revision a453940 New Change
[20] 125 lines
[+20]
126

    
   
126

   
127
########## Override the default Reducer Estimator logic #############
127
########## Override the default Reducer Estimator logic #############
128

    
   
128

   
129
# By default, the logic to estimate the number of reducers to use for a given job lives in:
129
# By default, the logic to estimate the number of reducers to use for a given job lives in:
130
#   org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
130
#   org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
131
# This logic can be relaced by implementing the following interface:
131
# This logic can be replaced by implementing the following interface:
132
#   org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigReducerEstimator
132
#   org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigReducerEstimator
133

    
   
133

   
134
# This class will be invoked to estimate the number of reducers to use.
134
# This class will be invoked to estimate the number of reducers to use.
135
# pig.exec.reducer.estimator = <fully qualified class name of a PigReducerEstimator implementation>
135
# pig.exec.reducer.estimator = <fully qualified class name of a PigReducerEstimator implementation>
136

    
   
136

   
137
# Optionally, you can supply a single String argument to pass to your PigReducerEstimator.
137
# Optionally, you can supply a single String argument to pass to your PigReducerEstimator.
138
# pig.exec.reducer.estimator.arg = <somevalue>
138
# pig.exec.reducer.estimator.arg = <somevalue>
139

    
   
139

   
140
#####################################################################
140
#####################################################################
141

    
   
141

   

    
   
142
###### Override the default Pig Stats Output Size Reader logic ######

    
   
143

   

    
   
144
# By default, the size of reducers output is computed as the total size of

    
   
145
# output files. But since not every storage is file-based, this logic is not

    
   
146
# always applicable. If that is the case, the logic can be replaced by

    
   
147
# implementing the following interface:

    
   
148
#   org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigStatsOutputSizeReader

    
   
149

   

    
   
150
# This class will be invoked to compute the size of reducers output.

    
   
151
# pig.stats.output.size.reader = <fully qualified class name of a PigStatsOutputSizeReader implementation>

    
   
152

   

    
   
153
# If you need to register more than one reader, you can register them as a comma

    
   
154
# separated list. Every reader implements a boolean supports(POStore sto) method.

    
   
155
# When there are more than one reader, they are consulted in order, and the

    
   
156
# first one whose supports() method returns true will be used.

    
   
157
#

    
   
158
#####################################################################

    
   
159

   
142
#pig.load.default.statements=
160
#pig.load.default.statements=
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/FileBasedOutputSizeReader.java
Revision e69de29 New Change
 
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigStatsOutputSizeReader.java
Revision e69de29 New Change
 
src/org/apache/pig/tools/pigstats/JobStats.java
Revision fca29d7 New Change
 
test/org/apache/pig/test/TestJobStats.java
Revision 42926bc New Change
 
  1. conf/pig.properties: Loading...
  2. src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/FileBasedOutputSizeReader.java: Loading...
  3. src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigStatsOutputSizeReader.java: Loading...
  4. src/org/apache/pig/tools/pigstats/JobStats.java: Loading...
  5. test/org/apache/pig/test/TestJobStats.java: Loading...