Review Board 1.7.22


PIG-3480 TFile-based tmpfile compression crashes in some cases

Review Request #14552 - Created Oct. 9, 2013 and updated

Aniket Mokashi
trunk
PIG-3480
Reviewers
pig
cheolsoo, dvryaboy, julien, rohini
pig
- Added a new parameter to make SequenceFileInterStorage optional.
- Added tests
- Refactored apis

 
trunk/conf/pig.properties
Revision 1530468 New Change
[20] 60 lines
[+20]
61
#pig.cachedbag.memusage=0.2
61
#pig.cachedbag.memusage=0.2
62
#pig.skewedjoin.reduce.memusagea=0.3
62
#pig.skewedjoin.reduce.memusagea=0.3
63
#pig.exec.nocombiner=false
63
#pig.exec.nocombiner=false
64
#opt.multiquery=true
64
#opt.multiquery=true
65
#pig.tmpfilecompression=false
65
#pig.tmpfilecompression=false

    
   
66
#pig.tmpfilecompression.codec=gz

    
   
67
#pig.tmpfilecompression.storage=seqfile
66

    
   
68

   
67
#value can be lzo or gzip
69
#value can be lzo or gzip
68
#pig.tmpfilecompression.codec=gzip
70
#pig.tmpfilecompression.codec=gzip
69
#pig.noSplitCombination=true
71
#pig.noSplitCombination=true
70

    
   
72

   
[+20] [20] 156 lines
trunk/src/org/apache/pig/PigConfiguration.java
Revision 1530468 New Change
 
trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
Revision 1530468 New Change
 
trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java
Revision 1530468 New Change
 
trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPartitionRearrange.java
Revision 1530468 New Change
 
trunk/src/org/apache/pig/backend/hadoop/executionengine/util/MapRedUtil.java
Revision 1530468 New Change
 
trunk/src/org/apache/pig/impl/io/InterStorage.java
Revision 1530468 New Change
 
trunk/src/org/apache/pig/impl/io/SequenceFileInterStorage.java
New File
 
trunk/src/org/apache/pig/impl/io/TFileStorage.java
Revision 1530468 New Change
 
trunk/src/org/apache/pig/impl/util/Utils.java
Revision 1530468 New Change
 
trunk/test/org/apache/pig/test/TestTmpFileCompression.java
Revision 1530468 New Change
 
  1. trunk/conf/pig.properties: Loading...
  2. trunk/src/org/apache/pig/PigConfiguration.java: Loading...
  3. trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java: Loading...
  4. trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java: Loading...
  5. trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPartitionRearrange.java: Loading...
  6. trunk/src/org/apache/pig/backend/hadoop/executionengine/util/MapRedUtil.java: Loading...
  7. trunk/src/org/apache/pig/impl/io/InterStorage.java: Loading...
  8. trunk/src/org/apache/pig/impl/io/SequenceFileInterStorage.java: Loading...
  9. trunk/src/org/apache/pig/impl/io/TFileStorage.java: Loading...
  10. trunk/src/org/apache/pig/impl/util/Utils.java: Loading...
  11. trunk/test/org/apache/pig/test/TestTmpFileCompression.java: Loading...