Review Board 1.7.22


MAPREDUCE-2584. Check for serializers early, and give out more information regarding missing serializers.

Review Request #885 - Created June 11, 2011 and submitted

Harsh J
trunk
MAPREDUCE-2584
Reviewers
hadoop-mapreduce
hadoop-mapreduce-git
As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..

This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
Added a test case that expects a failure if no io.serializers are present.

Diff revision 2

This is not the most recent revision of the diff. The latest diff is revision 4. See what's changed.

1 2 3 4
1 2 3 4

  1. src/java/org/apache/hadoop/mapred/MapTask.java: Loading...
  2. src/java/org/apache/hadoop/mapreduce/JobSubmitter.java: Loading...
  3. src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java: Loading...
src/java/org/apache/hadoop/mapred/MapTask.java
Revision 21599c2 New Change
[20] 829 lines
[+20] [+] private class MapOutputBuffer<K extends Object, V extends Object>
830
      comparator = job.getOutputKeyComparator();
830
      comparator = job.getOutputKeyComparator();
831
      keyClass = (Class<K>)job.getMapOutputKeyClass();
831
      keyClass = (Class<K>)job.getMapOutputKeyClass();
832
      valClass = (Class<V>)job.getMapOutputValueClass();
832
      valClass = (Class<V>)job.getMapOutputValueClass();
833
      serializationFactory = new SerializationFactory(job);
833
      serializationFactory = new SerializationFactory(job);
834
      keySerializer = serializationFactory.getSerializer(keyClass);
834
      keySerializer = serializationFactory.getSerializer(keyClass);

    
   
835
      if (this.keySerializer == null) {

    
   
836
        throw new IOException("Could not find a serializer for the Map-Output Key class: '" + keyClass.getCanonicalName() + "'.");

    
   
837
      }
835
      keySerializer.open(bb);
838
      keySerializer.open(bb);
836
      valSerializer = serializationFactory.getSerializer(valClass);
839
      valSerializer = serializationFactory.getSerializer(valClass);

    
   
840
      if (this.valSerializer == null) {

    
   
841
        throw new IOException("Could not find a serializer for the Map-Output Value class: '" + valClass.getCanonicalName() + "'.");

    
   
842
      }
837
      valSerializer.open(bb);
843
      valSerializer.open(bb);
838

    
   
844

   
839
      // output counters
845
      // output counters
840
      mapOutputByteCounter = reporter.getCounter(TaskCounter.MAP_OUTPUT_BYTES);
846
      mapOutputByteCounter = reporter.getCounter(TaskCounter.MAP_OUTPUT_BYTES);
841
      mapOutputRecordCounter =
847
      mapOutputRecordCounter =
[+20] [20] 939 lines
src/java/org/apache/hadoop/mapreduce/JobSubmitter.java
Revision 751d528 New Change
 
src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java
Revision 5fa329a New Change
 
  1. src/java/org/apache/hadoop/mapred/MapTask.java: Loading...
  2. src/java/org/apache/hadoop/mapreduce/JobSubmitter.java: Loading...
  3. src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java: Loading...