Review Board 1.7.22


MAPREDUCE-2584. Check for serializers early, and give out more information regarding missing serializers.

Review Request #885 - Created June 11, 2011 and submitted

Harsh J
trunk
MAPREDUCE-2584
Reviewers
hadoop-mapreduce
hadoop-mapreduce-git
As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..

This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
Added a test case that expects a failure if no io.serializers are present.
src/java/org/apache/hadoop/mapred/MapTask.java
Diff Revision 1 Diff Revision 4 - File Reverted
[20] 829 lines
[+20] [+] private class MapOutputBuffer<K extends Object, V extends Object>
830
      comparator = job.getOutputKeyComparator();
830
      comparator = job.getOutputKeyComparator();
831
      keyClass = (Class<K>)job.getMapOutputKeyClass();
831
      keyClass = (Class<K>)job.getMapOutputKeyClass();
832
      valClass = (Class<V>)job.getMapOutputValueClass();
832
      valClass = (Class<V>)job.getMapOutputValueClass();
833
      serializationFactory = new SerializationFactory(job);
833
      serializationFactory = new SerializationFactory(job);
834
      keySerializer = serializationFactory.getSerializer(keyClass);
834
      keySerializer = serializationFactory.getSerializer(keyClass);
835
      if (this.keySerializer == null) {

   
836
        throw new IOException("Could not find a serializer for the Map-Output Key class: '" + keyClass.getCanonicalName() + "'.");

   
837
      }

   
838
      keySerializer.open(bb);
835
      keySerializer.open(bb);
839
      valSerializer = serializationFactory.getSerializer(valClass);
836
      valSerializer = serializationFactory.getSerializer(valClass);
840
      if (this.valSerializer == null) {

   
841
        throw new IOException("Could not find a serializer for the Map-Output Value class: '" + valClass.getCanonicalName() + "'.");

   
842
      }

   
843
      valSerializer.open(bb);
837
      valSerializer.open(bb);
844

    
   
838

   
845
      // output counters
839
      // output counters
846
      mapOutputByteCounter = reporter.getCounter(TaskCounter.MAP_OUTPUT_BYTES);
840
      mapOutputByteCounter = reporter.getCounter(TaskCounter.MAP_OUTPUT_BYTES);
847
      mapOutputRecordCounter =
841
      mapOutputRecordCounter =
[+20] [20] 939 lines
src/java/org/apache/hadoop/mapreduce/JobSubmitter.java
Diff Revision 1 Diff Revision 4
 
src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java
Diff Revision 1 Diff Revision 4
 
  1. src/java/org/apache/hadoop/mapred/MapTask.java: Loading...
  2. src/java/org/apache/hadoop/mapreduce/JobSubmitter.java: Loading...
  3. src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java: Loading...