Review Board 1.7.22


HBASE-5209: Add active and backup masters to ClusterStatus

Review Request #3892 - Created Feb. 14, 2012 and updated

David Wang
0.94.0, 0.92.1, 0.90.7
HBASE-5209
Reviewers
hbase
hbase-git
Problem:
There is no method in the HBase client-facing APIs to determine which of the masters is currently active.  This can be especially useful in setups with multiple backup masters.

Solution:
Augment ClusterStatus to return the currently active master and the list of backup masters.

Notes:
* I uncovered a race condition in ActiveMasterManager, between when it determines that it did not win the original race to be the active master, and when it reads the ServerName of the active master.  If the active master goes down in that time, the read to determine the active master's ServerName will fail ungracefully and the candidate master will abort.  The solution incorporated in this patch is to check to see if the read of the ServerName succeeded before trying to use it.
* I fixed some minor formatting issues while going through the code.  I can take these changes out if it is considered improper to commit such non-related changes with the main changes.
* Ran mvn -P localTests test multiple times - no new tests fail
* Ran mvn -P localTests -Dtest=TestActiveMasterManager test multiple runs - no failures
* Ran mvn -P localTests -Dtest=TestMasterFailover test multiple runs - no failures
* Started active and multiple backup masters, then killed active master, then brought it back up (will now be a backup master)
  * Did the following before and after killing
    * hbase hbck -details - checked output to see that active and backup masters are reported properly
    * zk_dump - checked that active and backup masters are reported properly
* Started cluster with no backup masters to make sure change operates correctly that way
* Tested build with this diff vs. build without this diff, in all combinations of client and server
  * Verified that new client can run against old servers without incident and with the defaults applied.
  * Note that old clients get an error when running against new servers, because the old readFields() code in ClusterStatus does not handle exceptions of any kind.  This is not solvable, at least in the scope of this change.

12/02/15 15:15:38 INFO zookeeper.ClientCnxn: Session establishment complete on server haus02.sf.cloudera.com/172.29.5.33:30181, sessionid = 0x135834c75e20008, negotiated timeout = 5000
12/02/15 15:15:39 ERROR io.HbaseObjectWritable: Error in readFields
A record version mismatch occured. Expecting v2, found v3
        at org.apache.hadoop.io.VersionedWritable.readFields(VersionedWritable.java:46)
        at org.apache.hadoop.hbase.ClusterStatus.readFields(ClusterStatus.java:247)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:583)
        at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)

* Ran dev-support/test-patch.sh - no new issues fail:

-1 overall.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 7 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated -136 warning messages.  

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version ) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.
src/main/java/org/apache/hadoop/hbase/ClusterStatus.java
Revision 9df4c10 New Change
[20] 68 lines
[+20] [+] public class ClusterStatus extends VersionedWritable {
69
   *   <dt>3</dt> <dd>Added master and backupMasters</dd>
69
   *   <dt>3</dt> <dd>Added master and backupMasters</dd>
70
   * </dl>
70
   * </dl>
71
   */
71
   */
72
  private static final byte VERSION_MASTER_BACKUPMASTERS = 2;
72
  private static final byte VERSION_MASTER_BACKUPMASTERS = 2;
73
  private static final byte VERSION = 2;
73
  private static final byte VERSION = 2;
74
  private static final String UNKNOWN_SERVERNAME = "unknown";
74
  private static final String UNKNOWN_SERVERNAME = "#unknown#";
75

    
   
75

   
76
  private String hbaseVersion;
76
  private String hbaseVersion;
77
  private Map<ServerName, HServerLoad> liveServers;
77
  private Map<ServerName, HServerLoad> liveServers;
78
  private Collection<ServerName> deadServers;
78
  private Collection<ServerName> deadServers;
79
  private ServerName master;
79
  private ServerName master;
[+20] [20] 95 lines
[+20] [+] public boolean equals(Object o) {
175
      return false;
175
      return false;
176
    }
176
    }
177
    return (getVersion() == ((ClusterStatus)o).getVersion()) &&
177
    return (getVersion() == ((ClusterStatus)o).getVersion()) &&
178
      getHBaseVersion().equals(((ClusterStatus)o).getHBaseVersion()) &&
178
      getHBaseVersion().equals(((ClusterStatus)o).getHBaseVersion()) &&
179
      this.liveServers.equals(((ClusterStatus)o).liveServers) &&
179
      this.liveServers.equals(((ClusterStatus)o).liveServers) &&
180
      this.deadServers.equals(((ClusterStatus)o).deadServers) &&
180
      this.deadServers.containsAll(((ClusterStatus)o).deadServers) &&
181
      Arrays.equals(this.masterCoprocessors,
181
      Arrays.equals(this.masterCoprocessors,
182
                    ((ClusterStatus)o).masterCoprocessors) &&
182
                    ((ClusterStatus)o).masterCoprocessors) &&
183
      this.master.equals(((ClusterStatus)o).master) &&
183
      this.master.equals(((ClusterStatus)o).master) &&
184
      this.backupMasters.equals(((ClusterStatus)o).backupMasters);
184
      this.backupMasters.containsAll(((ClusterStatus)o).backupMasters);
185
  }
185
  }
186

    
   
186

   
187
  /**
187
  /**
188
   * @see java.lang.Object#hashCode()
188
   * @see java.lang.Object#hashCode()
189
   */
189
   */
[+20] [20] 163 lines
src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
Revision 7416ae2 New Change
 
src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Revision e2bbbd0 New Change
 
  1. src/main/java/org/apache/hadoop/hbase/ClusterStatus.java: Loading...
  2. src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java: Loading...
  3. src/main/java/org/apache/hadoop/hbase/master/HMaster.java: Loading...