Review Board 1.7.22


HBASE-5869 Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb

Review Request #4926 - Created April 28, 2012 and updated

Michael Stack
HBASE-5869
Reviewers
hbase
jxiang
hbase-git
Convert two zk users to pb: distributed log splitting and regions in transition.

Refactored distributed log splitting so we only serialize/deserialize in one location.
Less changes needed to do same for regions in transition.

Moves serialization/deserialization out of the ZKAssign, ZKSplit and into
the classes themselves so can encapsulate how serialization is done into one place
(try to make the ZK* classes just deal in bytes -- about 90% done).

Moved classes used by various packages up to top level to minimize imports
that are across package (zookeeper into protobuf and/or into regionserver and/or
master packages, etc).

A src/main/java/org/apache/hadoop/hbase/DeserializationException.java
  New generic deserialization exception.
A src/main/java/org/apache/hadoop/hbase/zookeeper/EmptyWatcher.java
D  src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
  Moved under zookeeper package.
A src/main/java/org/apache/hadoop/hbase/HBaseException.java
  New base hbase exception as suggested by hbase-5796.  New DeserializationException
  inherits from this.
A src/main/java/org/apache/hadoop/hbase/RegionTransition.java
  State of a region in transition.  Top-level because used by a
  few top-level packages.  Encapsulates pb serialization/deserialization.
M src/main/java/org/apache/hadoop/hbase/ServerName.java
  Add method to deserialize a ServeName, etc.  Encapsulates pb'ing.
M src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java
  Counters used by distributed log splitting.
A SplitLogTask
   Class that encapsulates log splitting state.  Also encapsulates pb'ing.
M src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
  Implement code for state.  Added functions to go from code to state and vice
  versa.  Used serializing.
M src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
  Remove unused imports.
D src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java
  Removed.  Replaced by RegionTransition moved to package top-level.
M src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java
M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
  Use new DeserializationException. Move to using new RegionTransition
  from RegionTransitionData class.  Pass deserialized class rather than
  byte array.  Remove duplicated code.
M src/main/java/org/apache/hadoop/hbase/master/HMaster.java
  Use new ServerName parse method rather than ZKUtil one.
M src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
M src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
  Redo to use new SplitLogTask and SplitLogCounter classes.
M src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
  expectPBMagicPrefix added
M src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
  Use new RegionTransition in place of RegionTransitionData.
M src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
  Define moved from ZKSplitLog to SplitLogManager.
M src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java
M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
  Changed method name from getZNodeData to toByteArray to match how we've
  named it elsewhere. Use new DeserializationException
M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java
  Use new RegionTransion class
M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
  Moved stuff that was in here up into SplitLogManager where better
  belongs.  Also moved serialization/deserialization up into the
  class itself: SplitLogTask.  Moved counters out to SplitLogCounter class.
M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
  Moved deserialization of ServerName out of here and up into ServerName.
M src/main/protobuf/ZooKeeper.proto
  Add two new classes, RegionTransition and SplitLogTask.

 

Diff revision 1

This is not the most recent revision of the diff. The latest diff is revision 3. See what's changed.

1 2 3
1 2 3

  1. src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java: Loading...
  2. src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java: Loading...
  3. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java: Loading...
  4. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java: Loading...
  5. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java: Loading...
  6. src/main/protobuf/ZooKeeper.proto: Loading...
  7. src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java: Loading...
  8. src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java: Loading...
  9. src/test/java/org/apache/hadoop/hbase/master/Mocking.java: Loading...
  10. src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java: Loading...
  11. src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java: Loading...
  12. src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java: Loading...
  13. src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java: Loading...
  14. src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java: Loading...
  15. src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitLogWorker.java: Loading...
  16. src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java: Loading...
  17. src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java: Loading...
  18. src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java: Loading...
  19. src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java: Loading...
This diff has been split across 2 pages: < 1 2
src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java
Revision f9575af New Change
[20] 18 lines
[+20]
19

    
   
19

   
20
import java.io.IOException;
20
import java.io.IOException;
21

    
   
21

   
22
import org.apache.hadoop.classification.InterfaceAudience;
22
import org.apache.hadoop.classification.InterfaceAudience;
23
import org.apache.hadoop.hbase.Abortable;
23
import org.apache.hadoop.hbase.Abortable;

    
   
24
import org.apache.hadoop.hbase.DeserializationException;
24
import org.apache.hadoop.hbase.ServerName;
25
import org.apache.hadoop.hbase.ServerName;
25
import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
26
import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
26
import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos;
27
import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos;
27
import org.apache.hadoop.hbase.protobuf.generated.ZooKeeperProtos;
28
import org.apache.hadoop.hbase.protobuf.generated.ZooKeeperProtos;
28
import org.apache.zookeeper.KeeperException;
29
import org.apache.zookeeper.KeeperException;
[+20] [20] 49 lines
[+20] [+] public ServerName getMasterAddress() {
78
   *
79
   *
79
   * @param refresh whether to refresh the data by calling ZK directly.
80
   * @param refresh whether to refresh the data by calling ZK directly.
80
   * @return Server name or null if timed out.
81
   * @return Server name or null if timed out.
81
   */
82
   */
82
  public ServerName getMasterAddress(final boolean refresh) {
83
  public ServerName getMasterAddress(final boolean refresh) {
83
    return ZKUtil.znodeContentToServerName(super.getData(refresh));
84
    try {

    
   
85
      return ServerName.parseFrom(super.getData(refresh));

    
   
86
    } catch (DeserializationException e) {

    
   
87
      LOG.warn("Failed parse", e);

    
   
88
      return null;

    
   
89
    }
84
  }
90
  }
85

    
   
91

   
86
  /**
92
  /**
87
   * Get master address.
93
   * Get master address.
88
   * Use this instead of {@link #getMasterAddress()} if you do not have an
94
   * Use this instead of {@link #getMasterAddress()} if you do not have an
[+20] [20] 8 lines
[+20] public ServerName getMasterAddress() {
97
  throws KeeperException, IOException {
103
  throws KeeperException, IOException {
98
    byte [] data = ZKUtil.getData(zkw, zkw.getMasterAddressZNode());
104
    byte [] data = ZKUtil.getData(zkw, zkw.getMasterAddressZNode());
99
    if (data == null){
105
    if (data == null){
100
      throw new IOException("Can't get master address from ZooKeeper; znode data == null");
106
      throw new IOException("Can't get master address from ZooKeeper; znode data == null");
101
    }
107
    }
102
    return ZKUtil.znodeContentToServerName(data);
108
    try {

    
   
109
      return ServerName.parseFrom(data);

    
   
110
    } catch (DeserializationException e) {

    
   
111
      KeeperException ke = new KeeperException.DataInconsistencyException();

    
   
112
      ke.initCause(e);

    
   
113
      throw ke;

    
   
114
    }
103
  }
115
  }
104

    
   
116

   
105
  /**
117
  /**
106
   * Set master address into the <code>master</code> znode or into the backup
118
   * Set master address into the <code>master</code> znode or into the backup
107
   * subdirectory of backup masters; switch off the passed in <code>znode</code>
119
   * subdirectory of backup masters; switch off the passed in <code>znode</code>
[+20] [20] 6 lines
[+20] public ServerName getMasterAddress() {
114
   * @throws KeeperException
126
   * @throws KeeperException
115
   */
127
   */
116
  public static boolean setMasterAddress(final ZooKeeperWatcher zkw,
128
  public static boolean setMasterAddress(final ZooKeeperWatcher zkw,
117
      final String znode, final ServerName master)
129
      final String znode, final ServerName master)
118
  throws KeeperException {
130
  throws KeeperException {
119
    return ZKUtil.createEphemeralNodeAndWatch(zkw, znode, getZNodeData(master));
131
    return ZKUtil.createEphemeralNodeAndWatch(zkw, znode, toByteArray(master));
120
  }
132
  }
121

    
   
133

   
122
  /**
134
  /**
123
   * Check if there is a master available.
135
   * Check if there is a master available.
124
   * @return true if there is a master set, false if not.
136
   * @return true if there is a master set, false if not.
[+20] [20] 5 lines
[+20] [+] public boolean hasMaster() {
130
  /**
142
  /**
131
   * @param sn
143
   * @param sn
132
   * @return Content of the master znode as a serialized pb with the pb
144
   * @return Content of the master znode as a serialized pb with the pb
133
   * magic as prefix.
145
   * magic as prefix.
134
   */
146
   */
135
   static byte [] getZNodeData(final ServerName sn) {
147
   static byte [] toByteArray(final ServerName sn) {
136
     ZooKeeperProtos.Master.Builder mbuilder = ZooKeeperProtos.Master.newBuilder();
148
     ZooKeeperProtos.Master.Builder mbuilder = ZooKeeperProtos.Master.newBuilder();
137
     HBaseProtos.ServerName.Builder snbuilder = HBaseProtos.ServerName.newBuilder();
149
     HBaseProtos.ServerName.Builder snbuilder = HBaseProtos.ServerName.newBuilder();
138
     snbuilder.setHostName(sn.getHostname());
150
     snbuilder.setHostName(sn.getHostname());
139
     snbuilder.setPort(sn.getPort());
151
     snbuilder.setPort(sn.getPort());
140
     snbuilder.setStartCode(sn.getStartcode());
152
     snbuilder.setStartCode(sn.getStartcode());
141
     mbuilder.setMaster(snbuilder.build());
153
     mbuilder.setMaster(snbuilder.build());
142
     return ProtobufUtil.prependPBMagic(mbuilder.build().toByteArray());
154
     return ProtobufUtil.prependPBMagic(mbuilder.build().toByteArray());
143
   }
155
   }
144
}
156
}
src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
Revision babde80 New Change
 
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java
Revision e94b672 New Change
 
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
Revision 30d7fe9 New Change
 
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
Revision 46a6fde New Change
 
src/main/protobuf/ZooKeeper.proto
Revision 961ab65 New Change
 
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
Revision c3a1889 New Change
 
src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
Revision d496d48 New Change
 
src/test/java/org/apache/hadoop/hbase/master/Mocking.java
Revision 676d6bb New Change
 
src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
Revision 36046f8 New Change
 
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
Revision 2669876 New Change
 
src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
Revision 14cdb90 New Change
 
src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java
Revision f8029ba New Change
 
src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java
Revision 0f7d54e New Change
 
src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitLogWorker.java
Revision 26b9865 New Change
 
src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
Revision 75b5aea New Change
 
src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java
Revision 07f8fc4 New Change
 
src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java
Revision 55a8c4a New Change
 
src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
Revision 4314572 New Change
 
  1. src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java: Loading...
  2. src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java: Loading...
  3. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java: Loading...
  4. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java: Loading...
  5. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java: Loading...
  6. src/main/protobuf/ZooKeeper.proto: Loading...
  7. src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java: Loading...
  8. src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java: Loading...
  9. src/test/java/org/apache/hadoop/hbase/master/Mocking.java: Loading...
  10. src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java: Loading...
  11. src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java: Loading...
  12. src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java: Loading...
  13. src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java: Loading...
  14. src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java: Loading...
  15. src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitLogWorker.java: Loading...
  16. src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java: Loading...
  17. src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java: Loading...
  18. src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java: Loading...
  19. src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java: Loading...
This diff has been split across 2 pages: < 1 2