Review Board 1.7.22


hbase-5453 Switch on-disk formats (reference files, HFile meta fields, etc) to PB

Review Request #5130 - Created May 15, 2012 and updated

Michael Stack
hbase-5453
Reviewers
hbase
hbase-git
A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
  New  class to hold clusterid in.
M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
  Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
  ClusterId under ZK got renamed as ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
  Hide the Reference#Range enums.  Don't let them out of this class.
  Make it so can do pb serialization.
M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
  Use new methods on Reference for getting top and bottom.
M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
  ClusterId under zk has been renamed ZKClusterId.
  Use new ClusterId class too.
M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
  Use new clusterid class.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
  Move the RegionInfo convertion up into HRegionInfo instead of here.
  Added generic toDelimitedByteArray helper.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
  Use new utility writing out .regioninfo files.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  Formatting.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
  Range in Reference is no longer public.
  Range in Reference is no longer public.
M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
  ClusterId got renamed ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
  Use new serialization utlity in HTD.
M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
  Generic method for writing dot file content.
M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
  Reference#Range is not public any more
M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
  Deprecated getHRegionInfo, etc.
D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
  Rename
A b/src/main/protobuf/ClusterId.proto
  Added file for ClusterId only since its written to fs and to zk.
A b/src/main/protobuf/FS.proto
  Protos for fs files.
M b/src/main/protobuf/ZooKeeper.proto
  Moved ClusterId out to own proto file
M b/src/main/protobuf/hbase.proto
  Added TableSchema and ColumnFamilySchema

 
Review request changed
Updated (May 17, 2012, 5:58 a.m.)
Address Gregory's two review comments.
Ship it!
Posted (May 17, 2012, 9:02 p.m.)

   

  
Your logic sounds correct and I agree we should avoid having to read the file each time.

It would be cool to just see an example of the old writable vs new pb size to ensure our intuition is correct.

Worst case (let's say they happen to be the same) we could write out the new pb stuff concatenated with the old way of writing and then they are guaranteed to be different sizes.  Hopefully we don't have to do something like that, though.