Review Board 1.7.22


hbase-5453 Switch on-disk formats (reference files, HFile meta fields, etc) to PB

Review Request #5130 - Created May 15, 2012 and updated

Michael Stack
hbase-5453
Reviewers
hbase
hbase-git
A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
  New  class to hold clusterid in.
M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
  Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
  ClusterId under ZK got renamed as ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
  Hide the Reference#Range enums.  Don't let them out of this class.
  Make it so can do pb serialization.
M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
  Use new methods on Reference for getting top and bottom.
M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
  ClusterId under zk has been renamed ZKClusterId.
  Use new ClusterId class too.
M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
  Use new clusterid class.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
  Move the RegionInfo convertion up into HRegionInfo instead of here.
  Added generic toDelimitedByteArray helper.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
  Use new utility writing out .regioninfo files.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  Formatting.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
  Range in Reference is no longer public.
  Range in Reference is no longer public.
M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
  ClusterId got renamed ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
  Use new serialization utlity in HTD.
M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
  Generic method for writing dot file content.
M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
  Reference#Range is not public any more
M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
  Deprecated getHRegionInfo, etc.
D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
  Rename
A b/src/main/protobuf/ClusterId.proto
  Added file for ClusterId only since its written to fs and to zk.
A b/src/main/protobuf/FS.proto
  Protos for fs files.
M b/src/main/protobuf/ZooKeeper.proto
  Moved ClusterId out to own proto file
M b/src/main/protobuf/hbase.proto
  Added TableSchema and ColumnFamilySchema

 

Diff revision 1

This is not the most recent revision of the diff. The latest diff is revision 4. See what's changed.

1 2 3 4
1 2 3 4

  1. src/main/java/org/apache/hadoop/hbase/ClusterId.java: Loading...
  2. src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java: Loading...
  3. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java: Loading...
  4. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java: Loading...
  5. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java: Loading...
  6. src/main/java/org/apache/hadoop/hbase/io/Reference.java: Loading...
  7. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java: Loading...
  8. src/main/java/org/apache/hadoop/hbase/master/HMaster.java: Loading...
  9. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java: Loading...
  10. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java: Loading...
  11. src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java: Loading...
  12. src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java: Loading...
  13. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java: Loading...
  14. src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java: Loading...
  15. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java: Loading...
  16. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java: Loading...
  17. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java: Loading...
  18. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java: Loading...
  19. src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java: Loading...
  20. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java: Loading...
This diff has been split across 2 pages: 1 2 >
src/main/java/org/apache/hadoop/hbase/ClusterId.java
New File

    
   
1
/*

    
   
2
 * Licensed to the Apache Software Foundation (ASF) under one

    
   
3
 * or more contributor license agreements.  See the NOTICE file

    
   
4
 * distributed with this work for additional information

    
   
5
 * regarding copyright ownership.  The ASF licenses this file

    
   
6
 * to you under the Apache License, Version 2.0 (the

    
   
7
 * "License"); you may not use this file except in compliance

    
   
8
 * with the License.  You may obtain a copy of the License at

    
   
9
 *

    
   
10
 *     http://www.apache.org/licenses/LICENSE-2.0

    
   
11
 *

    
   
12
 * Unless required by applicable law or agreed to in writing, software

    
   
13
 * distributed under the License is distributed on an "AS IS" BASIS,

    
   
14
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

    
   
15
 * See the License for the specific language governing permissions and

    
   
16
 * limitations under the License.

    
   
17
 */

    
   
18

   

    
   
19
package org.apache.hadoop.hbase;

    
   
20

   

    
   
21
import java.io.BufferedInputStream;

    
   
22
import java.io.IOException;

    
   
23
import java.io.InputStream;

    
   
24
import java.util.UUID;

    
   
25

   

    
   
26
import org.apache.hadoop.classification.InterfaceAudience;

    
   
27
import org.apache.hadoop.fs.FSDataInputStream;

    
   
28
import org.apache.hadoop.hbase.protobuf.ProtobufUtil;

    
   
29
import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos;

    
   
30
import org.apache.hadoop.hbase.util.Bytes;

    
   
31

   

    
   
32
import com.google.protobuf.InvalidProtocolBufferException;

    
   
33

   

    
   
34
/**

    
   
35
 * The identifier for this cluster.

    
   
36
 * It is serialized to the filesystem and up into zookeeper.  This is a container for the id.

    
   
37
 * Also knows how to serialize and deserialize the cluster id.

    
   
38
 */

    
   
39
@InterfaceAudience.Private

    
   
40
public class ClusterId {

    
   
41
  private final String id;

    
   
42

   

    
   
43
  /**

    
   
44
   * New ClusterID.  Generates a uniqueid.

    
   
45
   */

    
   
46
  public ClusterId() {

    
   
47
    this(UUID.randomUUID().toString());

    
   
48
  }

    
   
49

   

    
   
50
  ClusterId(final String uuid) {

    
   
51
    this.id = uuid;

    
   
52
  }

    
   
53

   

    
   
54
  /**

    
   
55
   * @return The clusterid serialized using pb w/ pb magic prefix

    
   
56
   */

    
   
57
  public byte [] toByteArray() {

    
   
58
    return ProtobufUtil.prependPBMagic(convert().toByteArray());

    
   
59
  }

    
   
60

   

    
   
61
  /**

    
   
62
   * @param bytes A pb serialized {@link ClusterId} instance with pb magic prefix

    
   
63
   * @return An instance of {@link ClusterId} made from <code>bytes</code>

    
   
64
   * @throws DeserializationException

    
   
65
   * @see {@link #toByteArray()}

    
   
66
   */

    
   
67
  public static ClusterId parseFrom(final byte [] bytes) throws DeserializationException {

    
   
68
    if (ProtobufUtil.isPBMagicPrefix(bytes)) {

    
   
69
      int pblen = ProtobufUtil.lengthOfPBMagic();

    
   
70
      ClusterIdProtos.ClusterId.Builder builder = ClusterIdProtos.ClusterId.newBuilder();

    
   
71
      ClusterIdProtos.ClusterId cid = null;

    
   
72
      try {

    
   
73
        cid = builder.mergeFrom(bytes, pblen, bytes.length - pblen).build();

    
   
74
      } catch (InvalidProtocolBufferException e) {

    
   
75
        throw new DeserializationException(e);

    
   
76
      }

    
   
77
      return convert(cid);

    
   
78
    } else {

    
   
79
      return new ClusterId(Bytes.toString(bytes));

    
   
80
    }

    
   
81
  }

    
   
82

   

    
   
83
  /**

    
   
84
   * Parses a {@link ClusterId} instance from the passed in stream.  Presumes the stream contains

    
   
85
   * nothing but a serialized clusterid.

    
   
86
   * @param in

    
   
87
   * @return An instance of ClusterId.

    
   
88
   * @throws IOException

    
   
89
   */

    
   
90
  public static ClusterId parseFrom(final FSDataInputStream in) throws IOException {

    
   
91
    // I need to be able to move back in the stream if this is not a pb serialization so I can

    
   
92
    // do the Writable decoding instead.

    
   
93
    InputStream is = in.markSupported()? in: new BufferedInputStream(in);

    
   
94
    int pblen = ProtobufUtil.lengthOfPBMagic();

    
   
95
    is.mark(pblen);

    
   
96
    byte [] pbuf = new byte[pblen];

    
   
97
    int read = is.read(pbuf);

    
   
98
    if (read != pblen) throw new IOException("read=" + read + ", wanted=" + pblen);

    
   
99
    if (ProtobufUtil.isPBMagicPrefix(pbuf)) {

    
   
100
      return convert(ClusterIdProtos.ClusterId.parseFrom(is));

    
   
101
    } else {

    
   
102
      // Presume Writables.  Need to reset the stream since it didn't start w/ pb.

    
   
103
      in.reset();

    
   
104
      String cid = in.readUTF();

    
   
105
      return new ClusterId(cid);

    
   
106
    }

    
   
107
  }

    
   
108

   

    
   
109
  /**

    
   
110
   * @return A pb instance to represent this instance.

    
   
111
   */

    
   
112
  ClusterIdProtos.ClusterId convert() {

    
   
113
    ClusterIdProtos.ClusterId.Builder builder = ClusterIdProtos.ClusterId.newBuilder();

    
   
114
    return builder.setClusterId(this.id).build();

    
   
115
  }

    
   
116

   

    
   
117
  /**

    
   
118
   * @param cid

    
   
119
   * @return A {@link ClusterId} made from the passed in <code>cid</code>

    
   
120
   */

    
   
121
  static ClusterId convert(final ClusterIdProtos.ClusterId cid) {

    
   
122
    return new ClusterId(cid.getClusterId());

    
   
123
  }

    
   
124
}
src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
Revision 5862f15 New Change
 
src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
Revision 8d83ff3 New Change
 
src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
Revision af89e3e New Change
 
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
Revision 5cac9af New Change
 
src/main/java/org/apache/hadoop/hbase/io/Reference.java
Revision 6360059 New Change
 
src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
Revision 9e4ada9 New Change
 
src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Revision 947ec5f New Change
 
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
Revision 5052878 New Change
 
src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
Revision ccc964e New Change
 
src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
Revision dabfbab New Change
 
src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
Revision 45cb6cf New Change
 
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java
New File
 
src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java
New File
 
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
Revision 058c006 New Change
 
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java
Revision 20c7738 New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Revision 9f16fee New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Revision 9d3898c New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
Revision 6a9f2fe New Change
 
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
Revision 5e1e16d New Change
 
  1. src/main/java/org/apache/hadoop/hbase/ClusterId.java: Loading...
  2. src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java: Loading...
  3. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java: Loading...
  4. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java: Loading...
  5. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java: Loading...
  6. src/main/java/org/apache/hadoop/hbase/io/Reference.java: Loading...
  7. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java: Loading...
  8. src/main/java/org/apache/hadoop/hbase/master/HMaster.java: Loading...
  9. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java: Loading...
  10. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java: Loading...
  11. src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java: Loading...
  12. src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java: Loading...
  13. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java: Loading...
  14. src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java: Loading...
  15. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java: Loading...
  16. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java: Loading...
  17. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java: Loading...
  18. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java: Loading...
  19. src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java: Loading...
  20. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java: Loading...
This diff has been split across 2 pages: 1 2 >