Review Board 1.7.22


HBASE-4536 - Allow CF to retain deleted rows

Review Request #2178 - Created Oct. 4, 2011 and submitted

Lars Hofhansl
trunk
HBASE-4536
Reviewers
hbase
jgray, tedyu
hbase
HBase timerange Gets and Scans allow to do "timetravel" in HBase. I.e. look at the state of the data at any point in the past, provided the data is still around.
This did not work for deletes, however. Deletes would always mask all puts in the past.
This change adds a flag that can be on HColumnDescriptor to enable retention of deleted rows.
These rows are still subject to TTL and/or VERSIONS.

This changes the following:
1. There is a new flag on HColumnDescriptor enabling that behavior.
2. Allow gets/scans with a timerange to retrieve rows hidden by a delete marker, if the timerange does not include the delete marker.
3. Do not unconditionally collect all deleted rows during a compaction.
4. Allow a "raw" Scan, which retrieves all delete markers and deleted rows.

The change is small'ish, but the logic is intricate, so please review carefully.
All tests pass now.

Changes between revision 1 and 12

1 2 3 4 5 6 7 8 9 10 11 12
1 2 3 4 5 6 7 8 9 10 11 12

  1. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java: Loading...
  2. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java: Loading...
  3. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Attributes.java: Loading...
  4. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java: Loading...
  5. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java: Loading...
  6. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java: Loading...
  7. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java: Loading...
  8. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java: Loading...
  9. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java: Loading...
  10. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java: Loading...
  11. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java: Loading...
  12. http://svn.apache.org/repos/asf/hbase/trunk/src/main/ruby/hbase/admin.rb: Loading...
  13. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java: Loading...
  14. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java: Loading...
  15. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java: Loading...
  16. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeepDeletes.java: Loading...
  17. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java: Loading...
  18. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java: Loading...
  19. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java: Loading...
  20. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java: Loading...
  21. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java: Loading...
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
Diff Revision 1 Diff Revision 12
[20] 85 lines
[+20] [+] public class HColumnDescriptor implements WritableComparable<HColumnDescriptor> {
86
  public static final String TTL = "TTL";
86
  public static final String TTL = "TTL";
87
  public static final String BLOOMFILTER = "BLOOMFILTER";
87
  public static final String BLOOMFILTER = "BLOOMFILTER";
88
  public static final String FOREVER = "FOREVER";
88
  public static final String FOREVER = "FOREVER";
89
  public static final String REPLICATION_SCOPE = "REPLICATION_SCOPE";
89
  public static final String REPLICATION_SCOPE = "REPLICATION_SCOPE";
90
  public static final String MIN_VERSIONS = "MIN_VERSIONS";
90
  public static final String MIN_VERSIONS = "MIN_VERSIONS";

    
   
91
  public static final String KEEP_DELETED_CELLS = "KEEP_DELETED_CELLS";
91

    
   
92

   
92
  /**
93
  /**
93
   * Default compression type.
94
   * Default compression type.
94
   */
95
   */
95
  public static final String DEFAULT_COMPRESSION =
96
  public static final String DEFAULT_COMPRESSION =
[+20] [20] 19 lines
[+20] public class HColumnDescriptor implements WritableComparable<HColumnDescriptor> {
115
   * Default setting for whether to serve from memory or not.
116
   * Default setting for whether to serve from memory or not.
116
   */
117
   */
117
  public static final boolean DEFAULT_IN_MEMORY = false;
118
  public static final boolean DEFAULT_IN_MEMORY = false;
118

    
   
119

   
119
  /**
120
  /**

    
   
121
   * Default setting for preventing deleted from being collected immediately.

    
   
122
   */

    
   
123
  public static final boolean DEFAULT_KEEP_DELETED = false;

    
   
124

   

    
   
125
  /**
120
   * Default setting for whether to use a block cache or not.
126
   * Default setting for whether to use a block cache or not.
121
   */
127
   */
122
  public static final boolean DEFAULT_BLOCKCACHE = true;
128
  public static final boolean DEFAULT_BLOCKCACHE = true;
123

    
   
129

   
124
  /**
130
  /**
[+20] [20] 24 lines
[+20] public class HColumnDescriptor implements WritableComparable<HColumnDescriptor> {
149
      DEFAULT_VALUES.put(COMPRESSION, DEFAULT_COMPRESSION);
155
      DEFAULT_VALUES.put(COMPRESSION, DEFAULT_COMPRESSION);
150
      DEFAULT_VALUES.put(TTL, String.valueOf(DEFAULT_TTL));
156
      DEFAULT_VALUES.put(TTL, String.valueOf(DEFAULT_TTL));
151
      DEFAULT_VALUES.put(BLOCKSIZE, String.valueOf(DEFAULT_BLOCKSIZE));
157
      DEFAULT_VALUES.put(BLOCKSIZE, String.valueOf(DEFAULT_BLOCKSIZE));
152
      DEFAULT_VALUES.put(HConstants.IN_MEMORY, String.valueOf(DEFAULT_IN_MEMORY));
158
      DEFAULT_VALUES.put(HConstants.IN_MEMORY, String.valueOf(DEFAULT_IN_MEMORY));
153
      DEFAULT_VALUES.put(BLOCKCACHE, String.valueOf(DEFAULT_BLOCKCACHE));
159
      DEFAULT_VALUES.put(BLOCKCACHE, String.valueOf(DEFAULT_BLOCKCACHE));

    
   
160
      DEFAULT_VALUES.put(KEEP_DELETED_CELLS, String.valueOf(DEFAULT_KEEP_DELETED));
154
  }
161
  }
155

    
   
162

   
156
  // Column family name
163
  // Column family name
157
  private byte [] name;
164
  private byte [] name;
158

    
   
165

   
[+20] [20] 104 lines
[+20] public class HColumnDescriptor implements WritableComparable<HColumnDescriptor> {
263
   */
270
   */
264
  public HColumnDescriptor(final byte [] familyName, final int maxVersions,
271
  public HColumnDescriptor(final byte [] familyName, final int maxVersions,
265
      final String compression, final boolean inMemory,
272
      final String compression, final boolean inMemory,
266
      final boolean blockCacheEnabled, final int blocksize,
273
      final boolean blockCacheEnabled, final int blocksize,
267
      final int timeToLive, final String bloomFilter, final int scope) {
274
      final int timeToLive, final String bloomFilter, final int scope) {
268
    this(familyName, DEFAULT_MIN_VERSIONS, maxVersions, compression, inMemory,
275
    this(familyName, DEFAULT_MIN_VERSIONS, maxVersions, DEFAULT_KEEP_DELETED,
269
        blockCacheEnabled, blocksize, timeToLive, bloomFilter, scope);
276
        compression, inMemory, blockCacheEnabled, blocksize, timeToLive,

    
   
277
        bloomFilter, scope);
270
  }
278
  }
271

    
   
279

   
272
  /**
280
  /**
273
   * Constructor
281
   * Constructor
274
   * @param familyName Column family name. Must be 'printable' -- digit or
282
   * @param familyName Column family name. Must be 'printable' -- digit or
[+20] [20] 15 lines
[+20] public class HColumnDescriptor implements WritableComparable<HColumnDescriptor> {
290
   * @throws IllegalArgumentException if passed a family name that is made of
298
   * @throws IllegalArgumentException if passed a family name that is made of
291
   * other than 'word' characters: i.e. <code>[a-zA-Z_0-9]</code> or contains
299
   * other than 'word' characters: i.e. <code>[a-zA-Z_0-9]</code> or contains
292
   * a <code>:</code>
300
   * a <code>:</code>
293
   * @throws IllegalArgumentException if the number of versions is &lt;= 0
301
   * @throws IllegalArgumentException if the number of versions is &lt;= 0
294
   */
302
   */
295
  public HColumnDescriptor(final byte [] familyName, final int minVersions,
303
  public HColumnDescriptor(final byte[] familyName, final int minVersions,
296
      final int maxVersions, final String compression, final boolean inMemory,
304
      final int maxVersions, final boolean keepDeletedRows,

    
   
305
      final String compression, final boolean inMemory,
297
      final boolean blockCacheEnabled, final int blocksize,
306
      final boolean blockCacheEnabled, final int blocksize,
298
      final int timeToLive, final String bloomFilter, final int scope) {
307
      final int timeToLive, final String bloomFilter, final int scope) {
299
    isLegalFamilyName(familyName);
308
    isLegalFamilyName(familyName);
300
    this.name = familyName;
309
    this.name = familyName;
301

    
   
310

   
302
    if (maxVersions <= 0) {
311
    if (maxVersions <= 0) {
303
      // TODO: Allow maxVersion of 0 to be the way you say "Keep all versions".
312
      // TODO: Allow maxVersion of 0 to be the way you say "Keep all versions".
304
      // Until there is support, consider 0 or < 0 -- a configuration error.
313
      // Until there is support, consider 0 or < 0 -- a configuration error.
305
      throw new IllegalArgumentException("Maximum versions must be positive");
314
      throw new IllegalArgumentException("Maximum versions must be positive");
306
    }
315
    }
307

    
   
316

   
308
    if (minVersions > maxVersions) {
317
    if (minVersions > 0) {
309
      throw new IllegalArgumentException("Minimum versions must be <= "
318
      if (timeToLive == HConstants.FOREVER) {

    
   
319
        throw new IllegalArgumentException("Minimum versions requires TTL.");

    
   
320
      }

    
   
321
      if (minVersions >= maxVersions) {

    
   
322
        throw new IllegalArgumentException("Minimum versions must be < "
310
          + "maximum versions.");
323
            + "maximum versions.");
311
    }
324
      }

    
   
325
    }
312

    
   
326

   
313
    setMaxVersions(maxVersions);
327
    setMaxVersions(maxVersions);
314
    setMinVersions(minVersions);
328
    setMinVersions(minVersions);

    
   
329
    setKeepDeletedRows(keepDeletedRows);
315
    setInMemory(inMemory);
330
    setInMemory(inMemory);
316
    setBlockCacheEnabled(blockCacheEnabled);
331
    setBlockCacheEnabled(blockCacheEnabled);
317
    setTimeToLive(timeToLive);
332
    setTimeToLive(timeToLive);
318
    setCompressionType(Compression.Algorithm.
333
    setCompressionType(Compression.Algorithm.
319
      valueOf(compression.toUpperCase()));
334
      valueOf(compression.toUpperCase()));
[+20] [20] 215 lines
[+20] [+] public boolean isInMemory() {
535
   */
550
   */
536
  public void setInMemory(boolean inMemory) {
551
  public void setInMemory(boolean inMemory) {
537
    setValue(HConstants.IN_MEMORY, Boolean.toString(inMemory));
552
    setValue(HConstants.IN_MEMORY, Boolean.toString(inMemory));
538
  }
553
  }
539

    
   
554

   

    
   
555
  public boolean getKeepDeletedCells() {

    
   
556
    String value = getValue(KEEP_DELETED_CELLS);

    
   
557
    if (value != null) {

    
   
558
      return Boolean.valueOf(value).booleanValue();

    
   
559
    }

    
   
560
    return DEFAULT_KEEP_DELETED;

    
   
561
  }

    
   
562

   

    
   
563
  /**

    
   
564
   * @param keepDeletedRows True if deleted rows should not be collected

    
   
565
   * immediately.

    
   
566
   */

    
   
567
  public void setKeepDeletedRows(boolean keepDeletedRows) {

    
   
568
    setValue(KEEP_DELETED_CELLS, Boolean.toString(keepDeletedRows));

    
   
569
  }

    
   
570

   
540
  /**
571
  /**
541
   * @return Time-to-live of cell contents, in seconds.
572
   * @return Time-to-live of cell contents, in seconds.
542
   */
573
   */
543
  public int getTimeToLive() {
574
  public int getTimeToLive() {
544
    String value = getValue(TTL);
575
    String value = getValue(TTL);
[+20] [20] 252 lines
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Attributes.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/ruby/hbase/admin.rb
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeepDeletes.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java
Diff Revision 1 Diff Revision 12
 
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
Diff Revision 1 Diff Revision 12
 
  1. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java: Loading...
  2. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/KeyValue.java: Loading...
  3. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Attributes.java: Loading...
  4. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java: Loading...
  5. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java: Loading...
  6. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java: Loading...
  7. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java: Loading...
  8. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java: Loading...
  9. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java: Loading...
  10. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java: Loading...
  11. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java: Loading...
  12. http://svn.apache.org/repos/asf/hbase/trunk/src/main/ruby/hbase/admin.rb: Loading...
  13. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java: Loading...
  14. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java: Loading...
  15. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java: Loading...
  16. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestKeepDeletes.java: Loading...
  17. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java: Loading...
  18. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java: Loading...
  19. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java: Loading...
  20. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java: Loading...
  21. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java: Loading...