Review Board 1.7.22


HBASE-2214 per scan max buffersize

Review Request #4726 - Created April 15, 2012 and updated

ferdy
0.94
HBASE-2214
Reviewers
hbase
tedyu
hbase
HBASE-2214 per scan max buffersize.
It works when running this test:



    new HBaseTestingUtility(conf).startMiniCluster();
 
    HBaseAdmin admin = new HBaseAdmin(conf);
    if (!admin.tableExists("test")) {
      HTableDescriptor tableDesc = new HTableDescriptor("test");
      tableDesc.addFamily(new HColumnDescriptor("fam"));
      admin.createTable(tableDesc);
    }
    
    
    HTable table = new HTable(conf, "test");
    Put put; 
    
    put = new Put(Bytes.toBytes("row1"));
    put.add(Bytes.toBytes("fam"),Bytes.toBytes("qual1"),Bytes.toBytes("val1"));
    table.put(put);
    
    put = new Put(Bytes.toBytes("row2"));
    put.add(Bytes.toBytes("fam"),Bytes.toBytes("qual2"),Bytes.toBytes("val2"));
    table.put(put);
    
    put = new Put(Bytes.toBytes("row3"));
    put.add(Bytes.toBytes("fam"),Bytes.toBytes("qual3"),Bytes.toBytes("val3"));
    table.put(put);
    
    table.flushCommits();
    //put a logging statement to ClientScanner#next() to see the effect.
    {
      System.out.println("returns all rows at once because of the caching");
      Scan scan = new Scan();
      scan.setCaching(100);
      ResultScanner scanner = table.getScanner(scan);
      scanner.next(100);
    }
    {
      System.out.println("returns one row at a time because of the maxResultSize");
      Scan scan = new Scan();
      scan.setCaching(100);
      scan.setMaxResultSize(1);
      ResultScanner scanner = table.getScanner(scan);
      scanner.next(100);
    }



See output:

returns all rows at once because of the caching
2012-04-25 22:18:47,494 DEBUG [main] client.ClientScanner(94): Creating scanner over test starting at key ''
2012-04-25 22:18:47,494 DEBUG [main] client.ClientScanner(206): Advancing internal scanner to startKey at ''
2012-04-25 22:18:47,499 DEBUG [main] client.ClientScanner(323): Rows returned 3
2012-04-25 22:18:47,502 DEBUG [main] client.ClientScanner(193): Finished with scanning at {NAME => 'test,,1335385126388.ed23a82f3d6ca2eab571918843796259.', STARTKEY => '', ENDKEY => '', ENCODED => ed23a82f3d6ca2eab571918843796259,}
returns one row at a time because of the maxResultSize
2012-04-25 22:18:47,504 DEBUG [main] client.ClientScanner(94): Creating scanner over test starting at key ''
2012-04-25 22:18:47,505 DEBUG [main] client.ClientScanner(206): Advancing internal scanner to startKey at ''
2012-04-25 22:18:47,514 DEBUG [main] client.ClientScanner(323): Rows returned 1
2012-04-25 22:18:47,517 DEBUG [main] client.ClientScanner(323): Rows returned 1
2012-04-25 22:18:47,522 DEBUG [main] client.ClientScanner(323): Rows returned 1

Changes between revision 1 and 2

1 2 3 4 5
1 2 3 4 5

  1. /src/main/java/org/apache/hadoop/hbase/client/Scan.java: Loading...
  2. /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java: Loading...
/src/main/java/org/apache/hadoop/hbase/client/Scan.java
Diff Revision 1 Diff Revision 2
This file contains only whitespace changes.
[20] 50 lines
[+20]
51
 * iterate over all rows.
51
 * iterate over all rows.
52
 * <p>
52
 * <p>
53
 * To scan everything for each row, instantiate a Scan object.
53
 * To scan everything for each row, instantiate a Scan object.
54
 * <p>
54
 * <p>
55
 * To modify scanner caching for just this scan, use {@link #setCaching(int) setCaching}.
55
 * To modify scanner caching for just this scan, use {@link #setCaching(int) setCaching}.
56
 * If caching is NOT set, we will use the caching value of the hosting {@link HTable}.  See 
56
 * If caching is NOT set, we will use the caching value of the hosting {@link HTable}.  See
57
 * {@link HTable#setScannerCaching(int)}. In addition to row caching, it is possible to specify a 
57
 * {@link HTable#setScannerCaching(int)}. In addition to row caching, it is possible to specify a
58
 * maximum buffer size, using {@link #setMaxBufferSize(long) setMaxBufferSize}. When both are used, 
58
 * maximum buffer size, using {@link #setMaxBufferSize(long) setMaxBufferSize}. When both are used,
59
 * single server requests are limited by either number of rows or maximum buffer size, whichever 
59
 * single server requests are limited by either number of rows or maximum buffer size, whichever
60
 * limit comes first. 
60
 * limit comes first.
61
 * <p>
61
 * <p>
62
 * To further define the scope of what to get when scanning, perform additional
62
 * To further define the scope of what to get when scanning, perform additional
63
 * methods as outlined below.
63
 * methods as outlined below.
64
 * <p>
64
 * <p>
65
 * To get all columns from specific families, execute {@link #addFamily(byte[]) addFamily}
65
 * To get all columns from specific families, execute {@link #addFamily(byte[]) addFamily}
[+20] [20] 262 lines
[+20] [+] public void setBatch(int batch) {
328
   * @param caching the number of rows for caching
328
   * @param caching the number of rows for caching
329
   */
329
   */
330
  public void setCaching(int caching) {
330
  public void setCaching(int caching) {
331
    this.caching = caching;
331
    this.caching = caching;
332
  }
332
  }
333
  
333

   
334
  /**
334
  /**
335
   * @return the maximum buffer size in bytes. See {@link #setMaxBufferSize(long)}
335
   * @return the maximum buffer size in bytes. See {@link #setMaxBufferSize(long)}
336
   */
336
   */
337
  public long getMaxBufferSize() {
337
  public long getMaxBufferSize() {
338
    return maxBufferSize;
338
    return maxBufferSize;
339
  }
339
  }
340
  
340

   
341
  /**
341
  /**
342
   * Set the maximum buffer size. The default is -1; this means that no specific
342
   * Set the maximum buffer size. The default is -1; this means that no specific
343
   * maximum buffer size will be set for this scan, and the global configured
343
   * maximum buffer size will be set for this scan, and the global configured
344
   * value will be used instead. (Defaults to unlimited). 
344
   * value will be used instead. (Defaults to unlimited).
345
   * 
345
   *
346
   * @param maxBufferSize The maximum buffer size in bytes.
346
   * @param maxBufferSize The maximum buffer size in bytes.
347
   */
347
   */
348
  public void setMaxBufferSize(long maxBufferSize) {
348
  public void setMaxBufferSize(long maxBufferSize) {
349
    this.maxBufferSize = maxBufferSize;
349
    this.maxBufferSize = maxBufferSize;
350
  }
350
  }
[+20] [20] 349 lines
/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Diff Revision 1 Diff Revision 2
 
  1. /src/main/java/org/apache/hadoop/hbase/client/Scan.java: Loading...
  2. /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java: Loading...