Review Board 1.7.22


FLUME-1821. Support configuration of hbase instances to be used in AsyncHBaseSink from flume config

Review Request #8864 - Created Jan. 7, 2013 and submitted

Hari Shreedharan
FLUME-1821
Reviewers
Flume
flume-git
AsyncHbasesink can now be configured to use zk quorum and zk root from configuration file.
All current tests pass. Added test to make sure new code works.
flume-ng-doc/sphinx/FlumeUserGuide.rst
Revision 63b8f9b New Change
[20] 1523 lines
[+20]
1524
which is specified by the configuration is used to convert the events into
1524
which is specified by the configuration is used to convert the events into
1525
HBase puts and/or increments. These puts and increments are then written
1525
HBase puts and/or increments. These puts and increments are then written
1526
to HBase. This sink provides the same consistency guarantees as HBase,
1526
to HBase. This sink provides the same consistency guarantees as HBase,
1527
which is currently row-wise atomicity. In the event of Hbase failing to
1527
which is currently row-wise atomicity. In the event of Hbase failing to
1528
write certain events, the sink will replay all events in that transaction.
1528
write certain events, the sink will replay all events in that transaction.
1529
This sink is still experimental.

   
1530
The type is the FQCN: org.apache.flume.sink.hbase.AsyncHBaseSink.
1529
The type is the FQCN: org.apache.flume.sink.hbase.AsyncHBaseSink.
1531
Required properties are in **bold**.
1530
Required properties are in **bold**.
1532

    
   
1531

   
1533
================  ============================================================  ====================================================================================
1532
================  ============================================================  ====================================================================================
1534
Property Name     Default                                                       Description
1533
Property Name     Default                                                       Description
1535
================  ============================================================  ====================================================================================
1534
================  ============================================================  ====================================================================================
1536
**channel**       --
1535
**channel**       --
1537
**type**          --                                                            The component type name, needs to be ``org.apache.flume.sink.hbase.AsyncHBaseSink``
1536
**type**          --                                                            The component type name, needs to be ``org.apache.flume.sink.hbase.AsyncHBaseSink``
1538
**table**         --                                                            The name of the table in Hbase to write to.
1537
**table**         --                                                            The name of the table in Hbase to write to.

    
   
1538
zookeeperQuorum   --                                                            The quorum spec. This is the value for the property ``hbase.zookeeper.quorum`` in hbase-site.xml

    
   
1539
znodeParent       /hbase                                                        The base path for the znode for the -ROOT- region. Value of ``zookeeper.znode.parent`` in hbase-site.xml
1539
**columnFamily**  --                                                            The column family in Hbase to write to.
1540
**columnFamily**  --                                                            The column family in Hbase to write to.
1540
batchSize         100                                                           Number of events to be written per txn.
1541
batchSize         100                                                           Number of events to be written per txn.
1541
timeout           --                                                            The length of time (in milliseconds) the sink waits for acks from hbase for
1542
timeout           --                                                            The length of time (in milliseconds) the sink waits for acks from hbase for
1542
                                                                                all events in a transaction. If no timeout is specified, the sink will wait forever.
1543
                                                                                all events in a transaction. If no timeout is specified, the sink will wait forever.
1543
serializer        org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
1544
serializer        org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
1544
serializer.*      --                                                            Properties to be passed to the serializer.
1545
serializer.*      --                                                            Properties to be passed to the serializer.
1545
================  ============================================================  ====================================================================================
1546
================  ============================================================  ====================================================================================
1546

    
   
1547

   

    
   
1548
Note that this sink takes the Zookeeper Quorum and parent znode information in

    
   
1549
the configuration. ``znodeParent`` is not mandatory, if not provided "/hbase" is

    
   
1550
used for this parameter.

    
   
1551

   

    
   
1552
If these are not provided in the configuration, then the sink

    
   
1553
will read this information from the first hbase-site.xml file in the classpath.

    
   
1554

   
1547
Example for agent named a1:
1555
Example for agent named a1:
1548

    
   
1556

   
1549
.. code-block:: properties
1557
.. code-block:: properties
1550

    
   
1558

   
1551
  a1.channels = c1
1559
  a1.channels = c1
[+20] [20] 1206 lines
flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java
Revision 1598f26 New Change
 
flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java
Revision 463c9c3 New Change
 
flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java
Revision c835172 New Change
 
  1. flume-ng-doc/sphinx/FlumeUserGuide.rst: Loading...
  2. flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java: Loading...
  3. flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java: Loading...
  4. flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java: Loading...