Review Board 1.7.22


FLUME-1972 ElasticSearchSink now allows the id of the document to be provided by a custom class

Review Request #10380 - Created April 9, 2013 and discarded

Edward Sargisson
flume-1.4
FLUME-1972
Reviewers
Flume
hshreedharan
flume-git
Elasticsearch allows clients to specify the id of the newly added document - and will generate one by default if none is supplied. If the user knows that the incoming event has a unique id then that id can be used as the elasticsearch document id. This change enhances the ElasticSearchSink to allow users to provide a custom class to decide what the id should be.
All unit tests and integration tests pass. A snapshot using commit 5b9d31f1ad228 and the patch for flume-1972 has passed our internal integration tests using customisations.
flume-ng-doc/sphinx/FlumeUserGuide.rst
Revision 38f2205 New Change
[20] 1769 lines
[+20]
1770
batchSize         100                                                                 Number of events to be written per txn.
1770
batchSize         100                                                                 Number of events to be written per txn.
1771
ttl               --                                                                  TTL in days, when set will cause the expired documents to be deleted automatically,
1771
ttl               --                                                                  TTL in days, when set will cause the expired documents to be deleted automatically,
1772
                                                                                      if not set documents will never be automatically deleted
1772
                                                                                      if not set documents will never be automatically deleted
1773
serializer        org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
1773
serializer        org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
1774
serializer.*      --                                                                  Properties to be passed to the serializer.
1774
serializer.*      --                                                                  Properties to be passed to the serializer.

    
   
1775
idProvider        org.apache.flume.sink.elasticsearch.ElasticSearchNullIdProvider     The class to use to calculate the id to give the document.

    
   
1776
idProvider.*      --                                                                  Properties to be passed to the idProvider.
1775
================  ==================================================================  =======================================================================================================
1777
================  ==================================================================  =======================================================================================================
1776

    
   
1778

   
1777
Example for agent named a1:
1779
Example for agent named a1:
1778

    
   
1780

   
1779
.. code-block:: properties
1781
.. code-block:: properties
[+20] [20] 7 lines
[+20]
1787
  a1.sinks.k1.clusterName = foobar_cluster
1789
  a1.sinks.k1.clusterName = foobar_cluster
1788
  a1.sinks.k1.batchSize = 500
1790
  a1.sinks.k1.batchSize = 500
1789
  a1.sinks.k1.ttl = 5
1791
  a1.sinks.k1.ttl = 5
1790
  a1.sinks.k1.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
1792
  a1.sinks.k1.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
1791
  a1.sinks.k1.channel = c1
1793
  a1.sinks.k1.channel = c1

    
   
1794
  

    
   
1795
Example for agent named a2 - which uses the provided ElasticSearchExtractIdFromHeaderIdProvider class to read a header called id and use that as the document id:

    
   
1796

   

    
   
1797
.. code-block:: properties

    
   
1798

   

    
   
1799
  a2.channels = c1

    
   
1800
  a2.sinks = k1

    
   
1801
  a2.sinks.k1.type = org.apache.flume.sink.elasticsearch.ElasticSearchSink

    
   
1802
  a2.sinks.k1.hostNames = 127.0.0.1:9200,127.0.0.2:9300

    
   
1803
  a2.sinks.k1.indexName = foo_index

    
   
1804
  a2.sinks.k1.indexType = bar_type

    
   
1805
  a2.sinks.k1.clusterName = foobar_cluster

    
   
1806
  a2.sinks.k1.batchSize = 500

    
   
1807
  a2.sinks.k1.ttl = 5

    
   
1808
  a2.sinks.k1.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer

    
   
1809
  a2.sinks.k1.idProvider = com.globalrelay.castellan.flume.sink.ElasticSearchExtractIdFromHeaderIdProvider

    
   
1810
  a2.sinks.k1.channel = c1
1792

    
   
1811

   
1793
Custom Sink
1812
Custom Sink
1794
~~~~~~~~~~~
1813
~~~~~~~~~~~
1795

    
   
1814

   
1796
A custom sink is your own implementation of the Sink interface. A custom
1815
A custom sink is your own implementation of the Sink interface. A custom
[+20] [20] 1229 lines
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchExtractIdFromHeaderIdProvider.java
New File
 
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchIdProvider.java
New File
 
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchNullIdProvider.java
New File
 
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java
Revision 1b3db14 New Change
 
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSinkConstants.java
Revision 7f75e22 New Change
 
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java
Revision 2edacdc New Change
 
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/ElasticSearchExtractIdFromHeaderIdProviderTest.java
New File
 
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java
Revision 94b95b1 New Change
 
  1. flume-ng-doc/sphinx/FlumeUserGuide.rst: Loading...
  2. flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchExtractIdFromHeaderIdProvider.java: Loading...
  3. flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchIdProvider.java: Loading...
  4. flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchNullIdProvider.java: Loading...
  5. flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java: Loading...
  6. flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSinkConstants.java: Loading...
  7. flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java: Loading...
  8. flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/ElasticSearchExtractIdFromHeaderIdProviderTest.java: Loading...
  9. flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java: Loading...