Review Board 1.7.22


Doc update for FLUME-2015, FLUME-1992, FLUME-1972 and FLUME-1782.

Review Request #10804 - Created April 26, 2013 and updated

Edward Sargisson
Reviewers
Flume
flume-git
Documentation for all currently known issues:
* Description of index name rolling
* required elasticsearch and lucene core jars
* correct heading level.
mvn site run and resulting FlumeUserGuide.html checked.

Diff revision 1 (Latest)

  1. flume-ng-doc/sphinx/FlumeUserGuide.rst: Loading...
flume-ng-doc/sphinx/FlumeUserGuide.rst
Revision 38f2205 New Change
[20] 1745 lines
[+20]
1746
  a1.sinks.k1.columnFamily = bar_cf
1746
  a1.sinks.k1.columnFamily = bar_cf
1747
  a1.sinks.k1.serializer = org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
1747
  a1.sinks.k1.serializer = org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
1748
  a1.sinks.k1.channel = c1
1748
  a1.sinks.k1.channel = c1
1749

    
   
1749

   
1750
ElasticSearchSink
1750
ElasticSearchSink
1751
'''''''''''''''''
1751
~~~~~~~~~~~~~~~~~

    
   
1752

   

    
   
1753
This sink writes data to an elasticsearch cluster. By default, events will be written so that the `Kibana <http://kibana.org>`_ graphical interface

    
   
1754
can display them - just as if `logstash <https://logstash.net>`_ wrote them. 

    
   
1755

   

    
   
1756
The elasticsearch and lucene-core jars required for your environment must be placed in the lib directory of the Apache Flume installation. 

    
   
1757
Elasticsearch requires that the major version of the client JAR match that of the server and that both are running the same minor version

    
   
1758
of the JVM. SerializationExceptions will appear if this is incorrect. To 

    
   
1759
select the required version first determine the version of elasticsearch and the JVM version the target cluster is running. Then select an elasticsearch client

    
   
1760
library which matches the major version. A 0.19.x client can talk to a 0.19.x cluster; 0.20.x can talk to 0.20.x and 0.90.x can talk to 0.90.x. Once the

    
   
1761
elasticsearch version has been determined then read the pom.xml file to determine the correct lucene-core JAR version to use. The Flume agent

    
   
1762
which is running the ElasticSearchSink should also match the JVM the target cluster is running down to the minor version.

    
   
1763

   

    
   
1764
Events will be written to a new index every day. The name will be <indexName>-yyyy-MM-dd where <indexName> is the indexName parameter. The sink

    
   
1765
will start writing to a new index at midnight UTC.

    
   
1766

   

    
   
1767
Events are serialized for elasticsearch by the ElasticSearchLogStashEventSerializer by default. This behaviour can be

    
   
1768
overridden with the serializer parameter. This parameter accepts implementations of org.apache.flume.sink.elasticsearch.ElasticSearchEventSerializer

    
   
1769
or org.apache.flume.sink.elasticsearch.ElasticSearchIndexRequestBuilderFactory. Implementing ElasticSearchEventSerializer is deprecated in favour of

    
   
1770
the more powerful ElasticSearchIndexRequestBuilderFactory.
1752

    
   
1771

   
1753
This sink writes data to ElasticSearch. A class implementing

   
1754
ElasticSearchEventSerializer which is specified by the configuration is used to convert the events into

   
1755
XContentBuilder which detail the fields and mappings which will be indexed. These are then then written

   
1756
to ElasticSearch. The sink will generate an index per day allowing easier management instead of dealing with

   
1757
a single large index

   
1758
The type is the FQCN: org.apache.flume.sink.elasticsearch.ElasticSearchSink
1772
The type is the FQCN: org.apache.flume.sink.elasticsearch.ElasticSearchSink
1759
Required properties are in **bold**.
1773
Required properties are in **bold**.
1760

    
   
1774

   
1761
================  ==================================================================  =======================================================================================================
1775
================  ======================================================================== =======================================================================================================
1762
Property Name     Default                                                             Description
1776
Property Name     Default                                                                  Description
1763
================  ==================================================================  =======================================================================================================
1777
================  ======================================================================== =======================================================================================================
1764
**channel**       --
1778
**channel**       --
1765
**type**          --                                                                  The component type name, needs to be ``org.apache.flume.sink.elasticsearch.ElasticSearchSink``
1779
**type**          --                                                                       The component type name, needs to be ``org.apache.flume.sink.elasticsearch.ElasticSearchSink``
1766
**hostNames**     --                                                                  Comma separated list of hostname:port, if the port is not present the default port '9300' will be used
1780
**hostNames**     --                                                                       Comma separated list of hostname:port, if the port is not present the default port '9300' will be used
1767
indexName         flume                                                               The name of the index which the date will be appended to. Example 'flume' -> 'flume-yyyy-MM-dd'
1781
indexName         flume                                                                    The name of the index which the date will be appended to. Example 'flume' -> 'flume-yyyy-MM-dd'
1768
indexType         logs                                                                The type to index the document to, defaults to 'log'
1782
indexType         logs                                                                     The type to index the document to, defaults to 'log'
1769
clusterName       elasticsearch                                                       Name of the ElasticSearch cluster to connect to
1783
clusterName       elasticsearch                                                            Name of the ElasticSearch cluster to connect to
1770
batchSize         100                                                                 Number of events to be written per txn.
1784
batchSize         100                                                                      Number of events to be written per txn.
1771
ttl               --                                                                  TTL in days, when set will cause the expired documents to be deleted automatically,
1785
ttl               --                                                                       TTL in days, when set will cause the expired documents to be deleted automatically,
1772
                                                                                      if not set documents will never be automatically deleted
1786
                                                                                           if not set documents will never be automatically deleted
1773
serializer        org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
1787
serializer        org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer The ElasticSearchIndexRequestBuilderFactory or ElasticSearchEventSerializer to use. Implementations of
1774
serializer.*      --                                                                  Properties to be passed to the serializer.
1788
                                                                                           either class are accepted but ElasticSearchIndexRequestBuilderFactory is preferred.
1775
================  ==================================================================  =======================================================================================================
1789
serializer.*      --                                                                       Properties to be passed to the serializer.

    
   
1790
================  ======================================================================== =======================================================================================================
1776

    
   
1791

   
1777
Example for agent named a1:
1792
Example for agent named a1:
1778

    
   
1793

   
1779
.. code-block:: properties
1794
.. code-block:: properties
1780

    
   
1795

   
[+20] [20] 1245 lines
  1. flume-ng-doc/sphinx/FlumeUserGuide.rst: Loading...