Review Board 1.7.22


FLUME-1782 Elastic Search sink does not use UTC to determine the correct index to write to

Review Request #10379 - Created April 9, 2013 and discarded

Edward Sargisson
flume-1.4
FLUME-1782
Reviewers
Flume
hshreedharan
flume-git
This change gets the timestamp from the event and uses it (in UTC) to determine the name of the index to write to. This is required to match the behaviour of Logstash so that Kibana can find the log events.
The previous code would use the current time and would do it in the timezone of the Flume agent's host.
All unit tests and integration tests pass. A snapshot using commit 5b9d31f1ad228 and the patch for flume-1972 has passed our internal integration tests using customisations.
flume-ng-doc/sphinx/FlumeUserGuide.rst
Revision 693c0d7 New Change
[20] 1745 lines
[+20]
1746
  a1.sinks.k1.columnFamily = bar_cf
1746
  a1.sinks.k1.columnFamily = bar_cf
1747
  a1.sinks.k1.serializer = org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
1747
  a1.sinks.k1.serializer = org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
1748
  a1.sinks.k1.channel = c1
1748
  a1.sinks.k1.channel = c1
1749

    
   
1749

   
1750
ElasticSearchSink
1750
ElasticSearchSink
1751
'''''''''''''''''
1751
~~~~~~~~~~~~~~~~~
1752

    
   
1752

   
1753
This sink writes data to ElasticSearch. A class implementing
1753
This sink writes data to elasticsearch. A class implementing
1754
ElasticSearchEventSerializer which is specified by the configuration is used to convert the events into
1754
ElasticSearchEventSerializer which is specified by the configuration is used to convert the events into
1755
XContentBuilder which detail the fields and mappings which will be indexed. These are then then written
1755
XContentBuilder which detail the fields and mappings which will be indexed. These are then then written
1756
to ElasticSearch. The sink will generate an index per day allowing easier management instead of dealing with
1756
to ElasticSearch. The sink will generate an index per day allowing easier management instead of dealing with
1757
a single large index
1757
a single large index. Note that the day is the UTC day. There is no provision for rolling the index in different timezones.
1758
The type is the FQCN: org.apache.flume.sink.elasticsearch.ElasticSearchSink
1758
The type is the FQCN: org.apache.flume.sink.elasticsearch.ElasticSearchSink
1759
Required properties are in **bold**.
1759
Required properties are in **bold**.
1760

    
   
1760

   
1761
================  ==================================================================  =======================================================================================================
1761
================  ==================================================================  =======================================================================================================
1762
Property Name     Default                                                             Description
1762
Property Name     Default                                                             Description
[+20] [20] 1263 lines
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java
Revision 1b3db14 New Change
 
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java
Revision 94b95b1 New Change
 
  1. flume-ng-doc/sphinx/FlumeUserGuide.rst: Loading...
  2. flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java: Loading...
  3. flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java: Loading...