Review Board 1.7.22


Doc Patch Review Request for FLUME-2065 and FLUME-2231

Review Request #15195 - Created Nov. 2, 2013 and updated

Ashish Paliwal
FLUME-2065, FLUME-2231
Reviewers
Flume
flume-git
Review Request for FLUME-2065
# Update config in Regex Extractor Interceptor to use a1 instead of agent to be in sync with rest of User Guide

FLUME-2231
# Added explicit mention of Ganglia Server for hosts description
NA, Doc patch

Diff revision 1 (Latest)

  1. flume-ng-doc/sphinx/FlumeUserGuide.rst: Loading...
flume-ng-doc/sphinx/FlumeUserGuide.rst
Revision 3a3038c New Change
[20] 2741 lines
[+20]
2742

    
   
2742

   
2743
If the Flume event body contained ``1:2:3.4foobar5`` and the following configuration was used
2743
If the Flume event body contained ``1:2:3.4foobar5`` and the following configuration was used
2744

    
   
2744

   
2745
.. code-block:: properties
2745
.. code-block:: properties
2746

    
   
2746

   
2747
  agent.sources.r1.interceptors.i1.regex = (\\d):(\\d):(\\d)
2747
  a1.sources.r1.interceptors.i1.regex = (\\d):(\\d):(\\d)
2748
  agent.sources.r1.interceptors.i1.serializers = s1 s2 s3
2748
  a1.sources.r1.interceptors.i1.serializers = s1 s2 s3
2749
  agent.sources.r1.interceptors.i1.serializers.s1.name = one
2749
  a1.sources.r1.interceptors.i1.serializers.s1.name = one
2750
  agent.sources.r1.interceptors.i1.serializers.s2.name = two
2750
  a1.sources.r1.interceptors.i1.serializers.s2.name = two
2751
  agent.sources.r1.interceptors.i1.serializers.s3.name = three
2751
  a1.sources.r1.interceptors.i1.serializers.s3.name = three
2752

    
   
2752

   
2753
The extracted event will contain the same body but the following headers will have been added ``one=>1, two=>2, three=>3``
2753
The extracted event will contain the same body but the following headers will have been added ``one=>1, two=>2, three=>3``
2754

    
   
2754

   
2755
Example 2:
2755
Example 2:
2756
~~~~~~~~~~
2756
~~~~~~~~~~
2757

    
   
2757

   
2758
If the Flume event body contained ``2012-10-18 18:47:57,614 some log line`` and the following configuration was used
2758
If the Flume event body contained ``2012-10-18 18:47:57,614 some log line`` and the following configuration was used
2759

    
   
2759

   
2760
.. code-block:: properties
2760
.. code-block:: properties
2761

    
   
2761

   
2762
  agent.sources.r1.interceptors.i1.regex = ^(?:\\n)?(\\d\\d\\d\\d-\\d\\d-\\d\\d\\s\\d\\d:\\d\\d)
2762
  a1.sources.r1.interceptors.i1.regex = ^(?:\\n)?(\\d\\d\\d\\d-\\d\\d-\\d\\d\\s\\d\\d:\\d\\d)
2763
  agent.sources.r1.interceptors.i1.serializers = s1
2763
  a1.sources.r1.interceptors.i1.serializers = s1
2764
  agent.sources.r1.interceptors.i1.serializers.s1.type = org.apache.flume.interceptor.RegexExtractorInterceptorMillisSerializer
2764
  a1.sources.r1.interceptors.i1.serializers.s1.type = org.apache.flume.interceptor.RegexExtractorInterceptorMillisSerializer
2765
  agent.sources.r1.interceptors.i1.serializers.s1.name = timestamp
2765
  a1.sources.r1.interceptors.i1.serializers.s1.name = timestamp
2766
  agent.sources.r1.interceptors.i1.serializers.s1.pattern = yyyy-MM-dd HH:mm
2766
  a1.sources.r1.interceptors.i1.serializers.s1.pattern = yyyy-MM-dd HH:mm
2767

    
   
2767

   
2768
the extracted event will contain the same body but the following headers will have been added ``timestamp=>1350611220000``
2768
the extracted event will contain the same body but the following headers will have been added ``timestamp=>1350611220000``
2769

    
   
2769

   
2770
Flume Properties
2770
Flume Properties
2771
----------------
2771
----------------
[+20] [20] 182 lines
[+20]
2954

    
   
2954

   
2955
=======================  =======  =====================================================================================
2955
=======================  =======  =====================================================================================
2956
Property Name            Default  Description
2956
Property Name            Default  Description
2957
=======================  =======  =====================================================================================
2957
=======================  =======  =====================================================================================
2958
**type**                 --       The component type name, has to be ``ganglia``
2958
**type**                 --       The component type name, has to be ``ganglia``
2959
**hosts**                --       Comma-separated list of ``hostname:port``
2959
**hosts**                --       Comma-separated list of ``hostname:port`` of Ganglia servers
2960
pollInterval             60       Time, in seconds, between consecutive reporting to ganglia server
2960
pollInterval             60       Time, in seconds, between consecutive reporting to Ganglia server
2961
isGanglia3               false    Ganglia server version is 3. By default, Flume sends in ganglia 3.1 format
2961
isGanglia3               false    Ganglia server version is 3. By default, Flume sends in Ganglia 3.1 format
2962
=======================  =======  =====================================================================================
2962
=======================  =======  =====================================================================================
2963

    
   
2963

   
2964
We can start Flume with Ganglia support as follows::
2964
We can start Flume with Ganglia support as follows::
2965

    
   
2965

   
2966
  $ bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=com.example:1234,com.example2:5455
2966
  $ bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=com.example:1234,com.example2:5455
[+20] [20] 347 lines
  1. flume-ng-doc/sphinx/FlumeUserGuide.rst: Loading...