Review Board 1.7.22


FLUME-1425: Create a SpoolDirectory Source and Client

Review Request #6377 - Created Aug. 4, 2012 and updated

Patrick Wendell
trunk
FlUME-1425
Reviewers
Flume
flume-git
This patch adds a spooling directory based source. The  idea is that a user can have a spool directory where files are deposited for ingestion into flume. Once ingested, the files are clearly renamed and the implementation guarantees at-least-once delivery semantics similar to those achieved within flume itself, even across failures and restarts of the JVM running the code.

This helps fill the gap for people who want a way to get reliable delivery of events into flume, but don't want to directly write their application against the flume API. They can simply drop log files off in a spooldir and let flume ingest asynchronously (using some shell scripts or other automated process).

Unlike the prior iteration, this patch implements a first-class source. It also extends the avro client to support spooling in a similar manner.
Extensive unit tests and I also built and played with this using a stub flume agent. If you look at the JIRA I have a configuration file for an agent that will print out Avro events to the command line - that's helpful when testing this.
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java
Revision da804d7 New Change
[20] 167 lines
[+20] [+] private ChannelSelectorType getKnownChannelSelector(String type) {
168
    /**
168
    /**
169
     * Syslog Udp Source
169
     * Syslog Udp Source
170
     *
170
     *
171
     * @see org.apache.flume.source.SyslogUDPSource
171
     * @see org.apache.flume.source.SyslogUDPSource
172
     */
172
     */

    
   
173
    SYSLOGUDP("org.apache.flume.conf.source.SyslogUDPSourceConfiguration"),
173

    
   
174

   
174
    SYSLOGUDP("org.apache.flume.conf.source.SyslogUDPSourceConfiguration");
175
    /**

    
   
176
     * Spool directory source

    
   
177
     *

    
   
178
     * @see org.apache.flume.source.SpoolDirectorySource

    
   
179
     */

    
   
180
    SPOOLDIR("org.apache.flume.conf.source.SpoolDirectorySourceConfiguration");
175

    
   
181

   
176
    private String srcConfigurationName;
182
    private String srcConfigurationName;
177

    
   
183

   
178
    private SourceConfigurationType(String src) {
184
    private SourceConfigurationType(String src) {
179
      this.srcConfigurationName = src;
185
      this.srcConfigurationName = src;
[+20] [20] 39 lines
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java
Revision abbbf1c New Change
 
flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
Revision 4a5ecae New Change
 
flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java
New File
 
flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java
New File
 
flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java
New File
 
flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java
New File
 
flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java
New File
 
flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java
New File
 
flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java
New File
 
flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java
New File
 
flume-ng-doc/sphinx/FlumeUserGuide.rst
Revision 953a670 New Change
 
  1. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java: Loading...
  2. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java: Loading...
  3. flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java: Loading...
  4. flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java: Loading...
  5. flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java: Loading...
  6. flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java: Loading...
  7. flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java: Loading...
  8. flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java: Loading...
  9. flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java: Loading...
  10. flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java: Loading...
  11. flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java: Loading...
  12. flume-ng-doc/sphinx/FlumeUserGuide.rst: Loading...