Review Board 1.7.22


FLUME-1685: ExecSource shouldn't die if the channel is full

Review Request #9003 - Created Jan. 17, 2013 and updated

Steve Hoffman
FLUME-1685
Reviewers
Flume
flume-git
Rather than kill the source outright when a channel fills, the channel should throw away the event (since there is nowhere to put it).  A basic back-off mechanism is also implemented so retries get spaced out the longer the channel remains full up to a configurable limit.

Patches are attached to FLUME-1685 including a backport to the 1.2 branch (which I use with Cloudera's cdh3u5 distribution)
A unit test was created to test the full channel condition.
This patch has been running in our production environment for 2 months and has successfully recovered several times from full channel events.
Review request changed
Updated (Jan. 17, 2013, 3:08 p.m.)
  • changed from Flume-1685: ExecSource shouldn't die if the channel is full to FLUME-1685: ExecSource shouldn't die if the channel is full
Just capitalized the FLUME in the title
Posted (Jan. 17, 2013, 3:12 p.m.)

   

  
.gitignore (Diff revision 1)
 
 
I added these other intellij files as ignores since you already included the iml files and we wouldn't want this committed accidentally.
flume-ng-core/pom.xml (Diff revision 1)
 
 
Added mocking framework for unit testing
You don't override an interface so I removed this.  IntelliJ (correctly) complains.
You don't override an interface so I removed this.  IntelliJ (correctly) complains.
Posted (Jan. 25, 2013, 11:19 p.m.)
Hi Steve,
I'm not a fan of dropping events. How about providing a configuration option here:
1. Default behavior: Upon ChannelException, hold the current Event batch in memory, back off exponentially (up to a configured MAX_BACKOFF), and retry until it succeeds. Do not read additional data from the InputStream until the put() succeeds.
2. Configuration option to drop when the channel fills up (i.e., this patch)

There is nothing in Flume today that knowingly throws away data by default.