Flume-1541: Implement a SinkSelector for LoadBalancingSinkProcessor that includes failover mechanics
Review Request #6939 - Created Sept. 6, 2012 and submitted
Added a callback to the SinkSelector interface to inform selectors of sink failures, adding a noop stub to the abstract source so existing selectors don't need changes. The new selector maintains counts of sequential failures as well as times for intended recovery and last failure time, which it uses to decide whether or not a sink should be added to the returned iterator. The iterator is generated in such a way that the round robin remains balanced.
Added 3 new tests to verify that dynamic rebalancing is done, that the sink becomes available again after timeout, and that the timeout increases with sequential failures All tests pass
Posted (Sept. 6, 2012, 6:14 a.m.)
Awesome... we definitely need exponential backoff on the load balancing sink processor. Only thing is, I wonder if we shouldn't just add it to the two existing selectors instead of implement new ones that essentially do the same thing. How about modifying the constructors to accept a boolean (backoffOnFailure) instead? Other than that it looks good, I saw only a couple minor edge case issues noted below.
Review request changed
Updated (Sept. 6, 2012, 7:55 a.m.)
Fixed the shift rollover Abstracted out the backoff mechanism to avoid code repetition between roundrobin and random. Added RANDOM_BACKOFF selector along with test for it(because of its random nature there is a negligible chance(something like 1/(2^49) that it fails if no random events are sent to the channel to be backed off). I retained the old implementations. If someone else wants merge the plain and backoff versions and switch modes between them with a flag, they're welcome to... I'd prefer to avoid the confusion. Added documents
Posted (Sept. 6, 2012, 8:07 a.m.)
Looks good. One thing, regarding the configuration "max_backoff_milis". Let's give it 2 Ls, and change it to use either dashes "max-backoff-millis" or camel-caps "maxBackoffMillis" since no other config options in Flume use underscores that I know of. Also, if you can strip the EOL whitespace that would be great.
Review request changed
Updated (Sept. 6, 2012, 9:26 a.m.)
- changed from pending to submitted