Review Board 1.7.22


Flume-1541: Implement a SinkSelector for LoadBalancingSinkProcessor that includes failover mechanics

Review Request #6939 - Created Sept. 6, 2012 and submitted

Juhani Connolly
FLUME-1541
Reviewers
Flume
flume-git
Added a callback to the SinkSelector interface to inform selectors of sink failures, adding a noop stub to the abstract source so existing selectors don't need changes.

The new selector maintains counts of sequential failures as well as times for intended recovery and last failure time, which it uses to decide whether or not a sink should be added to the returned iterator. The iterator is generated in such a way that the round robin remains balanced.
Added 3 new tests to verify that dynamic rebalancing is done, that the sink becomes available again after timeout, and that the timeout increases with sequential failures

All tests pass
flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSinkSelector.java
Revision 63397a5 New Change
[20] 60 lines
[+20] [+] public void setSinks(List<Sink> sinks) {
61
  }
61
  }
62

    
   
62

   
63
  protected List<Sink> getSinks() {
63
  protected List<Sink> getSinks() {
64
    return sinkList;
64
    return sinkList;
65
  }
65
  }

    
   
66

   

    
   
67
  @Override

    
   
68
  public void informSinkFailed(Sink failedSink) {

    
   
69
    // no-op

    
   
70
  }
66
}
71
}
flume-ng-core/src/main/java/org/apache/flume/sink/LoadBalancingSinkProcessor.java
Revision 18d4509 New Change
 
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoadBalancingSinkProcessor.java
Revision 1e9c94e New Change
 
flume-ng-doc/sphinx/FlumeUserGuide.rst
Revision ffed72b New Change
 
  1. flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSinkSelector.java: Loading...
  2. flume-ng-core/src/main/java/org/apache/flume/sink/LoadBalancingSinkProcessor.java: Loading...
  3. flume-ng-core/src/test/java/org/apache/flume/sink/TestLoadBalancingSinkProcessor.java: Loading...
  4. flume-ng-doc/sphinx/FlumeUserGuide.rst: Loading...