Review Board 1.7.22


Revised design for Spillable Mem Channel

Review Request #12060 - Created June 24, 2013 and updated

Roshan Naik
flume-1.5
FLUME-1227
Reviewers
Flume
flume-git
Revised design for Spillable Mem Channel.
We no longer have Spillable channel config pointing to another channel (by name) as in the previous design.

Spillable Channel instead derives from FileChannel (as per https://issues.apache.org/jira/browse/FLUME-1227?focusedCommentId=13628201&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13628201)

Essence of this design:
- SC derives from File channel and maintains an in memory queue. If memory queue is full, events are sent to disk overflow (i.e. File channel).
- SC maintains a 'Drain-Order' queue (DOQ) for remembering the order in which the incoming events were interleaved between main memory and 
- Put transaction: All the elements in the putList are committed to mem queue if it has space, else written to disk (ie file channel). Head of DOQ is updated to indicate where the elements put.
- Take transaction : Tail of DOQ is consulted to determine whether the next set of events are to be taken from the memory queue or from disk overflow. DOQ's tail is updated after events are taken out.


SC Configuration:
- Accepts all the File Channel settings
- Introduces one additional setting: 'memoryCapacity' which indicates the number of items it can hold in memory
 

Sample config ...

a1.channels = c1
a1.sinks = logger
a1.sources = src

a1.sources.src.type = exec
a1.sources.src.command = seq 1 100000
a1.sources.src.batchSize = 10
a1.sources.src.channels = c1

a1.sinks.logger.type = logger
a1.sinks.logger.channel = c1

a1.channels.c1.type = spillablememory
a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
a1.channels.c1.dataDirs = /tmp/flume/data
a1.channels.c1.memoryCapacity = 10
a1.channels.c1.keep-alive = 2
Wrote a set of Unit tests. A few are failing and need to be finished up.
pom.xml
Revision 15e6d9b New Change
[20] 923 lines
[+20]
924
        <artifactId>flume-jdbc-channel</artifactId>
924
        <artifactId>flume-jdbc-channel</artifactId>
925
        <version>1.5.0-SNAPSHOT</version>
925
        <version>1.5.0-SNAPSHOT</version>
926
      </dependency>
926
      </dependency>
927

    
   
927

   
928
      <dependency>
928
      <dependency>

    
   
929
        <groupId>org.apache.flume.flume-ng-channels</groupId>

    
   
930
        <artifactId>flume-spillable-memory-channel</artifactId>

    
   
931
        <version>1.4.0-SNAPSHOT</version>

    
   
932
      </dependency>

    
   
933

   

    
   
934
     <dependency>

    
   
935
       <groupId>org.apache.flume.flume-ng-channels</groupId>

    
   
936
       <artifactId>flume-recoverable-memory-channel</artifactId>

    
   
937
       <version>1.4.0-SNAPSHOT</version>

    
   
938
     </dependency>

    
   
939

   

    
   
940
      <dependency>
929
        <groupId>org.apache.flume.flume-ng-sinks</groupId>
941
        <groupId>org.apache.flume.flume-ng-sinks</groupId>
930
        <artifactId>flume-hdfs-sink</artifactId>
942
        <artifactId>flume-hdfs-sink</artifactId>
931
        <version>1.5.0-SNAPSHOT</version>
943
        <version>1.5.0-SNAPSHOT</version>
932
      </dependency>
944
      </dependency>
933

    
   
945

   
[+20] [20] 170 lines
flume-ng-channels/pom.xml
Revision 5832ab4 New Change
 
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
Revision 36f150b New Change
 
flume-ng-channels/flume-spillable-memory-channel/pom.xml
New File
 
flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java
New File
 
flume-ng-channels/flume-spillable-memory-channel/src/test/java/org/apache/flume/channel/TestSpillableMemoryChannel.java
New File
 
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java
Revision 26f4dd7 New Change
 
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java
Revision 15b8cc3 New Change
 
flume-ng-core/src/main/java/org/apache/flume/ChannelFullException.java
New File
 
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java
Revision 1370e66 New Change
 
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java
Revision 688323d New Change
 
flume-ng-dist/pom.xml
Revision 83332a9 New Change
 
flume-ng-doc/sphinx/FlumeUserGuide.rst
Revision 63cad21 New Change
 
flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgentConfiguration.java
Revision 6204bc5 New Change
 
flume-ng-node/pom.xml
Revision f1b0c65 New Change
 
  1. pom.xml: Loading...
  2. flume-ng-channels/pom.xml: Loading...
  3. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java: Loading...
  4. flume-ng-channels/flume-spillable-memory-channel/pom.xml: Loading...
  5. flume-ng-channels/flume-spillable-memory-channel/src/main/java/org/apache/flume/channel/SpillableMemoryChannel.java: Loading...
  6. flume-ng-channels/flume-spillable-memory-channel/src/test/java/org/apache/flume/channel/TestSpillableMemoryChannel.java: Loading...
  7. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java: Loading...
  8. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java: Loading...
  9. flume-ng-core/src/main/java/org/apache/flume/ChannelFullException.java: Loading...
  10. flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java: Loading...
  11. flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java: Loading...
  12. flume-ng-dist/pom.xml: Loading...
  13. flume-ng-doc/sphinx/FlumeUserGuide.rst: Loading...
  14. flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgentConfiguration.java: Loading...
  15. flume-ng-node/pom.xml: Loading...