Review Board 1.7.22


Revised design for Spillable Mem Channel

Review Request #12060 - Created June 24, 2013 and updated

Roshan Naik
flume-1.5
FLUME-1227
Reviewers
Flume
flume-git
Revised design for Spillable Mem Channel.
We no longer have Spillable channel config pointing to another channel (by name) as in the previous design.

Spillable Channel instead derives from FileChannel (as per https://issues.apache.org/jira/browse/FLUME-1227?focusedCommentId=13628201&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13628201)

Essence of this design:
- SC derives from File channel and maintains an in memory queue. If memory queue is full, events are sent to disk overflow (i.e. File channel).
- SC maintains a 'Drain-Order' queue (DOQ) for remembering the order in which the incoming events were interleaved between main memory and 
- Put transaction: All the elements in the putList are committed to mem queue if it has space, else written to disk (ie file channel). Head of DOQ is updated to indicate where the elements put.
- Take transaction : Tail of DOQ is consulted to determine whether the next set of events are to be taken from the memory queue or from disk overflow. DOQ's tail is updated after events are taken out.


SC Configuration:
- Accepts all the File Channel settings
- Introduces one additional setting: 'memoryCapacity' which indicates the number of items it can hold in memory
 

Sample config ...

a1.channels = c1
a1.sinks = logger
a1.sources = src

a1.sources.src.type = exec
a1.sources.src.command = seq 1 100000
a1.sources.src.batchSize = 10
a1.sources.src.channels = c1

a1.sinks.logger.type = logger
a1.sinks.logger.channel = c1

a1.channels.c1.type = spillablememory
a1.channels.c1.checkpointDir = /tmp/flume/checkpoint
a1.channels.c1.dataDirs = /tmp/flume/data
a1.channels.c1.memoryCapacity = 10
a1.channels.c1.keep-alive = 2
Wrote a set of Unit tests. A few are failing and need to be finished up.
Total:
37
Open:
7
Resolved:
30
Dropped:
0
Status:
From:
Description From Last Updated Status
I really think we need a better way of deciding if the events are in primary or secondary. We should ... Hari Shreedharan Aug. 30, 2013, 8:41 p.m. Open
Looks like none of the methods in this class are thread-safe and all methods are called from a synchronized(queueLock) block. ... Hari Shreedharan Aug. 30, 2013, 8:41 p.m. Open
What does this mean? Can you make the error message better here? Hari Shreedharan Aug. 30, 2013, 8:41 p.m. Open
lets have a different config to disable overflow, than overloading the same param. Hari Shreedharan Aug. 30, 2013, 8:41 p.m. Open
This essentially blocks more than 1 transaction at a time right? That will be a serious performance issue. Hari Shreedharan Oct. 16, 2013, 12:05 a.m. Open
else should be on this line Hari Shreedharan Oct. 31, 2013, 7:32 p.m. Open
In this instance, it would nice to make an exception for readability .. since the comment needs to appear before ... Roshan Naik Oct. 31, 2013, 11:33 p.m. Open
Review request changed
Updated (Dec. 18, 2013, 11:43 a.m.)
updating with patch v8.

This patch fixes the issues with
- the 80char line,
- The counter issue you noted plus other counters also fixed.. additional unit tests added to check counters.
- testTotalSemaphore failure.. (now renamed to testCounters)