Review Board 1.7.22


FLUME-1285 FileChannel has a dependency on Hadoop IO classes

Review Request #10829 - Created April 29, 2013 and updated

Israel Ekpo
flume-1.4
FLUME-1285
Reviewers
Flume
flume-git
Modified and Applied Fix from [~chris.nagy] to 1.4 branch

Tested on Ubuntu 12.04 with 1.6.0_33 64-bit

modified:   flume-ng-channels/flume-file-channel/pom.xml
modified:   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java
modified:   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java
new file:   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Writable.java
new file:   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/WritableUtils.java
modified:   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java
Tested on Ubuntu 12.04 with Java 1.6.0_33 64-bit

- Unit Tests for Flume NG file-based channel were all successful with these changes.
- Created and deployed local build successfully with the following settings

agent.sources = seqGenSrc
agent.channels = fileChannel
agent.sinks = loggerSink


agent.sources.seqGenSrc.type = seq
agent.sources.seqGenSrc.channels = fileChannel

agent.channels.fileChannel.type = file
agent.channels.c1.checkpointDir	= /flume-staging/file-channel/checkpoint
agent.channels.c1.dataDirs = /flume-staging/file-channel/data


agent.sinks.loggerSink.type = logger
agent.sinks.loggerSink.channel = fileChannel

Agent was started as follows

$ bin/flume-ng agent --conf-file conf/flume.properties --conf conf --name agent

Agent started up and loaded File Channel successfully.

Agent was also able to shut down successfully.
Posted (May 17, 2013, 4:37 a.m.)
why is the hadoop2 profile being removed ?  
Posted (May 22, 2013, 6:20 a.m.)

   

  
The profile was removed because the dependency on Hadoop for the channel is no longer there.

That is my take.
Posted (May 23, 2013, 2:56 p.m.)
Hey this looks pretty good! Quick question, when you did the manual testing did you generate some data with the pre-patched version and then process the data with the post-patched version?
  1. Hi Brock,
    
    I started from a clean slate.
    
    I built the code from scratch and then I started the agent with brand new events going from the source to the channel and then to the sink.
    
    Would you like me to build up events with the pre-patched version, stop the agent and then restart with the patched version?
    
    
  2. Hi,
    
    > Would you like me to build up events with the pre-patched version, stop the agent and then restart with the patched version?
    
    Yeah I don't expect any problem but it's probably something we should do...
    
    
  3. Ok,
    
    I will work on that over the weekend.
    
    Will update the review with my findings.