Review Board 1.7.22


FLUME-1660 Close "idle" hdfs handles

Review Request #7659 - Created Oct. 19, 2012 and submitted

Juhani Connolly
FLUME-1660
Reviewers
Flume
mpercy
flume-git
Added lastWrite to BucketWriter to verify when it was last updated

Added a thread to HDFSEventSink which verifies the last update of each active bucketWriter and closes them after the configurable timeout hdfs.closeIdleTimeout has passed.
Local machine testing was performed and the correct closing of files was confirmed, as well as the correct behavior of the configuration setting including disabling the feature(by using the default value for hdfs.closeIdleTimeout of 0)


There is one unrelated test failure which I'm not sure of(if anyone knows what's causing this, please let me know)

Failed tests:   testInOut(org.apache.flume.test.agent.TestFileChannel): Expected FILE_ROLL sink's dir to have only 1 child, but found 0 children. expected:<1> but was:<0>

All other tests pass.
flume-ng-doc/sphinx/FlumeUserGuide.rst
Revision c1303e0 New Change
[20] 1105 lines
[+20]
1106
hdfs.rollInterval       30            Number of seconds to wait before rolling current file
1106
hdfs.rollInterval       30            Number of seconds to wait before rolling current file
1107
                                      (0 = never roll based on time interval)
1107
                                      (0 = never roll based on time interval)
1108
hdfs.rollSize           1024          File size to trigger roll, in bytes (0: never roll based on file size)
1108
hdfs.rollSize           1024          File size to trigger roll, in bytes (0: never roll based on file size)
1109
hdfs.rollCount          10            Number of events written to file before it rolled
1109
hdfs.rollCount          10            Number of events written to file before it rolled
1110
                                      (0 = never roll based on number of events)
1110
                                      (0 = never roll based on number of events)

    
   
1111
hdfs.idleTimeout        0             Timeout after which inactive files get closed

    
   
1112
                                      (0 = disable automatic closing of idle files)
1111
hdfs.batchSize          100           number of events written to file before it is flushed to HDFS
1113
hdfs.batchSize          100           number of events written to file before it is flushed to HDFS
1112
hdfs.codeC              --            Compression codec. one of following : gzip, bzip2, lzo, snappy
1114
hdfs.codeC              --            Compression codec. one of following : gzip, bzip2, lzo, snappy
1113
hdfs.fileType           SequenceFile  File format: currently ``SequenceFile``, ``DataStream`` or ``CompressedStream``
1115
hdfs.fileType           SequenceFile  File format: currently ``SequenceFile``, ``DataStream`` or ``CompressedStream``
1114
                                      (1)DataStream will not compress output file and please don't set codeC
1116
                                      (1)DataStream will not compress output file and please don't set codeC
1115
                                      (2)CompressedStream requires set hdfs.codeC with an available codeC
1117
                                      (2)CompressedStream requires set hdfs.codeC with an available codeC
[+20] [20] 1178 lines
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
Revision 9f2c763 New Change
 
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
Revision e369604 New Change
 
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
Revision 6a8072e New Change
 
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
Revision fee4c8b New Change
 
  1. flume-ng-doc/sphinx/FlumeUserGuide.rst: Loading...
  2. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java: Loading...
  3. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java: Loading...
  4. flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java: Loading...
  5. flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java: Loading...