Review Board 1.7.22


HDFS Sink should check if file is closed and retry if it is not.

Review Request #11583 - Created May 31, 2013 and updated

Ted Malaska
trunk
2007
Reviewers
Flume
flume-git
We can use the new API added in HDFS-4525. We will need to use reflection though, so we can run against a version of HDFS which does not have this API.


 
flume-ng-doc/sphinx/FlumeUserGuide.rst
Revision 63cad21 New Change
[20] 1511 lines
[+20]
1512
hdfs.round              false         Should the timestamp be rounded down (if true, affects all time based escape sequences except %t)
1512
hdfs.round              false         Should the timestamp be rounded down (if true, affects all time based escape sequences except %t)
1513
hdfs.roundValue         1             Rounded down to the highest multiple of this (in the unit configured using ``hdfs.roundUnit``), less than current time.
1513
hdfs.roundValue         1             Rounded down to the highest multiple of this (in the unit configured using ``hdfs.roundUnit``), less than current time.
1514
hdfs.roundUnit          second        The unit of the round down value - ``second``, ``minute`` or ``hour``.
1514
hdfs.roundUnit          second        The unit of the round down value - ``second``, ``minute`` or ``hour``.
1515
hdfs.timeZone           Local Time    Name of the timezone that should be used for resolving the directory path, e.g. America/Los_Angeles.
1515
hdfs.timeZone           Local Time    Name of the timezone that should be used for resolving the directory path, e.g. America/Los_Angeles.
1516
hdfs.useLocalTimeStamp  false         Use the local time (instead of the timestamp from the event header) while replacing the escape sequences.
1516
hdfs.useLocalTimeStamp  false         Use the local time (instead of the timestamp from the event header) while replacing the escape sequences.

    
   
1517
hdfs.closeTries         1             Max number of times Flume should try to close a file in HDFS, if the version of HDFS supports ``isFileClosed`` method.
1517
serializer              ``TEXT``      Other possible options include ``avro_event`` or the
1518
serializer              ``TEXT``      Other possible options include ``avro_event`` or the
1518
                                      fully-qualified class name of an implementation of the
1519
                                      fully-qualified class name of an implementation of the
1519
                                      ``EventSerializer.Builder`` interface.
1520
                                      ``EventSerializer.Builder`` interface.
1520
serializer.*
1521
serializer.*
1521
======================  ============  ======================================================================
1522
======================  ============  ======================================================================
[+20] [20] 1719 lines
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
Revision bc3b383 New Change
 
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
Revision 2c2be6a New Change
 
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
Revision b8214be New Change
 
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
Revision 0383744 New Change
 
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFileSystemCloseRetryWrapper.java
New File
 
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFsDataOutputStreamCloseRetryWrapper.java
New File
 
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java
Revision ffbdde0 New Change
 
  1. flume-ng-doc/sphinx/FlumeUserGuide.rst: Loading...
  2. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java: Loading...
  3. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java: Loading...
  4. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java: Loading...
  5. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java: Loading...
  6. flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFileSystemCloseRetryWrapper.java: Loading...
  7. flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockFsDataOutputStreamCloseRetryWrapper.java: Loading...
  8. flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestUseRawLocalFileSystem.java: Loading...