Review Board 1.7.22


FLUME-1702 HDFSEventSink should write to a hidden file as opposed to a .tmp file

Review Request #8438 - Created Dec. 10, 2012 and submitted

Jarek Cecho
FLUME-1702
Reviewers
Flume
flume-git
I've added two additional HDFS sink properties "inUsePrefix" and "inUseSuffix" allowing user to override both prefix and suffix of temporal files.
Current unit tests seems to be passing. I've also added two additional tests to verify new functionality.
flume-ng-doc/sphinx/FlumeUserGuide.rst
Revision 649071020b2281f9a1fb772c0672c8879655e8dd New Change
[20] 1272 lines
[+20]
1273
**channel**             --
1273
**channel**             --
1274
**type**                --            The component type name, needs to be ``hdfs``
1274
**type**                --            The component type name, needs to be ``hdfs``
1275
**hdfs.path**           --            HDFS directory path (eg hdfs://namenode/flume/webdata/)
1275
**hdfs.path**           --            HDFS directory path (eg hdfs://namenode/flume/webdata/)
1276
hdfs.filePrefix         FlumeData     Name prefixed to files created by Flume in hdfs directory
1276
hdfs.filePrefix         FlumeData     Name prefixed to files created by Flume in hdfs directory
1277
hdfs.fileSuffix         --            Suffix to append to file (eg ``.avro`` - *NOTE: period is not automatically added*)
1277
hdfs.fileSuffix         --            Suffix to append to file (eg ``.avro`` - *NOTE: period is not automatically added*)

    
   
1278
hdfs.inUsePrefix        --            Prefix that is used for temporal files that flume actively writes into

    
   
1279
hdfs.inUseSuffix        ``.tmp``      Suffix that is used for temporal files that flume actively writes into
1278
hdfs.rollInterval       30            Number of seconds to wait before rolling current file
1280
hdfs.rollInterval       30            Number of seconds to wait before rolling current file
1279
                                      (0 = never roll based on time interval)
1281
                                      (0 = never roll based on time interval)
1280
hdfs.rollSize           1024          File size to trigger roll, in bytes (0: never roll based on file size)
1282
hdfs.rollSize           1024          File size to trigger roll, in bytes (0: never roll based on file size)
1281
hdfs.rollCount          10            Number of events written to file before it rolled
1283
hdfs.rollCount          10            Number of events written to file before it rolled
1282
                                      (0 = never roll based on number of events)
1284
                                      (0 = never roll based on number of events)
[+20] [20] 1473 lines
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
Revision d0ff6e34399886ee4a85d983243f76899a966b79 New Change
 
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
Revision 64ac2d73520be961617e3dd22b862305e4e00573 New Change
 
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
Revision b191ef3fd36be6cc4d8f754bb2ad9898994df03f New Change
 
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
Revision d23f09dd9e8ef5c257bdb3b08bf8deec57641b0b New Change
 
  1. flume-ng-doc/sphinx/FlumeUserGuide.rst: Loading...
  2. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java: Loading...
  3. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java: Loading...
  4. flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java: Loading...
  5. flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java: Loading...