Review Board 1.7.22


HIVE-4513 - disable hivehistory logs by default

Review Request #11029 - Created May 9, 2013 and updated

Thejas Nair
HIVE-4513
Reviewers
hive
hive-git
HiveHistory log files (hive_job_log_hive_*.txt files) store information about hive query such as query string, plan , counters and MR job progress information.

There is no mechanism to delete these files and as a result they get accumulated over time, using up lot of disk space. 
I don't think this is used by most people, so I think it would better to turn this off by default. Jobtracker logs already capture most of this information, though it is not as structured as history logs.

The change :
A new config parameter hive.session.history.enabled controls if the history-log is enabled. By default it is set to false.
SessionState initializes the HiveHIstory object. When this config is set to false, it creates a Proxy object that does not do anything. I did this instead of having SessionState return null, because that would add null checks in too many places. This keeps the code cleaner and avoids possibility of NPE.
As the proxy only works against interfaces, i created a HiveHistory interface, moved the implementation to HiveHistoryImpl. static functions were moved to HiveHistoryUtil .


 
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Revision 83f337b New Change
[20] 380 lines
[+20] [+] public class HiveConf extends Configuration {
381
    // session identifier
381
    // session identifier
382
    HIVESESSIONID("hive.session.id", ""),
382
    HIVESESSIONID("hive.session.id", ""),
383
    // whether session is running in silent mode or not
383
    // whether session is running in silent mode or not
384
    HIVESESSIONSILENT("hive.session.silent", false),
384
    HIVESESSIONSILENT("hive.session.silent", false),
385

    
   
385

   

    
   
386
    // Whether to enable history for this session

    
   
387
    HIVE_SESSION_HISTORY_ENABLED("hive.session.history.enabled", false),

    
   
388

   
386
    // query being executed (multiple per session)
389
    // query being executed (multiple per session)
387
    HIVEQUERYSTRING("hive.query.string", ""),
390
    HIVEQUERYSTRING("hive.query.string", ""),
388

    
   
391

   
389
    // id of query being executed (multiple per session)
392
    // id of query being executed (multiple per session)
390
    HIVEQUERYID("hive.query.id", ""),
393
    HIVEQUERYID("hive.query.id", ""),
[+20] [20] 861 lines
conf/hive-default.xml.template
Revision 0a6e433 New Change
 
hbase-handler/src/test/templates/TestHBaseCliDriver.vm
Revision c59e882 New Change
 
ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java
Revision 97436c5 New Change
 
ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java
New File
 
ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryProxyHandler.java
New File
 
ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryUtil.java
New File
 
ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryViewer.java
Revision fdd56db New Change
 
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
Revision ab369f0 New Change
 
ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
Revision a783303 New Change
 
ql/src/test/templates/TestCliDriver.vm
Revision a6ae6c3 New Change
 
  1. common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: Loading...
  2. conf/hive-default.xml.template: Loading...
  3. hbase-handler/src/test/templates/TestHBaseCliDriver.vm: Loading...
  4. ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java: Loading...
  5. ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java: Loading...
  6. ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryProxyHandler.java: Loading...
  7. ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryUtil.java: Loading...
  8. ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryViewer.java: Loading...
  9. ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java: Loading...
  10. ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java: Loading...
  11. ql/src/test/templates/TestCliDriver.vm: Loading...