Review Board 1.7.22


HIVE-4513 - disable hivehistory logs by default

Review Request #11029 - Created May 9, 2013 and updated

Thejas Nair
HIVE-4513
Reviewers
hive
hive-git
HiveHistory log files (hive_job_log_hive_*.txt files) store information about hive query such as query string, plan , counters and MR job progress information.

There is no mechanism to delete these files and as a result they get accumulated over time, using up lot of disk space. 
I don't think this is used by most people, so I think it would better to turn this off by default. Jobtracker logs already capture most of this information, though it is not as structured as history logs.

The change :
A new config parameter hive.session.history.enabled controls if the history-log is enabled. By default it is set to false.
SessionState initializes the HiveHIstory object. When this config is set to false, it creates a Proxy object that does not do anything. I did this instead of having SessionState return null, because that would add null checks in too many places. This keeps the code cleaner and avoids possibility of NPE.
As the proxy only works against interfaces, i created a HiveHistory interface, moved the implementation to HiveHistoryImpl. static functions were moved to HiveHistoryUtil .


 
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Revision 1672453 New Change
[20] 376 lines
[+20] [+] public class HiveConf extends Configuration {
377
    // session identifier
377
    // session identifier
378
    HIVESESSIONID("hive.session.id", ""),
378
    HIVESESSIONID("hive.session.id", ""),
379
    // whether session is running in silent mode or not
379
    // whether session is running in silent mode or not
380
    HIVESESSIONSILENT("hive.session.silent", false),
380
    HIVESESSIONSILENT("hive.session.silent", false),
381

    
   
381

   

    
   
382
    // Whether to enable history for this session

    
   
383
    HIVE_SESSION_HISTORY_ENABLED("hive.session.history.enabled", false),

    
   
384

   
382
    // query being executed (multiple per session)
385
    // query being executed (multiple per session)
383
    HIVEQUERYSTRING("hive.query.string", ""),
386
    HIVEQUERYSTRING("hive.query.string", ""),
384

    
   
387

   
385
    // id of query being executed (multiple per session)
388
    // id of query being executed (multiple per session)
386
    HIVEQUERYID("hive.query.id", ""),
389
    HIVEQUERYID("hive.query.id", ""),
[+20] [20] 846 lines
conf/hive-default.xml.template
Revision 3a7d1dc New Change
 
data/conf/hive-site.xml
Revision 544ba35 New Change
 
ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java
Revision e1c1ae3 New Change
 
ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java
New File
 
ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryProxyHandler.java
New File
 
ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryUtil.java
New File
 
ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryViewer.java
Revision fdd56db New Change
 
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
Revision 3d43451 New Change
 
ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
Revision a783303 New Change
 
  1. common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: Loading...
  2. conf/hive-default.xml.template: Loading...
  3. data/conf/hive-site.xml: Loading...
  4. ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java: Loading...
  5. ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java: Loading...
  6. ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryProxyHandler.java: Loading...
  7. ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryUtil.java: Loading...
  8. ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryViewer.java: Loading...
  9. ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java: Loading...
  10. ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java: Loading...