Review Board 1.7.22


Draft patch for review. Based on phase 1 mentioned in design doc.

Review Request #10857 - Created April 30, 2013 and updated

Roshan Naik
trunk
HIVE-5138
Reviewers
hive
hive-git
Draft patch for review. based on phase 1 mentioned in design doc. Deviates slighlty from doc in the follow ways... 
1) adds a couple of (temporary) rest calls to enable/disable streaming on a table. Later these will be replaced with support in DDL.
2) Also also HTTP methods are GET for easy testing with web browser
3) Authentication disabled on the new streaming HTTP methods


Usage Examples on db named 'sdb' & table named 'log' :

1) Setup db & table with single partition column 'date':
hcat -e "create database sdb; use sdb; create table log(msg string, region string) partitioned by (date string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE; "
2) To check streaming status:
http://localhost:50111/templeton/v1/streaming/status?database=sdb&table=log
3) Enable Streaming:
http://localhost:50111/templeton/v1/streaming/enable?database=sdb&table=log&col=date&value=1000
4) Get Chunk File to write to:
http://localhost:50111/templeton/v1/streaming/chunkget?database=sdb&table=log&schema=blah&format=blah&record_separator=blah&field_separator=blah
5) Commit Chunk File:
http://localhost:50111/templeton/v1/streaming/chunkcommit?database=sdb&table=log&chunkfile=/user/hive/streaming/tmp/sdb/log/2
6) Abort Chunk File:
http://localhost:50111/templeton/v1/streaming/chunkabort?database=sdb&table=log&chunkfile=/user/hive/streaming/tmp/sdb/log/3
7) Roll Partition:
http://localhost:50111/templeton/v1/streaming/partitionroll?database=sdb&table=log&partition_column=date&partition_value=3000
Manual testing only
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Revision 1af68a6 New Change
[20] 259 lines
[+20] [+] public class HiveConf extends Configuration {
260
    // Number of times to retry a connection to a Thrift metastore server
260
    // Number of times to retry a connection to a Thrift metastore server
261
    METASTORETHRIFTCONNECTIONRETRIES("hive.metastore.connect.retries", 3),
261
    METASTORETHRIFTCONNECTIONRETRIES("hive.metastore.connect.retries", 3),
262
    // Number of times to retry a Thrift metastore call upon failure
262
    // Number of times to retry a Thrift metastore call upon failure
263
    METASTORETHRIFTFAILURERETRIES("hive.metastore.failure.retries", 1),
263
    METASTORETHRIFTFAILURERETRIES("hive.metastore.failure.retries", 1),
264

    
   
264

   

    
   
265
    STREAMING_TMPDIR("hive.streaming.tempdir", "/user/hive/streaming/tmp"),

    
   
266

   
265
    // Number of seconds the client should wait between connection attempts
267
    // Number of seconds the client should wait between connection attempts
266
    METASTORE_CLIENT_CONNECT_RETRY_DELAY("hive.metastore.client.connect.retry.delay", 1),
268
    METASTORE_CLIENT_CONNECT_RETRY_DELAY("hive.metastore.client.connect.retry.delay", 1),
267
    // Socket timeout for the client connection (in seconds)
269
    // Socket timeout for the client connection (in seconds)
268
    METASTORE_CLIENT_SOCKET_TIMEOUT("hive.metastore.client.socket.timeout", 20),
270
    METASTORE_CLIENT_SOCKET_TIMEOUT("hive.metastore.client.socket.timeout", 20),
269
    METASTOREPWD("javax.jdo.option.ConnectionPassword", "mine"),
271
    METASTOREPWD("javax.jdo.option.ConnectionPassword", "mine"),
[+20] [20] 997 lines
hcatalog/webhcat/svr/pom.xml
Revision 16a870e New Change
 
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/HcatStreamingDelegator.java
New File
 
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Main.java
Revision 0f37278 New Change
 
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Server.java
Revision d8b6ce2 New Change
 
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestStreaming.java
New File
 
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/HiveJdbcClient.java
New File
 
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/WebHCatClient.java
New File
 
metastore/if/hive_metastore.thrift
Revision 3fea057 New Change
 
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Revision 39dda92 New Change
 
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
Revision e165b2d New Change
 
metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
Revision a2d6b1b New Change
 
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Revision a27243d New Change
 
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java
Revision e410c3a New Change
 
metastore/src/model/package.jdo
Revision c42b5b0 New Change
 
metastore/src/model/org/apache/hadoop/hive/metastore/model/MTable.java
Revision 2a78ce9 New Change
 
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
Revision 8066784 New Change
 
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
Revision 0f9b16c New Change
 
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
Revision 80fbbac New Change
 
  1. common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: Loading...
  2. hcatalog/webhcat/svr/pom.xml: Loading...
  3. hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/HcatStreamingDelegator.java: Loading...
  4. hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Main.java: Loading...
  5. hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Server.java: Loading...
  6. hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestStreaming.java: Loading...
  7. hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/HiveJdbcClient.java: Loading...
  8. hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/WebHCatClient.java: Loading...
  9. metastore/if/hive_metastore.thrift: Loading...
  10. metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java: Loading...
  11. metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java: Loading...
  12. metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java: Loading...
  13. metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java: Loading...
  14. metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java: Loading...
  15. metastore/src/model/package.jdo: Loading...
  16. metastore/src/model/org/apache/hadoop/hive/metastore/model/MTable.java: Loading...
  17. metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java: Loading...
  18. metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java: Loading...
  19. metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java: Loading...