Review Board 1.7.22


PIG-3538 Implement LIMIT in Tez

Review Request #14897 - Created Oct. 24, 2013 and updated

Alex Bain
tez
PIG-3538
Reviewers
pig
cheolsoo, daijy, mwagner, rohini
pig-git
Implement LIMIT in Tez by providing an implementation of visitLimit in TezCompiler.java.

UPDATED (Oct 24 4:37 PM):
1. I added a test to TestTezCompiler.java and a GLD file
2. I included Daniel's patch for a new e2e test
[abain@abain-ld pig]$ cat data/1.dat
1,orange
2,apple
3,strawberry

[abain@abain-ld pig]$ cat test3.pig
a = load './1.dat' using PigStorage(',') as (id:int, fruit:chararray);
b = LIMIT a 2;
STORE b INTO 'foo';

I ran with with "pig -x tez -f test3.pig" and got the following (correct results):

[abain@abain-ld pig]$ hadoop fs -ls /user/abain/foo
Found 2 items
-rw-r--r--   1 abain supergroup          0 2013-10-23 18:38 /user/abain/foo/_SUCCESS
-rw-r--r--   1 abain supergroup         17 2013-10-23 18:38 /user/abain/foo/part-r-00000

[abain@abain-ld pig]$ hadoop fs -cat /user/abain/foo/part-r-00000
1	orange
2	apple

UPDATED (Oct 24 4:37 PM):
1. ant -Dtestcase=TestTezCompiler test passes
2. I ran test-e2e-tez. The new test seems to pass (although something else failed).
Review request changed
Updated (Oct. 24, 2013, 11:47 p.m.)
New diff generated with --no-prefix
Posted (Oct. 24, 2013, 11:54 p.m.)
2. I ran test-e2e-tez. The new test seems to pass (although something else failed).
Yes, Checkin_2 fail for me. Should be a simple fix.
Ship it!
Posted (Oct. 25, 2013, 12:39 a.m.)
Ship It!