Review Board 1.7.22


PIG-3400 FS commands do not work with S3 paths

Review Request #13122 - Created July 31, 2013 and updated

Cheolsoo Park
PIG-3400
Reviewers
pig
pig-git
Makes fs utility commands work with s3 paths. Documents fs utility commands in the Pig manual. (They were not documented at all.)
All unit tests pass.

Manually verified the following commands with s3 paths:
pig -e 'ls s3://<path>'
pig -e 'mkdir s3://<path>'
pig -e 'rm s3://<path>'
pig -e 'cp s3://<path1> s3://<path2>'
pig -e 'mv s3://<path1> s3://<path2>'
pig -e 'copyToLocal s3://<path>/<file> .'
pig -e 'copyFromLocal <file> s3://<path>' 
Total:
3
Open:
2
Resolved:
0
Dropped:
1
Status:
From:
Posted (July 31, 2013, 10:07 p.m.)
http://pig.apache.org/docs/r0.11.1/cmds.html

  The fs commands should be documented as part of http://pig.apache.org/docs/r0.11.1/cmds.html#fs and not http://pig.apache.org/docs/r0.11.1/cmds.html#utillity-cmds. Utility cmds are grunt commands. 
  1. Actually, no. The commands that I am fixing are different from "fs" commands. It's confusing, but both of the following work as of now:
    
    grunt> fs ls <path>
    grunt> ls <path>
Posted (Aug. 1, 2013, 12:06 a.m.)

   

  
fs cat URI. Same for other fs commands. 
  1. No, actually "cat URI" is correct. If you look at GruntParser, processCat() is different from processFsCommand(). In fact, we should deprecate one of them, but as of now, they exist.
We should mention local and hdfs, but should not specifically state s3. With your fix, it will work with any file system implementation. So can mention that. Same for other fs commands.
  1. Sure I will replace s3 with "any other file system".
White spaces in many places
  1. Will fix.