Review Board 1.7.22


PIG-3541 Add diagnostic information to TezStats

Review Request #15031 - Created Oct. 29, 2013 and updated

Cheolsoo Park
tez
PIG-3541
Reviewers
pig
daijy, mwagner, rohini
pig-git
This patch includes the following changes:
* Implement Input/OutputStats for Tez. (This makes DUMP work.) As of now, counters cannot be retrieved from Tez DAG, so only filenames are reported.
* Add the error message from DAGStatus.getDiagnostic() for failed DAG. As of now, backend error messages or stack traces cannot be retrieved from Tez DAG, so only the id of failed vertex is reported.
* Factor out a few methods/fields that can be used by both MR and Tez into PigStats. Duplicate code between SimplePigStats and TezStats is minimal now.

* Updated TestTezLauncher by adding asserts for input/output stats.
* Ran ant test-tez.
* Verified reports for succeeded/failed DAGs-

  Success!
            Input(s): Successfully read records from: "hdfs://localhost:57063/user/cheolsoop/foo"                         
           Output(s): Successfully stored records in: "/user/cheolsoop/13"

  Failed!
        ErrorMessage: Vertex failed vertex_1383071498815_0006_1_01                                                        
                    : DAG failed due to vertex failure. failedVertices:1 killedVertices:0                                 

            Input(s): Failed to read data from "hdfs://localhost:57063/user/cheolsoop/foo"                                
           Output(s): Failed to produce result in "/user/cheolsoop/14" 
src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java
Revision 14da46e New Change
[20] 87 lines
[+20] [+] public void submit() {
88
            }
88
            }
89

    
   
89

   
90
            log.info("DAG Status: " + dagStatus);
90
            log.info("DAG Status: " + dagStatus);
91
            setJobState(dagState2JobState(dagStatus.getState()));
91
            setJobState(dagState2JobState(dagStatus.getState()));
92
            if (dagStatus.isCompleted()) {
92
            if (dagStatus.isCompleted()) {

    
   
93
                StringBuilder sb = new StringBuilder();

    
   
94
                for (String msg : dagStatus.getDiagnostics()) {

    
   
95
                    sb.append(msg);

    
   
96
                    sb.append("\n");

    
   
97
                }

    
   
98
                setMessage(sb.toString());
93
                break;
99
                break;
94
            }
100
            }
95

    
   
101

   
96
            try {
102
            try {
97
                Thread.sleep(1000);
103
                Thread.sleep(1000);
[+20] [20] 36 lines
src/org/apache/pig/backend/hadoop/executionengine/tez/TezJobControlCompiler.java
Revision 9d93968 New Change
 
src/org/apache/pig/tools/pigstats/JobStats.java
Revision 5eac24b New Change
 
src/org/apache/pig/tools/pigstats/PigStats.java
Revision e2eba6d New Change
 
src/org/apache/pig/tools/pigstats/mapreduce/MRJobStats.java
Revision 1a37848 New Change
 
src/org/apache/pig/tools/pigstats/mapreduce/MRPigStatsUtil.java
Revision 4bdcf19 New Change
 
src/org/apache/pig/tools/pigstats/mapreduce/SimplePigStats.java
Revision 5088563 New Change
 
src/org/apache/pig/tools/pigstats/tez/TezStats.java
Revision b0d7f45 New Change
 
src/org/apache/pig/tools/pigstats/tez/TezTaskStats.java
Revision bd45d8f New Change
 
test/org/apache/pig/tez/TestTezLauncher.java
Revision 8382a7d New Change
 
  1. src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java: Loading...
  2. src/org/apache/pig/backend/hadoop/executionengine/tez/TezJobControlCompiler.java: Loading...
  3. src/org/apache/pig/tools/pigstats/JobStats.java: Loading...
  4. src/org/apache/pig/tools/pigstats/PigStats.java: Loading...
  5. src/org/apache/pig/tools/pigstats/mapreduce/MRJobStats.java: Loading...
  6. src/org/apache/pig/tools/pigstats/mapreduce/MRPigStatsUtil.java: Loading...
  7. src/org/apache/pig/tools/pigstats/mapreduce/SimplePigStats.java: Loading...
  8. src/org/apache/pig/tools/pigstats/tez/TezStats.java: Loading...
  9. src/org/apache/pig/tools/pigstats/tez/TezTaskStats.java: Loading...
  10. test/org/apache/pig/tez/TestTezLauncher.java: Loading...