Review Board 1.7.22


SQOOP-924 Sqoop2: Devguide: Describe wire protocol

Review Request #13452 - Created Aug. 9, 2013 and submitted

Mengwei Ding
SQOOP-924
Reviewers
Sqoop
abe, hshreedharan, jarcec
sqoop-sqoop2
commit 5242f478e2106cfcfc19d0ada9307aa27fb9c829
Author: Mengwei Ding <mengwei.ding@gmail.com>
Date:   Mon Aug 19 15:42:52 2013 -0700

    SQOOP-924 Sqoop2: Devguide: Describe wire protocol

:000000 100644 0000000... 04908e8... A	docs/src/site/sphinx/RESTAPI.rst
:100644 100644 6b03e62... 07e3c47... M	docs/src/site/sphinx/index.rst

 

Diff revision 3 (Latest)

1 2 3
1 2 3

  1. docs/src/site/sphinx/RESTAPI.rst: Loading...
  2. docs/src/site/sphinx/index.rst: Loading...
docs/src/site/sphinx/RESTAPI.rst
New File

    
   
1
.. Licensed to the Apache Software Foundation (ASF) under one or more

    
   
2
   contributor license agreements.  See the NOTICE file distributed with

    
   
3
   this work for additional information regarding copyright ownership.

    
   
4
   The ASF licenses this file to You under the Apache License, Version 2.0

    
   
5
   (the "License"); you may not use this file except in compliance with

    
   
6
   the License.  You may obtain a copy of the License at

    
   
7

   

    
   
8
       http://www.apache.org/licenses/LICENSE-2.0

    
   
9

   

    
   
10
   Unless required by applicable law or agreed to in writing, software

    
   
11
   distributed under the License is distributed on an "AS IS" BASIS,

    
   
12
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

    
   
13
   See the License for the specific language governing permissions and

    
   
14
   limitations under the License.

    
   
15

   

    
   
16
=========================

    
   
17
Sqoop REST API Guide

    
   
18
=========================

    
   
19

   

    
   
20
This document will explain you how to use Sqoop Network API to allow external applications interacting with Sqoop server.

    
   
21
The REST API is a lower level API than the `Sqoop client API <ClientAPI.html>`_, which gives you the freedom to execute commands in Sqoop

    
   
22
server with any tools or programming language. Generally, the REST API is leveraged via HTTP requests and use ``JSON`` format to encode data content.

    
   
23

   

    
   
24
.. contents:: Table of Contents

    
   
25

   

    
   
26
Initialization

    
   
27
=========================

    
   
28

   

    
   
29
Before making any move, make sure that the Sqoop server is running.

    
   
30

   

    
   
31
Then find out three pieces of information about the Sqoop server: ``host``, ``port`` and ``webapp``, and keep them in mind.

    
   
32

   

    
   
33
To execute a function of Sqoop, you could assemble and send a HTTP request to an url of that function in Sqoop. Generally, the url

    
   
34
contain the ``host`` has the hostname, the ``port`` as the port number and ``webapp`` as the root directory of the url, which follows

    
   
35
any functions available in this page.

    
   
36

   

    
   
37
The request and its response might need to contain some additional parameters and contents. These parameters could be given via

    
   
38
HTTP header, body or both. All the content in the HTTP body is in ``JSON`` format.

    
   
39

   

    
   
40
Understand Connector, Connection, Job and Framework

    
   
41
===========================================================

    
   
42

   

    
   
43
To create and run a Sqoop task, we need to provide a bunch of parameters. All these parameters are provide via Input.

    
   
44

   

    
   
45
Some of these parameters are connector specific, which are needed for a connector. And the other parameters

    
   
46
might be required all the times. Sqoop provides these specifications via forms, each of which is a list of inputs. Each connector claims its own

    
   
47
forms. And the Sqoop framework claims global forms.

    
   
48

   

    
   
49
On the other hand, some of these parameters are stable for different jobs, like the url, the username and the password of database. And some

    
   
50
of them are job specific. Therefore Sqoop categorizes these parameters into 2 groups, which are connection object and job object. So, for

    
   
51
each connector and the framework, it has to provide forms for connection and job, respectively.

    
   
52

   

    
   
53
Objects

    
   
54
==============

    
   
55

   

    
   
56
This section covers all the objects that might exist in the request or response.

    
   
57

   

    
   
58
Form

    
   
59
----------

    
   
60

   

    
   
61
Before creating any connection or job, the first thing to do is getting familiar with all forms in the connector and the framework, via Get Connector

    
   
62
and Get Framework HTTP requests.

    
   
63

   

    
   
64
Each form is structured below:

    
   
65

   

    
   
66
+------------------+---------------------------------------------------------+

    
   
67
|   Field          | Description                                             |

    
   
68
+==================+=========================================================+

    
   
69
| ``id``           | The id of this form                                     |

    
   
70
+------------------+---------------------------------------------------------+

    
   
71
| ``inputs``       | A array of input fields of this form                    |

    
   
72
+------------------+---------------------------------------------------------+

    
   
73
| ``name``         | The name of this form                                   |

    
   
74
+------------------+---------------------------------------------------------+

    
   
75
| ``type``         | The type of this form (CONNECTION/JOB)                  |

    
   
76
+------------------+---------------------------------------------------------+

    
   
77

   

    
   
78
A typical form object is showing below:

    
   
79

   

    
   
80
::

    
   
81

   

    
   
82
  {

    
   
83
    "id":3,

    
   
84
    "inputs":[

    
   
85
      {

    
   
86
        "id":13,

    
   
87
        "name":"table.schemaName",

    
   
88
        "type":"STRING",

    
   
89
        "size":50,

    
   
90
        "sensitive":false

    
   
91
      }

    
   
92
    ],

    
   
93
    "name":"table",

    
   
94
    "type":"CONNECTION"

    
   
95
  }

    
   
96

   

    
   
97
Each input object in a form is structured below:

    
   
98

   

    
   
99
+------------------+---------------------------------------------------------+

    
   
100
|   Field          | Description                                             |

    
   
101
+==================+=========================================================+

    
   
102
| ``id``           | The id of this input                                    |

    
   
103
+------------------+---------------------------------------------------------+

    
   
104
| ``name``         | The name of this input                                  |

    
   
105
+------------------+---------------------------------------------------------+

    
   
106
| ``type``         | The data type of this input field                       |

    
   
107
+------------------+---------------------------------------------------------+

    
   
108
| ``size``         | The length of this input field                          |

    
   
109
+------------------+---------------------------------------------------------+

    
   
110
| ``sensitive``    | Whether this input contain sensitive information        |

    
   
111
+------------------+---------------------------------------------------------+

    
   
112

   

    
   
113
The connector and framework have both ``job-forms`` and ``conn-forms``, each of which is a array of forms. In ``job-forms``,

    
   
114
there are 2 arrays of forms, for ``IMPORT`` and ``EXPORT``, job respectively.

    
   
115

   

    
   
116
To send a filled form in the request, you should always use form id and input id to map the values to inputs. For example, the

    
   
117
following request contains a input value ``com.mysql.jdbc.Driver`` for input ``1`` in form ``1`` of connector and an input value

    
   
118
``10`` for input ``17`` in form ``4`` of framework.

    
   
119

   

    
   
120
::

    
   
121

   

    
   
122
  {

    
   
123
    "connector":[

    
   
124
      {

    
   
125
        "id":1,

    
   
126
        "inputs":[

    
   
127
          {

    
   
128
            "id":1,

    
   
129
            "name":"connection.jdbcDriver",

    
   
130
            "value":"com.mysql.jdbc.Driver",

    
   
131
            "type":"STRING",

    
   
132
            "size":128,

    
   
133
            "sensitive":false

    
   
134
          },

    
   
135
        ],

    
   
136
        "name":"connection",

    
   
137
        "type":"CONNECTION"

    
   
138
      }

    
   
139
    ],

    
   
140
    "connector-id":1,

    
   
141
    "framework":[

    
   
142
      {

    
   
143
        "id":4,

    
   
144
        "inputs":[

    
   
145
          {

    
   
146
            "id":17,

    
   
147
            "name":"security.maxConnections",

    
   
148
            "value":"10",

    
   
149
            "type":"INTEGER",

    
   
150
            "sensitive":false

    
   
151
          }

    
   
152
        ],

    
   
153
        "name":"security",

    
   
154
        "type":"CONNECTION"

    
   
155
      }

    
   
156
    ]

    
   
157
  }

    
   
158

   

    
   
159
Exception

    
   
160
---------------

    
   
161

   

    
   
162
Each operation on Sqoop server might return an exception in the Http response. Remember to take this into account.

    
   
163

   

    
   
164
The exception code and message could be found in both the header and body of the response, if happens.

    
   
165

   

    
   
166
Please jump to "Header Parameters" section to find how to get exception information from header.

    
   
167

   

    
   
168
In the body, the exception is expressed in ``JSON`` format. An example of the exception is:

    
   
169

   

    
   
170
::

    
   
171

   

    
   
172
  {

    
   
173
    "message":"DERBYREPO_0030:Unable to load specific job metadata from repository - Couldn't find job with id 2",

    
   
174
    "stack-trace":[

    
   
175
      {

    
   
176
        "file":"DerbyRepositoryHandler.java",

    
   
177
        "line":1111,

    
   
178
        "class":"org.apache.sqoop.repository.derby.DerbyRepositoryHandler",

    
   
179
        "method":"findJob"

    
   
180
      },

    
   
181
      {

    
   
182
        "file":"JdbcRepository.java",

    
   
183
        "line":451,

    
   
184
        "class":"org.apache.sqoop.repository.JdbcRepository$16",

    
   
185
        "method":"doIt"

    
   
186
      },

    
   
187
      {

    
   
188
        "file":"JdbcRepository.java",

    
   
189
        "line":90,

    
   
190
        "class":"org.apache.sqoop.repository.JdbcRepository",

    
   
191
        "method":"doWithConnection"

    
   
192
      },

    
   
193
      {

    
   
194
        "file":"JdbcRepository.java",

    
   
195
        "line":61,

    
   
196
        "class":"org.apache.sqoop.repository.JdbcRepository",

    
   
197
        "method":"doWithConnection"

    
   
198
      },

    
   
199
      {

    
   
200
        "file":"JdbcRepository.java",

    
   
201
        "line":448,

    
   
202
        "class":"org.apache.sqoop.repository.JdbcRepository",

    
   
203
        "method":"findJob"

    
   
204
      },

    
   
205
      {

    
   
206
        "file":"JobRequestHandler.java",

    
   
207
        "line":238,

    
   
208
        "class":"org.apache.sqoop.handler.JobRequestHandler",

    
   
209
        "method":"getJobs"

    
   
210
      }

    
   
211
    ],

    
   
212
    "class":"org.apache.sqoop.common.SqoopException"

    
   
213
  }

    
   
214

   

    
   
215
Form Validation Status

    
   
216
---------------------------

    
   
217

   

    
   
218
After submitting the forms of creating/updating connection/job, the server will validate these forms and send

    
   
219
back feedbacks to show the validation status.

    
   
220

   

    
   
221
There are 3 possible status:

    
   
222

   

    
   
223
+------------------+---------------------------------------------------------+

    
   
224
|   Status         | Description                                             |

    
   
225
+==================+=========================================================+

    
   
226
| ``FINE``         | No issues, no warnings. Everything is perfect           |

    
   
227
+------------------+---------------------------------------------------------+

    
   
228
| ``ACCEPTABLE``   | No issues, but might be some warnings. Good to go.      |

    
   
229
+------------------+---------------------------------------------------------+

    
   
230
| ``UNACCEPTABLE`` | The form has severe issues needed to be fixed           |

    
   
231
+------------------+---------------------------------------------------------+

    
   
232

   

    
   
233
An example of a good status is:

    
   
234

   

    
   
235
::

    
   
236

   

    
   
237
  {

    
   
238
   "status":"FINE",

    
   
239
   "messages":{}

    
   
240
  }

    
   
241

   

    
   
242
A bad status might be:

    
   
243

   

    
   
244
::

    
   
245

   

    
   
246
  {

    
   
247
    "message":"Can't load specified driver",

    
   
248
    "status":"UNACCEPTABLE"

    
   
249
  }

    
   
250

   

    
   
251
Job Status

    
   
252
-------------------

    
   
253

   

    
   
254
After submitting a job, you could look up the running status of it. There could be 7 possible status:

    
   
255

   

    
   
256
+-----------------------------+---------------------------------------------------------+

    
   
257
|   Status                    | Description                                             |

    
   
258
+=============================+=========================================================+

    
   
259
| ``BOOTING``                 | In the middle of submitting the job                     |

    
   
260
+-----------------------------+---------------------------------------------------------+

    
   
261
| ``FAILURE_ON_SUBMIT``       | Unable to submit this job to remote cluster             |

    
   
262
+-----------------------------+---------------------------------------------------------+

    
   
263
| ``RUNNING``                 | The job is running now                                  |

    
   
264
+-----------------------------+---------------------------------------------------------+

    
   
265
| ``SUCCEEDED``               | Job finished successfully                               |

    
   
266
+-----------------------------+---------------------------------------------------------+

    
   
267
| ``FAILED``                  | Job failed                                              |

    
   
268
+-----------------------------+---------------------------------------------------------+

    
   
269
| ``NEVER_EXECUTED``          | The job has never been executed since created           |

    
   
270
+-----------------------------+---------------------------------------------------------+

    
   
271
| ``UNKNOWN``                 | The status is unknown                                   |

    
   
272
+-----------------------------+---------------------------------------------------------+

    
   
273

   

    
   
274
Header Parameters

    
   
275
======================

    
   
276

   

    
   
277
For all Sqoop requests, the following header parameters are supported:

    
   
278

   

    
   
279
+---------------------------+----------+---------------------------------------------------------+

    
   
280
|   Parameter               | Required | Description                                             |

    
   
281
+===========================+==========+=========================================================+

    
   
282
| ``sqoop-user-name``       | true     | The name of the user who makes the requests             |

    
   
283
+---------------------------+----------+---------------------------------------------------------+

    
   
284

   

    
   
285
For all the responses, the following parameters in the HTTP message header are available:

    
   
286

   

    
   
287
+---------------------------+----------+------------------------------------------------------------------------------+

    
   
288
|   Parameter               | Required | Description                                                                  |

    
   
289
+===========================+==========+==============================================================================+

    
   
290
| ``sqoop-error-code``      | false    | The error code when some error happen in the server side for this request    |

    
   
291
+---------------------------+----------+------------------------------------------------------------------------------+

    
   
292
| ``sqoop-error-message``   | false    | The explanation for a error code                                             |

    
   
293
+---------------------------+----------+------------------------------------------------------------------------------+

    
   
294

   

    
   
295
So far, there are only these 2 parameters in the header of response message. They only exist when something bad happen in the server.

    
   
296
And they always come along with an exception message in the response body.

    
   
297

   

    
   
298
Functions

    
   
299
==================

    
   
300

   

    
   
301
The section elaborates all the functions that are supported by the Sqoop server.

    
   
302

   

    
   
303
/version - [GET] - Get Sqoop Version

    
   
304
-------------------------------------------

    
   
305

   

    
   
306
Get all the version metadata of Sqoop software in the server side.

    
   
307

   

    
   
308
* Method: ``GET``

    
   
309
* Format: ``JSON``

    
   
310
* Request Content: ``None``

    
   
311
* Fields of Response:

    
   
312

   

    
   
313
+---------------+---------------------------------------------------------+

    
   
314
|   Field       | Description                                             |

    
   
315
+===============+=========================================================+

    
   
316
| ``revision``  | The revision number of Sqoop source code                |

    
   
317
+---------------+---------------------------------------------------------+

    
   
318
| ``protocols`` | The version of network protocol                         |

    
   
319
+---------------+---------------------------------------------------------+

    
   
320
| ``date``      | The Sqoop release date                                  |

    
   
321
+---------------+---------------------------------------------------------+

    
   
322
| ``user``      | The user who made the release                           |

    
   
323
+---------------+---------------------------------------------------------+

    
   
324
| ``url``       | The url of the source code trunk                        |

    
   
325
+---------------+---------------------------------------------------------+

    
   
326
| ``version``   | The version of Sqoop in the server side                 |

    
   
327
+---------------+---------------------------------------------------------+

    
   
328

   

    
   
329

   

    
   
330
* Response Example:

    
   
331

   

    
   
332
::

    
   
333

   

    
   
334
  {

    
   
335
    "revision":"e56c977b56f4dc32a4cad06a328bad11e0d0055b",

    
   
336
    "protocols":["1"],

    
   
337
    "date":"Wed Aug  7 13:31:36 PDT 2013",

    
   
338
    "user":"mengwei.ding",

    
   
339
    "url":"git:\/\/mding-MBP.local\/Users\/mengwei.ding\/Documents\/workspace\/sqoop2\/common",

    
   
340
    "version":"2.0.0-SNAPSHOT"

    
   
341
  }

    
   
342

   

    
   
343
/v1/connector/[cid] - [GET] - Get Connector

    
   
344
---------------------------------------------------

    
   
345

   

    
   
346
Retrieve all the metadata of a given connector, such as its forms to be filled for jobs and connections, the explanation

    
   
347
for each fields of these forms.

    
   
348

   

    
   
349
Provide the id of the connector in the url ``[cid]`` part. If you provide ``all`` in the ``[cid]`` part in the url, you will

    
   
350
get the metadata of all connectors.

    
   
351

   

    
   
352
* Method: ``GET``

    
   
353
* Format: ``JSON``

    
   
354
* Request Content: ``None``

    
   
355
* Fields of Response:

    
   
356

   

    
   
357
+--------------------------+--------------------------------------------------------------------------+

    
   
358
|   Field                  | Description                                                              |

    
   
359
+==========================+==========================================================================+

    
   
360
| ``resources-connector``  | All resources for the given connector                                    |

    
   
361
+--------------------------+--------------------------------------------------------------------------+

    
   
362
| ``all``                  | All metadata about the given connector, such as id, name and all forms   |

    
   
363
+--------------------------+--------------------------------------------------------------------------+

    
   
364

   

    
   
365
If all connectors are retrieved, the ``resources-connector`` and ``all`` fields will become arrays and contain data for all connectors.

    
   
366

   

    
   
367
So far, the resource contains only explanations for fields of forms. For example, in the IMPORT job form, you could find a field called

    
   
368
``table.schemaName``. If you have no idea about what that field means, you could go the resource for help.

    
   
369

   

    
   
370
* Response Example:

    
   
371

   

    
   
372
::

    
   
373

   

    
   
374
  {

    
   
375
    "resources-connector":{

    
   
376
      "1":{

    
   
377
        "ignored.label":"Ignored",

    
   
378
        "table.partitionColumn.help":"A specific column for data partition",

    
   
379
        "table.label":"Database configuration",

    
   
380
        "table.boundaryQuery.label":"Boundary query",

    
   
381
        "ignored.help":"This is completely ignored",

    
   
382
        "ignored.ignored.label":"Ignored",

    
   
383
        "connection.jdbcProperties.help":"Enter any JDBC properties that should be supplied during the creation of connection.",

    
   
384
        "table.tableName.help":"Table name to process data in the remote database",

    
   
385
        "connection.jdbcDriver.label":"JDBC Driver Class",

    
   
386
        "connection.username.help":"Enter the username to be used for connecting to the database.",

    
   
387
        "table.help":"You must supply the information requested in order to create a job object.",

    
   
388
        "table.partitionColumn.label":"Partition column name",

    
   
389
        "ignored.ignored.help":"This is completely ignored",

    
   
390
        "table.partitionColumnNull.label":"Nulls in partition column",

    
   
391
        "table.warehouse.label":"Data warehouse",

    
   
392
        "table.boundaryQuery.help":"The boundary query for data partition",

    
   
393
        "connection.username.label":"Username",

    
   
394
        "connection.jdbcDriver.help":"Enter the fully qualified class name of the JDBC driver that will be used for establishing this connection.",

    
   
395
        "connection.label":"Connection configuration",

    
   
396
        "table.columns.label":"Table column names",

    
   
397
        "table.dataDirectory.label":"Data directory",

    
   
398
        "table.partitionColumnNull.help":"Whether there are null values in partition column",

    
   
399
        "connection.password.label":"Password",

    
   
400
        "table.warehouse.help":"The root directory for data",

    
   
401
        "table.sql.label":"Table SQL statement",

    
   
402
        "table.sql.help":"SQL statement to process data in the remote database",

    
   
403
        "table.schemaName.help":"Schema name to process data in the remote database",

    
   
404
        "connection.jdbcProperties.label":"JDBC Connection Properties",

    
   
405
        "table.columns.help":"Specific columns of a table name or a table SQL",

    
   
406
        "connection.connectionString.help":"Enter the value of JDBC connection string to be used by this connector for creating connections.",

    
   
407
        "table.dataDirectory.help":"The sub-directory under warehouse for data",

    
   
408
        "table.schemaName.label":"Schema name",

    
   
409
        "connection.connectionString.label":"JDBC Connection String",

    
   
410
        "connection.help":"You must supply the information requested in order to create a connection object.",

    
   
411
        "connection.password.help":"Enter the password to be used for connecting to the database.",

    
   
412
        "table.tableName.label":"Table name"

    
   
413
      }

    
   
414
    },

    
   
415
    "all":[

    
   
416
      {

    
   
417
        "id":1,

    
   
418
        "name":"generic-jdbc-connector",

    
   
419
        "class":"org.apache.sqoop.connector.jdbc.GenericJdbcConnector",

    
   
420
        "job-forms":{

    
   
421
          "IMPORT":[

    
   
422
            {

    
   
423
              "id":2,

    
   
424
              "inputs":[

    
   
425
                {

    
   
426
                  "id":6,

    
   
427
                  "name":"table.schemaName",

    
   
428
                  "type":"STRING",

    
   
429
                  "size":50,

    
   
430
                  "sensitive":false

    
   
431
                },

    
   
432
                {

    
   
433
                  "id":7,

    
   
434
                  "name":"table.tableName",

    
   
435
                  "type":"STRING",

    
   
436
                  "size":50,

    
   
437
                  "sensitive":false

    
   
438
                },

    
   
439
                {

    
   
440
                  "id":8,

    
   
441
                  "name":"table.sql",

    
   
442
                  "type":"STRING",

    
   
443
                  "size":2000,

    
   
444
                  "sensitive":false

    
   
445
                },

    
   
446
                {

    
   
447
                  "id":9,

    
   
448
                  "name":"table.columns",

    
   
449
                  "type":"STRING",

    
   
450
                  "size":50,

    
   
451
                  "sensitive":false

    
   
452
                },

    
   
453
                {

    
   
454
                  "id":10,

    
   
455
                  "name":"table.partitionColumn",

    
   
456
                  "type":"STRING",

    
   
457
                  "size":50,

    
   
458
                  "sensitive":false

    
   
459
                },

    
   
460
                {

    
   
461
                  "id":11,

    
   
462
                  "name":"table.partitionColumnNull",

    
   
463
                  "type":"BOOLEAN",

    
   
464
                  "sensitive":false

    
   
465
                },

    
   
466
                {

    
   
467
                  "id":12,

    
   
468
                  "name":"table.boundaryQuery",

    
   
469
                  "type":"STRING",

    
   
470
                  "size":50,

    
   
471
                  "sensitive":false

    
   
472
                }

    
   
473
              ],

    
   
474
              "name":"table",

    
   
475
              "type":"CONNECTION"

    
   
476
            }

    
   
477
          ],

    
   
478
          "EXPORT":[

    
   
479
            {

    
   
480
              "id":3,

    
   
481
              "inputs":[

    
   
482
                {

    
   
483
                  "id":13,

    
   
484
                  "name":"table.schemaName",

    
   
485
                  "type":"STRING",

    
   
486
                  "size":50,

    
   
487
                  "sensitive":false

    
   
488
                },

    
   
489
                {

    
   
490
                  "id":14,

    
   
491
                  "name":"table.tableName",

    
   
492
                  "type":"STRING",

    
   
493
                  "size":2000,

    
   
494
                  "sensitive":false

    
   
495
                },

    
   
496
                {

    
   
497
                  "id":15,

    
   
498
                  "name":"table.sql",

    
   
499
                  "type":"STRING",

    
   
500
                  "size":50,

    
   
501
                  "sensitive":false

    
   
502
                },

    
   
503
                {

    
   
504
                  "id":16,

    
   
505
                  "name":"table.columns",

    
   
506
                  "type":"STRING",

    
   
507
                  "size":50,

    
   
508
                  "sensitive":false

    
   
509
                }

    
   
510
              ],

    
   
511
              "name":"table",

    
   
512
              "type":"CONNECTION"

    
   
513
            }

    
   
514
          ]

    
   
515
        },

    
   
516
        "con-forms":[

    
   
517
          {

    
   
518
            "id":1,

    
   
519
            "inputs":[

    
   
520
              {

    
   
521
                "id":1,

    
   
522
                "name":"connection.jdbcDriver",

    
   
523
                "type":"STRING",

    
   
524
                "size":128,

    
   
525
                "sensitive":false

    
   
526
              },

    
   
527
              {

    
   
528
                "id":2,

    
   
529
                "name":"connection.connectionString",

    
   
530
                "type":"STRING",

    
   
531
                "size":128,

    
   
532
                "sensitive":false

    
   
533
              },

    
   
534
              {

    
   
535
                "id":3,

    
   
536
                "name":"connection.username",

    
   
537
                "type":"STRING",

    
   
538
                "size":40,

    
   
539
                "sensitive":false

    
   
540
              },

    
   
541
              {

    
   
542
                "id":4,

    
   
543
                "name":"connection.password",

    
   
544
                "type":"STRING",

    
   
545
                "size":40,

    
   
546
                "sensitive":true

    
   
547
              },

    
   
548
              {

    
   
549
                "id":5,

    
   
550
                "name":"connection.jdbcProperties",

    
   
551
                "type":"MAP",

    
   
552
                "sensitive":false

    
   
553
              }

    
   
554
            ],

    
   
555
            "name":"connection",

    
   
556
            "type":"CONNECTION"

    
   
557
          }

    
   
558
        ],

    
   
559
        "version":"2.0.0-SNAPSHOT"

    
   
560
      }

    
   
561
    ]

    
   
562
  }

    
   
563

   

    
   
564

   

    
   
565
/v1/framework - [GET]- Get Sqoop Framework

    
   
566
-----------------------------------------------

    
   
567

   

    
   
568
Retrieve all metadata of Sqoop framework. The metadata include all the form fields that are required to all Sqoop objects, such as connection and jobs.

    
   
569

   

    
   
570
* Method: ``GET``

    
   
571
* Format: ``JSON``

    
   
572
* Request Content: ``None``

    
   
573
* Fields of Response:

    
   
574

   

    
   
575
+--------------------------+----------------------------------------------------------------------------------------------------+

    
   
576
|   Field                  | Description                                                                                        |

    
   
577
+==========================+====================================================================================================+

    
   
578
| ``id``                   | The id for Sqoop framework (It should be always be 1, since there is always 1 framework out there) |

    
   
579
+--------------------------+----------------------------------------------------------------------------------------------------+

    
   
580
| ``resources``            | All resources for Sqoop framework                                                                  |

    
   
581
+--------------------------+----------------------------------------------------------------------------------------------------+

    
   
582
| ``framework-version``    | The version of Sqoop framework                                                                     |

    
   
583
+--------------------------+----------------------------------------------------------------------------------------------------+

    
   
584
| ``job-forms``            | Framework's Job Configuration forms                                                                |

    
   
585
+--------------------------+----------------------------------------------------------------------------------------------------+

    
   
586
| ``con-forms``            | Framework's connection configuration forms                                                         |

    
   
587
+--------------------------+----------------------------------------------------------------------------------------------------+

    
   
588

   

    
   
589
The framework and connector might contain several forms to be filled for job or connection object. Many parameters for job and connection

    
   
590
are categorize into different classes, which are know as forms. Each form has its id and name. In job and connection objects, they use

    
   
591
the id of the form to track these parameter inputs.

    
   
592

   

    
   
593
* Response Example:

    
   
594

   

    
   
595
::

    
   
596

   

    
   
597
  {

    
   
598
    "id":1,

    
   
599
    "resources":{

    
   
600
      "output.label":"Output configuration",

    
   
601
      "security.maxConnections.help":"Maximal number of connections that this connection object can use at one point in time",

    
   
602
      "output.storageType.label":"Storage type",

    
   
603
      "output.ignored.help":"This value is ignored",

    
   
604
      "input.label":"Input configuration",

    
   
605
      "security.help":"You must supply the information requested in order to create a job object.",

    
   
606
      "output.storageType.help":"Target on Hadoop ecosystem where to store data",

    
   
607
      "input.inputDirectory.help":"Directory that should be exported",

    
   
608
      "output.outputFormat.label":"Output format",

    
   
609
      "output.ignored.label":"Ignored",

    
   
610
      "output.outputFormat.help":"Format in which data should be serialized",

    
   
611
      "output.help":"You must supply the information requested in order to get information where you want to store your data.",

    
   
612
      "throttling.help":"Set throttling boundaries to not overload your systems",

    
   
613
      "input.inputDirectory.label":"Input directory",

    
   
614
      "throttling.loaders.label":"Loaders",

    
   
615
      "input.help":"Specifies information required to get data from Hadoop ecosystem",

    
   
616
      "throttling.extractors.label":"Extractors",

    
   
617
      "throttling.extractors.help":"Number of extractors that Sqoop will use",

    
   
618
      "security.label":"Security related configuration options",

    
   
619
      "throttling.label":"Throttling resources",

    
   
620
      "throttling.loaders.help":"Number of loaders that Sqoop will use",

    
   
621
      "output.outputDirectory.help":"Output directory for final data",

    
   
622
      "security.maxConnections.label":"Max connections",

    
   
623
      "output.outputDirectory.label":"Output directory"

    
   
624
    },

    
   
625
    "framework-version":"1",

    
   
626
    "job-forms":{

    
   
627
      "IMPORT":[

    
   
628
        {

    
   
629
          "id":5,

    
   
630
          "inputs":[

    
   
631
            {

    
   
632
              "id":18,

    
   
633
              "values":"HDFS",

    
   
634
              "name":"output.storageType",

    
   
635
              "type":"ENUM",

    
   
636
              "sensitive":false

    
   
637
            },

    
   
638
            {

    
   
639
              "id":19,

    
   
640
              "values":"TEXT_FILE,SEQUENCE_FILE",

    
   
641
              "name":"output.outputFormat",

    
   
642
              "type":"ENUM",

    
   
643
              "sensitive":false

    
   
644
            },

    
   
645
            {

    
   
646
              "id":20,

    
   
647
              "name":"output.outputDirectory",

    
   
648
              "type":"STRING",

    
   
649
              "size":255,

    
   
650
              "sensitive":false

    
   
651
            }

    
   
652
          ],

    
   
653
          "name":"output",

    
   
654
          "type":"CONNECTION"

    
   
655
        },

    
   
656
        {

    
   
657
          "id":6,

    
   
658
          "inputs":[

    
   
659
            {

    
   
660
              "id":21,

    
   
661
              "name":"throttling.extractors",

    
   
662
              "type":"INTEGER",

    
   
663
              "sensitive":false

    
   
664
            },

    
   
665
            {

    
   
666
              "id":22,

    
   
667
              "name":"throttling.loaders",

    
   
668
              "type":"INTEGER",

    
   
669
              "sensitive":false

    
   
670
            }

    
   
671
          ],

    
   
672
          "name":"throttling",

    
   
673
          "type":"CONNECTION"

    
   
674
        }

    
   
675
      ],

    
   
676
      "EXPORT":[

    
   
677
        {

    
   
678
          "id":7,

    
   
679
          "inputs":[

    
   
680
            {

    
   
681
              "id":23,

    
   
682
              "name":"input.inputDirectory",

    
   
683
              "type":"STRING",

    
   
684
              "size":255,

    
   
685
              "sensitive":false

    
   
686
            }

    
   
687
          ],

    
   
688
          "name":"input",

    
   
689
          "type":"CONNECTION"

    
   
690
        },

    
   
691
        {

    
   
692
          "id":8,

    
   
693
          "inputs":[

    
   
694
            {

    
   
695
              "id":24,

    
   
696
              "name":"throttling.extractors",

    
   
697
              "type":"INTEGER",

    
   
698
              "sensitive":false

    
   
699
            },

    
   
700
            {

    
   
701
              "id":25,

    
   
702
              "name":"throttling.loaders",

    
   
703
              "type":"INTEGER",

    
   
704
              "sensitive":false

    
   
705
            }

    
   
706
          ],

    
   
707
          "name":"throttling",

    
   
708
          "type":"CONNECTION"

    
   
709
        }

    
   
710
      ]

    
   
711
    },

    
   
712
    "con-forms":[

    
   
713
      {

    
   
714
        "id":4,

    
   
715
        "inputs":[

    
   
716
          {

    
   
717
            "id":17,

    
   
718
            "name":"security.maxConnections",

    
   
719
            "type":"INTEGER",

    
   
720
            "sensitive":false

    
   
721
          }

    
   
722
        ],

    
   
723
        "name":"security",

    
   
724
        "type":"CONNECTION"

    
   
725
      }

    
   
726
    ]

    
   
727
  }

    
   
728

   

    
   
729
/v1/connection/[xid] - [GET] - Get Connection

    
   
730
----------------------------------------------------

    
   
731

   

    
   
732
Retrieve all the metadata of a given connection, such as its values for different fields of connector form and sqoop framework form.

    
   
733

   

    
   
734
Provide the id of the connector in the url [xid] part. If you provide ``all`` in the [xid] part in the url, you will get the metadata of all connections.

    
   
735

   

    
   
736
* Method: ``GET``

    
   
737
* Format: ``JSON``

    
   
738
* Request Content: ``None``

    
   
739
* Fields of Response:

    
   
740

   

    
   
741
+--------------------------+---------------------------------------------------------------------------------------+

    
   
742
|   Field                  | Description                                                                           |

    
   
743
+==========================+=======================================================================================+

    
   
744
| ``resources-connector``  | All resources for the given connector                                                 |

    
   
745
+--------------------------+---------------------------------------------------------------------------------------+

    
   
746
| ``resources-framework``  | All resources related with Sqoop framework                                            |

    
   
747
+--------------------------+---------------------------------------------------------------------------------------+

    
   
748
| ``all``                  | All metadata about the given connection, such as id, name and all form input values   |

    
   
749
+--------------------------+---------------------------------------------------------------------------------------+

    
   
750

   

    
   
751
* Response Example:

    
   
752

   

    
   
753
::

    
   
754

   

    
   
755
  {

    
   
756
    "resources-connector":{

    
   
757
      "1":{

    
   
758
        "ignored.label":"Ignored",

    
   
759
        "table.partitionColumn.help":"A specific column for data partition",

    
   
760
        "table.label":"Database configuration",

    
   
761
        "table.boundaryQuery.label":"Boundary query",

    
   
762
        "ignored.help":"This is completely ignored",

    
   
763
        "ignored.ignored.label":"Ignored",

    
   
764
        "connection.jdbcProperties.help":"Enter any JDBC properties that should be supplied during the creation of connection.",

    
   
765
        "table.tableName.help":"Table name to process data in the remote database",

    
   
766
        "connection.jdbcDriver.label":"JDBC Driver Class",

    
   
767
        "connection.username.help":"Enter the username to be used for connecting to the database.",

    
   
768
        "table.help":"You must supply the information requested in order to create a job object.",

    
   
769
        "table.partitionColumn.label":"Partition column name",

    
   
770
        "ignored.ignored.help":"This is completely ignored",

    
   
771
        "table.partitionColumnNull.label":"Nulls in partition column",

    
   
772
        "table.warehouse.label":"Data warehouse",

    
   
773
        "table.boundaryQuery.help":"The boundary query for data partition",

    
   
774
        "connection.username.label":"Username",

    
   
775
        "connection.jdbcDriver.help":"Enter the fully qualified class name of the JDBC driver that will be used for establishing this connection.",

    
   
776
        "connection.label":"Connection configuration",

    
   
777
        "table.columns.label":"Table column names",

    
   
778
        "table.dataDirectory.label":"Data directory",

    
   
779
        "table.partitionColumnNull.help":"Whether there are null values in partition column",

    
   
780
        "connection.password.label":"Password",

    
   
781
        "table.warehouse.help":"The root directory for data",

    
   
782
        "table.sql.label":"Table SQL statement",

    
   
783
        "table.sql.help":"SQL statement to process data in the remote database",

    
   
784
        "table.schemaName.help":"Schema name to process data in the remote database",

    
   
785
        "connection.jdbcProperties.label":"JDBC Connection Properties",

    
   
786
        "table.columns.help":"Specific columns of a table name or a table SQL",

    
   
787
        "connection.connectionString.help":"Enter the value of JDBC connection string to be used by this connector for creating connections.",

    
   
788
        "table.dataDirectory.help":"The sub-directory under warehouse for data",

    
   
789
        "table.schemaName.label":"Schema name",

    
   
790
        "connection.connectionString.label":"JDBC Connection String",

    
   
791
        "connection.help":"You must supply the information requested in order to create a connection object.",

    
   
792
        "connection.password.help":"Enter the password to be used for connecting to the database.",

    
   
793
        "table.tableName.label":"Table name"

    
   
794
      }

    
   
795
    },

    
   
796
    "resources-framework":{

    
   
797
      "output.label":"Output configuration",

    
   
798
      "security.maxConnections.help":"Maximal number of connections that this connection object can use at one point in time",

    
   
799
      "output.storageType.label":"Storage type",

    
   
800
      "output.ignored.help":"This value is ignored",

    
   
801
      "input.label":"Input configuration",

    
   
802
      "security.help":"You must supply the information requested in order to create a job object.",

    
   
803
      "output.storageType.help":"Target on Hadoop ecosystem where to store data",

    
   
804
      "input.inputDirectory.help":"Directory that should be exported",

    
   
805
      "output.outputFormat.label":"Output format",

    
   
806
      "output.ignored.label":"Ignored",

    
   
807
      "output.outputFormat.help":"Format in which data should be serialized",

    
   
808
      "output.help":"You must supply the information requested in order to get information where you want to store your data.",

    
   
809
      "throttling.help":"Set throttling boundaries to not overload your systems",

    
   
810
      "input.inputDirectory.label":"Input directory",

    
   
811
      "throttling.loaders.label":"Loaders",

    
   
812
      "input.help":"Specifies information required to get data from Hadoop ecosystem",

    
   
813
      "throttling.extractors.label":"Extractors",

    
   
814
      "throttling.extractors.help":"Number of extractors that Sqoop will use",

    
   
815
      "security.label":"Security related configuration options",

    
   
816
      "throttling.label":"Throttling resources",

    
   
817
      "throttling.loaders.help":"Number of loaders that Sqoop will use",

    
   
818
      "output.outputDirectory.help":"Output directory for final data",

    
   
819
      "security.maxConnections.label":"Max connections",

    
   
820
      "output.outputDirectory.label":"Output directory"

    
   
821
    },

    
   
822
    "all":[

    
   
823
      {

    
   
824
        "id":1,

    
   
825
        "enabled":true,

    
   
826
        "updated":1375912819893,

    
   
827
        "created":1375912819893,

    
   
828
        "name":"First connection",

    
   
829
        "connector":[

    
   
830
          {

    
   
831
            "id":1,

    
   
832
            "inputs":[

    
   
833
              {

    
   
834
                "id":1,

    
   
835
                "name":"connection.jdbcDriver",

    
   
836
                "value":"com.mysql.jdbc.Driver",

    
   
837
                "type":"STRING",

    
   
838
                "size":128,

    
   
839
                "sensitive":false

    
   
840
              },

    
   
841
              {

    
   
842
                "id":2,

    
   
843
                "name":"connection.connectionString",

    
   
844
                "value":"jdbc%3Amysql%3A%2F%2Flocalhost%2Ftest",

    
   
845
                "type":"STRING",

    
   
846
                "size":128,

    
   
847
                "sensitive":false

    
   
848
              },

    
   
849
              {

    
   
850
                "id":3,

    
   
851
                "name":"connection.username",

    
   
852
                "value":"root",

    
   
853
                "type":"STRING",

    
   
854
                "size":40,

    
   
855
                "sensitive":false

    
   
856
              },

    
   
857
              {

    
   
858
                "id":4,

    
   
859
                "name":"connection.password",

    
   
860
                "type":"STRING",

    
   
861
                "size":40,

    
   
862
                "sensitive":true

    
   
863
              },

    
   
864
              {

    
   
865
                "id":5,

    
   
866
                "name":"connection.jdbcProperties",

    
   
867
                "type":"MAP",

    
   
868
                "sensitive":false

    
   
869
              }

    
   
870
            ],

    
   
871
            "name":"connection",

    
   
872
            "type":"CONNECTION"

    
   
873
          }

    
   
874
        ],

    
   
875
        "connector-id":1,

    
   
876
        "framework":[

    
   
877
          {

    
   
878
            "id":4,

    
   
879
            "inputs":[

    
   
880
              {

    
   
881
                "id":17,

    
   
882
                "name":"security.maxConnections",

    
   
883
                "value":"10",

    
   
884
                "type":"INTEGER",

    
   
885
                "sensitive":false

    
   
886
              }

    
   
887
            ],

    
   
888
            "name":"security",

    
   
889
            "type":"CONNECTION"

    
   
890
          }

    
   
891
        ]

    
   
892
      }

    
   
893
    ]

    
   
894
  }

    
   
895

   

    
   
896
/v1/connection - [POST] - Create Connection

    
   
897
---------------------------------------------------------

    
   
898

   

    
   
899
Create a new connection object. Try your best to provide values for as many as inputs of

    
   
900
connection forms from both connectors and framework.

    
   
901

   

    
   
902
* Method: ``POST``

    
   
903
* Format: ``JSON``

    
   
904
* Fields of Request:

    
   
905

   

    
   
906
+--------------------------+--------------------------------------------------------------------------------------+

    
   
907
|   Field                  | Description                                                                          |

    
   
908
+==========================+======================================================================================+

    
   
909
| ``all``                  | Request array, in which each element is an independent request                       |

    
   
910
+--------------------------+--------------------------------------------------------------------------------------+

    
   
911
| ``id``                   | The id of the connection. Useless here, because we don't know the id before creation |

    
   
912
+--------------------------+--------------------------------------------------------------------------------------+

    
   
913
| ``enabled``              | Whether to enable this connection (true/false)                                       |

    
   
914
+--------------------------+--------------------------------------------------------------------------------------+

    
   
915
| ``updated``              | The last updated time of this connection                                             |

    
   
916
+--------------------------+--------------------------------------------------------------------------------------+

    
   
917
| ``created``              | The creation time of this connection                                                 |

    
   
918
+--------------------------+--------------------------------------------------------------------------------------+

    
   
919
| ``name``                 | The name of this connection                                                          |

    
   
920
+--------------------------+--------------------------------------------------------------------------------------+

    
   
921
| ``connector``            | Filled inputs for connector forms for this connection                                |

    
   
922
+--------------------------+--------------------------------------------------------------------------------------+

    
   
923
| ``connector-id``         | The id of the connector used for this connection                                     |

    
   
924
+--------------------------+--------------------------------------------------------------------------------------+

    
   
925
| ``framework``            | Filled inputs for framework forms for this connection                                |

    
   
926
+--------------------------+--------------------------------------------------------------------------------------+

    
   
927

   

    
   
928

   

    
   
929
* Request Example:

    
   
930

   

    
   
931
::

    
   
932

   

    
   
933
  {

    
   
934
    "all":[

    
   
935
      {

    
   
936
        "id":-1,

    
   
937
        "enabled":true,

    
   
938
        "updated":1375919952017,

    
   
939
        "created":1375919952017,

    
   
940
        "name":"First connection",

    
   
941
        "connector":[

    
   
942
          {

    
   
943
            "id":1,

    
   
944
            "inputs":[

    
   
945
              {

    
   
946
                "id":1,

    
   
947
                "name":"connection.jdbcDriver",

    
   
948
                "value":"com.mysql.jdbc.Driver",

    
   
949
                "type":"STRING",

    
   
950
                "size":128,

    
   
951
                "sensitive":false

    
   
952
              },

    
   
953
              {

    
   
954
                "id":2,

    
   
955
                "name":"connection.connectionString",

    
   
956
                "value":"jdbc%3Amysql%3A%2F%2Flocalhost%2Ftest",

    
   
957
                "type":"STRING",

    
   
958
                "size":128,

    
   
959
                "sensitive":false

    
   
960
              },

    
   
961
              {

    
   
962
                "id":3,

    
   
963
                "name":"connection.username",

    
   
964
                "value":"root",

    
   
965
                "type":"STRING",

    
   
966
                "size":40,

    
   
967
                "sensitive":false

    
   
968
              },

    
   
969
              {

    
   
970
                "id":4,

    
   
971
                "name":"connection.password",

    
   
972
                "type":"STRING",

    
   
973
                "size":40,

    
   
974
                "sensitive":true

    
   
975
              },

    
   
976
              {

    
   
977
                "id":5,

    
   
978
                "name":"connection.jdbcProperties",

    
   
979
                "type":"MAP",

    
   
980
                "sensitive":false

    
   
981
              }

    
   
982
            ],

    
   
983
            "name":"connection",

    
   
984
            "type":"CONNECTION"

    
   
985
          }

    
   
986
        ],

    
   
987
        "connector-id":1,

    
   
988
        "framework":[

    
   
989
          {

    
   
990
            "id":4,

    
   
991
            "inputs":[

    
   
992
              {

    
   
993
                "id":17,

    
   
994
                "name":"security.maxConnections",

    
   
995
                "value":"10",

    
   
996
                "type":"INTEGER",

    
   
997
                "sensitive":false

    
   
998
              }

    
   
999
            ],

    
   
1000
            "name":"security",

    
   
1001
            "type":"CONNECTION"

    
   
1002
          }

    
   
1003
        ]

    
   
1004
      }

    
   
1005
    ]

    
   
1006
  }

    
   
1007

   

    
   
1008
* Fields of Response:

    
   
1009

   

    
   
1010
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1011
|   Field                  | Description                                                                          |

    
   
1012
+==========================+======================================================================================+

    
   
1013
| ``id``                   | The id assigned for this new created connection                                      |

    
   
1014
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1015
| ``connector``            | The validation status for the inputs of connector forms, provided by the request     |

    
   
1016
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1017
| ``framework``            | The validation status for the inputs of connector forms, provided by the request     |

    
   
1018
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1019

   

    
   
1020
* Response Example:

    
   
1021

   

    
   
1022
::

    
   
1023

   

    
   
1024
  {

    
   
1025
    "id":1,

    
   
1026
    "connector":{

    
   
1027
      "status":"FINE",

    
   
1028
      "messages":{

    
   
1029

   

    
   
1030
      }

    
   
1031
    },

    
   
1032
    "framework":{

    
   
1033
      "status":"FINE",

    
   
1034
      "messages":{

    
   
1035

   

    
   
1036
      }

    
   
1037
    }

    
   
1038
  }

    
   
1039

   

    
   
1040
/v1/connection/[xid] - [PUT] - Update Connection

    
   
1041
---------------------------------------------------------

    
   
1042

   

    
   
1043
Update an existing connection object with id [xid]. To make the procedure of filling inputs easier, the general practice

    
   
1044
is get the connection first and then change some of the inputs.

    
   
1045

   

    
   
1046
* Method: ``PUT``

    
   
1047
* Format: ``JSON``

    
   
1048
* Fields of Request:

    
   
1049

   

    
   
1050
The same as Create Connection.

    
   
1051

   

    
   
1052
* Fields of Response:

    
   
1053

   

    
   
1054
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1055
|   Field                  | Description                                                                          |

    
   
1056
+==========================+======================================================================================+

    
   
1057
| ``connector``            | The validation status for the inputs of connector forms, provided by the request     |

    
   
1058
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1059
| ``framework``            | The validation status for the inputs of connector forms, provided by the request     |

    
   
1060
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1061

   

    
   
1062
* Response Example:

    
   
1063

   

    
   
1064
::

    
   
1065

   

    
   
1066
  {

    
   
1067
    "connector":{

    
   
1068
      "status":"FINE",

    
   
1069
      "messages":{

    
   
1070

   

    
   
1071
      }

    
   
1072
    },

    
   
1073
    "framework":{

    
   
1074
      "status":"FINE",

    
   
1075
      "messages":{

    
   
1076

   

    
   
1077
      }

    
   
1078
    }

    
   
1079
  }

    
   
1080

   

    
   
1081
/v1/connection/[xid] - [DELETE] - Delete Connection

    
   
1082
---------------------------------------------------------

    
   
1083

   

    
   
1084
Delete a connection with id ``xid``.

    
   
1085

   

    
   
1086
* Method: ``DELETE``

    
   
1087
* Format: ``JSON``

    
   
1088
* Request Content: ``None``

    
   
1089
* Response Content: ``None``

    
   
1090

   

    
   
1091
/v1/connection/[xid]/enable - [PUT] - Enable Connection

    
   
1092
---------------------------------------------------------

    
   
1093

   

    
   
1094
Enable a connection with id ``xid``.

    
   
1095

   

    
   
1096
* Method: ``PUT``

    
   
1097
* Format: ``JSON``

    
   
1098
* Request Content: ``None``

    
   
1099
* Response Content: ``None``

    
   
1100

   

    
   
1101
/v1/connection/[xid]/disable - [PUT] - Disable Connection

    
   
1102
---------------------------------------------------------

    
   
1103

   

    
   
1104
Disable a connection with id ``xid``.

    
   
1105

   

    
   
1106
* Method: ``PUT``

    
   
1107
* Format: ``JSON``

    
   
1108
* Request Content: ``None``

    
   
1109
* Response Content: ``None``

    
   
1110

   

    
   
1111
/v1/job/[jid] - [GET] - Get Job

    
   
1112
----------------------------------------

    
   
1113

   

    
   
1114
Retrieve all the metadata of a given job, such as its values for different fields of connector form and sqoop framework form.

    
   
1115

   

    
   
1116
Provide the id of the job in the url [jid] part. If you provide ``all`` in the [jid] part in the url, you will get the metadata of all connections.

    
   
1117

   

    
   
1118
* Method: ``GET``

    
   
1119
* Format: ``JSON``

    
   
1120
* Request Content: ``None``

    
   
1121
* Fields of Response:

    
   
1122

   

    
   
1123
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1124
|   Field                  | Description                                                                          |

    
   
1125
+==========================+======================================================================================+

    
   
1126
| ``resources-connector``  | All resources for the given connector                                                |

    
   
1127
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1128
| ``resources-framework``  | All resources related with Sqoop framework                                           |

    
   
1129
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1130
| ``all``                  | All metadata about the given job, such as id, name and all form input values         |

    
   
1131
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1132

   

    
   
1133
* Response Example:

    
   
1134

   

    
   
1135
::

    
   
1136

   

    
   
1137
  {

    
   
1138
    "resources-connector":{

    
   
1139
      "1":{

    
   
1140
        "ignored.label":"Ignored",

    
   
1141
        "table.partitionColumn.help":"A specific column for data partition",

    
   
1142
        "table.label":"Database configuration",

    
   
1143
        "table.boundaryQuery.label":"Boundary query",

    
   
1144
        "ignored.help":"This is completely ignored",

    
   
1145
        "ignored.ignored.label":"Ignored",

    
   
1146
        "connection.jdbcProperties.help":"Enter any JDBC properties that should be supplied during the creation of connection.",

    
   
1147
        "table.tableName.help":"Table name to process data in the remote database",

    
   
1148
        "connection.jdbcDriver.label":"JDBC Driver Class",

    
   
1149
        "connection.username.help":"Enter the username to be used for connecting to the database.",

    
   
1150
        "table.help":"You must supply the information requested in order to create a job object.",

    
   
1151
        "table.partitionColumn.label":"Partition column name",

    
   
1152
        "ignored.ignored.help":"This is completely ignored",

    
   
1153
        "table.partitionColumnNull.label":"Nulls in partition column",

    
   
1154
        "table.warehouse.label":"Data warehouse",

    
   
1155
        "table.boundaryQuery.help":"The boundary query for data partition",

    
   
1156
        "connection.username.label":"Username",

    
   
1157
        "connection.jdbcDriver.help":"Enter the fully qualified class name of the JDBC driver that will be used for establishing this connection.",

    
   
1158
        "connection.label":"Connection configuration",

    
   
1159
        "table.columns.label":"Table column names",

    
   
1160
        "table.dataDirectory.label":"Data directory",

    
   
1161
        "table.partitionColumnNull.help":"Whether there are null values in partition column",

    
   
1162
        "connection.password.label":"Password",

    
   
1163
        "table.warehouse.help":"The root directory for data",

    
   
1164
        "table.sql.label":"Table SQL statement",

    
   
1165
        "table.sql.help":"SQL statement to process data in the remote database",

    
   
1166
        "table.schemaName.help":"Schema name to process data in the remote database",

    
   
1167
        "connection.jdbcProperties.label":"JDBC Connection Properties",

    
   
1168
        "table.columns.help":"Specific columns of a table name or a table SQL",

    
   
1169
        "connection.connectionString.help":"Enter the value of JDBC connection string to be used by this connector for creating connections.",

    
   
1170
        "table.dataDirectory.help":"The sub-directory under warehouse for data",

    
   
1171
        "table.schemaName.label":"Schema name",

    
   
1172
        "connection.connectionString.label":"JDBC Connection String",

    
   
1173
        "connection.help":"You must supply the information requested in order to create a connection object.",

    
   
1174
        "connection.password.help":"Enter the password to be used for connecting to the database.",

    
   
1175
        "table.tableName.label":"Table name"

    
   
1176
      }

    
   
1177
    },

    
   
1178
    "resources-framework":{

    
   
1179
      "output.label":"Output configuration",

    
   
1180
      "security.maxConnections.help":"Maximal number of connections that this connection object can use at one point in time",

    
   
1181
      "output.storageType.label":"Storage type",

    
   
1182
      "output.ignored.help":"This value is ignored",

    
   
1183
      "input.label":"Input configuration",

    
   
1184
      "security.help":"You must supply the information requested in order to create a job object.",

    
   
1185
      "output.storageType.help":"Target on Hadoop ecosystem where to store data",

    
   
1186
      "input.inputDirectory.help":"Directory that should be exported",

    
   
1187
      "output.outputFormat.label":"Output format",

    
   
1188
      "output.ignored.label":"Ignored",

    
   
1189
      "output.outputFormat.help":"Format in which data should be serialized",

    
   
1190
      "output.help":"You must supply the information requested in order to get information where you want to store your data.",

    
   
1191
      "throttling.help":"Set throttling boundaries to not overload your systems",

    
   
1192
      "input.inputDirectory.label":"Input directory",

    
   
1193
      "throttling.loaders.label":"Loaders",

    
   
1194
      "input.help":"Specifies information required to get data from Hadoop ecosystem",

    
   
1195
      "throttling.extractors.label":"Extractors",

    
   
1196
      "throttling.extractors.help":"Number of extractors that Sqoop will use",

    
   
1197
      "security.label":"Security related configuration options",

    
   
1198
      "throttling.label":"Throttling resources",

    
   
1199
      "throttling.loaders.help":"Number of loaders that Sqoop will use",

    
   
1200
      "output.outputDirectory.help":"Output directory for final data",

    
   
1201
      "security.maxConnections.label":"Max connections",

    
   
1202
      "output.outputDirectory.label":"Output directory"

    
   
1203
    },

    
   
1204
    "all":[

    
   
1205
      {

    
   
1206
        "connection-id":1,

    
   
1207
        "id":1,

    
   
1208
        "enabled":true,

    
   
1209
        "updated":1375913231253,

    
   
1210
        "created":1375913231253,

    
   
1211
        "name":"First job",

    
   
1212
        "connector":[

    
   
1213
          {

    
   
1214
            "id":2,

    
   
1215
            "inputs":[

    
   
1216
              {

    
   
1217
                "id":6,

    
   
1218
                "name":"table.schemaName",

    
   
1219
                "value":"test",

    
   
1220
                "type":"STRING",

    
   
1221
                "size":50,

    
   
1222
                "sensitive":false

    
   
1223
              },

    
   
1224
              {

    
   
1225
                "id":7,

    
   
1226
                "name":"table.tableName",

    
   
1227
                "value":"example",

    
   
1228
                "type":"STRING",

    
   
1229
                "size":50,

    
   
1230
                "sensitive":false

    
   
1231
              },

    
   
1232
              {

    
   
1233
                "id":8,

    
   
1234
                "name":"table.sql",

    
   
1235
                "type":"STRING",

    
   
1236
                "size":2000,

    
   
1237
                "sensitive":false

    
   
1238
              },

    
   
1239
              {

    
   
1240
                "id":9,

    
   
1241
                "name":"table.columns",

    
   
1242
                "value":"id%2Cdata",

    
   
1243
                "type":"STRING",

    
   
1244
                "size":50,

    
   
1245
                "sensitive":false

    
   
1246
              },

    
   
1247
              {

    
   
1248
                "id":10,

    
   
1249
                "name":"table.partitionColumn",

    
   
1250
                "value":"id",

    
   
1251
                "type":"STRING",

    
   
1252
                "size":50,

    
   
1253
                "sensitive":false

    
   
1254
              },

    
   
1255
              {

    
   
1256
                "id":11,

    
   
1257
                "name":"table.partitionColumnNull",

    
   
1258
                "value":"true",

    
   
1259
                "type":"BOOLEAN",

    
   
1260
                "sensitive":false

    
   
1261
              },

    
   
1262
              {

    
   
1263
                "id":12,

    
   
1264
                "name":"table.boundaryQuery",

    
   
1265
                "type":"STRING",

    
   
1266
                "size":50,

    
   
1267
                "sensitive":false

    
   
1268
              }

    
   
1269
            ],

    
   
1270
            "name":"table",

    
   
1271
            "type":"CONNECTION"

    
   
1272
          }

    
   
1273
        ],

    
   
1274
        "connector-id":1,

    
   
1275
        "type":"IMPORT",

    
   
1276
        "framework":[

    
   
1277
          {

    
   
1278
            "id":5,

    
   
1279
            "inputs":[

    
   
1280
              {

    
   
1281
                "id":18,

    
   
1282
                "values":"HDFS",

    
   
1283
                "name":"output.storageType",

    
   
1284
                "value":"HDFS",

    
   
1285
                "type":"ENUM",

    
   
1286
                "sensitive":false

    
   
1287
              },

    
   
1288
              {

    
   
1289
                "id":19,

    
   
1290
                "values":"TEXT_FILE,SEQUENCE_FILE",

    
   
1291
                "name":"output.outputFormat",

    
   
1292
                "value":"TEXT_FILE",

    
   
1293
                "type":"ENUM",

    
   
1294
                "sensitive":false

    
   
1295
              },

    
   
1296
              {

    
   
1297
                "id":20,

    
   
1298
                "name":"output.outputDirectory",

    
   
1299
                "value":"%2Ftmp%2Foutput",

    
   
1300
                "type":"STRING",

    
   
1301
                "size":255,

    
   
1302
                "sensitive":false

    
   
1303
              }

    
   
1304
            ],

    
   
1305
            "name":"output",

    
   
1306
            "type":"CONNECTION"

    
   
1307
          },

    
   
1308
          {

    
   
1309
            "id":6,

    
   
1310
            "inputs":[

    
   
1311
              {

    
   
1312
                "id":21,

    
   
1313
                "name":"throttling.extractors",

    
   
1314
                "type":"INTEGER",

    
   
1315
                "sensitive":false

    
   
1316
              },

    
   
1317
              {

    
   
1318
                "id":22,

    
   
1319
                "name":"throttling.loaders",

    
   
1320
                "type":"INTEGER",

    
   
1321
                "sensitive":false

    
   
1322
              }

    
   
1323
            ],

    
   
1324
            "name":"throttling",

    
   
1325
            "type":"CONNECTION"

    
   
1326
          }

    
   
1327
        ]

    
   
1328
      }

    
   
1329
    ]

    
   
1330
  }

    
   
1331

   

    
   
1332
/v1/job - [POST] - Create Job

    
   
1333
---------------------------------------------------------

    
   
1334

   

    
   
1335
Create a new job object. Try your best to provide values for as many as inputs of

    
   
1336
job forms from both connectors and framework.

    
   
1337

   

    
   
1338
* Method: ``POST``

    
   
1339
* Format: ``JSON``

    
   
1340
* Fields of Request:

    
   
1341

   

    
   
1342
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1343
|   Field                  | Description                                                                          |

    
   
1344
+==========================+======================================================================================+

    
   
1345
| ``all``                  | Request array, in which each element is an independent request                       |

    
   
1346
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1347
| ``connection-id``        | The id of the connection used for this job                                           |

    
   
1348
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1349
| ``id``                   | The id of the job. Useless here, because we don't know the id before creation        |

    
   
1350
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1351
| ``type``                 | The type of this job ("IMPORT"/"EXPORT")                                             |

    
   
1352
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1353
| ``enabled``              | Whether to enable this connection (true/false)                                       |

    
   
1354
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1355
| ``updated``              | The last updated time of this connection                                             |

    
   
1356
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1357
| ``created``              | The creation time of this connection                                                 |

    
   
1358
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1359
| ``name``                 | The name of this connection                                                          |

    
   
1360
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1361
| ``connector``            | Filled inputs for connector forms for this connection                                |

    
   
1362
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1363
| ``connector-id``         | The id of the connector used for this connection                                     |

    
   
1364
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1365
| ``framework``            | Filled inputs for framework forms for this connection                                |

    
   
1366
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1367

   

    
   
1368

   

    
   
1369
* Request Example:

    
   
1370

   

    
   
1371
::

    
   
1372

   

    
   
1373
  {

    
   
1374
    "all":[

    
   
1375
      {

    
   
1376
        "connection-id":1,

    
   
1377
        "id":-1,

    
   
1378
        "enabled":true,

    
   
1379
        "updated":1375920083970,

    
   
1380
        "created":1375920083970,

    
   
1381
        "name":"First job",

    
   
1382
        "connector":[

    
   
1383
          {

    
   
1384
            "id":2,

    
   
1385
            "inputs":[

    
   
1386
              {

    
   
1387
                "id":6,

    
   
1388
                "name":"table.schemaName",

    
   
1389
                "value":"test",

    
   
1390
                "type":"STRING",

    
   
1391
                "size":50,

    
   
1392
                "sensitive":false

    
   
1393
              },

    
   
1394
              {

    
   
1395
                "id":7,

    
   
1396
                "name":"table.tableName",

    
   
1397
                "value":"example",

    
   
1398
                "type":"STRING",

    
   
1399
                "size":50,

    
   
1400
                "sensitive":false

    
   
1401
              },

    
   
1402
              {

    
   
1403
                "id":8,

    
   
1404
                "name":"table.sql",

    
   
1405
                "type":"STRING",

    
   
1406
                "size":2000,

    
   
1407
                "sensitive":false

    
   
1408
              },

    
   
1409
              {

    
   
1410
                "id":9,

    
   
1411
                "name":"table.columns",

    
   
1412
                "value":"id%2Cdata",

    
   
1413
                "type":"STRING",

    
   
1414
                "size":50,

    
   
1415
                "sensitive":false

    
   
1416
              },

    
   
1417
              {

    
   
1418
                "id":10,

    
   
1419
                "name":"table.partitionColumn",

    
   
1420
                "value":"id",

    
   
1421
                "type":"STRING",

    
   
1422
                "size":50,

    
   
1423
                "sensitive":false

    
   
1424
              },

    
   
1425
              {

    
   
1426
                "id":11,

    
   
1427
                "name":"table.partitionColumnNull",

    
   
1428
                "value":"true",

    
   
1429
                "type":"BOOLEAN",

    
   
1430
                "sensitive":false

    
   
1431
              },

    
   
1432
              {

    
   
1433
                "id":12,

    
   
1434
                "name":"table.boundaryQuery",

    
   
1435
                "type":"STRING",

    
   
1436
                "size":50,

    
   
1437
                "sensitive":false

    
   
1438
              }

    
   
1439
            ],

    
   
1440
            "name":"table",

    
   
1441
            "type":"CONNECTION"

    
   
1442
          }

    
   
1443
        ],

    
   
1444
        "connector-id":1,

    
   
1445
        "type":"IMPORT",

    
   
1446
        "framework":[

    
   
1447
          {

    
   
1448
            "id":5,

    
   
1449
            "inputs":[

    
   
1450
              {

    
   
1451
                "id":18,

    
   
1452
                "values":"HDFS",

    
   
1453
                "name":"output.storageType",

    
   
1454
                "value":"HDFS",

    
   
1455
                "type":"ENUM",

    
   
1456
                "sensitive":false

    
   
1457
              },

    
   
1458
              {

    
   
1459
                "id":19,

    
   
1460
                "values":"TEXT_FILE,SEQUENCE_FILE",

    
   
1461
                "name":"output.outputFormat",

    
   
1462
                "value":"TEXT_FILE",

    
   
1463
                "type":"ENUM",

    
   
1464
                "sensitive":false

    
   
1465
              },

    
   
1466
              {

    
   
1467
                "id":20,

    
   
1468
                "name":"output.outputDirectory",

    
   
1469
                "value":"%2Ftmp%2Foutput",

    
   
1470
                "type":"STRING",

    
   
1471
                "size":255,

    
   
1472
                "sensitive":false

    
   
1473
              }

    
   
1474
            ],

    
   
1475
            "name":"output",

    
   
1476
            "type":"CONNECTION"

    
   
1477
          },

    
   
1478
          {

    
   
1479
            "id":6,

    
   
1480
            "inputs":[

    
   
1481
              {

    
   
1482
                "id":21,

    
   
1483
                "name":"throttling.extractors",

    
   
1484
                "type":"INTEGER",

    
   
1485
                "sensitive":false

    
   
1486
              },

    
   
1487
              {

    
   
1488
                "id":22,

    
   
1489
                "name":"throttling.loaders",

    
   
1490
                "type":"INTEGER",

    
   
1491
                "sensitive":false

    
   
1492
              }

    
   
1493
            ],

    
   
1494
            "name":"throttling",

    
   
1495
            "type":"CONNECTION"

    
   
1496
          }

    
   
1497
        ]

    
   
1498
      }

    
   
1499
    ]

    
   
1500
  }

    
   
1501

   

    
   
1502
* Fields of Response:

    
   
1503

   

    
   
1504
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1505
|   Field                  | Description                                                                          |

    
   
1506
+==========================+======================================================================================+

    
   
1507
| ``id``                   | The id assigned for this new created job                                             |

    
   
1508
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1509
| ``connector``            | The validation status for the inputs of connector forms, provided by the request     |

    
   
1510
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1511
| ``framework``            | The validation status for the inputs of framework forms, provided by the request     |

    
   
1512
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1513

   

    
   
1514
* Response Example:

    
   
1515

   

    
   
1516
::

    
   
1517

   

    
   
1518
  {

    
   
1519
    "id":1,

    
   
1520
    "connector":{

    
   
1521
      "status":"FINE",

    
   
1522
      "messages":{

    
   
1523

   

    
   
1524
      }

    
   
1525
    },

    
   
1526
    "framework":{

    
   
1527
      "status":"FINE",

    
   
1528
      "messages":{

    
   
1529

   

    
   
1530
      }

    
   
1531
    }

    
   
1532
  }

    
   
1533

   

    
   
1534
/v1/job/[jid] - [PUT] - Update Job

    
   
1535
---------------------------------------------------------

    
   
1536

   

    
   
1537
Update an existing job object with id [jid]. To make the procedure of filling inputs easier, the general practice

    
   
1538
is get the existing job object first and then change some of the inputs.

    
   
1539

   

    
   
1540
* Method: ``PUT``

    
   
1541
* Format: ``JSON``

    
   
1542
* Fields of Request:

    
   
1543

   

    
   
1544
The same as Create Job.

    
   
1545

   

    
   
1546
* Fields of Response:

    
   
1547

   

    
   
1548
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1549
|   Field                  | Description                                                                          |

    
   
1550
+==========================+======================================================================================+

    
   
1551
| ``connector``            | The validation status for the inputs of connector forms, provided by the request     |

    
   
1552
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1553
| ``framework``            | The validation status for the inputs of framework forms, provided by the request     |

    
   
1554
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1555

   

    
   
1556
* Response Example:

    
   
1557

   

    
   
1558
::

    
   
1559

   

    
   
1560
  {

    
   
1561
    "connector":{

    
   
1562
      "status":"FINE",

    
   
1563
      "messages":{

    
   
1564

   

    
   
1565
      }

    
   
1566
    },

    
   
1567
    "framework":{

    
   
1568
      "status":"FINE",

    
   
1569
      "messages":{

    
   
1570

   

    
   
1571
      }

    
   
1572
    }

    
   
1573
  }

    
   
1574

   

    
   
1575
/v1/job/[jid] - [DELETE] - Delete Job

    
   
1576
---------------------------------------------------------

    
   
1577

   

    
   
1578
Delete a job with id ``jid``.

    
   
1579

   

    
   
1580
* Method: ``DELETE``

    
   
1581
* Format: ``JSON``

    
   
1582
* Request Content: ``None``

    
   
1583
* Response Content: ``None``

    
   
1584

   

    
   
1585
/v1/job/[jid]/enable - [PUT] - Enable Job

    
   
1586
---------------------------------------------------------

    
   
1587

   

    
   
1588
Enable a job with id ``jid``.

    
   
1589

   

    
   
1590
* Method: ``PUT``

    
   
1591
* Format: ``JSON``

    
   
1592
* Request Content: ``None``

    
   
1593
* Response Content: ``None``

    
   
1594

   

    
   
1595
/v1/job/[jid]/disable - [PUT] - Disable Job

    
   
1596
---------------------------------------------------------

    
   
1597

   

    
   
1598
Disable a job with id ``jid``.

    
   
1599

   

    
   
1600
* Method: ``PUT``

    
   
1601
* Format: ``JSON``

    
   
1602
* Request Content: ``None``

    
   
1603
* Response Content: ``None``

    
   
1604

   

    
   
1605
/v1/submission/history/[jid] - [GET] - Get Job Submission History

    
   
1606
----------------------------------------------------------------------

    
   
1607

   

    
   
1608
Retrieve all job submission history of a given job, such as the status, counters and links for those submissions.

    
   
1609

   

    
   
1610
Provide the id of the job in the url [jid] part. If you provide ``all`` in the [jid] part in the url, you will get all job submission history of all jobs.

    
   
1611

   

    
   
1612
* Method: ``GET``

    
   
1613
* Format: ``JSON``

    
   
1614
* Request Content: ``None``

    
   
1615
* Fields of Response:

    
   
1616

   

    
   
1617
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1618
|   Field                  | Description                                                                          |

    
   
1619
+==========================+======================================================================================+

    
   
1620
| ``progress``             | The progress of the running Sqoop job                                                |

    
   
1621
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1622
| ``job``                  | The id of the Sqoop job                                                              |

    
   
1623
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1624
| ``creation-date``        | The submission timestamp                                                             |

    
   
1625
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1626
| ``last-update-date``     | The timestamp of the last status update                                              |

    
   
1627
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1628
| ``status``               | The status of this job submission                                                    |

    
   
1629
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1630
| ``external-id``          | The job id of Sqoop job running on Hadoop                                            |

    
   
1631
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1632
| ``external-link``        | The link to track the job status on Hadoop                                           |

    
   
1633
+--------------------------+--------------------------------------------------------------------------------------+

    
   
1634

   

    
   
1635
* Response Example:

    
   
1636

   

    
   
1637
::

    
   
1638

   

    
   
1639
  {

    
   
1640
    "all":[

    
   
1641
      {

    
   
1642
        "progress":-1.0,

    
   
1643
        "last-update-date":1375913666476,

    
   
1644
        "external-id":"job_201307221513_0009",

    
   
1645
        "status":"SUCCEEDED",

    
   
1646
        "job":1,

    
   
1647
        "creation-date":1375913630576,

    
   
1648
        "external-link":"http:\/\/localhost:50030\/jobdetails.jsp?jobid=job_201307221513_0009",

    
   
1649
        "counters":{

    
   
1650
          "org.apache.hadoop.mapreduce.JobCounter":{

    
   
1651
            "SLOTS_MILLIS_MAPS":59135,

    
   
1652
            "FALLOW_SLOTS_MILLIS_REDUCES":0,

    
   
1653
            "FALLOW_SLOTS_MILLIS_MAPS":0,

    
   
1654
            "TOTAL_LAUNCHED_MAPS":2,

    
   
1655
            "SLOTS_MILLIS_REDUCES":0

    
   
1656
          },

    
   
1657
          "org.apache.hadoop.mapreduce.TaskCounter":{

    
   
1658
            "MAP_INPUT_RECORDS":0,

    
   
1659
            "PHYSICAL_MEMORY_BYTES":231583744,

    
   
1660
            "SPILLED_RECORDS":0,

    
   
1661
            "COMMITTED_HEAP_BYTES":112721920,

    
   
1662
            "CPU_MILLISECONDS":20940,

    
   
1663
            "VIRTUAL_MEMORY_BYTES":1955266560,

    
   
1664
            "SPLIT_RAW_BYTES":223,

    
   
1665
            "MAP_OUTPUT_RECORDS":5

    
   
1666
          },

    
   
1667
          "org.apache.hadoop.mapreduce.FileSystemCounter":{

    
   
1668
            "FILE_WRITE_OPS":0,

    
   
1669
            "FILE_READ_OPS":0,

    
   
1670
            "FILE_LARGE_READ_OPS":0,

    
   
1671
            "FILE_BYTES_READ":0,

    
   
1672
            "HDFS_BYTES_READ":223,

    
   
1673
            "FILE_BYTES_WRITTEN":386286,

    
   
1674
            "HDFS_LARGE_READ_OPS":0,

    
   
1675
            "HDFS_WRITE_OPS":2,

    
   
1676
            "HDFS_READ_OPS":3,

    
   
1677
            "HDFS_BYTES_WRITTEN":72

    
   
1678
          },

    
   
1679
          "org.apache.sqoop.submission.counter.SqoopCounters":{

    
   
1680
            "ROWS_READ":5

    
   
1681
          }

    
   
1682
        }

    
   
1683
      },

    
   
1684
      {

    
   
1685
        "progress":-1.0,

    
   
1686
        "last-update-date":1375913554412,

    
   
1687
        "external-id":"job_201307221513_0008",

    
   
1688
        "status":"SUCCEEDED",

    
   
1689
        "job":1,

    
   
1690
        "creation-date":1375913501078,

    
   
1691
        "external-link":"http:\/\/localhost:50030\/jobdetails.jsp?jobid=job_201307221513_0008",

    
   
1692
        "counters":{

    
   
1693
          "org.apache.hadoop.mapreduce.JobCounter":{

    
   
1694
            "SLOTS_MILLIS_MAPS":54905,

    
   
1695
            "FALLOW_SLOTS_MILLIS_REDUCES":0,

    
   
1696
            "FALLOW_SLOTS_MILLIS_MAPS":0,

    
   
1697
            "TOTAL_LAUNCHED_MAPS":2,

    
   
1698
            "SLOTS_MILLIS_REDUCES":0

    
   
1699
          },

    
   
1700
          "org.apache.hadoop.mapreduce.TaskCounter":{

    
   
1701
            "MAP_INPUT_RECORDS":0,

    
   
1702
            "PHYSICAL_MEMORY_BYTES":218865664,

    
   
1703
            "SPILLED_RECORDS":0,

    
   
1704
            "COMMITTED_HEAP_BYTES":112918528,

    
   
1705
            "CPU_MILLISECONDS":2550,

    
   
1706
            "VIRTUAL_MEMORY_BYTES":1955266560,

    
   
1707
            "SPLIT_RAW_BYTES":223,

    
   
1708
            "MAP_OUTPUT_RECORDS":5

    
   
1709
          },

    
   
1710
          "org.apache.hadoop.mapreduce.FileSystemCounter":{

    
   
1711
            "FILE_WRITE_OPS":0,

    
   
1712
            "FILE_READ_OPS":0,

    
   
1713
            "FILE_LARGE_READ_OPS":0,

    
   
1714
            "FILE_BYTES_READ":0,

    
   
1715
            "HDFS_BYTES_READ":223,

    
   
1716
            "FILE_BYTES_WRITTEN":387362,

    
   
1717
            "HDFS_LARGE_READ_OPS":0,

    
   
1718
            "HDFS_WRITE_OPS":2,

    
   
1719
            "HDFS_READ_OPS":2,

    
   
1720
            "HDFS_BYTES_WRITTEN":72

    
   
1721
          },

    
   
1722
          "org.apache.sqoop.submission.counter.SqoopCounters":{

    
   
1723
            "ROWS_READ":5

    
   
1724
          }

    
   
1725
        }

    
   
1726
      }

    
   
1727
    ]

    
   
1728
  }

    
   
1729

   

    
   
1730
/v1/submission/action/[jid] - [GET]- Get The Latest Submission Of A Given Job

    
   
1731
---------------------------------------------------------------------------------

    
   
1732

   

    
   
1733
Retrieve the latest submission of a given job, such as the status, counters and links for those submissions.

    
   
1734

   

    
   
1735
This function is similar to ``/v1/submission/history/[jid]`` except that it always return one submission object.

    
   
1736

   

    
   
1737
* Method: ``GET``

    
   
1738
* Format: ``JSON``

    
   
1739
* Request Content: ``None``

    
   
1740
* Fields of Response:

    
   
1741

   

    
   
1742
The same as ``/v1/submission/history/[jid]``

    
   
1743

   

    
   
1744
* Response Example:

    
   
1745

   

    
   
1746
The same as ``/v1/submission/history/[jid]``

    
   
1747

   

    
   
1748
/v1/submission/action/[jid] - [POST]- Submit Job

    
   
1749
---------------------------------------------------------------------------------

    
   
1750

   

    
   
1751
Submit a job with id ``[jid]`` to make it run.

    
   
1752

   

    
   
1753
* Method: ``POST``

    
   
1754
* Format: ``JSON``

    
   
1755
* Request Content: ``None``

    
   
1756
* Response Content: ``None``

    
   
1757

   

    
   
1758
/v1/submission/action/[jid] - [DELETE]- Stop Job

    
   
1759
---------------------------------------------------------------------------------

    
   
1760

   

    
   
1761
Stop a job with id ``[jid]``.

    
   
1762

   

    
   
1763
* Method: ``DELETE``

    
   
1764
* Format: ``JSON``

    
   
1765
* Request Content: ``None``

    
   
1766
* Response Content: ``None``
docs/src/site/sphinx/index.rst
Revision 6b03e62 New Change
 
  1. docs/src/site/sphinx/RESTAPI.rst: Loading...
  2. docs/src/site/sphinx/index.rst: Loading...