Review Process

Manual review is one of the most basic quality evaluation methods, in which all or a subset of the labeled data are directly inspected by human reviewers during the validation stage.
The review can be performed on the entire labeled dataset or a sample of the dataset. A full inspection can improve the quality of your dataset, but will require more time.
In the review stage, the reviewer evaluates labeled data and can approve or reject them. In addition, the Suite provides role-based access control for features such as issue tracking, labeling status management, and review workflow so that customers can build an optimal workflow and communicate effectively especially during the data inspection and correction stages.

1. Assignee — Submit/Skip Labels

After completing a labeling task in Label Mode, an Assignee submits the task or skips the label that is not suitable for labeling.

• When labeling, the label status is In Progress. When the label is submitted or skipped, the label status changes to Submitted · Skipped.

• The review round starts at the same time as the label submission and skip, and the review status changes to Pending.

  *Review round: A series of processes in which a labeler submits a label and a reviewer reviews the label to decide whether to approve or reject it is called a "review round."

2. Reviewer - Review Labels

2-1) Launch Review Mode

There are two different ways to start the review mode:

• Run the Review Mode by Clicking Review on the top of the project sidebar

In this mode, Full-screen Review Mode is launched that allows you to review labels that are pending review one by one.

• Click the Labels tab on the Projects sidebar menu.
• Click on the label (not the Edit Label button) in the Labels table.

Modal Review Mode is launched that allows you to review only that label.

2-2. After reviewing the label, you can select whether to approve or reject the label with the Approve and Reject buttons on the top right.

　1) Approve

• Approving a label means that it is suitable for model training.
• Once the label is approved, the review status changes to Approved, and the review of the label ends.

　2) Reject

• Rejecting a label means that it is not suitable for model training.
• The reviewer must enter the reason when rejecting the label. This reason is recorded in the Issue Thread.
• Rejecting the label will change the review status to Rejected, and the label will be re-assigned to the previous labeler. The label status is also changed to In Progress.

3. Labeler - Revise Rejected Labels

3-1. The rejected label will be re-assigned to the previous labeler.

• The labeler can modify the label by checking the reason for rejection via the issue thread.
• When the labeler starts the Label Mode, the rejected label is surfaced first.

3-2. Labelers must resubmit labels after editing.

• When the labeler submits the labels after editing, the next review round begins.