Get Started with Team-Based Labeling

The Image Labeler app enables you to create an individual or team labeling project. The choice of which project to create depends on whether you are working in collaboration with other individuals to perform the tasks involved in a labeling project.

Individual Project — The project contains all the labels and images needed for your labeling project, and there is no need to distribute or track the project status. To get started using an individual labeling project, see Get Started with the Image Labeler.
Team Project — Work in collaboration with other individuals to perform tasks such as labeling, reviewing labeled data, sending feedback, and tracking the progress of the project. Once all the tasks are complete, you can combine the labeled images from the various tasks and export them into a ground truth object.

Team-based Project: Video Walkthrough and Step-by-Step Guide

This video provides a walkthrough describing how to set up a team-based project in the Image Labeler app.

In this video, we will show you how to set up a team-based labeling project in the Image Labeler app. The team-based mode of the Labeler app is very useful when you're working with multiple people to label and review a large number of images. In this video, we'll set up a project in the Labeler using a charming collection of photos featuring the pets of MathWorkers.

Before we jump into the app, let's look at the different responsibilities on a labeling team. There is a project owner who creates the project. The owner makes the project shareable, creates label definitions, and creates and distributes tasks to the team. Tasks are distributed to the labelers that consist of one or more people who use the predefined label definitions to annotate a set of images according to the task instructions. Finally, the labeled images are passed to reviewers who can accept, reject, or correct them.

Using a team-based project in the Image Labeler means that you can create and distribute tasks to multiple people on your team, all in one app. The process is iterative and versatile, and especially helpful when you have a large number of images to label. The workflow involved in using the multi-user mode is straightforward. The project owner sets up the project, creates label definitions and creates annotation and review tasks.

Task owners annotate the images assigned to them. The project owner can then create review tasks, where the reviewers can fix, reject, or accept the labeled images. If there are rejected images, it gets returned to the task owner who can then fix the rejected images and send for review.

At any time, the project owner can view the progress of the project. And whenever labeled images are accepted-- either a subset of them or all of them-- the project owner can choose to export the labels to a ground truth object. Now let's jump into the app.

In MATLAB, you can launch the Image Labeler app from the app's library or at the command line. By clicking the Apps tab, you can find the Image Labeler app listed in the Image Processing and Computer Vision section, or typing Image Labeler in the search box. Here, I'll launch it from the command line by just typing Image Labeler.

I'll choose New Team Project, and the app opens in the Team Project app environment. First I'll import my collection of images to be labeled by clicking Import and then From File, and I'll select all the images in this folder. Here are all the images loaded into the app for the project.

First I'm going to create the labels which I'll include in this project for my team to use for their tasks. I'll create three labels-- cat, dog and other. I'll select Add Label from the toolbar, and then I'll choose Rectangles, which is the label shape I'll use for all three label definitions.

First one here I named Dog and click OK. You'll see the labels appear on the left as they are created. Next, I'll do the same and name the label Cat.

Details on all the fields you see, and tutorials on labeling itself, is available in our doc. I'm just going to focus on the aspects of setting up a project. Now I have the last label named Other, and all three are listed.

Next, we'll establish team members. I'll create three. After selecting Add Team Members, I'll type each name in the field at the top of the dialog box and click the Add button after each one. You'll notice the names get populated in the table. Now I'll X out.

Now I want to create tasks for two of the people on the team. To do this, I need to create a task for each one of them. I also want to assign the number of images to label as evenly as possible between them. I'll select Add Label Task, where there are two options-- one to manually create a task, and the other one to do the image assignment for me by assigning the images between tasks automatically. That's the one I'll select.

I set the number of label tasks to 2, and I'm going to distribute all the images in the collection-- which here is 57. So when I click Create Task, you'll see the image distribution summary displayed where one task owner will get one extra image to label. You'll see on the left, I now have the two tasks listed ready to be named and assigned an owner. I'll select the first task, give it a name and assign it to an owner.

The image is listed as already pre-populated having used the auto distribution option. I'll do the same for the other task. I'll add an instruction and I'll copy the instruction on the other one.

Now I have one task assigned to Robin and the other to Kelly. To see a list of thumbnails assigned to either task, you can use the dropdown image filters list. The same image cannot be assigned to more than one task. Here, I selected Robin's task, and you see it's about half. Each task owner can only modify the images assigned to their task.

The Image Labeler app uses badges overlaid on the thumbnails to indicate the status of an image. Select the Image Status Legend to display the relevant badges and definitions that correspond to the task in process. The project owner will see all the badges, as you see here.

Next, I need to publish the tasks which will make them available to task owners. I could select Publish Single Tasks or All unpublished ones. This also saves the project.

This dialog box appears, which conveniently provides the path to copy the location of the saved project, which is handy to have and use to send to task owners. If the people who will be labeling your images do not have a MATLAB license, you can compile an executable of the Labeler app and send it to them along with the project. I'm not covering it in this video, but the link in this dialog box goes to our doc where you will find the details on how to create that executable. You can always get back to this dialog box from the Published Tasks dropdown menu. I see here the status of the tasks are now published.

After launching the app and opening the project file, the task owner selects their name from the dropdown menu and clicks Open. The app opens with the images and label definitions loaded. All I need to do is check for any task instructions and get going.

I'll select the appropriate label definition from the ROI Label Definition Panel, which is located on the left, and Label Images. You'll notice that in either the task owner mode for labeling or reviewing, only the options of the app are available to that task. This reduces potential errors or mishaps within the project.

The Project Owner mode can access all functionality. Autosave is on by default, which saves the annotations after I'm done annotating an image. However, the autosave can cause delays, so you can disable it under the Save Task dropdown. I can go back to modify any of the images, mark an image as labeled or unlabeled to modify it again.

I've annotated a few images here using the Dog, Cat and Other definitions. I'll send the ones I've done for review and get back to others another time. To do this, select Send For Review. This message comes up just to let me the status, and to confirm what I'm doing, and reminding me that the images I annotated will be locked once I send them.

You'll notice badges appear on the thumbnails as you go through the workflow. As I said earlier, you can always check the definitions by selecting the Image Status legend from the toolbar. That's it for the task owner. Everything gets saved in the project file and folder.

Now we'll create a review task. It's basically the same process as creating review tasks. After launching the app as the project owner, I'll see that there are some images to be reviewed, which we see are marked with a green badge.

Now we'll select Add Review Task, and you'll see a task get added to the Tasks list and the dialogue to create it in the Task Information panel. Here I'll give it a name, Owner, and instructions in the same way I did for the annotation tasks. And then I'll add which images I want reviewed in this task. Then I'll publish the tasks, choosing All Unpublished. The same dialog pops up, which provides the path to the project file, and a helpful link to the doc which explains the details of this step.

Opening the app as the reviewer is the same process as the others. Launch the app, open the project file, and select your name from the dropdown list under Task Owner. The app opens with whatever images are ready for review shown in the image browser.

The right-hand side panel provides Accept and Reject options. The large buttons are convenient for accepting or rejecting all the labels of the specific image you're reviewing. Alternatively, you can accept or reject individual labels in an image using the Radio buttons in the list.

The first image here with three dogs looks all set, so I click the Accept Image button. The reviewer can also correct, or modify, or fix a label. I'll modify this label to include the tail.

This one is clearly marked wrong. It must have been its disguise. I could fix it, but I'll reject the image instead. The others look good, and so I accept them and then send the reviewed images by selecting Send Review Results. Now these images are locked.

Now the labeling task owner will receive the rejected image back, ready to be labeled correctly. I'll open the app as that task owner and I see all the images that were accepted or locked, and the one that was rejected needs to be labeled. I'll continue on labeling additional images as well.

You can label them in any order. You can mark them Labeled one by one, or multi-select them. Now the process repeats as they mark them for review.

Now I'm going to open the project as the owner, and I can see the progress. I can also decide to export the labels for whichever images have been successfully reviewed and accepted. I'll do this by selecting Export and Accepted Images. Name the object, and select to Export to a File or the Workspace. I'm choosing File, but I can always drop the file into the workspace.

Let's summarize what we did. First, to set up the project, I created label definitions. I chose only rectangles. I added task owners-- I had three.

And then I created two tasks, and assigned one each to two task owners. Finally, I published the tasks. And if I needed to, I could create an executable of the app to send to the team members on the project-- which is great, because it means full licenses are not required, which is very cost-effective.

Then we launched the Image Labelers as the Task Owner to annotate images by selecting the name from the Task Owner's startup menu. After labeling images, we sent them for review. Next, the project owner creates one or more review tasks-- we created One-For-One Reviewer. When an image is marked for review, you'll notice the badge changes. You can always check the legend for badge definitions by selecting Image Status Legend in the Resources section of the toolbar.

Next, the annotated images are reviewed, where one or more images can be rejected or accepted. The task owner who did the annotation will see rejected images in their task session. The owner follows the same workflow as before, marking the images ready for review. Images that are already accepted by the reviewer remain locked.

Finally, the project owner, aside from monitoring the project at any time, exports all or a subset of labels related to the accepted images to a ground truth object. It can be exported to a MAT file or to the workspace. The exported labels can now be used in your applications, such as training an object detector or a segmentation network. For more details about the app and how to use it, you can check out our documentation and examples.

Video Player is loading.

Current Time 0:00

Duration 0:00

Loaded: 0%

Stream Type LIVE

Remaining Time 0:00

These steps walk you through how to set up an image labeling project, create labels, assign and distribute images and labels between team members, review the labeled images, provide feedback, and combine the work to export the labeled images from the app. The steps also include how to track progress for all the labeling and review tasks.