Communications Mining user guide

Introduction

Using the Check label and Missed label training modes is the part of the Refine phase where you try to identify any inconsistencies or missed labels in the messages that have already been reviewed. This is different to the Teach label step, which focuses on unreviewed messages that have predictions made by the platform, rather than assigned labels.

Check label shows you messages where the platform thinks the selected label may have been misapplied, that is, it potentially should not have been applied.

Missed label shows you messages that the platform thinks may be missing the selected label, that is, it potentially should have been applied but was not. In this case, the selected label will typically appear as a suggestion, as shown in the following image.

The suggestions from the platform in either mode are not necessarily correct, these are just the instances where the platform is unsure based on the training that's been completed so far. You can choose to ignore them if you disagree with the platform's suggestions after reviewing them.

Using these training modes is a very effective way of finding occurrences where the user may have not been consistent in applying labels. By using them you are able to correct these occasions and therefore improve the performance of the label.

When to use Check label and Missed label

The simplest answer of when to use either training mode is when they are one of the recommended actions in the Model Rating section or specific label view in the Validation page. For more details, check Understanding and improving model performance.

As a rule of thumb, any label that has a significant number of pinned examples but has low average precision, which can be indicated by red label warnings in the Validation page or in the label filter bars, will likely benefit from some corrective training in either Check label and Missed label mode.

When validating the performance of a model, the platform will determine whether it thinks a label has often been applied incorrectly, or where it thinks it's been regularly missed, and will prioritise whichever corrective action it thinks would be most beneficial for improving a label's performance.

Missed label is also a very useful tool if you have added a new label to an existing taxonomy with lots of reviewed examples. Once you've provided some initial examples for the new label concept, Missed label can quickly help you identify any examples in the previously reviewed messages where it should also apply. For more details, check Adding new labels to existing taxonomies.

Using Check label and Missed label

To reach either of these training modes, there are the following main options:

In each mode, the platform will show you up to 20 examples per page of reviewed messages where it thinks the selected label may have been applied incorrectly, Check label, or may be missing the selected label, Missed label.

Check label

In Check label, review each of the examples on the page to confirm that they are genuine examples of the selected label. If they are, move on without taking action. If they are not, remove the label by selecting the X button when hovering over it, and ensure you apply the correct labels instead.

Correcting labels added in error can have a major impact on the performance of a label, by ensuring that the model has correct and consistent examples from which to make predictions for that label.

Missed label

In Missed label, review each of the examples on the page to see whether the selected label has in fact been missed. If it has, select the label suggestion, as shown in the previous image, to apply the label. If it has not, ignore the suggestion and move on.

Just because the platform is suggesting a label on a reviewed message, does not mean the model considers it to be a prediction, nor will it count towards any statistics on the number of labels in a dataset. If a suggestion is wrong, you can just ignore it.

Partially annotated messages can be very detrimental to the ability of the model to predict a label, as when you do not apply a label to a message, you essentially tell the model that this is not an example of this label concept. If it is in fact a correct example, this can be very confusing for the model, particularly if there are other very similar examples that do have the label applied.

Adding labels that have been missed can therefore have a major impact on the performance of a label, by ensuring that the model has correct and consistent examples from which to make predictions for that label.