UiPath Documentation
ixp
latest
false
IXP in Flow user guide
  • Overview
  • Model building
    • Overview
    • Managing Extract nodes
    • Uploading sample documents​
    • Reviewing predictions
    • Validating predicted values
    • Model configuration
    • Bring your own subscription (BYOS)
  • Model evaluation
  • Model deployment
  • Consuming models
  • Developer
  • Frequently asked questions

Validating predicted values

Validate IXP model predictions in the Extract node of UiPath Maestro™ Flow to establish ground truth after refining prompt instructions, without directly training the model from annotations.

Overview​

The model does not currently learn directly from user annotations. To improve the performance of the model, modify your prompt instructions or model configuration.

Once you are happy with your prompt instructions, you can start validating the predictions of the model to establish ground truth for its performance evaluation. Before you do so, it is important to understand the differences between predictions and references.

Predictions and references​

Predictions are the predicted values, and are compared to ground truth.

Note:

Incorrect or missing predictions impact the model score and performance metrics in the Eval runs tab.

References are the matching references to the predicted values found in the document. The purpose of references is to help users find the right part of the document.

Note:

Incorrect or missing references do not impact the model score or performance metrics. However, it might be helpful to add or correct references to improve the human validation experience.

There are known limitations with references and highlighted spans, which are one of our key priorities to improve.

You might encounter one of the following scenarios:

  • Correct prediction and correct reference, but an incorrect span is highlighted. Usually, the first match that was found.
  • Correct prediction, but there is a missing reference and a highlighted span. In these scenarios, we provide a page number as a fallback.
Note:

Predictions and references are independent. This means that missing or incorrect references do not impact the predicted values.

Validating predicted values​

In the annotation view, each extracted value is highlighted in the document with the color of its field, as shown next to the field name in the annotation panel. Extracted values are highlighted as follows:

  • Filled - Extracted values. The fill is the same for confirmed and unconfirmed values. To check whether a value is confirmed, use its checkbox in the annotation panel: the checkbox is selected for confirmed values and cleared for unconfirmed values.

  • Solid border - The selected value. When you select a value in the annotation panel, its highlight in the document gets a solid border.

To validate the predicted values, follow these steps:

  1. Hover over the Flow Extract node, and select Open full view.

  2. In the Dataset tab, select Annotate for the document you want to validate. The model highlights the relevant span of text in the document and displays the generated predictions in the annotation panel. Values that are applicable to multiple instances of a field group, such as account number, have arrow buttons to help navigate to relevant sections. The buttons appear when you point to the values.

  3. Review the predictions and confirm the extracted values in one of the following ways:

    • To confirm a single value, select its checkbox.
    • To confirm an entire field group row, select the row checkbox.
    Note:

    Any changes you make but do not submit are saved as a draft, which you can return to and update later. For more details, check Draft annotations.

  4. Select Submit to submit your annotations.

You can configure additional fields in your extraction schema at any point from the Definition tab.

Example of validating predicted values in Action Center​

Stopping the validation​

It is recommended that you validate at least 10 examples for each field, so that the model can provide meaningful validation statistics.

Confirmed and unconfirmed references and values​

The validation process entails the following:

  • Confirming the extracted values if they are correct.
  • Annotating the expected values if the predicted values are incorrect.
  • Marking fields as missing if they are not present in the document.
  • Configuring any additional fields in your extraction schema.

This section shows what values and references look like in their confirmed state (validated) and unconfirmed state (not validated).

Note:

Any changes you make but do not submit are saved as a draft, which you can return to and update later. For more details, check Draft annotations.

Predicted values​

Unconfirmed values​

No values are marked as confirmed:

Figure 1. Unconfirmed values

Confirmed​

References are highlighted with a solid border:

Figure 2. Confirmed reference

Values are marked as confirmed:

Figure 3. Confirmed values

Validated values​

This section contains example scenarios of when the predictions of the model match or do not match the annotations of the user.

When the annotation matches the prediction, the value looks as follows:

Figure 4. Correct prediction

When the annotation does not match the prediction, the value looks as follows:

Figure 5. Incorrect prediction

Note:

Predictions are compared to annotations, not the references. References might be missing or incorrect, but this will not be flagged as a mismatch.

Annotating the expected value​

When a predicted value is incorrect, annotate the expected value. The annotation sets the ground truth, the true expected value, that evaluation runs compare the predictions of the model against. It does not change the model or its future predictions.

To annotate the expected value, proceed as follows:

  1. In the annotation panel, select the ellipsis for the field, and then select Change extracted value. For a field without a value, which shows Is missing, select the plus icon.

  2. Highlight the correct value in the document.

    Note:

    Any changes you make but do not submit are saved as a draft, which you can return to and update later. For more details, check Draft annotations.

Editing values​

In the annotation view of the Flow Extract node, edit the values as follows:

  1. Select Add value to create a new value with no reference in the document.

  2. Select the ellipsis for a field to use one of the following options:

    • Change extracted value
    • Revert to previous value
    • Mark as missing

  3. Select the ellipsis for a field group row to use one of the following options:

    • Highlight row
    • Delete row

  4. Confirm the values by selecting the checkbox of each value or field group row.

    Note:

    Any changes you make but do not submit are saved as a draft, which you can return to and update later. For more details, check Draft annotations.

To show each instance of a field group as a row in a table instead of a list in the annotation panel, go to the Definition tab, select the ellipsis for the field group, and turn on Display as a table.

Draft annotations​

Any edits you make to values and references on a document are stored as draft annotations until you submit them to the model as annotations. As a result, the document status appears as In progress, or as Annotated (Editing) for a document that was already annotated. To discard the draft, select Discard draft.

All of the previously mentioned actions, such as validating, annotating, and editing values, and then submitting the annotations, are kept in draft as you make changes.

If you refresh predictions while a document is in a draft state, the system preserves the draft annotations while repopulating new predictions, and matches the new set of predictions from the model to the draft annotations.

Was this page helpful?

Connect

Need help? Support

Want to learn? UiPath Academy

Have questions? UiPath Forum

Stay updated