Error Handling

Overview

This section explores strategies for identifying, surfacing, and recovering from errors within a workflow to maintain operational integrity and minimize downtime.

Executor Options

ExecutorOptions define how the workflow should behave during execution, this is currently scoped to behavior upon error. This allows you to control whether the workflow pauses for error handling or proceeds automatically, depending on the situation.

Here’s how to configure the executor options in your workflow definition:

workflow = Workflow(
    workcell=workcell,
    #...
    options=Options(
        executor=ExecutorOptions(
            behaviour_on_failure="pause"
        )
    )
)

class linq.workflow.ExecutorOptions(*, behaviour_on_failure: Literal['abort', 'pause'])

Options for the execution of a workflow.

behaviour_on_failure: Literal['abort', 'pause']: Behaviour of the system when a task fails. Defaults to cancelling the workflow.

The options for behaviour_on_failure will execute as follows:

"PAUSE": the workflow will stop at the point of failure, allowing an operator to manually decide the next steps. This is outlined below.
"ABORT": the workflow will immediately terminate when any task fails.

Tip

"PAUSE" is the recommended option.

Get Latest Error

This command retrieves the most recent error encountered by the workflow on the specified workcell. The resulting Error ID, can be used for mitigating the error through Error Run Controls below.

linq workcell error get-latest --workcell-id=<workcell-id>

This command will return the error ID and a list of labware ID’s affected by the error. Labwares affected are those whos labware flow is affected by the error.

Attention

The list of labware ID’s will only be returned as of the Novemeber release of LINQ on the 6th of Novemeber.

Error Run Controls

Once you have the Error ID, the next step is to decide how to respond to the error. You can use the CLI tool to interact with the error by adding respond to the following commands.

SKIP TASK: Bypass the current failed task and continue with the next one in the workflow. When skipping a transport task, be cautious as it skips both the robot and transport layer movements. Ensure that you manually place the labware on the next instrument if needed.
```
    linq workcell error respond --workcell-id= --error-id= --action= skip-task
```
RETRY TASK: Re-attempt the failed task. This option is focused mainly on scara and transport tasks. This is useful if the task is recoverable. If retry fails, you can manually move the plate to the next slot and Skip-TasK instead.
```
    linq workcell error respond --workcell-id= --error-id= --action= retry-task
```

ABORT LABWARE: Abort labware related to the error and continue execution for labware who’s flow is not affected by the error. Affected labware will not be continued and need to be cleared from the system before continuing.

Labware is considered affected when it is dependent on the labware journey of labware involved in the error, for example if a labware is used alongside an errored labware. Task dependencies do not influence affected labware.

    linq workcell error respond --workcell-id= --error-id= --action= abort-labware

This command will return the list of labware that is affected by the error and that needs to be removed from the workcell:

The following labware will be affected by the abort-labware action:
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ Labware ID ┃ Reason        ┃ Current Location     ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│ labware_1  │ some reason 1 │ instrument_x, slot 1 │
│ labware_2  │ some reason 2 │ instrument_y, slot 2 │
└────────────┴───────────────┴──────────────────────┘
Remove the labware from the workcell and then choose confirm, or choose cancel

This action can be confirmed or canceled. Canceling will not execute the abort-labware action or abort the run, other error handling actions can still be attempted after cancelling.

For a video tutorial on using abort-labware see the error handling tutorials page.

ABORT: Stop the current workflow and terminate execution entirely. This is the final option if the error is unrecoverable. You will be required to reset after aborting.
```
    linq workcell error respond --workcell-id= --error-id= --action= abort
```
RESET: This option resets the specific workcell, returning the workcell to its initial state. You will be required to abort the workflow before resetting the workcell.
```
    linq workcell error respond --workcell-id= --error-id= --action= reset
```

Available Failure Actions

The available_failure_actions parameter allows you to override default error handling behavior in both ActionTask and CodeTask.

By specifying options, you can control which recovery actions are available to the operator when a task fails.

wash_dilution_plate = ActionTask(
    ...
    available_failure_actions=AvailableFailureActions(
        complete_manually=True,
        retry=True,
        ignore=False,
        abort_labwares=True,
        advanced_options={
            "additional_labwares_to_abort": [
                "labware_id_1",
                "labware_id_2",
            ]
        }
    ),
)

Options

complete_manually: Allows manual completion of the task.
retry: Enables retrying the failed task.
ignore: Allows the workflow to continue without retrying or manual completion.
abort_labwares: Enables aborting labware affected by the failure.
advanced_options.additional_labwares_to_abort: Allows specifying additional labware IDs to abort if the task fails.

These options determine which actions are exposed to the operator when the workflow is paused on failure.