Error Handling

Overview

This section explores strategies for identifying, surfacing, and recovering from errors within a workflow to maintain operational integrity and minimize downtime.

Executor Options

ExecutorOptions define how the workflow should behave during execution. This allows you to control whether the workflow pauses for error handling or proceeds automatically, depending on the situation.

Here’s how to configure the executor options in your workflow definition:

workflow = Workflow(
    workcell=workcell,
    #...
    options=Options(
        executor=ExecutorOptions(
            behaviour_on_failure="pause"
        )
    )
)

class linq.workflow.ExecutorOptions(*, behaviour_on_failure: Literal['abort', 'pause'])

Options for the execution of a workflow.

behaviour_on_failure: Literal['abort', 'pause']: Behaviour of the system when a task fails. Defaults to cancelling the workflow.

The options for behaviour_on_failure will execute as follows:

"pause": the workflow will stop at the point of failure, allowing an operator to manually decide the next steps. This is outlined below.
"abort": the workflow will immediately terminate when any task fails.

Tip

"pause" is the recommended option.

Additional Executor Options

Beyond behaviour_on_failure, ExecutorOptions provides additional configuration for advanced error handling and execution control:

auto_ignore_broken_time_constraints: When set to True, automatically ignores time constraints that are violated during execution. Defaults to False.
abort_labwares_aborts_entire_batch: When set to True, aborting a labware will also abort all other labware that share the same batch ID. Defaults to False.
replanning_options: Advanced configuration for replanning behavior, including thresholds for underrun/overrun and optimization parameters.

Example with additional options:

workflow = Workflow(
    workcell=workcell,
    #...
    options=Options(
        executor=ExecutorOptions(
            behaviour_on_failure="pause",
            auto_ignore_broken_time_constraints=True,
            abort_labwares_aborts_entire_batch=True,
        )
    )
)

Get Latest Error

This command retrieves the most recent error encountered by the workflow on the specified workcell. The resulting Error ID can be used to mitigate the error through Error Run Controls below.

linq workcell error get-latest --workcell-id=<workcell-id>

This command will return the error ID and a list of affected labware IDs.

Error Run Controls

Once you have the Error ID, the next step is to decide how to respond to the error. You can use the CLI tool to interact with the error by adding respond to the following commands.

SKIP TASK: Bypass the current failed task and continue with the next task in the workflow. When skipping a transport task, be cautious, as it skips both the robot and transport layer movements. Ensure that you manually place the labware on the next instrument if needed.
```
    linq workcell error respond --workcell-id= --error-id= --action= skip-task
```
RETRY TASK: Re-attempt the failed task. This option is mainly intended for SCARA and transport tasks. This is useful when the task is recoverable. If retry fails, you can manually move the plate to the next slot and useSkip-Task instead.
```
    linq workcell error respond --workcell-id= --error-id= --action= retry-task
```

ABORT LABWARE: Abort labware related to the error and continue execution for labware whose flow is not affected by the error. Affected labware will not continue and must be cleared from the system before continuing.

Labware is considered affected when it depends on the journey of labware involved in the error, for example if a labware is used alongside errored labware. Task dependencies do not influence affected labware.

    linq workcell error respond --workcell-id= --error-id= --action= abort-labware

This command will return a list of labware affected by the error that need to be removed from the workcell:

The following labware will be affected by the abort-labware action:
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ Labware ID ┃ Reason        ┃ Current Location     ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│ labware_1  │ some reason 1 │ instrument_x, slot 1 │
│ labware_2  │ some reason 2 │ instrument_y, slot 2 │
└────────────┴───────────────┴──────────────────────┘
Remove the labware from the workcell and then choose confirm, or choose cancel

This action can be confirmed or canceled. Canceling will not execute the abort-labware action or abort the run. Other error handling actions can still be attempted after canceling.

ABORT: Stop the current workflow and terminate execution entirely. This is the final option if the error is unrecoverable. You will be required to reset the workcell after aborting.
```
    linq workcell error respond --workcell-id= --error-id= --action= abort
```
RESET: This option resets the specific workcell, returning it to its initial state. You will be required to abort the workflow before resetting the workcell.
```
    linq workcell error respond --workcell-id= --error-id= --action= reset
```

Available Failure Actions

The available_failure_actions parameter allows you to override default error-handling behavior in both ActionTask and CodeTask.

By specifying options, you can control which recovery actions are available to the operator when a task fails.

from linq.schema.workflow_api import AdvancedOptions

wash_dilution_plate = ActionTask(
    ...
    available_failure_actions=AvailableFailureActions(
        complete_manually=True,
        retry=True,
        ignore=False,
        abort_labwares=True,
        advanced_options=AdvancedOptions(
            additional_labwares_to_abort=[
                "labware_id_1",
                "labware_id_2",
            ]
        )
    ),
)

Options

complete_manually: Allows manual completion of the task.
retry: Enables retrying the failed task.
ignore: Allows the workflow to continue without retrying or manual completion.
abort_labwares: Enables aborting labware affected by the failure.
advanced_options.additional_labwares_to_abort: Allows specifying additional labware IDs to abort if the task fails.

These options determine which actions are exposed to the operator when the workflow is paused on failure.