The Work of Orchestration: How Human Expertise Shapes AI Execution

Expertise can shape more than one case

Orchestration means arranging how work reaches an agreed result across the people and systems involved. Some coordination can itself be automated. The human contribution includes choosing the purpose, designing the arrangement, resolving decisions outside delegated authority and improving the operation. It does not require a person to issue a separate instruction for every system action.

An expert might identify that a familiar exception usually arises because a particular record is missing. The useful contribution could be a better intake requirement, a source check or a clearer routing rule. Where the next search depends on what has been found, an agent may help within the limits set out in RePerspective Labs' Bounded Autonomy framework. That framework specifies permitted actions, required evidence and conditions for human intervention. A recurring, predictable route may be handled adequately by an existing workflow.

These choices can make expertise available beyond the cases an individual handles personally. They can also spread an error. A mistaken instruction used repeatedly deserves as much attention as a mistaken judgment in one case. The design therefore needs an accountable maintainer, evidence of its effects and a way to revise it.

How expertise shapes many cases

Early research gives a useful, limited view of this changing work. A June 2026 exploratory preprint based on interviews with 17 experienced developers identified preparation of controls, planning with agents, monitoring execution and reviewing results as forms of oversight. The study concerns software development and does not establish an enterprise-wide productivity effect. It does show why an account of orchestration should include work before and during execution as well as checking afterward. [1]

Turn judgment into a maintained operating design

Expertise often appears as a short instruction that conceals considerable judgment: check the relevant policy, obtain the right evidence, or send the difficult case to a specialist. Making that instruction usable requires explaining what relevant means, which source applies and what should happen when the evidence disagrees.

The work begins with representative cases. Ask an experienced employee to explain what changes their decision, including a case that looks ordinary but should take a different route. Translate those distinctions into instructions, source requirements and tests. Keep unresolved judgments visible so an authorized person can address them. Some decisions will continue to depend on expertise that cannot responsibly be reduced to a reusable rule.

The result should be specific enough to inspect and maintain. An instruction needs a scope and an owner. A knowledge source needs an applicable version. A test needs an expected result and a reason that result is acceptable. Employees should contribute their knowledge through a supported process; they should not acquire responsibility for changing policy merely because they can edit a prompt.

A hypothetical claims evidence service

The following proposed design extends the claims-evidence setting in Workflow, Agent, or Both? It is an illustration, not a reported implementation, measured result or recommendation about a particular insurance jurisdiction.

The service prepares evidence for a defined class of claims. Success is a source-linked packet accepted as ready for substantive review by a designated claims specialist, acting for the Outcome owner. It does not decide coverage or settle the claim. Material omissions and record conflicts must be resolved before acceptance; recording an unresolved issue does not make the packet complete.

Here, an experienced claims employee helps define the evidence checklist, source precedence and cases that should stop preparation. The team tests those choices against ordinary files, missing records and contradictory facts. Its authority arrangements coexist within the same service.

Follow the exception into an approved improvement

One file contains two records with different dates of loss. The system preserves both sources and marks the packet incomplete. A named claims specialist investigates what evidence is needed to establish the date. The Operating owner remains responsible for the waiting queue and the service commitment; unresolved work cannot disappear into an exception inbox.

Once the conflict is resolved, the packet is checked again against the acceptance criteria. The receiving specialist records acceptance only when those criteria hold. Coverage and settlement remain separate decisions. If the conflict cannot be resolved within this service, the case stays incomplete and follows the designated specialist route.

Resolve the case and improve later work

Suppose several such cases reveal that an old intake form is still supplying an obsolete field. The immediate case decisions provide evidence for investigation. They do not automatically authorize a new general rule. The Knowledge owner checks the meaning and applicability of the proposed correction with the relevant policy authority. The Technology owner implements the approved source or validation change, and the Operating owner prepares its use and support.

Test the correction against the cases that exposed the problem and against legitimate variations it must preserve. A check that flags every difference could create unnecessary holds. If the change improves the result under the agreed criteria, release it through the approved change process and check its effects in operation.

The specialist has contributed to a result in the current case and to a maintained improvement for later cases. That second contribution becomes credible when its effect can be demonstrated.

Provide a dependable execution environment

The surrounding software must make the proposed human role workable. A person cannot judge an exception if the service cannot show the relevant sources, what has already happened and which decision is waiting. Nor can a well-written instruction compensate for an interface that permits an unauthorized action.

Data supplies records and facts. Context connects the material that applies to the current case, including instructions, evidence and the work already completed. Anthropic's September 2025 engineering guidance treats context as a changing selection of instructions, tools and other information available to the model, with attention to relevance and limited capacity. More input is not automatically better input. [2]

An agent harness is the software surrounding a model that organizes its execution. For this service, the complete execution environment needs to preserve progress, enforce tool permissions, record actions and support interruption and recovery. The harness, existing applications and integration services may share those responsibilities. Leaders should establish where each responsibility is implemented and tested.

Boundaries make a delegation more governable, but do not eliminate model uncertainty. Anthropic's April 2026 account describes risks across the model, harness, tools and environment, and states that layered defenses do not guarantee protection. [3] In the proposed service, retrieval access therefore remains separate from permission to change a record or make a claims decision. The AI Agents and Systems of Record Insight develops the architecture behind dependable changes.

Fund the work people will carry

A protected trial can give domain experts room to redesign one service. Include people who will operate it after the project team steps back, and preserve the dependencies the service will face. A demonstration supported by exceptional staff and constant engineering attention provides limited evidence about ordinary operation.

The Outcome owner must fund the accepted result, including continuing time to maintain knowledge, tests and the agent harness.

RAND identifies infrastructure underinvestment among several causes of AI/ML project failure. [4] For an agent service, the implication is clear: unfunded maintenance and harness work can block the move from pilot to full production.

The Operating owner needs capacity for daily exceptions, support and recovery. These responsibilities should not be hidden in an assumption that employees will use time saved elsewhere. Establish which work will stop, which will continue and where the new responsibilities fit.

For the claims specialist, success might include a useful correction to an evidence rule, recognition of a consequential omission or coaching a colleague through an unfamiliar case. The role should not become an endless review queue with the old processing target unchanged. Some work may remain demanding or become more concentrated in difficult cases.

Maintain opportunities to learn the underlying operation. A newer colleague can assess selected cases independently before comparing their reasoning with the system and an experienced specialist. Rehearse missing information and service interruptions. The AI-Era Workforce Capability Framework provides the broader method for connecting responsibility to demonstrated capability and support.

Judge the service and the human contribution

Evaluate whether the changed design improves accepted evidence packets and reduces avoidable effort across preparation and receiving review. Track elapsed time, correction, incomplete work and the effort required to investigate exceptions. Examine difficult cases separately so an improved average does not conceal a growing unresolved queue.

Use Cost per accepted outcome for the capability: total recurring cost in the defined period divided by outcomes accepted by the owner in that period. In this example, the unit is one accepted evidence packet, with acceptance recorded by the designated specialist. Match the cost and outcome cohort, include the agreed follow-up needed to establish acceptance and disclose any lag.

Recurring cost includes technology use, human review and correction, failed attempts, exceptions, support and recovery. Repeated attempts add cost without adding accepted packets; one-time design and build costs remain in the investment case. With no accepted packets, the measure is not calculable. Released staff time becomes cash savings only when an expense actually changes.

Assess people against contributions they can influence. A well-founded escalation can protect the result even when it lengthens one case. A proposed instruction change deserves recognition when testing and subsequent evidence support it. Neither the number of agents supervised nor the volume of prompts is sufficient evidence of a useful contribution.