Build the Intelligence Layer

Knowledge becomes an operating dependency

Organizations can accumulate a great deal of information without making it usable at the moment of decision. Records may disagree, policies may be out of date, and an exception may depend on context held by one experienced colleague. Connecting a model to more documents does not settle which source has authority.

Begin with the operating design. For each decision, identify the knowledge required, how current it must be, the relevant permissions and what should happen when the evidence is incomplete. This keeps the build connected to a real operating obligation.

The intelligence layer may combine retrieval, rules, conventional software and models. Choose components for the work they must perform, the consequences of failure and the effort needed to sustain them. Technical sophistication is not itself a benefit in the operating case.

When two sources disagree, what tells the operation which one to trust?

Five questions to put to the build

Which knowledge is authoritative?

Name an owner for the policies, definitions and records that govern each important decision. Record provenance, version and effective date where they matter. Define how corrections reach the system and how superseded material is withdrawn. Knowledge that nobody maintains will eventually become an operating risk.

Does the system understand the relevant context?

Supply the context the task requires: process state, customer or supplier history, permitted actions and outstanding exceptions. Apply access boundaries throughout retrieval and action. Test that a plausible answer does not conceal a missing record, an unauthorised source or a conflict the system should surface.

How will reliability be judged?

Build an evaluation set from representative work, including rare but consequential failure types. Define success for the intended role: retrieving a source, making a recommendation and executing an action need different evidence. Include downstream consequences, review burden and the cost of a wrong answer. Keep some cases outside development feedback so the final evaluation tests more than familiarity.

What must the system refuse or defer?

Specify behavior for insufficient evidence, conflicting policy, unavailable services and requests outside the allowed scope. Confidence language alone does not establish correctness. Where the evidence cannot support an action, require abstention, a clarification request or routing to an authorized person.

Can this capability be maintained economically?

Estimate integration, evaluation, knowledge maintenance, monitoring and support costs alongside model usage. Identify dependencies on suppliers and specialist skills. Compare building, buying and connecting existing capabilities. Revisit the operating case if reliable performance requires more effort than the expected value can justify.

Retrieving the policy is not the whole decision

Hypothetical example, continuing the invoice opportunity.

The assistant retrieves a supplier exception policy that permits a particular approval path. A newer version narrows that permission, while an old training document still describes the earlier rule. Both documents contain convincing language.

The knowledge owner needs to establish precedence and effective dates. Evaluation should test which rule is retrieved, whether the explanation identifies its source and whether the proposed action stays within the current permission. Cases with unresolved policy conflicts should route to Finance rather than produce a confident recommendation.

First test: ask the team to demonstrate the normal case, a superseded policy, a missing receipt and an inaccessible record. Inspect the sources and resulting actions. If the system cannot reliably recognise a missing basis for action, reduce its role or address the knowledge gap before deployment.

• Can an owner identify the authoritative basis for each important decision?
• Have we tested outdated, conflicting, missing and inaccessible information?
• Does evaluation reflect the intended action and its consequences, including the effort required for human review?
• Can the system abstain and route work appropriately when its evidence is insufficient?
• Are permissions and action limits enforced, rather than left as written intentions?
• Do we know who maintains this capability and whether the full running cost still supports the operating case?

Choose the next commitment

Proceed to bounded deployment when evaluation supports the intended role and the operation can enforce its limits. Narrow the role when assistance is dependable but autonomous execution is not justified. Repair the knowledge or integration where missing foundations drive failure.

Carry the evaluation record into deployment. Success on test cases is evidence for an operational trial, not proof of performance under every live condition.

Record the review decision as Expand, Continue, Narrow, Redesign or Stop. The phase gate states the next delivery commitment; a pause, fallback or rollback remains an operational control.