Harness engineering

What it is and why it exists

How it works

• The harness exposes the same chat-completions contract it consumes. Callers cannot tell whether they are talking to a bare model or to the agent.
• A planner node calls the model. If the model answers, the run ends. If it requests tool calls, each call becomes a node and all of them run concurrently.
• Tool nodes append their results to the conversation and route back to the planner. Several tool nodes naming the same successor collapse into one planner run.
• A wave budget bounds the cycle. When the budget runs out, one final turn without tools answers from the evidence already gathered.
• Before the answer leaves, a deterministic guard compares it with the evidence from this request. A failed draft gets one corrective retry.

Where it sits in the build order

In the reference build

Process map

One request through the harness

1. 1
   AuthenticateKey, scope, rate limit, quota. Scope decides whether tools are offered. (decision)
2. 2
   TriageCheap gates first: regex, then embedding similarity. Some questions skip tools. (decision)
3. 3
   Compose promptBase instructions, at most one matched skill, the caller's memory section. (input)
4. 4
   PlannerModel call with tool schemas. Returns an answer or a set of tool calls. (model call)
5. 5
   Tool waveAll requested calls run at once. Hooks run before and after each. Results are capped. (tool)

   loops to step 4: back to planner, up to 6 rounds
6. 6
   GuardThe draft is checked against this request's evidence. One corrective retry on failure. (decision)
7. 7
   RespondStreamed or complete, in the same format the caller sent. (output)
8. 8
   RecordLogs and memory writes happen after the response and never block it. (storage)

Build steps

Each step states why it sits at this point. Open any step on its own. The first is open.

# The whole public surface of the loop.
async def run(payload: dict, engine_url: str, allow_tools: bool, key_id: str):
    """-> (openai_response_dict, prompt_tokens, completion_tokens, model_calls)"""

async def flat_loop(messages, tools, max_rounds=6):
    for _ in range(max_rounds):
        reply = await chat(messages, tools=tools)
        msg = reply["choices"][0]["message"]
        messages.append(msg)
        calls = msg.get("tool_calls") or []
        if not calls:
            return msg["content"]
        for call in calls:
            result = await call_tool(call["function"]["name"],
                                     json.loads(call["function"]["arguments"]))
            messages.append({"role": "tool",
                             "tool_call_id": call["id"],
                             "content": json.dumps(result)[:8000]})
    return None  # budget exhausted, handled in a later step

_EXTEND_KEYS = {"messages_append", "trace_notes"}
_SUM_KEYS = {"usage_prompt_tokens", "usage_completion_tokens", "model_calls"}

def merge_state(state: dict, update: dict) -> None:
    for key, value in update.items():
        if key in _EXTEND_KEYS:
            target = key.removesuffix("_append")
            state.setdefault(target, []).extend(value)
        elif key in _SUM_KEYS:
            state[key] = state.get(key, 0) + value
        else:
            state[key] = value

async def run_graph(entry_name, entry_fn, initial_state, max_waves=6):
    state = dict(initial_state)
    wave = {entry_name: entry_fn}
    trace = []
    for wave_idx in range(1, max_waves + 1):
        if not wave:
            break
        trace.append({"wave": wave_idx, "nodes": list(wave)})
        results = await asyncio.gather(*[fn(state) for fn in wave.values()])
        for update, _ in results:           # merge everything first
            if update:
                merge_state(state, update)
        next_wave = {}
        for _, edges in results:            # then route
            next_wave.update(edges)         # same key twice = fan-in
        wave = next_wave
    else:
        state["_budget_exhausted"] = True
    state["_trace"] = trace
    return state

def compose_system_prompt(messages, partition_key):
    parts = [read_agent_md()]
    skill = match_skill(last_user_text(messages))   # zero or one
    if skill:
        parts.append(skill.body)
    memory = compose_memory_section(partition_key)
    if memory:
        parts.append(memory)
    return "\n\n".join(parts)

def write_path_guard(name: str, args: dict) -> str | None:
    """Return a reason to block, or None to allow."""
    if name in WRITE_TOOLS:
        p = Path(args.get("path", "")).resolve()
        if not p.is_relative_to(ALLOWED_ROOT):
            return f"writes are limited to {ALLOWED_ROOT.name}/"
    return None

How it works with the other parts

• SkillsThe harness matches one skill per request and injects its body. Skills change behavior without changing harness code.
• MemoryMemory is read during prompt composition and written after the response.
• Guardrails and verificationHooks act on tool calls. The answer guard acts on the final draft. Both are called by the loop.
• EvaluationEvery request, tool call, skill match, and triage decision is logged as JSON lines. Evals and self-observation read those logs.
• Multi-agent patternsThe same wave executor runs sub-agents as nodes. A sub-agent is a planner with its own tools and budget.

What went wrong in the real build

The same idea on other platforms

Explain it back

Answer aloud first. Then open the answer and compare.

Terms

From the live build

Recent changes and files the sync job filed under this module.

• 2026-10-07 · changeOut of tool rounds: one final turn without tools answers from the evidence gathered; restate standing after promotion; ordered promotion queue
• 2026-10-07 · changeBUSY sentinel is held until a streamed response finishes, so the learner pauses for the whole chat answer
• 2026-10-06 · note · apps/agent-server/skills/spending-investigation.mdSpending orchestration
• 2026-07-31 · note · study-notes/03_Agent_Harness_Deep_Dive.md03 — The Agent Harness, Deep Dive
• 2026-07-30 · note · AGENT_LATENCY_INVESTIGATION_SUMMARY.mdAgent-Server Latency Investigation — Summary (2026-07-30)
• 2026-10-07 · source · apps/agent-server/agent_loop.py · 1531 linesbuilds the specific planner/tool graph on top of graph.py's generic engine, and exposes the same run() contract main.py already calls.
• 2026-08-21 · source · apps/agent-server/tools/budget_execution.py · 501 linesU7/U8: the account layer, and the reconciliation. Two tools: budget_execution the appropriation's own story: total resources -> obligated -> outlaid -> unobligated, by Treasury Account, program activity or object class. THE ONLY PLACE OUTLAYS LIVE.
• 2026-08-21 · source · apps/agent-server/tools/supplemental.py · 367 linessupplemental_appropriations: emergency and disaster money, by the law that created it.
• 2026-07-28 · source · apps/agent-server/graph.py · 111 linesminimal state-graph execution engine. Why this exists (v2 refinement, 2026-07-27): step 2's agent_loop.py was a flat sequential loop — call the model, run whatever tools it asked for in one batch, repeat.

Ask the tutor about this module