Embedding Studio
Two orchestrators, one hotel booking system.
The same multi agent workflow, run twice at the same time. Identical MCP server, identical specialist agents, identical final answer step. The only difference is the node that decides who goes next on every hop: a Sonnet call on the left, a typed decision on the right. The clocks are wall clock, measured per run, not from a benchmark file.
One thing to know before reading the clocks. Every call to Claude carries about 1.5s of time to first token before it does any work, measured on a one word prompt, so a workflow's wall clock is mostly a count of its sequential model calls. The left arm makes six: one supervisor call a hop, one a specialist agent, one to write the reply. The right arm makes one, the reply, and answers every other question with a typed decision. That is the whole mechanism.
Nothing here is sampled at a temperature. The supervisor and the judge are constrained to a JSON schema, every call runs at the lowest effort setting, and on the right arm the tool arguments come from deterministic extraction rather than from a model, so the same ask does the same work twice. That was not true of an earlier version, where the left arm asked for a 15 day forecast one run and a 3 day forecast the next and the answers disagreed because the facts underneath them did.
Pick an ask
Hardest first. The gap is widest where there is most to orchestrate.
- loading the asks…
Left arm
A language model supervises
LangGraph, nine MCP tools, three specialist agents. On every hop a Sonnet call is asked who goes next, with the tool list in the prompt. This is how it is normally built.
- supervisor 0.00s
- hops 0
- claude calls 0
- decisions 0
- tool calls 0
- tokens 0
Right arm
A decision model decides everything
Same graph, same MCP tools. Every choice is a typed decision with a calibrated confidence: which agent goes next, and whether a write needs a person. The tools then run from MCP's own schemas with no model involved. Claude is called once, at the end, to write the answer.
- supervisor 0.00s
- hops 0
- claude calls 0
- decisions 0
- tool calls 0
- tokens 0
What this does and does not show
- The arms differ in two places now, not one. The supervisor, and how an agent's tools get called: the left arm asks Sonnet which tool and with what arguments, the right arm runs them from MCP's schemas and deterministic extraction. So this is no longer an isolated test of the supervisor swap. It is the comparison between an orchestrator that asks a language model every question and one that asks it only to write the final answer.
- Deterministic extraction works here because the world is closed: five cities, a day of the month, a night count, a budget, a booking id. On open traffic, turning "the Tuesday after next" into a date is a language model's job, and the shape that survives is one extraction call up front rather than one per hop.
- Before writing the reply, both arms run any read-only tool the request has arguments for that has not been run yet. A read is free and MCP declares it idempotent, so answering while holding an unexecuted free read is strictly worse. It costs a measured 0.02s. It is also a patch, not a fix: on the two city ask the fitted router never chose the weather agent, and without the sweep the right arm answered "I have no forecast data". The repair for that is labelled rows, not a sweep.
- The decision supervisor is a fitted model. On a hand written held out set it routes correctly 80% of the time against 96% for Sonnet, and the reason to use it anyway is the
routefield: on the five states it got wrong it was below the band every time and acted on none of them. An llm judge has no field that separates its good answers from its bad ones. - A single run is one sample. Latency on both arms has a tail, so run the same ask twice before believing a gap of a few tenths.
- The hotel world is canned. This is about who gets called and when, not inventory.