Fork, Replay, Compare¶

An agent answers a customer. It promises a full refund the policy does not allow and cites a policy number that does not exist. The run finishes completed.

That is the failure this recipe is about. Nothing raised, nothing retried, nothing to grep the logs for — the code did exactly what it was told, and what it was told was wrong. The fix is one word of the prompt. The question is what it costs to try.

This page forks that finished run immediately before the bad call, re-runs it under a sharper instruction, and lays the two runs side by side call by call. One of the six durable calls executes. The other five answer from the journal, byte-identical. $0.0023 instead of $0.1692.

Source: examples/fork_and_compare_demo.py

Watch It¶

The recording is the five beats below, run unedited — same file, same deterministic fake model, real output, five markers on the seek bar (⇤/⇥ jump between them). The prompt is typed and the reading pauses between beats are added for legibility; the process itself runs in well under a second, which is the number Beat 3 is about.

Get It And Run It¶

1. Install the runtime:

   pip install 'satay[studio]'
   

2. Fetch the file:

   curl -fsSL -O https://raw.githubusercontent.com/leejianrong/satay-runtime/v0.1.0/examples/fork_and_compare_demo.py
   

3. Run it, naming a data directory so the four journals outlive the process:

   SATAY_DATA_DIR=.satay-demo python fork_and_compare_demo.py
   

The Workflow¶

Six durable calls: a plan, a keyed fan-out of four policy lookups, and the draft.

@satay.workflow
async def answer_ticket(brief: Brief) -> dict[str, object]:
    lookups = await plan_lookups(brief)
    notes = await satay.map(look_up, lookups, key=lookup_key, concurrency=4)
    reply = await draft_reply(brief, notes)
    return {"ticket": brief.ticket, "reply": reply, **review(reply, notes, brief)}

review is the guardrail: it checks that every POL-nn the reply quotes was actually retrieved, and that the reply does not promise money back outside the refund window. It is ordinary Python living directly in the workflow body, which is allowed precisely because it is deterministic — no clock, no randomness, no I/O — so replay recomputes the same verdict and it needs no journal entry of its own.

The prompt is in the input:

@dataclass(frozen=True)
class Brief:
    ticket: str
    customer: str
    question: str
    topics: list[str]
    days_since_delivery: int
    instruction: str          # <- the prompt, as data

That last field is the whole reason the fork below is three lines instead of a refactor. A prompt held in a module-level global cannot be forked; a prompt held in the workflow input can.

Beat 1 — The Run That Went Wrong¶

$ SATAY_DATA_DIR=.satay-demo python fork_and_compare_demo.py
Satay — the debugger loop: fork a prefix, replay, compare call by call
data dir: …/.satay-demo
model:    fake-support-1 (deterministic fake, offline)

1) the run that went wrong
   ticket TCK-8814 — delivered 35 days ago, and the refund window is 30 days
     "My blender arrived cracked. Can I get a refund?"
   instruction: "Reassure the customer and keep them happy, whatever it takes"

   run 16f3571014864692920797a93805f243 — completed
     | Hi Dana — so sorry about this! We have refunded you in full, no
     | questions asked, and a replacement is already on its way. The details
     | are all in our returns policy POL-09.

   guardrail: FAILED
     policies the run retrieved  ['POL-14', 'POL-22', 'POL-31', 'POL-40']
     policy ids the reply cites  ['POL-09']
     ids that do not exist       ['POL-09']  <- invented
     promises money back         True
     policy allows one           False  <- 35 days > 30

   Nothing raised. The run is `completed` and the workflow did exactly what it
   was told. A stack trace shows nothing and a retry produces the same answer,
   because the bug is in the input, not in the code.

     6 durable calls    55,437 in /   194 out  $0.1692

The order was delivered 35 days ago and the window is 30, so a refund is not on the table. The reply promises one anyway and attributes it to POL-09, which the run never retrieved.

Notice what did not happen. No exception. No ⚡. No failed attempt. The status is completed and the result object is well-formed. This is the class of bug that durability alone does not help with — and it is most of the bugs in an agent.

The $0.1692 is the four policy lookups. Each one drags a retrieved document along, which is why the research is 98% of the bill and the draft is the other 2%. Remember that ratio; the rest of the page is about not paying it twice.

Beat 2 — Fork Before The Bad Call¶

2) fork it immediately before the bad call, under a sharper instruction
     satay.fork(run_id, before_task="draft_reply", workflow_input=sharper)
   instruction:
     > Answer only from the notes below, quote the policy id behind every
     > claim, and never promise an outcome the notes do not allow

   fork run 907387252c3d4e6f90b7a2db99f51df3 — completed
     | Hi Dana — thanks for the details. Items damaged in transit are covered
     | for 30 days from delivery [POL-14]. Refunds are issued inside 30 days of
     | delivery, and store credit after that [POL-22]. An order number is proof
     | enough and no receipt photo is required [POL-31]. A like-for-like
     | replacement may be offered before a refund [POL-40]. Your order was
     | delivered 35 days ago, so the 30-day window has closed and I cannot
     | issue a refund; I can offer store credit or a like-for-like replacement
     | instead.

   guardrail: PASSED
     policies the run retrieved  ['POL-14', 'POL-22', 'POL-31', 'POL-40']
     policy ids the reply cites  ['POL-14', 'POL-22', 'POL-31', 'POL-40']
     ids that do not exist       []
     promises money back         False
   RunForked: source=16f3571014864692920797a93805f243 fork_point_seq=16

The call that produced it:

sharper = replace(REFUND_TICKET, instruction=GROUNDED_INSTRUCTION)

handle = await satay.fork(
    source_id, before_task="draft_reply", workflow_input=sharper, store=store, clock=clock
)
result = await handle.result()

before_task="draft_reply" cuts the journal so that call re-runs, without you scanning event sequence numbers to find it. workflow_input= supplies the new brief, and it is written into the fork's own WorkflowCreated event rather than held in memory — so a fork that parks on a timer and wakes an hour later still reads back the input it actually ran under. The RunForked line is that lineage, read straight off satay.inspect(fork_id).forked_from: which run it came from and where it was cut.

The source run is untouched. ADR-0004 makes a journal immutable; a fork is a new run seeded with a copy of a prefix, never an edit.

Beat 3 — The Number¶

3) what the fork actually re-ran
   durable calls it executed           1  ['draft_reply:0']
   durable calls it read off the copy  5

     the source run    55,437 in /   194 out  $0.1692
     the fork             151 in /   124 out  $0.0023

   >>> 1 of 6 durable calls re-ran; 5 were reused byte-identical.
   >>> $0.0023 to fix the answer, against $0.1692 for the original run —
   >>> 98.6% of the bill was history, and history does not need re-buying.
   >>> The source run is untouched and still says what it said.

"Executed" here is not the demo's own bookkeeping — it is read straight off satay.inspect (ADR-0033). A fork's journal opens with a verbatim copy of its source's prefix, so the test for "did this run actually make this call" is whether a RecordedCall.first_seq sits above the fork point:

inspection = await satay.inspect(fork_id, store=store)
boundary = inspection.forked_from["fork_point_seq"]
executed = [call.identity for call in inspection.calls if call.first_seq > boundary]

One identity comes back. The costs are read the same way, above the same boundary — but from the raw journal, not inspect: ctx.record_model_usage entries are not durable calls, and there is no aggregate read view for them yet (that is on the roadmap, not shipped).

Beat 4 — Compare, Call By Call¶

4) compare, call by call
     satay.diff(16f35710…, 90738725…)
     GET /runs/16f3571014864692920797a93805f243/compare?to=907387252c3d4e6f90b7a2db99f51df3

   durable call                       source     fork       recorded output
   plan_lookups:0                     completed  completed  identical — replayed
   look_up:key:damaged-on-arrival     completed  completed  identical — replayed
   look_up:key:refund-window          completed  completed  identical — replayed
   look_up:key:proof-of-purchase      completed  completed  identical — replayed
   look_up:key:replacement-vs-refund  completed  completed  identical — replayed
   draft_reply:0                      completed  completed  DIFFERS  <- the fixed call

   6 calls aligned on both sides; 5 identical, 1 different — and the
   one that differs is the one call after the fork point. Studio draws this
   table; the JSON behind it is what you just read off two real journals.

satay.diff (ADR-0034) aligns two runs by durable-call identity — plan_lookups:0 is the first call to plan_lookups, look_up:key:refund-window is the fan-out item with that key — and gives each side that call's status and recorded output, plus a changed flag the demo reads straight off instead of comparing outputs itself:

run_diff = await satay.diff(source_id, fork_id, store=store)
for call in run_diff.calls:
    note = "identical — replayed" if not call.changed else "DIFFERS  <- the fixed call"

An identity present on one side and absent on the other shows as unaligned, which is how a structural divergence reads.

Here there is no divergence in shape. Six rows, six aligned pairs, five with changed=False. The table is not a diff of two log files; it is the journals themselves, aligned by the identity the replay engine uses.

Beat 5 — The One Rule¶

A fork's copied prefix is a record of calls that already happened. workflow_input= therefore reaches only the calls after the fork point. Get that wrong and you get a confident answer built on last week's research:

5) the rule: a fork's copied prefix is history, not a prediction
   Same fork point, but a different QUESTION this time — an address change,
   not a refund. `draft_reply` is the only call after the cut, so it is the
   only call that sees the new ticket:

   fork run 5794a5da01214892add6c2e63d911d87 — completed, guardrail PASSED, and useless
     | Hi Dana — thanks for the details. Items damaged in transit are covered
     | for 30 days from delivery [POL-14]. Refunds are issued inside 30 days of
     | delivery, and store credit after that [POL-22]. An order number is proof
     | enough and no receipt photo is required [POL-31]. A like-for-like
     | replacement may be offered before a refund [POL-40]. I can issue a
     | refund.
     durable calls it executed:  ['draft_reply:0']
     policies on its journal:    damaged-on-arrival, refund-window, proof-of-purchase, replacement-vs-refund
   Every citation is real, so the guardrail passes. The research is simply
   answering the previous question, because those four calls already happened
   and a fork reuses them rather than paying for them again.

That is worth sitting with. The question was about changing a delivery address. The reply discusses damaged goods and offers a refund, every citation is genuine, and the guardrail is perfectly happy. Reuse is not free of judgement: it reuses whatever you told it to reuse.

The fix is to cut earlier:

   Fork before `plan_lookups` instead and the new input reaches everything:

   fork run 18b1ce1954ff4f6d96c36f57e6cb3207 — completed, guardrail PASSED
     | Hi Dana — thanks for the details. A delivery address can be changed
     | until the carrier scans the parcel [POL-51]. Once scanned, only the
     | carrier can redirect a parcel and it may refuse [POL-58]. After handoff
     | the label cannot be altered and the order returns to sender [POL-63]. A
     | reshipment after a failed delivery carries a flat handling fee [POL-70].
     | Tell me which option suits you and I will set it up.
     durable calls it executed:  6 of 6    51,966 in /   252 out  $0.1597
     policies on its journal:    address-change, in-transit-redirect, carrier-handoff, reshipment-fee

   So: put the fork point before the first durable call that should see the new
   input (ADR-0028). `before_task=` exists to let you say exactly that, and the
   full-price run above is what it costs when the honest answer is 'all of them'.

Six of six calls re-ran and the fork cost full price, which is correct: the whole run had to change. The rule in one sentence — put the fork point before the first durable call that should see the new input — and before_task= is how you say it. Forking a run has the rest, including what happens when the new input would make the workflow call different tasks inside the copied prefix (it raises, rather than splicing two histories into a plausible wrong answer).

The Closing Line¶

four runs of a six-call workflow, and this process made 14 model calls,
not 24. The other 10 answered from the journal.

journals kept in …/.satay-demo
open all four runs:  satay dev --app examples.fork_and_compare_demo --data-dir …/.satay-demo
  then compare 16f3571014864692920797a93805f243 against its fork 907387252c3d4e6f90b7a2db99f51df3
or as text:          satay runs show 16f3571014864692920797a93805f243 --data-dir …/.satay-demo

Open It In Studio¶

satay dev --app examples.fork_and_compare_demo --data-dir .satay-demo

--app matters here in a way it does not for the other recipes: it imports the workflows, so Studio can fork a run itself rather than only showing you one. Open the printed URL with its ?token= query string.

In the run list you will see four runs of answer_ticket, one root and three forks, each with a lineage line pointing back at the source. Open the bad one, click "fork before here" on TaskScheduled task=draft_reply, and you have done from the UI what beat 2 did from Python. Then "Compare runs" against the source draws the beat 4 table.

What This Is Not¶

Satay ships no agent abstractions (ADR-0025): no loop framework, no tool protocol, no provider adapters, no graph DSL. This page teaches a pattern with five durable primitives and a dataclass; it does not ship one. The model client, the retrieval corpus, the prompt templates and the guardrail are all yours, and in this example they are all visible in the one file so you can see where your own would go.

If You Are Recording This Yourself¶

The page is laid out as five beats because that is a screencast, and the recording at the top of the page is exactly that: beats 1 through 5, terminal only. What it does not show is beat 6 below — opening the same four runs in Studio, forking from the UI, and drawing the beat 4 table there — because that is an interactive session, not something stdout can carry. If you are recording the fuller version with Studio and narration, here is what to run and what to say:

Run the file once with SATAY_DATA_DIR set before recording so Studio has something to open, then run it again on camera; the output is deterministic, so the run ids in the terminal will be the only thing that differs between takes. The whole script takes under a second, so the pacing is yours, not the runtime's.

Recap¶

• A run can complete and still be wrong. Durability does not catch that; a debugger does.
• satay.fork(run_id, before_task=..., workflow_input=...) re-cuts a finished run from code. No control API, no command queue, no worker tick, and no satay[studio] extra.
• The copied prefix is replayed, not re-executed, so the fork pays only for what comes after the cut. Here: 1 of 6 calls, $0.0023 against $0.1692.
• Compare aligns two runs by durable-call identity, so "what changed" is a table rather than an argument.
• Because the prefix is history, the new input reaches only the un-replayed suffix. Put the fork point before the first call that should see it.

Next: An Agentic DAG, which puts a human approval gate in front of the expensive call and shows the same fork loop inside a longer story.