Interactive systems lab / 01
Retry & Idempotency Lab.
A task can arrive twice. A worker can die after a side effect. Step through both and see why durable intent matters.
Deliver the same job. What changes?
First move: Choose “Crash after effect” and compare the side-effect counts. Use Previous and Next to replay each delivery.
Configure failure
Call and hope
- Delivery 1Provider called
Delivery 1 performs the side effect.
- Delivery 1Worker crashed
Effect happened, but no completion record survived.
Persist, claim, verify
- IntakeIntent persisted
One job row and one stable operation key exist before delivery.
- Delivery 1Provider called
Worker sends the stable operation key with the request.
- Delivery 1Worker crashed
Next delivery reuses the provider key.
A stable provider key lets the retry retrieve the original effect without performing it twice.
Teaching model: one logical operation and one provider effect. A real integration also needs bounded leases, payload hashes, provider ledgers, and reconciliation for ambiguous outcomes.
Under the model
Exactly once needs a boundary.
Queues deliver work at least once. A job ledger lets a worker claim and complete one logical operation across process restarts. External providers add another boundary: a stable provider key can deduplicate a replay, while an unkeyed call that may have succeeded needs reconciliation before another send.
Real implementations also use leases, unique constraints, transactional outboxes, provider result ledgers, and operator recovery paths. This lab isolates one operation so the failure order is visible.
- Compare a crash before and after the provider call.
- Turn off provider keys and step into the uncertain outcome.
- Add deliveries after completion and watch the durable handler skip them.
Lesson 01 / Check your understanding
Can you explain the result?
A worker crashes after an unkeyed provider call. Local state cannot prove the result. What is safest?