ADR 0019: Data migrations, hot swap and replay

Status: accepted, 2026-10-04 (Phase 6, second slice)

Context

Doc 04 sections 4 to 6 ask for typed store evolution, rollout with instant rollback, and deterministic replay. ADR 0016 shipped store, svc, deploy plan and deploy local with no notion of a schema version: an item that no longer decoded as the store’s value type was a 500 with “a migration is needed”. As before, nothing here calls AWS. Everything runs on 127.0.0.1, and the real-deploy side is generated files that sspur never executes.

Decisions

#	Decision	Reason
1	A store’s schema version is `base32(blake3(type hash, JSON shape))[..12]`. The type hash is the value type’s definition hash (doc 02). The shape lists field names, variant names, primitive types and refinement text	Definition hashes leave names out, so a copy of the old type under a new name (`ItemV1`) has the old hash, which is what lets one file name both versions. But sum hashes are positional and leave out variant names, while stored JSON uses them, so the shape is part of the version
2	A migration is an ordinary function `fn migrate_S(old: OldT) -> V` (or `migrate_S_suffix`), pure, with V the store’s value type. It is matched to stored items by OldT’s schema version. The optional reverse `fn unmigrate_S(v: V) -> OldT` must pair with a migration of the same schema. Bad signatures are `E_MIGRATE_SIG`	Doc 04 sketched a `migrate S from #old to #new` node. A named function needs no new syntax, is typechecked, compiled natively and tested like any other definition (`test round_trip = unmigrate_Items(migrate_Items(x)) == x`), and is found by `q`, `edit` and `callers`. Totality comes from purity plus the checker: a trap in a migration is a 500 for that read, never a corrupted write
3	`sspur deploy migrate old.ssp new.ssp` compares stores by name. Same version: unchanged. Otherwise a structural diff of the shapes: added `Opt` fields, added variants and dropped refinements are compatible; any other change needs a migration function for the old version (migration), and without one it is breaking (`E_MIGRATE_MISSING`, which prints the old shape to copy). A changed key type is always breaking (`E_MIGRATE_KEY`). Removed stores are reported, and their tables are retained	Compatible changes decode as they are: an absent `Opt` field reads as none and extra fields are ignored. Refinements are assumed by native code and by SMT check elimination, so a new or changed refinement must be re-established by a function. Keys address items, so they can’t change in place
4	Items are `{pk, sv, v}` plus `v_<schema>` copies. Writes store the current schema in `sv` and, for each `unmigrate_`, the old-schema copy. A read takes, in order: its own schema’s copy, `v` when `sv` matches, the migration whose source is `sv`, `v` as the current type (compatible change or untagged item), then any migration whose source type decodes an untagged item	This is lazy read-time migration in the generated runtime (`rt_item_val`), in memory only. Writing back on read would give `read` a PutItem grant, which breaks ADR 0016’s least privilege. The reverse copy is what lets the old version run next to the new one: it reads `v_<its schema>` first
5	Side by side holds when every changed store is compatible without added variants, or has both a migration and its reverse. Otherwise `migrate` says “all at once only” and a local canary is refused	An old version can’t decode an added variant or a new-schema item without the copy. A rollback in that case still works for items the new version didn’t write
6	A store with a migration gets a backfill function `backfill_S` with exactly `dynamodb:Scan` and `dynamodb:PutItem` on that table. One invocation handles one page (`{"limit", "start"}` in, `{scanned, migrated, current, skipped, failed, next}` out). Each write is conditional on `v` being unchanged since the scan (`ConditionExpression "#v = :v"`), so a concurrent handler write wins and the item is counted as skipped	A page per call keeps each run inside Lambda’s time limit and makes it resumable. The condition makes it safe under live traffic and idempotent: a second run migrates nothing. The function is the same `bootstrap`, started with `SSPUR_BACKFILL=S` instead of `SSPUR_HANDLER`
7	Local hot swap. `deploy local` keeps a router with a stable version, an optional weighted canary and the previous version. Each version is a set of `bootstrap` processes under its own credentials (one grant set per hash) against the same emulated DynamoDB. A request picks its version once, holds a reference to it, and finishes there. `swap` starts the new processes before it changes the router. `rollback` drops a canary or swaps stable and previous. A version’s processes are stopped when nothing refers to it any more	No request sees a version that is half started or already stopped, so a swap drops nothing, and state is shared. The previous version keeps running, so rollback is a pointer change (pillar 17). `swap` runs `migrate::compare` against the stable version first and refuses breaking changes
8	The controls are `/_sspur/{swap,promote,rollback,backfill,status}` on the local port, with CLI wrappers `sspur deploy swap	promote
9	Real deploy. Per handler: an `AWS::Lambda::Version` named by the program hash with `DeletionPolicy: Retain`, a `live` alias that the API integration and invoke permission point at, an `UpdatePolicy: CodeDeployLambdaAliasUpdate`, a deployment group with automatic rollback, and two alarms: Lambda `Errors` on the alias, and a metric filter on the runtime’s `REQ <id> <status>` log line counting 5xx answers (the runtime returns handler failures as responses, which Lambda doesn’t count as errors). The `TrafficShift` parameter picks the CodeDeploy config (default `LambdaCanary10Percent5Minutes`). One `CodeDeployRole` can only get and update this service’s functions and aliases and describe its alarms	This is AWS’s own weighted alias routing, driven by CloudFormation, so `deploy.sh` stays one `cloudformation deploy`. Retained versions are what make a rollback instant. The template still passes `cfn-lint` and has no `*` action or resource
10	`rollback.sh`: during a shift it stops the CodeDeploy deployment with auto-rollback; afterwards `PREV=<hash>` redeploys that hash’s template (now uploaded next to its zip by `deploy.sh`) with `TrafficShift=CodeDeployDefault.LambdaAllAtOnce`. `backfill.sh` invokes the backfill function page by page. `deploy migrate --out DIR` writes `migrate.json` (classification, schemas, functions, backfill grants, and the expand, backfill, contract phases) with the two scripts	They are the operator’s commands, generated so nothing is written by hand, and never run by sspur
11	Recording. `deploy local --record FILE` appends one JSON line per request: id, svc, version, handler, method, path, query, body, status, response, and every DynamoDB call the handler made (`op`, table, request, result). Calls are attributed through the caller’s access key to the function, and through the function to the request it is serving: one process serves one request at a time	Handlers can only perform `db.*`, `log`, `div` and `fail` (ADR 0016), so requests plus db results are the complete input. There is no clock or randomness to record
12	Replay. `sspur deploy replay rec new.ssp` runs each recorded request alone against new.ssp. The emulated table is reset to what the recording observed before the request’s first write: GetItem results (including absent keys), DeleteItem results, and Scan pages, where a last page marks the table as fully known. A read outside that is reported, not guessed. Status, response JSON and the sequence of writes (key and decoded value) are compared. A response or write that differs only by added `null` fields is extended; that passes unless `--strict`. Any other difference exits 1	Per-request seeding makes replay independent of order and of state the recording never saw, and bit-for-bit deterministic (pillar 18). Comparing writes catches changes that don’t show in the response. Treating new null fields as extended lets a compatible schema change pass the gate

Results

crates/sspur-deploy/tests/evolve.rs on examples/crud with examples/crud/evolve/items_v2.ssp (adds note: Opt[Str]) and items_v3.ssp (replaces qty: Int by stock: Stock{on_hand, unit} with migrate_Items and unmigrate_Items):

Classification: v1 to v2 compatible and side by side, no backfill. v1 to v3 migration through migrate_Items, whose ItemV1 has the stored schema. v3 without the functions is breaking with E_MIGRATE_MISSING, without the reverse it is all at once only, a migration with an effect is E_MIGRATE_SIG, a type error in one is E_TYPE, an Int key is E_MIGRATE_KEY, and an added variant is compatible but not side by side.
Migration rollout: items written by v1, a 50% canary of v3 serving both versions (v3 returns migrated stock), promote, writes through v3, rollback (v1 reads v3’s items through the reverse copy), an old-schema write by v1, roll forward (migrated again), backfill (exactly the old items migrated, none failed, every item tagged and carrying a v1 copy), a second backfill migrates nothing, and v1 still reads everything after a final rollback.
Hot swap: a request in flight when the swap happens (the emulator delays its db call by 400 ms) finishes on v1 with status 200, and the next one is served by v2. Then 4 client threads issue more than 80 POST, GET and list requests while the service swaps, rolls back and rolls forward: every request succeeds and every POST is in the final list.
Replay: 11 recorded requests (creates, conflict, reads, a 404, update, list, deletes, a 400) replay against v1 with 11 identical. Against v2 they pass with some extended, and fail with --strict. Against a version that sorts the list by qty and makes deleting a missing item 204, exactly the list response and the second delete’s status differ.
The v3 template, with the backfill function, aliases, versions, alarms and CodeDeploy, passes cfn-lint 1.40. crates/sspur-cli/tests/deploy.rs runs deploy migrate, then deploy local --record, swap, backfill, rollback, status and replay through the CLI.

Real canary (2026-10-07)

examples/crud/items.ssp (v1) and examples/crud/evolve/items_v3.ssp (v3, migrate_Items and unmigrate_Items) were run through the generated scripts on a test account in us-west-2, then torn down completely.

Step	Result
`deploy.sh` v1, all at once; 6 items created	ok
`deploy.sh` v3, `LambdaCanary10Percent5Minutes`	First attempt failed and CloudFormation rolled back with v1 serving and zero errors: CodeDeploy could not call `cloudwatch:DescribeAlarms`. Fixed, second attempt ok
During the shift, 20 reads every 45 s	About 10% answered by v3 for five minutes, then 100%; 0 errors. v3 answered old items through the lazy migration
`backfill.sh`	6 scanned, 6 migrated, 0 failed; each item keeps a `v_<old schema>` copy for the old version. A second run migrated 0 (idempotent)
Write on v3, then `rollback.sh` with `PREV=<v1 hash>`	First attempt stopped at a CodeDeploy API error; fixed. v1 then served again and read the v3-created item with `qty` mapped back from `stock.on_hand`

Two bugs found and fixed:

cloudwatch:DescribeAlarms has no resource-level permissions, so the role’s alarm ARN list granted nothing and every alias update failed. The statement now uses Resource: "*" for that one read-only action. It is the only wildcard in the template, and tests/crud.rs asserts exactly that.
rollback.sh called list-deployments with an application but no deployment group, which the API rejects. It now walks the service’s deployment groups.

A version deployed with the old generator keeps the old role in its retained template, so rolling back to it reproduces bug 1. The test regenerated v1’s template before rolling back. Versions deployed from now on roll back cleanly.

Remaining

Item history from the .sspur store instead of two files: migrate could take a store root hash for the old side.
Chained migrations go one hop per function (migrate_S_v1 from v1, migrate_S_v2 from v2). There is no automatic composition yet.
SLO-driven gates (slo in doc 04) instead of the fixed alarms, and a contract step that drops v_<schema> copies once no old version runs.
Replay doesn’t compare log output, and recordings aren’t redacted by type annotation (doc 04 section 6).