Migrations and hot swap

A service’s stored data outlives its code. This tutorial changes the type a store holds, checks the change, replays recorded traffic against the new version, swaps it into a running service with no downtime, and rolls it back. Everything runs locally under sspur deploy local.

Version 1

A notes service with one store:

type NoteId = new Str

type Note = {id: NoteId, text: Str where _.len > 0}

type NoteErr = NotFound{id: Str} | Conflict{id: Str}

store Notes = table[NoteId, Note]

fn create(n: Note) -> Note ! db.read[Notes], db.write[Notes], fail[NoteErr]
= do
  if db.get(Notes, n.id).is_some then raise Conflict{id: n.id.raw}
  db.put(Notes, n.id, n)
  n

fn read(id: Str) -> Opt[Note] ! db.read[Notes]
= db.get(Notes, NoteId(id))

svc notes
  ep post "/notes" = create
  ep get "/notes/{id}" = read

Version 2

Version 2 adds tags. A note gets a required tags field, there is an endpoint to add a tag and one to list notes by tag. The old type, copied under a new name, and two functions describe how stored notes change:

type Note = {id: NoteId, text: Str where _.len > 0, tags: List[Str]}

type NoteV1 = {id: NoteId, text: Str where _.len > 0}

fn migrate_Notes(old: NoteV1) -> Note
= Note{id: old.id, text: old.text, tags: []}

fn unmigrate_Notes(n: Note) -> NoteV1
= NoteV1{id: n.id, text: n.text}

migrate_Notes turns a stored version 1 note into a version 2 note, and it is pure. unmigrate_Notes is optional: with it, every write also stores a version 1 copy, so the old version keeps reading notes the new one wrote, and running both side by side or rolling back is safe. The type’s name doesn’t matter, only its shape: NoteV1 has the same schema as version 1’s Note.

create keeps the request body of version 1, so existing clients keep working, and new tags go through their own endpoint:

type NewNote = {id: NoteId, text: Str where _.len > 0}

fn create(n: NewNote) -> Note ! db.read[Notes], db.write[Notes], fail[NoteErr]
= do
  if db.get(Notes, n.id).is_some then raise Conflict{id: n.id.raw}
  note = Note{id: n.id, text: n.text, tags: []}
  db.put(Notes, n.id, note)
  note

fn tag(id: Str, t: Str) -> Note ! db.read[Notes], db.write[Notes], fail[NoteErr]
= do
  old = db.get(Notes, NoteId(id)).ok_or(NotFound{id})
  note = old with tags := old.tags.push(t)
  db.put(Notes, NoteId(id), note)
  note

Check the change

$ sspur deploy migrate notes_v1.ssp notes_v2.ssp --out mig
migrate svc notes #1d67e631ca7fd732 -> #7cd30b84fad22a9a
  Notes migration schema ltijs64nicnc -> 2rjv4w4gfn3f via migrate_Notes, reverse unmigrate_Notes
    ! field tags: List[Str] added (required)
    note: writes keep a v_ltijs64nicnc copy made by unmigrate_Notes, so the old version reads new items
  backfill: backfill_Notes (dynamodb:Scan, dynamodb:PutItem on Notes); reads migrate lazily until it has run
  ok: old and new versions can run side by side

deploy migrate classifies every store’s change:

It also writes migrate.json, backfill.sh and rollback.sh for a real deploy. A missing or wrongly typed migrate_ function is E_MIGRATE_MISSING or E_MIGRATE_SIG.

Record real traffic

Run version 1 with --record, which appends every request, its response and each DynamoDB call to a file:

$ sspur deploy local notes_v1.ssp --port 8080 --record traffic.jsonl
listening on http://127.0.0.1:8080
...

$ curl -s -w ' %{http_code}\n' -X POST localhost:8080/notes -d '{"id":"n1","text":"buy milk"}'
{"id":"n1","text":"buy milk"} 201
$ curl -s -w ' %{http_code}\n' localhost:8080/notes/n1
{"id":"n1","text":"buy milk"} 200
$ curl -s -w ' %{http_code}\n' localhost:8080/notes/n9
{"error":"not found"} 404

Replay it against version 2

deploy replay runs each recorded request alone against the new code, with the store seeded from what the request read, and compares status, response and writes:

$ sspur deploy replay traffic.jsonl notes_v2.ssp
replayed 3 requests: 1 identical, 0 extended (only new null fields), 2 differ
  local-00000001 POST /notes response
    recorded: {"id":"n1","text":"buy milk"}
    new:      {"id":"n1","tags":[],"text":"buy milk"}
  local-00000001 POST /notes writes
    recorded: [{"key":{"S":"n1"},"op":"put","store":"Notes","value":{"id":"n1","text":"buy milk"}}]
    new:      [{"key":{"S":"n1"},"op":"put","store":"Notes","value":{"id":"n1","tags":[],"text":"buy milk"}}]
  local-00000002 GET /notes/n1 response
    recorded: {"id":"n1","text":"buy milk"}
    new:      {"id":"n1","tags":[],"text":"buy milk"}

Every status matches, and the only differences are the new tags field. That is the change we meant to make, so it is safe to ship. A response that only gains null fields counts as extended, which fails only with --strict. Replay exits with status 1 on any difference, so it can gate a deploy.

Swap and roll back

deploy swap starts version 2 beside the running version and sends new requests to it. In-flight requests finish where they started:

$ sspur deploy swap notes_v2.ssp --port 8080
...
$ sspur deploy status --port 8080
{"canary":null,"previous":{"inflight":0,"version":"1d67e631ca7fd732"},"stable":{"inflight":0,"version":"7cd30b84fad22a9a"},"svc":"notes"}

The old note reads through migrate_Notes, in memory, and new writes use the new shape:

$ curl -s -w ' %{http_code}\n' localhost:8080/notes/n1
{"id":"n1","text":"buy milk","tags":[]} 200
$ curl -s -w ' %{http_code}\n' -X POST localhost:8080/notes/n1/tags -d '"home"'
{"id":"n1","text":"buy milk","tags":["home"]} 201
$ curl -s -w ' %{http_code}\n' localhost:8080/tags/home
[{"id":"n1","text":"buy milk","tags":["home"]}] 200

deploy rollback makes the previous version live again. It reads n1 from the version 1 copy that unmigrate_Notes wrote alongside the tagged note:

$ sspur deploy rollback --port 8080
{"canary":null,"previous":{"inflight":0,"version":"7cd30b84fad22a9a"},"stable":{"inflight":0,"version":"1d67e631ca7fd732"},"svc":"notes"}
$ curl -s -w ' %{http_code}\n' localhost:8080/notes/n1
{"id":"n1","text":"buy milk"} 200

--weight P on swap sends only P% of new requests to the new version, as a canary, and deploy promote makes it live. A canary needs unmigrate_, because both versions write. deploy backfill rewrites stored items in the new shape, each write conditional on the item being unchanged since the scan.

For AWS, deploy plan turns the same model into Lambda versions, a live alias, CodeDeploy traffic shifting with alarms, and a backfill function. ADR 0019 has the details.