Build a web app

sspur new web-app writes SSPUR’s reference application: a notes service with user accounts, a JSON API and HTML pages, stored in SQLite. It is a complete program rather than a sketch: passwords are hashed with Argon2id, session tokens are stored as digests, forms carry CSRF tokens, logins are rate limited, every input is validated, and it comes with unit, end-to-end, concurrency and load tests, a debugging walkthrough, backups and a deploy guide. This page walks through the code it generates.

$ sspur new web-app notes
created notes from the web-app template

  cd notes
  sspur run              # then open http://127.0.0.1:8080
  sspur run tests/run.ssp

README.md has the rest: configuration, debugging, backups and deploying.

The layout

The program is split into modules, and each module is a package inside the app’s directory: models, storage, auth and views, plus sessions, pagination and passwords under packages/, written to be reused. sspur.toml lists them as path dependencies. A dependency inside the package’s own directory follows its source, so an edit to storage/lib.ssp takes effect at the next sspur run, while a dependency from elsewhere stays pinned by its hash in sspur.lock (Packages). main.ssp holds the configuration, the routes and the handlers.

Valid input is a type

The input rules are refinements on the types that request bodies decode into:

// An email address as a person types it: one `@`, a dot in the domain, no spaces.
pub type Email = Str where email_ok(_)

// A note's title: not blank, at most 200 characters.
pub type Title = Str where _.trim != "" and _.len <= 200

// A note's text, at most 20,000 characters.
pub type Body = Str where _.len <= 20000

// A tag: 1 to 32 letters, digits or dashes.
pub type Tag = Str where tag_ok(_)

// At most 10 tags.
pub type Tags = List[Tag] where _.len <= 10

// What a client sends to sign up or log in.
pub type Credentials = {email: Email, password: Str where _.len <= 1024}

// A new note; `body` and `tags` may be left out.
pub type NewNote = {title: Title, body: Opt[Body], tags: Opt[Tags]}

// A change to a note: the fields that are present replace the note's.
pub type NotePatch = {title: Opt[Title], body: Opt[Body], tags: Opt[Tags]}

The server decodes a body into the type and checks every where clause before the handler runs, so a handler never sees a blank title or eleven tags. A body that breaks a rule gets a 400 that names the field and the rule, body.title: must satisfy _.trim != "" and _.len <= 200. The HTML forms use the same types: the handler turns the form fields into JSON and decodes it with json.decode[NewNote], which checks the same clauses and returns an err instead.

Routes and the app’s state

A svc lists the routes. Here it takes a parameter, app, which the server is started with, and every endpoint parameter named app receives it: that is how handlers get the database and the keys without globals.

// Settings: app.json, then NOTES_<FIELD> environment variables (NOTES_PORT=9000).
type Config = {mode: Str where _ == "dev" or _ == "production", host: Str, port: Int where _ >= 0 and _ <= 65535, workers: Int where _ >= 0 and _ <= 64, db: Str where _ != "", secret: Secret[Str], secure_cookies: Opt[Bool], trust_proxy: Bool, static_dir: Str}

fn defaults() -> Config
= Config{mode: "dev", host: "127.0.0.1", port: 8080, workers: 0, db: "notes.db", secret: secret(""), secure_cookies: none, trust_proxy: false, static_dir: "public"}

// What every handler gets: the database, the key that signs CSRF tokens, the cookie settings and
// the login policy.
type App = {db: Db, key: Secret[Str], session: sessions.CookieOpts, anon: sessions.CookieOpts, auth: auth.Policy, trust_proxy: Bool}

// The JSON API and the HTML pages. Parameters named `app` get the App the server was started
// with, `req` the request, `{id}` the path segment, other GET parameters the query string and
// the other POST and PATCH parameter the JSON body.
svc api(app: App)
  ep get "/api/health" = health
  ep post "/api/signup" = signup
  ep post "/api/login" = login
  ep post "/api/logout" = logout
  ep get "/api/me" = me
  ep get "/api/notes" = list_notes
  ep post "/api/notes" = create_note
  ep get "/api/notes/{id}" = get_note
  ep patch "/api/notes/{id}" = update_note
  ep delete "/api/notes/{id}" = delete_note
  ep get "/" = home
  ep get "/login" = login_page
  ep post "/login" = login_submit
  ep get "/signup" = signup_page
  ep post "/signup" = signup_submit
  ep post "/logout" = logout_submit
  ep get "/notes/new" = new_note_page
  ep post "/notes" = create_note_submit
  ep get "/notes/{id}" = note_page
  ep post "/notes/{id}" = update_note_submit
  ep post "/notes/{id}/delete" = delete_note_submit

Parameters bind by name and type: {id} from the path, req: Request the raw request, q and limit from the query string of a GET, and the one other parameter of a POST or PATCH from the JSON body. Results become responses: a value is 200 with its JSON (201 for a POST), none is 404 and Unit 204. An error a handler raises picks its status by its variant’s name, which is why AppErr has Invalid, Unauthorized, Forbidden, NotFound, Conflict and TooMany.

A handler

Every API handler starts by finding its caller:

// The signed-in user of an API request: a Bearer token, or the session cookie with the
// x-csrf-token header on requests that change something.
fn caller(app: App, req: Request) -> User ! db, fail[DbErr], fail[AppErr], time
= do
  (token, cookie) = match sessions.bearer(req)
    | some(t) => (t, false)
    | none => (req.cookie(app.session.name).or(""), true)
  some(user) = auth.user_of(app.db, token, now()) else raise Unauthorized
  if cookie and unsafe_method(req.method) and not sessions.csrf_ok(app.key, sessions.digest(token), req.header("x-csrf-token").or("")) then raise Forbidden{why: "a request with the session cookie needs the x-csrf-token header from /api/me"}
  user

fn create_note(app: App, req: Request, n: NewNote) -> Note ! db, fail[DbErr], fail[AppErr], time, log
= do
  user = caller(app, req)
  note = storage.create_note(app.db, user.id, n, now())
  event(req, "note.created", some(user.id), some(note.id))
  note

The effects in the signature are the whole story of what create_note may do: use the database, raise a database or app error, read the clock, and log. event writes one structured line, whose fields become the fields of a JSON log line under SSPUR_LOG=json, next to the request id the server gave the request.

Storage

Each query is a function, values reach SQL only as ? parameters, and rows come back as records. A unique email becomes a Conflict:

// Creates an account; an email that is taken is `Conflict`.
pub fn create_user(c: Db, email: Str, password_hash: Str, now: Time) -> User ! db, fail[DbErr], fail[AppErr]
= catch db.one[User](c, "insert into users (email, password, created_at) values (?, ?, ?) returning id, email, created_at", [email, password_hash, now])
  | Constraint{msg} => raise Conflict{why: "an account with this email already exists"}
  | e => raise e

A change that reads before it writes runs in a transaction, so two edits at once can’t lose either:

// Applies `p` to the note `id` of `user`, reading and writing in one transaction so that two
// changes at once can't lose either.
pub fn update_note(c: Db, user: Int, id: Int, p: NotePatch, now: Time) -> Opt[Note] ! db, fail[DbErr]
= db.tx(c, () => do
  some(old) = note(c, user, id) else none
  tags = p.tags.map(models.normal_tags).or(old.tags)
  r = db.one[NoteRow](c, "update notes set title = ?, body = ?, tags = ?, updated_at = ? where id = ? and user_id = ? returning id, title, body, tags, created_at, updated_at", [p.title.map(_.trim).or(old.title), p.body.or(old.body), json.encode(tags), now, id, user])
  some(to_note(r)))

Listing uses SQLite’s full-text search and its JSON functions, with the optional filters written into the query rather than built as strings:

// A page of `user`'s notes, newest first, matching the search `q` and the tag `tag` when they
// are not empty.
pub fn list_notes(c: Db, user: Int, q: Str, tag: Str, r: PageReq) -> Page[Note] ! db, fail[DbErr]
= do
  m = search_terms(q)
  total = db.one[Int](c, "select count(*) from notes n where n.user_id = ?1 and (?2 = '' or n.id in (select rowid from notes_fts where notes_fts match ?2)) and (?3 = '' or exists (select 1 from json_each(n.tags) where value = ?3))", [user, m, tag.lower])
  rows = db.query[NoteRow](c, "select n.id, n.title, n.body, n.tags, n.created_at, n.updated_at from notes n where n.user_id = ?1 and (?2 = '' or n.id in (select rowid from notes_fts where notes_fts match ?2)) and (?3 = '' or exists (select 1 from json_each(n.tags) where value = ?3)) order by n.updated_at desc, n.id desc limit ?4 offset ?5", [user, m, tag.lower, r.limit, r.offset])
  pagination.page(rows.map(to_note), total, r)

The schema is a list of migration steps in the same file; db.migrate applies the new ones at startup.

Accounts and sessions

Logging in checks the rate limit for the account and for the client’s address, compares the password in constant time, and starts a session whose token goes to the client while only its SHA-256 is stored:

fn too_many(c: Db, p: Policy, key: Str, now: Time) -> Unit ! db, fail[DbErr], fail[AppErr]
= do
  (n, first) = storage.failures(c, key, now.sub(mins(p.window_mins)))
  if n >= p.max_failures then raise TooMany{retry_after: max(first.map(f => f.add(mins(p.window_mins)).since(now).secs).or(60), 1)}

// Signs in with an email and password. `client` is the caller's address when it is known (""
// otherwise); failures count against the account and the address, and past the limit every
// attempt is `TooMany` until the window passes. A wrong email or password is `Unauthorized`,
// the same for both.
pub fn log_in(c: Db, p: Policy, email: Str, password: Str, client: Str, now: Time) -> Session ! db, fail[DbErr], fail[AppErr], rand
= do
  e = models.normal_email(email)
  keys = ["email:" + e] + (if client == "" then [] else ["client:" + client])
  for k in keys
    too_many(c, p, k, now)
  match storage.account(c, e)
  | some(a) if passwords.verify(secret(password), a.password) => do
    _ = storage.clear_failures(c, "email:" + e)
    _ = if passwords.needs_rehash(a.password) then storage.set_password(c, a.id, passwords.hash(secret(password))) else 0
    start(c, User{id: a.id, email: a.email, created_at: a.created_at}, p, now)
  | found => do
    _ = found.is_some or passwords.verify(secret(password), decoy())
    for k in keys
      _ = storage.add_failure(c, k, now)
    raise Unauthorized

A wrong password and an unknown email give the same answer, and an unknown email still costs one Argon2id check, so neither the response nor its timing tells whether an account exists.

Forms and CSRF

The HTML pages are server-rendered forms that work without JavaScript. A CSRF token is an HMAC of the session’s digest under the server’s key, so it can’t be forged or carried to another session:

// The CSRF token for a form or request bound to `binding` (a session digest, or the value of an
// anonymous cookie): an HMAC under the server's key, so it can't be forged or moved to another
// session.
pub fn csrf_token(key: Secret[Str], binding: Str) -> Str
= crypto.hmac256(key, "csrf\n" + binding).base64url

// Whether `given` is the CSRF token for `binding`, compared in constant time.
pub fn csrf_ok(key: Secret[Str], binding: Str, given: Str) -> Bool
= binding != "" and crypto.equal(given, csrf_token(key, binding))

// Whether a form post carries the CSRF token of its session.
fn form_ok(app: App, req: Request, b: Browser) -> Bool
= sessions.csrf_ok(app.key, sessions.digest(b.token), req.form("csrf").or(""))

fn create_note_submit(app: App, req: Request) -> Response ! db, fail[DbErr], time, log
= match browser(app, req)
  | none => redirect("/login", [])
  | some(b) if not form_ok(app, req, b) => forbidden()
  | some(b) => match note_input(req)
    | err(e) => note_form_again(b, req, none, e)
    | ok(n) => do
      note = storage.create_note(app.db, b.user.id, n, now())
      event(req, "note.created", some(b.user.id), some(note.id))
      redirect("/", [])

The session cookie is HttpOnly and SameSite=Lax, and in production Secure with the __Host- prefix. Pages are sent with a strict content security policy, and every value a user wrote goes through views.esc. Login and signup forms have no session yet, so their token is bound to a random anon cookie instead.

Startup

// The settings, or why they are not usable. In production the secret must be set: it signs the
// CSRF tokens, and a random one would sign out everyone at every restart.
fn load_config() -> Res[Config, Str] ! fs, env
= match config.load(defaults(), env_var("NOTES_CONFIG").or("app.json"), "NOTES_")
  | err(e) => err(e)
  | ok(c) if c.mode == "production" and not c.secret.check(s => s.len >= 32) => err("NOTES_SECRET must be set to at least 32 random characters in production; make one with: openssl rand -base64 48")
  | ok(c) => ok(c)

type Started = {event: Str, mode: Str, db: Str, migrations: Int}

fn start(c: Config) -> Unit ! net, log, db, fail[DbErr], rand, time
= do
  conn = db.open(c.db)
  applied = db.migrate(conn, storage.migrations())
  _ = storage.sweep(conn, now(), now().sub(days(1)))
  key = if c.secret.check(s => s != "") then c.secret else do
    log.warn("NOTES_SECRET is not set: using a random key, so sessions end when the server restarts")
    secret(crypto.token(32))
  secure = c.secure_cookies.or(c.mode == "production")
  app = App{db: conn, key, session: sessions.cookie_opts(if secure then "__Host-session" else "session", 14, secure), anon: sessions.cookie_opts("anon", 1, secure), auth: auth.default_policy(), trust_proxy: c.trust_proxy}
  log.info(json.encode(Started{event: "started", mode: c.mode, db: c.db, migrations: applied}))
  http.serve_with(http.server(c.port) with host := c.host, workers := c.workers, gzip := true, health := "/healthz", files := [("/static", c.static_dir)], max_body := 65536, timeout_ms := 15000, api(app))
  log.info("stopped")

fn main() -> Unit ! net, log, env, fs, db, fail[DbErr], rand, time, proc
= match load_config()
  | ok(c) => start(c)
  | err(e) => do
    log.error("cannot start: " + e)
    exit(2)

config.load reads app.json and NOTES_* variables and checks the where clauses of Config, so NOTES_PORT=70000 stops the server with port: must satisfy _ >= 0 and _ <= 65535. In production a missing secret is an error too. The server shuts down gracefully on SIGTERM, which is what systemd sends.

Tests

Handlers are functions, so a test can call the svc directly, with an in-memory database:

test api_round_trip = catch do
    app = test_app()
    up = call(app, "POST", "/api/signup", [], "\{\"email\":\"ada@example.com\",\"password\":\"correct horse battery\"}")
    auth = [("authorization", "Bearer " + token_of(up))]
    made = call(app, "POST", "/api/notes", auth, "\{\"title\":\"Plan\",\"tags\":[\"work\"]}")
    id = made.json[Note].map(n => n.id).or(0)
    listed = call(app, "GET", "/api/notes?tag=work", auth, "").json[Page[Note]].map(p => p.total)
    changed = call(app, "PATCH", "/api/notes/{id}", auth, "\{\"body\":\"ship it\"}").json[Note].map(n => n.body)
    gone = call(app, "DELETE", "/api/notes/{id}", auth, "").status
    up.status == 201 and made.status == 201 and listed == ok(1) and changed == ok("ship it") and gone == 204 and call(app, "GET", "/api/notes/{id}", auth, "").status == 404
  | _ => false

The end-to-end test starts the real server as another process on a free port and reads the port from its output:

// Starts `sspur run main.ssp` on port 0 and waits (up to 10 minutes, for a first native build
// on a slow machine) for the line that says which port it got.
fn start(sspur: Str, tier: Str, db: Str) -> Running ! proc, rand, time, div, fail[E2eErr]
= do
  args = ["run"] + (if tier == "interp" then ["--interp"] else []) + ["main.ssp"]
  env = [("NOTES_PORT", "0"), ("NOTES_DB", db), ("NOTES_MODE", "production"), ("NOTES_SECURE_COOKIES", "false"), ("NOTES_SECRET", crypto.token(48)), ("NOTES_WORKERS", "8")]
  p = match proc.spawn(sspur, args, env)
    | ok(p) => p
    | err(e) => raise Failed{step: "start", why: e}
  deadline = now_ms() + 600000
  var seen = ""
  var port = 0
  while port == 0 and now_ms() < deadline and p.running()
    match p.line(1000)
    | some(l) => do
      seen := seen + l + "\n"
      port := port_in(l).or(0)
    | none => ()
  if port == 0 then raise Failed{step: "start", why: "the server did not say where it listens:\n" + seen}
  Running{p, base: "http://127.0.0.1:{port}", log: seen}

and then drives it with the HTTP client, including from many clients at once:

// Many clients at once: 8 people each create 20 notes, and 8 more clients add 20 each to one
// shared account. Every note must be there afterwards.
fn concurrency(base: Str) -> Int ! net, conc, fail[E2eErr]
= do
  tokens = (0..8).map(i => signup(base, "user{i}@example.com"))
  shared = signup(base, "shared@example.com")
  failures = atomic(0)
  for i in par(0..16)
    make_notes(base, if i < 8 then tokens[i] else shared, 20, failures)
  check("parallel requests", failures.load == 0, "{failures.load} requests failed")
  for i in 0..8
    check("user {i} notes", count(base, tokens[i], "par") == 20, "{count(base, tokens[i], "par")}")
  check("shared notes", count(base, shared, "par") == 160, "{count(base, shared, "par")} of 160: writes were lost")
  320

$ sspur test
5 passed, 0 failed
$ sspur test tests/storage.ssp
8 passed, 0 failed
$ sspur run tests/e2e.ssp
e2e passed (server native): 350 notes, 800 requests
...

sspur run tests/run.ssp runs every test natively and then in the interpreter, the end-to-end test included.

Run it

$ NOTES_PORT=8080 NOTES_SECRET=use-openssl-rand-base64-48-for-a-real-one sspur run &
info: {"event":"started","mode":"dev","db":"notes.db","migrations":4}
info: listening on http://127.0.0.1:8080

$ curl -s -w ' %{http_code}\n' localhost:8080/api/health
{"status":"ok","db":true} 200
$ curl -s -w ' %{http_code}\n' localhost:8080/api/signup -d '{"email":"ada","password":"correct horse battery"}'
{"error":"invalid request","detail":"body.email: must satisfy models.email_ok(_)"} 400
$ curl -s -w ' %{http_code}\n' localhost:8080/api/signup -d '{"email":"ada@example.com","password":"password123"}'
{"error":{"tag":"Invalid","field":"password","why":"the password is too easy to guess"}} 400
$ curl -s -w ' %{http_code}\n' localhost:8080/api/notes
{"error":"Unauthorized"} 401

The generated README has the rest: the configuration table, stopping in a handler with lldb (lldb -- sspur run --debug main.ssp, see Debugging), backups with sspur run tools/backup.ssp, and deploying on a Linux VM as a systemd service behind Caddy, with the unit file and Caddyfile in deploy/.