Blog ·
Ownership transfer was a one-row UPDATE. Then we added least-privilege.
Hardening queries to per-tenant roles and RLS quietly broke our ownership transfer: it retargeted one authorization store out of four. Transfer must move them all — idempotently, in one batch.
Multi-tenant Postgres, one schema per tenant database. On day one, every query ran as the shared owner role, so "transfer this database to the user who just signed in" was honestly a one-row registry UPDATE — flip tenant_id, done. We wrote that down as a design decision, and it was the right one at the time.
Least-privilege arrived; the transfer path didn't get the memo
Months later we hardened the query path. Every query now runs SET LOCAL ROLE tenant_<hash> against per-tenant grants, and a row-level-security policy whose USING clause bakes the tenant id in as a literal. Four places now store who owns this database: the role itself, its grants, the runner's role membership, and the policy literal. The transfer path still updated exactly one — the registry row.
So transfers kept working in the registry — the sidebar showed the database under its new owner — while every transferred database went permanently unqueryable: the role the session now switches to was never created, never granted USAGE, never made a WITH SET member, and the RLS literal still named the old tenant. Three authorization layers said "old owner"; one row said "new owner"; the row lost.
-- The transfer that "worked" — one of four authorization stores:
UPDATE databases SET tenant_id = :new_tenant WHERE id = :db_id;
-- What least-privilege actually requires, idempotently, in one batch:
DO $$ BEGIN
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'tenant_9047fe') THEN
CREATE ROLE tenant_9047fe NOLOGIN; -- 1. the role exists
END IF;
END $$;
GRANT USAGE ON SCHEMA app TO tenant_9047fe; -- 2. the grants
GRANT tenant_9047fe TO query_runner WITH SET TRUE; -- 3. the membership
ALTER POLICY tenant_isolation ON app.facts -- 4. the RLS literal
USING (tenant_id = 'tenant-9047fe');The error wore a cold-start costume
Worse, the failure was invisible. SET ROLE to a missing role fails with a deterministic SQLSTATE — 22023 (invalid parameter value: the role does not exist) or 42501 (insufficient privilege) — and our generic error branch re-labelled both as "couldn't reach the database," the same message a serverless cold start produces. No code was logged anywhere. For nine end-to-end runs we diagnosed retry timing on an error that was never transient.
The tell that finally broke the costume: creates succeeded while the transferred database failed in the same second — for minutes on end — with the correct SQL planned every time. Cold starts don't select their victims by ownership history. Deterministic failures wearing a transient error's label do.
Two general fixes
- An ownership transfer must retarget every place authorization state lives — role existence, grants, role membership, policy literals — idempotently, in one batch. The day you turn least-privilege on, grep for every writer of your tenant column: each one is a transfer path that just silently broke.
- A catch-all error branch must log the code it swallows. Re-labelling an error without recording the original SQLSTATE means you will re-diagnose the same bug from scratch, behind a message that actively points you at the wrong cause.
The general shape: an invariant added in one subsystem (queries run least-privileged) creates an obligation in another (transfer must move authorization state), and nothing but a human's memory connects the two. When you add a security layer, walk every path that writes the identity it keys on — not just the paths that read it.
(This is a build note from nlqdb, the database you query in plain English. Honest split: this is a Postgres multi-tenancy lesson from our adoption path, not a product feature.)