System Design
O6

Multi-tenant SaaS

Many customers share one system. Each tenant must see only its own data, must not slow the others, and needs its own limits, settings, audit trail and webhooks.

Not startedSaved in this browser only.
  1. 1Start with shared tables: tenant_id first in every key, and row-level security as a second check.
  2. 2Give each tenant its own limits: a token bucket per tenant, quotas, and a job queue that takes turns over tenants.
  3. 3Sign webhooks, retry with backoff, dead-letter after 5 attempts, and keep each subscription in order.
  4. 4Move to cells when one database or one blast radius is too big. Give the largest tenant its own cell.
O6
    A

    What goes wrong

    a leak, and a noisy neighbour
    1 · a query forgets the tenantService, tenant 1WHERE id = 7invoicestenant 1invoice 7tenant 2invoice 7tenant 3invoice 7✕ 2 leaks2 · one tenant floods the shared queuetenant 1: 300 jobs at oncetenant 2, last in4 workersoldest first✕ tenant 2 waits 490 ms (p99)for a job that takes 5 ms
    • A filter in application code depends on every query being right.
    • Shared CPU, connections and queues serve the loudest tenant first.

    Shared tables have two risks: one forgotten filter leaks another tenant's data, and one tenant's burst delays everyone.

    B

    Tenancy models

    where each tenant's rows live
    modelwherestatus
    Shared tables, filter in code onlyServiceNot approved
    Shared tables, tenant_id first, RLSPostgresApproved
    Schema per tenantPostgresHundreds of tenants
    Database per tenantPostgresFew large tenants
    Cells: shared tables per cellRouterAt scale
    Tenant id from the request bodyServiceNot approved

    Schema and database per tenant multiply the catalog and every migration by the number of tenants. Panel C shows by how much.

    I start with shared tables and row-level security. I move a tenant to its own cell only when its size or contract needs it.

    C

    Pick a model

    catalog costs measured on Postgres, multiplied by tenants
    model
    tenants
    Postgres · one databaseinvoices (tenant_id, id, ...)tenant 1invoice 1tenant 2invoice 2tenant 1invoice 3tenant 3invoice 4tenant 2invoice 5tenant 1invoice 61,000 tenants as rows
    databases
    1
    catalog entries
    12
    empty size
    124 kB
    one migration
    1 ALTER statement, 2.8 ms
    new tenant
    insert one tenants row
    no tenant filter in a query
    ✓ row-level security returns only the tenant of the transaction
    one noisy tenant
    ✕ slows every tenant on the server: CPU, I/O and connections are shared
    one database server down
    ✕ 100% of tenants
    connection pools
    1

    Measured per copy of 5 tables on the lab Postgres: 12 catalog entries, 124 kB as a schema, 8.0 MB as a database. A migration took 0.23 ms per schema. Cells of 500 tenants are an example size.

    At 10,000 tenants, a schema each means 120,000 catalog entries and 10,000 runs of every migration. Shared tables keep one copy.

    D

    Schema

    entities, then SQL
    same tenant onlytenantsPKidUQslugFKplanlimits, flagscellwhich stackstatusactive, deletingno RLS: the router reads itprojectsper tenantPKtenant_idleads the keyPKidnameUQ per tenantRLS: tenant_id = currentinvoicesper tenantPKtenant_idPKidFKproject_idwith tenant_idamount_centsbigintFK (tenant_id, project_id)flag_overridesper tenantPKtenant_idPKflagenabledwins over planRLS: tenant_id = currentaudit_logper tenantPKtenant_idPKidactorwhoaction, targetdid whatgrant SELECT, INSERT only
    keys
    (tenant_id, id). Two tenants can both have project 1.
    indexes
    Start with tenant_id, so a query reads one tenant's range.
    later
    Hash-partition or shard by tenant_id with no change to keys.
    tenants and tenant tablessql
    CREATE TABLE plans (
      code         text PRIMARY KEY,
      max_projects int  NOT NULL,
      jobs_per_min int  NOT NULL   -- the tenant's share of the shared job queue
    );
    
    CREATE TABLE tenants (
      id     bigint PRIMARY KEY,
      slug   text   NOT NULL UNIQUE,
      plan   text   NOT NULL REFERENCES plans (code),
      cell   int    NOT NULL DEFAULT 11,   -- which copy of the stack serves this tenant
      status text   NOT NULL DEFAULT 'active' CHECK (status IN ('active', 'suspended', 'deleting'))
    );
    
    -- Every tenant table starts its key with tenant_id.
    CREATE TABLE projects (
      tenant_id bigint NOT NULL REFERENCES tenants (id),
      id        bigint NOT NULL,
      name      text   NOT NULL,
      PRIMARY KEY (tenant_id, id)2,
      UNIQUE (tenant_id, name)3
    );
    
    CREATE TABLE invoices (
      tenant_id    bigint NOT NULL,
      id           bigint NOT NULL,
      project_id   bigint NOT NULL,
      amount_cents bigint NOT NULL CHECK (amount_cents >= 0),
      status       text   NOT NULL DEFAULT 'draft',
      PRIMARY KEY (tenant_id, id),
      FOREIGN KEY (tenant_id, project_id) REFERENCES projects (tenant_id, id)4
    );
    CREATE INDEX invoices_status ON invoices (tenant_id, status);
    
    -- Append-only: who did what, in which tenant.5
    CREATE TABLE audit_log (
      tenant_id bigint      NOT NULL,
      id        bigint      GENERATED ALWAYS AS IDENTITY,
      at        timestamptz NOT NULL DEFAULT now(),
      actor     text        NOT NULL,
      action    text        NOT NULL,
      target    text        NOT NULL,
      PRIMARY KEY (tenant_id, id)
    );
    1. 1The router reads this column to send the tenant to its copy of the stack.
    2. 2tenant_id first: every index range is one tenant.
    3. 3Names are unique inside a tenant, not across tenants.
    4. 4Both sides carry tenant_id, so an invoice cannot point at another tenant's project.
    5. 5The grants in panel G let the service insert and read audit rows, never change them.

    tenant_id leads every key, composite foreign keys keep links inside one tenant, and the audit table accepts inserts only.

    E

    The request path

    click a step; its path lights up
    Clientsigned tokenRoutertenant → cellServicestateless, N copiesRedisbuckets, flagsPostgresshared tables, RLS, queueWorkersfair scheduler, dispatcherTelemetrytenant_id on everything

    Step 1: Resolve tenant

    • Read the tenant from the signed token or the subdomain. Never from the request body.
    • Look up the tenant cell in the directory, and send the request there.

    If it fails

    Unknown or suspended tenant: refuse with 403 before any work.

    The router finds the tenant's cell, Redis limits the tenant, and Postgres scopes every query to the tenant through row-level security.

    F

    Capabilities used

    what each tool gives you
    toolcapabilitywhat it gives this designalso used for
    PostgresRow-level security policiesEvery query sees and writes only the tenant of the transaction.Per-user rows, regions
    Postgresset_config(..., is_local = true)The tenant setting ends at COMMIT, so a pooled connection carries nothing over.Request ids for audit
    PostgresRoles and GRANTThe service role owns nothing, so RLS applies. Audit rows are insert-only.Read-only reporting roles
    PostgresComposite primary and foreign keysKeys lead with tenant_id; links cannot cross tenants.Sharding keys
    PostgresFOR UPDATE SKIP LOCKEDWorkers take different tenants' turns at once without waiting.Job queues, outboxes
    PostgresData-modifying CTEsJobs and the tenant's ready count change in one statement.Counters beside inserts
    PostgresDISTINCT ONPicks the oldest pending event of each subscription: order per subscription.Latest row per group
    PostgresHash partitionsSplit the largest tables by tenant_id. Vacuum and indexes stay small.Time-series retention
    RedisScripts, key expiryA token bucket per tenant, refilled by the plan's rate.Rate limits, quotas
    RedisHashes with a TTLCache each tenant's plan and flags for a few seconds.Sessions
    ServiceHMAC-SHA256Signs each webhook with a secret per subscription.Signed URLs

    Postgres gives me row-level security, composite keys and a queue with SKIP LOCKED. Redis gives me a token bucket per tenant.

    G

    Row-level security

    pseudo code; attempts recorded from Postgres
    scope each requestpseudo code
    handle(request):
      tenant = tenant of the signed-in user          // never from the body1
      BEGIN
        SET LOCAL role = service role                // not the table owner2
        SET LOCAL app.tenant_id = tenant             // ends with the transaction3
        run the queries                              // no tenant filter needed
      COMMIT
    
    policy on every tenant table:
      a row is visible and writable only IF row.tenant_id = app.tenant_id4
    1. 1A client can write any tenant id in a body. The signed token is the only source.
    2. 2The owner bypasses row-level security unless the table forces it.
    3. 3A session setting would stay on the pooled connection for the next request.
    4. 4No tenant set: the setting is NULL, and NULL matches no row.
    Tested source SQL: policies and grants · SQL: scope a transaction
    SQL: policies and grantssql
    -- The tenant comes from the transaction. Unset, it is NULL, and NULL matches no row.
    CREATE FUNCTION current_tenant() RETURNS bigint LANGUAGE sql STABLE
      AS $$ SELECT nullif(current_setting('app.tenant_id', true), '')::bigint $$;
    
    ALTER TABLE projects ENABLE ROW LEVEL SECURITY;
    ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;
    ALTER TABLE audit_log ENABLE ROW LEVEL SECURITY;
    ALTER TABLE flag_overrides ENABLE ROW LEVEL SECURITY;
    CREATE POLICY tenant ON projects USING (tenant_id = current_tenant());
    CREATE POLICY tenant ON invoices USING (tenant_id = current_tenant());
    CREATE POLICY tenant ON audit_log USING (tenant_id = current_tenant());
    CREATE POLICY tenant ON flag_overrides USING (tenant_id = current_tenant());
    
    GRANT SELECT, INSERT, UPDATE, DELETE ON projects, invoices TO tenancy_app;
    GRANT SELECT ON plans, tenants, flags, flag_overrides TO tenancy_app;
    -- The service can add audit rows and read them, never change them.
    GRANT SELECT, INSERT ON audit_log TO tenancy_app;
    SQL: scope a transactionsql
    -- Each request runs as the service role, with its tenant set for this transaction only.
    SELECT set_config('role', 'tenancy_app', true), set_config('app.tenant_id', $1, true);
    attempt, asresult
    List projects with no WHERE clause · tenant 13 rows
    Read tenant 2's project by its key · tenant 10 rows
    Update every invoice of tenant 2 · tenant 10 rows
    Insert a project for tenant 2 · tenant 1Refused, 42501
    Move its own project to tenant 2 · tenant 1Refused, 42501
    Link an invoice to tenant 2's project · tenant 1Refused, 23503
    Change an audit row · tenant 1Refused, 42501
    Delete an audit row · tenant 1Refused, 42501
    No tenant set · service role0 rows
    Tenant set for the session; the next request reuses the connection · service role3 of 6 rows
    Connect as the table owner · owner6 of 6 rows

    Tenant 1 owns 3 of the 6 projects. 42501: the policy or a missing grant refused it. 23503: no such project in tenant 1.

    I set the tenant for each transaction, connect as a role that owns no tables, and let the policy filter every query.

    H

    Noisy neighbours

    limits per tenant
    methodwherestatus
    Token bucket per tenantRedisApproved
    Quotas: rows, storage, seatsPostgresApproved
    Turns over tenants in the queuePostgresApproved
    Cap on running jobs per tenantPostgresIdle workers
    Statement timeout per rolePostgresApproved
    Oldest job first, one queuePostgresNot approved
    Big tenant to its own cellRouterLargest

    A hard cap leaves workers idle while only one tenant has work. Turns keep every worker busy.

    Every shared resource gets a limit per tenant: requests, jobs, storage and connections.

    I

    A fair job queue

    pseudo code
    claim, in turnpseudo code
    enqueue(tenant, jobs):                          // one statement
      insert the jobs; tenant.ready += count(jobs)1
    
    claim(worker):                                   // one statement
      turn = tenant with ready > 0, served longest ago2,
             skipping tenants another worker claims right now3
      move turn to the back of the line; turn.ready -= 1
      take turn's oldest ready job4; mark it running
    1. 1A ready count per tenant. Without it, each claim probes the jobs of every tenant, and the lab rate fell to about 200 jobs a second.
    2. 2A tick from a sequence. A tenant that just got a job goes to the back.
    3. 3SKIP LOCKED on the tenant row. Four workers serve four different tenants at once.
    4. 4Inside one tenant, jobs still run oldest first.
    Tested source SQL: enqueue and claim in turn · Go: claim
    SQL: enqueue and claim in turnsql
    -- The jobs and the tenant's ready count change in one statement.
    WITH t AS (
      INSERT INTO tenant_queues (tenant_id, ready) VALUES ($1, $2)
      ON CONFLICT (tenant_id) DO UPDATE SET ready = tenant_queues.ready + EXCLUDED.ready
    )
    INSERT INTO jobs (tenant_id) SELECT $1 FROM generate_series(1, $2::int)
    RETURNING id;
    
    -- Round robin over tenants: lock the tenant with work that was served longest ago, move it to
    -- the back of the line, then take its oldest job.
    WITH turn AS (
      SELECT tenant_id FROM tenant_queues
      WHERE ready > 0
      ORDER BY last_served, tenant_id
      LIMIT 1
      FOR UPDATE SKIP LOCKED
    ), served AS (
      UPDATE tenant_queues SET last_served = nextval('serve_order'), ready = ready - 1
      WHERE tenant_id = (SELECT tenant_id FROM turn)
    )
    UPDATE jobs SET state = 'running', worker = $1
    WHERE id = (
      SELECT id FROM jobs
      WHERE tenant_id = (SELECT tenant_id FROM turn) AND state = 'ready'
      ORDER BY id LIMIT 1
      FOR UPDATE SKIP LOCKED
    )
    RETURNING id, tenant_id;
    Go: claimgo
    // Claim takes one ready job for a worker. ok is false when no job is ready, or when every
    // tenant with ready jobs is being claimed by another worker right now.
    func (q *Queue) Claim(ctx context.Context, worker int) (job Job, ok bool, err error) {
      sql := qq["claim_fifo"]
      if q.Sched == Fair {
        sql = qq["claim_fair"]
      }
      err = q.DB.QueryRow(ctx, sql, worker).Scan(&job.ID, &job.Tenant)
      if errors.Is(err, pgx.ErrNoRows) {
        return Job{}, false, nil
      }
      if err != nil {
        return Job{}, false, fmt.Errorf("claim (%s): %w", q.Sched, err)
      }
      return job, true, nil
    }
    

    Each claim locks the tenant that has waited longest, moves it to the back of the line, and takes its oldest job.

    J

    See it: a burst of 300 jobs

    recorded from Postgres; every bar is a job the lab ran
    workers0100200300400500600ms from the burstworker 1worker 2worker 3worker 4wait of each small-tenant job0 ms100 ms200 ms300 ms400 ms500 msx: when the job arrived · y: how long it waited for a worker
    tenant 1, burst of 300 tenants 2 to 6, one job each every 25 ms5 ms of work per job, 4 workers
    small tenants
    ✕ p50 371 ms, p99 490 ms; 75 of 75 jobs waited over 100 ms
    burst
    p99 wait 491 ms; all 300 done at 637 ms

    With oldest first, small tenants waited 490 ms at p99. With turns over tenants, 6.1 ms, and the burst finished in about the same time.

    K

    Per-tenant settings and flags

    pseudo code
    resolve a flagpseudo code
    flag_on(tenant, flag):
      IF tenant has an override for flag: RETURN the override1
      IF tenant.plan IN flag.plans:       RETURN true
      RETURN flag.default
    1. 1Turn a feature on for one tenant to test it, or off for one tenant that has a problem.
    Tested source SQL: flags and overrides · SQL: resolve
    SQL: flags and overridessql
    -- A flag is on for the plans it lists. A per-tenant override wins over the plan.
    CREATE TABLE flags (
      name       text    PRIMARY KEY,
      default_on boolean NOT NULL DEFAULT false,
      plans      text[]  NOT NULL DEFAULT '{}'
    );
    CREATE TABLE flag_overrides (
      tenant_id bigint  NOT NULL REFERENCES tenants (id),
      flag      text    NOT NULL REFERENCES flags (name),
      enabled   boolean NOT NULL,
      PRIMARY KEY (tenant_id, flag)
    );
    SQL: resolvesql
    -- The override, else the plan, else the default.
    SELECT coalesce(o.enabled, t.plan = ANY (f.plans) OR f.default_on)
    FROM tenants t
    JOIN flags f ON f.name = $2
    LEFT JOIN flag_overrides o ON o.tenant_id = t.id AND o.flag = f.name
    WHERE t.id = $1;
    per tenantexamples
    Plan limitsSeats, projects, request rate, jobs a minute
    Feature flagsPlan default plus overrides
    Data rulesRegion, retention, encryption key
    Sign-inSingle sign-on provider, required second factor

    A tenant's override wins over its plan, and the plan wins over the default. I cache the result for a few seconds.

    L

    Audit log and approvals

    who did what
    audit fieldwhy
    tenant_id, actor, action, targetWho did what to which object
    before and after, as JSONWhat changed
    request id, IP addressJoin to logs and traces
    Same transaction as the changeNo change without its row
    Grant INSERT and SELECT onlyNobody edits history
    approval statenext
    requestedapproved or rejected, by another user of the tenant
    approvedapplied once, with the request id as the key
    expiredafter a deadline with no decision

    Every change writes an audit row in the same transaction. Risky actions need a second person to approve them.

    M

    Webhooks to customers

    pseudo code; one subscription, recorded
    dispatch and verifypseudo code
    every tick:
      FOR EACH subscription:
        head = its oldest pending event              // later events wait1
        IF head.next_at > now: skip
        POST body, header t=now, v1=HMAC(secret, now + "." + body)2
        2xx:                     mark delivered
        fails, 5th attempt:      mark dead3           // the next event can go
        fails:                   d = 60 s × 2^(attempt - 1)
                                 next_at = now + random(d/2, d)4
    
    receiver:
      refuse IF the HMAC differs OR |now - t| > 300 s
      apply each event id once5
    1. 1One head per subscription keeps its events in order. A failing endpoint delays only its own events.
    2. 2Signing the timestamp with the body stops a replay with a new timestamp.
    3. 3A dead letter. The customer fixes the endpoint and asks for a redrive.
    4. 4Jitter. Endpoints that failed together do not retry together.
    5. 5A reply lost after the receiver applied the event brings a second copy. The event id stops it.
    Tested source Go: dispatch · Go: sign and verify · SQL: head of each subscription
    Go: dispatchgo
    // Tick sends every subscription's head event that is due at now, one request each.
    func (d *Dispatcher) Tick(ctx context.Context, now int64) (sent []Attempt, nextDue int64, err error) {
      rows, err := d.DB.Query(ctx, qw["heads"])
      if err != nil {
        return nil, 0, fmt.Errorf("read heads: %w", err)
      }
      heads, err := pgx.CollectRows(rows, func(r pgx.CollectableRow) (head, error) {
        var h head
        err := r.Scan(&h.sub, &h.seq, &h.eventID, &h.body, &h.attempts, &h.nextAt, &h.url, &h.secret)
        return h, err
      })
      if err != nil {
        return nil, 0, fmt.Errorf("read heads: %w", err)
      }
      nextDue = -1
      for _, h := range heads {
        if h.nextAt > now {
          if nextDue < 0 || h.nextAt < nextDue {
            nextDue = h.nextAt
          }
          continue
        }
        a := Attempt{At: now, Sub: h.sub, Seq: h.seq, EventID: h.eventID, N: h.attempts + 1}
        if a.Status, err = d.post(ctx, h, now); err != nil {
          return sent, 0, err
        }
        switch {
        case a.Status >= 200 && a.Status < 300:
          a.Result = "delivered"
          _, err = d.DB.Exec(ctx, qw["delivered"], h.sub, h.seq, a.Status)
        case a.N >= d.Backoff.MaxAttempts:
          a.Result = "dead"
          _, err = d.DB.Exec(ctx, qw["dead"], h.sub, h.seq, a.Status)
        default:
          a.Result = "retry"
          a.NextAt = now + d.Backoff.Delay(a.N, d.Rand)
          _, err = d.DB.Exec(ctx, qw["retry_later"], h.sub, h.seq, a.NextAt, a.Status)
          if nextDue < 0 || a.NextAt < nextDue {
            nextDue = a.NextAt
          }
        }
        if err != nil {
          return sent, 0, fmt.Errorf("save attempt %d of %s: %w", a.N, h.eventID, err)
        }
        sent = append(sent, a)
      }
      return sent, nextDue, nil
    }
    
    // post sends one signed request. It returns the status code, or 0 when no reply came in time.
    func (d *Dispatcher) post(ctx context.Context, h head, now int64) (int, error) {
      body := []byte(h.body)
      req, err := http.NewRequestWithContext(ctx, http.MethodPost, h.url, bytes.NewReader(body))
      if err != nil {
        return 0, fmt.Errorf("build request for %s: %w", h.eventID, err)
      }
      req.Header.Set("Content-Type", "application/json")
      req.Header.Set("Webhook-Id", h.eventID)
      req.Header.Set(SignatureHeader, Sign(h.secret, now, body))
      resp, err := d.Client.Do(req)
      if err != nil {
        return 0, nil // a timeout or a refused connection is a failed attempt: retry later
      }
      _, copyErr := io.Copy(io.Discard, resp.Body)
      if err := errors.Join(copyErr, resp.Body.Close()); err != nil {
        return 0, fmt.Errorf("read reply for %s: %w", h.eventID, err)
      }
      return resp.StatusCode, nil
    }
    
    Go: sign and verifygo
    // Sign returns the signature header for body sent at ts: an HMAC-SHA256 over "ts.body" with
    // the subscription's secret. Signing the timestamp stops a replay with a new timestamp.
    func Sign(secret string, ts int64, body []byte) string {
      return fmt.Sprintf("t=%d,v1=%s", ts, digest(secret, ts, body))
    }
    
    func digest(secret string, ts int64, body []byte) string {
      mac := hmac.New(sha256.New, []byte(secret))
      fmt.Fprintf(mac, "%d.", ts)
      mac.Write(body)
      return hex.EncodeToString(mac.Sum(nil))
    }
    
    // Verify checks a signature header on the receiving side. It compares in constant time and
    // refuses a timestamp more than tolerance seconds from now.
    func Verify(secret, header string, body []byte, now, tolerance int64) error {
      tsPart, sigPart, ok := strings.Cut(header, ",")
      tsText, ok1 := strings.CutPrefix(tsPart, "t=")
      sig, ok2 := strings.CutPrefix(sigPart, "v1=")
      if !ok || !ok1 || !ok2 {
        return fmt.Errorf("%w: malformed header", ErrBadSignature)
      }
      ts, err := strconv.ParseInt(tsText, 10, 64)
      if err != nil {
        return fmt.Errorf("%w: bad timestamp: %w", ErrBadSignature, err)
      }
      if !hmac.Equal([]byte(digest(secret, ts, body)), []byte(sig)) {
        return ErrBadSignature
      }
      if now-ts > tolerance || ts-now > tolerance {
        return fmt.Errorf("%w: signed at %d, now %d", ErrStale, ts, now)
      }
      return nil
    }
    
    SQL: head of each subscriptionsql
    -- The head of each subscription is its oldest pending event. Later events wait behind it, so a
    -- customer receives one subscription's events in order.
    SELECT DISTINCT ON (d.subscription_id)
      d.subscription_id, d.seq, d.event_id, d.body, d.attempts, d.next_at, s.url, s.secret
    FROM deliveries d
    JOIN subscriptions s ON s.id = d.subscription_id
    WHERE d.state = 'pending'
    ORDER BY d.subscription_id, d.seq;
    DispatcherCustomer endpoint0 sPOST evt_2_1, try 1, signed✕ 500: wait 59 s59 sPOST evt_2_1, try 2, signed✕ 500: wait 106 s165 sPOST evt_2_1, try 3, signed✓ 200, applied165 sPOST evt_2_2, try 1, signed✓ 200, applied
    endpointattemptsreceiver saw
    healthy3evt_1_1: applied; evt_1_2: applied; evt_1_3: applied
    fails twice4evt_2_1: applied; evt_2_2: applied
    down for 5 requests7evt_3_2: applied; evt_3_1: applied, older than the last event
    replies late once2evt_4_1: applied; evt_4_1: duplicate, ignored
    Signed with a guessed secret1Refused, 401
    A captured request, sent again later1Refused, 401

    Simulated clock in seconds, real HTTP and a real outbox table. On the endpoint that was down, the first event went dead at 614 s. The redrive at 1214 s delivered it after the second event, so the body carries a sequence number.

    I write each event to an outbox in the same transaction, sign it and retry with growing waits. After 5 attempts it goes dead, and the next event can go.

    N

    Export and deletion

    requests from a tenant
    requesthow
    Export all dataA job reads every table WHERE tenant_id = X, writes files to blob storage, and sends a link that expires.
    Delete a tenantMark it deleting, so the router refuses it. A job deletes rows in batches from every store, then the directory row.
    Delete one personDelete or anonymise their rows. Keep audit rows with a placeholder actor.
    BackupsDo not edit them. They expire at the end of their retention.

    Hash partitions by tenant do not make deletion cheaper: a partition holds many tenants. Only a cell or a database per tenant drops in one step.

    Export and deletion are background jobs per tenant. A deletion reaches every store, and backups expire within their retention.

    O

    Tenant-aware observability

    find one tenant's problem
    signaltenant detailstatus
    Logstenant_id field on every lineApproved
    Tracestenant_id on the root span, passed downApproved
    Metrics, one label per tenantOne series per tenant per metricNot approved
    Metrics, top tenants plus "other"A bounded label setApproved
    Usage per tenantRows in Postgres, summed per dayApproved
    SLO per planError budget for each tierPaid tiers

    tenant_id is on every log line and trace span. Metrics name only the largest tenants, so the number of series stays bounded.

    P

    Failure cases

    what breaks, and why tenants stay apart
    eventresultwhy it is safesaved by
    A query forgets the tenant filterIt asks for every row.The policy returns only the current tenant's rows.RLS
    The tenant is set for the sessionThe next request inherits it.Not safe. Set it per transaction; the lab leaked 3 rows otherwise.SET LOCAL
    The service connects as the table ownerRLS does not apply.Not safe. Use a role that owns nothing, or FORCE ROW LEVEL SECURITY.Roles
    A body names another tenantThe write asks for tenant 2.The tenant comes from the token, and the policy refuses the row.RLS
    A tenant floods the APIIts bucket empties.It gets 429. Other tenants keep their own buckets.Bucket
    A tenant floods the job queueA long backlog for one tenant.Workers take turns, so other tenants wait one job at most.Fair claim
    A customer endpoint is downIts events fail.Backoff, then a dead letter. Other subscriptions do not wait.Dispatcher
    A webhook reply is lostThe dispatcher retries.The receiver ignores the event id it already applied.Receiver
    A forged or replayed webhookWrong HMAC or old timestamp.The receiver returns 401.HMAC
    A bad migration or deployErrors after the change.With cells, it reaches one cell first; stop the rollout there.Cells
    Q

    Scale ladder

    start simple; climb only on a signal
    Each step adds one component1Shared tables, RLS2+ limits, fair queue3+ partitions4+ cells, router5+ own cell, largestmore load →
    Capacity against demand1001k10k100kDemand, average: 579 requests or jobs per secondDemand, average579Demand, 10× peak: 5,790 requests or jobs per secondDemand, 10× peak5,790Jobs, fair claim: 4,200 requests or jobs per secondJobs, fair claim4,200Reads, RLS: 15,900 requests or jobs per secondReads, RLS15,900Reads, 1 round trip: 61,000 requests or jobs per secondReads, 1 round trip61,000requests or jobs per second, log scale
    Routertenant → cellTenant directorysmall, cached, replicatedcell 1many small tenantsService, N copiesRedis: limits, flagsPostgres: shared tablescell 2many small tenantsService, N copiesRedis: limits, flagsPostgres: shared tablescell 3one large tenantService, N copiesRedis: limits, flagsPostgres: shared tables
    stepaddit handlesmove up when you see
    1Shared tables, tenant_id first, row-level security.About 15,900 tenant-scoped reads a second on the lab laptop. Adding a tenant is one row.One tenant's load shows in every tenant's p99.
    2Limits per tenant: a Redis bucket, quotas, turns in the job queue.Small tenants waited 6.1 ms at p99 under a burst of 300 jobs.The largest tables pass memory; vacuum and index builds take hours.
    3Hash partitions by tenant_id on the largest tables.Each partition vacuums and indexes alone. Queries name a tenant, so they read one partition.One primary nears its write limit, or one outage reaches too many tenants.
    4Cells: a full stack per group of tenants, a router and a tenant directory.Add cells as tenants grow. An outage reaches one cell's tenants.One tenant outgrows a shared cell, or its contract asks for isolation.
    5Its own cell for each of the largest tenants.The same code and tools as every other cell.Top of the ladder.

    Demand example, derived: 10,000 tenants send 50,000,000 requests a day ÷ 86,400 s ≈ 579 a second. Measured rates come from the lab, so read them as orders of magnitude.

    One Postgres with shared tables and row-level security carries thousands of tenants. I add limits per tenant first, partitions next, and cells when one database is too big a blast radius.

    R

    Drill

    predict, then reveal

    0 of 10 known

    1. Why put tenant_id first in every primary key and index?

    2. The code already filters by tenant_id. Why add row-level security?

    3. A request sets the tenant for the session, and the next request on that pooled connection forgets to set it. What happens?

    4. Why does a connection as the table owner see every row?

    5. One tenant enqueues 300 jobs at once. How do you keep the other tenants fast?

    6. When is a schema per tenant a good choice?

    7. What is a cell, and what does it give you?

    8. How do you sign a webhook, and why sign the timestamp too?

    9. Why deliver one subscription's events in order, and what breaks the order?

    10. How do you show per-tenant metrics without millions of series?

    S

    Numbers to say

    measured in the lab
    RLS
    About 15,900 reads a second against 17,600 with the same 4 round trips and no policy: about 10%.
    round trips
    One round trip read about 61,000 a second. Send the tenant setting with the query.
    queue
    Small tenants' p99 wait: 490 ms oldest first, 6.1 ms in turns.
    claims
    About 4,200 jobs a second in turns, 3,900 oldest first, 8 workers.
    schema
    12 catalog entries and 124 kB per tenant; 0.23 ms per schema to migrate.
    database
    8 MB empty per tenant; 112 ms to create.
    demand
    50 million requests a day is 579 a second.

    Postgres 16 on an 8-core laptop, 32 clients, 3 s per case, median of 3 runs. Other workloads shared the machine, so use these as orders of magnitude.