Upgrades

An upgrade replaces the image (and the document extractor's, which holds no state) and applies the migrations the new build carries. No other state moves: everything durable is in ClickHouse (and in the document bucket, for originals written with LOOM_DOCUMENT_OBJECT_STORE=s3), and the code mirrors are mounted unchanged. This page is the operator side. How a client sees unknown and removed fields is in the HTTP API reference. Which releases get fixes, and which SDK works with which server: supported versions.

How migrations run

The journal, numbering, runner directives and the rules for writing a migration are in migrations. What an operator needs:

At boot. The server applies pending files before it reports ready or starts workers (LOOM_BOOT_MIGRATE=true, the default). Every process, in any container, takes the KeeperMap lock in loom._migration_lock, so each file applies once. A server without a Keeper is refused.
Provider rebuilds. A boot with files pending first enrolls as a provider participant, then checks under the provider configuration lock (one Keeper key, held for that check only) that no provider rebuild is unfinished, and refuses to migrate if one is: Finish the provider rebuild before applying schema migrations. The files then apply under the migration lock alone. A reconfiguration started meanwhile waits until the booting process acknowledges it, so a rebuild never runs beside a migration.
A container killed while it migrates or reconfigures blocks the others only for a while. The migration lock is taken over once its holder stops renewing it (ttl_s). The provider lock (TTL 60 s, renewed every 20 s) is taken over once it expired and its holder's participant row expired too: no heartbeat for 60 s (one every 10 s) and no query tagged with its id in system.processes on any node. Containers booting together wait up to 120 s for the provider lock instead of failing. A lock whose holder has no participant row is freed with loom provider-lock clear --holder <holder> once that process is confirmed stopped; the command refuses while the holder renews, heartbeats or has a query running.
Mutations. A file is recorded only once its mutations are done. A failed mutation, or one running past LOOM_MIGRATE_MUTATION_WAIT_S (default 600), fails the file and stops the boot with the file named in the log.
Key changes. A migration that changes a sort or partition key rebuilds an empty table itself and fails the boot on one with rows. Run scripts/ops/table_rebuild.py for that table before the deploy; the strict dry run catches it.
Fresh install. A fresh install and an upgrade are one path: the server replays whatever the ledger lacks. On an empty database that is the whole journal and takes a few minutes.
Stranded ledger. A database whose ledger stopped partway through the files a baseline replaces is refused (stranded): migrate it with a build from before that baseline first.
Renumbered files. A ledger row whose file the repository no longer has is ledger-only, and --strict exits 1 on it, except when its recorded SHA-256 is a current file's: that is the same file under a new number. The runner reports it as renamed, records the new name without running it again if it is pending, and --strict passes. A file edited after it ran under its old name has another SHA-256, so it stays ledger-only: its new name runs as a pending file, and the old row is removed by hand (troubleshooting). /v1/health lists the old row under schema.ledger_only until then.

The CLI runs the same runner by hand:

loom migrate --status              # applied and pending; changes nothing
loom migrate --dry-run --strict    # pending files; exit 1 on ledger-only rows (not renames) or checksum drift
loom migrate                       # apply pending migrations
loom migrate --target N            # apply through file number N only

make migrate, make migrate-status and make migrate-dry run the same module from a checkout.

Upgrade procedure

Read the release's GitHub Release notes for operator steps (new settings, backfills a deployment must run) and the client-visible wire changes.
From the new build's checkout, against the production ClickHouse, run loom migrate --dry-run --strict. It lists what the boot will apply and stops on ledger problems (troubleshooting). If the release declares a table rebuild, run scripts/ops/table_rebuild.py for that table now. A deployment of the Postgres authority moving to a build that keeps knowledge document heads in Postgres unsets LOOM_RETENTION_DOCUMENTS_DAYS now, by hand: the newer build refuses to boot with it set, and under that authority no retention TTL reaches loom.documents, and a namespace's retention_days does not reach loom.chunks either (rolling upgrades, rule 38). A cell moving to a build that marks its sessions and carries the org triggers (rule 47) first points its LOOM_PG_DSN straight at Postgres, past any pooler (Postgres), and a later cell prepared and then backed out by unsetting LOOM_CELL is cleared first (troubleshooting): the newer build gates its processes on its state. On a later cell already prepared, the newer build's first migrating boot creates the org triggers, so an older pod still running there has every insert of a tenant org's row refused (an unmapped error to its callers) until it is replaced: roll the cell out in one step.
Deploy the new image. The first container to boot applies the migrations; the others wait for it and then find nothing pending.
Wait for /v1/health to report status: ok and an empty schema.pending before routing traffic to it. The zero-downtime pair does steps 2 to 4 for you, except unsetting the setting step 2 names: redeploy.sh runs the strict dry run in its preflight and swaps only after the new colour is ready.
Run the release's backfills, if any (maintenance).
Update client plugins when the release mentions a plugin change: hooks run on the client machine and change only through the client's plugin manager (claude plugin update loom@loom for Claude Code), never through a server deploy.
Once the last old replica has stopped, re-verify the erasures completed during the rollout: loom erasure-re-verify --completed-since <when step 3 began> --resume. An old replica's saga does not know the tables the new build added (rolling upgrades, rule 14). The command deletes only orphan row links: rows of entity_triple_rows and entity_edge_rows in each erasure's scope whose memory has no live row and no row written since the erasure's census. Without --resume it counts them and exits 1 if any are left. Data written after the erasure stays.
Turn on a feature the previous build cannot honour only once every replica runs the new build: memory_extract_agent_learnings, since the previous build does not erase loom.learnings_supporters (rolling upgrades, rule 23); LOOM_DOCUMENT_OBJECT_STORE=s3, since the previous build reads a bucket row's empty data and answers its download 410 BLOB_GONE (rule 9); and LOOM_PG_PAYLOAD_CLEAR_ENABLED, no sooner than one clearing horizon after the new build serves alone, since the previous build does not stamp a late delivery (rule 43).
A deployment with LOOM_CODE_ASK_MEMORY_NAMESPACE set, now or at any time before, that moves from a build whose cached code.ask answers do not name the memory they quoted (one before 086_code_answer_memory_scope.sql) to one that does, a roll forward after a rollback included: once the newer build serves alone, with no provider rebuild running, clear the answer cache with TRUNCATE TABLE <LOOM_CH_DATABASE>.code_answers (loom.code_answers by default). The old build's answers and tombstones name no memory, and an erasure or redact it ran during the rollout did not know the table (rolling upgrades, rule 36), so no saga can find what they hold. A rebuild publishes its copy of the table over it, so one running across the truncate brings the rows back. A truncate drops the parts at once, on every replica of a replicated table, with no mutation to wait for, and the loom user holds the grant. Each cleared question costs one synthesis on its next ask.
Moving to a build whose user erasure reaches what a consolidated summary of the user's memories fed, a roll forward after a rollback included: once the newer build serves alone, request again each user erasure without purge requested before then (the user_id rows of loom.erasure_requests, kept 90 days), including one the newer build finished: an older build's saga left the summary's history, chunks, vectors, graph rows and shared copies (rolling upgrades, rule 37). The new saga also erases what the user wrote since: request it for a user with no live memory row, or only where that is wanted. A user purge an older build completed left the same rows, and no later user erasure finds them: the user's keys are gone. One it ran past shared.tombstone also left the search rows of shared copies of the user's own memories, whichever build finished it. A shared copy an older build promoted with agent="*" names * for its source agent, and no erasure of the user it copied may find it. List them, and review each, with memory.demote for one that holds an erased user's text:
```
SELECT org, namespace, memory_key FROM loom.shared FINAL
WHERE tombstoned_at IS NULL AND arrayExists(x -> splitByChar('/', x)[3] = '*', promoted_from)
```
A deployment of the Postgres authority that moves to a build keeping knowledge document heads in Postgres (002_knowledge_documents.sql), a roll forward after a rollback included: once the newer build serves alone, run again every erasure that completed while a replica of an older build served since the newer build first served. An older build's erasure removes no head or delivery: an erased document's text stays in loom.knowledge_heads until the erasure runs again, and a delivery it left is published again by the delivery worker (rolling upgrades, rule 38). Do it before retrying a parked delivery.
A deployment of the Postgres authority that moves to a build keeping knowledge ingestion jobs (005_knowledge_jobs.sql) or the dataset catalog (007_knowledge_datasets.sql) in Postgres, a roll forward after a rollback included: once the newer build serves alone, run again every erasure that completed while a replica of an older build served since the newer build first served. An older build's erasure removes no job from loom.knowledge_jobs and no catalog row from loom.knowledge_datasets: the newer build's worker projects such a job back into loom.knowledge_ingestions and runs it, and the catalog rows stay until the erasure runs again (rolling upgrades, rules 40 and 42). Step 11's runs cover the heads, the jobs and the catalog rows alike; one run per erasure is enough. Ask again, too, each knowledge.delete and knowledge.bulk_delete an older build answered in that window: it removed the sources' datasets from loom.datasets alone.
Moving to a build that keeps document sources on any namespace, a roll forward after a rollback included: once the newer build serves alone, retire the sources an older build's namespace.delete, collection.delete or namespace erasure left active (rolling upgrades, rule 44). The sources on a deleted namespace:
```
SELECT s.org, s.namespace, s.name
FROM (SELECT org, namespace, name FROM loom.knowledge_sources FINAL WHERE active) AS s
INNER JOIN (SELECT org, namespace FROM loom.namespaces FINAL WHERE NOT active) AS n
    ON s.org = n.org AND s.namespace = n.namespace
```
and the sources registered before a namespace erasure completed that it did not retire (one an older build ran, or one an older build added while a newer build's erasure ran):
```
SELECT s.org, s.namespace, s.name, e.id
FROM (
    SELECT org, namespace, name, active, created_at, updated_at
    FROM loom.knowledge_sources FINAL
) AS s
INNER JOIN (
    SELECT id, org, completed_at,
           arrayJoin(if(scope_type = 'namespace', [scope_value],
                        if(JSONType(scope_value) = 'Array',
                           JSONExtract(scope_value, 'Array(String)'),
                           JSONExtract(scope_value, 'namespaces', 'Array(String)')))) AS namespace
    FROM loom.erasure_requests FINAL
    WHERE scope_type IN ('namespace', 'namespace_list') AND status = 'completed'
) AS e ON s.org = e.org AND s.namespace = e.namespace
WHERE s.created_at <= e.completed_at AND (s.active OR s.updated_at > e.completed_at)
```
For the first query, remove each listed source (knowledge.source.remove), or delete its namespace again with the newer build, which retires it. For the second, remove each listed source with knowledge.source.remove and delete_documents=true, which retires the documents it wrote back (it accepts a source already retired, by a later delete of its namespace for one), and delete with memory.delete the decision rows mined from them. A namespace erasure with the newer build does both, but it also erases everything written to the namespace since the first erasure; request it again only for a namespace nobody has written to since. A source registered after the erasure is not listed: the erasure never covered it. The second list is a worklist to act on once: a source removed as above stays listed, since its removal row is newer than the erasure.

A namespace erasure an older build started and the newer build finished needs nothing here: its final census finds the documents and decisions its source wrote back and reopens their steps, so the saga completes only once they are gone.
Moving to a build whose namespace erasure reaches loom.shared, a roll forward after a rollback included: once the newer build serves alone, clear the promoted rows an older build's namespace erasure left (rolling upgrades, rule 45). The query below lists each namespace that a namespace or namespace_list erasure in the ledger (kept 90 days) reached and that holds a promoted row, with the time the last one without a purge (erased_at) and with one (purged_at) ran its census. The statements below take rows promoted before that second, so a row promoted after the census stays, and so may one promoted earlier in the census's own second, which promoted_at's whole seconds cannot tell apart; the newer build's step already took such a row in an erasure it ran. Only a namespace_list purges, and an older build's namespace row may hold the flag it was asked for.
```
SELECT org, namespace,
       maxIf(counted_at, NOT purged) AS erased_at,
       maxIf(counted_at, purged) AS purged_at
FROM (
    -- The census's time: a row promoted after it was never in the erasure's reach.
    SELECT org, ifNull(verified_at, completed_at) AS counted_at,
           scope_type = 'namespace_list' AND purge AS purged,
           arrayJoin(if(scope_type = 'namespace', [scope_value],
                        if(JSONType(scope_value) = 'Array',
                           JSONExtract(scope_value, 'Array(String)'),
                           JSONExtract(scope_value, 'namespaces', 'Array(String)')))) AS namespace
    FROM loom.erasure_requests FINAL
    WHERE scope_type IN ('namespace', 'namespace_list') AND status = 'completed'
)
-- Only a namespace that holds a promoted row has anything to clear.
WHERE (org, namespace) IN (SELECT org, namespace FROM loom.shared)
GROUP BY org, namespace
```
For each, run what the newer build's step runs, over every version promoted before the erasure's census, a version a later promotion of the same key hides from FINAL included. Where purged_at is set, delete them:
```
ALTER TABLE loom.shared DELETE
WHERE org = '<org>' AND namespace = '<namespace>' AND promoted_at < '<purged_at>'
SETTINGS mutations_sync = 2
```
and where erased_at is set, tombstone the live ones left:
```
ALTER TABLE loom.shared UPDATE tombstoned_at = toDateTime('<the time you run it>')
WHERE org = '<org>' AND namespace = '<namespace>' AND tombstoned_at IS NULL
  AND promoted_at < '<erased_at>'
SETTINGS mutations_sync = 2
```
The setting waits for each statement on every replica. The time is written as a literal, as the step binds it, because a replicated table refuses a mutation that calls a non-deterministic function such as now(). A namespace erasure requested before the ledger's 90 days is not listed, and the promoted rows it left cannot be told apart from a live namespace's; they stay as the older build left them.

While a migration applies, the old build keeps serving: every migration is additive. The rules that let the two builds serve side by side are in rolling upgrades.

Rollback

Redeploy the previous image; with the pair, ./redeploy.sh --rollback. Before rolling back to a build without assistant-turn learnings, turn memory_extract_agent_learnings off and delete the rows of loom.learnings_supporters and the assistant_turn rows of loom.learnings_queue (extraction). The previous build serves a promoted bot rule to every agent in the namespace, not only its bot. To prevent that, delete the documents tagged source-kind:assistant_turn with knowledge.delete before rolling back; otherwise accept the wider serving. A namespace erasure the previous build runs once namespace guides exist leaves that namespace's guide; run the erasure again once the newer build serves (rolling upgrades, rule 24). A namespace erasure the previous build runs leaves the namespace's promoted rows live; step 14 of the upgrade procedure lists them once the newer build serves again (rule 45). After a rollback to a build without bot token maximums, every bot's tokens last an hour, and a bot that build saves keeps 3600 as its maximum; reapply a shorter or longer one once the newer build serves (rolling upgrades, rule 26). A user erasure the previous build runs leaves what a summary of the user's memories fed; request again each one run without purge once the newer build serves, as in step 10; a purge it completed is not found again (rule 37). Do not roll back past the build that adds the document bucket while any catalog row names an object in it: an older build answers those downloads 410 BLOB_GONE, its erasure deletes the rows and cannot reach their objects, and its sweep never walks the bucket (where originals are stored). After a rollback to a build before 086_code_answer_memory_scope.sql with a memory namespace bound to code.ask, clear the answer cache as in step 9 twice: once the older build serves alone, and again once the newer build serves again. The migration is already in the ledger, so the roll forward applies nothing that would prompt the second. Turn LOOM_PG_PAYLOAD_CLEAR_ENABLED off before rolling back, and on again no sooner than one clearing horizon after the newer build serves alone again; once a clearing pass has run, run loom reconcile only with a build that carries 010_revision_delivered_at.sql, start the deployment on that build after it, and roll back only once the newer build owes no claimable delivery for a cleared revision, which this returns as 0 (rule 43):

SELECT count(*) FROM loom.memory_deliveries d
JOIN loom.memory_revisions r USING (org, namespace, seq)
WHERE r.row IS NULL AND NOT d.settled AND d.lease_until <> 'infinity';

Until it does, let the newer build deliver them. The previous build never claims a parked delivery, so a parked one may stay. While the previous build serves, a retry leaves out the cleared revisions (Postgres operations), and no loom reconcile runs: it would release reconcile's holds to the previous build. Before rolling a prepared cell (one with a loom.cell_first row) back to a build without the cell org triggers, stop every pod of the newer build, then run only the trigger drop of A process with LOOM_CELL unset is gated, not its state transaction (the cell stays prepared), and run it again if a newer pod boots with migrations on before the rollback ends: the older build's sessions send no tenancy marker, so once the newer build has created the triggers each of its writes for an org the cell holds no tenancy row for is refused (on a later cell before attach, every tenant org), and its other writes pay the triggers' state read. There is no migration to undo: the previous build reads and writes the newer schema. Its version writes to loom.memory and the other ReplacingMergeTree memory tables keep the columns it does not know (rolling upgrades).

There are no down-migrations. To return to an earlier schema, restore a backup taken before the upgrade (backup and recovery).

Downgrading an installer install

Each installer bundle names the ClickHouse release it was built with, so installing an earlier Loom version also runs an earlier ClickHouse on a volume the newer one wrote, which it may not read. To roll back Loom alone, keep the newer ClickHouse: before running the earlier installer (it starts the stack at once), pin the image ClickHouse runs now in the install directory's .env. This prints the line to put there, replacing any earlier LOOM_CLICKHOUSE_IMAGE line:

dir=${LOOM_INSTALL_DIR:-$HOME/.loom}
echo "LOOM_CLICKHOUSE_IMAGE=$(docker inspect -f '{{.Config.Image}}' \
  "$(docker compose --project-directory "$dir" ps -q clickhouse)")"

Then run the earlier release's installer naming its version, as an older installer over a newer install refuses otherwise:

curl -fsSL https://github.com/ClickHouse/loom/releases/download/vX.Y.Z/install.sh -o loom-install.sh && LOOM_VERSION=X.Y.Z sh loom-install.sh && rm loom-install.sh

Run it from your home directory, or another directory only you can write to.

The installer keeps .env, so the pin holds ClickHouse through later upgrades too: delete the LOOM_CLICKHOUSE_IMAGE line before installing a newer Loom version, and do not leave LOOM_CLICKHOUSE_IMAGE exported in the shell, since the installer writes a value it is given back into .env. To go back on both, restore a backup taken before the upgrade.

What stays the same across an upgrade

Across an upgrade	What holds	Enforced by
The previous build against the new schema	keeps reading and writing every table it knew, so a rollback needs no schema step	`tests/unit/test_migrations_additive.py`
An applied migration file	never edited or deleted; a correction is a new file	the ledger checksum; `--dry-run --strict`
A client sending a field the server dropped	served; the field is named in `ignored_args`	`server/transport/arg_check.py`
The previous and new build serving together	each reads what the other wrote, new columns included, except a feature switched on only after the rollout (step 8)	`make test-mixed-version`

An upgrade can change or remove operations and fields. Check what a deployment serves with the capability catalogue (HTTP API), and list what changed between two builds with scripts/codegen/wire_changelog.py <previous> <new> (code generation).

Deployment: the pair that makes an upgrade a swap
Migrations: the rules a new migration follows
Rolling upgrades: the rules for builds N-1 and N
Troubleshooting