# ClickHouse Loom > Loom is a self-hosted MCP server that gives AI agents shared long-term memory, document search with citations and a code index. It keeps its data in ClickHouse, with Postgres as an option for live memory and knowledge state, and serves it over MCP, HTTP and the Python and TypeScript SDKs. Start with [the docs map](https://clickhouse.github.io/loom/index.md). To install Loom, a coding agent reads [agent setup](https://clickhouse.github.io/loom/getting-started/agent-setup.md) and follows [agent-setup.txt](https://github.com/ClickHouse/loom/releases/latest/download/agent-setup.txt), which each release attaches with its pins filled in; for Cline or another MCP client, it follows [llms-install.md](https://clickhouse.github.io/loom/llms-install.md). To connect to a running server: [connect a coding agent](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md). To decide whether Loom fits: [the overview](https://clickhouse.github.io/loom/getting-started/overview.md); for one need, [use cases](https://clickhouse.github.io/loom/use-cases/index.md). Every operation's fields are in [operations](https://clickhouse.github.io/loom/reference/operations.md) and the pages it links. This file holds the Getting started, Use cases and Concepts pages, the FAQ and the glossary in full; https://clickhouse.github.io/loom/llms.txt lists the other pages. --- # Overview > Source: https://clickhouse.github.io/loom/getting-started/overview/ (Markdown: https://clickhouse.github.io/loom/getting-started/overview.md) Loom is a server that stores memory, documents, conversations, an entity graph and a code index for AI agents, as rows in ClickHouse, with [Postgres](https://clickhouse.github.io/loom/operations/postgres.md) as an option for live memory and knowledge state. Agents and people reach it over MCP, HTTP, the Python and TypeScript SDKs, and the dashboard. It is Apache-2.0. This page helps you decide whether it fits. ## What it does - **Memory that lasts.** Facts an agent saves are there in later sessions and in other tools, one claim per key, with history, provenance and validity. It can be one person's memory or shared by many agents and people. - **Write-time checks.** A near-duplicate merges, a contradiction is flagged, a write against a pinned fact is refused. - **Search that abstains.** `memory.search` returns `abstain=true` with a reason code when nothing clears a gate calibrated for your embedder. - **Audit and erasure.** Every write that changes memory leaves a content-hashed audit row. `erasure.request` ends with a residue census over every table the read paths touch. - **A code index with its history.** Search, symbols, callers, impact, PR history with review comments and bisect candidates for your repositories, each answer naming the commit and index tier behind it. With a tracker source connected, the issue tracker too: duplicate checks, issue↔PR links and flaky tests. Memory and the code index are separate planes on one server. Use either alone or both. Memory needs a server and a writer token. The code index also needs a repository added (`LOOM_CODE_REPOS_JSON`, or **Repositories → Add repository** in the dashboard where the [repository-job sandbox](https://clickhouse.github.io/loom/operations/code-index-operations.md#repository-job-sandbox) runs) and `LOOM_CODE_INGEST_ENABLED=true`. The [Claude Code and Codex plugins](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md) use both. ## Where namespaces fit - An **org** is the tenant. - A **namespace** is the unit of sharing and access inside it. A deployment usually has one per team or project, one per bot that must stay isolated, one `personal-` per person (created on first use, and the default for that person's MCP clients), and one `code-[-]` per indexed branch. - Every member of a namespace reads every row in it. - `agent` records who wrote a row; it is not a boundary. - `user_id` / `read_scope=own_user` separates end users inside one bot's namespace. - Access is a role (viewer/writer/admin) on a namespace, carried by a token or added by a grant. [Identities and namespaces](https://clickhouse.github.io/loom/concepts/identities-and-namespaces.md) has the full model. ## When Loom fits - You use several AI tools and want them to share what you told any of them, in a personal namespace. - Several tools and people (Claude Code, Codex, Cursor, a support bot, a person at the dashboard) need the same memory. - You want SQL over what your agents know: facts, traces, costs and audit rows are ClickHouse tables. - You already run ClickHouse, or can. It is Loom's one required store; [Postgres](https://clickhouse.github.io/loom/operations/postgres.md) is an option for live memory and knowledge state. - An abstain or a flagged conflict is better than a confident wrong answer. - You need history, provenance, audit or verifiable erasure. - Coding agents need an index of your repositories. ## When it does not fit | Situation | Why not | Consider instead | | ------------------------------------------------------------ | ----------------------------------------------------------- | ----------------------------------------------- | | One agent, one process, short-lived memory | A server and a database are overhead when nothing is shared | An in-process memory library | | Sub-millisecond key lookups on a hot path | Every read is a database query | A cache in front of your own store | | Whole-graph analytics or long paths | Traversal is one bounded query per hop, at most four hops | A graph store, fed by `loom entity-export` | | You need an agent framework | Loom does not run the agent loop, prompts or model | An agent framework, with Loom as its memory | | A compliance guarantee out of the box | Erasure and audit are engineering primitives | Your compliance process, using those primitives | | Production on single-node ClickHouse OSS with no backup plan | No replication or point-in-time recovery | ClickHouse Cloud | | Frozen APIs | Operations and fields change between releases | Pin a release; check `ops.capabilities` | Validity is valid time only (`valid_at`, `valid_to`); what Loom recorded and when lives in the version history, not a second time axis. Some parts need an operator to turn them on: the code index, tracker sources, OAuth, telemetry export, and erasure in Langfuse, ClickStack. ## What each property costs | Property | Mechanism | Cost | | --------------------------------- | ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | One engine (ClickHouse authority) | Vector, lexical, graph, filters, audit and erasure are SQL over the same rows | Every read is a query. Readers pay `FINAL` on versioned rows. Derived rows (entities, summaries) land later, through workers. | | Abstain instead of a weak match | Gates derived at boot from the embedder's own cosine distribution (`LOOM_ADAPTIVE_THRESHOLDS`) | Some weak but correct matches are refused; set `min_similarity` or `floor_mode` per request. | | Facts that change | Merge, duplicate, conflict and pinned verdicts on each write; append-only `loom.memory_versions` | Each write compares against neighbours. A paraphrase can wait in the review queue until someone resolves it. | | Explainable ranking | `memory.search` with `explain=true` itemises each hit's score; hits carry `source_class` and `verified` | Above `top_k` 15, where `explain` is off by default, `explain=true` makes each hit larger. | | Sharing with access control | The identity tuple on every row; tokens and grants carry namespace roles | You mint and rotate tokens and grant roles. A cross-principal leakage suite gates every read op in CI. | | Erasure you can check | A resumable saga ending in a residue census; external backends fail closed | An erasure is a multi-step job. A backend that cannot confirm deletion leaves the saga `failed` for manual escalation. | | Code answers with freshness | Parser and compiler (SCIP) tiers; answers carry `as_of.commit` and a stale flag | A large repository takes hours to index. The compiler tier needs the toolchain. | Quality claims are yours to measure. The repository ships the benchmark harnesses and the raw results behind the published numbers ([results](https://clickhouse.github.io/loom/benchmarks/results.md)). Run the gates on your deployment with real providers before relying on a default ([running](https://clickhouse.github.io/loom/benchmarks/running.md)). ## Decision checklist Two or more answers in the right-hand column mean Loom is probably the wrong tool. | Question | Points to Loom | Points away | | ------------------------------------------------------------------------- | -------------- | ----------------------- | | Do two or more tools or agents need to share memory? | Yes | No, one process | | Can you run ClickHouse and operate a service? | Yes | No, you want it managed | | Must the data stay in your own infrastructure? | Yes | No preference | | Is an abstain or a flagged conflict better than a confident wrong answer? | Yes | Always return something | | Do you need history, provenance, audit or verifiable erasure? | Yes | No | | Is your hot path a key lookup under a millisecond? | No | Yes | | Do you need whole-graph analytics rather than a few hops? | No | Yes | | Can you absorb API changes between releases, or pin one? | Yes | No | | Will you pay for a real embedder and LLM, and bench on your data? | Yes | No | ## Prerequisites | You need | For | | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ClickHouse (OSS to evaluate, Cloud in production) | Everything. See [ClickHouse requirements](https://clickhouse.github.io/loom/operations/deployment.md#clickhouse-requirements). | | Postgres 16+ (optional) | Live memory and knowledge state under the [Postgres memory authority](https://clickhouse.github.io/loom/operations/postgres.md), chosen at a deployment's first boot; needs a synchronous standby unless `LOOM_PG_DURABILITY=local` ([requirements](https://clickhouse.github.io/loom/operations/postgres.md#requirements)). | | Docker with Compose 2.20 or later | The released local stack (one-line installer; on Windows, [from WSL 2](https://clickhouse.github.io/loom/getting-started/quickstart.md#fast-path-released-image-no-clone)) and server image. | | Python 3.12+ | Contributor-only `loom up`, server development and operator CLI work. | | An embedding provider | Real retrieval: OpenAI, Gemini, or the local CPU embedder the released stack uses. | | An LLM provider | Extraction, `memory.ask` synthesis, `code.ask`: OpenAI or Anthropic. | | Tokens and namespaces | Access. See [tokens and grants](https://clickhouse.github.io/loom/guides/tokens-and-grants.md). | | For the code index | Git access to the repositories, `LOOM_CODE_GITHUB_TOKEN` for PR history, and time for the first index. | ## Next [Evaluate Loom](https://clickhouse.github.io/loom/getting-started/evaluate.md) separates a product smoke test from a real quality evaluation. [Quickstart](https://clickhouse.github.io/loom/getting-started/quickstart.md) runs the local stack. [Deployment](https://clickhouse.github.io/loom/operations/deployment.md) runs it for a team. The [documentation map](https://clickhouse.github.io/loom/index.md) lists every page in reading order. --- # Use cases > Source: https://clickhouse.github.io/loom/use-cases/ (Markdown: https://clickhouse.github.io/loom/use-cases/index.md) Each page answers one question people and agents ask when they look for a tool: what Loom does for that need and where to start. | Need | Page | | -------------------------------------------------------------------------------- | ---------------------------------------------------------- | | Long-term and shared memory for AI agents (an MCP memory server) | [Agent memory](https://clickhouse.github.io/loom/use-cases/agent-memory.md) | | Knowledge and document search for agents (RAG) | [Document search](https://clickhouse.github.io/loom/use-cases/document-search.md) | | A code index and code intelligence for coding agents | [Code index](https://clickhouse.github.io/loom/use-cases/code-intelligence.md) | | What the code index gives per language (C++, Python, TypeScript, Go, Rust, Java) | [Code index by language](https://clickhouse.github.io/loom/use-cases/code-intelligence-by-language.md) | | Memory linked to the code it is about, flagged when the code changes | [Memory linked to code](https://clickhouse.github.io/loom/use-cases/memory-linked-to-code.md) | | Lower model cost for coding agents | [Coding-agent cost](https://clickhouse.github.io/loom/use-cases/coding-agent-cost.md) | | Shared context between agents in a multi-agent workflow | [Multi-agent](https://clickhouse.github.io/loom/use-cases/multi-agent.md) | | Memory, recall, audit and erasure for autonomous agents | [Autonomous agents](https://clickhouse.github.io/loom/use-cases/autonomous-agents.md) | | Agent runs, traces, tracker and PR history | [Runs and PR history](https://clickhouse.github.io/loom/use-cases/runs-and-pr-history.md) | | Self-hosted agent memory on ClickHouse, with Docker or Kubernetes | [Self-hosted on ClickHouse](https://clickhouse.github.io/loom/use-cases/self-hosted-clickhouse.md) | [When Loom does not fit](https://clickhouse.github.io/loom/getting-started/overview.md#when-it-does-not-fit) lists the needs another tool serves better. --- # Long-term and shared memory for AI agents > Source: https://clickhouse.github.io/loom/use-cases/agent-memory/ (Markdown: https://clickhouse.github.io/loom/use-cases/agent-memory.md) Loom is a self-hosted MCP server and HTTP API that gives AI agents long-term memory they share: a fact one agent saves in a session can be found by the next session, by another agent, or by a person at the dashboard. This page shows how that works and where to start. ## What you get - **One claim per key, in a namespace.** `memory.set` saves a fact; `memory.get` reads it back by key; `memory.search` and `loom_search` find it by meaning. A namespace is the unit of sharing: by default, every member reads every row in it. [Memory](https://clickhouse.github.io/loom/concepts/memory.md) - **Memory that can be corrected.** A change makes a new version with its history. The same fact after canonicalisation merges into the stored row, the same fact in other words is stored as a duplicate, a contradiction is flagged for review, and a write against a pinned fact is refused. [Conflicts and refusal](https://clickhouse.github.io/loom/features/memory/conflicts-and-refusal.md) · [History and provenance](https://clickhouse.github.io/loom/features/memory/history-and-provenance.md) - **Search that abstains.** When nothing clears the similarity floor, `memory.search` answers `abstain=true` with a reason code and labels each weak hit `below_threshold`; `loom_search` drops them. [Search and ask](https://clickhouse.github.io/loom/features/memory/search-and-ask.md) - **Long-term memory for your coding agent, shared across agents.** The Claude Code and Codex plugins load relevant context at session start and capture the session as it runs. Cursor, GitHub Copilot, Gemini CLI, Cline, Kilo Code, OpenCode, goose, Zed and other MCP clients connect to the same server, and an agent on a namespace finds the facts other agents saved there. [Connect a coding agent](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md) · [MCP clients](https://clickhouse.github.io/loom/features/integrations/mcp-clients.md) - **Memories linked to the code they are about.** A coding agent's lessons follow the definitions they name through renames and edits, and appear when an agent reads that code. [Memory linked to code](https://clickhouse.github.io/loom/use-cases/memory-linked-to-code.md) - **Per-user memory for your own agent.** The Python and TypeScript SDKs write and read memory per end user. [Your first agent](https://clickhouse.github.io/loom/getting-started/first-agent.md) ## Measured On 100 paired LongMemEval-S questions with the same `gpt-5` reader, answers from Loom's retrieved memories scored 94% against 90% from the whole history, with 91% fewer reader prompt tokens (excluding the one-time cost of building the memory); the accuracy difference is not statistically established. [With and without Loom](https://clickhouse.github.io/loom/benchmarks/results.md#with-and-without-loom) ## Start [Quickstart](https://clickhouse.github.io/loom/getting-started/quickstart.md) boots the local stack and saves a first memory over HTTP, Python and MCP. [When Loom does not fit](https://clickhouse.github.io/loom/getting-started/overview.md#when-it-does-not-fit) covers the cases an in-process memory library serves better. Some ops on this page are marked experimental on the [status page](https://clickhouse.github.io/loom/reference/status.md). Use a [Microsoft Agent Framework context provider](https://clickhouse.github.io/loom/features/integrations/frameworks.md#microsoft-agent-framework) to recall user facts before a run and record the conversation after it. Store the [Claude API memory tool](https://clickhouse.github.io/loom/features/integrations/frameworks.md#claude-api-memory-tool) file tree in Loom so agents in a namespace can share its contents. --- # Memory, recall, audit and erasure for autonomous agents > Source: https://clickhouse.github.io/loom/use-cases/autonomous-agents/ (Markdown: https://clickhouse.github.io/loom/use-cases/autonomous-agents.md) Loom gives autonomous agents memory they recall later, an audit record of who wrote what, and erasure of a scope everywhere Loom keeps it. These are server operations, stored in ClickHouse (or Postgres for live memory and knowledge state). ## What you get - **Memory and recall.** Versioned facts with provenance, search that abstains when nothing is close enough, and a ranking boost for memories that successful runs relied on. [Memory](https://clickhouse.github.io/loom/concepts/memory.md) · [Runs](https://clickhouse.github.io/loom/features/runs.md) - **Procedures.** Reusable how-tos the agent learns from doing: `procedure.add` stores one as a graph of steps, `procedure.match` finds the ones that fit a task, and the runs the agent reports teach Loom the proven path and the steps that fail. A revised graph is kept only if its runs do not do worse. [Procedures and learnings](https://clickhouse.github.io/loom/features/memory/procedures-and-learnings.md) - **Refusal.** A pinned fact refuses contradicting writes, and `blacklist.add` registers a pattern no write may store. [Conflicts and refusal](https://clickhouse.github.io/loom/features/memory/conflicts-and-refusal.md) - **Audit.** Each operation that changes memory leaves an audit row with a SHA-256 of its content, and a worker seals hourly windows into a hash chain, so an edit or deletion is detectable. The transport's audit writes are fail-open (a failed insert loses the row); under the Postgres authority a memory revision's audit row is stored with the revision. [Audit log](https://clickhouse.github.io/loom/features/observability/audit.md#limits-and-failure-modes) - **Erasure you can check.** `erasure.request` erases a scope from every table that holds memory content (tombstones by default; `purge=true` deletes the rows of a `user_id` or `namespace_list` scope) and ends with a residue census and a receipt. [Retention and erasure](https://clickhouse.github.io/loom/features/memory/retention-and-erasure.md) - **Rollback.** `namespace.rollback` returns every memory in a namespace to its version at a point in time. [History and provenance](https://clickhouse.github.io/loom/features/memory/history-and-provenance.md) - **Headless operation.** A bot gets its own identity (it signs itself up where self-signup is open), checks its connection before its first run, and can be diagnosed when it goes quiet. [Headless bots](https://clickhouse.github.io/loom/guides/headless-bots.md) Audit and erasure are engineering primitives, not a compliance certification. [Security](https://clickhouse.github.io/loom/architecture/security.md) Some ops on this page are marked experimental on the [status page](https://clickhouse.github.io/loom/reference/status.md). --- # Code intelligence for C++, Python, TypeScript, Go, Rust, Java and infrastructure code > Source: https://clickhouse.github.io/loom/use-cases/code-intelligence-by-language/ (Markdown: https://clickhouse.github.io/loom/use-cases/code-intelligence-by-language.md) Loom's code index answers a coding agent's structure questions in the languages a repository holds: where a symbol is defined, who calls it, and what a change touches. Needs a repository added and `LOOM_CODE_INGEST_ENABLED=true`. Every language gets the fast tier, parsed with tree-sitter grammars or hand-written scanners; C/C++, Python, TypeScript and Go can add a compiler tier that resolves names the way the compiler does. The full list of extensions and edges is the [language support table](https://clickhouse.github.io/loom/features/code-index/overview.md#language-support), and the compiler tiers' setup is in [code index operations](https://clickhouse.github.io/loom/operations/code-index-operations.md#compiler-tier). An answer names the tier behind it (`tier: "scip"` for the compiler tier, `"treesitter"` for the fast tier), and `code.coverage` reports what a namespace holds per language and tier. Read it before trusting `code.callers` or `code.impact` answers in a language you have not used Loom with. ## C and C++ Symbols, calls, includes, inheritance and overrides come from the fast tier; the compiler tier (`scip-clang`) resolves overloads, templates and namespaces from the project's build, so `code.symbol` with `uses_brief: true` on a qualified name such as `DB::IMergeTreeDataPart::getBytesOnDisk` returns the one definition, its call-site count and its top call sites with their enclosing functions, each marked `tier: "scip"`. The compiler tier needs the project's CMake build setup. ## Python Calls, imports, inheritance and overrides come from the fast tier, notebooks' code cells included; the compiler tier (`scip-python`) resolves names across modules. In Loom's own repository, `code.symbol` on `writer_actor` returns its definition in `server/memory/app/provenance.py` with its signature and docstring at the compiler tier, and `uses_brief: true` adds its call sites. ## TypeScript and JavaScript Calls (including `new`), imports, inheritance and overrides come from the fast tier, as do the script blocks of Vue, Svelte and Astro components; an Angular `.component.html` template is indexed as one unit with its component. The compiler tier (`scip-typescript`) resolves names through the project's `tsconfig.json`, `jsconfig.json` or `package.json`. ## Go Calls, interface embedding and structural implementation come from the fast tier, and SQL in Go strings becomes table references; the compiler tier (`scip-go`) resolves names across packages, monorepos included. ## Rust and Java Rust gets calls, `use` imports and trait implementations; Java gets calls, with imports binding names, and SQL in Java strings. Both have the fast tier only: a call bound in the same file has confidence 1.0, through the enclosing class 0.8 and by a repository-wide name match 0.6, a call it cannot bind is declined rather than guessed, and `code.impact` adds mapped tests and co-change partners at file level. ## SQL and migrations `.sql` files are indexed for search and grep by default. With `LOOM_CODE_SQL_STATEMENT_RECORDS` on, each statement becomes a record with its table references, so an agent asks which migrations and queries read or write a table. Turning it on for a namespace already indexed needs a `loom code backfill --paths-file` of the SQL files ([switches](https://clickhouse.github.io/loom/features/code-index/overview.md#language-support)). ## Infrastructure as code Terraform and HCL, Kubernetes manifests, Helm charts and Go templates, Jsonnet, CMake, Protocol Buffers and shell scripts are indexed with their references, includes and calls, so an agent follows a Helm `include` to its `define`, or a Terraform reference to what it names. Dockerfile, Makefile, Jenkinsfile and CI YAML jobs become symbols with `LOOM_CODE_HOST_ISLANDS` on, after a backfill of those files when the namespace is already indexed. ## Start 1. [Index a repository](https://clickhouse.github.io/loom/features/code-index/repositories.md). Which compiler tiers run for it is in [code index operations](https://clickhouse.github.io/loom/operations/code-index-operations.md#compiler-tier). 2. [Connect a coding agent](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md) on the `coding` toolset. 3. Ask a structure question; check `code.coverage` for the languages you rely on. ## Related - [A code index for coding agents](https://clickhouse.github.io/loom/use-cases/code-intelligence.md) - [Code operations](https://clickhouse.github.io/loom/features/code-index/operations.md) - [Adding language support](https://clickhouse.github.io/loom/development/adding-language-support.md) --- # A code index for coding agents > Source: https://clickhouse.github.io/loom/use-cases/code-intelligence/ (Markdown: https://clickhouse.github.io/loom/use-cases/code-intelligence.md) Loom gives coding agents a shared index of your git repositories, served over MCP, so an agent asks a structure question and gets the lines that answer it instead of reading whole files. Needs a repository added and `LOOM_CODE_INGEST_ENABLED=true`. How the index is built and kept fresh is in [the code index overview](https://clickhouse.github.io/loom/features/code-index/overview.md); what it gives in each language is in [code index by language](https://clickhouse.github.io/loom/use-cases/code-intelligence-by-language.md). ## What you get - **Definitions, callers and impact.** `code.symbol`, `code.callers` and `code.impact` answer with the file and line, and each answer names the commit and index tier behind it. [Code operations](https://clickhouse.github.io/loom/features/code-index/operations.md) - **Search by meaning and by exact string.** `code.search` finds where a mechanism lives; `code.grep` finds an exact message or identifier. - **History and pull requests.** Merged-PR history with review comments, a pre-PR duplicate check and bisect candidates for a regression. These need PR history turned on (`LOOM_CODE_PR_HISTORY_ENABLED`); review comments and open PRs also need a GitHub token. [History, PRs and releases](https://clickhouse.github.io/loom/features/code-index/history-and-prs.md) says what each needs. - **Lessons on the code.** `code.symbol` returns the memories linked to a definition, and `code.search`, `code.callers`, `code.callees`, `code.impact`, `code.outline` and a task's `code.context_pack` attach the lessons for the files they return, each with whether its code moved, changed or is gone. [Memory linked to code](https://clickhouse.github.io/loom/use-cases/memory-linked-to-code.md) - **Freshness you can see.** The index trails the branch by merged commits, and a stale answer says so. [Freshness](https://clickhouse.github.io/loom/features/code-index/freshness.md) - **Plugins and MCP clients.** The Claude Code plugin connects on the `coding` toolset and the Codex plugin on `core,coding,tracker,tasks`; both list the code tools. Cursor, GitHub Copilot, Gemini CLI, Cline, Kilo Code, OpenCode and other MCP clients list them with `?toolset=coding` on their URL (`loom install` adds it for Cursor; `--toolset coding` adds it for the others it writes). [Connect a coding agent](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md) · [MCP clients](https://clickhouse.github.io/loom/features/integrations/mcp-clients.md) ## Measured On the ClickHouse repository, Claude Code was asked to end each answer with up to three files. With Loom it named the right file for 96.4% of 36 conceptual questions (10 paired runs), against 91.4% without Loom, and the runs with Loom cost 21% less (the agent's model bill; Loom's own model calls excluded). On 200 SWE-bench Verified issues, `code.search` ranked a changed file in its first 10 results for 82% (file localization, not issue resolution), against 57% for BM25 over every indexed file and 79% for BM25 over `.py` files only; the `.py`-only gap is within noise. [With and without Loom](https://clickhouse.github.io/loom/benchmarks/results.md#with-and-without-loom) · [SWE-bench Verified file localization](https://clickhouse.github.io/loom/benchmarks/results.md#swe-bench-verified-file-localization) ## Start [Adding repositories](https://clickhouse.github.io/loom/features/code-index/repositories.md) turns the index on for a repository; [the code index overview](https://clickhouse.github.io/loom/features/code-index/overview.md) explains the tiers. Some ops on this page are marked experimental on the [status page](https://clickhouse.github.io/loom/reference/status.md). --- # Lower model cost for coding agents > Source: https://clickhouse.github.io/loom/use-cases/coding-agent-cost/ (Markdown: https://clickhouse.github.io/loom/use-cases/coding-agent-cost.md) With the Loom plugin and code index, Claude Code answered questions about the ClickHouse codebase at 6% to 21% lower model cost than without it on all three measured question sets. The savings were 20% on 24 answer-graded questions (answer quality 0.84 against 0.78), 21% on 36 conceptual questions and 6% on 24 multi-hop questions, where tracing a bug to its fix on a release branch cost the same both ways. ## What was measured Both arms ran Claude Code (`claude-sonnet-5`) on the same checkout: one plain session, one with the Loom plugin and the `coding` toolset. Cost is the whole session's model bill and excludes Loom's own server-side model calls, hosting and indexing. The comparison measures the plugin as a whole; it does not separate the code index from memory. Results are specific to the tested workloads. With Loom a question takes longer on the clock: the plugin's per-prompt search and the MCP round trips add latency. How per-question cost is compared against run-to-run noise is on the results page. [With and without Loom](https://clickhouse.github.io/loom/benchmarks/results.md#with-and-without-loom) has every set, the protocol and the artifacts, and [evaluate Loom](https://clickhouse.github.io/loom/getting-started/evaluate.md) shows how to measure it on your own repository. For memory read by a model instead of a whole chat history, see [agent memory](https://clickhouse.github.io/loom/use-cases/agent-memory.md#measured). ## How the plugin is designed to save - **Index answers instead of file reads.** `code.symbol`, `code.callers` and `code.search` return the lines that answer a structure question. [Code index](https://clickhouse.github.io/loom/use-cases/code-intelligence.md) - **Saved context instead of rediscovery.** Decisions and findings a session saves are loaded at the next session's start. [Shared memory](https://clickhouse.github.io/loom/use-cases/agent-memory.md) Loom also records what it spends on embedding and LLM calls per operation. [Cost and usage](https://clickhouse.github.io/loom/features/observability/cost-and-usage.md) --- # Document search and RAG for agents > Source: https://clickhouse.github.io/loom/use-cases/document-search/ (Markdown: https://clickhouse.github.io/loom/use-cases/document-search.md) Loom indexes your documents (Markdown and text, and PDF, `.docx` and `.pptx` files that its extractor service converts) and lets agents search them with citations over MCP, HTTP or the SDKs. It is the retrieval half of retrieval-augmented generation (RAG): your agent or model writes the answer from the passages Loom returns. ## What you get - **Chunks that follow the document's structure.** `knowledge.add` stores the text and, optionally, the original file; Loom splits it on its headings, windows each section and embeds every chunk. [Knowledge documents](https://clickhouse.github.io/loom/features/knowledge.md) - **Citations.** A hit names its document's `source_uri` and passage, so an agent can quote and link the source. - **One search across memory, documents and conversations.** `loom_search` runs the memory, knowledge, profile and conversation searches in one call, across the caller's memory namespaces. [Unified search](https://clickhouse.github.io/loom/features/unified-search.md) - **Document sources.** A namespace's admins can register sources Loom pulls documents from; the sync worker needs `LOOM_KNOWLEDGE_SOURCE_SYNC_ENABLED`. [Document sources](https://clickhouse.github.io/loom/features/knowledge.md#document-sources) - **Documents beside the code.** Documents from a source on a repository's trunk code namespace carry its `repo:` tag, so that repository's searches read them with its code. [Document sources](https://clickhouse.github.io/loom/features/knowledge.md#document-sources) · [Memory linked to code](https://clickhouse.github.io/loom/use-cases/memory-linked-to-code.md) - **A similarity floor.** `loom_search` drops knowledge hits below a similarity floor instead of handing the model a weak passage. [`loom_search`](https://clickhouse.github.io/loom/features/unified-search.md#loom_search) - **Access control on every passage.** Documents live in namespaces with viewer, writer and admin roles. [Namespaces and sharing](https://clickhouse.github.io/loom/features/namespaces-and-sharing.md) ## Measured On a committed corpus of PDF, DOCX and PPTX files (27 held-out files), Loom's extractor, with OCR on (its default), scored 0.999 text recall and 0.991 table cell F1 in the measured run ([document parsing](https://clickhouse.github.io/loom/benchmarks/results.md#document-parsing)). Retrieval quality over documents is measured only in part; [coverage and known gaps](https://clickhouse.github.io/loom/benchmarks/coverage.md) says what is still missing. ## Start The [quickstart](https://clickhouse.github.io/loom/getting-started/quickstart.md) runs the stack with the extractor service. Add a document from the dashboard's Library page, with `knowledge.add` for text, or, for a file the extractor converts, with `knowledge.upload` then `knowledge.ingest` ([ingesting files](https://clickhouse.github.io/loom/features/knowledge.md#ingesting-files)). Some ops on this page are marked experimental on the [status page](https://clickhouse.github.io/loom/reference/status.md). --- # Memory linked to code for coding agents > Source: https://clickhouse.github.io/loom/use-cases/memory-linked-to-code/ (Markdown: https://clickhouse.github.io/loom/use-cases/memory-linked-to-code.md) Loom links coding agents' memories to the functions, classes and files they name, follows them through refactors, and flags a memory when its code changes. By default, search leaves out a memory whose code is all gone unless a person pinned, locked or verified it, and every read that serves a lesson says when its code changed or is gone. It runs over MCP, HTTP and the SDKs on the repositories a self-hosted Loom indexes (the [code index](https://clickhouse.github.io/loom/use-cases/code-intelligence.md), `LOOM_CODE_INGEST_ENABLED=true`), and the `memory_code_links` worker that keeps the links is on by default. ## What you get - **Lessons attached to the code they are about.** A memory tagged `repo:` (the Claude Code and Codex plugins tag what their hooks capture) or written in the repository's code namespace is linked to each definition (function, class or other symbol) it names in backticks, or anywhere in its text as `A::b` or a call (`a.b()`, `name()`), that resolves to exactly one place (an unqualified name needs four or more characters); to definitions in the files it lists that it names by a bare camelCase or snake_case word; and to each file it lists that holds none of them. A qualified name not on trunk yet links if it merges within `LOOM_MEMORY_CODE_LINKS_PENDING_DAYS` (14 days). [Memories linked to code](https://clickhouse.github.io/loom/concepts/memory.md#memories-linked-to-code) - **Links that follow refactors.** On every indexed commit, each link reads `fresh`, `moved`, `changed` (naming the commit that changed it), `unresolved` or `missing`. A link records the definition's tokens, so whitespace, comments and most formatter changes leave it `fresh`. [Link states](https://clickhouse.github.io/loom/concepts/memory.md#memories-linked-to-code) - **Stale memories handled the way you choose.** The namespace's [`code_change_policy`](https://clickhouse.github.io/loom/reference/operations/namespace.md#namespaceregister) decides: `retire_removed` (the default) leaves a memory whose code is all gone out of search and flags changed code; `flag` only flags; `revise` also leaves gone-code memories out, and has the model check a changed one's claim against the old and new code: one that still holds is re-anchored, and a contradicted one gets a suggested correction. A memory a person pinned, locked or verified is never left out. Under every policy, search usually ranks a memory whose code changed below the next three results instead of hiding it; a held memory and a row found by name keep their place. Loom never rewrites the memory itself. [Policies](https://clickhouse.github.io/loom/concepts/memory.md#memories-linked-to-code) - **Lessons wherever the agent reads code.** `code.symbol` returns the memories linked to a definition as `lessons`; `code.search`, `code.callers`, `code.callees`, `code.impact`, `code.outline` and a task's `code.context_pack` attach the lessons for the files they return; each lesson carries its link state. The Claude Code and Codex hooks show a file's memories (`memory.for_file`) before an agent edits it, and the Cline hooks when it reads or edits one. [`code.symbol`](https://clickhouse.github.io/loom/reference/operations/code.md#codesymbol) · [Claude Code plugin](https://clickhouse.github.io/loom/features/integrations/claude-code.md) · [Codex plugin](https://clickhouse.github.io/loom/features/integrations/codex.md) · [Cline hooks](https://github.com/ClickHouse/loom/blob/main/integrations/cline/README.md) (experimental) - **Shared with the team.** Lessons are read from the repository's code namespace and the namespaces the call names (over MCP, the caller's own by default), so a lesson saved in the repository's code namespace reaches every agent that may read that namespace (a token narrowed to its own user or session reads no lessons). [Memories linked to code](https://clickhouse.github.io/loom/concepts/memory.md#memories-linked-to-code) - **Branches read their own code.** On another branch, a lesson is checked against that branch's own definition, so a release branch that still has the old text reads `fresh` (on a host with `LOOM_GIT_HISTORY=none`, which has no git mirror, a lesson carries trunk's state). [Memories linked to code](https://clickhouse.github.io/loom/concepts/memory.md#memories-linked-to-code) - **Documents beside the code.** Documents from a source registered on the repository's code namespace (its trunk's) carry the repository's tag, so searches over the repository read them with its code (the sync worker needs `LOOM_KNOWLEDGE_SOURCE_SYNC_ENABLED`). [Document sources](https://clickhouse.github.io/loom/features/knowledge.md#document-sources) - **A review list.** [`namespace.validate`](https://clickhouse.github.io/loom/reference/operations/namespace.md#namespacevalidate) lists the memories whose code changed, was not found again or is gone, and the dashboard shows each memory's links and their state. ![A memory's detail in the dashboard: its linked code, one definition current and one changed since a commit](https://clickhouse.github.io/loom/assets/screenshots/memory-linked-code.webp) ## Start [Quickstart](https://clickhouse.github.io/loom/getting-started/quickstart.md) boots Loom, and [adding repositories](https://clickhouse.github.io/loom/features/code-index/repositories.md) turns the code index on for a repository. Connect your coding agent with its plugin or over MCP ([connect a coding agent](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md)). Save a lesson that names a definition of four or more characters in backticks or as a call (`` `name()` ``), or a qualified one such as `` `Reader.run` ``. The Claude Code and Codex plugins tag what their hooks capture with `repo:`, and the session primer names the id ([how it is derived](https://clickhouse.github.io/loom/features/integrations/claude-code.md#configuration)). A lesson the agent saves itself with `loom_memory_set` links when it carries that tag (`tags: ["repo:"]`) or is written in the repository's code namespace; any client can do either. Over HTTP (or `agent.call` in the Python SDK), a lesson saved in your own namespace is read only when the call names that namespace in `lesson_namespaces`. Each link pass links writes at least 60 seconds old and passes run every 60 seconds, so once trunk is indexed, `code.symbol` on that definition returns the lesson under `lessons` within about two minutes. Some ops on this page are marked experimental on the [status page](https://clickhouse.github.io/loom/reference/status.md). --- # Shared context for multi-agent workflows > Source: https://clickhouse.github.io/loom/use-cases/multi-agent/ (Markdown: https://clickhouse.github.io/loom/use-cases/multi-agent.md) Loom gives the agents in a multi-agent workflow one place to share context: memory every agent in a namespace reads, task inboxes for handing work from one agent to another, and a record of decisions. Your agents and orchestrator run the workflow; Loom keeps what they learn and hand over. ## What you get - **Shared memory with provenance.** Every row records which agent wrote it, and under the default `auto_reinforce` policy a contradiction between two agents is flagged for review instead of silently overwritten. [Namespaces and sharing](https://clickhouse.github.io/loom/features/namespaces-and-sharing.md) · [Namespace conflict policy](https://clickhouse.github.io/loom/features/memory/conflicts-and-refusal.md#namespace-conflict-policy) - **Task inboxes.** Each agent has a durable queue with leases, retries and review rules; a claimed task has one owner at a time. [Tasks, decisions and policies](https://clickhouse.github.io/loom/features/tasks-and-decisions.md) - **Decisions.** What was decided in a case like this, why, and whether it worked. [Decisions](https://clickhouse.github.io/loom/features/tasks-and-decisions.md#decisions) - **Working memory.** Text an agent is given every time instead of finding it by search, separate from long-term memory. [Working memory and profiles](https://clickhouse.github.io/loom/features/memory/working-memory-and-profiles.md) - **Context packs.** `context.pack` builds the bounded bundle an agent loads at the start of a task. [Unified search and context packs](https://clickhouse.github.io/loom/features/unified-search.md) - **Session briefs.** The resumable state of one line of work: goal, status, decisions, dead ends and what is done. One agent saves it with `session.brief_save` and the next loads it with `session.brief_get`. [Session briefs](https://clickhouse.github.io/loom/features/conversations-and-sessions.md#session-briefs) - **Framework adapters.** The SDKs present Loom through agent frameworks' own memory interfaces. [Agent frameworks](https://clickhouse.github.io/loom/features/integrations/frameworks.md) ## Start [Integrate Loom into an agent](https://clickhouse.github.io/loom/guides/agent-integration.md) covers the identity each agent sends and what to write; [tokens and grants](https://clickhouse.github.io/loom/guides/tokens-and-grants.md) gives each agent its own token. Some ops on this page are marked experimental on the [status page](https://clickhouse.github.io/loom/reference/status.md). --- # Agent runs, traces and PR history > Source: https://clickhouse.github.io/loom/use-cases/runs-and-pr-history/ (Markdown: https://clickhouse.github.io/loom/use-cases/runs-and-pr-history.md) Loom records the work coding agents do, as runs and searchable session conversations, and connects it to your code history. Runs carry goals and outcomes, and model costs when the Claude Code plugin reports the session's turn usage (on by default). The history covers merged pull requests, review comments, GitHub issues and, with a tracker source, incidents and CI runs. Conversation recording, PR history and the tracker are each off until turned on. An agent or a person can then ask what was tried before, what was decided and which change introduced a regression. ## What you get - **Runs.** `run.begin` and `run.complete` record a task's goal, status, the memories the agent relied on and what it wrote. [Runs](https://clickhouse.github.io/loom/features/runs.md) - **Conversations and session briefs.** Recorded sessions are searchable, and a session brief lets the next session resume. Recording is off until the namespace owner turns it on with `conversation.set_logging(enabled=true)`. [Conversations and sessions](https://clickhouse.github.io/loom/features/conversations-and-sessions.md) · [Trace and resume long agent runs](https://clickhouse.github.io/loom/guides/trace-and-resume.md) - **PR history.** History of a symbol, path or line, review briefs, a pre-PR duplicate check and bisect candidates, as on the [code index page](https://clickhouse.github.io/loom/use-cases/code-intelligence.md). - **Tracker records.** Duplicate issue checks, issue-to-PR links and flaky tests, read live from the team's own ClickHouse. Needs `LOOM_TRACKER_ORG` and a registered tracker source. [Tracker plane](https://clickhouse.github.io/loom/features/tracker/overview.md) - **Traces and cost.** Loom emits OpenTelemetry and records its own model spend per operation; Langfuse and ClickStack are optional sinks. [Telemetry](https://clickhouse.github.io/loom/features/observability/telemetry.md) · [Cost and usage](https://clickhouse.github.io/loom/features/observability/cost-and-usage.md) ## Measured On public ClickHouse issues, `tracker.duplicate_check` ranked the older issue first for 91 of 104 recorded duplicates (0.875) and called 2 of 100 issues with no recorded link a duplicate. The result is one tracker's, and its issues (stack traces, error codes, consistently linked duplicates) are easier than many trackers'. [Duplicate check on public ClickHouse issues](https://clickhouse.github.io/loom/benchmarks/results.md#duplicate-check-on-public-clickhouse-issues) ## Start [Connect a coding agent](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md) installs the plugins that capture sessions once recording is on; the [code index](https://clickhouse.github.io/loom/use-cases/code-intelligence.md) needs a repository added and `LOOM_CODE_INGEST_ENABLED=true`. Some ops on this page are marked experimental on the [status page](https://clickhouse.github.io/loom/reference/status.md). --- # Self-hosted agent memory on ClickHouse > Source: https://clickhouse.github.io/loom/use-cases/self-hosted-clickhouse/ (Markdown: https://clickhouse.github.io/loom/use-cases/self-hosted-clickhouse.md) Loom runs on your own infrastructure, with Docker or on Kubernetes with its Helm chart, and stores agent memory, documents, conversations and code indexes as tables in ClickHouse. A deployment can choose Postgres at first boot for live memory and knowledge state. Data is stored only in your stores (the hosted model providers and trace sinks you configure receive what they process), and what your agents know is queryable with SQL. ## What you get - **Docker or Kubernetes.** The one-line installer starts the local stack; the Helm chart runs N identical replicas behind one URL. [Deployment](https://clickhouse.github.io/loom/operations/deployment.md) · [Kubernetes](https://clickhouse.github.io/loom/operations/kubernetes.md) · [Running N replicas](https://clickhouse.github.io/loom/operations/replicas.md) - **ClickHouse OSS or ClickHouse Cloud.** [Choosing the ClickHouse](https://clickhouse.github.io/loom/operations/deployment.md#choosing-the-clickhouse) · [ClickHouse Cloud and Managed Postgres](https://clickhouse.github.io/loom/operations/clickhouse-cloud.md) - **One store for vectors, text and audit.** Under the default ClickHouse authority, vector, lexical and graph search, filters, audit and erasure are SQL in one database. [Storage authority](https://clickhouse.github.io/loom/architecture/storage-authority.md) - **Multi-user.** Orgs, namespaces, roles, tokens and single sign-on. [Identities and namespaces](https://clickhouse.github.io/loom/concepts/identities-and-namespaces.md) · [Single sign-on](https://clickhouse.github.io/loom/operations/sso.md) - **Local models.** The released stack embeds on CPU with a bundled local model, and the embedding provider accepts an OpenAI-compatible endpoint you run. [Models and API keys](https://clickhouse.github.io/loom/getting-started/models-and-keys.md) ## Measured A synthetic load of memories with 2.4 versions per key and 3072-dimension embeddings (two vectors per row) took about 13.6 KB per key in ClickHouse, almost all of it the embedding row; a smaller embedder takes less, and parts with few rows carry somewhat more per row until they merge. [Memory at scale](https://clickhouse.github.io/loom/benchmarks/results.md#memory-at-scale) ## Start [Quickstart](https://clickhouse.github.io/loom/getting-started/quickstart.md) · [Prerequisites](https://clickhouse.github.io/loom/getting-started/overview.md#prerequisites) · [Upgrades](https://clickhouse.github.io/loom/operations/upgrades.md) · [Backup and recovery](https://clickhouse.github.io/loom/operations/backup-and-recovery.md) --- # Frequently asked questions > Source: https://clickhouse.github.io/loom/faq/ (Markdown: https://clickhouse.github.io/loom/faq.md) Short answers to the first questions people ask. Each links to the page that explains it. ## Using it ### What is the fastest way to try Loom? If a Loom server already exists, install the Claude Code or Codex plugin and point it at the server: [connect a coding agent](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md). To run a server on a laptop, `loom up` boots one in a few minutes: [quickstart](https://clickhouse.github.io/loom/getting-started/quickstart.md). ### How do I give Claude Code, Codex or Cursor long-term memory? Use the Claude Code or Codex plugins for session-start context and capture, or connect any other MCP client with `loom install --client ` (see [MCP clients](https://clickhouse.github.io/loom/features/integrations/mcp-clients.md)). Boot a server via the [quickstart](https://clickhouse.github.io/loom/getting-started/quickstart.md). Cursor gets the MCP entry and the `loom` and `loom-code` skills (no hooks for session-start context; not yet tested in Cursor); transcript capture comes from the separate `sessionEnd` hook. See [Connect a coding agent](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md) and [Cursor](https://clickhouse.github.io/loom/features/integrations/cursor.md). ### Why are my search results nonsense? The install runs a fake embedder: its `.env` (a checkout's, or `~/.loom/.env`) names `fake-ngram-*`, whose vectors make retrieval meaningless. Switch its embedder, which re-embeds what is stored ([Models and keys](https://clickhouse.github.io/loom/getting-started/models-and-keys.md)); a fresh install of the released stack embeds with a local model already. A hosted embedder retrieves better on paraphrase-heavy questions ([providers](https://clickhouse.github.io/loom/operations/deployment.md#providers)). ### Why does `memory.search` say `abstain` instead of returning something? No hit cleared the similarity floor calibrated for your embedder, so Loom says it doesn't know rather than hand back the nearest weak match. The reason is in the response. See [retrieval and ranking](https://clickhouse.github.io/loom/concepts/retrieval-and-ranking.md). ### Why was my write flagged as a conflict, or refused? A write close in meaning to an existing memory is marked `conflicted` and linked to it; a close write against a pinned memory is refused with `409 PINNED_CONFLICT`; a write matching a blacklist pattern gets `422 BLACKLISTED`. Resolve conflicts in the dashboard's Review queue or with `conflicts.resolve`. See [conflicts and refusal](https://clickhouse.github.io/loom/features/memory/conflicts-and-refusal.md). ### Can I use Loom just for myself? Yes. Run the [local stack](https://clickhouse.github.io/loom/getting-started/quickstart.md), which has no login and one namespace, and connect your agents with the [plugin or `loom install`](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md). On a server with login, each person gets a `personal-` namespace that their MCP clients write to by default and nobody else reads without a grant. Memory saved from Claude Code is then there for Codex, Cursor and the dashboard. ### Can two agents see each other's memories? Every member of a namespace can read its rows. Each agent writes into its own layer, and an admin can copy a memory into the namespace's shared layer with `memory.promote`. A token can read several namespaces while writing one. See [identities and namespaces](https://clickhouse.github.io/loom/concepts/identities-and-namespaces.md). ### When should I use knowledge documents instead of memories? A memory is one short claim that can be revised. A document, a long page or anything over a couple of kilobytes goes in the knowledge plane with `knowledge.add`, which chunks and embeds it. See [memory](https://clickhouse.github.io/loom/concepts/memory.md#what-goes-where) and [knowledge documents](https://clickhouse.github.io/loom/features/knowledge.md). ### How do I make Loom forget something? `memory.delete` tombstones one memory (restorable for a grace window); `erasure.request` removes everything a user or namespace owns, across every table, and verifies nothing is left. Loom deletes nothing on its own schedule unless an operator sets a retention window. See [retention and erasure](https://clickhouse.github.io/loom/features/memory/retention-and-erasure.md). ### How do I find out why a result ranked where it did? Call `memory.search` with `explain=true`: each hit's `score_components` carry the ranking signals and, after a reranker, its move (`rerank_from`, `rerank_pos`, `rerank_delta`) and, when the reranker changed the order, its own fused score (`rank_score_pre_rerank`). `memory.search_trace` scores the vector leg alone. See [search and ask](https://clickhouse.github.io/loom/features/memory/search-and-ask.md). ### Can Loom answer questions about my code? Yes, for repositories the server indexes: `code.search`, `code.symbol`, `code.callers`, `code.impact` and more, each answer saying which commit and index tier backs it. See the [code index](https://clickhouse.github.io/loom/features/code-index/overview.md) and [adding a repository](https://clickhouse.github.io/loom/features/code-index/repositories.md). ### Why doesn't my coding agent use the code index? The plugin attaches the index only when the session starts inside a git checkout whose remote is an indexed repository you can read. Start the session from the repository's checkout, check the repository is listed under **Repositories** in the dashboard (`/repositories`), and see [when the code index is used](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md#when-the-code-index-is-used). ## Running it ### Do I need ClickHouse Cloud? No. Loom runs against ClickHouse OSS too, and the laptop stack starts one in Docker. Cloud is the intended production path because single-node OSS has no replication or point-in-time recovery. See [deployment](https://clickhouse.github.io/loom/operations/deployment.md). ### Can I use Anthropic or Gemini models? Yes. The LLM provider can be OpenAI or Anthropic, and embeddings can come from OpenAI, Gemini or a local CPU model. Changing the embedding model changes the vector space, so run `loom reembed` afterwards. See [LLM and embeddings](https://clickhouse.github.io/loom/architecture/llm-architecture.md). ### How do I give someone access? Mint a token (`loom mint-token`, or the dashboard's Settings → Tokens) and grant a role on a namespace. See [tokens and grants](https://clickhouse.github.io/loom/guides/tokens-and-grants.md). ### Something is wrong. Where do I start? `loom doctor` on the server, `/loom:loom-status` in Claude Code, and `/v1/health`. Then [troubleshooting](https://clickhouse.github.io/loom/operations/troubleshooting.md), which is organised by symptom. ## The project ### Why is an operation missing from my MCP tool list? The MCP listing depends on the toolset (`?toolset=` or `LOOM_MCP_TOOLSET`, default `core`), and code tools are listed only with `LOOM_MCP_CODE_TOOLS=true`. Under `all`, an op that neither the core nor the atomic roster lists and that is marked `experimental` is left out. Every op is still callable by name through `loom_op` (reads only on `readonly`), over HTTP and from the SDKs. See [MCP toolsets](https://clickhouse.github.io/loom/reference/mcp.md#toolsets). ### How good is the retrieval, and how do you know? Loom ships its benchmark suite with the method, datasets and exact commands. Only results reproducible from committed files are stated. See [benchmarks](https://clickhouse.github.io/loom/benchmarks/overview.md). ### How is Loom different from putting a vector database next to my app? Under the default ClickHouse authority, one ClickHouse database holds the rows, vectors, text index, graph, audit and erasure state, so nothing has to be kept in sync. The [Postgres memory authority](https://clickhouse.github.io/loom/architecture/storage-authority.md#why-two-stores) holds live memory and knowledge state in Postgres, and Loom keeps the two stores in step itself. Each write is checked against what is stored. The [overview](https://clickhouse.github.io/loom/getting-started/overview.md) lists what each property costs and when Loom does not fit. --- # What's included > Source: https://clickhouse.github.io/loom/getting-started/whats-included/ (Markdown: https://clickhouse.github.io/loom/getting-started/whats-included.md) This page lists Loom's main capabilities, self-hosted from the Apache-2.0 repository: one section per area, each row linked to the page that covers it. The [docs index](https://clickhouse.github.io/loom/index.md) maps the rest of the docs. Several capabilities must be turned on; the linked page says how, and the boolean switches that load false are listed in [status](https://clickhouse.github.io/loom/reference/status.md). The embedder is your choice: the bundled local model, a hosted provider or an OpenAI-compatible endpoint you run. Extraction, summaries and cited answers need a real LLM ([models and API keys](https://clickhouse.github.io/loom/getting-started/models-and-keys.md)). ## Memory | Capability | What it does | Page | | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | | Shared long-term memory | One claim per key in a namespace, judged against the facts already there before it lands | [Writing memory](https://clickhouse.github.io/loom/features/memory/writing.md) | | Typed records | Rows with exact fields for data that has identity, answering predicate questions without parsing text | [Typed records](https://clickhouse.github.io/loom/features/memory/writing.md#typed-records) | | Search and ask | Ranked hits, or an answer that cites memory keys; both abstain when memory does not support one | [Search and ask](https://clickhouse.github.io/loom/features/memory/search-and-ask.md) | | Conflicts and refusal | A close but different value is flagged and the namespace owner decides which fact stands; a registered pattern refuses matching writes | [Conflicts and refusal](https://clickhouse.github.io/loom/features/memory/conflicts-and-refusal.md) | | History and provenance | Every write is a new version, with who wrote it and how it entered memory | [History and provenance](https://clickhouse.github.io/loom/features/memory/history-and-provenance.md) | | Extraction from transcripts | The server decides what in a transcript is durable | [Extraction](https://clickhouse.github.io/loom/features/memory/extraction.md) | | Import | Facts another memory system already distilled, written with the same checks as any write | [Importing](https://clickhouse.github.io/loom/features/memory/writing.md#importing-from-another-memory-system) | | Change webhooks | A signed event when a memory is written, revised, deleted, restored, redacted or erased, posted to a URL a namespace admin registers | [Change webhooks](https://clickhouse.github.io/loom/features/memory/writing.md#change-webhooks) | | Working memory and profiles | Small named blocks and profile text an agent is given every time | [Working memory and profiles](https://clickhouse.github.io/loom/features/memory/working-memory-and-profiles.md) | | Procedures and learnings | Steps learned from reported runs, reflections and a review queue of candidate facts | [Procedures and learnings](https://clickhouse.github.io/loom/features/memory/procedures-and-learnings.md) | | Memories linked to code | A memory about code follows the definitions it names as they move or change | [Memories linked to code](https://clickhouse.github.io/loom/concepts/memory.md#memories-linked-to-code) | | Entity graph | Entities and their relations beside the memory rows | [Entity graph](https://clickhouse.github.io/loom/features/entity-graph.md) | ## Documents, code and work records | Capability | What it does | Page | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Document search | Documents split into sections, embedded, searched and cited by passage; searches that found nothing are grouped into knowledge gaps | [Knowledge documents](https://clickhouse.github.io/loom/features/knowledge.md) · [Knowledge gaps](https://clickhouse.github.io/loom/features/knowledge.md#knowledge-gaps) | | Files and document sources | PDF, DOCX, PPTX, XLSX, CSV and more parsed by an extractor service; registered sources synced into a namespace | [Ingesting files](https://clickhouse.github.io/loom/features/knowledge.md#ingesting-files) · [Document sources](https://clickhouse.github.io/loom/features/knowledge.md#document-sources) | | Unified search | Memory, profile, documents and past conversations in one call, and a context pack for a task | [Unified search](https://clickhouse.github.io/loom/features/unified-search.md) | | Ask AI | An assistant in the dashboard that answers about what a namespace holds, with a citation for every claim; its model answers run on a chat server you deploy beside Loom | [Dashboard](https://clickhouse.github.io/loom/features/dashboard.md#ask-ai) · [Chat server setup](https://clickhouse.github.io/loom/features/integrations/librechat.md#the-deployed-assistant) | | Code index | Definitions, callers, impact and search by meaning over your git repositories | [Code index](https://clickhouse.github.io/loom/features/code-index/overview.md) | | PR and release history | Merged pull requests joined against today's symbols, callers and tests | [History, PRs and releases](https://clickhouse.github.io/loom/features/code-index/history-and-prs.md) | | Tracker | Issues, pull requests, incidents and CI runs read live from your own ClickHouse | [Tracker plane](https://clickhouse.github.io/loom/features/tracker/overview.md) | | Conversations and sessions | Searchable verbatim transcripts, recorded once an admin turns logging on for the namespace, and session briefs an agent resumes from | [Conversations and sessions](https://clickhouse.github.io/loom/features/conversations-and-sessions.md) · [Recording](https://clickhouse.github.io/loom/features/conversations-and-sessions.md#recording-is-the-owners-choice) | | Runs | One record per task execution, with the memories it relied on | [Runs](https://clickhouse.github.io/loom/features/runs.md) | | Tasks, decisions and policies | Task inboxes, a decision record and policies for coordination between agents | [Tasks, decisions and policies](https://clickhouse.github.io/loom/features/tasks-and-decisions.md) | ## Access, governance and operations | Capability | What it does | Page | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | Namespaces and sharing | The unit of isolation and sharing inside an org; every read is scoped to one | [Namespaces and sharing](https://clickhouse.github.io/loom/features/namespaces-and-sharing.md) | | Tokens and roles | Tokens, grants and roles per namespace | [Tokens and grants](https://clickhouse.github.io/loom/guides/tokens-and-grants.md) | | Single sign-on and SCIM | OIDC sign-in for the dashboard, CLI and MCP clients, and deprovisioning through SCIM | [Single sign-on](https://clickhouse.github.io/loom/operations/sso.md) | | Audit log | One row per audited operation or server event, its free-text values stored as hashes, sealed into a verified checkpoint chain | [Audit log](https://clickhouse.github.io/loom/features/observability/audit.md) | | Retention and erasure | Expiry, configured retention, and erasure of a scope everywhere Loom keeps it | [Retention and erasure](https://clickhouse.github.io/loom/features/memory/retention-and-erasure.md) | | Sensitive values | A `high`-sensitivity memory is never embedded and is readable by key only, with an audited read; detection is configurable | [Memory types](https://clickhouse.github.io/loom/concepts/memory.md#memory-types) | | Model-call controls | An org admin can switch off every model call for the org's data, and a deployment can route them through an inference gateway | [Switching model calls off](https://clickhouse.github.io/loom/getting-started/models-and-keys.md#switching-an-orgs-model-calls-off) · [Inference gateway](https://clickhouse.github.io/loom/getting-started/models-and-keys.md#through-an-inference-gateway) | | Cost and usage | Every embedding and LLM dollar recorded against the operation that spent it | [Cost and usage](https://clickhouse.github.io/loom/features/observability/cost-and-usage.md) | | Telemetry | A per-operation ledger, OpenTelemetry traces, logs and metrics, and query attribution | [Telemetry](https://clickhouse.github.io/loom/features/observability/telemetry.md) | | Dashboard | The web app where people decide conflicts, proposals, tasks, grant requests and policies | [Dashboard](https://clickhouse.github.io/loom/features/dashboard.md) | | SQL access | Read-only ClickHouse views over a tenant's memories, history, conversations and request ledger, also readable from Postgres | [SQL surface](https://clickhouse.github.io/loom/reference/sql-surface.md) · [Query from Postgres](https://clickhouse.github.io/loom/guides/query-from-postgres.md) | | Replicas and Kubernetes | Identical replicas behind one URL, and a Helm chart | [Running N replicas](https://clickhouse.github.io/loom/operations/replicas.md) · [Kubernetes](https://clickhouse.github.io/loom/operations/kubernetes.md) | | Postgres memory authority | Optional: live memory and document heads in Postgres, chosen at first boot | [Postgres](https://clickhouse.github.io/loom/operations/postgres.md) | | Backup and recovery | ClickHouse backups, plus the document bucket and Postgres where used | [Backup and recovery](https://clickhouse.github.io/loom/operations/backup-and-recovery.md) | | Upgrades | The image is replaced and its migrations apply at boot; a fresh install takes the same path | [Upgrades](https://clickhouse.github.io/loom/operations/upgrades.md) | | Namespace copy and OKF bundles | A namespace copied to another Loom on a schedule, or exported and imported as an OKF bundle | [Copying a namespace](https://clickhouse.github.io/loom/operations/okf-sync.md) · [OKF bundles](https://clickhouse.github.io/loom/features/integrations/okf.md) | | Usage export | One UTC day of usage and a snapshot of the namespaces, written to S3 as Parquet for a data warehouse | [Analytics export](https://clickhouse.github.io/loom/operations/analytics-export.md) | ## Ways to connect MCP for any MCP client, plugins for Claude Code and Codex, framework adapters, an HTTP API, Python and TypeScript SDKs, and the `loom` CLI: [Integrations](https://clickhouse.github.io/loom/features/integrations/index.md) · [MCP clients](https://clickhouse.github.io/loom/features/integrations/mcp-clients.md) · [Frameworks](https://clickhouse.github.io/loom/features/integrations/frameworks.md) · [HTTP API](https://clickhouse.github.io/loom/reference/http-api.md) · [Python SDK](https://clickhouse.github.io/loom/reference/python-sdk.md) · [TypeScript SDK](https://clickhouse.github.io/loom/reference/typescript-sdk.md) · [CLI](https://clickhouse.github.io/loom/reference/cli.md). --- # Glossary > Source: https://clickhouse.github.io/loom/glossary/ (Markdown: https://clickhouse.github.io/loom/glossary.md) Loom's terms, alphabetically, each defined once. Code identifiers are in backticks. **Abstain** — `memory.search` returning `abstain=true` with a reason code instead of the nearest weak match, when the best hit is below the abstain gate calibrated for the embedder. It separates "nothing was stored" from "something unrelated scored 0.3". See [retrieval and ranking](https://clickhouse.github.io/loom/concepts/retrieval-and-ranking.md). **Actor** — the server-stamped column naming who wrote a row: `human:` or `process:`, with the kind taken from the token. Never read from the request body. See [identities and namespaces](https://clickhouse.github.io/loom/concepts/identities-and-namespaces.md). **Agent** — the third element of the identity tuple: the name of the program writing and reading (`claude-code`, `support-bot`). It records authorship; it is not a read boundary: every member of a namespace reads every agent's rows. **Attestation** — the `verified_*` columns recording that a row was checked. An edit leaves them in place and marks them stale; a reader decides trust by comparing the attested content hash. See [history and provenance](https://clickhouse.github.io/loom/features/memory/history-and-provenance.md). **Audit log** — `loom.audit_log`: one content-hashed row per audited operation, grouped into hash-chained hourly checkpoints. Rows carry hashes, not content. See [audit](https://clickhouse.github.io/loom/features/observability/audit.md). **Belief revision** — the design lens for the memory lifecycle: Expand (`memory.set`), Revise (`memory.update_by_id`), Contract (`memory.delete`), Reinforce (`memory.pin`), Refuse (`blacklist.add`). See [how a write is judged](https://clickhouse.github.io/loom/concepts/memory.md#how-a-write-is-judged). **Blacklist** — registered substrings or regexes that refuse matching writes (`422 BLACKLISTED`); `sweep=true` also tombstones existing matches. **Chunk** — a piece of a knowledge document or a source file, stored and embedded separately so retrieval can return the passage rather than the whole document. **Code link** — a memory's link to a definition or file it names, followed through renames and edits. Its state is `fresh`, `moved`, `changed`, `unresolved` or `missing`. A code name not yet on trunk waits up to `memory_code_links_pending_days` (14 days). See [memories linked to code](https://clickhouse.github.io/loom/concepts/memory.md#memories-linked-to-code). **Code namespace** — a namespace of kind `code` holding one indexed git repository: its chunks, symbols, edges, commits and PR history. See [code index](https://clickhouse.github.io/loom/features/code-index/overview.md). **Commit coverage** — whether every index tier has processed a given commit; `code.index_status(commit=...)` answers it, and `code.coverage.watch` notifies when it becomes true. See [freshness](https://clickhouse.github.io/loom/features/code-index/freshness.md). **Compiler tier** — code-index facts derived from a compiler-grade indexer (SCIP) rather than the parser tier; call edges and references resolved by type information. **Conflict** — a write semantically close to an existing memory (cosine ≥ `LOOM_CONFLICT_THRESHOLD`). Both rows are marked `conflicted` and cross-linked; a person or agent resolves them. See [conflicts and refusal](https://clickhouse.github.io/loom/features/memory/conflicts-and-refusal.md). **Consolidation** — the background pass that merges fragmented memories about the same subject and records lineage (`memory_belief_sources`, `support_type='derived_from'`). **Content hash** — `content_sha256` of a memory's value. It deduplicates exact repeats, backs the `if_match_sha256` precondition and anchors audit rows. **Context pack** — `context.pack`: a bounded bundle of working memory, profile, relevant memories, procedures and decisions, loaded once at the start of a session. See [unified search](https://clickhouse.github.io/loom/features/unified-search.md#contextpack). **Dashboard** — the web application at the server root, for people: memory, activity, sources, review queue, tasks, repositories and settings. See [dashboard](https://clickhouse.github.io/loom/features/dashboard.md). **Decision** — a first-class record of a choice (`decision.record`) with reasoning, rejected alternatives and, later, its outcome (`decision.outcome`). See [tasks and decisions](https://clickhouse.github.io/loom/features/tasks-and-decisions.md). **Entity edge** — a weighted co-occurrence link between two entities in `loom.entity_edges`, with a validity interval; the graph `facts.traverse` walks. See [entity graph](https://clickhouse.github.io/loom/features/entity-graph.md). **Erasure saga** — the resumable, fail-closed sequence of steps `erasure.request` runs to remove a subject's data from every table (and configured external backends), ending in a residue census. See [retention and erasure](https://clickhouse.github.io/loom/features/memory/retention-and-erasure.md). **Extraction** — turning conversation messages into candidate memories (`memory.set_from_messages`), with an LLM and quality filters. See [extraction](https://clickhouse.github.io/loom/features/memory/extraction.md). **Extractor stamp** — a hash of the code-index extractor modules' code tokens. A change to it re-chunks every indexed repository; comments and docstrings don't count. See [code-index architecture](https://clickhouse.github.io/loom/architecture/code-index-architecture.md). **FINAL** — the ClickHouse modifier that collapses a `ReplacingMergeTree` table to the latest version per key at read time. Loom reads `loom.memory` with it. See [ClickHouse](https://clickhouse.github.io/loom/architecture/clickhouse.md). **Fence** — the token `task.claim` returns with a lease; later `task.progress`, `task.heartbeat`, `task.complete` and `task.fail` calls must present it, so a lapsed claimant cannot write. **Grant** — a row giving a subject a role on a namespace (`grants.*`). See [tokens and grants](https://clickhouse.github.io/loom/guides/tokens-and-grants.md). **Identity tuple** — `(org, namespace, agent, user_id, session_id)` on every row. The first three are required; the last two default to `'-'`. See [identities and namespaces](https://clickhouse.github.io/loom/concepts/identities-and-namespaces.md). **`ignored_args`** — the response field naming request fields the server did not use (unknown or retired), so a client learns about drift without a failure. **Inbox** — a named queue of tasks owned by a namespace (`task.*`, `inbox.*`). See [tasks and decisions](https://clickhouse.github.io/loom/features/tasks-and-decisions.md). **Knowledge document** — a document added with `knowledge.add`, chunked and embedded for retrieval. Distinct from a memory: it is reference material, not a belief. See [knowledge documents](https://clickhouse.github.io/loom/features/knowledge.md). **Learning** — a candidate waiting in `loom.learnings_queue` for a person's decision (`learnings.list`, `learnings.review`): a fact several writers back, promoted to the shared layer; a conversation-seeded fact, promoted to a namespace memory row; or a rule a bot stated about itself, promoted to the namespace's knowledge. **`log_comment`** — the ClickHouse query setting Loom fills with the calling op, worker or CLI command, so `system.query_log` attributes every statement. **`maturity`** — an op's declared `stable`/`experimental` field. An experimental op that neither the core nor the atomic toolset lists is left out of the `all` listing; it is still callable by name. See [MCP toolsets](https://clickhouse.github.io/loom/reference/mcp.md#toolsets). **MCP tool** — the Model Context Protocol projection of an op, named `loom_` with dots replaced by underscores (`loom_memory_set`), plus a few synthetic tools such as `loom_search` and `loom_op`. See [MCP](https://clickhouse.github.io/loom/reference/mcp.md). **Memory** — one row in `loom.memory`: a key, a value, its identity tuple, type, tags, validity and provenance. See [memory](https://clickhouse.github.io/loom/concepts/memory.md). **Memory key** — the readable identifier of a memory within its namespace and agent, conventionally `//`. A write to an existing key supersedes it. **Memory type** — the kind of a memory: `semantic`, `episodic`, `procedural`, `preference`, `profile` and others; the type decides whether it is embedded and how it is searched. **Mentor hint** — a short usage suggestion Loom attaches to results or serves through `mentor.brief`, pointing an agent at a plane or op it is not using well. **Namespace** — the unit of storage, sharing and access: a named workspace inside an org. Kinds are `memory`, `code` and `bench`. See [namespaces and sharing](https://clickhouse.github.io/loom/features/namespaces-and-sharing.md). **Operation (op)** — one declared entry in the registry (`server.operations.registry.ALL_OPERATIONS`), served at `POST /v1/`, as an MCP tool and through the SDKs. See [operations reference](https://clickhouse.github.io/loom/reference/operations.md). **Org** — the top-level tenant; the first element of the identity tuple. **Pinned** — a memory marked by `memory.pin`. A conflicting write against a pinned memory is rejected with `409 PINNED_CONFLICT`. **Plane** — a product area with its own ops and tables: memory, knowledge, conversation, graph, code, tracker, runs and tasks, audit. **Procedure** — a reusable how-to (`procedure.add` / `match`), stored as a memory of type `procedural`. See [procedures and learnings](https://clickhouse.github.io/loom/features/memory/procedures-and-learnings.md). **Profile** — an agent's or user's standing context (`profile.*`), loaded first by `context.pack`. **Projection** — a row derived from a write (an embedding, a graph edge, a summary). A projection that fails is recorded and retried (`ops.projection_failures.*`). **Read scope** — `read_scope`, which bounds a read inside one namespace: `namespace` (every row, the default), `own_user` or `own_session`. A token's claim is a ceiling a request can narrow but not widen. See [identities and namespaces](https://clickhouse.github.io/loom/concepts/identities-and-namespaces.md#read-scope-inside-a-namespace). **Record** — a structured memory validated against a registered record schema (`memory_schema.*`). Not to be confused with a tracker record. **ReplacingMergeTree (RMT)** — the ClickHouse engine behind most Loom tables: a new version row replaces the old one at merge or `FINAL` read time, so every write path must carry every column forward. See [ClickHouse](https://clickhouse.github.io/loom/architecture/clickhouse.md). **Residue census** — the last step of an erasure saga: a count of live rows in scope over every table the read paths touch. Any residue fails the saga. **Role** — `viewer`, `writer`, `admin`, `org_admin`, `platform_admin`, in increasing privilege. Each op declares the minimum it needs. **Run** — one agent execution, bracketed by `run.begin` and `run.complete`, recording what it recalled and wrote. See [runs](https://clickhouse.github.io/loom/features/runs.md). **Session** — a conversation or working period, the fifth element of the identity tuple; a **session brief** (`session.brief_*`) is its saved summary for resuming later. **Shared layer** — one canonical row per `(org, namespace, memory_key)`, promoted from the rows agents write (`memory.promote` / `memory.demote`, table `loom.shared`). "Private" for the other layer means not promoted, not hidden. **Source class** — the server-stamped provenance of a row: `asserted`, `extracted`, `observed`, `derived`, `imported` or `external`. A request may only downgrade it. **Tier** — which extractor produced a code-index fact (parser or compiler); answers say which tier backs them. **Tombstone** — a higher-version row marking a memory deleted. `memory.restore` can undo it within the grace window; nothing is physically deleted by `memory.delete`. **Toolset** — the MCP listing a client asks for (`core`, `coding`, `atomic`, `tasks`, `tracker`, `readonly`, `all`); it changes what is listed, not what the token may call. **Tracker** — the plane that reads an issue tracker's records (issues, PRs, CI runs) from a configured source for duplicate checks, bisect and test history. See [tracker](https://clickhouse.github.io/loom/features/tracker/overview.md). **Trust mode** — the acceptance policy an inbox owner sets for a poster; Loom stores it and never changes it itself. **Valid time** — `valid_at` / `valid_to` on memories (`valid_from` / `valid_to` on entity edges): when a fact was true in the world. Loom records valid time only; it is not bi-temporal. **Version** — the monotonically increasing column a `ReplacingMergeTree` keeps the highest of; every write of a memory (including a delete) is a new version. **Worker** — a supervised background loop registered as a row in `server/workers/registry.py` (`_WORKERS`). See [workers](https://clickhouse.github.io/loom/reference/workers.md). **Working memory** — scratchpad blocks (`working_memory.*`) for the current task: not embedded, not ranked, loaded ahead of durable memory by `context.pack`. See [working memory and profiles](https://clickhouse.github.io/loom/features/memory/working-memory-and-profiles.md). --- # Identities and namespaces > Source: https://clickhouse.github.io/loom/concepts/identities-and-namespaces/ (Markdown: https://clickhouse.github.io/loom/concepts/identities-and-namespaces.md) Loom answers two questions on every call: whose row is this (the identity tuple, plus two columns the server stamps), and may this caller touch it (the token, the namespace's grants and its classification). This page explains the model. Minting and revoking tokens is in [Tokens and grants](https://clickhouse.github.io/loom/guides/tokens-and-grants.md); enforcement is in [Security](https://clickhouse.github.io/loom/architecture/security.md). ## Where namespaces fit - An **org** is the tenant. - A **namespace** is the unit of sharing and access inside it. A deployment usually has one per team or project, one per bot that must stay isolated, one `personal-` per person, and one `code-[-]` per indexed branch. - Every member of a namespace reads every row in it. - `agent` records who wrote a row; it is not a boundary. - `user_id` / `read_scope=own_user` separates end users inside one bot's namespace. - Access is a role (viewer/writer/admin) on a namespace, carried by a token or added by a grant. ## The identity tuple Every row carries five identity columns: ``` (org, namespace, agent, user_id, session_id) ``` | Part | Meaning | Default | | ------------ | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | `org` | The tenant. Nothing crosses an org boundary except through a wildcard operator token. | (required; single-user mode fills it, below) | | `namespace` | The unit of isolation and sharing inside an org. Every read is scoped to `(org, namespace)`. | (required; single-user mode fills it, below) | | `agent` | Who wrote the row (`support-bot`, `claude-code`). Authorship, not a read boundary. | (required; single-user HTTP fills it on some ops, [HTTP reference](https://clickhouse.github.io/loom/reference/http-api.md#request)) | | `user_id` | The end user the row is about. | `-` | | `session_id` | One conversation or run of a conversation. | `-` | Many operations also take `run_id` (default `-`) to scope a write or read to one task execution. The token fixes the org and the namespaces a caller may reach; the rest of the tuple travels in each request body. The SDKs bind `(org, namespace, agent)` when the client is constructed and pass `user_id`, `session_id` and `run_id` per call, `-` for each one a call does not name. Where one of these fields filters a read (searches, asks, the startup pack, conversation and run listings, `memory.update`'s key lookup), `-` means no filter: a search that names no run reads every run's rows. Where the field is the row's address (`memory.get`, `memory.list`, `memory.stack`, working memory, profile writes, saved messages), `-` is the row written with no user, session or run. **Reads span every author.** A read defaults to `agent="*"`: every row of the namespace, whoever wrote it. Passing an `agent` only narrows. `user_id` and `session_id` on a read filter to rows written under them. **Why the tuple leads the tables.** The sort keys of `loom.memory` and `loom.memory_embeddings` start with `(org, namespace, agent, user_id, session_id)`, so a read scoped to one namespace is a primary-key range in ClickHouse rather than a scan, and naming an agent narrows it. A read across agents, the default, filters `user_id` and `session_id` inside the namespace's range: the key skips granules on them only as far as the agent column allows, and no skip index covers them. Any scoped read still reads at least a granule of every part its range spans ([memory at scale](https://clickhouse.github.io/loom/benchmarks/results.md#memory-at-scale)). ### What the server stamps Two more columns are decided by the server on every write and are never read from the request body, except that a caller may downgrade `source_class` (`server/memory/app/provenance.py` for `actor`, `server/memory/domain/provenance.py` for `source_class`): | Column | Values | Decided by | | -------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `actor` | `human:` or `process:` (a service account) | The verified token (`writer_actor`). A queued job or CLI import writes as the principal that started it. | | `source_class` | `asserted`, `extracted`, `observed`, `derived`, `imported`, `external` | The operation and the `source_trace` convention (`source_class_for`). | A request may declare only a downgrade of `source_class`: `external` (content from a tool or page), `imported` or `derived`. Asking for anything else is a 422. The reason is that provenance feeds ranking and `memory.ask`'s choice of evidence: if a caller could label scraped text `asserted`, the label would mean nothing. `agent` is caller-supplied because it names a role in the system, while `actor` is the authenticated principal behind it. ## Orgs and namespaces An **org** is a tenant; a **namespace** is where rows live and the unit access is granted on. A namespace comes into being when it is registered (`namespace.register`, which makes the caller its `admin`), claimed at sign-up, provisioned by the server (the single-user namespace, and a `personal-` namespace for each signed-in person), or created for an indexed repository. Its directory row in `loom.namespaces` carries its configuration: description, kind, visibility, join policy, conflict and proposal policy, cost cap, retention and retrieval defaults. | Namespace kind | Holds | Read by | | -------------- | ------------------------------------------------- | ---------------------------------------------------------------------------- | | `memory` | Facts, documents, conversations: the default kind | memory, knowledge and conversation operations, and fan-outs | | `code` | One indexed repository and branch | the `code.*` operations, addressed by name; `loom_search` under a code scope | | `bench` | A benchmark corpus | benchmarks; its traffic is classed `bench` | A search that fans out over all of a caller's namespaces spans the `memory` kind only (`LOOM_MEMORY_SEARCH_FANOUT_KINDS`): sweeping every code index for a personal fact costs a statement per index and finds nothing. **Why namespaces, not per-agent stores.** The common case is several agents and people that should see the same facts: a team's deploy window, a customer's history. Making the namespace the read boundary means sharing is the default inside it and absent across it, and a person or bot is given access by granting a role on a namespace, not by copying rows. ### Visibility, joining and restriction | Setting | Values | Effect | | ------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `visibility` | `private` (default), `public` | A public namespace is listed in `namespace.browse` and can be joined. Visibility grants no read by itself. | | `join_policy` | `open`, `approval`, `closed`, `inherit` (unset) | What `namespace.join` does. `inherit` resolves to `open` for a public namespace that does not require approval, and `approval` otherwise. | | `requires_approval` | `true`, `false` | Read only under `inherit`: every join then waits for an admin (`grants.approve`). | | `classification` | `standard` (default), `restricted` | Switches every widening default off (below). | On an `open` namespace a `viewer` join is active at once and a `writer` or `admin` join is pending until an admin approves it; on `approval` every join is pending; `closed` refuses joins. A **restricted** namespace (`classification=restricted` on its directory row, or on its entry in `LOOM_CODE_REPOS_JSON`) admits a call only when the token belongs to the namespace's org, names the namespace literally (not through `*`), holds a role there that reaches the operation's role, and the token's own subject holds a live direct grant. No org-readable rule, public visibility, wildcard, inherited service-account grant, `org_admin` or `platform_admin` bypasses it, and a refusal does not reveal whether the namespace exists. Fan-outs, context packs and rollups skip restricted namespaces the caller cannot read, and each admitted request is audited. The step-by-step setup is in [Tokens and grants](https://clickhouse.github.io/loom/guides/tokens-and-grants.md#restricted-namespaces-classificationrestricted). ## Roles Roles are ordered, and a higher role satisfies every check for a lower one. Each operation declares the minimum role it needs (`GET /v1/operations` lists it). | Role | Grants | How it is held | | ---------------- | ------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `viewer` | Read. | Token claim or namespace grant. | | `writer` | Read, and write memory, knowledge and the other data planes. | Token claim or namespace grant. | | `admin` | Administer one namespace: grants, configuration, policy, restores, promotion to the shared layer. | Token claim or namespace grant; registering a namespace makes you its admin. | | `org_admin` | Org-wide views (for example cost without a namespace). | A recognised gate with no in-product assignment path. | | `platform_admin` | Cross-org operator views. | Conferred at login to the handles and verified emails in `LOOM_PLATFORM_ADMINS`, and to the tokens of a bot registered `--platform-admin`; minted only for a service account, with an expiry. | Grants carry only `viewer`, `writer` or `admin`, and so do the tokens Loom mints, but for a `platform_admin` one: org-wide power is an operator decision in configuration or at bot registration, not something a token can carry away. Code namespaces have readers (`viewer`) and maintainers (`admin`) and no `writer`. ## Who may call what A call on `(org, namespace)` is allowed when the namespace is not restricted (or passes the restricted checks) and either the token's own role for that namespace reaches the operation's role, or a live grant does. - **The token** (`sub`, `org`, `namespaces`, `roles`, optional `ns_roles` and `ns_write`) says which namespaces the caller reads, in priority order, and with what role. A **scoped** token (`ns_roles`, per-namespace roles) is authoritative for exactly the namespaces it lists. A **flat** token applies its `roles` to every namespace it names. `ns_write` narrows writes to a subset, so one token can read a team namespace and write only its own: ``` namespaces = ["team-shared", "agent-private"] ns_write = ["agent-private"] ``` - **Grants** (`loom.namespace_grants`, one row per `(org, namespace, subject)`) add a role on one namespace above what the token carries. - **A claim goes no further than the grant.** Where the subject holds a grant row on a namespace the token names, of any status, the claim reaches no further than that grant: a revoked or lowered grant (`grants.revoke`, `grants.create` with `replace=true`, `namespace.leave`, a released claim, a restore) ends every older token's claim there, and a pending or rejected request caps it at no access. A scoped token keeps its own role there, capped at the live grant. - **What is not touched.** A namespace the subject never held a grant on keeps its claim; revoke the token to take that away. - **When it applies.** The process that changed the grant applies it at once, every other within `LOOM_ACL_WATERMARK_POLL_SECONDS` (default 1 s), on this server or another. - **Service accounts inherit.** A service account with no grant of its own on a namespace uses its creator's grant there, so a token minted from the dashboard follows the person who made it, except on a restricted namespace. - **Operations addressed by an org-wide id**, such as `memory.get_by_id`, authorize the caller on the scope that holds the row. When the caller lacks the op's role on the row's namespace, they answer as for a missing row, so a refusal never confirms that the id exists ([row-scoped operations](https://clickhouse.github.io/loom/architecture/security.md#row-scoped-operations)). Why tokens plus grants: the token is verified locally on every call without a lookup, which keeps the hot path cheap; grants let an admin give and take access without re-issuing tokens. Revocations are shared through ClickHouse and reach every replica within the revocation cache interval. ## Reading several namespaces A search can read several namespaces at once: `namespace_scope` is `this` (the addressed namespace, the default) or `all_mine` (the addressed namespace plus the caller's own). The `loom_search` MCP tool and `context.pack` use the same fan-out. Memory search lists the namespaces in scope in `namespaces_searched` (a tiered search can stop before the last of them) and the ones the cap left out in `namespaces_skipped`. The caller's own namespaces depend on the token: | Token | Namespaces `all_mine` adds | | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | | Flat (most tokens; `loom mint-token`, self-signup) | The namespaces the token names, plus every namespace the subject holds a grant on | | Scoped (`ns_roles`; a token for _Only what you choose_) | Exactly the namespaces the token lists, in that order; grants added later do not join | | Delegated with `*` (a token for _Everything you belong to_) | The creator's live grants, so a namespace the person joins later is searched too | | Delegated naming namespaces | The ones it names where the creator still holds a grant | | None, auth off (`single-user`) | The namespaces the single user holds a grant on: the default one and every one they registered | Then the list is narrowed: - **Kinds.** Only namespaces of the kinds in `LOOM_MEMORY_SEARCH_FANOUT_KINDS` (default `memory`) are swept. Code and benchmark namespaces are read by name or with their own tools. The exception is MCP `loom_search` under a code scope: it also reads the scoped repository's trunk code namespace, for its documents and decisions, when the caller may read it ([unified search](https://clickhouse.github.io/loom/features/unified-search.md#which-namespaces-it-reads)). - **Restricted namespaces** stay only when the caller passes the restricted check. - **The cap.** Besides the addressed namespace, at most `LOOM_MEMORY_SEARCH_FANOUT_MAX_NAMESPACES` (default 32) are read, in the caller's list order: a scoped token's order, or a delegated token's named list; otherwise by name. A scoped token keeps its first ones; for anyone else the cut drops public namespaces before private ones. Memory, knowledge and conversation search name the rest in `namespaces_skipped` (every page of a paged memory search too), and `loom_search` says so in its text; address one by name to read it. The addressed namespace is always searched, and first: the fan-out only widens a search. Memory search then reads in tiers. For a scoped token the other namespaces follow in its order, and the search stops at the first with a confident match. For anyone else the addressed namespace and the private ones are read first, and the public ones only when that first pass comes up short. Some namespaces a caller can read are not its own and are never swept: those reached through a `*` claim or an org or platform role, public namespaces it has not joined, and namespaces it created by writing into them (they get an owner row but no grant, so a harness that writes a namespace per task does not widen every later search). Address them by name. A search covers one org, the token's own. ## Read scope inside a namespace Membership is the read boundary, which is right for a team's shared memory and wrong for one agent that serves many end users through one token: a prompt from one user could pull another user's memories into the answer. `read_scope` bounds a read inside the namespace: | `read_scope` | A read returns | | --------------------- | ----------------------------------------------------- | | `namespace` (default) | every row of the namespace | | `own_user` | rows written under the `user_id` the request names | | `own_session` | rows written under the `session_id` the request names | `memory.search`, `memory.search_trace`, `memory.ask`, `memory.recall`, `context.pack`, `namespace.inventory`, `profile.render` and `profile.structured` take it; `profile.structured` refuses `own_session`, since a user's profile spans their sessions. A token can carry a `read_scope` claim (and a pinned `read_user_id`) that is a ceiling: a request can narrow it but never widen it, and a scoped read without the `user_id` or `session_id` it binds to is refused with `IDENTITY_INCOMPLETE` instead of silently widening. Mint `own_user` tokens for multi-user agents. Both transports resolve the scope once, when they admit the call, and every read of memory rows in that call carries it, whichever op made the read: `memory.get`, `get_by_id` (another user's id answers `found=false`), `list`, `bulk_get`, `history`, `timeline`, `related`, `evidence`, `conflicts.list`, `memory.proposals`, session briefs, digests, procedures, working memory and the `loom_search` legs. Under a scope narrower than `namespace`: - `user_id="*"` is refused with `AUTH_INSUFFICIENT`, and so is `session_id="*"` under `own_session`. - A token pinned with `read_user_id` acts only as that user, on writes too: a body naming another `user_id` is refused with `AUTH_INSUFFICIENT` on every op, and a body that names none (or `-`) runs as the pinned user. - A write's answer names no row outside the scope: a merge into, a duplicate of or a pinned conflict with another user's row reports the verdict without that row's id, hash or key, and `conflicts.list` drops such peers from `conflict_with`. A pinned row still blocks a close write from any user (a namespace-wide invariant): a foreign pinned row within the conflict distance refuses the write, unnamed, however many of the writer's own rows sit closer. The narrowed writer's check reads the namespace's pinned rows (their type taken from the memory row) beside its own scoped read, so it costs at most about twice a namespace-wide writer's. A deferred `memory.set_from_messages` (`defer=true`, or `extraction_defer_agents`) stores the request's scope with it and replays under that scope. `run.begin` with another user's `run_id` starts a new run under a fresh id, and `run.complete` on it answers `NOT_FOUND`. A begin with a caller-supplied id claims it in ClickHouse Keeper (`loom.agent_run_claims`) and releases the claim once its run row is written, so of two users' concurrent begins on one id exactly one keeps it. A begin that fails before its row keeps the claim, so no other user's begin can take the id. While Keeper fails a begin only checks the run row; then a narrowed token keeps a supplied id only to re-open its own existing run, and a new one gets a server-made id, so a client that makes the id itself must use the returned `run_id`. - Session briefs are kept per user (per session under `own_session`): two users saving the same `brief_key` get two briefs, each sees only its own, and a namespace-wide token lists them under keys prefixed `~~` (or `~~~`). - `code.search` breaks ties with the scope's own user's recent files, whatever `user_id` the body names. - A `context.pack` leg the scope cannot bound (conversations, which carry no user, and the runs manifest, which reads every session of a user) answers nothing. - `namespace.inventory` counts memories and runs, and the agents, authors and subjects on its memory rows, inside the scope; documents and members namespace-wide (as `knowledge.*` and `grants.list` read them), and lists records (one per key, whoever wrote it last) and integrations (`webhook.list` is refused) as not counted (`complete: false`). - A blacklist pattern refuses every user's writes but records no user (its `created_by` is the token subject, which every per-user token of one backend shares), so every `blacklist.*` op is refused with `AUTH_INSUFFICIENT`: patterns are added, listed, removed and swept with a namespace-wide token. A pattern still refuses a narrowed token's write, and that 422 names no pattern id or reason. - `policy.decisions` lists only the caller's own decisions. `memory.unlock` and `memory.unpin` (admin) reach only the caller's own rows, like `memory.lock`. - An op that cannot bound its answer to one user (the entity graph `facts.*`, `conversation.*`, namespace exports, admin and curation ops, `ops.projection_failures.*`, `erasure.*`, `audit.list`, `cost.agent_summary`, `agents.connected`) is refused with `AUTH_INSUFFICIENT`, and so is a write to a table keyed without a user, where it would overwrite another user's row: `memory.record.write`, `decision.record`, `procedure.record_run` and `context.offload`. `memory.set_from_messages` skips its `record_schema` projection into records and says so in `degraded`. A token whose scope does not resolve (an `own_session` ceiling and a body naming no session, or an unpinned `own_user` token and a body naming no user) is refused on every read it scopes and on every write whose body carries a user or session (and `cost.summary`), rather than run as user `-`. Under `own_session`, ops with no session field (`cost.summary`, `memory.outcome`) are therefore always refused. The code-index ops (`code.*`) carry a user for attribution only and are admitted; a memory read inside them is refused. Namespace-wide rules (`policy.set`, `policy.delete`, `memory_schema.register` / `propose` / `approve`, `blacklist.*`) need a namespace-scoped token, and so does `policy.check`, which evaluates the caller's rules over every user's calls. The list is `server/identity/domain/read_scope.py` (`SCOPED_OPS`, `NARROW_SAFE_OPS`), and an op it does not name is refused. ## Agent-written rows and the shared layer Inside a namespace, rows in `loom.memory` are keyed by the full tuple: the same `memory_key` written by two agents, or for two users, is two rows. That is the per-agent ("private") layer, and it is what `memory.search` reads by default (`scope="private"`). "Private" here means _not promoted_, not _hidden_: every reader of the namespace can read these rows unless `read_scope` narrows it. The **shared layer**, `loom.shared`, holds one row per `(org, namespace, memory_key)`: a namespace-owned, canonical version of a fact that no longer belongs to the agent that learned it. `memory.promote` copies a row into it, with a rationale and an optional approver, and `memory.demote` retires it (the response says whether that happened inside the promotion's `reversible_for_days` window, default 7, up to 90). A search with `scope="shared"` reads that layer, and `memory.lineage` walks a shared row back to its promotion. `memory.promote` and `memory.demote` need `admin`. `memory.cross_promote` copies a row into another namespace, with provenance, for a caller who is admin on both. A `definition` is promoted only: - From its live namespace-wide row (agent, user and session `'-'`, `active`, not retired by a supersession and not past its `expires_at` or `valid_to`) under a `definition/` key, to that same key; a row of another type is never promoted under a `definition/` key. Else 422 `DEFINITION_PROMOTION_REFUSED`. - With the hash it approves (`approved_sha256`, the row's `content_sha256`): refused without it (422 `VALIDATION_FAILED`), and with 409 `PRECONDITION_FAILED` when the text no longer has it. - With a value that is a JSON object holding a non-empty string `term` and, optionally, a list of string `aliases`; else 422 `VALIDATION_FAILED`. - Not labelled `medium` and not reading as `medium` or `high` (422 `SENSITIVE_DEFINITION_REFUSED`; a `high` label answers `HIGH_SENSITIVITY_PROMOTION_REFUSED` first, as for any row). - Within `memory_shared_definitions_max`: a new key at the ceiling, or any promotion while the namespace is past it, is refused (409 `DEFINITION_CEILING`). - Under a name no other live shared definition holds as its term or an alias (409 `DEFINITION_NAME_TAKEN`, naming the keys and the names), names compared folded: format characters dropped, NFKC, whitespace collapsed, case-folded, then NFKC again. One namespace's definition promotions, with their ceiling and name checks, and its definition demotes run one at a time, the last to run deciding whether the definition is shared (503 `OVERLOADED` when another holds the namespace past a 10 s wait). Demoting a definition gives back its name and its place. The token's actor is its `promoted_by`: `approver` and `promoted_from_candidate_id` are ignored and named in `ignored_args`, and no close shared row under another key refuses it. `learnings.review` promotes no candidate of type `definition` to memory. A shared definition is **governed** while it was promoted from exactly one row, the namespace-wide row under its key, and that row is `active`, not retired by a supersession, not past its `expires_at` or `valid_to`, neither labelled nor graded `medium` or `high`, and holds the text the shared copy holds (the served hash of both, so a store that orders an object's keys differently is no change). An edit, a relabel to `medium` or `high` or a tombstone of the source ends it; reverting the edit to the approved text makes it governed again with no new promotion. `memory.search` leaves out a shared definition whose namespace-wide source is gone (deleted, or tombstoned by the expiry sweep) or labelled `medium` or `high`, since the shared layer has no label of its own. When a source cannot be read, it leaves out every shared definition. A copy promoted from a row other than the namespace-wide one is judged by the namespace-wide row under its key: withheld when there is none, served on that row's label otherwise. The copy itself stays: a relabel back, or the source written again, serves it again, and `memory.demote` retires it. `memory.definitions.list` (admin) lists the namespace's live shared definitions, the most recently promoted first and at most twice `memory_shared_definitions_max`, with `count` of them all. Each entry carries its key and `promoted_at`; one within the ceiling also carries its term and aliases, its `verdict` (`governed` or `lapsed`), the `lapse_reason` (`not_namespace_wide`, `no_term`, `gone`, `superseded`, `conflicted`, `duplicate`, `not_active` for any other status, `expired`, `ended` past its `valid_to`, `changed` or `sensitive`) and, for a governed one, `lost`: each name it loses to another governed definition that holds it, as `name` and the `winner`'s key (the latest promotion wins, a tie settled by its promotion event). One past the ceiling is `capped`, and its source is not read. Why two layers: agents write freely and often redundantly under their own identity, while a curated fact the whole namespace should rely on needs one canonical key, an owner's decision and an audit trail. Promotion is that decision. ## Identity modes How a caller gets a token at all is the deployment's identity mode (`LOOM_IDENTITY_PROVIDER`): - **`single-user`**: no authentication; a call that names no namespace resolves to one (`LOOM_DEFAULT_NAMESPACE`, default `local`, in the org `LOOM_SELF_SIGNUP_ORG`, default `dev`) on MCP for every op but `code.*`. Over HTTP, a call with no token gets the org on every op but `code.*` and `tracker.*`, and the namespace on the everyday memory and search ops; which ops, and the agent each gets, are in the [HTTP reference](https://clickhouse.github.io/loom/reference/http-api.md#request). An `all_mine` search also reads the namespaces the user registered. For a laptop only. - **`trust-network`**: self-signup with email and password; the first principal to claim a namespace name owns it. An agent can sign itself up (`loom init --agent`) and a person claims it later with the code it printed, which works once and is replaced by the person's own password. - **`gateway`**: an upstream identity layer authenticates people and mints short-lived Loom tokens with the shared secret; Loom only verifies. A subject in `LOOM_TRUSTED_CALLERS` may act for an end user with `X-Loom-On-Behalf-Of`, and every ledger row records the real caller. A service whose users have no Loom account signs a short-lived token per request for a `user:` instead, with a key registered for it as an end-user issuer ([act for a service's end users](https://clickhouse.github.io/loom/guides/tokens-and-grants.md#act-for-a-services-end-users)). ## Collections A collection is a namespace named `col-…` that only grants open: no token claim or org admin role reaches it. A collection can sit inside another and inherit its grants, and a grant on one can name a `user:`, `group:` or `agent:` principal, or `everyone` (every request acting in the org), besides a person or a service account. People name a collection by its title, and the server derives its `col-…` name from it; the name stays its address for good. [Sharing knowledge](https://clickhouse.github.io/loom/architecture/sharing.md) describes the model and how a name is derived. People see a collection as a collection, in the Library, by its title and its place there; the namespace that stores it is what grants, search scope, audit and erasure key on, and it never appears in the dashboard's namespace picker. A namespace named `acl-` and 24 hex digits is a mirror: it holds the documents of a document source that follows its store's permissions, each shown only to a reader the store admits ([document sources](https://clickhouse.github.io/loom/architecture/document-sources.md#the-mirror)). No token reaches one (only a platform admin's erasure names one), so `namespace.register` and `namespace.restore` refuse the form. When `LOOM_IDENTITY_PROVIDER` is empty the server infers the mode: auth off gives `single-user`, auth on with `LOOM_SELF_SIGNUP=open` gives `trust-network`, auth on otherwise gives `gateway`. A non-loopback bind with sign-up open also needs `trust-network` set explicitly. [Deployment](https://clickhouse.github.io/loom/operations/deployment.md#identity-modes) lists each mode's settings. ## Related - [Tokens and grants](https://clickhouse.github.io/loom/guides/tokens-and-grants.md): minting, claims, grants, restricted namespaces, history policy and revocation, step by step. - [Namespaces and sharing](https://clickhouse.github.io/loom/features/namespaces-and-sharing.md): the namespace operations and the dashboard's sharing views. - [Security](https://clickhouse.github.io/loom/architecture/security.md): how authentication and authorization are enforced. - [Memory](https://clickhouse.github.io/loom/concepts/memory.md): the row the tuple addresses. --- # Memory > Source: https://clickhouse.github.io/loom/concepts/memory/ (Markdown: https://clickhouse.github.io/loom/concepts/memory.md) A memory is one claim an agent should find again: "deploys happen on Tuesdays", "the payments team owns the billing service". Loom stores each claim as a row in `loom.memory`, under a readable key, with who wrote it, when it was true, where it came from and whether anyone vouched for it. This page covers the row, how a write is judged against what is stored, and how rows change over time. [Retrieval and ranking](https://clickhouse.github.io/loom/concepts/retrieval-and-ranking.md) covers how rows are found. ## Rows in ClickHouse Memories, document chunks, conversation turns, entity edges, code symbols, runs, audit entries and erasure requests are all rows in ClickHouse, created by numbered migrations the server applies on boot. Under the [Postgres memory authority](https://clickhouse.github.io/loom/architecture/storage-authority.md#why-two-stores), Postgres holds live memory and knowledge state, and ClickHouse keeps the vectors, search and history. Uploaded document originals are ClickHouse rows too (`loom.document_blobs`), unless the deployment keeps their bytes in an S3-compatible bucket. Outside ClickHouse, Postgres and that bucket, Loom keeps the code index's git mirrors, a cache of the remotes ClickHouse records, and the rate-limit buckets on a shared volume. - **The identity tuple leads the sort keys.** `loom.memory` is ordered by `(org, namespace, agent, user_id, session_id, memory_key)`, so scoping a read to a namespace is a primary-key range; a user narrows it fully only under a named agent, a session only under a named agent and user. [Identities and namespaces](https://clickhouse.github.io/loom/concepts/identities-and-namespaces.md) covers the tuple. - **One engine.** Under the default ClickHouse authority, vector search, lexical search, the graph walk, filters, audit and erasure are SQL over the same rows, with no second store to keep in step. The cost is that every read is a query, not a key-value hit. Under the Postgres authority, point reads (by key or id, outside an `as_of` replay) are answered from Postgres. - **A write is a new version.** The memory tables are `ReplacingMergeTree(version)` read with `FINAL`. A revision, tombstone, pin or restore inserts a new version row. ## The write, derive, read loop ```mermaid flowchart LR W["Writer: memory.set,
set_from_messages,
knowledge.add"] --> G["Write checks:
lock, pin, blacklist,
precondition, conflict"] G --> T[("ClickHouse rows:
memory, embeddings,
chunks, audit_log")] T --> D["Workers derive:
entities, deferred extraction,
consolidation, calibration,
usage folds"] D --> T R["Reader: memory.search,
memory.ask, context.pack"] --> L["Retrieval legs:
vector, lexical, graph,
chunks, digests"] T --> L L --> K["Fuse and rank,
gate or abstain"] K --> A["Answer with provenance"] A --> F["run.complete,
memory.outcome"] F --> T ``` 1. **Write.** `memory.set` runs its checks ([below](https://clickhouse.github.io/loom/concepts/memory.md#how-a-write-is-judged)), embeds the value inline, and inserts the memory row, its embedding row, verbatim chunks for long values and a hash-only audit row. If the inline embed fails, the call fails and nothing is written. `memory.set_from_messages` has the LLM pick the durable facts in a transcript and writes through the same bulk path. 2. **Derive.** Workers add what is too slow for the write path: entity tags (`entity_extractor`), deferred extraction (`extraction_queue`), consolidated facts, gate recalibration, and usage folded into ranking signals (`usage_fold`). Derived rows carry `source_class=derived`. 3. **Read.** `memory.search` runs several legs (vector, lexical, entity graph, verbatim chunks, session digests), fuses them, ranks, and returns hits or abstains. `memory.ask` writes a grounded answer; `context.pack` builds a budgeted prompt block. 4. **Feed back.** A run that names the memories it relied on (`run.complete` `memories_used`, or `memory.outcome`) credits them; a bad outcome records harm. This becomes a bounded, decaying ranking term. It never deletes a row or ranks it below its relevance. The loop is eventually consistent. A read right after a write finds the fact through its inline embedding; entities and extracted facts arrive after the worker's next pass. ## The `loom.memory` row One row is one version of one memory. | Group | Columns | Decides | | ---------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------- | | Identity | `org`, `namespace`, `agent`, `user_id`, `session_id`, `run_id` | Who may read it and who wrote it | | Address | `memory_key`, `memory_id` | How a caller names it (key) and how Loom names one row (id) | | Content | `value`, `memory_type`, `tags`, `entities`, `files`, `confidence` | What it says and how it is classified | | Lifecycle | `status`, `pinned`, `conflict_with`, `duplicate_of`, `superseded_by`, `version` | Whether it is served, and what replaced it | | Time | `created_at`, `updated_at`, `valid_at`, `valid_to`, `invalid_at`, `expires_at` | When it was written, when it was true, when it left service | | Provenance | `actor`, `source_class`, `content_sha256`, `verified_by`, `verified_sha256` | Who wrote it, how Loom learned it, whether anyone attested it | | Guards | `lock_mode`, `lock_until`, `sensitivity`, `on_expire` | Which writes it refuses and how it is read | The full column list is in the [schema reference](https://clickhouse.github.io/loom/reference/schema.md). A searchable memory also has a row in `loom.memory_embeddings`: vectors for the value and the key, lowercased, stemmed and lemmatised copies of the excerpt for lexical matching, and the `source_class` and `verified` flags the ranker reads. ## Keys A key names one attribute of one subject, as a readable path `//`: ``` sre/case/1234/evidence/dns-sidecar project/loom/deploy-window user/preferences/editor ``` - **One fact per key.** Conflict checks, supersession and history work per key. A key holding a list of unrelated facts cannot be partly revised. - **A revision rewrites the same key.** Loom keeps every version, so a numbered copy (`…-v2`) or a key made from a hash of the value only hides the revision. Hash-made keys are the `hash_keys` finding of `namespace.audit`. - **Facets go in `tags` and `entities`,** not in the key. The storage identity is `(org, namespace, agent, user_id, session_id, memory_key)`: the same key written by two agents or in two sessions is two rows. Under the namespace policy `newest_wins`, the newest write for a key retires every older live row with that key in the namespace, whoever wrote it. Extraction mints keys from the subject and attribute, never from the value, so a later session that states the same attribute lands on the same key. Under `auto_reinforce` and `manual_review`, the conflict resolver judges the rows of one key that two sessions of the same user (not the anonymous user `-` or a user_id sent empty, whose sessions are separate tasks) wrote, from a point write or a bulk write, as it judges any close pair: it retires the older row of an update it is sure of (except under `manual_review`), flags a contradiction for review and leaves two different facts under a reused key live. Under `append_only` nothing is judged. ## Memory types `memory_type` is a free string, default `semantic`. A few values change how the row is stored and found (`server/memory/domain/memory_types.py`): | Type | Embedded | In unfiltered search | Use for | | ---------------------- | -------- | -------------------- | -------------------------------------------------------- | | `semantic` (default) | yes | yes | Durable facts. | | `preference` | yes | yes | Preferences that should rank in recall. | | `procedural` | yes | yes | Procedures (`procedure.*`). | | `reflection` | yes | no | Lessons from runs; read with `memory_type="reflection"`. | | `episodic` | no | no | Session events replayed by recency. | | `profile` | no | no | Always-loaded context fetched by label (`profile.*`). | | any other (`decision`) | yes | yes | Domain types are searchable by default. | - **Reflections are hidden** from unfiltered search and `memory.ask`: a lesson about a process is not a claim about the world. - **A `procedural` write** whose value is not a procedure body (a non-empty `situation` and at least two `steps`) is stored as `semantic`, tagged `retyped_from:procedural`, and the response says so in `retyped_from`. - **Sensitivity.** A row with `sensitivity="high"` is never embedded, so no search leg finds it. It is readable by key only, and that read is audited. With `sensitivity` unset the server detects credential shapes, card numbers, US Social Security numbers and stated PINs (`high`), and email, phone and IBAN shapes (`medium`) (`LOOM_MEMORY_SENSITIVITY_AUTODETECT`). ## Status | Status | Meaning | Served by default | | ------------ | -------------------------------------------------------------------------- | ------------------------- | | `active` | The live belief. | yes | | `conflicted` | Close to a peer but saying something different; linked by `conflict_with`. | yes | | `duplicate` | Near-identical to a canonical row (`duplicate_of`), which answers for it. | no (`include_duplicates`) | | `superseded` | A later row replaced it (`superseded_by`, retired at `invalid_at`). | no (`include_superseded`) | | `proposed` | A fact on trial, decided with `memory.proposal_decide`. | no (`include_proposed`) | | `rejected` | Marked wrong (`memory.reject`, or a rejected proposal). | no | | `tombstoned` | Deleted by its writer (`memory.delete`). | no | The served set is defined once, in `server/memory/domain/temporal_sql.py` (`SERVED_STATUSES`). A conflicted row stays served: hiding one side of a contradiction would let the store answer confidently with a fact it has reason to doubt. ## How a write is judged Five operations cover the memory lifecycle: | Lens | Operation | Effect | | --------- | --------------------- | -------------------------------------------------------------------------- | | Expand | `memory.set` | Learn a fact under a key. | | Revise | `memory.update_by_id` | Correct one row by id (`memory.update` does the same by key). | | Contract | `memory.delete` | Tombstone a key. `memory.restore` revives it within 14 days of the delete. | | Reinforce | `memory.pin` | Store and pin: no later write that does not itself pin replaces it. | | Refuse | `blacklist.add` | Refuse every future write matching a substring or regex. | A searchable `memory.set` runs these checks in order: 1. **Lock and pin guards** on the row under the key. `memory.lock` (`read_only` or `no_delete`) answers 423 `LOCKED`. A changed value under a pinned key is 409 `PINNED_CONFLICT`. A byte-identical restatement passes every guard. 2. **Blacklist.** A value matching a registered pattern is 422 `BLACKLISTED`; in a bulk write the entry is skipped. Replicas cache patterns and poll a one-row version table (`loom.blacklist_version`), so a new pattern reaches every replica within seconds. 3. **Precondition.** `if_match_sha256` makes the write land only if the live row's hash still matches; otherwise 409 `PRECONDITION_FAILED`. The current hash rides every `memory.get`, `memory.get_by_id` and `memory.set` response (`content_sha256`). 4. **Restatement.** A value byte-identical to the live row writes a new version that carries the live row's `reinforcement_count + 1`. The bulk path (`memory.bulk_set`, used by `set_from_messages`) treats any live row with the same hash in `(org, namespace, agent, user_id)` as the fact (`LOOM_MEMORY_BULK_DEDUP_SCOPE=identity`). 5. **Peer search and verdict.** Active rows of the same `memory_type` whose embedding is close to the new value are **peers**, compared closest first; a write that names its `user_id` takes them from that user's rows only ([how a write is judged](https://clickhouse.github.io/loom/features/memory/conflicts-and-refusal.md#how-a-write-is-judged)): - same canonical form (lowercased, punctuation stripped, whitespace collapsed) → `merge`: nothing written, the peer is reinforced, `merged_into_key` names it; - a close pinned peer → `rejected`, 409 `PINNED_CONFLICT` (under a namespace-wide token, a write that names its user and restates another user's or a shared pinned value is written); - under `newest_wins`, a peer that states the same subject and predicate with a different object → `refines`: the peer is retired, unless its `valid_at` is later than the new row's, which answers `conflict`; - cosine at or above the duplicate threshold with the same content words and no differing value (a number, time, amount or unit; "27:10" / "26:30") → `duplicate`; - cosine at or above the conflict threshold but different → `conflict`, both rows `conflicted`; - no close peer → `clear`. 6. **Insert** into `loom.memory` and `loom.memory_embeddings`, supersession, entity projection and one hash-only row in `loom.audit_log`. A non-searchable type or a `high`-sensitivity row skips the peer search: it has no embedding to compare. `dry_run=true` runs every check and writes nothing, which answers "new fact, restatement or contradiction?" before you commit to a key. The response carries `conflict_verdict`, up to five `peers`, and a `next` hint. - **Thresholds.** With `LOOM_ADAPTIVE_THRESHOLDS` on and a real embedder, the conflict and duplicate thresholds come from the embedder's own similarity distribution at boot (`server/memory/app/embedder_calibration.py`). The scalars `LOOM_CONFLICT_THRESHOLD` (0.85) and `LOOM_DUPLICATE_THRESHOLD` (0.97) are the fallback. A namespace or memory type can pin its own; `calibration.get` reports which is in force. - **The conflict resolver.** Cosine says two statements are about the same thing, not whether they agree. A leader-only worker judges each write with a close peer after it lands, one LLM call per row: it retires the older row of an update it is sure of under the default `auto_reinforce` and under `newest_wins` (the old value stays readable with `include_superseded` and `as_of`), flags a contradiction, and clears a false flag ([conflicts](https://clickhouse.github.io/loom/features/memory/conflicts-and-refusal.md#the-conflict-resolver)). No verdict on the write itself calls an LLM. - **Run scope.** A write under a `run_id` whose canonical match belongs to another run is stored as a `duplicate`, not merged, so a run-scoped read still finds it. - **Namespace policy.** `default_conflict_policy` decides what a contradiction does (`auto_reinforce`, `manual_review`, `newest_wins`, `append_only`). `memory.update`, `memory.update_by_id` and `memory.bulk_set` run the same check. Verdict details, the policies, the Review queue and `conflicts.resolve` are in [conflicts and refusal](https://clickhouse.github.io/loom/features/memory/conflicts-and-refusal.md). **Pins protect what the user said.** A pinned row is never replaced by a write that does not itself pin, by key or by a close value under another key. To change it, use `memory.update` (keeps the pin) or `memory.pin` (re-pins). `memory.for_file` lists pinned rows first; no other read orders by the pin. **Consolidation keeps lineage.** `memory.consolidate` (and the consolidation worker, when a namespace opts in) merges fragments about one subject into canonical rows with `source_class='derived'`, and writes one `loom.memory_belief_sources` row per source (`support_type='derived_from'`). Deleting a source retires a consolidated row when none of its sources survive, and downgrades it to `conflicted` when some do (`dependents_tombstoned`, `dependents_downgraded` in the `memory.delete` response). ### A memory's lifecycle ```mermaid stateDiagram-v2 [*] --> active: memory.set (clear, refines) [*] --> conflicted: memory.set (conflict) [*] --> duplicate: memory.set (duplicate) [*] --> proposed: memory.set(proposed=true) active --> conflicted: a later close write conflicted --> active: conflicts.resolve (revise, allow) conflicted --> tombstoned: conflicts.resolve (contract, or peer revised) proposed --> active: memory.proposal_decide (accept) proposed --> conflicted: memory.proposal_decide (accept, contradicted on trial) proposed --> rejected: memory.proposal_decide (reject) active --> rejected: memory.reject active --> superseded: newer write (newest_wins, refines, conflict resolver) superseded --> active: memory.restore active --> tombstoned: memory.delete, blacklist sweep, expiry tombstoned --> active: memory.restore (within 14 days) ``` Every arrow is a new version; the old one stays in `loom.memory_versions`. ## Tags, entities and files - `tags` are free labels (`case:1234`, `kind:evidence`). A search filters on them (`tags`), boosts with them (`tags_soft`), or sinks them (`demote_tags`). A namespace's `tag_registry` can attach retention, dedup and conflict policy to a tag. Tags keep the case they were written in: `memory.list` and `knowledge.list` match them in any case, and every other tag filter, `memory.search` and `memory.apply` included, matches them exactly. - `entities` are the named things the value mentions, stored in the canonical form search reads them in (lowercased; articles, possessives and punctuation dropped; the head noun singular), so `Billing Services` and `billing service` are one entity. A writer may pass them; otherwise the `entity_extractor` worker fills them. They drive the entity boost and the [entity graph](https://clickhouse.github.io/loom/features/entity-graph.md). - `files` are repository paths the memory is about; `memory.for_file` serves them before an edit of those files. `memory.for_file` renders one line per memory. A decision or a constraint prints whole, up to the ceiling below, and does not count against the request's `token_budget` (300 tokens by default); the other rows are held to it, cut to one line of 220 characters. A row that does not fit is left out whole while the rows after it still get their turn, and a closing `- … more` line, which counts against the budget too, says how many were left out and how to list them. The header and the removal footer are always printed, and the `… more` line whenever a row is left out, so a budget smaller than those can be exceeded. Decisions and constraints share one ceiling, `memory_for_file_whole_tokens` (1,500 tokens, about 6,000 characters), so a file governed by dozens of them cannot flood the prompt. They print in the block's order; one that would pass the ceiling is left out, and the ones after it that still fit print. A closing line counts those left out and names the first five keys, each read with `memory.get` (in the namespace it names, for a team row) or found with `loom_search`. A row left out with the newer row that supersedes it is counted on the `… more` line instead, never named as one to read. The default keeps the whole block under the 10,000 characters a coding-agent host passes from a hook to the model as it is. Each line: - A memory from an extra namespace (the repository's code namespace) is headed `Team rule (by , )`, or `Decision (by , )` for a decision (`memory_type` `decision` or the tag `decision`). - A line lists its `constraint`, `gotcha` and `decision` kinds in one bracket (`[constraint, decision]`; a line headed `Decision` leaves out `decision`). - `(pinned)` marks a pinned row. A pin is the row's own flag: a member of a code namespace can set it on a row they write there, so `(pinned)` on a team row says someone froze it, not that a maintainer did. - A decision nobody pinned, such as one `loom decisions extract` wrote, carries no pin mark: it is served as it is. - `(outcome: failed)` marks any decision whose recorded outcome failed, pinned or not. - A decision tagged `source-changed` (its source document no longer says it; see [extracting decisions](https://clickhouse.github.io/loom/features/tasks-and-decisions.md#extract-decisions-from-a-document)) reads `(source changed)`, pinned or not. - A note says when its linked code changed or went. - A string value is printed as text, not as the JSON it is stored as, and a row other than a decision or constraint ends in `…` when the read or the line cut it; a cut object or list prints as stored. - A decision with a source carries ` — source: ` after its value, whole, so a `knowledge:` citation keeps its `#section` and resolves through `knowledge.get`. - A newer row the conflict check linked to an older one of its type is placed above it, and a team line says which it supersedes (`supersedes decision` when the older is tagged `decision`). Two typed decisions the check linked each keep their place in the block, with no such note, since each decision key is its own decision. When the budget or the ceiling leaves out any row that supersedes others, in any namespace, every row it supersedes is left out with it, so an older row never reads as current. ## Memories linked to code What this gives a coding agent, and where to start: [memory linked to code](https://clickhouse.github.io/loom/use-cases/memory-linked-to-code.md). A memory about code is linked to the definitions and files it names, and the links follow the code. The `memory_code_links` worker links each memory tagged `repo:` (the coding plugins tag their sessions) or written in a repository's code namespace against that repository's trunk: - each backticked name, `A::b` and `a.b()` call that resolves to exactly one definition; the memory's `files` pick between namesakes, and a name defined more than once is not linked; - a bare camelCase or snake_case word of the prose only to a definition in the memory's `files`; - each file the memory names that holds none of those definitions. A qualified name trunk does not define yet waits for `memory_code_links_pending_days` (14), so a memory about unmerged code links once it merges. A link records the definition's fingerprint: all its overloads and the conditions that select them, as tokens, so whitespace, comments (not a Go `//go:` directive or a Python `# type:` comment) and most of what formatters change (trailing commas and other optional separators, a JavaScript `;`, quote style and the escapes it needs, the case of a number literal, a Python docstring's indentation, parentheses in Python, JavaScript and TypeScript) leave it as it was. Joining adjacent string literals, and any other edit, does not. What a file defines is read from the file at the repository's indexed head in its bare git mirror. On every indexed commit the worker follows the commit's renames and edits: | Link state | When | | ------------ | ------------------------------------------------------------------------------------------------------------------ | | `fresh` | The definition is where it was, with the text the memory was written against. | | `moved` | Its file moved, or it was renamed with nothing else changed, or it moved to another file. | | `changed` | Its text changed after the memory was written (`changed_since` names the commit), renamed or not. | | `unresolved` | Its name is gone and a new definition in its place may be it, but none certainly is. Served, with a review signal. | | `missing` | The code is gone: nothing new took its place. The link keeps its place, so a revert finds it again. | ![A memory's detail in the dashboard: its linked code, one definition current and one changed since a commit](https://clickhouse.github.io/loom/assets/screenshots/memory-linked-code.webp) A memory's detail in the dashboard lists these links with their state ([clip](https://clickhouse.github.io/loom/assets/screenshots/clip-memory-linked-code.webm)). A link follows its definition through file moves, renames (with an edit, when the new name is related to the old), and its class's rename within the file. A namesake in another class or namespace never inherits a link, and a `missing` link comes back only when its own name does. A name gone from its file is looked for in the files changed since its last check as the mirror holds them, beside the index, so a definition moved by a commit the index has not taken in whole (past `code_ingest_max_files_per_commit`) is followed when the link is rechecked. When `memory_code_links_mirror_naming_files` or more of those files name it, or more than `memory_code_links_mirror_changed_files` changed under the entry's prefixes, or the diff cannot be read, or the paths are more than git takes in one command or searches in its time, the mirror adds nothing and the index decides. When git cannot run the search, or a definition it finds cannot be fingerprinted, a live link the index's namesakes do not settle is `unresolved`. Files the ingest would not index (outside the entry's `path_prefixes`, refused by its denylist, over the byte cap, binary) add no definition; a generated file, which the ingest declares without searching its text, does. A file the parser fails on keeps its links as they were until it parses. When the fingerprint rules change, every link is re-hashed from the commits its hashes were taken at. When its old text is gone from git, a link that matched it takes the text the next check finds and a changed one stays `changed`; one whose old text no longer parses keeps its last state, and the worker reports it. Loom never writes the memory. The namespace's `code_change_policy` (`namespace.register`) decides what reads make of a change: | Policy | Removed code | Changed code | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `retire_removed` (default) | When every link of a memory is `missing` (a pending or `unresolved` one counts as not), search (and `memory.ask`, `context.pack`) leaves it out before it picks its results; `memory.get` still shows it. It is back when the code is. | Flagged. | | `flag` | Flagged. | Flagged. | | `revise` | As `retire_removed`. | The claim is re-checked against the old and new code by the LLM: one that holds is re-anchored, a contradicted one gets a corrected claim as the link's `suggestion`. | Under every policy, a memory that a search still serves ranks lower when its worst link is `changed`, `unresolved` or `missing`: it moves below the next `memory_search_code_changed_places` (3) rows that rank by score (a row also moving down, or already moved, is not counted), never out of the results nor below a row served only as filler (under the similarity floor, or flagged below the abstain threshold), after the confidence and abstain checks have read them in relevance order; `score_components.code_changed_places` says how far among its own search's rows (`loom_search` places its merged repository decisions by similarity among the rows its search did not rank down). These are never moved down: - a memory a person holds; - a memory whose only such links are into code the caller may not read, or are links nothing keeps current (`memory_code_links_enabled` off); - a row placed by an identifier match or by an `anchor_first` or `recent_first` tier, which a move does not count as a place either; - every row a search serves for a query whose answer is a date order (`loom_search`'s merged repository decisions, placed by similarity, still move); - the rows of a later page (`page=`), served in their parked order; - a chunk, record or summary hit, which carries no `code_state`. A reader block (`render="reader"`, `loom_search`'s block) whose question carries a time word (`when`, `after`, `since`, `latest`, …) lists its rows by date, so the move does not show in it; `results` keeps the move. `memory.ask` re-sorts the search's rows by similarity and recency while `memory_ask_recency_weight` is above 0 (0.2 by default), and its identifier lookups place their own rows; a `context.pack` startup read with no query does not search. A move does not show in any of these. `revise` judges a changed link once per edit, within a per-pass budget (`memory_code_links_revise_per_pass`). A memory labelled `high` sensitivity is never sent to the model. A memory a person pinned, locked (until the lock lapses) or verified, and one naming more code than the link cap, is only flagged. Links made for an earlier value of a memory (its text, files, repository or user) are ignored by every read until it is relinked. Links are kept against the trunk. On another branch, a lesson compares the branch's own definition, under any of the link's last 16 names, with the text the memory was written against: a release branch that still has the old text reads `fresh` after trunk changed it, and one that has code trunk removed still gets its lesson. Where the branch's files cannot be read (a host with `LOOM_GIT_HISTORY=none` has no git mirror), a lesson carries trunk's last state. The links are read where the code is: - `code.symbol` attaches the linked memories as `lessons`, each with its `code_state`. They come from the repository's team namespace and the `lesson_namespaces` the call names (over MCP the caller's memory home when it names none), never from every namespace the caller can read. - `code.search`, `code.callers`, `code.callees`, `code.impact`, `code.outline` and a task's `code.context_pack` attach `lessons` for the answer's top files (`code_result_lessons_paths`): from the same namespaces, less any other repository's code namespace, the memories linked to code in each file or naming it by its path or a tail of it (a bare filename, `search/handler.py`; an absolute path is trusted to name a file below the root that it ends in), each with its `code_state`, within `code_result_lessons_tokens`. Each row carries `path`, `memory_key`, `namespace`, `excerpt`, `code_state`, `changed_since` (the commit that put its code in a changed, unresolved or missing state) when one is known, and `supersedes_key` when it replaces an older row; a row another row read for the answer supersedes is not served. `lessons_left_out` counts the rows the cap, a cut or that rule left out. A decision or constraint is served whole or left out. The read runs after the answer, never enters the search memo, and is skipped for `include_lessons: false`, with `code_result_lessons` off, or for a token narrowed to its own user or session. - `memory.for_file` serves memories linked to code in the file beside those whose `files` name it. A line whose code changed, was not found again or went says so and names the first indexed commit at which it read so; `context.pack` decision lines carry the same note, and its memory lines a `code_state=@` label. - `memory.get` returns the links as `code` (with any `suggestion`); `memory.search` hits read from a memory's own row carry its worst link as `code_state` (a chunk, record or summary hit carries none and is never left out); `memory.ask` notes the memories whose code moved, changed or went. - `namespace.validate` lists memories whose code changed, was not found again or is gone, in the repositories the caller may read. ## Time | Column | Axis | Meaning | | ----------------- | ------ | --------------------------------------------------------------------------------------- | | `valid_at` | world | When the fact became true, as the writer stated it. `NULL` = unknown. | | `valid_to` | world | When the fact stopped being true. A read drops a row past its `valid_to`. | | `temporal_anchor` | world | The date the fact is about (an event date), set by extraction. | | `created_at` | record | When Loom wrote the row. | | `invalid_at` | record | When Loom retired the row (supersession). | | `tombstoned_at` | record | When the writer deleted it. | | `expires_at` | record | When the row leaves the served set, if its writer or the namespace owner set an expiry. | An `as_of` read replays one axis. `as_of_basis="valid_time"` (the default) asks what was true at T: rows whose best world date (`valid_at`, else `temporal_anchor`, else `created_at`) is at or before T. `as_of_basis="knowledge_time"` asks what Loom had learned by T: rows written by T and not retired, deleted or invalidated by T. Under `newest_wins`, a templated substitution retires the peer only if the peer's `valid_at` is not later than the new row's, so a backfill of old facts does not retire current ones. This is valid time plus a retirement stamp, not a bi-temporal model: supersession, not exclusion by date, decides which of two rows is current. A full interval model would need every writer to state when each fact ended, which agents rarely know. ## Versions and history Every change (value edit, tombstone, restore, lock, status flip) is a new row with a higher nanosecond `version` and every other column carried forward. Merges drop old versions, so `loom.memory` is the current state, not a history. The history is `loom.memory_versions`, an append-only mirror filled by a materialized view on every insert. `memory.history` reads every version of a key; `namespace.rollback` returns every head in a namespace to its version at a point in time; `memory.redact` scrubs a historical value and keeps its audit entry. `content_sha256` on each version is what `if_match_sha256` compares and what an attestation binds to. ## Provenance The server stamps two columns on every write and never takes them from the body: - `actor`: `human:` or `process:` (a service account), from the token. - `source_class`: how Loom came to hold the content, from the operation and the token. | `source_class` | Means | | -------------- | ----------------------------------------------------------------------------- | | `asserted` | A caller stated it directly (`memory.set`). | | `extracted` | The extraction pipeline inferred it from a transcript. | | `observed` | Loom recorded it from activity it saw. | | `derived` | Computed from other memories (consolidation output, session digests). | | `imported` | Loaded from another system by an importer. | | `external` | Declared by the writer as outside content: a tool result, a page, a document. | A request may only declare a downgrade (`derived`, `external` or `imported`); anything else is a 422. Text an agent copies from its environment is what an attacker controls, so `memory.ask` keeps unattested `external` rows out of its synthesis (`LOOM_MEMORY_ASK_EXCLUDE_UNVERIFIED_EXTERNAL`, default on), and `LOOM_MEMORY_SEARCH_TRUST_WEIGHT` can lift asserted and attested rows in ranking. - **Lineage.** Consolidation writes `loom.memory_belief_sources` rows; a session digest lists the keys it summarised in `session_summaries.derived_from`. `memory.provenance` says where a row came from, `memory.evidence` returns the conversation turns an extracted fact was drawn from, and `memory.lineage` walks how a shared memory was promoted. - **Attestation.** `memory.verify` records who vouched for the content, bound to its hash. Reads report `verification`: `human` or `process` while the content is unchanged, `stale` once it changes, `none` if never attested. An edit keeps the attestation and reads as `stale`; it never transfers trust to new content. ## What goes where | It is… | Plane | Why not memory | | ---------------------------------------------------------- | --------------------------------------------------- | --------------------------------------------------------------------------------------------------- | | A fact still true next week that another session needs | **Durable memory**, `memory.set` | It is memory. | | A transcript the user wants remembered | `memory.set_from_messages` (extraction) | One large row is findable only as a whole; extraction writes one short fact per key. | | The current plan, a lease, a cursor, "where was I" | **Working memory**, `working_memory.*` | Coordination state changes every step; ranking it pollutes recall. | | A large tool result you may re-read | `context.offload` / `context.fetch` | It is a blob to fetch by key, not a claim to rank. | | The resumable state of one line of work | **Session briefs**, `session.brief_*` | A brief is restored by choice, never ranked into another task's answer. | | A document, spec or runbook; anything over a few kilobytes | **Knowledge**, `knowledge.add` / `knowledge.upsert` | Documents are chunked by heading; a memory is embedded from an excerpt of at most 1,200 characters. | | A chat turn log | **Conversations**, `conversation.*` | Turns are evidence, not beliefs; `memory.recall` reads the turns an extraction used. | | Data with identity and exact fields (a PR, a ticket) | **Records**, `memory.record.*` | Records fold field by field and filter exactly. | | Always-loaded persona or preferences | **Profile**, `profile.*` | Fetched by label every time; ranking it against questions wastes the slot. | For documents: store the document in knowledge and write one short memory under a readable key that says what it is and points at it. `context.pack` loads working memory ahead of the durable sections, so an agent sees its own state first. ## Retention Loom deletes only what its writer asked for. No memory table has a default TTL and no sweep retires rows by age. A row leaves the served set when its writer deletes or supersedes it, when its writer gave it an expiry (`ttl_seconds`, `valid_to`), or when the namespace owner or operator configured a retention window. `memory.restore` revives a tombstone within 14 days. [Retention and erasure](https://clickhouse.github.io/loom/features/memory/retention-and-erasure.md) covers the settings and the erasure saga. ## One fact over time In March a user says "I live in Delhi"; in June, "I moved to Berlin." - **June write.** Extraction mints the key from the subject and attribute, so both statements land on the same key. The June row does not overwrite March: the Delhi version stays in `loom.memory_versions`, readable with `memory.history`. - **A misfiled fact.** If Berlin lands under another key, the peer search finds the Delhi row close but different and marks both `conflicted`, linked by `conflict_with`. The namespace policy decides what follows (see [how a write is judged](https://clickhouse.github.io/loom/concepts/memory.md#how-a-write-is-judged)). - **"Where did they live in April?"** An `as_of` read answers it, by when the fact was true or by what Loom knew at that date ([time](https://clickhouse.github.io/loom/concepts/memory.md#time)). - **"Who said so?"** `actor` and `source_class` record the writer and how the fact arrived; `memory.evidence` returns the turns an extracted fact came from. - **A bad import.** `namespace.rollback` returns every head in the namespace to its version at a point in time, and runs as a dry run unless told otherwise. ## Example Write a fact, probe a change before writing it, then update against the hash you read: ```python import asyncio import os from datetime import datetime, timezone from loom import Datapack async def main() -> None: dp = Datapack( org="acme", namespace="eng-tooling", base_url=os.environ["LOOM_BASE_URL"], api_key=os.environ["LOOM_API_KEY"], ) agent = dp.agent("release-bot") key = "project/billing/deploy-window" out = await agent.memory.set( memory_key=key, value="Billing deploys only on Tuesdays and Thursdays, never on Fridays.", tags=["kind:constraint", "service:billing"], entities=["billing"], valid_at=datetime.now(timezone.utc), ) print(out["conflict_verdict"], out["content_sha256"]) probe = await agent.memory.set( memory_key="project/billing/deploy-days", value="Billing deploys only on Thursdays.", dry_run=True, ) if probe["conflict_verdict"] == "conflict": peer = probe["peers"][0] # closest peer, with its key current = await agent.memory.get(memory_key=peer["memory_key"]) await agent.memory.update( memory_key=peer["memory_key"], value="Billing deploys only on Thursdays.", if_match_sha256=current["content_sha256"], ) asyncio.run(main()) ``` A 409 `PRECONDITION_FAILED` on the update means another writer changed the row after the read: read again and decide again. Over HTTP the write is `POST /v1/memory.set`; over MCP it is the `loom_memory_set` tool. ## Related - [Identities and namespaces](https://clickhouse.github.io/loom/concepts/identities-and-namespaces.md), [retrieval and ranking](https://clickhouse.github.io/loom/concepts/retrieval-and-ranking.md) - [Writing memories](https://clickhouse.github.io/loom/features/memory/writing.md), [conflicts and refusal](https://clickhouse.github.io/loom/features/memory/conflicts-and-refusal.md), [history and provenance](https://clickhouse.github.io/loom/features/memory/history-and-provenance.md), [working memory and profiles](https://clickhouse.github.io/loom/features/memory/working-memory-and-profiles.md), [knowledge](https://clickhouse.github.io/loom/features/knowledge.md) - [Memory architecture](https://clickhouse.github.io/loom/architecture/memory-architecture.md): the write path in code - [Operations](https://clickhouse.github.io/loom/reference/operations.md#memory), [settings](https://clickhouse.github.io/loom/reference/configuration.md#memory) --- # Retrieval and ranking > Source: https://clickhouse.github.io/loom/concepts/retrieval-and-ranking/ (Markdown: https://clickhouse.github.io/loom/concepts/retrieval-and-ranking.md) This page explains how `memory.search` finds rows. It runs several retrieval legs over the namespace at once, fuses their candidates into one pool, ranks the pool by relevance plus bounded correction terms, and then decides whether the best hit is good enough to answer from. A weak result is reported as **abstain** ("Loom does not know"), not as the nearest noise. Request and response fields are in [search and ask](https://clickhouse.github.io/loom/features/memory/search-and-ask.md). ![The dashboard's Recall view: each hit's similarity, entity boost and score](https://clickhouse.github.io/loom/images/dashboard-recall.png) ```mermaid flowchart LR Q["query"] --> P["plan: mode, budget, HyDE"] P --> F["fact leg: value + key vectors"] P --> L["lexical leg: tokens, stems, BM25"] P --> C["verbatim chunks"] P --> R["records"] P --> S["side legs: summaries, entity boost"] P --> G["graph: entity edges, triples"] F --> U["fused pool"] L --> U C --> U R --> U S --> U G --> U U --> K["rank: relevance + bounded terms"] K --> X["optional reranker"] X --> V{"top hit above abstain threshold?"} V -- yes --> A["hits"] V -- no --> B["abstain + abstain_code"] ``` ## The retrieval legs Each leg answers a failure of the others. A leg that has nothing to add returns nothing; a leg that errors fails open and the response names it in `degraded`. | Leg | Reads | Finds what the others miss | | --------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | **Fact, vector** | `loom.memory_embeddings` value vector (no vector index) | Paraphrases: the question and the stored claim share meaning, not words. | | **Fact, key axis** | The same table's `memory_key` vector | A question that names the attribute a key encodes. Scores `least(value distance, key distance)`. | | **Lexical** | Lowercased, stemmed and lemmatised excerpts and the key | Exact identifiers, numbers and rare words that embed poorly. Stemming runs in ClickHouse (`stem`). | | **BM25** | Per-token IDF and corpus length roll-ups | Rare-term weighting for `bm25_hybrid`, with a live tail so new rows count at once. | | **Verbatim chunks** | `loom.memory_chunks` | A passage of a long `memory.set` value folds onto the row whose value cuts it; a transcript's is a hit. | | **Session digests** | `loom.session_summaries` | "What happened in that session"; with digest descent the top digests pull their sessions' best facts in. | | **Entity** | `loom.entity_vectors`, `loom.entity_summaries` | Rows about an entity the query names, even when the wording differs. | | **Graph** | `loom.entity_edges`, `loom.entity_triples` | Rows one hop from the query's entities, and rows carrying a matching (subject, relation, object) triple. | | **Records** | `loom.memory_records` | Typed records whose fields match. | | **Key match** | Stemmed key parts | A row whose key names exactly the attribute asked for, as a tie-break (`LOOM_MEMORY_SEARCH_KEY_MATCH_MODE`). | | **Computed evidence** | `loom.derived_facts`, event dates | Aggregates ("how many"), date differences and the latest value of an attribute, computed in ClickHouse. | **Why so many legs.** A single vector leg misses identifiers and numbers (they embed poorly), passages deep in a long value (only the first part is embedded), and rows that name an entity in other words. Each leg is a query over a table Loom already writes, so adding one costs a statement, not a second system. **The vector legs scan, with no vector index.** The scan is exact, or, where [codes-first reads](https://clickhouse.github.io/loom/architecture/clickhouse.md#vector-storage) apply, a pass over quantized codes keeps a candidate pool and rescores it at full precision. **The key axis** is on per namespace when its keys are reused (`LOOM_MEMORY_SEARCH_KEY_AXIS=auto`, threshold `LOOM_MEMORY_SEARCH_KEY_AXIS_AUTO_REUSE_SHARE`): reading two vector columns costs roughly twice the bytes, and it only helps where keys carry meaning. **Planning and budgets.** `retrieval_budget` (`fast`, `standard`, `deep`) decides which expensive legs run. Under `standard`, cheap recall runs first and the LLM legs (HyDE, query decomposition, the iterative retry) spend only when the first pass is weak, bounded by `LOOM_MEMORY_SEARCH_FALLBACK_MAX_ROUNDS` and `LOOM_MEMORY_SEARCH_FALLBACK_MAX_WALL_MS`. Under `fast` (or the `fast` recipe) a read makes no LLM call; it costs embeddings and ClickHouse reads. HyDE embeds an LLM-written hypothetical answer instead of the question, which closes the gap between how questions and stored claims are phrased; `hyde=false` forces the raw query. ## Fusion `search_mode` picks how the fact pool is ordered. - A request that sets neither `search_mode` nor `recipe` gets the namespace's `retrieval_defaults` mode, else `LOOM_MEMORY_SEARCH_DEFAULT_MODE` (`semantic`). The wire schema shows `rrf`; the server fills the real default. - The planner's shape rules apply on top: an identifier in the query takes the lexical fusion. - `recipe` applies a named preset (`fast`, `balanced`, `quality`, `verbatim`, `code`, `temporal`, `graph`) and wins over both defaults. | Mode | Ordering | | ------------- | -------------------------------------------------------------------------------------- | | `semantic` | Cosine similarity plus the rank terms below. | | `rrf` | Reciprocal rank fusion of the lexical and semantic orders, `1/(k + rank)` with k = 60. | | `hybrid` | `alpha × lexical + (1 − alpha) × cosine`, plus the rank terms. | | `bm25_hybrid` | IDF-weighted BM25 fused with cosine. | | `lexical` | Token overlap on content and key; no vector order. | | `graph` | Cosine plus entity-graph signal: one-hop overlap and matched triples. | | `temporal` | Rows dated within a window of the question's resolved date, cosine within the window. | | `related` | Tag and entity overlap with the request's `tags` / `entity_filter`. | **Why reciprocal rank fusion exists as a mode.** Lexical overlap scores spread over 0 to 1, while cosines from a real embedder sit in a narrow band; a weighted sum of the two under-weights one of them unless alpha is tuned per corpus. RRF combines ranks, not scores, so it needs no tuning. Loom also lowers the lexical weight for code- or path-shaped queries, whose shared path tokens are near-noise (`server/memory/app/retrieval/fusion.py`, `_rrf_lexical_weight`). Ties in one order break by the other, then by key and the row's namespace, writer and session, so two identical searches return the same window. ## The rank expression For the cosine-ordered modes the score of one fact row is relevance plus terms that may reorder near-ties but are bounded so they cannot outrank relevance (`server/memory/app/retrieval/fact_rank.py`): | Term | What it adds | Controlled by | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | Similarity | `1 − cosine distance` of the best vector. | the embedder | | Entity boost | Up to 0.5 × the share of `entity_filter` entities the row carries; entity-vector matches from the query. | `entity_filter` | | Recency, chronology, date | A lift on a recency-intent ("latest", "current") or ordering question, and for rows anchored near the asked date. | the query's own cues; the pool-level variant `LOOM_MEMORY_SEARCH_RECENCY_POOL_WEIGHT` defaults to 0 (off) | | Prior group | Stored `confidence` × `confidence_weight`, soft tag and category boosts, the preference boost; capped together. | `confidence_weight`, `tags_soft`, `prior_cap` (`LOOM_MEMORY_PRIOR_CAP`) | | Reinforcement | Recalls and positive outcomes, decaying with a half-life; a `harm` event takes earned boost back. | `LOOM_MEMORY_SEARCH_REINFORCEMENT_WEIGHT` (0.15), `…_HALF_LIFE_DAYS` (90) | | Trust | `+0.5` asserted or observed, `−0.25` imported, `−1` external, `+1` when attested; times the weight. | `LOOM_MEMORY_SEARCH_TRUST_WEIGHT` (default 0 = off) | | Demotions | `demote_tags` sinks a row to the bottom without dropping it; rows labelled `sensitivity:medium` or `:high` are lowered, except when the query asks for a contact detail or a bank account (an IBAN). | `demote_tags`, `LOOM_MEMORY_SEARCH_SENSITIVITY_DEMOTE` (0.1) | **Pinned is not a rank term.** A pin protects a row from being overwritten ([how a write is judged](https://clickhouse.github.io/loom/concepts/memory.md#how-a-write-is-judged)); it does not make it rank higher. A fact is returned because it answers the question. `personalization="none"` drops every user-derived term (preference, the prior group, reinforcement) and keeps relevance, so the two rankings can be compared on one server. `LOOM_MEMORY_SEARCH_SOURCE_CLASS_BOOST_ELIGIBLE` limits which provenance classes may earn those terms at all: leaving out `extracted` keeps an inference from outranking a statement on its priors. **Reranker.** `rerank` on the request, or the namespace's `rerank_endpoint`, sends the pool to a reranker (`builtin:openai[:model]`, `builtin:cohere[:model]`, or the name of an endpoint the operator configured in `LOOM_RERANK_ENDPOINTS_JSON`; a URL is refused). The pool is widened first (`LOOM_MEMORY_SEARCH_POOL_MULTIPLIER_RERANK`), because a reranker only reorders what the first stage surfaced. It fails open and never drops a candidate. Pass `rerank` when the order of the pool matters more than one more call per search. To measure that on your own queries, run each query with `rerank` and with `rerank="none"` (which also turns off a namespace's `rerank_endpoint`), both with `explain=true` and on the budget you serve (a `fast` or LLM-free read makes no `builtin:openai` call), and compare the two windows, reading each hit's `score_components` as [Why did this rank here](https://clickhouse.github.io/loom/concepts/retrieval-and-ranking.md#why-did-this-rank-here) explains. No committed run compares the same searches with and without a reranker ([results](https://clickhouse.github.io/loom/benchmarks/results.md#longmemeval-s)). After ranking, the served window is shaped: `duplicate` and `superseded` rows are hidden unless asked for, at most `LOOM_MEMORY_SEARCH_PER_KEY_CAP` rows per key are kept, and `exclude_keys` removes rows the caller already holds. ## Abstain Every search ends with a verdict on its own result. The response carries `abstain`, `abstain_code` (`no_match`, `below_threshold`, `entity_absent`, `low_calibrated_confidence`, `page_expired`), a prose `abstain_reason`, a `retrieval_confidence` and a `retrieval_risk` of `low`, `medium`, `high` or `abstain`. The codes are defined in [search and ask](https://clickhouse.github.io/loom/features/memory/search-and-ask.md). Rows in the window whose own similarity is below the threshold carry `below_threshold: true`, so a caller that wants "abstain means abstain" per row can filter on it. **Where the thresholds come from.** A fixed similarity floor is calibrated for one embedding model; swap the model and the distribution of cosines shifts, so the same number starts filtering good hits or stops filtering noise. With `LOOM_ADAPTIVE_THRESHOLDS` on (the default), a probe at boot embeds three small corpora and derives every gate from the embedder's own distribution (`server/memory/app/embedder_calibration.py`): | Probe corpus | Gates derived | | --------------------- | ----------------------------------------------------------------------------------------------- | | Distractor pairs | The noise floor (90th percentile + 0.02) and the abstain threshold (floor + 0.05). | | Question → fact pairs | The trigger gates: query rewrite, HyDE fallback and rerank skip (10th, 25th, 75th percentiles). | | Paraphrase pairs | The write-time conflict and duplicate thresholds (median, 75th percentile). | A per-namespace calibration, when one is stored, overrides the cluster-wide one, and the `recalibration` worker repeats the probe (`LOOM_RECALIBRATION_INTERVAL_SECONDS`) so the gates follow a silently revised embedder. `calibration.get` shows which tier is in force. The scalar settings (`LOOM_MEMORY_SEARCH_MIN_SIMILARITY` 0.22, `LOOM_MEMORY_SEARCH_ABSTAIN_THRESHOLD` 0.30, `LOOM_MEMORY_SEARCH_HYDE_FALLBACK_THRESHOLD` 0.45, `LOOM_MEMORY_SEARCH_QUERY_REWRITE_THRESHOLD` 0.38) apply to fake embedders, a failed probe, or adaptivity switched off; in that case you own re-benchmarking after an embedder change. **The trade-off.** A floor trades recall for precision. An agent given the five nearest rows to an off-topic question will use them; an abstain tells it to ask or say it does not know. A floor set too high hides answers phrased unlike the question. Loom's position is that an explicit abstain an agent can act on is worth more than a filled window, and it leaves the caller the choice per read: - `min_similarity` sets this read's floor; - `floor_mode` decides what happens to rows under it: `readmit` (default: appended after the confident rows, tagged below the floor, so `top_k` is filled), `ids_only` (appended with an empty body: the key says a weak match exists without putting its text in context) or `drop`. An empty result from a scope with deferred extraction in flight reports `pending_extraction`: a save is on its way, so the empty answer is not evidence of absence. ## Why did this rank here `explain=true` on `memory.search` attaches `score_components` to each served hit: the ranking signals, the RRF score and weight when that mode ran, and after a reranker each row's move (`rerank_from`, `rerank_pos`, `rerank_delta`) and, when the reranker changed the order, its own fused score (`rank_score_pre_rerank`). A memory whose linked code changed and that moved down carries how many places (`code_changed_places`; when it moves is in [memories linked to code](https://clickhouse.github.io/loom/concepts/memory.md#memories-linked-to-code)). `memory.search_trace` takes the same request and scores the vector leg alone (no fusion, assembly or reranker): every candidate with its `similarity`, `entity_boost`, `rank_score`, `rank` and the plane it came from (`source`: `fact` or `chunk`). With `LOOM_SEARCH_TRACE_LOG_ENABLED=true` every search also appends a row to `loom.search_trace_log` (returned keys, scores, abstain, risk, latency), readable with `memory.search_trace_log`; the query text is stored only when the namespace consents to text capture. The dashboard's Recall Lab (`/recall`) shows the rows the Memories list shows for a query: keyword matches, then `memory.search` hits with their `score_components`. `memory.ask` retrieves the same way and then writes an answer from a narrower pool; see [search and ask](https://clickhouse.github.io/loom/features/memory/search-and-ask.md). The `loom_search` MCP tool (the `search.query` operation) and `context.pack` search several planes at once; see [unified search](https://clickhouse.github.io/loom/features/unified-search.md). ## Limits - Search reads only vectors of the live embedding model. Rows embedded by another model are invisible until `loom reembed` regenerates them. - Only the first part of a long value is embedded; the chunk leg is what finds a passage further in. A value that is really a document belongs in knowledge. - The abstain gate reads the rank-1 row. Under `rrf` a lexical-only match can hold rank 1 while a better cosine sits at rank 2; `LOOM_MEMORY_SEARCH_ABSTAIN_TOP_SIM_SCOPE=window` reads the best cosine in the window instead. - Calibration derives gates from small built-in probe corpora. A corpus unlike them (code, a single narrow domain) may want a per-namespace calibration. ## Related - [Search and ask](https://clickhouse.github.io/loom/features/memory/search-and-ask.md): every request field and response field. - [Unified search](https://clickhouse.github.io/loom/features/unified-search.md): `loom_search` and `context.pack` in detail. - [Entity graph](https://clickhouse.github.io/loom/features/entity-graph.md): where the entity and graph legs come from. - [Memory](https://clickhouse.github.io/loom/concepts/memory.md): the fields ranking reads (`source_class`, `status`, time). - [Performance tuning](https://clickhouse.github.io/loom/guides/performance-tuning.md). - Operations: [memory](https://clickhouse.github.io/loom/reference/operations.md#memory), [calibration](https://clickhouse.github.io/loom/reference/operations.md#calibration), [context](https://clickhouse.github.io/loom/reference/operations.md#context). Settings: [MemorySearch](https://clickhouse.github.io/loom/reference/configuration.md#memorysearch), [MemoryAsk](https://clickhouse.github.io/loom/reference/configuration.md#memoryask), [Mcp](https://clickhouse.github.io/loom/reference/configuration.md#mcp). --- # Connect a coding agent > Source: https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent/ (Markdown: https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md) This page connects Claude Code, Codex, Cursor or another MCP client to a running Loom server. Install the plugin (or wire the MCP entry), point it at the server, and work as usual. The next session starts with what the agent saved. What the Claude Code and Codex plugins send to Loom is in [data handling](https://clickhouse.github.io/loom/features/integrations/claude-code.md#data-handling). Conversation recording is off by default; the namespace owner turns it on with the **Record conversations** switch on the dashboard's Conversations page, or with `conversation.set_logging`. To run a server yourself, start with the [quickstart](https://clickhouse.github.io/loom/getting-started/quickstart.md); to have your agent install a pinned release and prove it works, give it the latest release's `https://github.com/ClickHouse/loom/releases/latest/download/agent-setup.txt` ([Set up Loom with your coding agent](https://clickhouse.github.io/loom/getting-started/agent-setup.md)). ## What you need ![Settings → Tokens: a new token and the Claude Code setup it prints](https://clickhouse.github.io/loom/images/dashboard-connect.png) - **The server URL**, for example `https://loom.example.com`, or `http://localhost:7777` for a stack started with `loom up`. - **A writer token for your namespace**, except against a single-user server on the same machine: there the plugins and the install links for [Cursor](https://clickhouse.github.io/loom/features/integrations/cursor.md#mcp-tools) and [VS Code](https://clickhouse.github.io/loom/features/integrations/vscode.md#local-single-user-server) connect without one. Create one in the dashboard at `/settings` → **Tokens**, or ask an operator to run `loom mint-token --subject you --org --namespaces --roles writer`. On a server with self-signup open, the Claude Code plugin can sign the machine up by itself on the first session. See [what happens with no credential](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md#no-token-yet). ## Claude Code ```bash claude plugin marketplace add ClickHouse/loom claude plugin install loom@loom \ --config server_url=https://loom.example.com \ --config token= ``` Both `--config` values are optional; without them Claude Code asks once. Start a new session. Loom's session primer arrives at the top, and from then on: - the hooks send Loom what [data handling](https://clickhouse.github.io/loom/features/integrations/claude-code.md#data-handling) lists; - the `loom-memory`, `loom-code`, `loom-session` and other skills tell Claude when to save a fact, when to search, and how to use the code index; - `loom_search`, `loom_memory_set` and the code tools are available as MCP tools. Everything the plugin installs, and how to configure it, is on the [Claude Code](https://clickhouse.github.io/loom/features/integrations/claude-code.md) page. ## Codex ```bash codex plugin marketplace add ClickHouse/loom codex plugin add loom@loom ``` Then run `codex` interactively once and [approve the plugin's hooks](https://clickhouse.github.io/loom/features/integrations/codex.md#approve-the-hooks); until then the tools work but nothing is captured. The plugin's MCP entry points at `http://localhost:7777`. For another server, write its URL and your token to `~/.loom/config.json` and override the MCP entry in `~/.codex/config.toml`, as shown in [Codex: the server](https://clickhouse.github.io/loom/features/integrations/codex.md#the-server). The Codex plugin ships the same hooks and skills; the differences are on the [Codex](https://clickhouse.github.io/loom/features/integrations/codex.md) page. Don't also run `codex mcp add loom` with the plugin installed: an entry with the same name replaces the plugin's. ## Cursor and other MCP clients On a Cursor Teams or Enterprise plan, a team admin can import the [Cursor plugin](https://clickhouse.github.io/loom/features/integrations/cursor.md#plugin) (untested in Cursor), which carries the MCP entry and Loom's skills. For Cursor or VS Code against a single-user server on the same machine, no token is needed: use the [Cursor](https://clickhouse.github.io/loom/features/integrations/cursor.md#mcp-tools) or [VS Code](https://clickhouse.github.io/loom/features/integrations/vscode.md#local-single-user-server) install link. Otherwise, in the dashboard, open **Settings → Tokens**, fill in the **New token** form (press **New token** first if a token exists) and press **Create token**, then choose your client (Cursor, VS Code, Cline, Windsurf, Zed or Claude Desktop) under **Connect**: the dialog shows the entry for its config file, with the token in it. For any other MCP client, copy the entry from [MCP clients](https://clickhouse.github.io/loom/features/integrations/mcp-clients.md). With the `loom` CLI installed (`pip install -e .` from a clone of the repository), one command writes the entry instead: ```bash loom install --client cursor --server-url https://loom.example.com --token "$LOOM_API_KEY" ``` Restart the client afterwards. `--client` also takes `claude-desktop`, `vscode`, `cline`, `windsurf` and `zed`; without it, `loom install` wires every client it finds. These clients get the tools; transcript capture needs a hook, which [Cursor](https://clickhouse.github.io/loom/features/integrations/cursor.md) has and the others take from a [transcript template](https://clickhouse.github.io/loom/features/integrations/index.md#transcript-capture-hooks). [MCP clients](https://clickhouse.github.io/loom/features/integrations/mcp-clients.md) covers toolsets and the entry the command writes. Then add the [agent instructions](https://clickhouse.github.io/loom/features/integrations/agent-instructions.md) block to a coding client's rule file, so the agent knows when to call the tools; for a client other than Cursor, add `?toolset=coding` to the entry's URL (`--toolset coding` with `loom install`) so the tools the block names are listed. ## Check that it works 1. In Claude Code, run `/loom:loom-status`. It reports the server the plugin calls, whether the token is accepted and can write the namespace, and any captured batches that never reached the server. 2. Tell the agent something durable, for example "our deploy window is Tuesday and Thursday only". In a new session, ask "when can we deploy?". The answer comes from Loom. 3. Open the dashboard (`/`). The saved fact is there, and the session's conversation too while conversation recording is on; you can correct or delete either. `loom doctor --client` adds the local wiring checks when you have the CLI. ## When the code index is used The plugin attaches the code index only when the session starts inside a git checkout of an indexed repository: - It reads the checkout's remotes (`origin`, then `upstream`, then the others) and looks them up among the repositories the server indexes. A fork with no index of its own uses its parent's index. - You need read access to that repository's code namespace. The dashboard's **Repositories** page shows what you can read and lets you join. - On a match, the session's code questions are scoped to that repository, and grep and whole-file reads of indexed paths are answered from the index as of its latest indexed commit. - With no match (a directory outside git, a repository nobody indexed, or one you can't read), nothing changes: the agent reads local files, and memory works as usual. To use a repository's index anyway, name it: `LOOM_CODE_REPO=/` (for example `ClickHouse/loom`, or just `loom` when one repository you can read has that name), the Claude Code plugin's `code_repo` option, or `"code_repo"` in `~/.loom/config.json` (the setting a Codex session reads). It is a fallback: outside any git checkout, or in a checkout whose repository has no index (a mirror with a different URL), the primer names that repository's namespace and every `loom_code_*` call defaults to it. Inside an indexed checkout, the checkout's own index still wins; to search another repository, pass its namespace to the call. `local_first` (`LOOM_CODE_LOCAL_FIRST=on`) keeps the index out of grep and file reads, for example while you work on uncommitted changes. Details: [Claude Code](https://clickhouse.github.io/loom/features/integrations/claude-code.md#the-code-shunt). ## No token yet On a server started with `LOOM_SELF_SIGNUP=open`, the first Claude Code session that finds no token signs the machine up. It writes the token to `~/.loom/config.json` (mode 0600) and a claim code to `~/.loom/claim-.txt`, and the primer says how to claim the namespace: the code works once, at the server's `/oauth/agent-claim` page, with a password of your own. `LOOM_AUTO_SIGNUP=off` turns this off. A server without self-signup leaves the session without a token, and the primer says how to get one. ## Next - [Your first agent](https://clickhouse.github.io/loom/getting-started/first-agent.md): the same memory from your own code with the SDK. - [Agent integration](https://clickhouse.github.io/loom/guides/agent-integration.md): what to save, when to search, and mistakes to avoid. - [Code index](https://clickhouse.github.io/loom/features/code-index/overview.md): ask Loom about a repository the server indexes. - [Troubleshooting](https://clickhouse.github.io/loom/operations/troubleshooting.md), starting from `/loom:loom-status`. --- # Evaluate Loom > Source: https://clickhouse.github.io/loom/getting-started/evaluate/ (Markdown: https://clickhouse.github.io/loom/getting-started/evaluate.md) Evaluate the decision you need to make, not every harness in the repository. This guide starts with a no-cost product smoke test, then moves to a workload-specific quality test. The public benchmark suite is the final step when you need to audit or reproduce a published claim. ## Choose an evaluation | Decision | Data | Providers | Typical output | | ----------------------------------------------- | -------------------------------------------------------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------------- | | Does Loom install and connect correctly? | Four test facts and one document | Stock fake providers are enough | A fact survives a new session; source, history and audit rows appear in the dashboard | | Is memory useful for our agent? | 25–50 real questions with expected memory keys | Real embedder; real LLM for extraction or `memory.ask` | recall@k, rank@1, abstention, answer accuracy, latency and cost | | Is code intelligence useful for our repository? | Representative symbols, caller sets and issue-shaped questions | Real embedder; indexed repository | exact definitions, caller F1, file recall, abstention, tokens and latency | | Can we reproduce a published result? | The named public dataset and committed result artifact | The pinned models or documented substitutes | A comparable run with the same slice, reader, judge and search mode | Do not use the fake embedder (a checkout's `.env`) or the fake LLM to judge quality: they exist so the stack can boot without credentials and integration tests can be deterministic. The released stack's local embedder retrieves for real, so a first look at search is meaningful on it; judge answers and extraction with a real LLM. ## Before you run Write down these five things first: 1. **Workload.** The users, repositories and question types the decision covers. 2. **Baseline.** What the same user or agent does without Loom: full context, grep, BM25, another memory system, or no retrieval. 3. **Precision gates.** The minimum recall, answer accuracy or task success you will accept. Quality comes before cost. 4. **Economy gates.** Maximum p95 latency, tokens, model cost, index time and storage. 5. **Instrument.** Loom commit, ClickHouse deployment, provider models, model parameters, hardware, dataset version and random seed. Pre-registering the gates prevents a convenient metric or slice from becoming the result after the run. Keep failures and abstentions visible; report request errors separately instead of silently counting or discarding them as retrieval misses. ## 1. Product smoke test This takes about 15 minutes and proves wiring, persistence and visibility—not semantic quality. 1. [Boot the local stack](https://clickhouse.github.io/loom/getting-started/quickstart.md#1-boot-the-stack) and confirm `/v1/health` reports `status: ok`. 2. [Write and search a fact](https://clickhouse.github.io/loom/getting-started/quickstart.md#4-first-memory-over-http). 3. [Connect a coding agent](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md), tell it a durable preference, open a fresh session and ask for that preference. 4. Correct the fact, then confirm the dashboard shows its source and version history. 5. Run `loom demo` with a document and confirm the returned chunks cite that document ([document smoke test](https://clickhouse.github.io/loom/getting-started/quickstart.md#optional-ask-a-question-about-a-document)). Pass only when the new session recalls the fact, the correction replaces the current answer, an unrelated question abstains, and the dashboard shows the writer and history. The result says nothing about production recall while health reports `fake_embedding` or `fake_llm`. ## 2. Evaluate your workload Configure [real providers](https://clickhouse.github.io/loom/operations/deployment.md#providers), reset or re-embed data if the vector width changed, and record the resolved models from `/v1/health`. Use an isolated namespace and keep the same indexed data for every paired arm. ### Memory and RAG Build a set of at least 25–50 questions from real tasks. Include: - direct facts, paraphrases, multi-hop questions and time-sensitive updates; - questions whose answer should have been superseded; - plausible but unanswerable questions, to measure false answers and abstention; - near-duplicate distractors that differ in the value that matters. For each answerable memory question, label the expected `memory_key` before running. For document RAG, label the expected document and chunk, query `knowledge.search`, and score those identifiers with the same retrieval metrics. Measure retrieval and synthesis separately: - **Retrieval:** recall@k, rank@1, MRR, false-positive rate and abstain precision/recall. - **Answer:** correctness under one pinned reader and judge, plus citation correctness. - **Economy:** p50/p95 latency, input/output tokens, model cost, construction cost and stored rows. For an existing memory corpus, save a JSON dataset with this shape (the supplied harness targets `memory.search`; use an equivalent scorer for `knowledge.search`): ```json { "org": "acme", "namespace": "support-eval", "questions": [{ "query": "When can we deploy?", "gold_keys": ["deploy_window"] }] } ``` Then run retrieval without changing the corpus: ```bash export PYTHONPATH=$PWD LOOM_API_KEY= .venv/bin/python bench/retrieval_recall.py \ --dataset /path/to/retrieval_qa.json --base-url http://localhost:7777 \ --top-k 10 --out /tmp/loom-retrieval.csv ``` Set `--min-recall` and `--min-mrr` only from your pre-registered gates. Use `memory.ask` or your production reader on the same questions for answer accuracy; recall and answer accuracy are different measurements. ### Code intelligence Index a repository that resembles your production repositories. Do not start with only the repository Loom was tuned on. Label: - definitions with the expected file and qualified symbol; - callers with the complete expected caller set; - conceptual and issue-shaped questions with acceptable files in the top 5 and top 10; - out-of-repository questions that should abstain; - the tests or dependants expected from an impact query. Run the code-agent acceptance report against an indexed repository: ```bash export PYTHONPATH=$PWD .venv/bin/python bench/acceptance/code_agent.py \ --base-url http://localhost:7777 --token-file /path/to/token \ --namespace ``` The built-in fixture targets ClickHouse and is useful for checking a deployment. For a product decision, replace or extend it with your labelled questions and compare the same agent tasks with ordinary repository tools only. Report task success first, then turns, tokens, cost and wall time. Tool-schema overhead depends on the client, so record whether it defers MCP tool definitions. ### Keep the comparison fair - Use the same model, prompt, data snapshot and question order in paired arms. - Keep extraction/index construction cost separate from per-question cost, then state the number of questions needed to amortize construction. - Disable learning from evaluation reads (`auto_touch: false`, `reinforce: false`) so an early query cannot improve a later one. - Report sample size, per-category results and uncertainty. Do not promote a small smoke slice to a general claim. - Save per-question outputs, including errors and misses—not only aggregate scores. ## 3. Audit or reproduce published results The README is a scorecard, not the evidence ledger. For every published number: 1. Open the matching section in [Benchmark results](https://clickhouse.github.io/loom/benchmarks/results.md). 2. Inspect its committed artifact under [`bench/results/`](https://github.com/ClickHouse/loom/blob/main/bench/results). 3. Confirm the dataset slice, Loom commit, models, reader, judge, search mode and seed. 4. Run the exact reproduction command from the result section. 5. Interpret the delta using the [methodology](https://clickhouse.github.io/loom/benchmarks/methodology.md), including judge calibration, bootstrap intervals and known bias. Third-party datasets are not all redistributed by this repository. Their source and license are listed under [datasets](https://clickhouse.github.io/loom/benchmarks/running.md#datasets). Full public runs can take hours and incur provider charges; start with the documented smoke slice, then run the complete protocol before publishing a comparison. ## What to publish Publish a result in the README only when it is representative, understandable in one row, and backed by a committed artifact and reproduction command. Put the full category table, models, prompts, uncertainty, costs, failure analysis and limitations in the [results ledger](https://clickhouse.github.io/loom/benchmarks/results.md). Put stable evaluation rules in [methodology](https://clickhouse.github.io/loom/benchmarks/methodology.md), and commands and dataset acquisition in [running](https://clickhouse.github.io/loom/benchmarks/running.md). Every public claim should state: - the task and sample size; - the metric and whether it is retrieval, answer quality or end-to-end success; - the baseline and paired delta, when one exists; - models and judge, or that no LLM judge was used; - uncertainty or the granularity of a one-question change; - construction and query cost separately; - the committed artifact and exact reproduction command; - material negative findings and where the result does not generalize. Do not publish every available number in the README. A long scoreboard hides the product decision, encourages invalid cross-benchmark comparisons and makes stale claims hard to spot. The README should lead readers to evidence; the docs should carry the evidence. ## Next - [Benchmark coverage](https://clickhouse.github.io/loom/benchmarks/coverage.md): what has a public measurement, what has only a harness, and what has no public evidence yet. - [Benchmark overview](https://clickhouse.github.io/loom/benchmarks/overview.md): choose a harness by the signal it measures. - [Running benchmarks](https://clickhouse.github.io/loom/benchmarks/running.md): providers, tokens, commands and datasets. - [Benchmark methodology](https://clickhouse.github.io/loom/benchmarks/methodology.md): comparison rules and limitations. - [Benchmark results](https://clickhouse.github.io/loom/benchmarks/results.md): committed measurements and artifacts. --- # Your first agent > Source: https://clickhouse.github.io/loom/getting-started/first-agent/ (Markdown: https://clickhouse.github.io/loom/getting-started/first-agent.md) This tutorial builds a small agent with the Python SDK. It continues from the [quickstart](https://clickhouse.github.io/loom/getting-started/quickstart.md): the stack runs on `http://localhost:7777` and the Python SDK is installed ([quickstart step 5](https://clickhouse.github.io/loom/getting-started/quickstart.md#5-first-memory-from-python)). Every step runs on the stock stack. Extraction and `memory.ask` synthesis need a real LLM; ranking is meaningful on the released stack's local embedder, not on a checkout's fake one ([providers](https://clickhouse.github.io/loom/operations/deployment.md#providers)). ## 1. Give the agent an identity Every row Loom stores carries the identity tuple `(org, namespace, agent, user_id, session_id)`. The `Agent` binds the first three; `user_id` and `session_id` travel per call and default to `-`. ```python import asyncio from loom import Agent USER = "u-42" # the end user this conversation is about def make_agent() -> Agent: return Agent( org="dev", namespace="local", agent="support-bot", # authorship: every row and ledger entry names it base_url="http://localhost:7777", consumer="support-demo", # the product this process speaks for ) ``` - **`namespace`** is the sharing unit. Every member of the namespace reads every row in it, whichever agent wrote it; `agent` records who wrote a row, it is not a read boundary. On the stock stack the only namespace is `dev/local`. - **`agent`** must be a real name. `""`, `"-"` and `"*"` are replaced with a name derived from the script, with a warning, because per-agent cost, provenance and advice need to tell bots apart. - **`consumer`** names the product this process speaks for and is stamped on every request. A search counts as a ranking signal only when the reader names a consumer and ties the read to a run or session, so set it from the start. - The token comes from `api_key=`, then `LOOM_API_KEY`, then `~/.loom/token`. None is needed in single-user mode. Why the tuple: one agent serving many users must not answer one user from another's facts, and one namespace shared by several agents must still say who wrote what. The [identities and namespaces](https://clickhouse.github.io/loom/concepts/identities-and-namespaces.md) page explains the model. ## 2. Store facts and handle abstain ```python async def remember_and_recall(a: Agent) -> None: res = await a.memory.set( "support/pref/seat", "Prefers an aisle seat on long flights.", tags=["travel"], user_id=USER, ) print("stored:", res["memory_key"], res["status"], res.get("conflict_verdict", "")) hits = await a.memory.search("which seat should I book?", user_id=USER, top_k=3) if hits.abstain: # Loom does not know. Say so; do not answer from the weak rows. print("no answer:", hits.abstain_code, hits.abstain_reason) return for hit in hits: print(hit.memory_key, round(hit.similarity, 3), hit.content_excerpt) ``` - **One claim per key.** A key reads like a path (`//`). Writing the same key again revises the fact; writing a different key whose value is close to an existing fact is checked against it. The response's `status` and `conflict_verdict` say what happened (a conflict with a pinned fact is rejected with `PINNED_CONFLICT`). [Conflicts and refusal](https://clickhouse.github.io/loom/features/memory/conflicts-and-refusal.md) covers the verdicts. - **`user_id` on the read filters to that user's rows.** Without it the search spans the whole namespace. - **`abstain` is the answer, not an error.** When the best hit is below a similarity gate calibrated for the embedder, the result still carries the rows but sets `abstain=True` with a machine-readable `abstain_code` (`no_match`, `below_threshold`, `entity_absent`, `low_calibrated_confidence`, `page_expired`). Branch on the code, not the free-text reason. Each row below the gate also carries `below_threshold=True`. `pending_extraction` above zero means deferred extraction has not finished yet, so an empty result is not evidence of absence. With the fake embedder the ranking is by character n-grams, so which query abstains is arbitrary; with a real embedder a question about something never stored abstains. [Retrieval and ranking](https://clickhouse.github.io/loom/concepts/retrieval-and-ranking.md) explains the gates and the ranking terms. For a grounded natural-language answer instead of hits, `await a.memory.ask("Which seat does the user prefer?", user_id=USER)` returns `answer`, `abstain` and the `memory_keys` it cited. It needs a real LLM; with the fake one there is no synthesis. ## 3. Hold a conversation as a session A `Session` is what an agent loop keeps between turns and across restarts. It binds one `(user_id, session_id)` scope and four verbs to it: ```python async def converse(a: Agent) -> None: async with a.session("thread-42", user_id=USER) as s: resumed = await s.resume(query="book a flight for the user") system_prompt = resumed.system # the context pack, rendered history = resumed.messages # the transcript tail, oldest first; [] when new print(system_prompt[:300], len(history)) await s.append( {"role": "user", "content": "Book me on the Friday flight to Berlin."}, {"role": "assistant", "content": "Booked 14C, an aisle seat."}, ) # leaving the block calls close(): buffered turns go to extraction ``` - **`resume()`** is `context.pack` for this scope plus the last turns of the transcript: exactly what a restarted process needs to continue. - **`append()`** writes each turn to the conversation transcript (`conversation.append`) at once, and every `flush_every` turns (default 8) sends the batch to `memory.set_from_messages`, where the server extracts the facts worth keeping. `close()` flushes the rest. On the fake LLM the transcript is stored but extraction writes nothing. - **`offload()` / `fetch()`** park a large tool result once and keep only a pointer in the transcript; **`note()`** appends to the working-memory block the next pack carries. If the namespace has conversation logging turned off, the transcript is not stored (the SDK logs a warning) and extraction still runs. [Conversations and sessions](https://clickhouse.github.io/loom/features/conversations-and-sessions.md) covers threads, digests and session briefs. ## 4. Load context at session start Outside a session, one call assembles the bounded block an agent wants before its first turn: working memory first, then the profile, relevant memories, matching procedures and decisions, within a token budget. ```python async def session_start(a: Agent) -> str: pack = await a.context.pack("book a flight", mode="startup", user_id=USER) print(pack.section_count, "sections,", pack.tokens_estimate, "tokens") return str(pack) # the text to put in the system prompt ``` `mode` (`startup`, `task`, `debug`, `handoff`) shifts the budget between sections; `token_budget` must be between 200 and 12000 (the server rejects values outside it). `pack.stable_prefix` is the part that does not change between queries for this identity, which is where to put a provider cache breakpoint. A section that fails degrades to empty instead of failing the pack. Working memory is scratchpad state, not retrieval memory: it is loaded first and never embedded or ranked ([Working memory and profiles](https://clickhouse.github.io/loom/features/memory/working-memory-and-profiles.md)). ## 5. Record a task as a run A run is one agent execution: what it was asked, what it relied on and how it ended. Naming the memories a run relied on is the feedback loop: those keys are credited in search ranking, and a later grade can count the outcome against them. ```python async def handle_task(a: Agent) -> None: question = "which seat should I book?" hits = await a.memory.search(question, user_id=USER, top_k=3) async with a.run_scope(goal="Pick a seat for the Berlin booking", user_id=USER) as run: if hits.abstain: run.final_response = "I don't have the user's seat preference." else: best = hits[0] run.used(best.memory_key) # relied on, not merely retrieved run.final_response = f"Booked an aisle seat ({best.content_excerpt})" await a.memory.set( "support/booking/berlin", "Booked seat 14C on the Friday flight to Berlin.", user_id=USER, ) print(run.completed) # the run.complete response ``` `run_scope` calls `run.begin` on entry and `run.complete` on exit, with status `completed` (or the `run.status` the block set), `failed` when the block raises, or `cancelled` when it is cancelled (either still propagates after the ledger row is written). Writes inside the block carry the run's `run_id`, and the keys a grounded `memory.ask` answer cites inside it (not an abstention) are added to `memories_used` automatically. A write under a run is scoped to it: restating a fact that another run wrote is stored as a `duplicate`, not merged. The same two calls are available directly as `a.run.begin(goal=...)` and `a.run.complete(run_id, status=..., memories_used=[...], final_response=...)`; `run.complete` also takes a `reflection` (what worked, what failed, lessons) that is stored as a `reflection` memory. `a.run.get(run_id)` replays a run with what it wrote, recalled and cost. [Runs](https://clickhouse.github.io/loom/features/runs.md) covers reflections, retention and grades. ## 6. Put it together ```python async def main() -> None: async with make_agent() as a: await remember_and_recall(a) await session_start(a) await converse(a) await handle_task(a) asyncio.run(main()) ``` Run it, then open `http://localhost:7777/memories` to see the rows with their writer agent, `/runs` for the run, and `/conversations` for the thread. Blocking code uses `loom.SyncAgent`, which takes the same arguments and exposes every method as a blocking call. ## The same agent in TypeScript The TypeScript SDK (`@clickhouse/loom`; [install](https://clickhouse.github.io/loom/reference/typescript-sdk.md#install)) has the same identity model and session, with shorter verb names. It has no run helper; runs go through the typed `call`. ```ts import { Agent } from "@clickhouse/loom"; const USER = "u-42"; const bot = new Agent("dev", "local", "support-bot", { baseUrl: "http://localhost:7777", consumer: "support-demo", }); await bot.remember("support/pref/seat", "Prefers an aisle seat on long flights.", { userId: USER, tags: ["travel"], }); const hits = await bot.recall("which seat should I book?", { userId: USER, top_k: 3 }); if (hits.abstain) console.log("no answer:", hits.abstain_code); else for (const h of hits.results) console.log(h.memory_key, h.content_excerpt); const s = bot.session("thread-42", { userId: USER }); const { system, messages } = await s.resume({ query: "book a flight for the user" }); console.log(system.slice(0, 300), messages.length); await s.append({ role: "user", content: "Book me on the Friday flight to Berlin." }); await s.close(); const run = await bot.call( "run.begin", { goal: "Pick a seat for the Berlin booking" }, { userId: USER }, ); // run_id is part of the identity scope, like userId await bot.call( "run.complete", { status: "completed", memories_used: ["support/pref/seat"], final_response: "Booked an aisle seat.", }, { runId: run.run_id }, ); ``` `bot.contextPack({ query, mode: "startup" })` is `context.pack`, and `bot.ask(query)` is `memory.ask`. The [TypeScript SDK reference](https://clickhouse.github.io/loom/reference/typescript-sdk.md) covers the full surface. ## 7. Connect a coding agent A coding agent reads the same namespace through MCP tools. What you stored from Python is visible to it when its token reads that namespace. [Connect a coding agent](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md) covers the plugins and `loom install`. ## Next - [Memory](https://clickhouse.github.io/loom/concepts/memory.md): what happens between the write and the read. - [Writing memory](https://clickhouse.github.io/loom/features/memory/writing.md) and [Search and ask](https://clickhouse.github.io/loom/features/memory/search-and-ask.md): every field of the calls above. - [Agent integration guide](https://clickhouse.github.io/loom/guides/agent-integration.md): patterns for a production agent loop. - [Python SDK reference](https://clickhouse.github.io/loom/reference/python-sdk.md). --- # Quickstart > Source: https://clickhouse.github.io/loom/getting-started/quickstart/ (Markdown: https://clickhouse.github.io/loom/getting-started/quickstart.md) This page boots Loom on a laptop and saves a first memory over HTTP, Python and MCP. It needs no API key and no account. A fresh install of the released stack embeds with a small local model the image carries, so search returns meaningful hits with no network (an upgraded one keeps the embedder its `.env` names); its LLM is fake, so extraction and `memory.ask` answers produce nothing. A checkout's `.env` uses fake providers for both. **The stock Docker installation is single-user:** no login, localhost-only access and a default `dev/local` namespace. One person can connect multiple agents without enabling multi-user mode. **Team deployments are also supported in OSS**, with authenticated accounts and namespace access controls; [start in multi-user mode](https://clickhouse.github.io/loom/getting-started/quickstart.md#start-in-multi-user-mode) or follow [Deployment → Identity modes](https://clickhouse.github.io/loom/operations/deployment.md#identity-modes). Both modes support the same Loom features; real LLMs and embeddings need provider configuration in either mode. ## Prerequisites | Need | For | | ------------------------------ | ------------------------------------------------------------------------ | | Docker with the compose plugin | The stack: ClickHouse and the Loom server. | | Python 3.12 or newer | Contributor-only `loom` CLI. The SDK supports 3.11+. | | Node.js 20 or newer | The TypeScript SDK (optional). | | Free port `7777` | Loom. Checkout development also exposes ClickHouse on `8123` and `9000`. | | `jq` | Reading the `curl` output in step 4 (optional). | ## 1. Boot the stack ### Fast path: released image, no clone Prerequisite: Docker Desktop or Docker Engine with Compose 2.20 or later. On Windows, run the command in a WSL 2 terminal where `docker` works: Docker Desktop with its WSL integration turned on for that distribution, or Docker Engine installed in it. The installer is a POSIX shell script. ```bash curl -fsSL https://github.com/ClickHouse/loom/releases/latest/download/install.sh -o loom-install.sh && sh loom-install.sh && rm loom-install.sh ``` The command saves the installer in the current directory and then runs that file, so run it from your home directory, or another directory only you can write to: in a shared one, someone else can replace `loom-install.sh`, or point it at another file, before it runs. A failed download stops with curl's error and a non-zero exit; after a failed install, fix what it reports and run the whole command again. The installer downloads `loom-stack.tar.gz` from the latest GitHub Release, verifies its SHA-256, installs the Compose and ClickHouse configuration under `~/.loom`, pulls the matching `ghcr.io/clickhouse/loom:X.Y.Z` and `ghcr.io/clickhouse/loom-extractor:X.Y.Z` (the document extractor service) images, and waits for `http://localhost:7777/v1/health`. The server image contains the API, workers, migrations and the built React dashboard at the server root; no Python or Node installation is needed. Python, TypeScript and Go compiler indexers are included and enabled, and so is the C++ one (`scip-clang`) on `amd64`. A repository indexed with the TypeScript indexer gets its dependencies installed automatically. Both images support Linux `amd64` and `arm64`, including Docker Desktop on Apple Silicon. Once Loom answers, the installer: 1. **Checks a memory.** A new install saves a test memory, reads it back and deletes it, which shows that writes and reads work and leaves no live memory (its deleted version shows under Recently deleted). A rerun after that check passed, or an upgrade, skips it. 2. **Learns whether sign-in is on.** With sign-in on, it connects no agent and says where tokens come from. 3. **Offers to connect your coding agents.** On a terminal it asks, for each one it finds: Claude Code and Codex through their plugins, VS Code through `code --add-mcp`. `LOOM_CONNECT` answers instead, for a run without a terminal too, and only the clients it names are looked at: a comma list of `claude-code`, `codex` and `vscode`, `all`, or `none` alone; any other value stops the install. Without a terminal and without `LOOM_CONNECT`, no agent is looked at or changed. A plugin or MCP server set up before (by the plugin, `claude mcp add`, `codex mcp add` or `loom install`) is left as it is; for a Claude Code plugin the installer prints the command that points it at this install. Claude Code's plugin runs the Node.js 18 or later on PATH, so without one the installer installs no plugin and prints the `claude mcp add` line for the MCP server alone; Codex's finds Node.js through nvm and the like too, so it is installed, with the `codex mcp add` line beside it. Codex's plugin always uses `http://localhost:7777`, so for an install elsewhere it gets a link to its setup page instead. VS Code is not offered in a remote session or WSL. A client `LOOM_CONNECT` names that is not on PATH is named. 4. **Opens the dashboard** on a desktop, for a new install; `LOOM_NO_OPEN=1` skips that. Before it starts Loom, a new install on a terminal with Ollama answering on `localhost:11434` is also asked whether to run Loom's model jobs on it (extraction, and the assistant once enabled), with no API key; `LOOM_LOCAL_MODELS=ollama` answers yes and `off` no ([a model server on your machine](https://clickhouse.github.io/loom/getting-started/models-and-keys.md#a-model-server-on-your-machine-no-api-key)). Run from WSL, these steps act inside WSL: connect an agent that runs on Windows from Windows with the [connect guide](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md), and open the dashboard address the installer prints in a Windows browser. Ctrl-C stops the run at any step; a clone an agent CLI started for its plugin may still finish in the background. The stack runs its own ClickHouse container. To use a ClickHouse you already run, a self-managed server or a ClickHouse Cloud service (26.4 or later, with a Keeper), pass its URL; the installer asks for the password when the URL has none, creates the `loom` database and a read-only reader user, or prints the SQL an admin has to run ([choosing the ClickHouse](https://clickhouse.github.io/loom/operations/deployment.md#choosing-the-clickhouse)): ```bash curl -fsSL https://github.com/ClickHouse/loom/releases/latest/download/install.sh -o loom-install.sh \ && sh loom-install.sh --clickhouse https://loom@abc123.us-east-1.aws.clickhouse.cloud:8443 && rm loom-install.sh ``` Each release's `install.sh` installs that release, so the copy `latest` serves installs the latest release, and a release's own URL, below, installs that release every time. Over a newer install, an older release's copy refuses to install, rather than start an older Loom and ClickHouse on newer data; to downgrade on purpose, follow [downgrading an installer install](https://clickhouse.github.io/loom/operations/upgrades.md#downgrading-an-installer-install). To pin a release, use its own URL: ```bash curl -fsSL https://github.com/ClickHouse/loom/releases/download/v1.2.3/install.sh -o loom-install.sh && sh loom-install.sh && rm loom-install.sh ``` To inspect everything before starting, download the files and check them: ```bash curl -fLO https://github.com/ClickHouse/loom/releases/latest/download/loom-stack.tar.gz curl -fLO https://github.com/ClickHouse/loom/releases/latest/download/loom-stack.tar.gz.sha256 curl -fL https://github.com/ClickHouse/loom/releases/latest/download/install.sh -o loom-install.sh sha256sum -c loom-stack.tar.gz.sha256 tar -tzf loom-stack.tar.gz ``` Then, once what they print looks right, install from the checked bundle: ```bash LOOM_INSTALL_BUNDLE="$PWD/loom-stack.tar.gz" \ LOOM_INSTALL_CHECKSUM="$PWD/loom-stack.tar.gz.sha256" sh loom-install.sh && rm loom-install.sh ``` `LOOM_PORT`, `LOOM_BIND_HOST`, `LOOM_IMAGE_REPOSITORY`, `LOOM_EXTRACTOR_IMAGE_REPOSITORY` and `LOOM_CLICKHOUSE_IMAGE` given to the installer are written to the install directory's `.env`, so later `docker compose` commands there and later installer runs use the same port and images. A `LOOM_BIND_HOST` other than loopback publishes Loom beyond this host, and the server refuses to start that way without `LOOM_PUBLIC_BASE_URL` and sign-in: install with `LOOM_SKIP_START=1`, set them in `.env` (below), then run the install command again without `LOOM_SKIP_START=1`. Each release runs the ClickHouse release that was latest when it was built, so running the install command again for the same Loom version keeps its ClickHouse; installing a newer Loom version moves it, unless `.env` pins `LOOM_CLICKHOUSE_IMAGE` (delete that line to follow the releases again). Running the install command again updates the managed Compose/configuration files and reconciles the containers. It preserves `~/.loom/.env`, adding the extractor's URL and a token to one without, and the named Docker volumes. To stop without deleting data: ```bash docker compose --project-directory "$HOME/.loom" down ``` Add `--volumes` only for an intentional, irreversible reset. ### Try it in GitHub Codespaces [![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/ClickHouse/loom?quickstart=1) The repository's dev container runs the latest release in a codespace, with nothing on your machine: it installs Loom with the installer above, starts it, and connects Claude Code to it. The first start downloads about 3 GB of images and takes a few minutes. Open the dashboard at the address the creation log's last lines print (`Loom is running in the codespace. Dashboard: https://-7777.app.github.dev`, the **Ports** tab's forwarded address for port 7777); the installer's `http://127.0.0.1:7777/` above it is the codespace's own address. In the codespace's terminal, run `cd ~ && claude` (outside the checkout, whose instructions are for working on Loom itself), sign in, and say "Remember that I never deploy on Friday"; in a new `claude` session, ask "When can I deploy?". The try-out runs single-user, with no sign-in. In the browser, port 7777 is forwarded privately: its address answers only your GitHub account. Keep it private: public, or private to an organization, opens a server with no sign-in to everyone it is shared with. Agents inside the codespace, such as its `claude`, use `http://localhost:7777`; the dashboard's connection lines name the forwarded address instead, which needs a GitHub sign-in that agents in the codespace do not have. Opened in desktop VS Code, the codespace also forwards the port to your laptop's `localhost:7777`, where it is as open as a local install to every program on the laptop, and agents on the laptop can reach it there; the forward needs that local port free, so stop a local Loom first. Use the dashboard from the browser address: from the laptop's `localhost:7777` the page opens, but the server refuses its requests, so it shows no data; only the forwarded address is this server's own page. The install lives in `/workspaces/.loom`, so a rebuilt codespace keeps its `.env` and Compose project, though its memories live in Docker volumes a rebuild may not keep; a rebuild also runs the latest release's installer over it, which moves it to that release. Wherever these docs name `~/.loom` or `"$HOME/.loom"`, a codespace uses `/workspaces/.loom`. Codespaces offers to open port 7777 as soon as it listens; the dashboard answers once the creation log prints "Loom is running in the codespace". If the log ends with an error instead, run `sh .devcontainer/start-loom.sh` in the codespace's terminal: it keeps what the first run set up. ### Start in multi-user mode For a fresh released-image installation with accounts and sign-in from the first boot, prepare the stack files without starting containers: ```bash curl -fsSL https://github.com/ClickHouse/loom/releases/latest/download/install.sh -o loom-install.sh \ && LOOM_SKIP_START=1 sh loom-install.sh && rm loom-install.sh ``` Edit `~/.loom/.env` before starting Docker. Replace the existing identity and auth entries, and add the signup setting: ```dotenv LOOM_IDENTITY_PROVIDER=trust-network LOOM_AUTH_REQUIRED=true LOOM_SELF_SIGNUP=open ``` Keep the generated `COMPOSE_PROJECT_NAME` and `LOOM_JWT_SECRET`. Then pull and start the stack (with your own ClickHouse, run `ch-setup` first, as the installer said): ```bash docker compose --project-directory "$HOME/.loom" pull docker compose --project-directory "$HOME/.loom" run --rm loom loom ch-setup # your own ClickHouse only docker compose --project-directory "$HOME/.loom" up -d ``` After the [health checks](https://clickhouse.github.io/loom/getting-started/quickstart.md#check-the-stack) pass, open `http://localhost:7777/` and sign up. SDK and MCP clients need tokens; use the org and namespaces granted to your account, not the single-user `dev/local` defaults in the examples below. This enables multi-user authentication, not public networking: the stack still binds to localhost. Before sharing it with a team, configure [operator access](https://clickhouse.github.io/loom/operations/deployment.md#identity-modes) and [TLS](https://clickhouse.github.io/loom/operations/deployment.md#proxy-and-tls). Configure [real models and API keys](https://clickhouse.github.io/loom/getting-started/models-and-keys.md) as well; selecting multi-user mode does not replace the fake LLM. For an already running single-user stack, follow the [switching instructions](https://clickhouse.github.io/loom/getting-started/quickstart.md#2-open-the-dashboard) instead; container recreation preserves the existing data volumes. ### Contributor path: build a checkout Prerequisite: Docker Compose 2.34 or later, which the checkout's compose file needs to pull ClickHouse daily. ```bash git clone https://github.com/ClickHouse/loom && cd loom python3 -m venv .venv && . .venv/bin/activate pip install -e . loom up ``` `loom up` does three things: 1. Creates the two files git does not track: `.env` from `.env.example`, and `deploy/clickhouse-config/named_collections.xml` from its template. It never overwrites either; it gives an existing `.env` without an extractor token one. 2. Runs the dev compose stack with `--build`. 3. Polls `http://localhost:7777/v1/health` until the server is ready, printing the pending-migration count. It gives up after `--timeout` (300 s) with the last health body and log lines. The first contributor boot builds the image and applies every schema migration, so allow a few minutes. `loom up` then prints the dashboard URL (`http://localhost:7777/`), a `curl` call and a Python snippet that save a first memory, and a line that wires Claude Code. When auth is on, it also mints a writer token and puts it in those snippets. | Flag | Does | | ------------ | ---------------------------- | | `--rebuild` | Rebuild after a code change. | | `--no-build` | Reuse the cached image. | | `--open` | Open the dashboard. | | `--port` | Use another Loom port. | | `--env-file` | Point at another env file. | Re-running `loom up` on a running stack only re-checks health. `make dev` does the same without the CLI and runs: ```bash docker compose --env-file .env -f compose.yaml -f dev/docker-compose.dev.yml --profile bundled up -d --build ``` With `compose.yaml` first, Compose reads the repository root's `.env` for the `${VAR:-default}` substitutions; `--env-file .env` only says so explicitly. `loom down` stops the stack and preserves its volumes. `loom down --delete-data` permanently deletes this installation's named volumes only after you type the generated Compose project name; `--yes` is the non-interactive confirmation. The Makefile targets do the same: `make down` preserves data and `make down-all` deletes it. ### What the stock `.env` boots The checkout and release-bundle `.env` templates both select single-user mode. The release bundle sets `LOOM_IDENTITY_PROVIDER=single-user` explicitly; the checkout leaves it empty and infers the same mode from `LOOM_AUTH_REQUIRED=false`. | Setting | Value | Effect | | ---------------------------------------------------------------------------- | ------------- | ------------------------------------------------------------------------- | | `LOOM_AUTH_REQUIRED=false`, `LOOM_IDENTITY_PROVIDER=single-user` (or empty) | `single-user` | No login. Default namespace `dev/local`. SDK and MCP calls need no token. | | `LOOM_LLM_PROVIDER=fake`, `LOOM_LLM_MODEL=fake-echo-1` | fake LLM | Extraction, entity tagging and `memory.ask` synthesis produce nothing. | | `LOOM_EMBEDDING_MODEL=local`, `LOOM_EMBEDDING_DIM=384` (release bundle) | bge-small | Semantic search on CPU, from the model files the image carries. | | `LOOM_EMBEDDING_MODEL=fake-ngram-3072`, `LOOM_EMBEDDING_DIM=3072` (checkout) | fake embedder | Search ranks by character n-grams. Enough to check wiring, not recall. | Both installers generate a unique `COMPOSE_PROJECT_NAME`, a random JWT signing secret, a random extractor token, and `LOOM_BIND_HOST=127.0.0.1`. Separate clones therefore do not silently share ClickHouse volumes, and Loom plus ClickHouse are reachable only from this computer. For real providers, open **Settings → Models & keys**, choose your providers and models, enter their API keys, then click **Test & apply**. One OpenAI API key can serve both the LLM and embeddings. No Docker restart is needed; embedding changes trigger a verified rebuild, with progress and retry shown in the dashboard. For environment-based deployment instead, replace the provider lines in `~/.loom/.env` (released stack) or `.env` (checkout). Setting `OPENAI_API_KEY` alone changes nothing: the stock provider lines name their providers explicitly. Moving off the local or fake embedder re-embeds what is stored. ```bash LOOM_EMBEDDING_MODEL=text-embedding-3-large LOOM_EMBEDDING_DIM=3072 LOOM_LLM_PROVIDER=openai LOOM_LLM_MODEL=gpt-6-luna OPENAI_API_KEY=sk-... ``` For Claude, OpenAI, hosted embeddings, subscription-only clients and deployments with one LLM provider, follow [Configure models and API keys](https://clickhouse.github.io/loom/getting-started/models-and-keys.md). Only after editing the released stack's `.env`, apply it with `docker compose --project-directory "$HOME/.loom" up -d --force-recreate loom`; a plain restart does not reload the container's environment. Advanced settings are in [Deployment → Providers](https://clickhouse.github.io/loom/operations/deployment.md#providers). ### Check the stack ```bash curl -fsS http://localhost:7777/v1/health | jq '{status, fake_embedding, fake_llm, providers_degraded, problems}' docker compose --project-directory "$HOME/.loom" exec loom loom doctor ``` - `/v1/health` says `status: ok` when ClickHouse is reachable and no problem other than `code_tier_failing` (a repository's failing code-index tier) is an error. `fake_embedding`, `fake_llm` and `providers_degraded` say which mode you are in; `problems` lists each issue with its fix; `schema.pending` lists unapplied migrations. - `loom doctor` prints one line per server-side check. `--json` gives JSON. From a source checkout stack, run `docker compose --env-file .env -f compose.yaml -f dev/docker-compose.dev.yml exec loom loom doctor`. ## 2. Open the dashboard `http://localhost:7777/` opens straight into the dashboard in `single-user` mode; in multi-user mode, sign up or sign in first. In `single-user` mode the dashboard and agent defaults use `dev/local` (`LOOM_DEFAULT_NAMESPACE` renames the namespace). This is a convenience mode, not a user-isolation boundary: anyone who can reach the server can access its data and administration. Keep it localhost-only. To try sign-up instead (`trust-network`: email and password, and the first person to claim a namespace name becomes its admin), replace these entries in `~/.loom/.env` (released stack) or `.env` (checkout): ```dotenv LOOM_IDENTITY_PROVIDER=trust-network LOOM_AUTH_REQUIRED=true LOOM_SELF_SIGNUP=open ``` Setting only `LOOM_AUTH_REQUIRED=true` is insufficient when `single-user` is explicitly selected: that mode keeps authentication off. Keep the installer's generated `LOOM_JWT_SECRET` and existing volumes, then recreate the server to load the environment: ```bash # Released stack: docker compose --project-directory "$HOME/.loom" up -d --force-recreate loom # Checkout stack (run from the repository root): docker compose --env-file .env -f compose.yaml -f dev/docker-compose.dev.yml --profile bundled up -d --force-recreate loom ``` The dashboard then opens on its sign-in page, which has a **Sign up** tab, and every SDK and MCP call needs a token. The identity modes are in [Identities and namespaces](https://clickhouse.github.io/loom/concepts/identities-and-namespaces.md#identity-modes). Naming `trust-network` is also what lets a non-loopback `LOOM_BIND_HOST` start with sign-up open, so anyone who reaches the port can then sign up. Before making the server reachable by a team, follow the [deployment guide](https://clickhouse.github.io/loom/operations/deployment.md#identity-modes) for operator access and [TLS](https://clickhouse.github.io/loom/operations/deployment.md#proxy-and-tls). ## 3. Get a token In single-user mode a token is optional. Minting one anyway keeps your scripts working when auth is turned on, as long as they name their `org` (or send it blank), `namespace` and `agent`: a call that sends a credential, as the SDK does when it finds any ([Python SDK → Credentials](https://clickhouse.github.io/loom/reference/python-sdk.md#credentials)), gets none of them filled in, and a blank `org` is the token's own. - **Dashboard:** **Settings → Tokens → New token**. By default the token reads every namespace and repository you belong to (a restricted repository only when you add it), but no collection, and saves to the namespace you are in. _Only what you choose_ limits it to the namespaces, collections and repositories you tick (none at first); the ticked ones are searched in the order listed, and you change it with a row's arrows, or on a desktop by dragging the row's grip. The result shows the token once, with the setup for your MCP client or a script. - **CLI, as the operator:** run it inside the server container, which holds the signing secret: ```bash docker compose --project-directory "$HOME/.loom" exec loom loom mint-token --subject my-bot --org dev --namespaces local --roles writer ``` Add `--ttl-days 30` for anything that leaves your machine. `loom mint-token --help` lists the scoping options (`--namespace-role`, `--read-scope`, `--traffic-class`). - **A bot that signs itself up** (trust-network only): `loom init --agent nightly-triage --server-url http://localhost:7777` creates the namespace `agent-nightly-triage`, saves a writer token to `~/.loom/token` and the URL to `~/.loom/config.json`, and prints a claim code. A person enters it once at `/oauth/agent-claim` with a password of their own to take ownership. Save the token where the SDK, the CLI and the plugins look for it: ```bash mkdir -p ~/.loom && (umask 077; printf '%s' "" > ~/.loom/token) export LOOM_API_KEY="" ``` Claims, roles and grants are in [Tokens and grants](https://clickhouse.github.io/loom/guides/tokens-and-grants.md). ## 4. First memory over HTTP Every operation is a `POST /v1/` with a JSON body. A fresh install with the stock `.env`, from the installer or a checkout, is single-user: no token, and an everyday memory or search call (`memory.set`, `memory.get`, `memory.search` and a few more) that names no `org` or `namespace` runs in the install's own, org `dev` and namespace `local`, the ones MCP and the dashboard use; a `memory.set` that names no `agent` is recorded as agent `http`, and a search reads every agent. So the calls below need only their own fields ([which ops](https://clickhouse.github.io/loom/reference/http-api.md#request)). If a call without a token answers 401 `AUTH_INVALID`, auth is on: add `-H "Authorization: Bearer $LOOM_API_KEY"` and name your account's `org`, `namespace` and `agent`; a header with an empty token is refused. ```bash curl -sX POST http://localhost:7777/v1/memory.set -H 'Content-Type: application/json' \ -d '{"memory_key":"deploy_window","value":"Tue and Thu only; never Friday."}' curl -sX POST http://localhost:7777/v1/memory.get -H 'Content-Type: application/json' \ -d '{"memory_key":"deploy_window"}' | jq '{found, value}' curl -sX POST http://localhost:7777/v1/memory.search -H 'Content-Type: application/json' \ -d '{"query":"when can we deploy?"}' | jq '.results[0].content_excerpt' ``` The `memory.set` response carries `memory_id`, `content_sha256` and `status`. `memory.get` reads one key back: `found` says whether the key holds a live value, and `value` is the value as written. The search response carries `results` and `abstain`, which says whether the best hit is strong enough to answer from. Search reads every agent's memories in the namespace, including those added in the dashboard; name an `agent` to read only that agent's. `GET /v1/operations` lists every operation, and lets you try each one from the browser. Shapes are in the [HTTP API reference](https://clickhouse.github.io/loom/reference/http-api.md). ## 5. First memory from Python ```bash pip install clickhouse-loom # or, from a clone: pip install ./sdk/python ``` A release not yet on PyPI installs from its assets ([installing from a GitHub Release](https://clickhouse.github.io/loom/development/release-process.md#installing-from-a-github-release)). The example names `org` and `namespace`, so it works with or without the token from step 3, which the SDK sends when it finds one. On a single-user server, a call with no token may leave them out: `Agent(agent="quickstart", base_url=...)` runs in `dev`/`local` too. ```python import asyncio from loom import Agent async def main(): async with Agent( org="dev", namespace="local", agent="quickstart", # names the writer in the audit log base_url="http://localhost:7777", ) as a: # sends a credential it finds; none needed in single-user mode await a.memory.set("standup_time", "Daily standup is at 9:30.") row = await a.memory.get("standup_time") # None when the key is unset print(row["value"] if row else "unset") for hit in await a.memory.search("when is standup?"): print(hit.content_excerpt) asyncio.run(main()) ``` Where the SDK finds its server and credential is in [Python SDK → Credentials](https://clickhouse.github.io/loom/reference/python-sdk.md#credentials). The TypeScript SDK installs with `npm install @clickhouse/loom` ([from a clone](https://clickhouse.github.io/loom/reference/typescript-sdk.md#install)). The client surfaces are in the [Python SDK](https://clickhouse.github.io/loom/reference/python-sdk.md) and [TypeScript SDK](https://clickhouse.github.io/loom/reference/typescript-sdk.md) references. ## 6. First memory from a coding agent If the installer connected your agent, skip to the last paragraph of this section. For Claude Code, install the Loom plugin. It adds hooks (context at session start, capture at the end) and skills on top of the MCP tools: ```bash claude plugin marketplace add ClickHouse/loom claude plugin install loom@loom ``` On a single-user server on this machine, keep the default server URL and leave the token empty when Claude Code asks for the plugin's settings. Without the plugin, add the MCP entry directly: ```bash claude mcp add --transport http loom "http://localhost:7777/mcp?toolset=coding" -s user ``` Both reach a single-user server on this machine without a token of yours. With auth on, create a token in **Settings → Tokens** and choose **Claude Code** under **Connect**: the dialog shows the setup for the plugin and for a `claude mcp add` entry, with the token in it. Then: 1. Overwrite `~/.loom/config.json` with the dialog's `config.json` block plus the namespace the token saves to (`"namespace": ""`): an `org` or `namespace` a single-user session left in the file outranks the token, and without one the plugin writes to the token's first namespace other than `*`, or `coding`. Add back any `code_repo` or `agent` you had set. 2. Leave `LOOM_API_KEY` (step 3 may have exported one: `unset LOOM_API_KEY`) and the plugin's `token` option empty, or set them to the new token: the plugin reads both before that file ([configuration](https://clickhouse.github.io/loom/features/integrations/claude-code.md#configuration)). 3. Remove a `claude mcp add` entry made without a token (`claude mcp remove loom -s user`) before running the dialog's command, which refuses a name that exists. `loom install --client ` writes the MCP entry for Claude Desktop, Claude Code, Codex, Cursor, VS Code, Cline, Windsurf and Zed; with no flag it wires every client it finds. [Connect a coding agent](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md) covers Codex and Cursor. Start a new session and say _"Remember that I never deploy on Friday."_ The agent calls `loom_memory_set`. In a fresh session, ask _"When can I deploy?"_ The agent calls `loom_search` and answers from the stored fact. The model decides when to save and search from the tool descriptions; there are no trigger words. ## 7. See it in the dashboard | Page | Shows | | ----------------- | ------------------------------------------------- | | `/memories` | The namespace's memories, with writer and time. | | `/admin/activity` | The audit log: which agent wrote which key, when. | | `/admin/usage` | Cost per operation and agent. | The rest is in [Dashboard](https://clickhouse.github.io/loom/features/dashboard.md). ## Optional: index a repository on the released stack Adding a repository from the dashboard is refused on this stack with `SANDBOX_UNAVAILABLE`: it needs the [repository-job sandbox](https://clickhouse.github.io/loom/operations/code-index-operations.md#repository-job-sandbox), which this stack leaves off because Docker's default confinement blocks it. Only list repositories you trust: their build steps (TypeScript installs, Go module downloads, CMake) run without the sandbox, in the `loom` container, whose environment holds the JWT secret and provider keys. `LOOM_CODE_REPOS_JSON` is one line holding every repository as a JSON array ([Configure the entry](https://clickhouse.github.io/loom/operations/code-index-operations.md#configure-the-entry)). In `~/.loom/.env`, uncomment that line (add it if it is missing) and edit it. `dev` is the org a single-user stack runs as, and the org multi-user accounts sign up into (`LOOM_SELF_SIGNUP_ORG`); if you changed that, use your account's org here and in `--org` below: ```dotenv LOOM_CODE_REPOS_JSON=[{"org":"dev","namespace":"code-myrepo","repo":"acme/myrepo","git_branch":"main","public":true}] ``` - Loom picks the compiler-tier indexers from the repository's build files (`go.mod`, `package.json`, `pyproject.toml`, a root `CMakeLists.txt`, …); `scip_producer` names one instead ([Compiler tier](https://clickhouse.github.io/loom/operations/code-index-operations.md#compiler-tier)). C/C++ also needs a configured build tree, so without one a C/C++ repository gets only the parser tier. - For a private repository, add a GitHub token (`LOOM_CODE_GITHUB_TOKEN` in `.env`, or `github_token` on the entry) and drop `"public":true`: it is for repositories public on their host, and lets anyone in the org join the index as a reader at once from **Repositories**. Decide before the first `loom code init`: it sets the namespace's join policy from `public`, and editing `public` later does not change it (a maintainer changes it with `code.repo.set_access`). Then recreate the `loom` container and run the first index: ```bash docker compose --project-directory "$HOME/.loom" up -d --force-recreate loom docker compose --project-directory "$HOME/.loom" exec loom loom code init --namespace code-myrepo --org dev docker compose --project-directory "$HOME/.loom" exec loom loom code backfill --namespace code-myrepo --org dev --embed ``` `backfill --embed` runs on the released stack's local embedder. Under a fake embedder (a checkout's `.env`, or an upgraded install that still names `fake-ngram-*`) it refuses and exits; to try the wiring anyway, add `--allow-fake-embedder`. Choosing a model later in **Settings → Models & keys** re-embeds the code index; after an `.env` change it needs its own re-embedding, as [Models and keys](https://clickhouse.github.io/loom/getting-started/models-and-keys.md) describes. In multi-user mode a listed repository has no maintainer, so without `"public":true` a reader's join request waits with nobody to approve it. After `loom code init`, give someone access as the operator with a token minted in the container: `admin` makes them the maintainer, who approves readers from then on, and `viewer` lets them read. The subject is the namespace the account signed up with (its login handle, the subject of its tokens); a self-signup email is never verified, so a grant to it admits no one: ```bash ADMIN_TOKEN=$(docker compose --project-directory "$HOME/.loom" exec -T loom loom mint-token --subject operator --org dev --namespaces code-myrepo --roles admin --ttl-days 1) curl -s http://localhost:7777/v1/grants.create -H "Authorization: Bearer $ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d '{"org":"dev","namespace":"code-myrepo","subject":"your-namespace","role":"admin"}' ``` ## Optional: ask a question about a document ```bash loom demo ``` `loom demo` uploads this repository's `AGENTS.md`, ingests it with `knowledge.ingest` (the stack's reference extractor parses it) and asks a question about it with `knowledge.synthesize`, printing the answer and the chunks it cited. With the fake LLM it prints the top `knowledge.search` hits instead. `--file` and `--question` point it at your own document. ## Common problems | Symptom | Cause and fix | | ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `set_from_messages` writes no facts | The LLM is the fake one. Follow [Models and keys](https://clickhouse.github.io/loom/getting-started/models-and-keys.md) to configure one. | | Search results look arbitrary | The embedder is a fake one (`fake-ngram-*` in `.env`). Switch it as [Models and keys](https://clickhouse.github.io/loom/getting-started/models-and-keys.md) says, which re-embeds what is stored. Do not delete your volumes to change models. | | 401 `AUTH_INVALID` after turning auth on | No token, or one signed with another `LOOM_JWT_SECRET`. Re-mint with the Compose command in [Get a token](https://clickhouse.github.io/loom/getting-started/quickstart.md#3-get-a-token). | | The dashboard returns 503 | The server runs from source (`make server`) without a built frontend. Run `make frontend-build`. | | Released stack does not become healthy | Run `docker compose --project-directory "$HOME/.loom" logs --tail 100 loom clickhouse`. See [Troubleshooting](https://clickhouse.github.io/loom/operations/troubleshooting.md). | | Adding a repository answers `SANDBOX_UNAVAILABLE` | The stack runs without the repository-job sandbox. Index the repository through `LOOM_CODE_REPOS_JSON` ([Index a repository](https://clickhouse.github.io/loom/getting-started/quickstart.md#optional-index-a-repository-on-the-released-stack)). | | The installer says port `7777` is in use | Another program, or another Loom, listens there. Stop it, or run the install command again with `LOOM_PORT=7778` in front of `sh loom-install.sh`; `.env` keeps the port. | | The installer says port `7777` does not answer | A program holds it without answering, or `LOOM_BIND_HOST`, `LOOM_PORT` or a firewall keeps this host from reaching it. | | The installer could not check port `7777` | Curl failed for the reason its exit code names; check `LOOM_BIND_HOST` and `LOOM_PORT`. | | Under Docker Desktop on WSL, the install stops with `Ports are not available` | Windows holds or reserves the port, which the installer's check inside WSL cannot see. `netsh interface ipv4 show excludedportrange protocol=tcp` lists the reserved ranges; run the install command again with a `LOOM_PORT` outside them in front of `sh loom-install.sh`. | | A clone's port `8123` or `9000` is taken | Remap the ClickHouse host ports ([local stack](https://clickhouse.github.io/loom/operations/deployment.md#local-stack-from-a-clone)). | | The installer could not pull the images | Docker reaches the registry itself: behind a proxy, give the Docker daemon the [proxy settings](https://docs.docker.com/engine/daemon/proxy/). | ## Next - [Evaluate Loom](https://clickhouse.github.io/loom/getting-started/evaluate.md): turn this smoke test into a workload-specific quality, latency and cost evaluation. - [Connect a coding agent](https://clickhouse.github.io/loom/getting-started/connect-a-coding-agent.md): Claude Code, Codex or Cursor against this stack. - [Your first agent](https://clickhouse.github.io/loom/getting-started/first-agent.md): identity, abstain handling, sessions, context packs and runs from the SDK. - [Deployment](https://clickhouse.github.io/loom/operations/deployment.md): running Loom for a team. --- # Set up Loom with your coding agent > Source: https://clickhouse.github.io/loom/getting-started/agent-setup/ (Markdown: https://clickhouse.github.io/loom/getting-started/agent-setup.md) This page is for a coding agent, Claude Code or Codex, asked to install Loom on the machine it runs on and to prove that the install works. It names what to run, what not to do, what each check must see, and the report to give. Every step is written to be run as is; the steps a person has to take are marked. Each release attaches this page as plain text, with its own version and commit filled in: ```text https://github.com/ClickHouse/loom/releases/latest/download/agent-setup.txt https://github.com/ClickHouse/loom/releases/download/v/agent-setup.txt ``` ## Pins This copy names no release, so its commands do not run as they are. Follow the `agent-setup.txt` of the release you were asked to install, where every `` and `` below is filled in; with no release named, follow the latest one's: `https://github.com/ClickHouse/loom/releases/latest/download/agent-setup.txt`. ## Rules - Do not use `sudo`, and do not install or change system packages. Docker with the compose plugin must already run for your user, and `curl`, `tar`, `python3`, Node.js 18 or newer (the plugin's hooks run on it) and your own CLI (`claude` or `codex`) must be on `PATH`; if one is missing, stop and report it. - Set no provider key and ask for none. The stack boots and every check below passes without one. - Before step 1, stop and report it if any `LOOM_*` variable is set in your environment (the installer, the plugin and the SDKs all read them: a port, a bind address, another server), if `~/.loom` exists, or if your CLI already knows a `loom`: a plugin (`claude plugin list`, `codex plugin list`), a marketplace (`claude plugin marketplace list`, `codex plugin marketplace list`) or an MCP server (`claude mcp list`, `codex mcp list`). Whether to change any of them is the person's call. With none of them, the stack answers on `localhost:7777` only and the plugin connects to it. - Never weaken a check to make it pass: do not edit the check script, skip a step, change what a check expects, or report `PASS` for a step that did not run. A `FAIL` line is a correct report. ## 1. Install the release ```sh mkdir -p ~/loom-agent-setup curl -fsSL "https://github.com/ClickHouse/loom/releases/download/v/install.sh" \ -o ~/loom-agent-setup/loom-install.sh && sh ~/loom-agent-setup/loom-install.sh \ && rm ~/loom-agent-setup/loom-install.sh ``` The installer verifies the bundle's SHA-256, installs under `~/.loom`, starts the stack and waits for the server's `/v1/health`. If the command fails, the report is its `install` line alone; do not go on. The [quickstart](https://clickhouse.github.io/loom/getting-started/quickstart.md#1-boot-the-stack) says what the installer does in full. ## 2. Install the plugin The plugin comes from the same release's source, so it matches the server: ```sh curl -fsSL "https://github.com/ClickHouse/loom/archive/refs/tags/v.tar.gz" \ | tar -xz -C ~/loom-agent-setup ``` Keep `~/loom-agent-setup`: it is the marketplace's source. Then, for Claude Code: ```sh claude plugin marketplace add ~/loom-agent-setup/loom- claude plugin install loom@loom ``` or for Codex: ```sh codex plugin marketplace add ~/loom-agent-setup/loom- codex plugin add loom@loom ``` The installed stack is single-user, so the plugin connects to `http://localhost:7777` with no token. **The person** now starts a new session of the agent (Codex: an interactive `codex` run, where they also approve the plugin's hooks), gives it this text again and says "continue at step 3". Steps 3 and 4 run in that session. ## 3. Run the check Save this script as `loom-setup-check.py` and run `python3 loom-setup-check.py`. It uses only the Python standard library and talks to Loom over HTTP. It writes into the namespace `agent-setup-check`, tags what it writes with its own run, searches only that run's rows, and deletes them when it is done. ```python title="loom-setup-check.py" import json import os import pathlib import random import sys import urllib.error import urllib.parse import urllib.request PIN_VERSION = "" PIN_COMMIT = "" # Loom's own tests point the check elsewhere; an agent sets neither. BASE = os.environ.get("SETUP_CHECK_URL", "http://localhost:7777") WHO = {"org": os.environ.get("SETUP_CHECK_ORG", "dev"), "namespace": "agent-setup-check"} AGENT = "agent-setup" SESSION = "setup-1" RUN = f"{random.randrange(100_000):05d}" TAGS = [f"setup-check:{RUN}"] FACT = f"The setup check orchard has {RUN} pear trees." KEY = f"setup/check/orchard-{RUN}" DOC = f"setup-check://irrigation-{RUN}" NOTE = f"Irrigation valve {RUN} opens at five in the morning, every second day." LOOPBACK = ("localhost", "127.0.0.1", "::1") checks = [] def call(op, **body): req = urllib.request.Request( f"{BASE}/v1/{op}", data=json.dumps({**WHO, **body}).encode(), headers={"Content-Type": "application/json"}, ) try: with urllib.request.urlopen(req, timeout=60) as resp: return json.load(resp) except urllib.error.HTTPError as e: raise RuntimeError(f"{op} answered {e.code}: {e.read()[:200].decode(errors='replace')}") def check(name, probe): try: grade, detail = probe() except Exception as e: # any failure is this check's FAIL line grade, detail = "FAIL", f"{type(e).__name__}: {e}" checks.append((name, grade, detail)) return grade == "PASS" def hit_problem(hits, field, wanted, text_field): """Why the first hit is not the row this run wrote, or None when it is.""" if not hits["results"]: return "no hits" top = hits["results"][0] if top[field] != wanted: return f"the first hit is {top[field]}, not {wanted}" if RUN not in top[text_field]: return f"{wanted} came back without its text" return None def bind_host(): """LOOM_BIND_HOST from the installed stack's .env, or "" when it sets none.""" env = pathlib.Path.home() / ".loom" / ".env" lines = env.read_text().splitlines() if env.exists() else [] values = [line.split("=", 1)[1].strip() for line in lines if line.startswith("LOOM_BIND_HOST=")] return values[-1] if values else "" def server(): if PIN_VERSION.startswith("<"): return "FAIL", "this copy names no release: use the release's agent-setup.txt" if urllib.parse.urlsplit(BASE).hostname not in LOOPBACK: return "FAIL", f"{BASE} is not a loopback address" host = bind_host() if host not in ("", *LOOPBACK): return "FAIL", f"~/.loom/.env publishes Loom on {host}" with urllib.request.urlopen(f"{BASE}/v1/health", timeout=30) as resp: health = json.load(resp) version, commit = health.get("version"), health.get("build_commit") # A release candidate's server reports the final version it leads to. final = PIN_VERSION.split("-")[0] if (version, commit) != (final, PIN_COMMIT): return "FAIL", f"serves {version} at {commit}; the pins name {final} at {PIN_COMMIT}" return "PASS", f"{BASE} serves {version} at {commit[:12]}" def memory(): call("memory.set", agent=AGENT, memory_key=KEY, value=FACT, tags=TAGS, session_id=SESSION) # A new session reads with no session_id: it sees what the last one wrote. hits = call( "memory.search", query="how many pear trees does the setup check orchard have", tags=TAGS ) problem = hit_problem(hits, "memory_key", KEY, "content_excerpt") if problem: return "FAIL", problem similarity = float(hits["results"][0]["similarity"]) return "PASS", f"a new session finds {KEY} (similarity {similarity:.2f})" def knowledge(): call("knowledge.add", agent=AGENT, source_uri=DOC, title="Irrigation", content=NOTE, tags=TAGS) hits = call("knowledge.search", query="when does the irrigation valve open", tags=TAGS) problem = hit_problem(hits, "source_uri", DOC, "content") if problem: return "FAIL", problem similarity = float(hits["results"][0]["similarity"]) return "PASS", f"knowledge.search finds {DOC} (similarity {similarity:.2f})" def cleanup(): gone = call("memory.delete", agent=AGENT, memory_key=KEY, session_id=SESSION)["found"] dropped = call("knowledge.delete", source_uri=DOC)["deleted"] if not (gone and dropped): return "FAIL", f"memory deleted: {gone}, document deleted: {dropped}" return "PASS", "deleted the memory and the document it wrote" # Nothing is written to a server the pins or the loopback rule refuse. if check("server", server): check("memory", memory) check("knowledge", knowledge) check("cleanup", cleanup) width = max(len(name) for name, _, _ in checks) for name, grade, detail in checks: print(f" [{grade}] {name:<{width}} {detail}") failed = [name for name, grade, _ in checks if grade == "FAIL"] if failed: print(f"\n{len(failed)} check(s) failed: {', '.join(failed)}") sys.exit(1) print(f"\nAll {len(checks)} checks passed.") ``` What each check must see: | Check | Passes when | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `server` | The URL and `~/.loom/.env`'s `LOOM_BIND_HOST` are loopback, and `/v1/health` names the pinned version (a release candidate's final) and commit. | | `memory` | A fact written in one session is the first hit of a search that names no session, as the next session's would be, with its own text and a numeric similarity. | | `knowledge` | A document added with `knowledge.add` is the first hit of `knowledge.search`, with its own text and a numeric similarity. | | `cleanup` | The memory and the document the script wrote are deleted. | When `server` fails, the script writes nothing and runs no other check. ## 4. Check the agent's connection Call your `loom_search` tool once, with the query `agent setup check`, and run `claude plugin list` (or `codex plugin list`). The line is `[PASS] agent loom_search answered` when the tool answers without an error and the list shows `loom@loom` enabled, otherwise `[FAIL] agent `. ## Report Give the script's lines, then the `agent` line, then one summary that counts the `agent` line too, in `loom doctor`'s format: ```text [PASS] server http://localhost:7777 serves 1.2.3 at 0123456789ab [PASS] memory a new session finds setup/check/orchard-04211 (similarity 0.91) [PASS] knowledge knowledge.search finds setup-check://irrigation-04211 (similarity 0.77) [PASS] cleanup deleted the memory and the document it wrote [PASS] agent loom_search answered All 5 checks passed. ``` On a failure the summary reads ` check(s) failed: `, and the first `FAIL` line is the one to fix. A step that stops the run is one line and its summary: - a rule that stops you before step 1: `[FAIL] setup `; - a failed install: `[FAIL] install `; - a failed step 2: `[FAIL] plugin `; each followed by `1 check(s) failed: `. --- # Configure models and API keys > Source: https://clickhouse.github.io/loom/getting-started/models-and-keys/ (Markdown: https://clickhouse.github.io/loom/getting-started/models-and-keys.md) Your coding agent, Loom's internal LLM and Loom's embedding model are separate choices. You can connect Claude Code to a Loom server using OpenAI models, or Codex to a server using Claude. The agent does not need to use the same provider as Loom. | Choice | What it does | Where to configure it | | ------------------------------------------------ | ---------------------------------------------------------------------- | ---------------------------------------------------------- | | Your agent: Claude Code, Codex or another client | Runs your coding or assistant workflow and calls Loom's tools. | In that client; connect it with a Loom token. | | Loom's LLM | Extracts facts, summarizes sessions and generates answers inside Loom. | The server's provider, model and API key settings. | | Loom's embedding model | Makes memories, documents and code searchable by meaning. | The server's embedding model, dimensions and provider key. | `LOOM_API_KEY` is a token for accessing **Loom**. It is different from `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` and `GEMINI_API_KEY`, which pay for the server's model calls. Keep provider keys on the server; connecting an agent does not transfer its subscription credentials or API keys to Loom. ## What works without provider keys? The released Docker installation embeds with `local` (bge-small, 384 dimensions, on CPU from files the image carries) and selects the `fake-echo-1` LLM, unless the installer set a local model server ([a model server on your machine](https://clickhouse.github.io/loom/getting-started/models-and-keys.md#a-model-server-on-your-machine-no-api-key)). It boots without provider keys or network: storage, the dashboard, structural code indexing and semantic search work, though a hosted embedder retrieves better on paraphrase-heavy questions. The fake LLM does not produce meaningful extracted facts, summaries or generated answers. A checkout's `.env` selects `fake-ngram-3072` embeddings, which match on character overlap. An upgraded installation keeps the embedder its `~/.loom/.env` names; switching it to `local` is a model change like any other, and re-embeds what is stored. For stronger retrieval over paraphrase-heavy content, configure a hosted embedding provider. For automatic fact extraction, summaries and generated answers, configure an LLM too. An LLM alone does not replace an embedding model; an embedding model alone does not generate answers. With no embedding configured, a checkout's server uses its fake embedder; it does not skip embedding. Some hosted providers also fall back to fakes when their key is missing. Check `fake_embedding`, `fake_llm`, `providers_degraded` and `problems` in `/v1/health`, then save and retrieve a test fact and try an answer. An invalid key or inaccessible model can still cause provider calls to fail; a running server is not proof of working model access. Use `LOOM_REQUIRE_REAL_PROVIDERS=true` in environment-based deployments to make fake or keyless providers a degraded-health condition. ## Choose a setup | What your organization has | Loom LLM | Embeddings | | ---------------------------------------------- | ------------------------- | ------------------------------------------------------ | | OpenAI API access | An available OpenAI model | OpenAI; one API key can serve both roles. | | Anthropic plus an approved embedding provider | An available Claude model | OpenAI, Gemini or a compatible endpoint, with its key. | | Anthropic API access only | An available Claude model | Add an embedding provider and its API key. | | No provider key; a model server on the machine | A local model (`local`) | The `local` embedder the image carries. | **One OpenAI API key is enough for both roles; an Anthropic key is not required.** In Models & keys, choose one main LLM and leave per-job overrides blank. All enabled LLM jobs use that model, including verification and judging. Advanced overrides are optional and require another provider's key only when you explicitly select one. Choose OpenAI, Google Gemini or an **OpenAI-compatible embedding endpoint**. OpenAI embeddings use an embedding model such as `text-embedding-3-large`, not a GPT chat model. Claude can handle Loom's LLM jobs, but requires a separate embedding provider and key. Anthropic does not provide its own embedding model; Voyage is not a built-in Loom provider. [Anthropic's embedding guidance](https://platform.claude.com/docs/en/build-with-claude/embeddings) Loom's internal model jobs use provider API keys, not coding-agent CLI logins or subscription credentials. Your external agent's choice of provider is independent. ## Set the server configuration ### Recommended: use the dashboard As a platform administrator, open **Settings → Models & keys** (also available during Setup). Choose your LLM and embedding provider, enter their model IDs and API keys, then click **Test & apply**. Leave per-job overrides blank to use one LLM for every enabled job. Saved dashboard settings override the environment and are encrypted in ClickHouse. **No Docker restart or Compose command is needed.** Key and LLM updates apply live; embedding changes start the rebuild described below. **Use environment configuration** removes those overrides and restores this server's startup settings; it does not reload an edited `.env` file. Changing the embedding space requires a rebuild. The dashboard's embedding choices are hosted providers. To keep the released stack's `local` embedder and add a real LLM, set only the LLM lines in `~/.loom/.env` ([Claude for the LLM](https://clickhouse.github.io/loom/getting-started/models-and-keys.md#claude-for-the-llm) shows them) and recreate the container: the embedding space is unchanged, so nothing is re-embedded. For dashboard embedding changes, Loom queues that rebuild automatically and keeps data-plane requests paused until verification, publication and cleanup finish. Models & keys shows the phase and progress; a failed job stays paused and offers retry from its saved cursor. Keep the ClickHouse volume and the original `LOOM_JWT_SECRET`, which seals the stored provider keys, when recreating containers. Rebuilds make provider calls and may take time and incur fees proportional to the indexed content. The rebuild publishes to local MergeTree tables only. The first embedding choice on an install with no stored vectors, such as a fresh install that has stored nothing yet, switches directly with no rebuild. Once vectors exist on replicated or shared storage, such as ClickHouse Cloud, Models & keys locks the embedding fields and the server refuses an embedding change before pausing anything: change the embedding model in the environment instead and re-embed each index as described above. Lost remote-process recovery remains a documented limitation in [provider reconfiguration](https://clickhouse.github.io/loom/architecture/provider-reconfiguration.md). The token settings are separate: they configure agent access to Loom, not model-provider credentials. ### Alternative: configure the environment For automated deployments, edit `~/.loom/.env` (or the directory selected with `LOOM_INSTALL_DIR`). For a source checkout, edit the repository's `.env`. Replace existing entries instead of appending duplicate keys. These settings are the startup baseline; saved dashboard overrides take precedence. Environment edits require container recreation or a native server restart, as described below. ### OpenAI for both roles ```dotenv LOOM_LLM_PROVIDER=openai LOOM_LLM_MODEL=gpt-6-luna OPENAI_API_KEY= LOOM_EMBEDDING_PROVIDER=openai LOOM_EMBEDDING_MODEL=text-embedding-3-large LOOM_EMBEDDING_DIM=3072 LOOM_REQUIRE_REAL_PROVIDERS=true ``` The examples use Loom's shipped model defaults. Choose a model enabled for your API account. Setting a key alone does not change the stock installer configuration, which names its providers explicitly (the local embedder and the fake LLM). ### Claude for the LLM ```dotenv LOOM_LLM_PROVIDER=anthropic LOOM_LLM_MODEL=claude-haiku-4-5 ANTHROPIC_API_KEY= LOOM_REQUIRE_REAL_PROVIDERS=true ``` Then choose an embedding configuration: ```dotenv # OpenAI embeddings LOOM_EMBEDDING_PROVIDER=openai LOOM_EMBEDDING_MODEL=text-embedding-3-large LOOM_EMBEDDING_DIM=3072 OPENAI_API_KEY= ``` ```dotenv # Or Gemini embeddings LOOM_EMBEDDING_PROVIDER=gemini LOOM_EMBEDDING_MODEL=gemini-embedding-001 LOOM_EMBEDDING_DIM=3072 GEMINI_API_KEY= ``` For another service that implements the OpenAI embeddings API, configure its API root, model ID and native output width: ```dotenv LOOM_EMBEDDING_PROVIDER=compatible LOOM_EMBEDDING_BASE_URL=https://embedding-provider.example/v1 LOOM_EMBEDDING_MODEL=your-embedding-model LOOM_EMBEDDING_DIM=768 LOOM_EMBEDDING_API_KEY= # Enable only if the endpoint accepts the optional dimensions parameter: LOOM_EMBEDDING_SEND_DIMENSIONS=false ``` The base URL is the API root; Loom appends `/embeddings`. Use HTTPS for remote services. HTTP is supported for private self-hosted endpoints, but transmits the key and content unencrypted. Compatibility requires indexed float-array responses with the configured width; a provider using a different API needs its own adapter. Changing the endpoint, model or width requires rebuilding existing embeddings. If your organization currently has only Anthropic API access, obtain access to an embedding provider before enabling semantic search. ### Keep every LLM job on your chosen provider Loom supports different models for different jobs, but you do not need accounts with multiple LLM providers. Add these settings **after** `LOOM_LLM_MODEL` to have the normally provider-specific jobs use your selected model: ```dotenv LOOM_MEMORY_EXTRACT_VERIFY_MODELS=${LOOM_LLM_MODEL} LOOM_SESSION_BRIEF_LLM_MODEL= LOOM_CODE_BISECT_LLM_RERANK_MODEL= LOOM_TRACKER_DUPCHECK_RERANK_MODEL= LOOM_RETRIEVAL_JUDGE_MODEL= ``` Compose and Loom's dotenv loader expand the verifier's model from the earlier entry. The empty values above inherit the main LLM. This keeps the capture verifier enabled; an empty verifier list would disable it. Remove any other cross-provider model overrides you added previously, including fallback, synthesis and reranking models. These settings matter: some shipped advanced defaults name Claude or OpenAI models explicitly, and an override with no provider key can fall back to the fake client. Using one model across jobs is a convenience configuration, not the configuration behind every published benchmark. Tune individual jobs later using [models per job](https://clickhouse.github.io/loom/operations/deployment.md#models-per-job). ### A model server on your machine, no API key `LOOM_LLM_PROVIDER=local` runs Loom's LLM jobs on a chat-completions server you run: [Ollama](https://ollama.com), llama.cpp's `llama-server`, or any server that speaks the OpenAI chat-completions format. Loom sends it no key. With the released stack's `local` embedder, nothing leaves the machine. A new install run from a terminal offers this when Ollama answers on `localhost:11434`; `LOOM_LOCAL_MODELS=ollama` in front of `sh loom-install.sh` sets it without asking, on a new install or an existing one, and `LOOM_LOCAL_MODELS=off` never offers it. It writes: ```dotenv LOOM_LLM_PROVIDER=local LOOM_LLM_MODEL=qwen2.5:3b LOOM_LOCAL_LLM_BASE_URL=http://host.docker.internal:11434/v1 LOOM_ASSISTANT_PROVIDER=local LOOM_ASSISTANT_MODEL=qwen2.5:3b LOOM_ASSISTANT_ANSWER_CEILING_USD=0 LOOM_MEMORY_EXTRACT_VERIFY_MODELS= LOOM_MEMORY_CONFLICT_RESOLVER_ENABLED=false LOOM_LLM_REQUEST_TIMEOUT_S=900 LOOM_LLM_RETRY_ATTEMPTS=1 LOOM_EXTRACTION_QUEUE_MAX_BATCHES_PER_PASS=1 LOOM_EXTRACTION_QUEUE_MAX_PASS_SECONDS=3600 LOOM_EXTRACTION_QUEUE_MAX_ATTEMPTS=2 LOOM_ENTITY_EXTRACT_ENABLED=false LOOM_SESSION_BRIEF_LLM_MODEL= LOOM_CODE_BISECT_LLM_RERANK_MODEL= LOOM_TRACKER_DUPCHECK_RERANK_MODEL= LOOM_RETRIEVAL_JUDGE_MODEL= ``` The preset turns off the two jobs that remove facts on a model's judgment, whose thresholds were set on hosted models: the capture verifier, which drops a fact every verifier model calls not durable, and the background [conflict resolver](https://clickhouse.github.io/loom/features/memory/conflicts-and-refusal.md#the-conflict-resolver), which retires an older row it judges replaced. A small model's misjudgment there would lose facts. Writes still flag conflicts for the Review queue; set `LOOM_MEMORY_CONFLICT_RESOLVER_ENABLED=true` with the key for `LOOM_CONFLICT_RESOLVER_MODEL` (`gpt-6.1-sol`, OpenAI) to judge them. On an existing install, a configuration saved in Models & keys wins over `.env`: after the restart, choose **Use environment configuration** there for the preset to apply. Pull the model first (`ollama pull qwen2.5:3b`, about 2 GB), and give Ollama a context window that holds Loom's prompts: extraction sends about 15,000 tokens, and Ollama cuts a longer prompt to its window without an error, so run it with `OLLAMA_CONTEXT_LENGTH=32768` (the window `qwen2.5` takes). A CPU can take minutes to read a prompt that long, so the preset waits `LOOM_LLM_REQUEST_TIMEOUT_S=900` seconds for a reply rather than the hosted default of 300, and tries it once (`LOOM_LLM_RETRY_ATTEMPTS=1`): a server that timed out answers no faster a second time. The worker that replays the captures the coding plugins send gives a pass a deadline, so the preset lets a deferred-extraction pass run an hour on one batch (`LOOM_EXTRACTION_QUEUE_MAX_PASS_SECONDS`, by default four of its intervals, at least 60 s and at most 900; a batch's lease is kept at least a minute longer). On a `local` provider a capture cut short (at that deadline, or by Loom shutting down) or whose call ran out its timeout spends one of its `LOOM_EXTRACTION_QUEUE_MAX_ATTEMPTS=2` attempts, so a session too long for the machine ends `failed` (`memory.extraction_status` shows `cancelled` or the timeout) instead of holding the queue: a long session can need several calls in a row, each up to the timeout. On a server that answers one request at a time the timeout also counts the wait behind other calls, and the assistant and `memory.ask` share the server, so heavy use of them during a capture can fail it too; raise `LOOM_EXTRACTION_QUEUE_MAX_ATTEMPTS` if it does. Entity tagging (`LOOM_ENTITY_EXTRACT_ENABLED`) is off: it would queue behind extraction on a server that answers one request at a time, and its pass, cut at its deadline, writes nothing, so search does without entity matches. The other workers that call a model are not tuned for a local one: observe extraction (off by default), whose pass of up to 10 sessions is cut at 900 s and starts again from the newest session, and conversation titles and structured profiles, which run once conversation logging or a profile schema is set up. For llama.cpp, point `LOOM_LOCAL_LLM_BASE_URL` at its `/v1` root (`http://host.docker.internal:8080/v1`) and set `LOOM_LLM_MODEL` to the name it serves; for a local assistant, also set `LIBRECHAT_LOCAL_LLM_BASE_URL` to that root and `LOOM_ASSISTANT_MODEL` to that name. A server run from a checkout reaches the model at `localhost`, the setting's default (`http://localhost:11434/v1`). A per-job model whose name starts with `gpt`, `o1`, `o3`, `o4`, `chatgpt` or `claude` goes to that hosted provider whatever `LOOM_LLM_PROVIDER` says; a local model named that way (`gpt-oss`) runs locally as `LOOM_LLM_MODEL`, and in a per-job setting only when that setting names the same model. - **Reaching the host.** The containers reach the machine as `host.docker.internal`. Docker Desktop forwards that to services on loopback; with Docker Engine on Linux, Ollama must listen beyond loopback (`OLLAMA_HOST=0.0.0.0`, with port 11434 closed to the network). Under the Helm chart with `networkPolicy.enabled`, add the server's port to `networkPolicy.extraEgress`. - **The assistant.** `LOOM_ASSISTANT_PROVIDER=local` moves the assistant's agent to LibreChat's `local` endpoint, which calls `LIBRECHAT_LOCAL_LLM_BASE_URL` (default `http://host.docker.internal:11434/v1`), and needs `LOOM_ASSISTANT_MODEL`. The assistant answers by calling Loom's tools, so pick a model the server runs with tool calls (Ollama marks them). A local model has no price, so each answer counts `LOOM_ASSISTANT_ANSWER_CEILING_USD` ($0.10 by default) against the assistant's share of the namespace's daily model cost cap, as any unpriced model does: at the default cap and share, five answers a day. The preset sets the ceiling to `0`, so local answers cost nothing; it is also what an answer cut before its usage arrived costs, so keep it at its default when a hosted model serves the assistant. Under the Helm chart, add `LOCAL_LLM_BASE_URL` to the assistant's Secret; the chart's policy lets LibreChat reach only DNS, its database, Loom and TCP 443, so the model server needs a NetworkPolicy of your own that selects the LibreChat pod. - **Quality and speed.** Loom's prompts and thresholds are tuned on hosted models, and no published benchmark covers a local one, so treat a small model as a way to try Loom with no key or to keep everything on the machine, not as an equal: expect extraction, `memory.ask` answers and the assistant's tool use to be weaker, by an amount nobody has measured. A small model on a CPU is slow on long prompts such as extraction, so a busy capture queue backs up; a GPU, or a larger model on a machine with the memory for it, helps. For a team server, a hosted model answers better. ### Through an inference gateway A gateway that speaks the OpenAI or Anthropic API in front of the providers (for budgets, cost attribution or one key for many services) takes Loom's calls once its API roots point there; the model names stay what the gateway serves: ```bash LOOM_OPENAI_BASE_URL=https://gateway.example/v1 # chat and embeddings LOOM_ANTHROPIC_BASE_URL=https://gateway.example # Messages: /v1/messages OPENAI_API_KEY= ``` Off its default, the Anthropic key is also sent as `Authorization: Bearer`, the scheme gateways take. With a gateway key, require the gateway, so a wrong root can never send prompts and the key anywhere else. A token exchange requires it by itself. ```bash LOOM_INFERENCE_GATEWAY_REQUIRED=true LOOM_INFERENCE_GATEWAY_HOSTS=gateway.example # comma-separated; exact names, any case ``` While the gateway is required, model calls and the memory content Loom posts to classifiers and rerankers go only to the hosts `LOOM_INFERENCE_GATEWAY_HOSTS` lists. A host matches exactly: no wildcard, and `gateway.example` does not admit `api.gateway.example` or `gateway.example.attacker.test`; an IP address matches only if it is listed. - **At start**, Loom refuses to load with no `LOOM_INFERENCE_GATEWAY_HOSTS`, or while `LOOM_OPENAI_BASE_URL`, `LOOM_ANTHROPIC_BASE_URL`, `LOOM_EMBEDDING_BASE_URL`, `LOOM_INFERENCE_TOKEN_URL` (and `LOOM_LOCAL_LLM_BASE_URL` while `LOOM_LLM_PROVIDER=local`) or an entry of `LOOM_RERANK_ENDPOINTS_JSON` names another host, while the embedder is Gemini's, or while `LOOM_CODE_SEARCH_RERANK_DEFAULT` or `LOOM_KNOWLEDGE_SEARCH_RERANK_DEFAULT` is `builtin:cohere`. Both API roots count, the one for a provider you do not use included (their defaults are the providers' own APIs): a model named per request may be either provider's. - **At call time**, a model call to any other host is refused before it leaves, on every redirect hop too, whatever named it: a model named per request, a `LOOM_CODE_SEARCH_VECTOR_MODELS` entry. A refused call is not retried, and `/v1/health` reports the provider as failing. - **Namespaces**: `namespace.register` refuses a `classify_endpoint` that is not https at the default port of a listed host (422), and one stored before is never called: memory.set keeps the caller's type, with a warning in the log. `builtin:cohere` is refused as a rerank choice (422), and a namespace that stored it reranks with nothing, with a warning in the log. - **Dashboard**: saving a model configuration with an endpoint at another host (or the Gemini embedder) answers 422, and the model check answers `gateway_required`. A stored one that the check refuses is not applied, its serving vector settings included, and `/v1/health` reports `model_configuration_refused`. The check reads only the embedding, so a refused configuration always embeds differently from the settings in force, and the vector tables may hold that other space: every data-plane request on that server is refused (503 `GATEWAY_REQUIRED`, which neither SDK retries, with no `Retry-After`) and the server is not ready, until a configuration the check accepts replaces it or it is reset to the environment's. A rolling restart then stops at that server: replace the configuration before rolling. - **Rolling it out:** a previous build ignores both settings, so turn them on once every replica runs this one. A deployment that already exchanges a token refuses to start on this build until `LOOM_INFERENCE_GATEWAY_HOSTS` lists its gateway, and while it embeds with Gemini, names a rerank endpoint at another host or defaults code search to `builtin:cohere`; requests naming `builtin:cohere` get a 422, and namespaces that stored it or an off-gateway `classify_endpoint` stop reranking or classifying until it is changed. Set the hosts and move the embedder behind the gateway (`LOOM_EMBEDDING_PROVIDER=compatible`, `LOOM_EMBEDDING_BASE_URL` at the gateway) first, or the rollout stops at the first upgraded replica. A gateway that accepts workload identity (RFC 8693 token exchange) needs no key at all: Loom exchanges its workload's OIDC token there, renews the answer five minutes before it runs out (halfway, for one living under ten minutes), and sends it with every call. With no key to infer them from, name the provider and the models: ```bash LOOM_INFERENCE_TOKEN_URL=https://gateway.example/oauth/token # https LOOM_INFERENCE_GATEWAY_HOSTS=gateway.example # every root's host LOOM_INFERENCE_WORKLOAD_TOKEN_FILE=/var/run/secrets/tokens/gateway # rotated by the platform LOOM_INFERENCE_TOKEN_SCOPE="chat embeddings" # if the gateway asks for one LOOM_OPENAI_BASE_URL=https://gateway.example/v1 # both roots required: LOOM_ANTHROPIC_BASE_URL=https://gateway.example # the token is the gateway's LOOM_LLM_PROVIDER=openai LOOM_LLM_MODEL=gpt-6-luna LOOM_EMBEDDING_MODEL=text-embedding-3-large ``` Loom refuses to start while either provider's root, or the token endpoint, is at a host `LOOM_INFERENCE_GATEWAY_HOSTS` does not list: a model named per request may be either provider's, and the gateway's token would go there. In the dashboard, model changes are checked through the gateway too. On Kubernetes the token file is a projected service-account token with the gateway's audience; the gateway must trust the cluster's OIDC issuer and match Loom's service account (`system:serviceaccount::`), not just the issuer, since every pod of the cluster can get a token from it. A gateway that verifies the workload's token itself takes it as the bearer, with no exchange: set the bearer mode instead of a token endpoint. Loom reads the file at most once a minute, and again at once when the gateway answers 401, so a rotated token is picked up without a restart: ```bash LOOM_INFERENCE_WORKLOAD_TOKEN_MODE=bearer LOOM_INFERENCE_WORKLOAD_TOKEN_FILE=/var/run/secrets/tokens/gateway # sent as it is LOOM_INFERENCE_GATEWAY_HOSTS=gateway.example # required, as with the exchange LOOM_OPENAI_BASE_URL=https://gateway.example/v1 LOOM_ANTHROPIC_BASE_URL=https://gateway.example LOOM_EMBEDDING_PROVIDER=compatible # embeddings through the gateway too LOOM_EMBEDDING_BASE_URL=https://gateway.example/v1 ``` Name the provider and the models as in the exchange example. The token goes only to the gateway's hosts: the bearer mode requires the gateway, as the exchange does. A compatible embedding endpoint given its own `LOOM_EMBEDDING_API_KEY` sends that key instead. #### Naming the customer on every call A gateway that meters or budgets per customer reads the customer from a header. Name the headers, and every chat, embedding, rerank and classify request carries the org whose data it holds, and the feature that made it: ```bash LOOM_INFERENCE_CUSTOMER_HEADER=X-Customer-Id # the org LOOM_INFERENCE_FEATURE_HEADER=X-Feature # optional, with the customer header: hyde, memory.search, worker.embedding LOOM_INFERENCE_SYSTEM_CUSTOMER=loom # calls that belong to no org LOOM_INFERENCE_CUSTOMER_REQUIRED=true # a cell serving several customers: refuse to start without the header LOOM_INFERENCE_CUSTOMER_FORMAT='cloud:{org}' # the gateway's customer id for an org; default the org id LOOM_INFERENCE_FEATURE_MAP_JSON='{"hyde": "search", "*": "memory"}' # Loom's feature names to the gateway's ``` - A request names the org it addresses, as authorized for its token; a caller cannot name an org its token does not reach. - A background job names the org of the rows it works on, and makes one call per org: the embedding worker, `loom reembed` and a model rebuild split their batches by org. - The deployment's own calls (the startup check of the embedder, the recalibration worker's fixed pairs, the dashboard's provider check) name `LOOM_INFERENCE_SYSTEM_CUSTOMER`. - The sampled retrieval judge (`LOOM_RETRIEVAL_JUDGE_SAMPLE_RATE`) sends the judged org's query and memories, so each judged search adds a call that names that org. - With the customer header set, a model call that names no org is refused (`MODEL_CALL_UNATTRIBUTED`) instead of being sent: it is a call path that forgot its org, and the gateway would bill it to no one. - An org id is sent percent-encoded where it needs to be. Cached answers make no call: an embedding or a model answer served from Loom's cache is not metered again. The customer is still checked before the cache, so a path that names none is refused even on a hit. #### Naming the agent on every call A gateway that bills or limits each agent reads whom a call is for from headers. Map each header to a template, and every chat, embedding, rerank and classify request to the gateway carries it: ```bash LOOM_INFERENCE_IDENTITY_HEADERS_JSON='{"X-Tenant": "{tenant}", "X-Agent": "{agent}", "X-Actor": "{email}"}' LOOM_INFERENCE_GATEWAY_HOSTS=gateway.example # required: identity headers go only to these hosts LOOM_INFERENCE_TENANT_CLAIM=tenant_id # the workload trust claim {tenant} names (the default) ``` - The placeholders are `{org}`, `{namespace}`, `{subject}`, `{user_id}`, `{agent}` (the agent of a workload trust's token, `sa::`), `{tenant}` (the value that trust requires of its `LOOM_INFERENCE_TENANT_CLAIM` claim) and `{email}` (a signed-in person's address). - A header whose template renders empty is not sent: with bare placeholders, a call with no agent behind it (a person's, or the deployment's own) carries no agent header. The headers naming `{agent}` or `{tenant}` go together or not at all, so an agent never goes without its tenant. - A call that names no org is refused (`MODEL_CALL_UNATTRIBUTED`) unless it is the deployment's own: run `loom` commands that call models with `--org`, and with `--namespace` while the headers name the agent (one with no namespace is refused rather than billed to the gateway's default agent). - Background jobs name the identity of the rows they work on: the writer of a memory or of a deferred extraction, else the agent whose own namespace the row is in. A job batching rows makes one call per identity. - No other host gets the identity headers: a reranker, a classifier or a provider outside `LOOM_INFERENCE_GATEWAY_HOSTS` never learns whom a call is for. - `LOOM_INFERENCE_IDENTITY_REFUSALS_JSON` lists the answers your gateway gives for one identity, each by its status and the exact `code`, `type` or `message` of its error body (under `error`). Match by `code` where the gateway gives one. The default matches an inference proxy's per-agent codes and the budget refusal of the metering behind it: ```json { "refused": [ { "status": 429, "code": "budget_exceeded" }, { "status": 429, "type": "rate_limit_error", "message": "budget exceeded" }, { "status": 403, "code": "agent_denied" }, { "status": 400, "code": "identity_incomplete" }, { "status": 400, "code": "identity_malformed" } ], "wait": [{ "status": 429, "code": "agent_request_limit" }] } ``` The second entry is matched by type and message because the metering's Anthropic-shaped answer on `/v1/messages` carries no code. - Never list an answer the gateway also gives for its own failure: `token_unavailable` (it cannot fetch a credential, which a credential-service outage also answers) or `identity_missing` (the request named no agent and the gateway has no default, a configuration problem). Listed, `/v1/health` would stop seeing the outage or the misconfiguration. Deployment-level answers stay provider failures too: `caller_*`, `scope_not_granted` (a route the token's scopes do not cover), `route_not_found`, `upstream_*`, `misconfigured`, `no_identity`. - An answer under `refused` makes that request answer 403 `MODEL_CALL_REFUSED`, and one under `wait` 503 `MODEL_CALL_DEFERRED` with the gateway's `Retry-After`; a wait of two seconds or less is waited out once first, and the call asked again. Nothing retries either answer, `/v1/health` does not read them as a provider failure, and background jobs hold that identity's calls to the same model, for `LOOM_INFERENCE_REFUSAL_COOLDOWN_SECONDS` (10 minutes by default) after a refusal and the `Retry-After` after a wait, while other work goes on. Any answer the table does not list is the provider's, as always. - How each value is found: [identity headers](https://clickhouse.github.io/loom/architecture/llm-architecture.md#identity-headers). #### Switching an org's model calls off An org admin can stop every model call for the org's data, with or without a gateway, from the dashboard's Guardrails page or over the API: ```bash curl -s -X POST "$LOOM_BASE_URL/v1/org.set_model_calls" -H "Authorization: Bearer $ORG_ADMIN_TOKEN" \ -d '{"org":"acme","enabled":false,"reason":"customer turned AI off"}' ``` - While off, no chat, embedding, rerank or classify request is sent for the org, and no cached vector or answer is served to it either: the switch is a kill switch. A request that needs one is refused (403 `MODEL_CALLS_DISABLED`): `memory.set` of a memory that is embedded on write, or in a namespace with a classify endpoint, `memory.search`, and `tracker.similar`, `tracker.duplicate_check` and `code.bisect_candidates` when they embed, judge or rerank, among them. A read by key (`memory.get`) and a delete make no model call and answer. The local embedder runs in-process and sends nothing, but it is switched off too. A call that names the org is checked even when Loom makes it for itself; only its own calls that name no org (the startup check, calibration) go on. - `"enabled": true` switches it back on, and the waiting work runs. - **A platform admin's lock.** A platform admin can add `"locked": true`: the org's own admins can then still switch it off, but switching it back on or unlocking it is refused (403, naming who locked it) until a platform admin sends `"locked": false`. A platform admin's change that sends no `locked` keeps the lock. [`org.model_calls`](https://clickhouse.github.io/loom/reference/operations/org.md#orgmodel_calls) reads the setting, the lock and who set it. - **Offboarding an org** is manual: a platform admin sends `"enabled": false, "locked": true`. The org's cached query vectors expire after 30 days. No Loom path switches an org off by itself. - Every server applies a change within `LOOM_MODEL_CALLS_POLICY_REFRESH_SECONDS` (default 10; the process that took it, on its next call). A server that cannot read the switch for longer than `LOOM_MODEL_CALLS_POLICY_MAX_STALE_SECONDS` (default 60, 15 to 300) makes no model call for any org until it can. - Switch an org off only once every replica runs a build that knows the switch: an older one keeps calling models for the org. What waits, and for how long, while an org is off: | Work | While the org is off | | --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Memory embeddings (the embedding worker) | Not picked: the memories stay, without a vector, and are embedded once the org is back on. Never dropped. | | Extraction requests (the extraction queue) | Stay pending, unclaimed. Once a request has waited `LOOM_MODEL_CALLS_OFF_MAX_WAIT_DAYS` (default 7, 1 to 29) since it was queued and since the org was switched off, it is marked failed with a `memory_drops` record, reason `model_calls_off`. The record is written synchronously first: a request whose record cannot be written stays pending. The limit stays under 30 days because the queue's 30-day TTL would remove the request first, with no record. | | Entity tagging, profile folds, period digests, conversation titles, reflections, observation distillation | Not selected; done once the org is back on. They are derived from rows that stay, so nothing is dropped. | | Projection repairs | Not scanned; retried once the org is back on. | | Pending conflict judgements | Not re-judged. One older than the sweep's hour when the org comes back is past the sweep, as a judgement lost in an outage is: the write stands as made. | | Code ingest, code summaries, code embedding recovery | The org's repositories are skipped, so its whole code index, lexical search included, stays as it was. A stub the recovery scan moved past is found by its periodic full scan. | | Code-link judgements | Left due. | | Tracker embeddings, tracker link scoring | The namespace is skipped before its cursor moves. | | Retrieval-judge samples | Dropped: a sample is not owed work. | | `loom reembed` | The org's rows are left out, counted (`skipped_switched_off`) and reported; run it again once the org is back on. `--delete-old` keeps the old vectors of every org whose rows it left. | | A model change's rebuild | Completes; the org's rows are staged without a vector and counted in the rebuild's progress (`rows_waiting`). Once the org is back on, memory embeddings, code chunks and code summaries are embedded again by their recovery workers; a row of any other table has no vector until it is written again or a later rebuild covers it. | #### Without a workload token Where the platform gives Loom no token (a VM, a container host), Loom signs its own: set an audience instead of the token file, and the gateway trusts Loom as an issuer. ```bash LOOM_INFERENCE_TOKEN_URL=https://gateway.example/oauth/token LOOM_INFERENCE_GATEWAY_HOSTS=gateway.example LOOM_INFERENCE_TOKEN_AUDIENCE=loom-inference # what the gateway's entry for Loom names LOOM_PUBLIC_BASE_URL=https://loom.example # the issuer is /v1 LOOM_OPENAI_BASE_URL=https://gateway.example/v1 LOOM_ANTHROPIC_BASE_URL=https://gateway.example LOOM_LLM_PROVIDER=openai LOOM_LLM_MODEL=gpt-6-luna LOOM_EMBEDDING_MODEL=text-embedding-3-large ``` Each exchange sends a new token that lives five minutes: issuer `/v1`, subject `loom`, the audience above, signed RS256 with the newest RS256 key of the signing keyring. Create one first with `loom signing-key rotate --algorithm RS256`, unless `LOOM_JWT_SECRET` is an RSA key, which signs as RS256 already. That key signs Loom's own tokens too from then on, which every Loom of the deployment verifies; an HS256 secret is never used, since the gateway could not verify it. Spend is attributed to that one subject. The gateway's entry for Loom names that issuer, the audience, subject `loom`, RS256, and the keys at `/v1/.well-known/jwks.json` (OIDC discovery is at `/v1/.well-known/openid-configuration`). The gateway must be able to fetch that URL over https; where it cannot reach Loom, publish a copy of the key set somewhere it can. Each Loom publishes a key at its next keyring refresh after `rotate` writes it, and the key signs only `--activate-in` later (300 s by default): rotate with an `--activate-in` longer than the gateway keeps its copy of the keys, or than copying them takes. A retired key stays trusted at the gateway until its copy is refreshed. `loom signing-key retire` refuses to retire the last active RS256 key when `LOOM_INFERENCE_TOKEN_AUDIENCE` is set in its own environment, so run it with the servers'. Loom (the server and every `loom` command) refuses these settings without an https `LOOM_PUBLIC_BASE_URL`, with `LOOM_JWT_AUDIENCE` as the audience, or with an issuer its own verifier trusts, and its own verifier never accepts a token with the gateway's audience. ## Apply changes and check the result **Dashboard changes:** wait for Models & keys to report the active configuration. If a rebuild is running, follow its progress there; retry a failed job from the same page. Do not recreate containers to apply a dashboard change. **Environment changes only:** for the released stack, recreate the server container: ```bash docker compose --project-directory "$HOME/.loom" up -d --force-recreate loom docker compose --project-directory "$HOME/.loom" exec loom loom doctor curl -fsS http://localhost:7777/v1/health ``` Use your installation directory and port if different. Recreate the container: a plain `docker compose restart` does not reload its environment from `.env`. Native servers must restart with the updated environment. For the checkout stack, run `docker compose --env-file .env -f compose.yaml -f dev/docker-compose.dev.yml --profile bundled up -d --build --force-recreate loom`. Health reports `fake_llm`, `fake_embedding`, `providers_degraded` and actionable `problems`. With both real providers configured, the fake flags should be false. This checks configuration and health; also save and retrieve a test fact and try an answer to confirm that your account can call the chosen models. Changing the LLM or rotating an API key does not require rebuilding embeddings. Changing the embedding model does, even when its dimensions stay the same. For environment changes (rather than the dashboard workflow), existing memory, document, tracker and code indexes need their corresponding re-embedding or re-ingest procedures; `loom reembed` alone is not a universal rebuild of every index. If dimensions change, check the existing schema's vector constraints before applying the new configuration. Preserve your data and follow [maintenance](https://clickhouse.github.io/loom/operations/maintenance.md#re-embed-and-re-extract) and [code-index operations](https://clickhouse.github.io/loom/operations/code-index-operations.md).