Map / Outline
State and storage
Where facts live between calls, who may write them, and how concurrent writers stay safe.
What it is and why it exists
What
State is everything that outlives one model call: the conversation so far, run state inside a request, conversation history across requests, the knowledge store, cursors that mark how far a background job has read, and locks.
Why
A model call has no memory. Each kind of state has a different lifetime and a different owner, and mixing them causes the worst bugs. The reference build's single-writer stores failed when a nightly job and a daily job wrote at once, and one file was corrupted.
How it works
- Run state is a dict passed through the executor. Merge rules say which keys accumulate and which overwrite.
- Conversation state is the message list, stored per thread so a later request can recall it.
- Durable knowledge lives in Postgres. Multi-version concurrency means readers never block writers.
- Schemas separate ownership. Dropping one schema deletes everything the agent taught itself and leaves ingested documents untouched.
- Background jobs keep cursors so they resume where they stopped and never process a log line twice.
Where it sits in the build order
Needs first
Nothing. You can start here.
Unlocks
- RAG and knowledge graphChunks, vectors, the file ledger, and graph edges need a store that survives concurrent readers and a nightly writer. The reference build lost weeks to single-writer stores before moving to Postgres.
- Harness engineeringThe loop is a state machine. Concurrent tool results have to merge into one conversation without overwriting each other, so the merge rules are defined before the loop that relies on them.
- MemoryMemory is durable state with concurrent writers: live chats and background jobs. It needs the storage guarantees from the state module.
In the reference build
| Path | Role |
|---|---|
| apps/agent-server/graph.py | merge_state: the rules for run state. |
| apps/agent-server/pgstore.py | Connection handling and schema setup for Postgres. |
| apps/agent-server/pgschema.sql | Schemas by owner: learned knowledge, file ledger, chunk indexes. |
| apps/agent-server/conversation_store.py | Threads and messages across requests. |
The same idea on other platforms
| Platform | How this module maps |
|---|---|
| Databricks | Delta tables for durable state. Lakebase for Postgres-style transactional state such as threads and checkpoints. Run state lives in your agent code. |
| IBM watsonx | Orchestrate manages thread state for its agents. Durable domain state goes in watsonx.data or a database you connect. |
| Codex | Session history is managed by the tool and can be resumed. Durable project state is files in the repo. |
| Cursor | Chat history is managed by the editor. Durable state is files, plus whatever your MCP servers store. |
| Claude Code / Agent SDK | Sessions can be resumed. The SDK exposes session ids. Durable state is files and your own stores. |
| Another machine | Postgres in a container gives the same guarantees. SQLite is fine for one writer. |
Explain it back
Answer aloud first. Then open the answer and compare.
Name three kinds of state and their lifetimes.
A strong answerRun state lasts one request. Conversation state lasts a thread. Knowledge and memory last until retired. Mixing them leads to leaks across users or loss across requests.
Why did the reference build leave single-file stores?
A strong answerThey allow one writer at a time. Two scheduled jobs collided, one database was corrupted, and a vector segment became unreadable. Postgres gives concurrent access and crash recovery.
From the live build
Recent changes and files the sync job filed under this module.
- Conversation learner ignores eval threads; report treats findings from unchanged threads as no new traffic
- Memory store on Postgres, second attempt (qualified upsert columns); recall skips the current thread, memory briefing cannot fail a request
- Memory store on Postgres (mem schema), recall skips the current thread, memory briefing cannot fail a request
- Conversation recall: mem.turns index, recall_conversations tool, recall gate in the daily cycle
- Audit fixes: self-gap dedupe, SQLite stores in the nightly backup; agent quiz
- Tests: isolate conversations.db and the retrieval trace
- Baseline 2026-09-30: code and scripts after the Postgres cutover, pinned MCP servers, nightly pg_dump
- the quarantined learned-knowledge store (BRAIN-ARCHITECTURE.md §3.1). Deliberately modelled on memory_store.py, right down to the conventions: contextmanager connection, schema-if-not-exists, 0600 permissions, PRAGMA journal_mode=TRUNCATE, and idempotent additive migrations via ...
- learn from what people actually said. python3 nightly/conversation_learn.py # report, write nothing python3 nightly/conversation_learn.py --apply # record python3 nightly/conversation_learn.py --all # ignore the cursor WHY THIS EXISTS =============== This system had three ways ...
- the agent can look up what was said before (2026-10-01). Thin wrapper over conversation_index.search(). Registered after nightly/recall_eval.py passed its gate (see evals/memory/history.jsonl).
- searchable index of past conversations (2026-10-01). The audit scored memory 2 of 5: conversations.db keeps every exchange for 90 days and nothing can search it, so "what did we decide about X last week" has no answer.
- copy memory.db into Postgres schema mem (2026-10-01). Idempotent. agent:self observations are collapsed to the newest row per distinct finding (10,998 rows were 26 findings re-logged hourly). venv/bin/python3 ../ops/migrate_memory_pg.py # migrate + verify
- collects every tool into the two things agent_loop.py needs: the OpenAI-style schema list to hand Ollama (so the model knows what it can call), and a name -> async-callable map to actually execute a call.
- nightly dump of the knowledge store (2026-09-30). WHAT: pg_dump --format=custom of the whole build-host database (brain.* learned atoms, kb.* file ledger, kb_finance / kb_k12 chunk vectors).
- regression tests for the learned-knowledge store. Run: python3 test_brain_store.py These are not exhaustive unit tests; they are guards on the SIX properties that make brain_store.py safe to run unattended.