๐Ÿ” seaglass โ€” Architecture Reference

Local semantic + keyword search over iMessage history, exposed as an MCP server for Grogu. Generated as a living document โ€” click through the pipeline, filter the eval table, and toggle detail levels below.

01 Overview

seaglass builds a local, offline search index over a user's iMessage history (~/Library/Messages/chat.db) and exposes it as an MCP server with two tools: search_messages and get_conversation. Grogu (the user's CLI agent) calls this server instead of doing raw SQL LIKE queries against chat.db directly, giving it real semantic recall ("what did we decide about the trip") on top of exact keyword matching, date-range/participant filtering, and conversation-level context expansion.

Design decision: no persistent daemon (yet). Each invocation pays a cold model-load cost (embedding model + cross-encoder reranker) and exits. This was an explicit, deliberate choice โ€” see ยง7 Latency for the real measured cost of that decision, and why it may get revisited.
385,949
messages indexed
17,525
chunks in index.db
19.4 MB
index.db size
0.41
recall@final (n=32)
~4.1s
cold invocation
~0.75s
warm query (2nd+)

02 Pipeline (click a stage)

This is the exact call sequence for search_messages, shared verbatim by both the live MCP server (seaglass/mcp_server.py) and the eval harness (seaglass/eval/score.py) โ€” the eval always exercises the real filter/query-parsing layer, never a bypass straight to the index.

Click a stage above for details, or use โ† / โ†’ keys.

03 Runtimes & the chat.db lifecycle

Two things people ask about repeatedly: what actually executes the ML models, and how seaglass can safely read from chat.db โ€” a database that Messages.app itself has open and is actively writing to โ€” without corrupting it or racing a live write.

Runtimes

seaglass has no custom inference server or GPU runtime of its own. Both the embedding model and the cross-encoder reranker run in-process via Apple's MLX (seaglass/index/embed.py, seaglass/search/rerank.py) โ€” MLX targets Apple Silicon's unified memory directly, so weights are loaded once per process and matmuls run on-GPU with no serialization/IPC round-trip to a separate server. There is deliberately no persistent daemon (see the design decision in ยง1): each MCP invocation is a fresh Python process that imports mlx.core, loads model weights from disk, and pays a one-time MLX kernel-compilation cost on the very first query (see the ~1.8s cold-invocation number in ยง10). Everything else โ€” SQLite (index.db, chat_snapshot.db, live chat.db reads), the FastAPI app server, the MCP stdio server, and the pywebview desktop shell โ€” is plain CPython; MLX is the only place the app leaves pure-Python/SQLite.

The one place GitHub Copilot CLI (GHCP) is invoked is not a runtime seaglass hosts itself โ€” query-assist parsing and golden-set generation shell out to the copilot CLI binary as a subprocess (see ยง5 and ยง11); seaglass never talks to a model-serving API directly for those steps.

chat.db: snapshot during indexing, live afterwards

~/Library/Messages/chat.db is owned by Messages.app, which holds it open and writes to it continuously (new messages, read receipts, reactions). seaglass never opens a writable handle to it and never queries it directly during a full (re)build โ€” instead:

Build/sync time โ€” snapshot, then index the snapshot. run_build() (seaglass/app/indexstate.py) first copies chat_db_source (the live, real chat.db) to a private chat_snapshot.db file using SQLite's own online backup mechanism, taken via a short-lived read-only connection so Messages.app's writes are never blocked and the copy is always transactionally consistent. The entire embed โ†’ rerank โ†’ chunk โ†’ write pipeline then runs exclusively against that frozen snapshot, never the live file. This is what makes a build safe to run while iMessage is actively receiving texts, and it's also why index.db's content is only ever as fresh as the snapshot taken at the last build/sync.
After the build โ€” live, read-only polling for staleness. Once an index exists, the running app needs to know how far the snapshot has drifted from reality without re-copying the whole database every time. SearchEngine.status() (seaglass/app/engine.py) opens a second, independent read-only connection directly to the live chat_db_source โ€” via connect_readonly() (seaglass/imessage/source.py), which ATTACHes it under a file:...?mode=ro URI so it can never write โ€” and compares its most recent message timestamp against the newest chunk already in index.db. This connection is cached and reused across polls (the status endpoint is hit roughly once a minute by the desktop UI) rather than re-attached every time, and it degrades to "unknown/0" rather than throwing if the live file is missing or unreadable (e.g. Full Disk Access revoked). This live connection is used only for counting how stale the index is โ€” it is never used to actually pull message content into the index; that only ever happens against a fresh snapshot, via the same "Sync now" build path described above.

In short: chat_snapshot.db is the only thing the retrieval pipeline ever reads message bodies from, and it is only ever refreshed by taking a brand-new consistent snapshot of the live chat.db โ€” the live file itself is touched only for cheap, read-only "how far behind are we" checks between builds, never for content.

macOS privacy grants, and why the app ships as a bundle

Both of the things above depend on permissions macOS grants per app identity, not per user. A .app inherits none of the access the terminal that built it had, which makes the failure modes look like seaglass bugs rather than missing grants:

  • Full Disk Access gates the live chat.db. Without it the staleness check above silently reads nothing, so the app claims to be up to date forever.
  • Contacts gates name resolution (seaglass/imessage/contacts.py). Without it every sender renders as a raw phone number.
Both denials are surfaced, never absorbed. status() reports live_chat_readable and contacts_available, and the UI turns each into a banner or chip with a button that deep-links to the exact Settings pane (POST /api/system/open-settings, whitelisted panes only). A grant only takes effect for a newly launched process, so the banner also offers POST /api/system/relaunch, which waits for this pid to actually exit before reopening โ€” otherwise the replacement finds a live lock file and just opens a browser tab at the old instance. Shutdown closes the webview window rather than raising SIGTERM: Python runs signal handlers on the main thread, and that thread is parked inside the Cocoa event loop.
Contacts can only be granted by asking. Its Settings pane has no "+" โ€” it lists only apps that have already requested access. So POST /api/system/request-contacts triggers the system prompt on demand, from the running app. It cannot happen during warmup, which runs on a background thread before there is an event loop to present a prompt on. The prompt is also refused outright (it flashes and vanishes) unless the bundle's Info.plist carries NSContactsUsageDescription, and the bundle is ad-hoc code-signed so the privacy database has a stable identity to attach the grant to.
A bundle's CFBundleExecutable must be Mach-O. Current macOS refuses to launch one whose executable is a shell script (_LSOpenURLsWithCompletionHandler โ€ฆ error -10669), so seaglass/app/installapp.py compiles a tiny C stub at install time that execls the real launcher. Launching from a terminal instead (start.sh) works and keeps the terminal's grants, but the Dock tile then reads "Python", because it comes from LaunchServices' record of the running binary โ€” CPython's own Python.app.

04 Storage layer โ€” index.db

A single SQLite file holds three cooperating tables per chunk plus two virtual (derived) indexes. No text is ever duplicated between them โ€” full text lives compressed in chunks.body_semantic; the FTS5 index is contentless (inverted index only, no stored copy).

Show full column comments

05 Query parsing

seaglass/search/parse.py turns a raw natural-language query into a ParsedQuery: a semantic text (fed to the embedder + BM25) plus structured extractions โ€” date_from/date_to, has_media, and people_participant (chat participant handles, resolved via ContactIndex).

PatternExtracted asNotes
"last week", "in march"date_from / date_to Ambiguous bare month words ("may", "march") require corroboration (a digit or a second unambiguous date word elsewhere in the query) before being treated as a date.
"with Sam", "from Sam"people_participant Case-insensitivity is scoped to just the from|with keyword via inline (?i:...), not the whole pattern โ€” needed so the capitalized-name heuristic still works.
"photo", "pic", "video"has_media = true Maps to the pre-computed chunks.has_attachment column.
Known simplification: "from X" (sender) and "with X" (participant) are not yet distinguished โ€” both resolve to the broader "current chat membership" semantics. This can only make results too permissive, never miss a relevant chunk. Documented as a revisit-if-eval-shows-it-hurts-precision item.

06 Hybrid retrieval โ€” dense + sparse, RRF fused

seaglass/search/retrieve.py implements the baseline retriever: a pre-filter followed by two independent candidate-generation paths, fused by Reciprocal Rank Fusion (RRF).

StageMechanismTop-K
Pre-filterSQL predicate over chunks (dates, media, participant chat_ids) โ€” builds a candidate chunk_id set, or None meaning "unconstrained"โ€”
Densesqlite-vec int8 cosine KNN over chunks_vec, constrained to the pre-filter set if present200
SparseFTS5 BM25 over chunks_fts (porter/unicode61 tokenizer), same constraint200
FuseReciprocal Rank Fusion, k=6050 (FUSED_TOP_K)
Recency reserveLast 10 of the 50 fused slots (RECENCY_RESERVED_SLOTS) are refilled with the newest matched chunks not already present50 (unchanged)
Note: FTS5's bm25() returns negative values โ€” best matches sort ascending, the opposite of dense-search convention. Any code touching sparse scores directly has to account for this.
Note: sqlite-vec's int8 column type will silently misinterpret a bare blob parameter as float32 โ€” every insert/query wraps the vector with vec_int8(...) explicitly.
Why reserve slots for recency instead of weighting it into RRF? BM25 ordering among hundreds of near-identical matches is essentially arbitrary โ€” a one-word message sent today landed at sparse rank 171 of 228 for the query "sweet", and 1/(60+171) can never reach the fused top 50. Adding a weighted recency arm to RRF was measured first: the target only entered the pool at recency_weight = 1.0, i.e. recency made fully co-equal with relevance, which distorts every other query. Reserved slots are bounded instead โ€” the top 40 stay purely relevance-ordered, the candidate count stays exactly 50 so the reranker's cost is unchanged, and unused slack is handed back to the fusion ranking so the list is never short.

07 Rerank + session aggregation

seaglass/search/rank.py + rerank.py โ€” deliberately kept separate from retrieval so the baseline (no rerank) and full pipeline could be evaluated independently against the golden set.

  1. Cross-encoder rerank (ms-marco-MiniLM-L-12-v2) scores each of the 50 fused candidates against the query text directly (not just embedding similarity), cut to top 12 (RERANK_TOP_K). The model is run in eval mode for deterministic scoring. The last 3 of those 12 slots (RERANK_RECENT_SLOTS) are reserved for the newest candidates โ€” every fused candidate is scored in the same batched forward pass regardless of the cut, so this costs no extra model compute; it only decides which already-scored chunks are allowed to reach aggregation.
  2. Session aggregation groups the top-12 reranked chunks by (chat_id, local calendar day) โ€” approximating "that day we talked about X" โ€” into up to 8 sessions (MAX_SESSIONS), combining member-chunk scores per session.
  3. Recency shaping multiplies each session's score by 1 + 0.6 × 0.5age / 365d (RECENCY_MAX_BOOST, RECENCY_HALF_LIFE_DAYS), so a session from today is worth up to 1.6× one from the distant past. Bounded on purpose: this breaks ties between comparably relevant sessions rather than letting a weak recent match outrank a strong old one. Two of the eight session slots (RECENT_SESSION_SLOTS) are then reserved for the newest matches.
  4. Expansion pads each session with ยฑ2 time-adjacent chunks (EXPANSION_RADIUS) for conversational context, even if those chunks weren't independently retrieved.
Verbatim matches: putting the lexical signal back. Reserved slots get a recent chunk into the results, but they don't decide its rank. The remaining gap: if the user types "classic" and someone literally texted "classic" an hour ago, that is almost certainly the message they mean โ€” whatever the cross-encoder thinks of the surrounding 20 messages. After reranking, the candidate pool is re-queried against chunks_fts with the query wrapped as an FTS5 phrase (exact_phrase_chunk_ids), and sessions containing a verbatim hit get +1.2 × 0.5age / 30d added before the recency multiplier (LEXICAL_MATCH_BOOST, LEXICAL_HALF_LIFE_DAYS). The half-life is deliberately 30 days โ€” a twelfth of the general recency half-life โ€” because the signal being modelled is recent verbatim match: any common word appears verbatim across years of history, so an old literal match earns almost nothing and the usual relevance ordering still decides. Measured on the real 386k-message index: the query "classic" moved today's match from rank 7 to rank 1.
Why recency needs reserved slots and not just a multiplier. The cross-encoder scores a whole conversation chunk, so a query matching one short message inside a long window scores terribly: a message sent today reading exactly "Sweet" sits in a 22-message chunk that the reranker scored -4.89 (sigmoid 0.0075) for the query "sweet" โ€” rank 32 of 50. No bounded recency multiplier can rescue that, so relevance alone would let the user conclude search simply cannot find what they just typed. Note the sparse/BM25 arm did find that chunk; the lexical signal is what gets discarded once the cross-encoder takes over. The reservations at all three stages (candidate pool โ†’ rerank cut โ†’ session slots) form one chain that puts it back, each bounded so the majority of every result list stays purely relevance-ordered.
Day-boundary grouping uses the local system timezone, not UTC โ€” a deliberate judgment call since chat.db timestamps carry no stored timezone of their own, and local wall-clock day is the closest approximation to how a person actually remembers a conversation.
Pagination without re-ranking. "Load more" must not change what was on page 1, and re-running the cross-encoder per page would be both slow and unstable. So rerank_candidates(top_k=None) scores every candidate in one pass and order_sessions() returns the complete ordering; the engine caches that ordering (LRU, keyed by everything affecting rank except offset) and serves later pages by slicing it. Page 1 is built from the top-12 rerank cut with recency reservations applied; deeper pages append the remaining scored pool in pure score order, so adding pages is strictly additive and never reshuffles what the user already read. The cache is invalidated on warmup() because a rebuild renumbers chunk ids, and its key must include every filter โ€” the cached value is the reranked candidate pool, and filters are exactly what narrow that pool, so an omitted filter made toggling it replay the previous query's results. Cached pages return in 0.13โ€“0.18 s versus ~1.4 s cold.
A query with no words is a browse, not a search. An empty (or filters-only) query has nothing to match on, but embedding the empty string still yields a direction in vector space โ€” so the pipeline used to return dozens of confidently-scored, meaningless sessions. A blank semantic now short-circuits to the newest chunks in the candidate set (recency_ranked), which is what "show me messages from Alice" with no query text actually means. It skips both models entirely: 0.04 s versus ~1.3 s, and the filters still apply.
MLX memory: cap the Metal buffer cache. Activity Monitor showed the app at 7.30 GB while ps RSS read 170 MB. footprint -p located it: 7144 MB under IOAccelerator (graphics) โ€” MLX's Metal buffer cache, which on Apple Silicon lives in unified memory and counts toward physical footprint but not RSS. It is not a leak: active memory stayed flat at 254 MB across repeated query rounds while cache grew and never shrank, retaining the high-water mark of the largest batch ever run (an index build sets a very high one). seaglass/mlxmem.py caps it at 512 MB (SEAGLASS_MLX_CACHE_MB) at model load and releases it after a build. Measured cost: none โ€” 0.63 s vs 0.64 s over six embed+rerank rounds, because a single query's working set is far smaller than the retained cache. Steady state drops from ~5.3 GB to ~1.2 GB.

08 MCP server surface

seaglass/mcp_server.py exposes four tools over stdio, single-process, guarded by a global lock (_pipeline_lock) since MLX has no cross-thread safety guarantee, plus a separate _init_lock around all lazy singleton getters (index connection, chat connection, embedding model, reranker, contact index).

ToolPurposeKey params
search_messagesRanked, hydrated conversation sessions with message-id citations for a topical/semantic queryquery, max_sessions=8, redact=false
get_conversationDrill-down: raw messages from one chat_id, optionally centered on a timestampchat_id, around_ts, limit=50
index_statusChunk/vector counts, configured DB paths, hydration/contacts availability, and how many messages have arrived since the index was built โ€” lets a caller sanity-check the index before trusting search resultsโ€”
sync_indexBring the index up to date with the live Messages database, by proxying to the running desktop app's build endpointwait=false
Ranking lives in one place. The MCP server used to reimplement retrieve โ†’ rerank โ†’ aggregate. Every ranking improvement made in the app therefore skipped Grogu silently โ€” the recency reservation and the recent-verbatim-match boost both landed that way โ€” so the same query answered differently depending on which door it came in through, with the MCP answer being the worse one and nothing saying so. It now delegates to the app's SearchEngine.
The running app answers when there is one. If the desktop app is up and serving the same index, the MCP server proxies the query to it over loopback instead of loading a second copy of the models โ€” saving ~1 GB of RSS and a multi-second cold start. With no app running it falls back to its own lazy in-process pipeline, so Grogu still works when seaglass is closed.
Results declare their freshness and their ordering. Every search payload carries index_stale / n_messages_since_index (a stale answer is otherwise indistinguishable from a complete one) and ordering, which is recent for a time-ordered browse and relevance otherwise โ€” a caller honouring a limit needs to know whether "first" means most relevant or most recent. The freshness read costs ~0.25s against the live chat.db, so it is cached for 30s and invalidated after a build.
Behavior note: get_conversation's around_ts path searches the whole chat by fetching cheap (ROWID, date) pairs, locating the closest position to the target timestamp in Python, then fetching just that window by ROWID โ€” so drilling into an old thread in a long chat still finds the right window regardless of chat length.

09 Grogu integration

Grogu (src/grogu_mcp.py + src/grogu_imessage.py) launches seaglass-mcp as a stdio subprocess per invocation, configured via ~/.copilot/mcp-config.json (which supplies SEAGLASS_INDEX_DB and SEAGLASS_CHAT_DB as environment variables to the subprocess). If seaglass is available, search_via_seaglass is preferred over the legacy SQL LIKE fallback.

search_via_seaglass requests seaglass's default max_sessions=8 regardless of the caller's message limit, and flattens each returned session's messages plus context_messages into one result list. The SQL LIKE fallback path normalizes Apple-epoch timestamps to unix seconds so both paths return dates in the same format.

grogu imessage status reports whether seaglass is configured and reachable (seaglass: true/false) along with the index's own freshness (seaglass_index.n_messages_since_index); grogu imessage search transparently uses seaglass when available and falls back to SQL LIKE otherwise, warning on stderr when it does. A search against a lagging index prints a staleness warning and points at grogu imessage sync.

Person-and-recency queries. "latest from Adrian" is the most ordinary thing anyone asks, and it was wrong in four independent ways: from X only narrowed to chats X is in, so a group chat answered with someone else's messages; the filler words "latest"/"recent"/"the last thing" were embedded, and the noise vector decided the ranking; browse-mode pseudo-scores saturated the aggregation sigmoid so the ranking silently became "whichever day has the most chunks"; and chat.db stamps outgoing 1:1 messages with the recipient's handle_id, so a handle filter kept both halves of the conversation. seaglass now resolves a sender filter, narrows candidates to chunks that person actually spoke in, and routes contentless queries to a time-ordered browse.

10 Measured performance

Numbers below were captured against a real 17,525-chunk index. Toggle between memory and latency views.

Latency Memory (RSS)
ScenarioTimeNotes
Full cold CLI invocation~4.0โ€“4.1sNew process, embedding model + reranker load from scratch, HuggingFace revision-check HTTP calls, then the actual query
First query, models already loaded in-process~1.8sOne-time MLX kernel-compilation cost even though weights are resident
Steady-state query (2nd+)~0.3โ€“0.9s (avg ~0.75s)No measurable growth across repeated queries
Once warm, queries are ~5โ€“13ร— faster than a cold invocation. Nearly all of the ~4s cold cost is process/model-load overhead, not search work itself. seaglass currently runs as a per-invocation process rather than a persistent daemon; this is the main lever available for reducing latency further.

11 Evaluation results

Golden set: 32 hand-reviewed queries against real message history, scored by seaglass/eval/score.py โ€” which runs the exact same parse โ†’ filter โ†’ retrieve โ†’ rerank โ†’ aggregate pipeline as the live MCP server, so these numbers reflect real end-user behavior, not a shortcut.

Categorynrecall@50recall@12recall@finalgap1gap2kill

recall@50 95% Wilson CI: [0.45, 0.77] (n=32 โ€” still wide; treat per-category splits as directional only). gap1 = recall@12 โˆ’ recall@50 (rerank loss), gap2 = recall@final โˆ’ recall@12 (aggregation/expansion loss), kill = rate the pre-rerank filter discarded the true positive outright.

12 CLI & operational commands

# build (or resume) an index from a chat.db snapshot
python -m seaglass.cli build <chat.db> <index.db>

# ad-hoc search from the terminal (bypasses MCP, same pipeline)
python -m seaglass.cli search <index.db> "lisbon trip" --chat-db <chat.db>

# score a golden set against a built index
python -m seaglass.eval.score <index.db> <golden.jsonl> --chat-db <chat.db>

# run as an MCP server (what Grogu launches as a subprocess)
seaglass-mcp   # entry point: seaglass.mcp_server:main
  # env: SEAGLASS_INDEX_DB, SEAGLASS_CHAT_DB