01 Overview
seaglass builds a local, offline search index over a user's iMessage
history (~/Library/Messages/chat.db) and exposes it as an
MCP server with two tools:
search_messages and get_conversation. Grogu (the user's CLI agent)
calls this server instead of doing raw SQL LIKE queries against
chat.db directly, giving it real semantic recall ("what did we decide about the
trip") on top of exact keyword matching, date-range/participant filtering, and
conversation-level context expansion.
02 Pipeline (click a stage)
This is the exact call sequence for search_messages, shared verbatim by both the
live MCP server (seaglass/mcp_server.py) and the eval harness
(seaglass/eval/score.py) โ the eval always exercises the real filter/query-parsing
layer, never a bypass straight to the index.
03 Runtimes & the chat.db lifecycle
Two things people ask about repeatedly: what actually executes the ML models, and how
seaglass can safely read from chat.db โ a database that Messages.app itself
has open and is actively writing to โ without corrupting it or racing a live write.
Runtimes
seaglass has no custom inference server or GPU runtime of its own. Both the embedding
model and the cross-encoder reranker run in-process via
Apple's MLX
(seaglass/index/embed.py, seaglass/search/rerank.py) โ MLX targets
Apple Silicon's unified memory directly, so weights are loaded once per process and matmuls
run on-GPU with no serialization/IPC round-trip to a separate server. There is deliberately
no persistent daemon (see the design decision in ยง1):
each MCP invocation is a fresh Python process that imports mlx.core, loads
model weights from disk, and pays a one-time MLX kernel-compilation cost on the very first
query (see the ~1.8s cold-invocation number in ยง10).
Everything else โ SQLite (index.db, chat_snapshot.db, live chat.db reads), the FastAPI app
server, the MCP stdio server, and the pywebview desktop shell โ is plain CPython; MLX is the
only place the app leaves pure-Python/SQLite.
The one place GitHub Copilot CLI (GHCP) is invoked is not a runtime seaglass hosts
itself โ query-assist parsing and golden-set generation shell out to the copilot
CLI binary as a subprocess (see ยง5 and ยง11);
seaglass never talks to a model-serving API directly for those steps.
chat.db: snapshot during indexing, live afterwards
~/Library/Messages/chat.db is owned by Messages.app, which holds it open and
writes to it continuously (new messages, read receipts, reactions). seaglass never opens a
writable handle to it and never queries it directly during a full (re)build โ instead:
run_build() (seaglass/app/indexstate.py) first copies
chat_db_source (the live, real chat.db) to a private
chat_snapshot.db file using SQLite's own online backup mechanism, taken via a
short-lived read-only connection so Messages.app's writes are never blocked and the copy is
always transactionally consistent. The entire embed โ rerank โ chunk โ write pipeline then
runs exclusively against that frozen snapshot, never the live file. This is what makes a
build safe to run while iMessage is actively receiving texts, and it's also why
index.db's content is only ever as fresh as the snapshot taken at the last
build/sync.
SearchEngine.status()
(seaglass/app/engine.py) opens a second, independent read-only connection
directly to the live chat_db_source โ via
connect_readonly() (seaglass/imessage/source.py), which
ATTACHes it under a file:...?mode=ro URI so it can never write โ
and compares its most recent message timestamp against the newest chunk already in
index.db. This connection is cached and reused across polls (the status
endpoint is hit roughly once a minute by the desktop UI) rather than re-attached every time,
and it degrades to "unknown/0" rather than throwing if the live file is missing or
unreadable (e.g. Full Disk Access revoked). This live connection is used only for
counting how stale the index is โ it is never used to actually pull message content into the
index; that only ever happens against a fresh snapshot, via the same "Sync now" build path
described above.
In short: chat_snapshot.db is the only thing the retrieval pipeline ever reads
message bodies from, and it is only ever refreshed by taking a brand-new consistent
snapshot of the live chat.db โ the live file itself is touched only for cheap,
read-only "how far behind are we" checks between builds, never for content.
macOS privacy grants, and why the app ships as a bundle
Both of the things above depend on permissions macOS grants per app identity, not
per user. A .app inherits none of the access the terminal that built it had,
which makes the failure modes look like seaglass bugs rather than missing grants:
- Full Disk Access gates the live
chat.db. Without it the staleness check above silently reads nothing, so the app claims to be up to date forever. - Contacts gates name resolution
(
seaglass/imessage/contacts.py). Without it every sender renders as a raw phone number.
status() reports
live_chat_readable and contacts_available, and the UI turns each
into a banner or chip with a button that deep-links to the exact Settings pane
(POST /api/system/open-settings, whitelisted panes only). A grant only takes
effect for a newly launched process, so the banner also offers
POST /api/system/relaunch, which waits for this pid to actually exit before
reopening โ otherwise the replacement finds a live lock file and just opens a browser tab at
the old instance. Shutdown closes the webview window rather than raising SIGTERM: Python
runs signal handlers on the main thread, and that thread is parked inside the Cocoa event
loop.
POST
/api/system/request-contacts triggers the system prompt on demand, from the running
app. It cannot happen during warmup, which runs on a background thread before there is an
event loop to present a prompt on. The prompt is also refused outright (it flashes and
vanishes) unless the bundle's Info.plist carries
NSContactsUsageDescription, and the bundle is ad-hoc code-signed so the privacy
database has a stable identity to attach the grant to.
CFBundleExecutable must be Mach-O. Current macOS
refuses to launch one whose executable is a shell script
(_LSOpenURLsWithCompletionHandler โฆ error -10669), so
seaglass/app/installapp.py compiles a tiny C stub at install time that
execls the real launcher. Launching from a terminal instead
(start.sh) works and keeps the terminal's grants, but the Dock tile then reads
"Python", because it comes from LaunchServices' record of the running binary โ CPython's own
Python.app.
04 Storage layer โ index.db
A single SQLite file holds three cooperating tables per chunk plus two virtual (derived)
indexes. No text is ever duplicated between them โ full text lives compressed in
chunks.body_semantic; the FTS5 index is contentless (inverted index only,
no stored copy).
05 Query parsing
seaglass/search/parse.py turns a raw natural-language query into a
ParsedQuery: a semantic text (fed to the embedder + BM25) plus
structured extractions โ date_from/date_to, has_media,
and people_participant (chat participant handles, resolved via
ContactIndex).
| Pattern | Extracted as | Notes |
|---|---|---|
"last week", "in march" | date_from / date_to |
Ambiguous bare month words ("may", "march") require corroboration (a digit or a second unambiguous date word elsewhere in the query) before being treated as a date. |
"with Sam", "from Sam" | people_participant |
Case-insensitivity is scoped to just the from|with keyword via inline
(?i:...), not the whole pattern โ needed so the capitalized-name heuristic
still works. |
"photo", "pic", "video" | has_media = true |
Maps to the pre-computed chunks.has_attachment column. |
06 Hybrid retrieval โ dense + sparse, RRF fused
seaglass/search/retrieve.py implements the baseline retriever: a pre-filter
followed by two independent candidate-generation paths, fused by Reciprocal Rank Fusion (RRF).
| Stage | Mechanism | Top-K |
|---|---|---|
| Pre-filter | SQL predicate over chunks (dates, media, participant chat_ids) โ builds a candidate chunk_id set, or None meaning "unconstrained" | โ |
| Dense | sqlite-vec int8 cosine KNN over chunks_vec, constrained to the pre-filter set if present | 200 |
| Sparse | FTS5 BM25 over chunks_fts (porter/unicode61 tokenizer), same constraint | 200 |
| Fuse | Reciprocal Rank Fusion, k=60 | 50 (FUSED_TOP_K) |
| Recency reserve | Last 10 of the 50 fused slots (RECENCY_RESERVED_SLOTS) are refilled with the newest matched chunks not already present | 50 (unchanged) |
bm25() returns negative
values โ best matches sort ascending, the opposite of dense-search
convention. Any code touching sparse scores directly has to account for this.
sqlite-vec's int8 column type will silently
misinterpret a bare blob parameter as float32 โ every insert/query wraps the vector with
vec_int8(...) explicitly.
1/(60+171) can never reach the fused top 50.
Adding a weighted recency arm to RRF was measured first: the target only entered the
pool at recency_weight = 1.0, i.e. recency made fully co-equal with
relevance, which distorts every other query. Reserved slots are bounded instead โ the
top 40 stay purely relevance-ordered, the candidate count stays exactly 50 so the
reranker's cost is unchanged, and unused slack is handed back to the fusion ranking so
the list is never short.
07 Rerank + session aggregation
seaglass/search/rank.py + rerank.py โ deliberately kept
separate from retrieval so the baseline (no rerank) and full pipeline could be evaluated
independently against the golden set.
- Cross-encoder rerank (
ms-marco-MiniLM-L-12-v2) scores each of the 50 fused candidates against the query text directly (not just embedding similarity), cut to top 12 (RERANK_TOP_K). The model is run in eval mode for deterministic scoring. The last 3 of those 12 slots (RERANK_RECENT_SLOTS) are reserved for the newest candidates โ every fused candidate is scored in the same batched forward pass regardless of the cut, so this costs no extra model compute; it only decides which already-scored chunks are allowed to reach aggregation. - Session aggregation groups the top-12 reranked chunks by
(chat_id, local calendar day)โ approximating "that day we talked about X" โ into up to 8 sessions (MAX_SESSIONS), combining member-chunk scores per session. - Recency shaping multiplies each session's score by
1 + 0.6 × 0.5age / 365d(RECENCY_MAX_BOOST,RECENCY_HALF_LIFE_DAYS), so a session from today is worth up to 1.6× one from the distant past. Bounded on purpose: this breaks ties between comparably relevant sessions rather than letting a weak recent match outrank a strong old one. Two of the eight session slots (RECENT_SESSION_SLOTS) are then reserved for the newest matches. - Expansion pads each session with ยฑ2 time-adjacent chunks
(
EXPANSION_RADIUS) for conversational context, even if those chunks weren't independently retrieved.
chunks_fts with the query wrapped as an FTS5
phrase (exact_phrase_chunk_ids), and sessions containing a verbatim
hit get +1.2 × 0.5age / 30d added before the recency
multiplier (LEXICAL_MATCH_BOOST, LEXICAL_HALF_LIFE_DAYS).
The half-life is deliberately 30 days โ a twelfth of the general recency half-life โ
because the signal being modelled is recent verbatim match: any common word
appears verbatim across years of history, so an old literal match earns almost nothing and
the usual relevance ordering still decides. Measured on the real 386k-message index: the
query "classic" moved today's match from rank 7 to rank 1.
-4.89
(sigmoid 0.0075) for the query "sweet" โ rank 32 of 50. No bounded
recency multiplier can rescue that, so relevance alone would let the user conclude search
simply cannot find what they just typed. Note the sparse/BM25 arm did find that
chunk; the lexical signal is what gets discarded once the cross-encoder takes over. The
reservations at all three stages (candidate pool โ rerank cut โ session slots) form one
chain that puts it back, each bounded so the majority of every result list stays purely
relevance-ordered.
chat.db timestamps carry no stored timezone of their own, and
local wall-clock day is the closest approximation to how a person actually remembers a
conversation.
rerank_candidates(top_k=None) scores every candidate in one pass and
order_sessions() returns the complete ordering; the engine caches
that ordering (LRU, keyed by everything affecting rank except offset) and serves
later pages by slicing it. Page 1 is built from the top-12 rerank cut with recency
reservations applied; deeper pages append the remaining scored pool in pure score order,
so adding pages is strictly additive and never reshuffles what the user already read.
The cache is invalidated on warmup() because a rebuild renumbers chunk ids, and its key must include every filter โ the cached value is the reranked candidate pool, and filters are exactly what narrow that pool, so an omitted filter made toggling it replay the previous query's results.
Cached pages return in 0.13โ0.18 s versus ~1.4 s cold.
semantic now short-circuits
to the newest chunks in the candidate set (recency_ranked), which is what
"show me messages from Alice" with no query text actually means. It skips both models
entirely: 0.04 s versus ~1.3 s, and the filters still apply.
ps RSS read 170 MB. footprint -p located
it: 7144 MB under IOAccelerator (graphics) โ MLX's Metal buffer cache, which
on Apple Silicon lives in unified memory and counts toward physical footprint but not RSS.
It is not a leak: active memory stayed flat at 254 MB across repeated query
rounds while cache grew and never shrank, retaining the high-water mark of the
largest batch ever run (an index build sets a very high one). seaglass/mlxmem.py
caps it at 512 MB (SEAGLASS_MLX_CACHE_MB) at model load and releases it
after a build. Measured cost: none โ 0.63 s vs 0.64 s over six embed+rerank
rounds, because a single query's working set is far smaller than the retained cache.
Steady state drops from ~5.3 GB to ~1.2 GB.
08 MCP server surface
seaglass/mcp_server.py exposes four tools over stdio, single-process, guarded by
a global lock (_pipeline_lock) since MLX has no cross-thread safety guarantee, plus
a separate _init_lock around all lazy singleton getters (index connection, chat
connection, embedding model, reranker, contact index).
| Tool | Purpose | Key params |
|---|---|---|
search_messages | Ranked, hydrated conversation sessions with message-id citations for a topical/semantic query | query, max_sessions=8, redact=false |
get_conversation | Drill-down: raw messages from one chat_id, optionally centered on a timestamp | chat_id, around_ts, limit=50 |
index_status | Chunk/vector counts, configured DB paths, hydration/contacts availability, and how many messages have arrived since the index was built โ lets a caller sanity-check the index before trusting search results | โ |
sync_index | Bring the index up to date with the live Messages database, by proxying to the running desktop app's build endpoint | wait=false |
SearchEngine.
index_stale / n_messages_since_index (a
stale answer is otherwise indistinguishable from a complete one) and
ordering, which is recent for a time-ordered browse and
relevance otherwise โ a caller honouring a limit needs to know whether
"first" means most relevant or most recent. The freshness read costs ~0.25s against
the live chat.db, so it is cached for 30s and invalidated after a build.
get_conversation's around_ts
path searches the whole chat by fetching cheap (ROWID, date) pairs, locating
the closest position to the target timestamp in Python, then fetching just that window by
ROWID โ so drilling into an old thread in a long chat still finds the right window
regardless of chat length.
09 Grogu integration
Grogu (src/grogu_mcp.py + src/grogu_imessage.py) launches
seaglass-mcp as a stdio subprocess per invocation, configured via
~/.copilot/mcp-config.json (which supplies SEAGLASS_INDEX_DB and
SEAGLASS_CHAT_DB as environment variables to the subprocess). If seaglass is
available, search_via_seaglass is preferred over the legacy SQL LIKE
fallback.
search_via_seaglass requests seaglass's default max_sessions=8
regardless of the caller's message limit, and flattens each returned session's
messages plus context_messages into one result list. The SQL
LIKE fallback path normalizes Apple-epoch timestamps to unix seconds so both paths
return dates in the same format.
grogu imessage status reports whether seaglass is configured and reachable
(seaglass: true/false) along with the index's own freshness
(seaglass_index.n_messages_since_index); grogu imessage search
transparently uses seaglass when available and falls back to SQL LIKE
otherwise, warning on stderr when it does. A search against a lagging index prints a
staleness warning and points at grogu imessage sync.
from X only
narrowed to chats X is in, so a group chat answered with someone else's
messages; the filler words "latest"/"recent"/"the last thing" were embedded, and the
noise vector decided the ranking; browse-mode pseudo-scores saturated the aggregation
sigmoid so the ranking silently became "whichever day has the most chunks"; and
chat.db stamps outgoing 1:1 messages with the recipient's handle_id, so a
handle filter kept both halves of the conversation. seaglass now resolves a sender
filter, narrows candidates to chunks that person actually spoke in, and routes
contentless queries to a time-ordered browse.
10 Measured performance
Numbers below were captured against a real 17,525-chunk index. Toggle between memory and latency views.
| Scenario | Time | Notes |
|---|---|---|
| Full cold CLI invocation | ~4.0โ4.1s | New process, embedding model + reranker load from scratch, HuggingFace revision-check HTTP calls, then the actual query |
| First query, models already loaded in-process | ~1.8s | One-time MLX kernel-compilation cost even though weights are resident |
| Steady-state query (2nd+) | ~0.3โ0.9s (avg ~0.75s) | No measurable growth across repeated queries |
11 Evaluation results
Golden set: 32 hand-reviewed queries against real message history, scored by
seaglass/eval/score.py โ which runs the exact same
parse โ filter โ retrieve โ rerank โ aggregate pipeline as the live MCP server, so these
numbers reflect real end-user behavior, not a shortcut.
| Category | n | recall@50 | recall@12 | recall@final | gap1 | gap2 | kill |
|---|
recall@50 95% Wilson CI: [0.45, 0.77] (n=32 โ still wide; treat per-category
splits as directional only). gap1 = recall@12 โ recall@50 (rerank loss),
gap2 = recall@final โ recall@12 (aggregation/expansion loss), kill =
rate the pre-rerank filter discarded the true positive outright.
12 CLI & operational commands
# build (or resume) an index from a chat.db snapshot
python -m seaglass.cli build <chat.db> <index.db>
# ad-hoc search from the terminal (bypasses MCP, same pipeline)
python -m seaglass.cli search <index.db> "lisbon trip" --chat-db <chat.db>
# score a golden set against a built index
python -m seaglass.eval.score <index.db> <golden.jsonl> --chat-db <chat.db>
# run as an MCP server (what Grogu launches as a subprocess)
seaglass-mcp # entry point: seaglass.mcp_server:main
# env: SEAGLASS_INDEX_DB, SEAGLASS_CHAT_DB