fix(session-query): harden SQLite search reconciliation

This commit is contained in:
Hypatia May
2026-07-15 12:10:24 +08:00
parent ecf90ff382
commit f88ca85ffd
40 files changed
+1315 -227

No files matched your search

@@ -1,22 +1,22 @@
# @deepseek-ai/dsh-session-query-sqlite
SQLite FTS5 implementation of `ctx.sessionSearch`. The service searches the live-preferred logical session corpus, groups cross-session results by their strongest event, and keeps provider-specific BM25 scores private.
SQLite FTS5 implementation of `ctx.sessionSearch`. The service searches the live-preferred logical session corpus and groups cross-session results by their strongest event.
## Search contract
`searchSessions(request, exec?)` returns `SessionSearchHit` pages across the corpus; `searchEvents(request, exec?)` returns `SessionEventSearchHit` pages within one session. Queries are required, trimmed, whitespace-normalized literal phrases. FTS5 syntax such as quotes, `OR`, `NEAR`, and `*` is treated as data rather than executable MATCH syntax. Metadata filters are parameterized SQL predicates applied before ranking.
Ordering is deterministic: relevance first, then event time, session id where applicable, and seq. Cross-session results expose the selected event as `bestMatch`; both scopes return plain-text snippets bounded in Unicode code points. Cursors are opaque, bind to the normalized request and service instance, and fail when the relevant generation changes. A within-session cursor survives unrelated-session changes; a cross-session cursor does not.
Relevance is source-comparable across persistent and TEMP tables: actual FTS5 highlighted-match span count descending, then stored document code-point length ascending. Event time, session id where applicable, and seq break remaining ties. Cross-session results expose the selected event as `bestMatch`; both scopes derive whitespace-normalized plain text from FTS5 highlight positions and bound it in Unicode code points. Cursors are opaque branded values, bind to the normalized request and service instance, and fail when the relevant generation changes. A within-session cursor survives unrelated-session changes; a cross-session cursor does not.
All three surfaces (`current`, `shadowed`, and `log-only`) are searchable by default. Pass a surface filter to narrow them.
## Source and index lifecycle
The service requires `ctx.sessions` and observes optional `ctx.sessionPersistence` dynamically. One serialized state machine observes complete sources, extracts shared semantic documents, reconciles changes transactionally, and runs the query. Stable fingerprints preserve unchanged persisted rows and generations; new, changed, and deleted durable sessions reconcile on the next search. Source or transaction failure commits nothing, and the next search retries.
The service requires `ctx.sessions` and observes optional `ctx.sessionPersistence` dynamically. One serialized state machine compares lightweight durable snapshot revisions, loads only new or changed logs, extracts shared semantic documents, reconciles changes transactionally, and runs the query. Repeated queries and unchanged reopen load no full durable logs; new, changed, deleted, or load-repaired sources reconcile on the next stable observation. Source or transaction failure commits nothing, and the next search retries.
Persisted FTS rows live in a dedicated derived database. Connection-local TEMP tables hold live rows, which shadow the durable base for the same session and reveal it when the live owner disappears. Unmounting persistence hides durable rows without discarding the cache; remounting reconciles it. Closing or reopening the database drops every live overlay while retaining persisted rows.
The database is disposable but reset is guarded: a recognized incompatible search schema rebuilds in place, while an unrelated or canonical database is refused. Never point `path` at the session-persistence database.
The database is disposable but reset is guarded: a recognized incompatible search schema rebuilds in place, while an unrelated or canonical database is refused before mutating journal mode. Never point `path` at the session-persistence database. Exactly one service in one process owns a derived-index path; external writers or a second process are unsupported because generations and TEMP shadow state are connection-owned.
## Configuration
@@ -30,6 +30,6 @@ The database is disposable but reset is guarded: a recognized incompatible searc
## Tokenizer and limits
The index uses FTS5 `unicode61`. In the implementation experiment it supported the two-character query `AI` and produced an index about 2.1× smaller than the trigram alternative. The trade-off is token/phrase recall rather than arbitrary substring recall: `AI` does not match the token `BRAID`. Use `ctx.sessionQuery.filterEvents()` with a `text` clause when a literal whitespace-flexible substring scan is required.
The index uses FTS5 `unicode61`. In the implementation experiment it supported the two-character query `AI` and produced an index about 2.1× smaller than the trigram alternative. The trade-off is token/phrase recall rather than arbitrary substring recall: `AI` does not match the token `BRAID`. Use `ctx.sessionQuery.filterEvents()` with a `text` clause when a literal whitespace-flexible substring scan is required. NUL is rejected in queries; reserved highlight markers and NUL in documents are normalized before indexing so presentation markers cannot collide with source text.
Abort signals stop queued work and caller waits around asynchronous source observation. Node's synchronous `DatabaseSync` API cannot interrupt a MATCH statement already executing on the JavaScript thread; the signal is checked immediately before and after the serialized observation/reconciliation boundary.
@@ -6,12 +6,17 @@
import { createHash, randomUUID } from 'node:crypto'
import { DatabaseSync } from 'node:sqlite'
import { Context } from 'cordis'
import { Context, type Fiber } from 'cordis'
import z from 'schemastery'
import type { Session, SessionEvent, SessionHeader, SessionId } from '@deepseek-ai/dsh-session'
import type SessionPersistence from '@deepseek-ai/dsh-session-persistence'
import type {
SessionPersistenceRevision,
SessionPersistenceSnapshot,
} from '@deepseek-ai/dsh-session-persistence'
import {
SessionQueryError,
SessionSearchCursor,
SessionSearchService,
assertSessionHeadersCompatible,
buildSessionEventSearchDocuments,
@@ -22,6 +27,7 @@ import type {
SessionEventSearchRequest,
SessionSearchExecContext,
SessionSearchHit,
SessionSearchCursor as SessionSearchCursorValue,
SessionSearchPage,
SessionSearchRequest,
} from '@deepseek-ai/dsh-session-query'
@@ -32,6 +38,8 @@ import {
import {
type NormalizedEventRequest,
type NormalizedSessionRequest,
FTS_HIGHLIGHT_END,
FTS_HIGHLIGHT_START,
buildEventWhere,
buildSessionWhere,
makeSnippet,
@@ -39,6 +47,7 @@ import {
normalizeSessionRequest,
quoteFtsData,
requestFingerprint,
sanitizeFtsText,
} from './query.ts'
export {
@@ -78,19 +87,30 @@ interface ResolvedConfig {
interface ObservedSession {
header: SessionHeader
events: SessionEvent[]
documents: SessionEventSearchDocument[]
fingerprint: string
}
interface ObservedPersistedSession {
header: SessionHeader
revision: SessionPersistenceRevision
loaded?: ObservedSession
}
interface Observation {
persistence: SessionPersistence | undefined
persistenceRevision: number
persisted: Map<SessionId, ObservedSession>
persisted: Map<SessionId, ObservedPersistedSession>
live: Map<SessionId, ObservedSession>
}
interface IndexedRow {
interface IndexedPersistedRow {
id: string
revision: string
generation: number
}
interface IndexedLiveRow {
id: string
fingerprint: string
generation: number
@@ -109,8 +129,9 @@ interface SearchRow {
type: string
time: number
surface: string
text: string
score: number
marked_text: string
match_count: number
document_length: number
}
interface CursorPayload {
@@ -149,27 +170,32 @@ export class SessionSearchSqlite extends SessionSearchService {
private _localGeneration = 0
private _tail: Promise<void> = Promise.resolve()
private _closed = false
private _closePromise: Promise<void> | undefined
private readonly _optionalPersistenceFiber: Fiber
constructor(ctx: Context, config: Config) {
super(ctx)
this.config = resolveConfig(config)
this._ready = this._open()
ctx.effect(() => {
const fiber = ctx.inject(['sessionPersistence'], (childCtx: Context) => {
const service = childCtx.sessionPersistence
const binding = {}
this._persistenceBinding = binding
this._persistence = service
// Attach a rejection observer immediately; callers still receive the same
// rejection from `_ready`, including when no search is ever attempted.
void this._ready.catch(() => undefined)
this._optionalPersistenceFiber = ctx.inject(['sessionPersistence'], (childCtx: Context) => {
const service = childCtx.sessionPersistence
const binding = {}
this._persistenceBinding = binding
this._persistence = service
this._persistenceRevision += 1
childCtx.effect(() => () => {
/* v8 ignore next -- a stale optional-service disposer cannot clear a replacement */
if (this._persistenceBinding !== binding) return
this._persistenceBinding = undefined
this._persistence = undefined
this._persistenceRevision += 1
childCtx.effect(() => () => {
/* v8 ignore next -- a stale optional-service disposer cannot clear a replacement */
if (this._persistenceBinding !== binding) return
this._persistenceBinding = undefined
this._persistence = undefined
this._persistenceRevision += 1
}, 'sessionSearchSqlite.persistenceBinding')
})
return () => void fiber.dispose()
}, 'sessionSearchSqlite.persistenceBinding')
})
ctx.effect(() => {
return () => this._optionalPersistenceFiber.dispose()
}, 'sessionSearchSqlite.optionalPersistence')
ctx.effect(() => async () => this.close(), 'sessionSearchSqlite.close')
}
@@ -179,17 +205,18 @@ export class SessionSearchSqlite extends SessionSearchService {
exec?: SessionSearchExecContext,
): Promise<SessionSearchPage<SessionSearchHit>> {
const normalized = normalizeSessionRequest(request, this.config)
return this._serialized(exec?.signal, async () => {
await this._ensureReady(exec?.signal)
await this._reconcile(exec?.signal)
assertNotAborted(exec?.signal)
const signal = exec?.signal
return this._serialized(signal, async () => {
await this._ensureReady(signal)
await this._reconcile(signal)
assertNotAborted(signal)
const generation = String(this._globalGeneration)
const fingerprint = requestFingerprint(normalized)
const offset = normalized.cursor === undefined
? 0
: decodeCursor(normalized.cursor, this._instance, 'sessions', fingerprint, generation)
const rows = this._querySessions(normalized, offset)
return page(rows, normalized.limit, row => this._sessionHit(row, normalized.query), cursorOffset => encodeCursor({
return page(rows, normalized.limit, row => this._sessionHit(row), cursorOffset => encodeCursor({
version: 1,
instance: this._instance,
scope: 'sessions',
@@ -205,17 +232,18 @@ export class SessionSearchSqlite extends SessionSearchService {
exec?: SessionSearchExecContext,
): Promise<SessionSearchPage<SessionEventSearchHit>> {
const normalized = normalizeEventRequest(request, this.config)
return this._serialized(exec?.signal, async () => {
await this._ensureReady(exec?.signal)
await this._reconcile(exec?.signal)
assertNotAborted(exec?.signal)
const signal = exec?.signal
return this._serialized(signal, async () => {
await this._ensureReady(signal)
await this._reconcile(signal)
assertNotAborted(signal)
const generation = this._targetGeneration(normalized.sessionId)
const fingerprint = requestFingerprint(normalized)
const offset = normalized.cursor === undefined
? 0
: decodeCursor(normalized.cursor, this._instance, 'events', fingerprint, generation)
const rows = this._queryEvents(normalized, offset)
return page(rows, normalized.limit, row => this._eventHit(row, normalized.query), cursorOffset => encodeCursor({
return page(rows, normalized.limit, row => this._eventHit(row), cursorOffset => encodeCursor({
version: 1,
instance: this._instance,
scope: 'events',
@@ -227,8 +255,12 @@ export class SessionSearchSqlite extends SessionSearchService {
}
/** Close the database after every accepted operation reaches quiescence. */
async close(): Promise<void> {
if (this._closed) return
close(): Promise<void> {
this._closePromise ??= this._close()
return this._closePromise
}
private async _close(): Promise<void> {
this._closed = true
await this._tail
try {
@@ -287,20 +319,20 @@ export class SessionSearchSqlite extends SessionSearchService {
}
private async _reconcile(signal: AbortSignal | undefined): Promise<void> {
const observation = await this._observeStable(signal)
assertNotAborted(signal)
const db = this._requireDb()
const persistedRows = db.prepare(
'SELECT id, fingerprint, generation FROM persisted_sessions',
).all() as unknown as IndexedRow[]
'SELECT id, revision, generation FROM persisted_sessions',
).all() as unknown as IndexedPersistedRow[]
const liveRows = db.prepare(
'SELECT id, fingerprint, generation FROM temp.live_sessions',
).all() as unknown as IndexedRow[]
).all() as unknown as IndexedLiveRow[]
const persistedById = new Map(persistedRows.map(row => [row.id as SessionId, row]))
const liveById = new Map(liveRows.map(row => [row.id as SessionId, row]))
const observation = await this._observeStable(persistedById, signal)
assertNotAborted(signal)
const persistentChanges = observation.persistence === undefined
? []
: [...observation.persisted.values()].filter(entry => persistedById.get(entry.header.id)?.fingerprint !== entry.fingerprint)
: [...observation.persisted.values()].filter(entry => entry.loaded !== undefined)
const persistentDeletes = observation.persistence === undefined
? []
: persistedRows.filter(row => !observation.persisted.has(row.id as SessionId))
@@ -327,13 +359,17 @@ export class SessionSearchSqlite extends SessionSearchService {
db.exec('BEGIN IMMEDIATE')
began = true
for (const row of persistentDeletes) this._deleteSession('persisted', row.id as SessionId)
for (const entry of persistentChanges) this._replaceSession('persisted', entry, nextMainGeneration)
for (const entry of persistentChanges) {
/* v8 ignore next -- observation loads every entry whose revision differs */
if (entry.loaded === undefined) throw new Error(`missing loaded revision for session "${entry.header.id}"`)
this._replacePersistedSession(entry.loaded, entry.revision, nextMainGeneration)
}
if (persistentChanges.length > 0 || persistentDeletes.length > 0) {
db.prepare('UPDATE search_state SET global_generation = ? WHERE singleton = 1').run(nextMainGeneration)
}
for (const row of liveDeletes) this._deleteSession('live', row.id as SessionId)
for (const { entry, generation } of liveReplacements) {
this._replaceSession('live', entry, generation)
this._replaceLiveSession(entry, generation)
}
db.exec('COMMIT')
} catch (error: unknown) {
@@ -360,21 +396,39 @@ export class SessionSearchSqlite extends SessionSearchService {
this._lastPersistenceRevision = observation.persistenceRevision
}
private async _observeStable(signal: AbortSignal | undefined): Promise<Observation> {
private async _observeStable(
indexed: ReadonlyMap<SessionId, IndexedPersistedRow>,
signal: AbortSignal | undefined,
): Promise<Observation> {
for (;;) {
assertNotAborted(signal)
const persistence = this._persistence
const persistenceRevision = this._persistenceRevision
const persisted = new Map<SessionId, ObservedSession>()
let persisted = new Map<SessionId, ObservedPersistedSession>()
if (persistence !== undefined) {
try {
const headers = await waitWithAbort(persistence.list(), signal)
for (const listed of headers) {
const loaded = await waitWithAbort(persistence.load(listed.id), signal)
assertSessionHeadersCompatible(listed, loaded.meta)
persisted.set(listed.id, observeSession(loaded.meta, loaded.events))
const canReuseIndexed = this._lastPersistenceRevision === undefined
|| this._lastPersistenceRevision === persistenceRevision
const before = await waitWithAbort(persistence.listSnapshots(), signal)
persisted = materializePersistenceSnapshots(before)
for (const entry of persisted.values()) {
if (canReuseIndexed && indexed.get(entry.header.id)?.revision === entry.revision) continue
const loaded = await waitWithAbort(persistence.load(entry.header.id), signal)
assertSessionHeadersCompatible(entry.header, loaded.meta)
entry.loaded = observeSession(loaded.meta, loaded.events)
}
const after = materializePersistenceSnapshots(
await waitWithAbort(persistence.listSnapshots(), signal),
)
if (!samePersistenceSnapshots(persisted, after)) continue
if (this._persistenceRevision !== persistenceRevision) continue
} catch (error: unknown) {
if (isAbort(error) || signal?.aborted) {
throw new SessionQueryError('session-search aborted', 'SESSION_QUERY_ABORTED', {
cause: error,
})
}
if (this._persistenceRevision !== persistenceRevision) continue
if (error instanceof SessionQueryError) throw error
throw new SessionQueryError(
`session-search persistence observation failed: ${errorMessage(error)}`,
@@ -414,13 +468,50 @@ export class SessionSearchSqlite extends SessionSearchService {
}
}
private _replaceSession(source: 'persisted' | 'live', entry: ObservedSession, generation: number): void {
this._deleteSession(source, entry.header.id)
private _replacePersistedSession(
entry: ObservedSession,
revision: SessionPersistenceRevision,
generation: number,
): void {
this._deleteSession('persisted', entry.header.id)
const db = this._requireDb()
const sessionTable = source === 'persisted' ? 'persisted_sessions' : 'temp.live_sessions'
const docsTable = source === 'persisted' ? 'persisted_docs' : 'temp.live_docs'
db.prepare(`
INSERT INTO ${sessionTable}
INSERT INTO persisted_sessions
(id, version, created_at, cwd, parent_session, seed_length, revision, generation)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
`).run(
entry.header.id,
entry.header.version,
entry.header.createdAt,
entry.header.cwd ?? null,
entry.header.parentSession ?? null,
entry.header.seedLength ?? null,
revision,
generation,
)
const insert = db.prepare(`
INSERT INTO persisted_docs (text, session_id, seq, type, time, surface, codepoint_length)
VALUES (?, ?, ?, ?, ?, ?, ?)
`)
for (const document of entry.documents) {
const text = sanitizeFtsText(document.text)
insert.run(
text,
document.sessionId,
document.seq,
document.type,
document.time,
document.surface,
Array.from(text).length,
)
}
}
private _replaceLiveSession(entry: ObservedSession, generation: number): void {
this._deleteSession('live', entry.header.id)
const db = this._requireDb()
db.prepare(`
INSERT INTO temp.live_sessions
(id, version, created_at, cwd, parent_session, seed_length, fingerprint, generation)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
`).run(
@@ -434,11 +525,20 @@ export class SessionSearchSqlite extends SessionSearchService {
generation,
)
const insert = db.prepare(`
INSERT INTO ${docsTable} (text, session_id, seq, type, time, surface)
VALUES (?, ?, ?, ?, ?, ?)
INSERT INTO temp.live_docs (text, session_id, seq, type, time, surface, codepoint_length)
VALUES (?, ?, ?, ?, ?, ?, ?)
`)
for (const document of entry.documents) {
insert.run(document.text, document.sessionId, document.seq, document.type, document.time, document.surface)
const text = sanitizeFtsText(document.text)
insert.run(
text,
document.sessionId,
document.seq,
document.type,
document.time,
document.surface,
Array.from(text).length,
)
}
}
@@ -455,19 +555,16 @@ export class SessionSearchSqlite extends SessionSearchService {
ranked AS (
SELECT *, ROW_NUMBER() OVER (
PARTITION BY session_id
ORDER BY score ASC, time DESC, seq DESC
ORDER BY match_count DESC, document_length ASC, time DESC, seq DESC
) AS event_rank
FROM filtered
)
SELECT * FROM ranked
WHERE event_rank = 1
ORDER BY score ASC, time DESC, session_id ASC, seq DESC
ORDER BY match_count DESC, document_length ASC, time DESC, session_id ASC, seq DESC
LIMIT ? OFFSET ?
`).all(
quoteFtsData(request.query),
this._persistence === undefined ? 0 : 1,
this._persistence === undefined ? 0 : 1,
quoteFtsData(request.query),
...selectedDocumentsParams(request.query, this._persistence !== undefined),
...sessionWhere.params,
...eventWhere.params,
request.limit + 1,
@@ -483,13 +580,10 @@ export class SessionSearchSqlite extends SessionSearchService {
${selected.sql}
SELECT * FROM matched
WHERE ${where}
ORDER BY score ASC, time DESC, seq DESC
ORDER BY match_count DESC, document_length ASC, time DESC, seq DESC
LIMIT ? OFFSET ?
`).all(
quoteFtsData(request.query),
this._persistence === undefined ? 0 : 1,
this._persistence === undefined ? 0 : 1,
quoteFtsData(request.query),
...selectedDocumentsParams(request.query, this._persistence !== undefined),
request.sessionId,
...eventWhere.params,
request.limit + 1,
@@ -515,23 +609,23 @@ export class SessionSearchSqlite extends SessionSearchService {
)
}
private _sessionHit(row: SearchRow, query: string): SessionSearchHit {
private _sessionHit(row: SearchRow): SessionSearchHit {
return {
header: rowHeader(row),
live: row.live === 1,
persisted: row.persisted === 1,
bestMatch: this._eventHit(row, query),
bestMatch: this._eventHit(row),
}
}
private _eventHit(row: SearchRow, query: string): SessionEventSearchHit {
private _eventHit(row: SearchRow): SessionEventSearchHit {
return {
sessionId: row.session_id as SessionId,
seq: row.seq,
type: row.type as SessionEventSearchHit['type'],
time: row.time,
surface: row.surface as SessionEventSearchHit['surface'],
snippet: makeSnippet(row.text, query, this.config.snippetChars),
snippet: makeSnippet(row.marked_text, this.config.snippetChars),
}
}
@@ -548,7 +642,7 @@ export class SessionSearchSqlite extends SessionSearchService {
function selectedDocumentsSql(): { sql: string } {
return {
sql: `WITH matched AS (
sql: `WITH candidates AS (
SELECT
pd.session_id AS session_id,
ps.version AS version,
@@ -562,8 +656,8 @@ function selectedDocumentsSql(): { sql: string } {
pd.type AS type,
CAST(pd.time AS INTEGER) AS time,
pd.surface AS surface,
pd.text AS text,
bm25(persisted_docs) AS score
highlight(persisted_docs, 0, ?, ?) AS marked_text,
CAST(pd.codepoint_length AS INTEGER) AS document_length
FROM persisted_docs AS pd
JOIN persisted_sessions AS ps ON ps.id = pd.session_id
WHERE persisted_docs MATCH ?
@@ -585,15 +679,39 @@ function selectedDocumentsSql(): { sql: string } {
ld.type AS type,
CAST(ld.time AS INTEGER) AS time,
ld.surface AS surface,
ld.text AS text,
bm25(live_docs) AS score
highlight(live_docs, 0, ?, ?) AS marked_text,
CAST(ld.codepoint_length AS INTEGER) AS document_length
FROM temp.live_docs AS ld
JOIN temp.live_sessions AS ls ON ls.id = ld.session_id
WHERE live_docs MATCH ?
), matched AS (
SELECT *,
(
length(CAST(marked_text AS BLOB))
- length(CAST(replace(marked_text, ?, '') AS BLOB))
) / ? AS match_count
FROM candidates
)`,
}
}
function selectedDocumentsParams(query: string, persistenceVisible: boolean): Array<string | number> {
const expression = quoteFtsData(query)
const visible = persistenceVisible ? 1 : 0
return [
FTS_HIGHLIGHT_START,
FTS_HIGHLIGHT_END,
expression,
visible,
visible,
FTS_HIGHLIGHT_START,
FTS_HIGHLIGHT_END,
expression,
FTS_HIGHLIGHT_START,
Buffer.byteLength(FTS_HIGHLIGHT_START, 'utf8'),
]
}
function observeLive(session: Session): ObservedSession {
return observeSession(
structuredClone(session.header),
@@ -606,7 +724,6 @@ function observeSession(header: SessionHeader, events: readonly SessionEvent[]):
const detachedEvents = events.map(event => structuredClone(event))
return {
header: detachedHeader,
events: detachedEvents,
documents: buildSessionEventSearchDocuments(detachedHeader.id, detachedEvents),
fingerprint: createHash('sha256')
.update(JSON.stringify({ header: detachedHeader, events: detachedEvents }))
@@ -614,6 +731,49 @@ function observeSession(header: SessionHeader, events: readonly SessionEvent[]):
}
}
function materializePersistenceSnapshots(
snapshots: readonly SessionPersistenceSnapshot[],
): Map<SessionId, ObservedPersistedSession> {
if (!isRuntimeArray(snapshots)) throw new Error('persistence snapshots must be an array')
const result = new Map<SessionId, ObservedPersistedSession>()
for (const snapshot of snapshots) {
if (typeof snapshot.revision !== 'string') {
throw new Error('persistence snapshot revision must be a string')
}
const header = structuredClone(snapshot.header)
if (result.has(header.id)) {
throw new Error(`persistence listed duplicate session "${header.id}"`)
}
result.set(header.id, { header, revision: snapshot.revision })
}
return result
}
function samePersistenceSnapshots(
before: ReadonlyMap<SessionId, ObservedPersistedSession>,
after: ReadonlyMap<SessionId, ObservedPersistedSession>,
): boolean {
if (before.size !== after.size) return false
for (const [id, first] of before) {
const second = after.get(id)
if (
second === undefined
|| first.revision !== second.revision
|| !sameHeader(first.header, second.header)
) return false
}
return true
}
function sameHeader(a: SessionHeader, b: SessionHeader): boolean {
return a.version === b.version
&& a.id === b.id
&& a.createdAt === b.createdAt
&& a.cwd === b.cwd
&& a.parentSession === b.parentSession
&& a.seedLength === b.seedLength
}
function rowHeader(row: SearchRow): SessionHeader {
return {
version: row.version,
@@ -629,7 +789,7 @@ function page<Row, Item>(
rows: readonly Row[],
limit: number,
convert: (row: Row) => Item,
nextCursor: (offset: number) => string,
nextCursor: (offset: number) => SessionSearchCursorValue,
offset: number,
): SessionSearchPage<Item> {
const hasMore = rows.length > limit
@@ -639,12 +799,12 @@ function page<Row, Item>(
}
}
function encodeCursor(payload: CursorPayload): string {
return Buffer.from(JSON.stringify(payload), 'utf8').toString('base64url')
function encodeCursor(payload: CursorPayload): SessionSearchCursorValue {
return SessionSearchCursor(Buffer.from(JSON.stringify(payload), 'utf8').toString('base64url'))
}
function decodeCursor(
cursor: string,
cursor: SessionSearchCursorValue,
instance: string,
scope: CursorPayload['scope'],
fingerprint: string,
@@ -762,4 +922,8 @@ function errorMessage(error: unknown): string {
return error instanceof Error ? error.message : 'unknown error'
}
function isRuntimeArray(value: unknown): boolean {
return Array.isArray(value)
}
export default SessionSearchSqlite
@@ -2,16 +2,24 @@
import {
SessionQueryError,
filterSessionEventDocuments,
filterSessionResults,
materializeSessionEventResultFilters,
materializeSessionResultFilters,
} from '@deepseek-ai/dsh-session-query'
import type {
SessionAvailability,
SessionEventMetadataFilter,
SessionEventResultFilter,
SessionEventSearchRequest,
SessionResultFilter,
SessionSearchCursor,
SessionSearchRequest,
} from '@deepseek-ai/dsh-session-query'
/** Collision-free marker inserted before an FTS5 match by `highlight()`. */
export const FTS_HIGHLIGHT_START = '\uFDD0'
/** Collision-free marker inserted after an FTS5 match by `highlight()`. */
export const FTS_HIGHLIGHT_END = '\uFDD1'
/** Limit defaults needed to normalize a search request. */
export interface QueryLimits {
/** Page size used when the request omits one. */
@@ -26,7 +34,7 @@ export interface NormalizedSessionRequest {
sessionFilters: readonly SessionResultFilter[]
eventFilters: readonly SessionEventMetadataFilter[]
limit: number
cursor?: string
cursor?: SessionSearchCursor
}
/** Normalized within-session request. */
@@ -35,7 +43,7 @@ export interface NormalizedEventRequest {
query: string
filters: readonly SessionEventMetadataFilter[]
limit: number
cursor?: string
cursor?: SessionSearchCursor
}
/** Parameterized SQL predicate fragment. */
@@ -56,16 +64,15 @@ export function normalizeSessionRequest(
request: SessionSearchRequest,
limits: QueryLimits,
): NormalizedSessionRequest {
const sessionFilters = request.sessionFilters ?? []
const eventFilters = request.eventFilters ?? []
filterSessionResults([], sessionFilters)
filterSessionEventDocuments([], eventFilters)
const sessionFilters = materializeSessionResultFilters(request.sessionFilters ?? [])
const eventFilters = materializeMetadataFilters(request.eventFilters ?? [])
const cursor = materializeCursor(request.cursor)
return {
query: normalizeQuery(request.query),
sessionFilters,
eventFilters,
limit: normalizeLimit(request.limit, limits),
...request.cursor === undefined ? {} : { cursor: request.cursor },
...cursor === undefined ? {} : { cursor },
}
}
@@ -79,14 +86,17 @@ export function normalizeEventRequest(
request: SessionEventSearchRequest,
limits: QueryLimits,
): NormalizedEventRequest {
const filters = request.filters ?? []
filterSessionEventDocuments([], filters)
if (typeof request.sessionId !== 'string') {
throw new SessionQueryError('session-search session id must be text', 'SESSION_QUERY_INVALID_FILTER')
}
const filters = materializeMetadataFilters(request.filters ?? [])
const cursor = materializeCursor(request.cursor)
return {
sessionId: request.sessionId,
query: normalizeQuery(request.query),
filters,
limit: normalizeLimit(request.limit, limits),
...request.cursor === undefined ? {} : { cursor: request.cursor },
...cursor === undefined ? {} : { cursor },
}
}
@@ -115,9 +125,23 @@ export function buildSessionWhere(filters: readonly SessionResultFilter[]): SqlW
case 'availability': {
const availability = [...new Set(filter.values)]
if (availability.length === 0) clauses.push('0')
else if (availability.length === 1) clauses.push(`${availability[0]} = 1`)
else if (availability.length === 1) {
const value = availability[0] as SessionAvailability
switch (value) {
case 'live':
clauses.push('live = 1')
break
case 'persisted':
clauses.push('persisted = 1')
break
default:
unknownAvailability(value)
}
}
break
}
default:
unknownFilter(filter)
}
}
return { sql: clauses.join(' AND '), params }
@@ -145,6 +169,8 @@ export function buildEventWhere(filters: readonly SessionEventMetadataFilter[]):
case 'surface':
addList(clauses, params, 'surface', filter.values)
break
default:
unknownFilter(filter)
}
}
return { sql: clauses.join(' AND '), params }
@@ -159,6 +185,18 @@ export function quoteFtsData(query: string): string {
return `"${query.replaceAll('"', '""')}"`
}
/**
* Remove reserved marker collisions before text enters FTS5 or MATCH.
* @param text - extracted document text or normalized caller query.
* @returns text with reserved noncharacters mapped to replacement characters.
*/
export function sanitizeFtsText(text: string): string {
return text
.replaceAll('\0', '\uFFFD')
.replaceAll(FTS_HIGHLIGHT_START, '\uFFFD')
.replaceAll(FTS_HIGHLIGHT_END, '\uFFFD')
}
/**
* Build the stable normalized request identity stored in opaque cursors.
* @param request - normalized request whose filter ordering is canonicalized.
@@ -185,19 +223,16 @@ export function requestFingerprint(request: NormalizedSessionRequest | Normalize
/**
* Build a whitespace-normalized excerpt no longer than `maxChars`.
* @param text - complete extracted semantic document.
* @param query - normalized literal query used to position the excerpt.
* @param markedText - complete document with FTS5 `highlight()` markers.
* @param maxChars - maximum result length in Unicode code points.
* @returns bounded plain-text snippet.
*/
export function makeSnippet(text: string, query: string, maxChars: number): string {
const clean = text.replace(/\s+/gu, ' ').trim()
export function makeSnippet(markedText: string, maxChars: number): string {
const { text: clean, matchStart } = normalizeMarkedText(markedText)
const characters = Array.from(clean)
if (characters.length <= maxChars) return clean
if (maxChars === 1) return '…'
const foundUnits = clean.toLowerCase().indexOf(query.toLowerCase())
const found = foundUnits < 0 ? -1 : Array.from(clean.slice(0, foundUnits)).length
let start = found < 0 ? 0 : Math.max(0, found - Math.floor(maxChars / 3))
let start = Math.max(0, matchStart - Math.floor(maxChars / 3))
let prefix = start > 0 ? '…' : ''
let suffix = '…'
let contentLength = maxChars - prefix.length - suffix.length
@@ -216,6 +251,28 @@ export function makeSnippet(text: string, query: string, maxChars: number): stri
return `${prefix}${characters.slice(start, end).join('')}${suffix}`
}
function normalizeMarkedText(markedText: string): { text: string; matchStart: number } {
const characters: string[] = []
let matchStart: number | undefined
for (const character of markedText) {
if (character === FTS_HIGHLIGHT_START) {
matchStart ??= characters.length
continue
}
if (character === FTS_HIGHLIGHT_END) continue
if (/\s/u.test(character)) {
if (characters.length > 0 && characters.at(-1) !== ' ') characters.push(' ')
} else {
characters.push(character)
}
}
if (characters.at(-1) === ' ') characters.pop()
return {
text: characters.join(''),
matchStart: matchStart ?? 0,
}
}
function normalizeQuery(value: string): string {
if (typeof value !== 'string') {
throw new SessionQueryError('session-search query must be text', 'SESSION_QUERY_INVALID_QUERY')
@@ -227,7 +284,44 @@ function normalizeQuery(value: string): string {
'SESSION_QUERY_INVALID_QUERY',
)
}
return query
if (query.includes('\0')) {
throw new SessionQueryError(
'session-search query must not contain NUL',
'SESSION_QUERY_INVALID_QUERY',
)
}
return sanitizeFtsText(query)
}
function materializeCursor(cursor: SessionSearchCursor | undefined): SessionSearchCursor | undefined {
if (cursor === undefined) return undefined
if (typeof cursor !== 'string') {
throw new SessionQueryError('session-search cursor must be text', 'SESSION_QUERY_INVALID_CURSOR')
}
return cursor
}
function materializeMetadataFilters(
filters: readonly SessionEventMetadataFilter[],
): SessionEventMetadataFilter[] {
const candidates: readonly SessionEventResultFilter[] = filters
for (const filter of candidates) {
switch (filter.kind) {
case 'seq':
case 'time':
case 'type':
case 'surface':
break
case 'text':
throw new SessionQueryError(
'session-search metadata filters do not accept text clauses',
'SESSION_QUERY_INVALID_FILTER',
)
default:
unknownFilter(filter)
}
}
return materializeSessionEventResultFilters(filters) as SessionEventMetadataFilter[]
}
function normalizeLimit(value: number | undefined, limits: QueryLimits): number {
@@ -310,3 +404,18 @@ function compareNullable(a: string | null, b: string | null): number {
if (b === null) return 1
return a.localeCompare(b)
}
function unknownAvailability(value: never): never {
throw new SessionQueryError(
`session availability filter contains unknown value "${String(value)}"`,
'SESSION_QUERY_INVALID_FILTER',
)
}
function unknownFilter(filter: never): never {
const kind = (filter as { kind?: unknown }).kind
throw new SessionQueryError(
`session filter contains unknown kind ${typeof kind === 'string' ? `"${kind}"` : '(missing)'}`,
'SESSION_QUERY_INVALID_FILTER',
)
}
@@ -5,7 +5,7 @@ import { mkdir } from 'node:fs/promises'
import { dirname, resolve } from 'node:path'
/** Current derived-index schema version. Incompatible versions reset in place. */
export const SESSION_QUERY_SQLITE_SCHEMA_VERSION = 1
export const SESSION_QUERY_SQLITE_SCHEMA_VERSION = 2
/** SQLite application id protecting unrelated databases from derived resets. */
export const SESSION_QUERY_SQLITE_APPLICATION_ID = 0x44534851
@@ -24,8 +24,6 @@ export async function openSearchDatabase(path: string, journalMode: JournalMode)
if (actual !== ':memory:') await mkdir(dirname(actual), { recursive: true, mode: 0o700 })
const db = new DatabaseSync(actual)
try {
// journalMode is a validated closed union, not caller-controlled SQL.
db.exec(`PRAGMA journal_mode = ${journalMode.toUpperCase()}`)
const { application_id: applicationId } = db.prepare('PRAGMA application_id').get() as { application_id: number }
const { user_version: version } = db.prepare('PRAGMA user_version').get() as { user_version: number }
const userTables = listUserTables(db)
@@ -38,6 +36,9 @@ export async function openSearchDatabase(path: string, journalMode: JournalMode)
if (applicationId === SESSION_QUERY_SQLITE_APPLICATION_ID && version !== SESSION_QUERY_SQLITE_SCHEMA_VERSION) {
resetDerivedSchema(db)
}
// Apply mutating pragmas only after refusing foreign or canonical files.
// journalMode is a validated closed union, not caller-controlled SQL.
db.exec(`PRAGMA journal_mode = ${journalMode.toUpperCase()}`)
ensurePersistentSchema(db)
ensureTemporarySchema(db)
return db
@@ -78,7 +79,7 @@ function ensurePersistentSchema(db: DatabaseSync): void {
cwd TEXT,
parent_session TEXT,
seed_length INTEGER,
fingerprint TEXT NOT NULL,
revision TEXT NOT NULL,
generation INTEGER NOT NULL
) STRICT
`)
@@ -90,6 +91,7 @@ function ensurePersistentSchema(db: DatabaseSync): void {
type UNINDEXED,
time UNINDEXED,
surface UNINDEXED,
codepoint_length UNINDEXED,
tokenize = 'unicode61'
)
`)
@@ -117,6 +119,7 @@ function ensureTemporarySchema(db: DatabaseSync): void {
type UNINDEXED,
time UNINDEXED,
surface UNINDEXED,
codepoint_length UNINDEXED,
tokenize = 'unicode61'
)
`)
@@ -1,9 +1,11 @@
import { describe, expect, it } from 'vitest'
import { SessionId } from '@deepseek-ai/dsh-session'
import type { SessionQueryErrorCode } from '@deepseek-ai/dsh-session-query'
import { SessionSearchCursor, type SessionQueryErrorCode } from '@deepseek-ai/dsh-session-query'
import {
buildEventWhere,
buildSessionWhere,
FTS_HIGHLIGHT_END,
FTS_HIGHLIGHT_START,
makeSnippet,
normalizeEventRequest,
normalizeSessionRequest,
@@ -32,13 +34,13 @@ describe('SQLite search request normalization', () => {
sessionFilters: [{ kind: 'availability', values: ['live'] }],
eventFilters: [{ kind: 'surface', values: ['current'] }],
limit: 3,
cursor: 'next',
cursor: SessionSearchCursor('next'),
}, limits)).toEqual({
query: 'needle',
sessionFilters: [{ kind: 'availability', values: ['live'] }],
eventFilters: [{ kind: 'surface', values: ['current'] }],
limit: 3,
cursor: 'next',
cursor: SessionSearchCursor('next'),
})
expect(normalizeEventRequest({ sessionId: SessionId('s'), query: 'needle' }, limits)).toEqual({
sessionId: SessionId('s'),
@@ -50,13 +52,13 @@ describe('SQLite search request normalization', () => {
sessionId: SessionId('s'),
query: 'needle',
filters: [{ kind: 'seq', from: 1 }],
cursor: 'next',
cursor: SessionSearchCursor('next'),
}, limits)).toEqual({
sessionId: SessionId('s'),
query: 'needle',
filters: [{ kind: 'seq', from: 1 }],
limit: 2,
cursor: 'next',
cursor: SessionSearchCursor('next'),
})
})
@@ -65,11 +67,39 @@ describe('SQLite search request normalization', () => {
.toThrow(expectCode('SESSION_QUERY_INVALID_QUERY'))
expect(() => normalizeSessionRequest({ query: ' \n ' }, limits))
.toThrow(expectCode('SESSION_QUERY_INVALID_QUERY'))
expect(() => normalizeSessionRequest({ query: 'bad\0query' }, limits))
.toThrow(expectCode('SESSION_QUERY_INVALID_QUERY'))
expect(() => normalizeEventRequest({ sessionId: 1 as never, query: 'x' }, limits))
.toThrow(expectCode('SESSION_QUERY_INVALID_FILTER'))
expect(() => normalizeEventRequest({
sessionId: SessionId('s'),
query: 'x',
cursor: 1 as never,
}, limits)).toThrow(expectCode('SESSION_QUERY_INVALID_CURSOR'))
expect(() => normalizeSessionRequest({
query: 'x',
eventFilters: [{ kind: 'text', text: 'x' } as never],
}, limits)).toThrow(expectCode('SESSION_QUERY_INVALID_FILTER'))
expect(() => normalizeSessionRequest({
query: 'x',
eventFilters: [{} as never],
}, limits)).toThrow(expectCode('SESSION_QUERY_INVALID_FILTER'))
for (const limit of [1.5, 0, 4]) {
expect(() => normalizeEventRequest({ sessionId: SessionId('s'), query: 'x', limit }, limits))
.toThrow(expectCode('SESSION_QUERY_INVALID_LIMIT'))
}
})
it('materializes owned filter values during normalization', () => {
const values = ['live'] as Array<'live' | 'persisted'>
const filter = { kind: 'availability' as const, values }
const request = { query: 'needle', sessionFilters: [filter] }
const normalized = normalizeSessionRequest(request, limits)
values[0] = 'persisted'
request.sessionFilters.push({ kind: 'availability', values: ['persisted'] })
expect(normalized.sessionFilters).toEqual([{ kind: 'availability', values: ['live'] }])
})
})
describe('SQLite search predicate compilation', () => {
@@ -120,6 +150,17 @@ describe('SQLite search predicate compilation', () => {
{ kind: 'surface', values: [] },
])).toEqual({ sql: '0 AND 0', params: [] })
})
it('rejects runtime-unknown filter discriminants in both SQL builders', () => {
expect(() => buildSessionWhere([{ kind: 'future' } as never]))
.toThrow(expectCode('SESSION_QUERY_INVALID_FILTER'))
expect(() => buildEventWhere([{ kind: 'future' } as never]))
.toThrow(expectCode('SESSION_QUERY_INVALID_FILTER'))
expect(() => buildSessionWhere([{ kind: 'availability', values: ['future'] } as never]))
.toThrow(expectCode('SESSION_QUERY_INVALID_FILTER'))
expect(() => buildSessionWhere([{} as never]))
.toThrow(expectCode('SESSION_QUERY_INVALID_FILTER'))
})
})
describe('SQLite query identity and presentation', () => {
@@ -169,11 +210,13 @@ describe('SQLite query identity and presentation', () => {
})
it('normalizes, bounds, and positions snippets by Unicode code point', () => {
expect(makeSnippet(' short\ntext ', 'absent', 20)).toBe('short text')
expect(makeSnippet('abcdef', 'f', 1)).toBe('…')
expect(makeSnippet('abcdefghij', 'absent', 5)).toBe('abcd…')
expect(makeSnippet('abcdefghij', 'c', 5)).toBe('…bcd…')
expect(makeSnippet('abcdef', 'f', 2)).toBe('a…')
expect(makeSnippet('abcdef', 'f', 5)).toBe('…cdef')
expect(makeSnippet(' short\ntext ', 20)).toBe('short text')
expect(makeSnippet(`abcde${FTS_HIGHLIGHT_START}f${FTS_HIGHLIGHT_END}`, 1)).toBe('…')
expect(makeSnippet('abcdefghij', 5)).toBe('abcd…')
expect(makeSnippet(`ab${FTS_HIGHLIGHT_START}c${FTS_HIGHLIGHT_END}defghij`, 5)).toBe('…bcd…')
expect(makeSnippet(`abcde${FTS_HIGHLIGHT_START}f${FTS_HIGHLIGHT_END}`, 2)).toBe('a…')
expect(makeSnippet(`abcde${FTS_HIGHLIGHT_START}f${FTS_HIGHLIGHT_END}`, 5)).toBe('…cdef')
expect(makeSnippet(` x—${FTS_HIGHLIGHT_START}café${FTS_HIGHLIGHT_END}\n y `, 20))
.toBe('x—café y')
})
})
@@ -1,18 +1,25 @@
import { afterEach, describe, expect, it } from 'vitest'
import { Context } from 'cordis'
import { afterEach, describe, expect, it, vi } from 'vitest'
import { Context, type Fiber } from 'cordis'
import { DatabaseSync } from 'node:sqlite'
import { mkdtemp, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import SessionStore, { SESSION_FORMAT_VERSION, SessionId } from '@deepseek-ai/dsh-session'
import type { SessionEvent, SessionHeader, SessionId as SessionIdType } from '@deepseek-ai/dsh-session'
import SessionPersistence from '@deepseek-ai/dsh-session-persistence'
import SessionPersistence, { SessionPersistenceRevision } from '@deepseek-ai/dsh-session-persistence'
import type { SessionPersistenceSnapshot } from '@deepseek-ai/dsh-session-persistence'
import SessionPersistenceSqlite from '@deepseek-ai/dsh-session-persistence-sqlite'
import SessionSearchSqlite, {
SESSION_QUERY_SQLITE_APPLICATION_ID,
SESSION_QUERY_SQLITE_SCHEMA_VERSION,
} from '@deepseek-ai/dsh-session-query-sqlite'
import type { SessionQueryErrorCode } from '@deepseek-ai/dsh-session-query'
import {
SessionQueryError,
SessionSearchCursor,
type SessionAvailability,
type SessionQueryErrorCode,
type SessionSearchRequest,
} from '@deepseek-ai/dsh-session-query'
const temporaryDirectories: string[] = []
@@ -48,19 +55,36 @@ function expectCode(code: SessionQueryErrorCode): Error {
class TestPersistence extends SessionPersistence {
static entries = new Map<SessionIdType, { meta: SessionHeader; events: SessionEvent[] }>()
static revisions = new Map<SessionIdType, number>()
static nextRevision = 0
static loads = new Map<SessionIdType, number>()
static loadEffect: ((entry: { meta: SessionHeader; events: SessionEvent[] }) => void) | undefined
static listGate: Promise<void> | undefined
static listStarted: (() => void) | undefined
static snapshotEffect: (() => void | Promise<void>) | undefined
static snapshotOverride: (() => SessionPersistenceSnapshot[]) | undefined
static failure: unknown
static reset(entries: readonly { meta: SessionHeader; events: SessionEvent[] }[] = []): void {
this.entries = new Map(entries.map(entry => [entry.meta.id, structuredClone(entry)]))
this.entries = new Map()
this.revisions = new Map()
this.loads = new Map()
this.loadEffect = undefined
for (const entry of entries) this.set(entry)
this.listGate = undefined
this.listStarted = undefined
this.snapshotEffect = undefined
this.snapshotOverride = undefined
this.failure = undefined
}
static set(entry: { meta: SessionHeader; events: SessionEvent[] }): void {
this.entries.set(entry.meta.id, structuredClone(entry))
this.revisions.set(entry.meta.id, ++this.nextRevision)
}
create(meta: SessionHeader): Promise<void> {
TestPersistence.entries.set(meta.id, { meta: structuredClone(meta), events: [] })
TestPersistence.set({ meta, events: [] })
return Promise.resolve()
}
@@ -68,13 +92,21 @@ class TestPersistence extends SessionPersistence {
const entry = TestPersistence.entries.get(id)
if (entry === undefined) return Promise.reject(new Error('missing test session'))
entry.events.push(...structuredClone(events))
TestPersistence.revisions.set(id, ++TestPersistence.nextRevision)
return Promise.resolve()
}
async load(id: SessionIdType): Promise<{ meta: SessionHeader; events: SessionEvent[] }> {
TestPersistence.loads.set(id, (TestPersistence.loads.get(id) ?? 0) + 1)
if (TestPersistence.failure !== undefined) throw TestPersistence.failure
const entry = TestPersistence.entries.get(id)
if (entry === undefined) throw new Error('missing test session')
if (TestPersistence.loadEffect !== undefined) {
const effect = TestPersistence.loadEffect
TestPersistence.loadEffect = undefined
effect(entry)
TestPersistence.revisions.set(id, ++TestPersistence.nextRevision)
}
return structuredClone(entry)
}
@@ -84,6 +116,20 @@ class TestPersistence extends SessionPersistence {
if (TestPersistence.failure !== undefined) throw TestPersistence.failure
return [...TestPersistence.entries.values()].map(entry => structuredClone(entry.meta))
}
async listSnapshots(): Promise<SessionPersistenceSnapshot[]> {
TestPersistence.listStarted?.()
await TestPersistence.listGate
if (TestPersistence.failure !== undefined) throw TestPersistence.failure
const snapshots = TestPersistence.snapshotOverride?.()
?? [...TestPersistence.entries.values()].map(entry => ({
header: structuredClone(entry.meta),
revision: SessionPersistenceRevision(`test:${TestPersistence.revisions.get(entry.meta.id)}`),
}))
await TestPersistence.snapshotEffect?.()
return snapshots
}
}
async function liveContext(config: ConstructorParameters<typeof SessionSearchSqlite>[1] = { path: ':memory:' }): Promise<Context> {
@@ -175,6 +221,44 @@ describe('SQLite session search', () => {
await expect(ctx.sessionSearch.searchSessions({ query: '*' })).resolves.toEqual({ items: [] })
})
it('ranks live and persisted matches on one source-comparable contract', async () => {
const persisted = header('z-persisted')
TestPersistence.reset([
{ meta: persisted, events: messageEvents('needle needle', 10) },
...Array.from({ length: 12 }, (_, index) => ({
meta: header(`filler-${index}`),
events: messageEvents('needle', 10),
})),
])
const ctx = await liveContext()
const persistence = await ctx.plugin(TestPersistence)
ctx.sessions.create(SessionId('a-live'), {
seed: messageEvents('needle needle', 10),
meta: { createdAt: persisted.createdAt },
})
const result = await ctx.sessionSearch.searchSessions({
query: 'needle',
sessionFilters: [{ kind: 'id', values: [SessionId('a-live'), persisted.id] }],
})
expect(result.items.map(item => item.header.id)).toEqual([SessionId('a-live'), persisted.id])
await persistence.dispose()
})
it('positions snippets from FTS5 matches across diacritics and punctuation', async () => {
const ctx = await liveContext({ path: ':memory:', snippetChars: 14 })
const session = ctx.sessions.create(SessionId('snippet'), {
seed: messageEvents('long long long—café,\nnext value', 10),
})
const page = await ctx.sessionSearch.searchEvents({ sessionId: session.id, query: 'CAFE' })
expect(page.items).toHaveLength(1)
expect(page.items[0]!.snippet).toContain('café')
expect(page.items[0]!.snippet).toContain('—')
expect(page.items[0]!.snippet).not.toContain('\n')
expect(Array.from(page.items[0]!.snippet).length).toBeLessThanOrEqual(14)
})
it('binds cursors to requests and only invalidates within-session pages for target changes', async () => {
const ctx = await liveContext({ path: ':memory:', defaultLimit: 1, maxLimit: 5 })
const target = ctx.sessions.create(SessionId('target'), {
@@ -193,7 +277,7 @@ describe('SQLite session search', () => {
if (eventPage.nextCursor === undefined || sessionPage.nextCursor === undefined) throw new Error('expected cursors')
const eventKeys = eventPage.items.map(item => `${item.sessionId}:${item.seq}`)
let eventCursor: string | undefined = eventPage.nextCursor
let eventCursor: ReturnType<typeof SessionSearchCursor> | undefined = eventPage.nextCursor
while (eventCursor !== undefined) {
const next = await ctx.sessionSearch.searchEvents({
sessionId: target.id,
@@ -208,7 +292,7 @@ describe('SQLite session search', () => {
expect(new Set(eventKeys).size).toBe(eventKeys.length)
const sessionIds = sessionPage.items.map(item => item.header.id)
let sessionCursor: string | undefined = sessionPage.nextCursor
let sessionCursor: ReturnType<typeof SessionSearchCursor> | undefined = sessionPage.nextCursor
while (sessionCursor !== undefined) {
const next = await ctx.sessionSearch.searchSessions({ query: 'needle', limit: 1, cursor: sessionCursor })
sessionIds.push(...next.items.map(item => item.header.id))
@@ -251,6 +335,7 @@ describe('SQLite session search', () => {
{ sessionId: session.id, query: 'needle', limit: 4 },
{ sessionId: session.id, query: 'needle', filters: [{ kind: 'seq', from: 2, to: 1 }] },
{ sessionId: session.id, query: 'needle', filters: [{ kind: 'surface', values: ['future'] }] },
{ sessionId: session.id, query: 'bad\0query' },
] as const) {
await expect(ctx.sessionSearch.searchEvents(request as never)).rejects.toBeInstanceOf(Error)
}
@@ -258,7 +343,24 @@ describe('SQLite session search', () => {
query: 'needle',
sessionFilters: [{ kind: 'availability', values: ['remote' as never] }],
})).rejects.toThrow(expectCode('SESSION_QUERY_INVALID_FILTER'))
await expect(ctx.sessionSearch.searchEvents({ sessionId: session.id, query: 'needle', cursor: 'not-json' }))
await expect(ctx.sessionSearch.searchSessions({
query: 'needle',
sessionFilters: [{ kind: 'future' } as never],
})).rejects.toThrow(expectCode('SESSION_QUERY_INVALID_FILTER'))
await expect(ctx.sessionSearch.searchSessions({
query: 'needle',
eventFilters: [{ kind: 'future' } as never],
})).rejects.toThrow(expectCode('SESSION_QUERY_INVALID_FILTER'))
await expect(ctx.sessionSearch.searchEvents({
sessionId: session.id,
query: 'needle',
filters: [{ kind: 'future' } as never],
})).rejects.toThrow(expectCode('SESSION_QUERY_INVALID_FILTER'))
await expect(ctx.sessionSearch.searchEvents({
sessionId: session.id,
query: 'needle',
cursor: SessionSearchCursor('not-json'),
}))
.rejects.toThrow(expectCode('SESSION_QUERY_INVALID_CURSOR'))
await expect(ctx.sessionSearch.searchEvents({ sessionId: SessionId('absent'), query: 'needle' }))
.rejects.toThrow(expectCode('SESSION_QUERY_SESSION_NOT_FOUND'))
@@ -280,6 +382,37 @@ describe('SQLite session search', () => {
})
describe('SQLite reconciliation and source lifecycle', () => {
it('owns queued request and filter values before waiting for the serializer', async () => {
const durable = header('owned')
TestPersistence.reset([{ meta: durable, events: messageEvents('durable needle') }])
const ctx = await liveContext()
const persistence = await ctx.plugin(TestPersistence)
let release!: () => void
TestPersistence.listGate = new Promise<void>((resolve) => { release = resolve })
let markStarted!: () => void
const started = new Promise<void>((resolve) => { markStarted = resolve })
TestPersistence.listStarted = () => {
TestPersistence.listStarted = undefined
markStarted()
}
const blocking = ctx.sessionSearch.searchSessions({ query: 'needle' })
await started
const availability: SessionAvailability[] = ['persisted']
const request: SessionSearchRequest = {
query: 'needle',
sessionFilters: [{ kind: 'availability', values: availability }],
}
const queued = ctx.sessionSearch.searchSessions(request)
request.query = 'absent'
availability[0] = 'live'
release()
await expect(blocking).resolves.toMatchObject({ items: [{ header: durable }] })
await expect(queued).resolves.toMatchObject({ items: [{ header: durable }] })
await persistence.dispose()
})
it('mounts persistence dynamically, shadows with TEMP live rows, reveals, and hides on unmount', async () => {
const shared = header('shared', 10, { cwd: '/work' })
const durable = header('durable', 5)
@@ -311,7 +444,7 @@ describe('SQLite reconciliation and source lifecycle', () => {
.rejects.toThrow(expectCode('SESSION_QUERY_SESSION_NOT_FOUND'))
})
it('restarts observation when persistence unmounts during an asynchronous list', async () => {
it('discards a stale list rejection when persistence unmounts during observation', async () => {
const durable = header('racing')
TestPersistence.reset([{ meta: durable, events: messageEvents('durable needle') }])
const ctx = await liveContext()
@@ -328,10 +461,140 @@ describe('SQLite reconciliation and source lifecycle', () => {
const search = ctx.sessionSearch.searchSessions({ query: 'needle' })
await started
await persistenceFiber.dispose()
TestPersistence.failure = new Error('stale backend rejection')
release()
await expect(search).resolves.toEqual({ items: [] })
})
it('retries against a replacement after the prior binding rejects', async () => {
const durable = header('replacement')
TestPersistence.reset([{ meta: durable, events: messageEvents('durable needle') }])
const ctx = await liveContext()
const prior = await ctx.plugin(TestPersistence)
let rejectPrior!: (reason: unknown) => void
TestPersistence.listGate = new Promise<void>((_resolve, reject) => { rejectPrior = reject })
let markStarted!: () => void
const started = new Promise<void>((resolve) => { markStarted = resolve })
TestPersistence.listStarted = () => {
TestPersistence.listStarted = undefined
markStarted()
}
const search = ctx.sessionSearch.searchSessions({ query: 'needle' })
await started
await prior.dispose()
TestPersistence.listGate = undefined
const replacement = await ctx.plugin(TestPersistence)
rejectPrior(new Error('stale prior binding'))
await expect(search).resolves.toMatchObject({ items: [{ header: durable }] })
await replacement.dispose()
})
it('reloads a replacement source even when its opaque revisions collide', async () => {
const durable = header('colliding-replacement')
TestPersistence.reset([{ meta: durable, events: messageEvents('old content') }])
const revision = TestPersistence.revisions.get(durable.id)!
const ctx = await liveContext()
const prior = await ctx.plugin(TestPersistence)
await expect(ctx.sessionSearch.searchSessions({ query: 'old' }))
.resolves.toMatchObject({ items: [{ header: durable }] })
await prior.dispose()
TestPersistence.set({ meta: durable, events: messageEvents('new needle') })
TestPersistence.revisions.set(durable.id, revision)
const replacement = await ctx.plugin(TestPersistence)
const internals = ctx.sessionSearch as unknown as {
_lastPersistenceRevision: number
_persistenceRevision: number
}
expect(internals._persistenceRevision).not.toBe(internals._lastPersistenceRevision)
const page = await ctx.sessionSearch.searchSessions({ query: 'new needle' })
expect(TestPersistence.loads.get(durable.id)).toBe(2)
expect(page).toMatchObject({ items: [{ header: durable }] })
await expect(ctx.sessionSearch.searchSessions({ query: 'old' })).resolves.toEqual({ items: [] })
expect(TestPersistence.loads.get(durable.id)).toBe(2)
await replacement.dispose()
})
it('retries when a successful observation belongs to a source unmounted during listing', async () => {
const durable = header('successful-unmount')
TestPersistence.reset([{ meta: durable, events: messageEvents('durable needle') }])
const ctx = await liveContext()
const persistence = await ctx.plugin(TestPersistence)
let lists = 0
TestPersistence.snapshotEffect = async () => {
lists += 1
if (lists === 2) await persistence.dispose()
}
await expect(ctx.sessionSearch.searchSessions({ query: 'needle' })).resolves.toEqual({ items: [] })
expect(lists).toBe(2)
})
it('retries when the snapshot population changes during observation', async () => {
const first = header('first')
const added = header('added-during-list')
TestPersistence.reset([{ meta: first, events: messageEvents('first needle') }])
const ctx = await liveContext()
await ctx.plugin(TestPersistence)
TestPersistence.snapshotEffect = () => {
TestPersistence.snapshotEffect = undefined
TestPersistence.set({ meta: added, events: messageEvents('added needle') })
}
const page = await ctx.sessionSearch.searchSessions({ query: 'needle' })
expect(page.items.map(item => item.header.id).sort()).toEqual([added.id, first.id].sort())
expect(TestPersistence.loads.get(first.id)).toBe(2)
expect(TestPersistence.loads.get(added.id)).toBe(1)
})
it('retries if the source revision changes while live sessions are observed', async () => {
const durable = header('live-boundary-retry')
TestPersistence.reset([{ meta: durable, events: messageEvents('durable needle') }])
const ctx = await liveContext()
await ctx.plugin(TestPersistence)
const internals = ctx.sessionSearch as unknown as { _persistenceRevision: number }
const originalList = ctx.sessions.list.bind(ctx.sessions)
let bumped = false
const list = vi.spyOn(ctx.sessions, 'list').mockImplementation(() => {
if (!bumped) {
bumped = true
internals._persistenceRevision += 1
}
return originalList()
})
await expect(ctx.sessionSearch.searchSessions({ query: 'needle' }))
.resolves.toMatchObject({ items: [{ header: durable }] })
expect(TestPersistence.loads.get(durable.id)).toBe(2)
list.mockRestore()
})
it('rejects malformed snapshots and preserves typed persistence failures', async () => {
const durable = header('invalid-snapshot')
TestPersistence.reset([{ meta: durable, events: messageEvents('durable needle') }])
const ctx = await liveContext()
await ctx.plugin(TestPersistence)
TestPersistence.snapshotOverride = () => 'not-an-array' as never
await expect(ctx.sessionSearch.searchSessions({ query: 'needle' }))
.rejects.toThrow(expectCode('SESSION_QUERY_PERSISTENCE_FAILED'))
TestPersistence.snapshotOverride = () => [{ header: durable, revision: 1 as never }]
await expect(ctx.sessionSearch.searchSessions({ query: 'needle' }))
.rejects.toThrow(expectCode('SESSION_QUERY_PERSISTENCE_FAILED'))
TestPersistence.snapshotOverride = () => [
{ header: durable, revision: SessionPersistenceRevision('duplicate:1') },
{ header: durable, revision: SessionPersistenceRevision('duplicate:2') },
]
await expect(ctx.sessionSearch.searchSessions({ query: 'needle' }))
.rejects.toThrow(expectCode('SESSION_QUERY_PERSISTENCE_FAILED'))
TestPersistence.snapshotOverride = undefined
const typed = new SessionQueryError('typed persistence failure', 'SESSION_QUERY_PERSISTENCE_FAILED')
TestPersistence.failure = typed
await expect(ctx.sessionSearch.searchSessions({ query: 'needle' })).rejects.toBe(typed)
})
it('rejects immutable header conflicts between live and persisted sources', async () => {
const shared = header('conflict', 10)
TestPersistence.reset([{ meta: shared, events: messageEvents('persisted needle') }])
@@ -358,6 +621,9 @@ describe('SQLite reconciliation and source lifecycle', () => {
const firstPersistence = await first.plugin(TestPersistence)
const firstSearch = await first.plugin(SessionSearchSqlite, { path })
await first.sessionSearch.searchSessions({ query: 'needle' })
expect(Object.fromEntries(TestPersistence.loads)).toEqual({ unchanged: 1, changed: 1, deleted: 1 })
await first.sessionSearch.searchSessions({ query: 'needle' })
expect(Object.fromEntries(TestPersistence.loads)).toEqual({ unchanged: 1, changed: 1, deleted: 1 })
await firstSearch.dispose()
await firstPersistence.dispose()
@@ -368,14 +634,20 @@ describe('SQLite reconciliation and source lifecycle', () => {
const added = header('added')
TestPersistence.entries.delete(deleted.id)
TestPersistence.entries.set(changed.id, { meta: changed, events: messageEvents('changed needle') })
TestPersistence.entries.set(added.id, { meta: added, events: messageEvents('added needle') })
TestPersistence.set({ meta: changed, events: messageEvents('changed needle') })
TestPersistence.set({ meta: added, events: messageEvents('added needle') })
const second = new Context()
await second.plugin(SessionStore)
const secondPersistence = await second.plugin(TestPersistence)
const secondSearch = await second.plugin(SessionSearchSqlite, { path })
const result = await second.sessionSearch.searchSessions({ query: 'needle' })
expect(result.items.map(item => item.header.id).sort()).toEqual([added.id, changed.id, unchanged.id].sort())
expect(Object.fromEntries(TestPersistence.loads)).toEqual({
unchanged: 1,
changed: 2,
deleted: 1,
added: 1,
})
await secondSearch.dispose()
await secondPersistence.dispose()
@@ -409,10 +681,27 @@ describe('SQLite reconciliation and source lifecycle', () => {
await expect(second.sessionSearch.searchSessions({ query: 'live' })).resolves.toEqual({ items: [] })
await expect(second.sessionSearch.searchSessions({ query: 'persisted' }))
.resolves.toMatchObject({ items: [{ header: shared, live: false, persisted: true }] })
expect(TestPersistence.loads.get(shared.id)).toBe(1)
await searchAgain.dispose()
await persistenceAgain.dispose()
})
it('refreshes the stored revision after a mutating load repair', async () => {
const durable = header('repair')
TestPersistence.reset([{ meta: durable, events: messageEvents('before repair') }])
TestPersistence.loadEffect = (entry) => {
entry.events = messageEvents('repaired needle')
}
const ctx = await liveContext()
await ctx.plugin(TestPersistence)
await expect(ctx.sessionSearch.searchSessions({ query: 'repaired' }))
.resolves.toMatchObject({ items: [{ header: durable }] })
expect(TestPersistence.loads.get(durable.id)).toBe(2)
await ctx.sessionSearch.searchSessions({ query: 'repaired' })
expect(TestPersistence.loads.get(durable.id)).toBe(2)
})
it('recovers on the next search after source and SQLite transaction failures', async () => {
TestPersistence.reset([{ meta: header('durable'), events: messageEvents('durable needle') }])
const ctx = await liveContext()
@@ -462,15 +751,18 @@ describe('SQLite schema, cancellation, and real persistence integration', () =>
const foreignPath = await temporaryPath('foreign.db')
const foreign = new DatabaseSync(foreignPath)
foreign.exec('PRAGMA journal_mode = WAL')
foreign.exec('CREATE TABLE canonical(value TEXT)')
foreign.exec("INSERT INTO canonical VALUES ('safe')")
foreign.close()
const foreignCtx = await liveContext({ path: foreignPath })
const foreignCtx = await liveContext({ path: foreignPath, journalMode: 'delete' })
await expect(foreignCtx.sessionSearch.searchSessions({ query: 'needle' }))
.rejects.toThrow(expectCode('SESSION_QUERY_INDEX_FAILED'))
const stillForeign = new DatabaseSync(foreignPath)
expect(stillForeign.prepare('SELECT value FROM canonical').get()).toEqual({ value: 'safe' })
expect(stillForeign.prepare('PRAGMA journal_mode').get()).toEqual({ journal_mode: 'wal' })
stillForeign.close()
await (foreignCtx.sessionSearch as SessionSearchSqlite).close()
const otherAppPath = await temporaryPath('other-app.db')
const otherApp = new DatabaseSync(otherAppPath)
@@ -479,6 +771,25 @@ describe('SQLite schema, cancellation, and real persistence integration', () =>
const otherAppCtx = await liveContext({ path: otherAppPath })
await expect(otherAppCtx.sessionSearch.searchSessions({ query: 'needle' }))
.rejects.toThrow(expectCode('SESSION_QUERY_INDEX_FAILED'))
await (otherAppCtx.sessionSearch as SessionSearchSqlite).close()
})
it('observes asynchronous open rejection even when no query is made', async () => {
const path = await temporaryPath('never-queried.db')
const foreign = new DatabaseSync(path)
foreign.exec('CREATE TABLE canonical(value TEXT)')
foreign.close()
const unhandled: unknown[] = []
const onUnhandled = (reason: unknown) => { unhandled.push(reason) }
process.on('unhandledRejection', onUnhandled)
try {
const ctx = await liveContext({ path })
await new Promise<void>((resolve) => { setImmediate(resolve) })
expect(unhandled).toEqual([])
await (ctx.sessionSearch as SessionSearchSqlite).close()
} finally {
process.off('unhandledRejection', onUnhandled)
}
})
it('cancels both queued and in-flight source waits without committing them', async () => {
@@ -518,7 +829,7 @@ describe('SQLite schema, cancellation, and real persistence integration', () =>
releaseBlocking()
await expect(blocking).resolves.toEqual({ items: [] })
TestPersistence.entries.set(SessionId('uncommitted'), {
TestPersistence.set({
meta: header('uncommitted'),
events: messageEvents('durable needle'),
})
@@ -560,14 +871,38 @@ describe('SQLite schema, cancellation, and real persistence integration', () =>
await started
const queued = search.searchSessions({ query: 'needle' })
const closing = search.close()
const repeatedClose = search.close()
expect(repeatedClose).toBe(closing)
release()
await expect(accepted).resolves.toEqual({ items: [] })
await expect(queued).rejects.toThrow(expectCode('SESSION_QUERY_INDEX_FAILED'))
await closing
await Promise.all([closing, repeatedClose])
await expect(search.searchSessions({ query: 'needle' }))
.rejects.toThrow(expectCode('SESSION_QUERY_INDEX_FAILED'))
await search.close()
expect(search.close()).toBe(closing)
})
it('awaits optional-persistence child-fiber quiescence on disposal', async () => {
TestPersistence.reset()
const ctx = new Context()
await ctx.plugin(SessionStore)
const search = await ctx.plugin(SessionSearchSqlite, { path: ':memory:' })
const persistence = await ctx.plugin(TestPersistence)
const optional = (ctx.sessionSearch as unknown as {
_optionalPersistenceFiber: Fiber
})._optionalPersistenceFiber
let release!: () => void
const cleanup = new Promise<void>((resolve) => { release = resolve })
optional.ctx.effect(() => () => cleanup)
let settled = false
const disposing = search.dispose().then(() => { settled = true })
await Promise.resolve()
expect(settled).toBe(false)
release()
await disposing
await persistence.dispose()
})
it('combines the real SQLite persistence backend with the real search service keylessly', async () => {