부모 노트: Personal LLM Memory Lab

Executive decision

Formal coupled outcome: no eligible winner.

K3 policy incremental compiler × Q2 policy graph retrieval was the diagnostic leader at 31/80 end-to-end held-out cases (38.75%), but every K × Q pair failed at least one deployment hard gate. Keep K3 × Q2 as the next architecture target, not a production winner.

Current operation:

  1. Capture immutable evidence under raw/.
  2. Use deterministic ingest plus the governed compile workflow and compile Skill for writes.
  3. Keep metadata-v2 Markdown under wiki/ as the only durable memory.
  4. Apply policy scope before discovery or ranking.
  5. Use D1 BM25F + trust/time rerank + graph/source closure + bounded packing for reads.
  6. Require human review for trust promotion, contradiction resolution, sensitive routing, and forgetting verification.

Separate the failure domains

  • K: Was the source compiled into the correct durable memory transition?
  • Q conditional on K: Did retrieval correctly use the representation produced by a fixed compiler?
  • K × Q: Did the complete pipeline produce an acceptable, bounded answer premise?

The query pack and compilation pack each contain 100 authored cases: 20 development and 80 held-out. The compilation held-out suite was frozen after independent repairs and executed exactly once.

Query-only result

F1 was the formal query-only winner under the frozen context tie-breaker. D1 is the operational default.

  • A0 lexical: 70.85% macro; 1,630.5 median context chars; 18.76 ms p95.
  • B1 BM25F: 74.28%; 1,609.0 chars; 18.59 ms.
  • C1 graph/source closure: 86.73%; 2,361.0 chars; 18.14 ms.
  • D1 policy/trust: 87.98%; 2,378.5 chars; 18.20 ms.
  • E1 hybrid RRF: 87.98%; 2,350.0 chars; 26.15 ms.
  • F1 adaptive: 87.98%; 2,294.0 chars; 25.36 ms.

D1, E1, and F1 had the same 71/80 micro success. D1 had 28.2% lower p95 than F1, no dense-model dependency, and 3.7% more median context. B1 → C1 was the only supported large adjacent improvement: +12.45 points, Holm-adjusted p=.019531.

Compilation result

  • K0 digest: 7.50% hard-gate cases; 7.50% exact plan; 68.75% decision accuracy; 27.50% / 20.82% action precision / recall.
  • K1 schema: 31.25%; 26.25%; 70.00%; 41.92% / 39.23%.
  • K2 claim graph: 38.75%; 40.00%; 70.00%; 50.10% / 49.15%.
  • K3 policy incremental: 63.75%; 42.50%; 87.50%; 55.52% / 55.33%.

K3 led, with 100% privacy invariant, 98.75% conflict invariant, 95% lineage invariant, 70% review-queue accuracy, and 66.25% minimal-write exactness. It did not pass every compiler gate.

Coupled K × Q result

End-to-end held-out success:

  • K0: Q0 0.00%; Q1 0.00%; Q2 0.00%.
  • K1: Q0 7.50%; Q1 8.75%; Q2 11.25%.
  • K2: Q0 17.50%; Q1 22.50%; Q2 27.50%.
  • K3: Q0 28.75%; Q1 33.75%; Q2 38.75%.

For K3 × Q2, failure attribution was compilation 47, query 2, success 31. The leading bottleneck is compilation. Holding Q2 fixed, every compiler step produced a significant paired gain after Holm correction. Holding K3 fixed, Q0 → Q1 and Q1 → Q2 each added 5 points, but neither survived multiplicity correction.

The joint-case median context was 234.0 characters; the separate 88-probe query median was 234.5. Case-conditional success was 58.75%; probe success was 62.50%. These are different units and must not be merged.

Registered Q3 required a real dense implementation. The available trigram proxy was ineligible, so its forced-zero joint score is not evidence against dense retrieval.

Cost and scale

All token values are proxies, not tokenizer or billing counts. Query-only reports characters; compilation/scaling uses ceil(UTF-8 bytes / 4).

At 5,000 synthetic units:

  • K3 representation: 12.26 MB; compiler build p50: 17.35 ms.
  • K3 Q0/Q1/Q2 p95: approximately 137.0/191.2/222.8 ms.
  • K3 Q0/Q1/Q2 median contexts: 4,840/4,807.5/3,848 characters.
  • K3 × Q2 disposable-index build p50: 1,034.10 ms.

The current implementation exhaustively scores approved documents and rebuilds disposable state. A 50,000-unit number is only a linear extrapolation, not a capacity test.

Robustness and operability

Only 76/320 test-only metamorphic comparisons passed:

  • conflict-monotonic: 16/16;
  • policy-monotonic: 16/16;
  • equivalent-plan: 40/160;
  • minimal-delta: 4/128.

This blocks production promotion of K3.

The query Skill improved 18/20 → 20/20 exact routing cases; the compile Skill improved 6/20 → 8/20 exact plans. Neither small sequential A/B was statistically significant. The Skills remain useful procedural boundaries: one for reads, one for writes.

Direct and MCP-adapter semantics matched 29/29 synthetic cases, but the captured runtime lacked the Python mcp package. Live MCP operability is unproven. Agents can use the Wiki now through CLI and Skills; MCP is optional transport.

Management contract

The executable prompt catalog covers eight primary workflows: ingest, compile/cascade, query, research, reconcile, semantic audit, schema change, and privacy/forgetting. A task selects exactly one primary workflow and applies the shared contract for scope, allowed reads/writes, stop conditions, provenance, and completion evidence. Operational prompts and evaluation logs are never durable claim evidence.

Next implementation and test boundary

  1. Build one typed compiler core shared by production and evaluation.
  2. Raise equivalent-plan and minimal-delta consistency before a new freeze.
  3. Add a persistent, fingerprint-keyed sparse/graph index.
  4. Measure real registered dense Q3 only after policy filtering.
  5. Add answer-model and user-reviewed real-task evaluation without retaining sensitive prompts.
  6. Measure longitudinal review burden, trust decay, contradictions, and forgetting.
  7. Freeze a new suite only after implementation and preregistration are complete.

Evidence and limitations

Local durable synthesis:

wiki/syntheses/personal-llm-memory-architecture-evaluation.md

Immutable local results ledger:

raw/research/2026-07-15-personal-memory-ab-results.md

Conditional-query provenance addendum:

raw/research/2026-07-15-personal-memory-ab-results-addendum.md

Canonical compilation run:

operations/experiments/personal-memory-compilation-ab/test-dea77412ed6c/

Canonical query-only run:

operations/experiments/personal-memory-ab/20260714T234707Z-test-9ded220536/

These local paths were not present in the llm-wiki tree at migration time. This Notion report is kept as the remaining copy.

No relevant internal Notion page was found, so unrelated workspace content was excluded. This report does not prove generated-answer quality, universal privacy, live MCP, dense Q3, production throughput, or 50,000-unit capacity.