skills/memory-onboarding/references/schema-guide.md

# Schemas, Observations & Relations

How to make notes structured enough to query and consistent enough to trust. This is the mechanical core of the system the onboarding builds.

## Contents

1. [Note anatomy](#note-anatomy)
2. [Observations](#observations)
3. [Relations](#relations)
4. [Picoschema syntax](#picoschema-syntax)
5. [Creating a schema](#creating-a-schema)
6. [Validation & drift](#validation--drift)
7. [Design guidelines](#design-guidelines)

---

## Note anatomy

Every well-formed note has: frontmatter (title, type, tags — generated by `write_note` from its parameters), a **body** of substantive prose, an **Observations** section, and a **Relations** section.

```markdown
# Fix Gutter Downspout

| Notes | Modification Date | Approved By |
|:--|:--|:--|
| Initial Document | July 12, 2026 | Dana Reyes |

Back downspout detached during the June storm; water is pooling against the
foundation on the north side. Needs a bracket and probably a new elbow section.
Hardware store trip required — measure the downspout diameter first (looks like 3x4).

## Observations
- [status] open
- [priority] high
- [due] July 20, 2026
- [context] home-maintenance

## Relations
- part_of [[House Projects]]
- relates_to [[June Storm Damage]]
```

**Write the body generously.** Search retrieves chunks from note bodies, so richer prose makes notes *more* discoverable, not less. Prose carries meaning; observations carry precision. A note that is only bullet-point observations reads like a database row and loses the context that makes it useful a year later.

## Observations

Categorized, atomic facts: `- [category] content #optional-tag`

- Categories are **free-form** — `[decision]`, `[status]`, `[symptom]`, `[cost]`, `[lesson]` — whatever fits. Consistency within a note type is what matters (and is what schemas enforce).
- **One fact per line.** `- [decision] Use JWT with 15-minute expiry` not a paragraph of three decisions.
- **Categories are queryable** — searching by category finds every note carrying that kind of fact.
- Tags add cross-cutting findability: `- [risk] no SLA on the API #vendor #reliability`.

In this system, observations do double duty: they're facts, and they're the **fields** schemas validate — for scalar, enum, and array fields. A Task schema requiring `status` means every task note carries `- [status] ...` as an observation. (Entity-reference fields validate against relations instead — see below.)

## Relations

Typed, directional links: `- relation_type [[Target Note Title]]`

- relation_type is a descriptive snake_case verb: `part_of`, `depends_on`, `assigned_to`, `replaces`, `paid_by`. Invent freely.
- Wiki-links in body prose also create edges (untyped `references`). Use the Relations section for the structural links, prose links for narrative mentions.
- Targets may not exist yet — forward links resolve on creation.
- `build_context` on a `memory://` URL walks these edges to assemble connected context — this is what makes the graph worth building.

## Picoschema syntax

Schemas are notes (conventionally in a `Schemas/` folder) whose frontmatter metadata defines fields:

```yaml
schema:
  status(enum): "[open, in-progress, blocked, done], current state"
  due?: string, due date if any (Month D, YYYY)
  priority?(enum): "[low, medium, high], urgency"
  project?: Project, parent project note
  steps?(array): string, ordered sub-steps
```

- Types: `string`, `integer`, `number`, `boolean`
- `?` suffix = optional field
- `(enum)` = fixed value list — use for anything status-like
- `(array)` = list field
- A capitalized type name (`Person`, `Project`) makes it a **relation field**, validated against the note's Relations section (the relation type must equal the field name: `- project [[Target Note]]`) — it creates a graph edge, not an observation
- Every field gets a comma-then-description: it's documentation the next assistant reads

## Creating a schema

```python
write_note(
  title="Task",
  directory="Schemas",
  note_type="schema",
  metadata={
    "entity": "Task",
    "version": 1,
    "schema": {
      "status(enum)": "[open, in-progress, blocked, done], current state",
      "due?": "string, due date (Month D, YYYY)",
      "priority?(enum)": "[low, medium, high], urgency",
      "project?": "Project, parent project"
    },
    "settings": {"validation": "warn"}
  },
  content="""# Task

Schema for task notes. Tasks live in Tasks/, titled with a short imperative
description. Status moves open → in-progress → done; blocked is a holding state.

## Observations
- [convention] Every task carries status as an observation
- [convention] Done tasks get a close-date appended before archiving"""
)
```

Key points:

- **`validation: warn`, not `strict`** — the modes are `warn`, `strict`, and `off`. Warn logs findings without blocking anything; `strict` escalates the same findings to errors. Start every schema on warn — a new user fighting rejected writes will abandon the system. Move a schema to `strict` only if something automated consumes the notes and malformed ones break it.
- **Content notes declare their type** via `note_type` matching the schema's entity (e.g. `note_type="task"`).
- **Scalar, enum, and array fields must appear as observations** in the note body (`- [status] open`) to satisfy validation — frontmatter-only values don't count. Putting them in both places gives you metadata search *and* schema validation. **Entity-reference fields are different**: they're satisfied by a relation, not an observation — a line in `## Relations` whose type matches the field name (`project?: Project` is satisfied by `- project [[Kitchen Renovation]]`).
- **Version in metadata** — bump on breaking changes (new required field, removed field, type change). Additive optional fields don't need a bump.

## Validation & drift

- `schema_validate(note_type="task")` — check all notes of a type, or a single note via `identifier=`. It reports **missing required fields** and **invalid enum values** — that's the whole contract. Fields the schema doesn't declare are only counted as informational "unmatched" metadata (schemas are a subset, not a straitjacket), and scalar values are not type-checked, so a clean validate doesn't guarantee more than required-field presence and enum conformance. Use `schema_diff` to surface undeclared fields.
- `schema_diff(note_type="task")` — finds drift: fields notes use that the schema doesn't define (candidates to add as optional), and schema fields nobody uses (candidates to drop).
- `schema_infer(note_type="task")` — when notes of a type already exist without a schema, infer one from their actual structure instead of designing from scratch.

Evolution loop: `schema_diff` → edit the schema note (append changelog row, bump version if breaking) → `schema_validate` → fix outliers. Suggest the user run this monthly-ish once the system has real volume.

## Design guidelines

- **Every note type gets a schema; every note gets observations.** These two are not scale-dependent. A domain the user wants "loose" gets a one-field schema and a one-observation minimum, not an exemption — unstructured notes can't be queried, validated, or trusted by a future assistant, and retrofitting structure later is far more expensive than carrying one line now.
- **2–4 required fields, everything else optional.** Required fields are a tax on every note; each one must earn its place. Status almost always earns it. `notes_from_that_one_meeting` does not.
- **Enums for anything status-like.** Free-text statuses drift (`done`, `Done`, `finished`, `complete ✓`) and become unqueryable. Define the vocabulary in the schema and list it in the domain instruction note.
- **Schema per entity type, not per folder.** A `Person` schema serves contacts, family, and coworkers even if they live in different folders.
- **Don't over-constrain.** A schema describes common structure; it is not a straitjacket. If the user fights a field during the first week, make it optional or delete it.
- **Schemas are documentation.** Even at `warn` with validation never run, the schema note tells the next assistant exactly what a conforming note looks like. Write field descriptions accordingly.