# ADR 0004: RAG metadata + chunking for vector store

- **Status:** Accepted
- **Date:** 2026-04-09
- **Deciders:** AI team, Platform team

## Context

The first cut of the vector store (`encoach_vector`) stored one embedding
per source record, keyed only by `(model, res_id)`. This had two problems:

1. **Long documents dominated similarity scores.** A 20 000-character lesson
   would embed as one vector and out-vote shorter, more relevant passages.
2. **No tenancy filtering.** Retrieval could not be scoped to a particular
   course, subject, entity, or taxonomy topic, which meant RAG pulled content
   from unrelated tenants on multi-entity deployments.

The quality gate (`encoach_quality_gate`) also needed a way to deduplicate
re-ingested content so that re-running the indexer did not explode the table.

## Decision

Extend `encoach.vector.embedding` with RAG metadata columns:

| Field | Purpose |
|-------|---------|
| `course_id` | Scope to a specific course. |
| `subject_id` | Scope to a subject/domain. |
| `entity_id` | Tenancy filter — critical for institutional deployments. |
| `taxonomy` | Free-form tag (e.g. `"IELTS/writing/task1"`). |
| `content_hash` | SHA-256 of the raw chunk; used for dedup. |
| `chunk_index`, `chunk_total` | Position in the parent document. |

Chunking policy (see `encoach_vector.services.embedding_service`):

- Content ≤ 2 000 chars → embedded as a single chunk.
- Content > 2 000 chars → split on paragraph boundaries with ~200-char
  overlap, each chunk embedded individually.
- Each chunk stores its `content_hash`; the uniqueness constraint is
  `(model, res_id, chunk_index, content_hash)` so re-indexing is idempotent.

The indexer (`encoach_vector.services.indexer`) declares per-model metadata
mapping (which field feeds `course_id`, which feeds `subject_id`, etc.) so
adding a new source model is a single config entry.

`similarity_search` accepts any subset of the metadata as a filter and
applies it as a SQL `WHERE` clause before the vector distance computation.

## Consequences

- Positive: retrieval quality improves dramatically on long documents.
- Positive: multi-tenant deployments can scope RAG to a single entity.
- Positive: re-indexing is safe (idempotent) and cheap.
- Negative: the embedding table grows roughly linearly with document length.
  Mitigated by the `content_hash` dedup and by keeping only the latest
  revision per source record.
- Follow-up: expose a management action to purge embeddings for a retired
  course or entity.

## Alternatives considered

- **Use an external vector DB (Pinecone, Weaviate).** Rejected — pgvector is
  already in the Postgres image, keeping ops surface small. Can be revisited
  if we outgrow it.
- **Chunk-per-sentence instead of paragraph.** Rejected — too many tiny
  chunks, each losing context; paragraph-sized chunks strike a better
  recall/precision balance for our domain.