93c530eef2
- .github/workflows/ci.yml: two jobs — frontend (tsc --noEmit, lint, build, Playwright) and backend (Postgres 16 + odoo:19 --test-enable --test-tags encoach_api) — catches regressions before merge. - docs/adr/: start an Architecture Decision Record trail with 0001 canonical directory layout, 0002 JWT refresh flow, 0003 paginated response envelope, 0004 RAG metadata + chunking. - docs/PROJECT_SUMMARY.md §21 Hardening Release: full recap of the AI quality loop, compliance, Paymob, i18n, and CI work shipped in this drop, plus new DB tables, REST routes, frontend routes, verification results, and operator-facing configuration. - README.md refreshed for the v4 split-repo doctrine and the new feature surface. - new_project/DEPRECATED.md: formal retirement notice pointing at backend/ as the canonical tree. Made-with: Cursor
2.8 KiB
ADR 0004: RAG metadata + chunking for vector store
- Status: Accepted
- Date: 2026-04-09
- Deciders: AI team, Platform team
Context
The first cut of the vector store (encoach_vector) stored one embedding
per source record, keyed only by (model, res_id). This had two problems:
- Long documents dominated similarity scores. A 20 000-character lesson would embed as one vector and out-vote shorter, more relevant passages.
- No tenancy filtering. Retrieval could not be scoped to a particular course, subject, entity, or taxonomy topic, which meant RAG pulled content from unrelated tenants on multi-entity deployments.
The quality gate (encoach_quality_gate) also needed a way to deduplicate
re-ingested content so that re-running the indexer did not explode the table.
Decision
Extend encoach.vector.embedding with RAG metadata columns:
| Field | Purpose |
|---|---|
course_id |
Scope to a specific course. |
subject_id |
Scope to a subject/domain. |
entity_id |
Tenancy filter — critical for institutional deployments. |
taxonomy |
Free-form tag (e.g. "IELTS/writing/task1"). |
content_hash |
SHA-256 of the raw chunk; used for dedup. |
chunk_index, chunk_total |
Position in the parent document. |
Chunking policy (see encoach_vector.services.embedding_service):
- Content ≤ 2 000 chars → embedded as a single chunk.
- Content > 2 000 chars → split on paragraph boundaries with ~200-char overlap, each chunk embedded individually.
- Each chunk stores its
content_hash; the uniqueness constraint is(model, res_id, chunk_index, content_hash)so re-indexing is idempotent.
The indexer (encoach_vector.services.indexer) declares per-model metadata
mapping (which field feeds course_id, which feeds subject_id, etc.) so
adding a new source model is a single config entry.
similarity_search accepts any subset of the metadata as a filter and
applies it as a SQL WHERE clause before the vector distance computation.
Consequences
- Positive: retrieval quality improves dramatically on long documents.
- Positive: multi-tenant deployments can scope RAG to a single entity.
- Positive: re-indexing is safe (idempotent) and cheap.
- Negative: the embedding table grows roughly linearly with document length.
Mitigated by the
content_hashdedup and by keeping only the latest revision per source record. - Follow-up: expose a management action to purge embeddings for a retired course or entity.
Alternatives considered
- Use an external vector DB (Pinecone, Weaviate). Rejected — pgvector is already in the Postgres image, keeping ops surface small. Can be revisited if we outgrow it.
- Chunk-per-sentence instead of paragraph. Rejected — too many tiny chunks, each losing context; paragraph-sized chunks strike a better recall/precision balance for our domain.