93c530eef2
- .github/workflows/ci.yml: two jobs — frontend (tsc --noEmit, lint, build, Playwright) and backend (Postgres 16 + odoo:19 --test-enable --test-tags encoach_api) — catches regressions before merge. - docs/adr/: start an Architecture Decision Record trail with 0001 canonical directory layout, 0002 JWT refresh flow, 0003 paginated response envelope, 0004 RAG metadata + chunking. - docs/PROJECT_SUMMARY.md §21 Hardening Release: full recap of the AI quality loop, compliance, Paymob, i18n, and CI work shipped in this drop, plus new DB tables, REST routes, frontend routes, verification results, and operator-facing configuration. - README.md refreshed for the v4 split-repo doctrine and the new feature surface. - new_project/DEPRECATED.md: formal retirement notice pointing at backend/ as the canonical tree. Made-with: Cursor
68 lines
2.8 KiB
Markdown
68 lines
2.8 KiB
Markdown
# ADR 0004: RAG metadata + chunking for vector store
|
|
|
|
- **Status:** Accepted
|
|
- **Date:** 2026-04-09
|
|
- **Deciders:** AI team, Platform team
|
|
|
|
## Context
|
|
|
|
The first cut of the vector store (`encoach_vector`) stored one embedding
|
|
per source record, keyed only by `(model, res_id)`. This had two problems:
|
|
|
|
1. **Long documents dominated similarity scores.** A 20 000-character lesson
|
|
would embed as one vector and out-vote shorter, more relevant passages.
|
|
2. **No tenancy filtering.** Retrieval could not be scoped to a particular
|
|
course, subject, entity, or taxonomy topic, which meant RAG pulled content
|
|
from unrelated tenants on multi-entity deployments.
|
|
|
|
The quality gate (`encoach_quality_gate`) also needed a way to deduplicate
|
|
re-ingested content so that re-running the indexer did not explode the table.
|
|
|
|
## Decision
|
|
|
|
Extend `encoach.vector.embedding` with RAG metadata columns:
|
|
|
|
| Field | Purpose |
|
|
|-------|---------|
|
|
| `course_id` | Scope to a specific course. |
|
|
| `subject_id` | Scope to a subject/domain. |
|
|
| `entity_id` | Tenancy filter — critical for institutional deployments. |
|
|
| `taxonomy` | Free-form tag (e.g. `"IELTS/writing/task1"`). |
|
|
| `content_hash` | SHA-256 of the raw chunk; used for dedup. |
|
|
| `chunk_index`, `chunk_total` | Position in the parent document. |
|
|
|
|
Chunking policy (see `encoach_vector.services.embedding_service`):
|
|
|
|
- Content ≤ 2 000 chars → embedded as a single chunk.
|
|
- Content > 2 000 chars → split on paragraph boundaries with ~200-char
|
|
overlap, each chunk embedded individually.
|
|
- Each chunk stores its `content_hash`; the uniqueness constraint is
|
|
`(model, res_id, chunk_index, content_hash)` so re-indexing is idempotent.
|
|
|
|
The indexer (`encoach_vector.services.indexer`) declares per-model metadata
|
|
mapping (which field feeds `course_id`, which feeds `subject_id`, etc.) so
|
|
adding a new source model is a single config entry.
|
|
|
|
`similarity_search` accepts any subset of the metadata as a filter and
|
|
applies it as a SQL `WHERE` clause before the vector distance computation.
|
|
|
|
## Consequences
|
|
|
|
- Positive: retrieval quality improves dramatically on long documents.
|
|
- Positive: multi-tenant deployments can scope RAG to a single entity.
|
|
- Positive: re-indexing is safe (idempotent) and cheap.
|
|
- Negative: the embedding table grows roughly linearly with document length.
|
|
Mitigated by the `content_hash` dedup and by keeping only the latest
|
|
revision per source record.
|
|
- Follow-up: expose a management action to purge embeddings for a retired
|
|
course or entity.
|
|
|
|
## Alternatives considered
|
|
|
|
- **Use an external vector DB (Pinecone, Weaviate).** Rejected — pgvector is
|
|
already in the Postgres image, keeping ops surface small. Can be revisited
|
|
if we outgrow it.
|
|
- **Chunk-per-sentence instead of paragraph.** Rejected — too many tiny
|
|
chunks, each losing context; paragraph-sized chunks strike a better
|
|
recall/precision balance for our domain.
|