yamen/full_encoach_platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(backend): Phase 2/3 hardening release

Browse Source 
Roadmap P0 — platform safety & ops
- Merge duplicate encoach.student.attempt/answer models into encoach_scoring
  and drop the stale encoach_exam_template copies.
- Remove duplicate /api/exam/* routes; canonicalize on one controller tree.
- Gate raw-SQL seeds in seed_demo_data.py behind an explicit env flag.
- Add /api/health and /api/health/ready (DB + LLM reachability) endpoints.
- Fix docker-compose + ship odoo-docker.conf for container-local runs.
- Enforce OpenAI request_timeout=30s and @jwt_required on all AI/coach routes.
- Promote canonical cefr_mapper to encoach_ai.services.cefr_mapper.
- JWT cache TTL=30s + invalidation hook on user mutation.

Roadmap P1 — exam correctness & data provenance
- Wire QualityChecker + IeltsValidator into exam submit with a
  pending_review gate (encoach_ai.services.question_validator).
- Populate RAG metadata (course_id, subject_id, entity_id, taxonomy) on
  encoach_vector embeddings and add a chunking pipeline (>2000 chars).
- Add provenance fields on encoach.question (model, prompt_hash, log_id)
  and validate LLM output with schema before DB insert.
- Unify response envelope to {items,total,page,size}.
- Approval reject rollback with savepoint atomicity.
- Ticket notifications on status/assignee change.

Roadmap P2 — performance & observability
- Reports: replace Python loops with SQL read_group aggregations.
- X-Request-ID middleware + structured JSON logs.
- In-process/Prometheus counters and openapi.py controller exporting a
  spec by scanning @http.route decorators.
- Paymob real checkout + HMAC-SHA512 webhook verification, backed by a
  new encoach.paymob.order model and ir.config_parameter credentials.
- JWT refresh tokens + revocation table.
- Composite DB indexes on hot report/ticket/attempt paths.

Roadmap P3 — human-in-the-loop & compliance
- Human-in-the-loop exam review workflow (pending_review → publish) with
  new review controller and status transitions.
- encoach.ai.prompt model + versioning + admin editor endpoints (one
  active version per key, render-preview dry run).
- Student feedback loop → encoach.ai.feedback (upsert per user/subject,
  admin triage + resolve endpoints).
- GDPR export (/api/gdpr/export) and right-to-erasure (/api/gdpr/delete)
  with anonymization, tombstone record, and admin-self-erasure guard.
- HttpCase smoke tests for /api/health and /api/health/ready.

Made-with: Cursor
This commit is contained in:
Yamen Ahmad
2026-04-19 14:16:09 +04:00
parent 47d09a3ce5
commit dcf5ea6941
70 changed files with 4216 additions and 711 deletions
Download Patch File Download Diff File Expand all files Collapse all files

57
backend/custom_addons/encoach_vector/models/embedding.py
View File

@@ -28,9 +28,17 @@ class EncoachEmbedding(models.Model):
content_text = fields.Text()
metadata_json = fields.Text(default='{}')
course_id = fields.Integer(string='Course ID', index=True)
subject_id = fields.Integer(string='Subject ID', index=True)
entity_id = fields.Integer(string='Entity ID', index=True)
taxonomy = fields.Char(string='Taxonomy', index=True)
content_hash = fields.Char(string='Content SHA256', index=True)
chunk_index = fields.Integer(string='Chunk #', default=0)
chunk_total = fields.Integer(string='Total Chunks', default=1)
_content_unique = models.Constraint(
'UNIQUE(content_type, content_id)',
'Each content item can only have one embedding.',
'UNIQUE(content_type, content_id, chunk_index)',
'Each content item chunk can only have one embedding.',
)
@api.model
@@ -82,27 +90,46 @@ class EncoachEmbedding(models.Model):
return index_all(self.env)
@api.model
def similarity_search(self, query_vector, *, content_type=None, limit=10):
"""Find similar embeddings using cosine distance."""
def similarity_search(self, query_vector, *, content_type=None, limit=10,
course_id=None, subject_id=None, entity_id=None,
taxonomy=None):
"""Find similar embeddings using cosine distance.
Optional metadata filters (course/subject/entity/taxonomy) are applied
as SQL equality predicates to narrow the candidate set before ranking.
"""
vec_str = '[' + ','.join(str(v) for v in query_vector) + ']'
where = "WHERE embedding IS NOT NULL"
params = [vec_str, limit]
conditions = ["embedding IS NOT NULL"]
extra_params = []
if content_type:
where += " AND content_type = %s"
params = [vec_str, content_type, limit]
conditions.append("content_type = %s")
extra_params.append(content_type)
if course_id:
conditions.append("course_id = %s")
extra_params.append(int(course_id))
if subject_id:
conditions.append("subject_id = %s")
extra_params.append(int(subject_id))
if entity_id:
conditions.append("entity_id = %s")
extra_params.append(int(entity_id))
if taxonomy:
conditions.append("taxonomy = %s")
extra_params.append(taxonomy)
where = "WHERE " + " AND ".join(conditions)
query = f"""
SELECT id, content_type, content_id, content_text, metadata_json,
course_id, subject_id, entity_id, taxonomy,
chunk_index, chunk_total,
1 - (embedding <=> %s::vector) AS similarity
FROM encoach_embedding
{where}
ORDER BY embedding <=> %s::vector
LIMIT %s
"""
if content_type:
self.env.cr.execute(query, (vec_str, content_type, vec_str, limit))
else:
self.env.cr.execute(query, (vec_str, vec_str, limit))
params = [vec_str] + extra_params + [vec_str, limit]
self.env.cr.execute(query, params)
results = []
for row in self.env.cr.dictfetchall():
@@ -117,6 +144,12 @@ class EncoachEmbedding(models.Model):
'content_id': row['content_id'],
'text': row['content_text'],
'metadata': metadata,
'course_id': row['course_id'] or None,
'subject_id': row['subject_id'] or None,
'entity_id': row['entity_id'] or None,
'taxonomy': row['taxonomy'] or None,
'chunk_index': row['chunk_index'],
'chunk_total': row['chunk_total'],
'similarity': round(row['similarity'], 4),
})
return results

157
backend/custom_addons/encoach_vector/services/embedding_service.py
View File

@@ -1,13 +1,81 @@
"""Embedding service — encode text and manage vector storage."""
import hashlib
import json
import logging
import re
import time
_logger = logging.getLogger(__name__)
_model_instance = None
CHUNK_CHAR_LIMIT = 2000
CHUNK_OVERLAP = 200
_PARA_SPLIT = re.compile(r"\n\s*\n+")
_SENT_SPLIT = re.compile(r"(?<=[.!?])\s+")
def _chunk_text(text, *, limit=CHUNK_CHAR_LIMIT, overlap=CHUNK_OVERLAP):
"""Split long text into semantically-aware chunks of at most ``limit`` chars.
Preserves paragraph and sentence boundaries where possible. Each chunk
after the first retains ``overlap`` chars of tail context for continuity.
Short inputs return a single-chunk list unchanged.
"""
text = (text or "").strip()
if not text:
return []
if len(text) <= limit:
return [text]
chunks = []
buf = ""
for para in _PARA_SPLIT.split(text):
para = para.strip()
if not para:
continue
candidate = (buf + "\n\n" + para) if buf else para
if len(candidate) <= limit:
buf = candidate
continue
if buf:
chunks.append(buf)
buf = ""
if len(para) <= limit:
buf = para
continue
for sent in _SENT_SPLIT.split(para):
sent = sent.strip()
if not sent:
continue
cand = (buf + " " + sent) if buf else sent
if len(cand) <= limit:
buf = cand
continue
if buf:
chunks.append(buf)
if len(sent) <= limit:
buf = sent
else:
for i in range(0, len(sent), limit - overlap):
chunks.append(sent[i:i + limit])
buf = ""
if buf:
chunks.append(buf)
if overlap > 0 and len(chunks) > 1:
with_overlap = [chunks[0]]
for i in range(1, len(chunks)):
tail = chunks[i - 1][-overlap:]
with_overlap.append((tail + " " + chunks[i]).strip())
chunks = with_overlap
return chunks
def _sha256(text):
return hashlib.sha256((text or "").encode("utf-8")).hexdigest()
def _get_model():
"""Lazy-load the sentence-transformers model (cached across calls)."""
@@ -47,42 +115,77 @@ class EmbeddingService:
return [e.tolist() for e in embeddings]
def upsert(self, content_type, content_id, text, metadata=None):
"""Encode and store (or update) a single embedding.
"""Encode and store (or update) one embedding per text chunk.
Long ``text`` is split by :func:`_chunk_text` into chunks of at most
``CHUNK_CHAR_LIMIT`` chars. Each chunk is stored as a separate row
keyed by ``(content_type, content_id, chunk_index)``. Dedicated
metadata columns (course_id, subject_id, entity_id, taxonomy,
content_hash) are populated from ``metadata`` when present.
Returns:
encoach.embedding record
list of encoach.embedding records (one per chunk). Empty list
if input text is blank.
"""
if not text or not text.strip():
return None
return []
existing = self.Embedding.search([
metadata = dict(metadata or {})
chunks = _chunk_text(text)
if not chunks:
return []
vectors = self.encode(chunks)
existing_all = self.Embedding.search([
('content_type', '=', content_type),
('content_id', '=', content_id),
], limit=1)
])
by_idx = {rec.chunk_index: rec for rec in existing_all}
stale = [rec for rec in existing_all if rec.chunk_index >= len(chunks)]
if stale:
self.Embedding.browse([r.id for r in stale]).unlink()
vectors = self.encode([text])
meta_str = json.dumps(metadata or {})
records = []
total = len(chunks)
for idx, (chunk, vec) in enumerate(zip(chunks, vectors)):
chunk_hash = _sha256(chunk)
base_vals = {
'content_text': chunk[:10000],
'metadata_json': json.dumps(metadata),
'course_id': int(metadata.get('course_id') or 0) or False,
'subject_id': int(metadata.get('subject_id') or 0) or False,
'entity_id': int(metadata.get('entity_id') or 0) or False,
'taxonomy': metadata.get('taxonomy') or False,
'content_hash': chunk_hash,
'chunk_index': idx,
'chunk_total': total,
}
rec = by_idx.get(idx)
if rec:
if rec.content_hash != chunk_hash or rec.chunk_total != total:
rec.write(base_vals)
rec.set_embedding(vec)
else:
rec.write({k: v for k, v in base_vals.items()
if k in ('metadata_json', 'chunk_total')})
else:
rec = self.Embedding.create({
'content_type': content_type,
'content_id': content_id,
**base_vals,
})
rec.set_embedding(vec)
records.append(rec)
return records
if existing:
existing.write({
'content_text': text[:10000],
'metadata_json': meta_str,
})
existing.set_embedding(vectors[0])
return existing
record = self.Embedding.create({
'content_type': content_type,
'content_id': content_id,
'content_text': text[:10000],
'metadata_json': meta_str,
})
record.set_embedding(vectors[0])
return record
def search(self, query, *, content_type=None, limit=10):
def search(self, query, *, content_type=None, limit=10,
course_id=None, subject_id=None, entity_id=None,
taxonomy=None):
"""Semantic search — encode query and find similar content.
Optional metadata filters narrow the candidate pool to a specific
course / subject / entity / taxonomy before ranking.
Returns:
list of dicts with text, metadata, similarity score
"""
@@ -95,6 +198,10 @@ class EmbeddingService:
vectors[0],
content_type=content_type,
limit=limit,
course_id=course_id,
subject_id=subject_id,
entity_id=entity_id,
taxonomy=taxonomy,
)
latency = int((time.time() - t0) * 1000)
_logger.info("Vector search for '%s' returned %d results in %dms",

45
backend/custom_addons/encoach_vector/services/indexer.py
View File

@@ -11,6 +11,8 @@ MODEL_CONFIG = [
'text_field': 'name',
'description_field': 'description',
'metadata_fields': [],
'course_field': 'id',
'subject_field': 'subject_id',
},
{
'model': 'encoach.resource',
@@ -18,13 +20,19 @@ MODEL_CONFIG = [
'text_field': 'name',
'description_field': 'content',
'metadata_fields': ['type', 'cefr_level', 'difficulty'],
'course_field': 'course_id',
'subject_field': 'subject_id',
'entity_field': 'entity_id',
'taxonomy_field': 'type',
},
{
'model': 'encoach.question',
'content_type': 'question',
'text_field': 'name',
'text_field': 'stem',
'description_field': None,
'metadata_fields': ['question_type', 'difficulty', 'skill'],
'subject_field': 'subject_id',
'taxonomy_field': 'skill',
},
{
'model': 'encoach.course.module',
@@ -32,6 +40,8 @@ MODEL_CONFIG = [
'text_field': 'name',
'description_field': 'description',
'metadata_fields': ['skill'],
'course_field': 'course_id',
'taxonomy_field': 'skill',
},
{
'model': 'encoach.ai.generation.log',
@@ -39,6 +49,7 @@ MODEL_CONFIG = [
'text_field': 'brief',
'description_field': 'generated_content',
'metadata_fields': ['course_type', 'status'],
'taxonomy_field': 'course_type',
},
{
'model': 'encoach.chapter.material',
@@ -46,10 +57,21 @@ MODEL_CONFIG = [
'text_field': 'name',
'description_field': 'description',
'metadata_fields': ['type'],
'course_field': 'course_id',
'taxonomy_field': 'type',
},
]
def _safe_id(record, field):
if not field or not hasattr(record, field):
return None
val = getattr(record, field)
if hasattr(val, 'id'):
return val.id or None
return int(val) if val else None
def _get_text(record, config):
"""Extract indexable text from a record."""
parts = []
@@ -69,13 +91,32 @@ def _get_text(record, config):
def _get_metadata(record, config):
"""Extract metadata dict from a record."""
"""Extract metadata dict from a record, including structured
course/subject/entity/taxonomy fields used by the RAG retriever."""
meta = {}
for f in config.get('metadata_fields', []):
if hasattr(record, f):
val = getattr(record, f)
if val:
meta[f] = str(val) if not isinstance(val, (int, float, bool)) else val
course_id = _safe_id(record, config.get('course_field'))
if course_id:
meta['course_id'] = course_id
subject_id = _safe_id(record, config.get('subject_field'))
if subject_id:
meta['subject_id'] = subject_id
entity_id = _safe_id(record, config.get('entity_field'))
if entity_id:
meta['entity_id'] = entity_id
taxonomy_field = config.get('taxonomy_field')
if taxonomy_field and hasattr(record, taxonomy_field):
tx = getattr(record, taxonomy_field)
if tx:
meta['taxonomy'] = (
str(tx.display_name) if hasattr(tx, 'display_name') else str(tx)
)
return meta