encoach_frontend_new_v3/docs/04-AI-Stack-Report.md

# EnCoach - AI Stack Technical Report

**Analysis Date:** March 8, 2026

---

## Table of Contents

1. [AI Stack Overview](#1-ai-stack-overview)
2. [OpenAI GPT (Content Generation & Grading)](#2-openai-gpt--content-generation--grading)
3. [OpenAI Whisper (Speech-to-Text)](#3-openai-whisper--speech-to-text)
4. [AWS Polly (Text-to-Speech)](#4-aws-polly--text-to-speech)
5. [FAISS + Sentence Transformers (RAG Training Tips)](#5-faiss--sentence-transformers--rag-training-tips)
6. [ELAI (AI Avatar Video Generation)](#6-elai--ai-avatar-video-generation)
7. [GPTZero (AI Writing Detection)](#7-gptzero--ai-writing-detection)
8. [End-to-End Data Flows](#8-end-to-end-data-flows)
9. [Frontend AI Integration Points](#9-frontend-ai-integration-points)
10. [Environment Variables & Configuration](#10-environment-variables--configuration)

---

## 1. AI Stack Overview

The EnCoach platform uses **6 AI/ML services** working together to power automated IELTS exam generation, grading, and personalized training.

```
┌─────────────────────────────────────────────────────────┐
│                    Frontend (Next.js)                    │
│  ExamEditor, ExamPage, Training, AIDetection components │
└────────────────────────┬────────────────────────────────┘
                         │ HTTP (JWT)
                         ▼
┌─────────────────────────────────────────────────────────┐
│                  Backend (FastAPI)                       │
│                                                         │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐              │
│  │ OpenAI   │  │ Whisper  │  │ AWS Polly│              │
│  │ GPT-4o   │  │ (local)  │  │ (cloud)  │              │
│  │ GPT-3.5  │  │ base     │  │ neural   │              │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘              │
│       │              │              │                    │
│  Content Gen    Transcription    TTS Audio               │
│  Grading        Speaking eval    Listening MP3s          │
│  Evaluation                                              │
│                                                         │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐              │
│  │ FAISS +  │  │  ELAI    │  │ GPTZero  │              │
│  │ SentTrans│  │ (cloud)  │  │ (cloud)  │              │
│  │ (local)  │  │ avatars  │  │ detect   │              │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘              │
│       │              │              │                    │
│  RAG Tips       Avatar Videos   AI Detection             │
│  Training       Speaking        Writing eval             │
│                                                         │
└─────────────────────────────────────────────────────────┘
```

| Service | Type | Purpose | Model/Version |
|---|---|---|---|
| **OpenAI GPT** | Cloud API | Content generation, grading, evaluation | `gpt-4o`, `gpt-3.5-turbo` |
| **OpenAI Whisper** | Local (self-hosted) | Speech-to-text transcription | `base` (~1 GB, 4 instances) |
| **AWS Polly** | Cloud API | Text-to-speech for listening | Neural engine, 11 voices |
| **FAISS + Sentence Transformers** | Local (self-hosted) | RAG-based training tips | `all-MiniLM-L6-v2` + `IndexFlatL2` |
| **ELAI** | Cloud API | AI avatar video generation | ElevenLabs/Azure voices |
| **GPTZero** | Cloud API | AI-generated text detection | v2/predict/text |

---

## 2. OpenAI GPT — Content Generation & Grading

### 2.1 Configuration

| Setting | Value |
|---|---|
| **Default Model** | `gpt-4o` |
| **Secondary Model** | `gpt-3.5-turbo` |
| **Max Tokens** | 4,097 (300 reserved) |
| **Response Format** | `json_object` |
| **Retry Limit** | 2 (on blacklist or missing fields) |
| **Content Filter** | Blacklisted words (religious, sexual, political terms) |

**Temperature settings:**

| Context | Temperature | Behavior |
|---|---|---|
| Grading | 0.1 | Near-deterministic, consistent evaluation |
| Tips / Summaries | 0.2 | Low creativity, factual output |
| Content Generation | 0.7 | Higher creativity for passages and tasks |

### 2.2 Content Generation

GPT generates all IELTS exam content. Each module has specific prompt templates:

#### Reading Module
- **Model:** `gpt-4o`, temperature 0.7
- **Generates:** Passages with title and text body
- **Difficulty scaling:**
  - Passage 1: easy
  - Passage 2: hard
  - Passage 3: very hard
- **Exercise types generated:** Fill in the blanks, True/False/Not Given, Matching headings, Multiple choice
- **Prompt pattern:** System prompt defines JSON schema → User prompt specifies passage difficulty, topic, and word count

#### Listening Module
- **Model:** `gpt-4o`, temperature 0.7
- **Generates:** Conversation scripts and monologues
- **Output format:**
  - Conversations: `{"conversation": [{"name", "gender", "text"}]}` — 2 or 4 speakers
  - Monologues: `{"monologue": "..."}` — social or academic context
- **Exercise types generated:** Same as reading (fill blanks, T/F/NG, matching, MC)

#### Writing Module
- **Task 1 (General):** Letter prompts — `gpt-3.5-turbo`
- **Task 1 (Academic):** Image-based prompts — `gpt-4o`
- **Task 2 (Essay):** Essay prompts — `gpt-4o`

#### Speaking Module
- **Model:** `gpt-4o`, temperature 0.7
- **Part 1:** 5 questions across 2 topics
- **Part 2:** 1 question + 3 follow-up prompts
- **Part 3:** 5 discussion questions

#### Level Test
- **Generates:** Multiple choice questions at varying difficulty levels
- **Supports:** Standard level, UTAS format, custom levels

### 2.3 Grading / Evaluation

GPT evaluates student answers using IELTS band scoring criteria:

#### Writing Grading
- **Model:** `gpt-4o`, temperature 0.1 (Task 1) / 0.7 (Task 2)
- **Runs in parallel:** Evaluation + Perfect Answer + Spelling Fix + GPTZero
- **Output JSON:**
  ```json
  {
    "comment": "Detailed commentary...",
    "overall": 6.5,
    "task_response": {
      "Task Achievement": { "score": 7, "comment": "..." },
      "Coherence and Cohesion": { "score": 6, "comment": "..." },
      "Lexical Resource": { "score": 7, "comment": "..." },
      "Grammatical Range and Accuracy": { "score": 6, "comment": "..." }
    }
  }
  ```
- **Perfect Answer:** GPT generates an ideal answer for comparison
- **Spelling Fix:** `gpt-3.5-turbo` at temperature 0.2 corrects transcription errors

#### Speaking Grading
- **Flow:** Audio → Whisper transcription → GPT-4o grading
- **Criteria:** Fluency & Coherence, Lexical Resource, Grammar, Pronunciation
- **Perfect Answers:** `gpt-4o` (Part 1), `gpt-3.5-turbo` (Parts 2/3)

#### Exam Summary
- **Model:** `gpt-3.5-turbo`, temperature 0.2
- **Uses:** OpenAI function calling (`save_evaluation_and_suggestions`)
- **Output:** Overall evaluation, suggestions, bullet points

### 2.4 Other GPT Uses
- **Training tips selection:** GPT selects relevant tips from FAISS results
- **Whisper overlap fix:** GPT-4o merges overlapping transcription segments
- **Short answer grading:** Grades fill-in-the-blank and short text answers

---

## 3. OpenAI Whisper — Speech-to-Text

### 3.1 Configuration

| Setting | Value |
|---|---|
| **Model** | `base` (~1 GB) |
| **Instances** | 4 (round-robin) |
| **Loading** | `in_memory=True` |
| **Concurrency** | `ThreadPoolExecutor` with 4 workers |
| **Language** | English |
| **Precision** | `fp16=False` |

### 3.2 How It Works

Whisper runs **locally** on the Cloud Run container (not via OpenAI's API). Four model instances are loaded into memory at startup and assigned to workers via round-robin.

**Processing pipeline:**

```
Audio Input
    │
    ▼
Resample to 16 kHz mono float32 (librosa)
    │
    ▼
Split into 30-second chunks (1/4 overlap)
    │
    ▼
Transcribe each chunk (Whisper base model)
    │
    ▼
Concatenate transcriptions
    │
    ▼
Fix overlapping text (GPT-4o removes duplicated words)
    │
    ▼
Final transcript
```

**Retry:** 3 attempts via `tenacity` library.

### 3.3 Where It's Connected

| Feature | Connection |
|---|---|
| **Speaking Grading** | Student audio → Whisper → transcript → GPT grading |
| **Audio Transcription** | Uploaded audio → Whisper → listening script for exam editor |

---

## 4. AWS Polly — Text-to-Speech

### 4.1 Configuration

| Setting | Value |
|---|---|
| **Engine** | Neural |
| **Output Format** | MP3 |
| **Region** | `eu-west-1` |
| **Max Chunk Size** | 3,000 characters (split at sentence boundaries) |

### 4.2 Available Voices

| Voice | Language/Accent |
|---|---|
| Danielle | American English |
| Gregory | American English |
| Kevin | American English |
| Ruth | American English |
| Stephen | American English |
| Arthur | British English |
| Olivia | Australian English |
| Ayanda | South African English |
| Aria | New Zealand English |
| Kajal | Indian English |
| Niamh | Irish English |

### 4.3 How It Works

**Listening exam audio generation:**

```
Generated Dialog/Monologue (from GPT)
    │
    ▼
Assign voices to speakers (random for monologue, per-speaker for dialog)
    │
    ▼
Split text into ≤3000 char chunks at sentence boundaries
    │
    ▼
AWS Polly Neural TTS → MP3 audio bytes
    │
    ▼
Concatenate audio segments
    │
    ▼
Upload to Firebase Storage
    │
    ▼
Return download URL
```

### 4.4 Where It's Connected

| Feature | Connection |
|---|---|
| **Listening Exam Audio** | GPT script → Polly TTS → MP3 → Firebase Storage → student playback |
| **Listening Instructions** | "Recording has now finished" scripts → Stephen voice → MP3 |

---

## 5. FAISS + Sentence Transformers — RAG Training Tips

### 5.1 Configuration

| Setting | Value |
|---|---|
| **Embeddings Model** | `all-MiniLM-L6-v2` (Sentence Transformers) |
| **Index Type** | `faiss.IndexFlatL2` (exact L2 search) |
| **Top-K** | 5 results per query |
| **Data Source** | `pathways_2_rw_with_ids.json` |

### 5.2 Knowledge Base Categories

| Category | Index File |
|---|---|
| `ct_focus` | `./faiss/ct_focus_tips_index.faiss` |
| `language_for_writing` | `./faiss/language_for_writing_tips_index.faiss` |
| `reading_skill` | `./faiss/reading_skill_tips_index.faiss` |
| `strategy` | `./faiss/strategy_tips_index.faiss` |
| `word_link` | `./faiss/word_link_tips_index.faiss` |
| `word_partners` | `./faiss/word_partners_tips_index.faiss` |
| `writing_skill` | `./faiss/writing_skill_tips_index.faiss` |

**Metadata:** `./faiss/tips_metadata.pkl` (pickle file with tip IDs and text)

### 5.3 How RAG Works

```
Student Exam Performance
    │
    ▼
GPT analyzes performance → generates queries (text + category)
    │
    ▼
Sentence Transformers encodes query → embedding vector
    │
    ▼
FAISS L2 search → top 5 matching tips per category
    │
    ▼
GPT selects most relevant tips from retrieved results
    │
    ▼
Tips stored in MongoDB and displayed to student
```

### 5.4 Where It's Connected

| Feature | Connection |
|---|---|
| **Training Module** | After exam → analyze weak areas → retrieve personalized tips → display training content |
| **Walkthrough** | Tips linked to specific reading/writing skills for guided learning |

---

## 6. ELAI — AI Avatar Video Generation

### 6.1 Configuration

| Setting | Value |
|---|---|
| **API Endpoint** | `https://apis.elai.io/api/v1/videos` |
| **Auth** | Bearer token (`ELAI_TOKEN`) |
| **Animation** | `fade_in` |
| **Language** | English |

### 6.2 Available Avatars

| Avatar | Gender | Voice Provider |
|---|---|---|
| Gia | Female | ElevenLabs |
| Vadim | Male | Azure |
| Orhan | Male | ElevenLabs |
| Flora | Female | Azure |
| Scarlett | Female | ElevenLabs |
| Parker | Male | Azure |
| Ethan | Male | ElevenLabs |

Each avatar has a unique `avatar_code`, `avatar_url`, `avatar_canvas` dimensions, `voice_id`, and `voice_provider`.

### 6.3 How It Works

```
Speaking Task Text (from GPT)
    │
    ▼
Select Avatar (user choice from available list)
    │
    ▼
Build video config (slide, avatar, canvas, logo, voice settings)
    │
    ▼
POST to ELAI API → create video
    │
    ▼
POST render request → start processing
    │
    ▼
Poll GET status every 10 seconds until "ready"
    │
    ▼
Return video URL for playback
```

### 6.4 Where It's Connected

| Feature | Connection |
|---|---|
| **Speaking Exam** | AI avatar presents speaking questions via video |
| **Exam Editor** | Teachers generate speaking videos while creating exams |

---

## 7. GPTZero — AI Writing Detection

### 7.1 Configuration

| Setting | Value |
|---|---|
| **API Endpoint** | `https://api.gptzero.me/v2/predict/text` |
| **Auth** | `x-api-key` header |
| **Multilingual** | `false` |

### 7.2 How It Works

```
Student's Writing Submission
    │
    ▼
POST to GPTZero API with document text
    │
    ▼
Response: predicted_class, confidence, per-sentence AI probability
    │
    ▼
Returned as part of writing evaluation
    │
    ▼
Frontend displays AI Detection component
```

**Response fields:**
- `predicted_class`: `ai`, `mixed`, or `human`
- `confidence_category`: confidence level
- `class_probabilities`: probability distribution
- `sentences`: per-sentence analysis with `highlight_sentence_for_ai` flag

### 7.3 Where It's Connected

| Feature | Connection |
|---|---|
| **Writing Grading** | Runs in parallel with GPT evaluation, perfect answer, and spelling fix |
| **Frontend UI** | `AIDetection` component displays results with radial progress, segmented bars, and highlighted AI-generated sentences |

---

## 8. End-to-End Data Flows

### 8.1 Exam Generation Flow

```
Teacher clicks "Generate" in Exam Editor
    │
    ▼
Frontend: POST /api/exam/generate/{module}/{sectionId}
    │
    ▼
Next.js API Route: proxies to BACKEND_URL/{module}/...
    │
    ▼
FastAPI Controller → Service
    │
    ├── Reading: GPT-4o generates passage + exercises
    ├── Listening: GPT-4o generates script → Polly TTS → MP3 → Firebase
    ├── Writing: GPT-4o/3.5 generates task prompt
    └── Speaking: GPT-4o generates questions → ELAI creates avatar video
    │
    ▼
Response with generated content → stored in exam editor state
```

### 8.2 Writing Grading Flow

```
Student submits writing answer
    │
    ▼
Frontend: POST /api/evaluate/writing
    │
    ▼
Next.js API: inserts pending evaluation in MongoDB → proxies to backend
    │
    ▼
FastAPI runs 4 tasks IN PARALLEL:
    ├── GPT-4o: Grade against IELTS band criteria (temp 0.1)
    ├── GPT-4o: Generate perfect answer for comparison
    ├── GPT-3.5: Fix spelling/transcription errors (temp 0.2)
    └── GPTZero: Detect AI-generated content
    │
    ▼
Combined result stored in MongoDB evaluation collection
    │
    ▼
Frontend polls /api/evaluate/status until complete
    │
    ▼
UI shows: band scores, detailed comments, perfect answer, AI detection
```

### 8.3 Speaking Grading Flow

```
Student records audio response
    │
    ▼
Frontend: POST /api/evaluate/speaking (FormData with audio)
    │
    ▼
Next.js API: inserts pending evaluation → proxies to backend
    │
    ▼
FastAPI pipeline:
    │
    ▼
Whisper (local): Transcribe audio → text
    │
    ▼
GPT-4o: Grade transcript against IELTS speaking criteria (temp 0.1)
    │
    ▼
GPT-4o/3.5: Generate perfect answer
    │
    ▼
Result stored in MongoDB
    │
    ▼
Frontend polls and displays scores + transcript + perfect answer
```

### 8.4 Training / Personalized Tips Flow

```
Student completes an exam
    │
    ▼
Frontend: POST /api/training
    │
    ▼
Backend analyzes exam performance with GPT
    │
    ▼
GPT generates search queries + categories
    │
    ▼
Sentence Transformers encodes queries → FAISS L2 search
    │
    ▼
Top 5 tips per category retrieved
    │
    ▼
GPT selects most relevant tips
    │
    ▼
Training content stored in MongoDB → displayed to student
```

---

## 9. Frontend AI Integration Points

### 9.1 API Routes (Next.js → FastAPI)

| Frontend Route | Backend Route | AI Services Used |
|---|---|---|
| `POST /api/evaluate/writing` | `BACKEND_URL/grade/writing/{task}` | GPT-4o, GPT-3.5, GPTZero |
| `POST /api/evaluate/speaking` | `BACKEND_URL/grade/speaking/{task}` | Whisper, GPT-4o, GPT-3.5 |
| `GET /api/exam/generate/reading/{id}` | `BACKEND_URL/reading/{passage}` | GPT-4o |
| `GET /api/exam/generate/listening/{id}` | `BACKEND_URL/listening/{section}` | GPT-4o |
| `GET /api/exam/generate/writing/{id}` | `BACKEND_URL/writing/{task}` | GPT-4o / GPT-3.5 |
| `GET /api/exam/generate/speaking/{id}` | `BACKEND_URL/speaking/{task}` | GPT-4o |
| `POST /api/exam/media/listening` | `BACKEND_URL/listening/media` | AWS Polly |
| `POST /api/exam/media/speaking` | `BACKEND_URL/speaking/media` | ELAI |
| `POST /api/transcribe` | `BACKEND_URL/listening/transcribe` | Whisper |
| `POST /api/training` | `BACKEND_URL/training/` | GPT, FAISS, Sentence Transformers |
| `GET /api/exam/avatars` | `BACKEND_URL/speaking/avatars` | ELAI |

### 9.2 Key UI Components

| Component | Purpose |
|---|---|
| `AIDetection.tsx` | Displays AI detection results (radial progress, highlighted sentences) |
| `GenerateBtn.tsx` | Brain icon button with spinner for content generation |
| `generateVideos.ts` | Manages ELAI video creation + polling loop |
| `ExamPage.tsx` | Triggers writing/speaking evaluation + polls for results |
| `useEvaluationPolling.tsx` | Hook that polls evaluation status until grading completes |
| `generation.tsx` | Page for generating exams (gated by permissions) |

### 9.3 Permissions

| Permission | Controls |
|---|---|
| `generate_reading` | Reading passage generation |
| `generate_listening` | Listening script generation |
| `generate_writing` | Writing prompt generation |
| `generate_speaking` | Speaking task generation |
| `generate_level` | Level test generation |

---

## 10. Environment Variables & Configuration

### 10.1 Required API Keys

| Variable | Service | Where Used |
|---|---|---|
| `OPENAI_API_KEY` | OpenAI GPT-4o / GPT-3.5 | Content generation, grading, evaluation |
| `AWS_ACCESS_KEY_ID` | AWS Polly | Text-to-speech |
| `AWS_SECRET_ACCESS_KEY` | AWS Polly | Text-to-speech |
| `ELAI_TOKEN` | ELAI | Avatar video generation |
| `GPT_ZERO_API_KEY` | GPTZero | AI writing detection |

### 10.2 Backend Service Wiring (Dependency Injection)

```
DI Container
├── llm           → OpenAI(client=AsyncOpenAI)
├── tts           → AWSPolly(client=polly_client)
├── stt           → OpenAIWhisper(model="base", num_models=4)
├── vid_gen       → ELAI(client=http_client, token, avatars, conf)
├── ai_detector   → GPTZero(client=http_client, key)
├── training_kb   → TrainingContentKnowledgeBase(embeddings=SentenceTransformer)
│
├── Controllers
│   ├── ReadingController    → uses llm
│   ├── ListeningController  → uses llm, tts
│   ├── WritingController    → uses llm, ai_detector
│   ├── SpeakingController   → uses llm, stt, vid_gen
│   ├── GradeController      → uses llm, stt, ai_detector
│   ├── LevelController      → uses llm
│   └── TrainingController   → uses llm, training_kb
```

### 10.3 Cost Drivers

| Service | Cost Model | Usage Pattern |
|---|---|---|
| **OpenAI GPT-4o** | Per token (input + output) | Every generation, every grading — highest cost |
| **OpenAI GPT-3.5** | Per token (cheaper) | Summaries, spelling, some writing tasks |
| **AWS Polly** | Per character (Neural) | Every listening exam audio |
| **ELAI** | Per video minute | Every speaking exam video |
| **GPTZero** | Per API call | Every writing grading |
| **Whisper (local)** | Compute only (no API cost) | Every speaking grading + transcription |
| **FAISS (local)** | Compute only (no API cost) | Every training session |