encoach_frontend_new_v2/docs/04-AI-Stack-Report.md

EnCoach - AI Stack Technical Report

Analysis Date: March 8, 2026

---

Table of Contents

1. AI Stack Overview
2. OpenAI GPT (Content Generation & Grading)
3. OpenAI Whisper (Speech-to-Text)
4. AWS Polly (Text-to-Speech)
5. FAISS + Sentence Transformers (RAG Training Tips)
6. ELAI (AI Avatar Video Generation)
7. GPTZero (AI Writing Detection)
8. End-to-End Data Flows
9. Frontend AI Integration Points
10. Environment Variables & Configuration

---

1. AI Stack Overview

The EnCoach platform uses 6 AI/ML services working together to power automated IELTS exam generation, grading, and personalized training.

┌─────────────────────────────────────────────────────────┐
│                    Frontend (Next.js)                    │
│  ExamEditor, ExamPage, Training, AIDetection components │
└────────────────────────┬────────────────────────────────┘
                         │ HTTP (JWT)
                         ▼
┌─────────────────────────────────────────────────────────┐
│                  Backend (FastAPI)                       │
│                                                         │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐              │
│  │ OpenAI   │  │ Whisper  │  │ AWS Polly│              │
│  │ GPT-4o   │  │ (local)  │  │ (cloud)  │              │
│  │ GPT-3.5  │  │ base     │  │ neural   │              │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘              │
│       │              │              │                    │
│  Content Gen    Transcription    TTS Audio               │
│  Grading        Speaking eval    Listening MP3s          │
│  Evaluation                                              │
│                                                         │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐              │
│  │ FAISS +  │  │  ELAI    │  │ GPTZero  │              │
│  │ SentTrans│  │ (cloud)  │  │ (cloud)  │              │
│  │ (local)  │  │ avatars  │  │ detect   │              │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘              │
│       │              │              │                    │
│  RAG Tips       Avatar Videos   AI Detection             │
│  Training       Speaking        Writing eval             │
│                                                         │
└─────────────────────────────────────────────────────────┘

Service	Type	Purpose	Model/Version
OpenAI GPT	Cloud API	Content generation, grading, evaluation	gpt-4o, gpt-3.5-turbo
OpenAI Whisper	Local (self-hosted)	Speech-to-text transcription	base (~1 GB, 4 instances)
AWS Polly	Cloud API	Text-to-speech for listening	Neural engine, 11 voices
FAISS + Sentence Transformers	Local (self-hosted)	RAG-based training tips	all-MiniLM-L6-v2 + IndexFlatL2
ELAI	Cloud API	AI avatar video generation	ElevenLabs/Azure voices
GPTZero	Cloud API	AI-generated text detection	v2/predict/text

---

2. OpenAI GPT — Content Generation & Grading

2.1 Configuration

Setting	Value
Default Model	gpt-4o
Secondary Model	gpt-3.5-turbo
Max Tokens	4,097 (300 reserved)
Response Format	json_object
Retry Limit	2 (on blacklist or missing fields)
Content Filter	Blacklisted words (religious, sexual, political terms)

Temperature settings:

Context	Temperature	Behavior
Grading	0.1	Near-deterministic, consistent evaluation
Tips / Summaries	0.2	Low creativity, factual output
Content Generation	0.7	Higher creativity for passages and tasks

2.2 Content Generation

GPT generates all IELTS exam content. Each module has specific prompt templates:

Reading Module

• Model: gpt-4o, temperature 0.7
• Generates: Passages with title and text body
• Difficulty scaling:
  • Passage 1: easy
  • Passage 2: hard
  • Passage 3: very hard
• Exercise types generated: Fill in the blanks, True/False/Not Given, Matching headings, Multiple choice
• Prompt pattern: System prompt defines JSON schema → User prompt specifies passage difficulty, topic, and word count

Listening Module

• Model: gpt-4o, temperature 0.7
• Generates: Conversation scripts and monologues
• Output format:
  • Conversations: {"conversation": [{"name", "gender", "text"}]} — 2 or 4 speakers
  • Monologues: {"monologue": "..."} — social or academic context
• Exercise types generated: Same as reading (fill blanks, T/F/NG, matching, MC)

Writing Module

• Task 1 (General): Letter prompts — gpt-3.5-turbo
• Task 1 (Academic): Image-based prompts — gpt-4o
• Task 2 (Essay): Essay prompts — gpt-4o

Speaking Module

• Model: gpt-4o, temperature 0.7
• Part 1: 5 questions across 2 topics
• Part 2: 1 question + 3 follow-up prompts
• Part 3: 5 discussion questions

Level Test

• Generates: Multiple choice questions at varying difficulty levels
• Supports: Standard level, UTAS format, custom levels

2.3 Grading / Evaluation

GPT evaluates student answers using IELTS band scoring criteria:

Writing Grading

• Model: gpt-4o, temperature 0.1 (Task 1) / 0.7 (Task 2)
• Runs in parallel: Evaluation + Perfect Answer + Spelling Fix + GPTZero
• Output JSON:

  {
    "comment": "Detailed commentary...",
    "overall": 6.5,
    "task_response": {
      "Task Achievement": { "score": 7, "comment": "..." },
      "Coherence and Cohesion": { "score": 6, "comment": "..." },
      "Lexical Resource": { "score": 7, "comment": "..." },
      "Grammatical Range and Accuracy": { "score": 6, "comment": "..." }
    }
  }
  

• Perfect Answer: GPT generates an ideal answer for comparison
• Spelling Fix: gpt-3.5-turbo at temperature 0.2 corrects transcription errors

Speaking Grading

• Flow: Audio → Whisper transcription → GPT-4o grading
• Criteria: Fluency & Coherence, Lexical Resource, Grammar, Pronunciation
• Perfect Answers: gpt-4o (Part 1), gpt-3.5-turbo (Parts 2/3)

Exam Summary

• Model: gpt-3.5-turbo, temperature 0.2
• Uses: OpenAI function calling (save_evaluation_and_suggestions)
• Output: Overall evaluation, suggestions, bullet points

2.4 Other GPT Uses

• Training tips selection: GPT selects relevant tips from FAISS results
• Whisper overlap fix: GPT-4o merges overlapping transcription segments
• Short answer grading: Grades fill-in-the-blank and short text answers

---

3. OpenAI Whisper — Speech-to-Text

3.1 Configuration

Setting	Value
Model	base (~1 GB)
Instances	4 (round-robin)
Loading	in_memory=True
Concurrency	ThreadPoolExecutor with 4 workers
Language	English
Precision	fp16=False

3.2 How It Works

Whisper runs locally on the Cloud Run container (not via OpenAI's API). Four model instances are loaded into memory at startup and assigned to workers via round-robin.

Processing pipeline:

Audio Input
    │
    ▼
Resample to 16 kHz mono float32 (librosa)
    │
    ▼
Split into 30-second chunks (1/4 overlap)
    │
    ▼
Transcribe each chunk (Whisper base model)
    │
    ▼
Concatenate transcriptions
    │
    ▼
Fix overlapping text (GPT-4o removes duplicated words)
    │
    ▼
Final transcript

Retry: 3 attempts via tenacity library.

3.3 Where It's Connected

Feature	Connection
Speaking Grading	Student audio → Whisper → transcript → GPT grading
Audio Transcription	Uploaded audio → Whisper → listening script for exam editor

---

4. AWS Polly — Text-to-Speech

4.1 Configuration

Setting	Value
Engine	Neural
Output Format	MP3
Region	eu-west-1
Max Chunk Size	3,000 characters (split at sentence boundaries)

4.2 Available Voices

Voice	Language/Accent
Danielle	American English
Gregory	American English
Kevin	American English
Ruth	American English
Stephen	American English
Arthur	British English
Olivia	Australian English
Ayanda	South African English
Aria	New Zealand English
Kajal	Indian English
Niamh	Irish English

4.3 How It Works

Listening exam audio generation:

Generated Dialog/Monologue (from GPT)
    │
    ▼
Assign voices to speakers (random for monologue, per-speaker for dialog)
    │
    ▼
Split text into ≤3000 char chunks at sentence boundaries
    │
    ▼
AWS Polly Neural TTS → MP3 audio bytes
    │
    ▼
Concatenate audio segments
    │
    ▼
Upload to Firebase Storage
    │
    ▼
Return download URL

4.4 Where It's Connected

Feature	Connection
Listening Exam Audio	GPT script → Polly TTS → MP3 → Firebase Storage → student playback
Listening Instructions	"Recording has now finished" scripts → Stephen voice → MP3

---

5. FAISS + Sentence Transformers — RAG Training Tips

5.1 Configuration

Setting	Value
Embeddings Model	all-MiniLM-L6-v2 (Sentence Transformers)
Index Type	faiss.IndexFlatL2 (exact L2 search)
Top-K	5 results per query
Data Source	pathways_2_rw_with_ids.json

5.2 Knowledge Base Categories

Category	Index File
ct_focus	./faiss/ct_focus_tips_index.faiss
language_for_writing	./faiss/language_for_writing_tips_index.faiss
reading_skill	./faiss/reading_skill_tips_index.faiss
strategy	./faiss/strategy_tips_index.faiss
word_link	./faiss/word_link_tips_index.faiss
word_partners	./faiss/word_partners_tips_index.faiss
writing_skill	./faiss/writing_skill_tips_index.faiss

Metadata: ./faiss/tips_metadata.pkl (pickle file with tip IDs and text)

5.3 How RAG Works

Student Exam Performance
    │
    ▼
GPT analyzes performance → generates queries (text + category)
    │
    ▼
Sentence Transformers encodes query → embedding vector
    │
    ▼
FAISS L2 search → top 5 matching tips per category
    │
    ▼
GPT selects most relevant tips from retrieved results
    │
    ▼
Tips stored in MongoDB and displayed to student

5.4 Where It's Connected

Feature	Connection
Training Module	After exam → analyze weak areas → retrieve personalized tips → display training content
Walkthrough	Tips linked to specific reading/writing skills for guided learning

---

6. ELAI — AI Avatar Video Generation

6.1 Configuration

Setting	Value
API Endpoint	https://apis.elai.io/api/v1/videos
Auth	Bearer token (ELAI_TOKEN)
Animation	fade_in
Language	English

6.2 Available Avatars

Avatar	Gender	Voice Provider
Gia	Female	ElevenLabs
Vadim	Male	Azure
Orhan	Male	ElevenLabs
Flora	Female	Azure
Scarlett	Female	ElevenLabs
Parker	Male	Azure
Ethan	Male	ElevenLabs

Each avatar has a unique avatar_code, avatar_url, avatar_canvas dimensions, voice_id, and voice_provider.

6.3 How It Works

Speaking Task Text (from GPT)
    │
    ▼
Select Avatar (user choice from available list)
    │
    ▼
Build video config (slide, avatar, canvas, logo, voice settings)
    │
    ▼
POST to ELAI API → create video
    │
    ▼
POST render request → start processing
    │
    ▼
Poll GET status every 10 seconds until "ready"
    │
    ▼
Return video URL for playback

6.4 Where It's Connected

Feature	Connection
Speaking Exam	AI avatar presents speaking questions via video
Exam Editor	Teachers generate speaking videos while creating exams

---

7. GPTZero — AI Writing Detection

7.1 Configuration

Setting	Value
API Endpoint	https://api.gptzero.me/v2/predict/text
Auth	x-api-key header
Multilingual	false

7.2 How It Works

Student's Writing Submission
    │
    ▼
POST to GPTZero API with document text
    │
    ▼
Response: predicted_class, confidence, per-sentence AI probability
    │
    ▼
Returned as part of writing evaluation
    │
    ▼
Frontend displays AI Detection component

Response fields:

• predicted_class: ai, mixed, or human
• confidence_category: confidence level
• class_probabilities: probability distribution
• sentences: per-sentence analysis with highlight_sentence_for_ai flag

7.3 Where It's Connected

Feature	Connection
Writing Grading	Runs in parallel with GPT evaluation, perfect answer, and spelling fix
Frontend UI	AIDetection component displays results with radial progress, segmented bars, and highlighted AI-generated sentences

---

8. End-to-End Data Flows

8.1 Exam Generation Flow

Teacher clicks "Generate" in Exam Editor
    │
    ▼
Frontend: POST /api/exam/generate/{module}/{sectionId}
    │
    ▼
Next.js API Route: proxies to BACKEND_URL/{module}/...
    │
    ▼
FastAPI Controller → Service
    │
    ├── Reading: GPT-4o generates passage + exercises
    ├── Listening: GPT-4o generates script → Polly TTS → MP3 → Firebase
    ├── Writing: GPT-4o/3.5 generates task prompt
    └── Speaking: GPT-4o generates questions → ELAI creates avatar video
    │
    ▼
Response with generated content → stored in exam editor state

8.2 Writing Grading Flow

Student submits writing answer
    │
    ▼
Frontend: POST /api/evaluate/writing
    │
    ▼
Next.js API: inserts pending evaluation in MongoDB → proxies to backend
    │
    ▼
FastAPI runs 4 tasks IN PARALLEL:
    ├── GPT-4o: Grade against IELTS band criteria (temp 0.1)
    ├── GPT-4o: Generate perfect answer for comparison
    ├── GPT-3.5: Fix spelling/transcription errors (temp 0.2)
    └── GPTZero: Detect AI-generated content
    │
    ▼
Combined result stored in MongoDB evaluation collection
    │
    ▼
Frontend polls /api/evaluate/status until complete
    │
    ▼
UI shows: band scores, detailed comments, perfect answer, AI detection

8.3 Speaking Grading Flow

Student records audio response
    │
    ▼
Frontend: POST /api/evaluate/speaking (FormData with audio)
    │
    ▼
Next.js API: inserts pending evaluation → proxies to backend
    │
    ▼
FastAPI pipeline:
    │
    ▼
Whisper (local): Transcribe audio → text
    │
    ▼
GPT-4o: Grade transcript against IELTS speaking criteria (temp 0.1)
    │
    ▼
GPT-4o/3.5: Generate perfect answer
    │
    ▼
Result stored in MongoDB
    │
    ▼
Frontend polls and displays scores + transcript + perfect answer

8.4 Training / Personalized Tips Flow

Student completes an exam
    │
    ▼
Frontend: POST /api/training
    │
    ▼
Backend analyzes exam performance with GPT
    │
    ▼
GPT generates search queries + categories
    │
    ▼
Sentence Transformers encodes queries → FAISS L2 search
    │
    ▼
Top 5 tips per category retrieved
    │
    ▼
GPT selects most relevant tips
    │
    ▼
Training content stored in MongoDB → displayed to student

---

9. Frontend AI Integration Points

9.1 API Routes (Next.js → FastAPI)

Frontend Route	Backend Route	AI Services Used
POST /api/evaluate/writing	BACKEND_URL/grade/writing/{task}	GPT-4o, GPT-3.5, GPTZero
POST /api/evaluate/speaking	BACKEND_URL/grade/speaking/{task}	Whisper, GPT-4o, GPT-3.5
GET /api/exam/generate/reading/{id}	BACKEND_URL/reading/{passage}	GPT-4o
GET /api/exam/generate/listening/{id}	BACKEND_URL/listening/{section}	GPT-4o
GET /api/exam/generate/writing/{id}	BACKEND_URL/writing/{task}	GPT-4o / GPT-3.5
GET /api/exam/generate/speaking/{id}	BACKEND_URL/speaking/{task}	GPT-4o
POST /api/exam/media/listening	BACKEND_URL/listening/media	AWS Polly
POST /api/exam/media/speaking	BACKEND_URL/speaking/media	ELAI
POST /api/transcribe	BACKEND_URL/listening/transcribe	Whisper
POST /api/training	BACKEND_URL/training/	GPT, FAISS, Sentence Transformers
GET /api/exam/avatars	BACKEND_URL/speaking/avatars	ELAI

9.2 Key UI Components

Component	Purpose
AIDetection.tsx	Displays AI detection results (radial progress, highlighted sentences)
GenerateBtn.tsx	Brain icon button with spinner for content generation
generateVideos.ts	Manages ELAI video creation + polling loop
ExamPage.tsx	Triggers writing/speaking evaluation + polls for results
useEvaluationPolling.tsx	Hook that polls evaluation status until grading completes
generation.tsx	Page for generating exams (gated by permissions)

9.3 Permissions

Permission	Controls
generate_reading	Reading passage generation
generate_listening	Listening script generation
generate_writing	Writing prompt generation
generate_speaking	Speaking task generation
generate_level	Level test generation

---

10. Environment Variables & Configuration

10.1 Required API Keys

Variable	Service	Where Used
OPENAI_API_KEY	OpenAI GPT-4o / GPT-3.5	Content generation, grading, evaluation
AWS_ACCESS_KEY_ID	AWS Polly	Text-to-speech
AWS_SECRET_ACCESS_KEY	AWS Polly	Text-to-speech
ELAI_TOKEN	ELAI	Avatar video generation
GPT_ZERO_API_KEY	GPTZero	AI writing detection

10.2 Backend Service Wiring (Dependency Injection)

DI Container
├── llm           → OpenAI(client=AsyncOpenAI)
├── tts           → AWSPolly(client=polly_client)
├── stt           → OpenAIWhisper(model="base", num_models=4)
├── vid_gen       → ELAI(client=http_client, token, avatars, conf)
├── ai_detector   → GPTZero(client=http_client, key)
├── training_kb   → TrainingContentKnowledgeBase(embeddings=SentenceTransformer)
│
├── Controllers
│   ├── ReadingController    → uses llm
│   ├── ListeningController  → uses llm, tts
│   ├── WritingController    → uses llm, ai_detector
│   ├── SpeakingController   → uses llm, stt, vid_gen
│   ├── GradeController      → uses llm, stt, ai_detector
│   ├── LevelController      → uses llm
│   └── TrainingController   → uses llm, training_kb

10.3 Cost Drivers

Service	Cost Model	Usage Pattern
OpenAI GPT-4o	Per token (input + output)	Every generation, every grading — highest cost
OpenAI GPT-3.5	Per token (cheaper)	Summaries, spelling, some writing tasks
AWS Polly	Per character (Neural)	Every listening exam audio
ELAI	Per video minute	Every speaking exam video
GPTZero	Per API call	Every writing grading
Whisper (local)	Compute only (no API cost)	Every speaking grading + transcription
FAISS (local)	Compute only (no API cost)	Every training session