Presentation Evaluator — Prototype Plan

Overview

A post-hoc analysis system that evaluates academic student presentations (~10 min) from recorded video. The system processes a single front-facing camera recording, extracts speech and body language metrics, and produces a descriptive report benchmarked against TED talk norms.

Target users: Students and instructors in academic settings Goal: Provide actionable, descriptive feedback on presentation delivery (not slide content)

---

System Architecture

Recorded Video (MP4/MOV, ~10 min, front-facing camera)
    │
    ├──► ffmpeg ──► Audio Track (WAV)
    │                   │
    │                   ├──► Whisper (faster-whisper)
    │                   │       └── Transcript + word timestamps + confidence
    │                   │               │
    │                   │               ├──► Speech Metrics (derived)
    │                   │               │       ├── WPM (rolling + overall)
    │                   │               │       ├── Pause count/duration
    │                   │               │       ├── Filler words (um, uh, like, you know)
    │                   │               │       └── Hedge phrases (sort of, I think, maybe)
    │                   │               │
    │                   │               └──► Language Metrics (spaCy + textstat)
    │                   │                       ├── Vocabulary complexity (Flesch-Kincaid)
    │                   │                       ├── Sentence length variety
    │                   │                       ├── Word repetition frequency
    │                   │                       └── LLM content evaluation (Ollama)
    │                   │
    │                   └──► Prosody Analysis (parselmouth + librosa)
    │                           ├── Pitch (F0) mean, range, variance
    │                           ├── Pitch contour over time
    │                           ├── Volume (RMS energy) dynamics
    │                           ├── Volume variance / dynamic range
    │                           └── Monotone detection (low pitch variance segments)
    │
    └──► Video Frames (OpenCV)
            │
            └──► MediaPipe (per-frame)
                    ├── Pose Estimation
                    │       ├── Upper body movement magnitude
                    │       ├── Posture openness (shoulder width vs arm position)
                    │       ├── Fidgeting score (high-frequency small movements)
                    │       ├── Sway / weight shifting
                    │       └── Stillness vs. purposeful movement
                    │
                    ├── Hand Tracking
                    │       ├── Gesture frequency (hands moving > threshold)
                    │       ├── Gesture amplitude (range of hand motion)
                    │       ├── Hands visible vs. hidden (pockets, behind back)
                    │       └── Gesture variety (spatial distribution)
                    │
                    └── Face Mesh (468 landmarks + iris)
                            ├── Gaze direction (iris position relative to eye corners)
                            ├── Eye contact with camera (gaze within threshold cone)
                            ├── Eye contact duration / frequency
                            ├── Facial expressiveness (landmark variance over time)
                            └── Smile detection (mouth landmark geometry)

    All metrics ──► Time-Aligned DataFrame (sentence-level segments)
                        │
                        ├──► JSON output (structured metrics)
                        └──► Text report (Ollama-generated summary)

---

TED Talk Benchmarks

These are established norms from public speaking research and TED talk analyses. The system reports where the student falls relative to these ranges.

Metric	TED Talk Norm / Recommended Range	Source
Speaking rate	130–170 WPM (ideal ~150)	Various public speaking research
Pause frequency	1–2 pauses per sentence, 0.5–1.5s each	Rhetorical pause studies
Filler word rate	< 1 per minute (top speakers)	TED talk corpus analyses
Pitch variation	F0 std dev > 20–30 Hz (varies by gender)	Prosody research
Volume dynamic range	> 6 dB variation	Vocal coaching literature
Eye contact	> 60–70% of time looking at audience/camera	Presentation coaching
Gesture rate	1–3 purposeful gestures per sentence	TED talk gesture studies
Movement	Moderate (not frozen, not pacing)	Presentation coaching

Phase 1 approach: Hard-code these as configurable reference ranges. Later, we could build a small TED talk analysis pipeline to derive empirical benchmarks from actual TED recordings.

---

Pipeline Modules

Module 1: Ingestion (ingest.py)

• Accept video file path (MP4, MOV, WEBM)
• Extract audio track via ffmpeg → WAV (16kHz mono)
• Extract video frames via OpenCV at configurable FPS (e.g., 5 fps for body language — no need for 30fps)
• Return paths to audio file and frame directory

Module 2: Speech Analysis (speech.py)

• Transcription: faster-whisper (large-v3 model) → segments with word-level timestamps and confidence
• Speaking rate: Calculate WPM per sentence and rolling 30-second windows
• Pauses: Detect gaps between words/segments > 0.3s; classify as short (0.3–1s), medium (1–2s), long (2s+)
• Filler words: Pattern match against configurable list from transcript
• Hedge phrases: Multi-word pattern matching
• Clarity proxy: Average Whisper confidence per sentence (low confidence ≈ unclear speech)
• Output: DataFrame with one row per sentence, columns for all metrics

Module 3: Prosody Analysis (prosody.py)

• Pitch extraction: parselmouth Praat pitch tracking → F0 contour
• Pitch metrics: Mean F0, F0 range, F0 standard deviation (overall and per-sentence)
• Volume: librosa RMS energy → volume contour
• Volume metrics: Mean dB, dynamic range, volume variance per sentence
• Monotone segments: Flag sentences where F0 std dev < threshold
• Output: DataFrame aligned to speech segments by timestamp

Module 4: Body Language Analysis (body_language.py)

• Process extracted frames through MediaPipe (Pose + Hands + Face Mesh)
• Per-frame extraction:
  • 33 pose landmarks (x, y, z, visibility)
  • 21 hand landmarks per hand (when visible)
  • 468 face landmarks + iris positions
• Derived metrics (computed over sentence-aligned windows):
  • Movement magnitude: Mean velocity of shoulder/hip keypoints
  • Fidgeting score: High-frequency oscillation in hand/shoulder keypoints
  • Posture openness: Angle between elbows and torso midline
  • Gesture frequency: Count of hand displacement events > threshold
  • Gesture amplitude: Mean displacement magnitude during gestures
  • Hand visibility: Percentage of frames where hands are detected
  • Gaze direction: Iris center position relative to eye corner landmarks → angle
  • Eye contact ratio: Percentage of frames where gaze is within N degrees of camera
  • Facial expressiveness: Variance of mouth/eyebrow landmarks over time
• Output: DataFrame with one row per sentence-window, columns for all metrics

Module 5: Language Analysis (language.py)

• Input: Transcript from Module 2
• Readability: textstat Flesch-Kincaid grade level, Gunning Fog, etc.
• Vocabulary diversity: Type-token ratio (TTR), hapax legomena ratio
• Repetition: Most repeated content words, n-gram frequency
• Sentence structure: spaCy dependency parse depth, clause count variety
• Output: Document-level and per-sentence metrics

Module 6: LLM Evaluation (llm_eval.py)

• Input: Full transcript + summary metrics from other modules
• Model: Ollama (local) — suggest llama3.1:8b or mistral depending on what's available
• Prompt design: Structured prompt asking for:
  • Content organization assessment (intro, body, conclusion structure)
  • Clarity of explanation
  • Audience engagement indicators from language
  • Specific actionable suggestions
• Output: Structured JSON with scores and text feedback per category

Module 7: Report Generation (report.py)

• Input: All DataFrames from modules 2–6
• Merge: Time-align all metrics into a master DataFrame
• Benchmark comparison: For each metric, compute percentile relative to TED talk norms
• JSON export: Full structured output with all raw + derived metrics
• Text report: Pass metrics to Ollama with a report-generation prompt
  • Overall summary
  • Strengths (metrics in good range)
  • Areas for improvement (metrics outside range)
  • Specific moments to review (timestamps of notable events)

Module 8: Pipeline Orchestrator (pipeline.py)

• CLI entry point: python pipeline.py --input video.mp4 --output results/
• Runs all modules in sequence (audio extraction → parallel speech/prosody/video → language → LLM → report)
• Logging throughout with logging module
• Progress tracking with tqdm
• Saves intermediate results for debugging

---

Dependencies

# Core
opencv-python          # Video frame extraction
mediapipe              # Pose, hands, face mesh
faster-whisper         # Speech-to-text (runs on CPU/MPS)
parselmouth            # Praat pitch analysis
librosa                # Audio feature extraction
soundfile              # Audio I/O (librosa dependency)

# NLP
spacy                  # Linguistic analysis
textstat               # Readability metrics

# LLM
ollama                 # Local LLM client

# Data & Utilities
pandas                 # Data alignment and export
numpy                  # Numerical operations
tqdm                   # Progress bars
click                  # CLI interface

# System
ffmpeg-python          # Audio extraction (requires ffmpeg installed)

System requirements:

• Python 3.11+
• ffmpeg installed via Homebrew (brew install ffmpeg)
• Ollama installed and running locally
• spaCy English model: python -m spacy download en_core_web_sm
• ~4GB disk for Whisper large-v3 model (downloaded on first run)

---

Project Structure

presentation-evaluator/
├── PLAN.md
├── README.md
├── requirements.txt
├── Makefile
├── .gitignore
├── .env.example
│
├── src/
│   ├── __init__.py
│   ├── pipeline.py          # Orchestrator + CLI
│   ├── ingest.py            # Video/audio extraction
│   ├── speech.py            # Whisper transcription + speech metrics
│   ├── prosody.py           # Pitch and volume analysis
│   ├── body_language.py     # MediaPipe pose/hands/face analysis
│   ├── language.py          # NLP content metrics
│   ├── llm_eval.py          # Ollama-based evaluation
│   ├── report.py            # Report generation
│   ├── benchmarks.py        # TED talk reference ranges
│   └── utils.py             # Shared helpers (time alignment, etc.)
│
├── config/
│   └── default.yaml         # Thresholds, model settings, benchmark values
│
├── tests/
│   ├── test_speech.py
│   ├── test_prosody.py
│   ├── test_body_language.py
│   └── test_pipeline.py
│
└── output/                  # Generated results (gitignored)
    ├── metrics.json
    ├── metrics.csv
    └── report.txt

---

Makefile Targets

.PHONY: setup dev run test clean

setup:                  # Create venv, install deps, download models
dev:                    # Run pipeline on a short sample clip (~1 min)
run:                    # Run full pipeline on input video
test:                   # Run test suite
clean:                  # Remove output artifacts

---

Build Phases

Phase 1: Foundation (start here)

• Project setup (venv, deps, structure)
• Ingestion module (ffmpeg audio extraction + frame extraction)
• Speech module (Whisper transcription + basic metrics: WPM, pauses, fillers)
• Prosody module (pitch + volume analysis)
• Basic pipeline orchestration + JSON output
• Milestone: Can process a video and get speech/prosody metrics as JSON

Phase 2: Body Language

• MediaPipe pose processing on extracted frames
• Movement, posture, fidgeting metrics
• Hand tracking + gesture metrics
• Face mesh + gaze/eye contact estimation
• Time-align body metrics to speech segments
• Milestone: Full multimodal metrics in JSON output

Phase 3: Language + LLM

• spaCy/textstat language analysis
• Ollama integration for content evaluation
• Benchmark comparison logic
• Report generation (structured text report via Ollama)
• Milestone: Complete pipeline producing JSON + text report

Phase 4: Polish

• CLI with click (input validation, help text, progress output)
• Config file for all thresholds/settings
• Error handling and edge cases
• Test suite
• Documentation
• Milestone: Robust, documented prototype ready for user testing

---

Key Design Decisions

1. Frame rate for video analysis: 5 fps is sufficient for body language. At 10 min that's 3,000 frames — manageable on a Mac. Configurable if needed.

2. Sentence-level alignment: Whisper provides segment boundaries. All other metrics are windowed to these boundaries for a unified time axis.

3. Descriptive, not prescriptive: The system reports "Your speaking rate was 185 WPM. TED speakers typically average 130–170 WPM." It does not say "You spoke too fast."

4. No custom model training: Everything uses pretrained models and signal processing. The LLM (Ollama) provides qualitative assessment using prompt engineering only.

5. Gaze estimation simplicity: MediaPipe iris landmarks give us a rough gaze vector. For a single front-facing camera, we define "eye contact" as gaze directed within a configurable cone toward the camera. This is approximate but useful.

6. Whisper model size: Start with large-v3 for accuracy. Can drop to medium or small if processing time is an issue on Mac.