AI Context & Architecture Guide for Gemini

This document provides context for AI coding assistants to understand the architecture, design patterns, and workflows of the trans-summary project.

1. Architecture Overview

The project follows a Core-Wrapper pattern to separate business logic from user interfaces.

• Core Logic (pipeline.py): Contains the implementation of all major processing steps (formatting, summarizing, generating). It handles API interactions, file I/O, and logic. It is UI-agnostic.
• Shared Utilities (transcript_utils.py): Provides robust primitives for logging, API calls (with retry/validation), text normalization, and regex extraction.
• Configuration (config.py): Central source of truth for file paths, model selection, and constants.
• Interfaces:
  • GUI (ts_gui.py): Tkinter-based interface that calls pipeline.py functions in threads.
  • CLI Wrappers (transcript_*.py): Simple scripts that parse arguments and call pipeline.py functions.

2. Data Pipeline Flow

Files move through specific states, denoted by suffixes and directories:

1. Ingestion: source/*.txt (Raw text)
   • Action: format_transcript
2. Formatting: formatted/* - formatted.md (Clean Markdown)
   • Action: add_yaml
3. Metadata: formatted/* - yaml.md (Formatted + Frontmatter)
   • Action: summarize_transcript
4. Extraction: projects/* - structural-themes.md, interpretive-themes.md, topics.md, key-terms.md, lenses-ranked.md, bowen-references.md, emphasis-items.md
5. Synthesis: projects/* - summary-generated.md, abstract-generated.md
6. Publication: projects/*.html, projects/*.pdf

3. Key Design Patterns

API Interaction

• Centralized Calls: All Claude API calls go through transcript_utils.call_claude_with_retry.
• 7-Level Validation: Responses are rigorously validated (message type, role, stop reason, content existence, text validity, token usage) to prevent silent failures.
• Truncation Safety: The system automatically detects and rejects responses truncated by token limits.
• Streaming: Long generations use streaming to prevent timeouts.

Text Processing

• Regex over JSON: The pipeline prefers robust Regex extraction (see transcript_utils.extract_section) over strict JSON parsing for large text blocks, as it is more forgiving of LLM formatting minor errors.
• Fuzzy Matching: Validation scripts (transcript_validate_*.py) use difflib.SequenceMatcher to verify that generated quotes exist in the source text, handling minor punctuation/whitespace differences.

Prompt Management

• External Files: Prompts are stored as Markdown files in prompts/.
• Template Injection: pipeline._fill_prompt_template handles Jinja-like variable replacement (e.g., {{transcript}}).

4. Critical Files Map

File	Responsibility
pipeline.py	The Brain. Orchestrates all logic. If adding a feature, start here.
config.py	The Settings. Paths, models, constants.
transcript_utils.py	The Toolbox. Logging, API wrappers, text helpers.
html_generator.py	The Publisher. Logic for HTML/PDF creation.
summary_validation.py	The Critic. Deep semantic validation for summaries.
abstract_validation.py	The Critic. Deep semantic validation for abstracts.
ts_gui.py	The Face. Tkinter GUI implementation.

5. Validation Strategy

The project emphasizes correctness. Every generation step has a corresponding validation step:

• Format: validate_format checks word preservation.
• Headers: validate_headers checks header relevance via LLM.
• Abstract/Summary: validate_coverage checks if required topics are present.
• Quotes: validate_emphasis_items ensures quotes are not hallucinations.

6. Common Tasks for AI Agents

• Adding a new pipeline step:

  1. Define prompt in prompts/.
  2. Add logic to pipeline.py.
  3. Add CLI wrapper transcript_new_step.py.
  4. Update ts_gui.py to include a button.

• Updating Prompts:

  • Modify files in prompts/.
  • Ensure placeholders (e.g., {{transcript}}) match what pipeline.py provides.

• Changing Models:

  • Update DEFAULT_MODEL or AUX_MODEL in config.py.

• Testing & QA:

  • Always run pytest tests after making logic changes.
  • Run ruff check . to ensure code style compliance.
  • Add new tests in tests/ if introducing new logic.

7. Workflows

• Request Logging:
  • Automatically track user requests and task completion status in REQUEST_LOG.md.
  • Update this log at the start and end of significant tasks.