Pipeline Orchestration

# Pipeline Orchestration

This document defines the **final, locked production pipeline**. The flow below is the only approved sequence for transforming a creative brief into publishable media, commerce assets, and marketing campaigns. Any enhancement must plug into one of these stages and preserve the human approval checkpoint and Atlas-only distribution posture.

## End-to-End Flow

```mermaid
flowchart LR
    Brief["Creative Brief"] --> Intake["Intake & Project Setup"]
    Intake --> Planning["Sage + Director Planning"]
    Planning --> Visuals["Visuals (Character DNA)"]
    Visuals --> Shots["Cinematographer & DP Shotlist"]
    Shots --> Images["Ideogram + Flux Illustration"]
    Images --> Approvals{{"Human Approval Gate"}}
    Approvals --> Magnific["Atlas-Managed Magnific Upscale"]
    Magnific --> Kling["Atlas-Managed Kling Renders"]
    Kling --> Composer["Dynamic Layout + Purchase Psychology"]
    Composer --> Commerce["Commerce Sheet (Approval Gate)"]
    Commerce --> Shopify["Shopify & Distribution (Atlas)"]
    Shopify --> Marketing["Atlas Marketing & Social Ops"]
```

## Neuro-Aesthetic & Commerce Enhancements

- **CharacterConsistencyEngine** extends the Visuals stage with studio DNA prompts for Ideogram + Flux while preserving the existing marketing writer agent for copy.
- **DynamicBookComposer** runs immediately after the human approval gate, deriving saliency-aware layout zones that respect the locked KDP specs before the commerce branch executes.
- **PurchasePsychologyOptimizer** turns the approved illustration stack into conversion-ready frames consumed by marketing nodes and Atlas promo automation.
- **StyleFingerprintEngine** + **OmniChannelPublisher** attach a reusable brand fingerprint and omni-channel preview manifests to the product-sheet approval gate so Shopify/Atlas flows remain human-controlled.
- **Tooling remains identical**: imagery via Ideogram + Flux, Kling via API with Atlas fallback, Magnific via Atlas adapters, and Atlas continues to be the only browser automation surface.

### Immutable Rules
- **Single orchestrator** – `packages/forge/pipeline_orchestrator.py:create_orchestrator()` is the only entry point.
- **Approvals required** – Assets cannot be upscaled, rendered, or published until approvals are logged in state.
- **Atlas everywhere** – All publishing, commerce, and marketing calls must flow through OpenAI Atlas connectors.

## Stage Detail

### 1. Intake & Project Setup
- **Entry point**: `orchestrator.run(initial_state)` or `python -m examples.run_full_pipeline_demo`.
- **Key tasks**: normalize briefs, allocate `project_id`, seed logs, capture governance requirements.
- **Outputs**: base state with `initial_request`, `project_manifest`, and audit trail entries.

### 2. Planning & Styling
- **Agents**: `SageAgent`, `DirectorAgent`.
- **Responsibilities**: produce `story_analysis`, emotional arcs, and a style bible. All artifacts attach to state for downstream tooling.
- **Notes**: Planning runs deterministically so repeated executions keep the same scene slugs.

### 3. Visual Development
- **Agents**: `VisualiserAgent`, `CinematographerAgent`, `dp.run`.
- **Responsibilities**: translate story analysis into prompt matrices, camera plans, and a canonical `shot_list`.
- **Deliverables**: deterministic scene slugs (e.g., `scene-001`) used throughout approvals and storage.

### 4. Asset Generation & Human Review
- **Tools**: Ideogram integration via `packages/app/tools`.
- **Flow**: generate image candidates, push them for human review, capture selections in `data/<project_id>/approvals.json`.
- **Gate**: the orchestrator calls `require_approval` and halts until every scene has an approved asset.

### 5. Enhancement & Assembly
- **Tools**: `MagnificBrowser`, `assemble_full_story`, `compose_book`.
- **Responsibilities**: upscale approved art, render Kling shots (with Atlas fallback retrieval), stitch master videos and promos, compose EPUB and print-ready PDFs.
- **Outputs**: `images_upscaled/`, `video/master/`, `ebook/`, and associated manifests.

### 6. Gate B, Publishing & Commerce Packaging
- **Agents**: `PublishingAgent`, `PipelineOrchestrator.approval_gate_node`, `DistributionAgent`.
- **Atlas connectors**: KDP, Apple Books, Google Play Books, Shopify.
- **Tasks**: package deliverables, generate `state/approval_sheet.json` for the Interactive Design Sheet UI, accept decisions via `/api/approvals/submit`, regenerate Shopify manifests/CSV, and optionally stage drafts with `scripts/shopify_stage_listings.py`.
- **Rule**: Distribution stays paused (`distribution_requires_approval=True`) until every required candidate is approved. Change requests flow into `state/change_requests.json` and trigger engine regeneration.

### 7. Marketing & Analytics
- **Driver**: `packages/app/ops/marketing_blitz.py` with Atlas marketing connectors.
- **Scope**: launch ad campaigns, schedule social posts, trigger email flows, and sync KPI dashboards.
- **Outputs**: marketing artifacts in `final_output/<project_id>/marketing/` and Atlas campaign telemetry written back to state.

## Operational Commands

| Command | Purpose |
| --- | --- |
| `python -m examples.run_full_pipeline_demo` | Execute the full pipeline with sample inputs.
| `python pipeline.py plan --project <id>` | Inspect planning output and approval status.
| `python pipeline.py approve --project <id>` | Record approvals from the terminal (writes `approvals.json`).
| `python pipeline.py run-publish --project <id>` | Re-run only the publishing branch once assets are approved.
| `python pipeline.py run-video --project <id>` | Re-render video branch using cached approvals and upscales.
| `python pipeline.py approve-and-upload --project <id>` | Trigger Atlas uploads after manual verification.

## Artifacts & Manifests

- `data/<project_id>/approvals.json` – Canonical record of human-approved assets.
- `data/<project_id>/images/` – Raw Ideogram outputs.
- `data/<project_id>/images_upscaled/` – Magnific outputs managed by Atlas.
- `data/<project_id>/video/` – Kling renders and assembled promos.
- `final_output/<project_id>/ebook/` – EPUB, print PDF, cover.
- `state/approval_sheet.json` – Latest Gate B candidate sheet consumed by the approvals UI.
- `state/approval_decisions.json` – Persisted approval/needs changes/reject decisions.
- `state/change_requests.json` – Outstanding regeneration tasks created by Gate B.
- `state/spec_reports/*.json` – Fulfilment template compliance reports for reviewer download.
- `manifests/shopify_listing.jsonl` – Shopify manifest regenerated after approvals.
- `exports/shopify_products.csv` – Shopify CSV export aligned with the manifest.
- `final_output/<project_id>/marketing/` – Campaign copy, schedules, and KPI exports.

## Change Management

1. File an issue or ADR describing the enhancement and the stage it extends.
2. Update the Mermaid diagram here and in `README.md` to show the new sub-step.
3. Prove compliance with human approval and Atlas routing in tests (`tests/test_pipeline_orchestrator.py`).
4. Update this document, `README.md`, and `docs/ARCHITECTURE.md` **before** merging code.

Enhancements that attempt to fork the pipeline, bypass approvals, or perform direct (non-Atlas) uploads will be rejected.