# Pipeline Orchestration This document defines the **final, locked production pipeline**. The flow below is the only approved sequence for transforming a creative brief into publishable media, commerce assets, and marketing campaigns. Any enhancement must plug into one of these stages and preserve the human approval checkpoint and Atlas-only distribution posture. ## End-to-End Flow ```mermaid flowchart LR Brief["Creative Brief"] --> Intake["Intake & Project Setup"] Intake --> Planning["Sage + Director Planning"] Planning --> Visuals["Visuals (Character DNA)"] Visuals --> Shots["Cinematographer & DP Shotlist"] Shots --> Images["Ideogram + Flux Illustration"] Images --> Approvals{{"Human Approval Gate"}} Approvals --> Magnific["Atlas-Managed Magnific Upscale"] Magnific --> Kling["Atlas-Managed Kling Renders"] Kling --> Composer["Dynamic Layout + Purchase Psychology"] Composer --> Commerce["Commerce Sheet (Approval Gate)"] Commerce --> Shopify["Shopify & Distribution (Atlas)"] Shopify --> Marketing["Atlas Marketing & Social Ops"] ``` ## Neuro-Aesthetic & Commerce Enhancements - **CharacterConsistencyEngine** extends the Visuals stage with studio DNA prompts for Ideogram + Flux while preserving the existing marketing writer agent for copy. - **DynamicBookComposer** runs immediately after the human approval gate, deriving saliency-aware layout zones that respect the locked KDP specs before the commerce branch executes. - **PurchasePsychologyOptimizer** turns the approved illustration stack into conversion-ready frames consumed by marketing nodes and Atlas promo automation. - **StyleFingerprintEngine** + **OmniChannelPublisher** attach a reusable brand fingerprint and omni-channel preview manifests to the product-sheet approval gate so Shopify/Atlas flows remain human-controlled. - **Tooling remains identical**: imagery via Ideogram + Flux, Kling via API with Atlas fallback, Magnific via Atlas adapters, and Atlas continues to be the only browser automation surface. ### Immutable Rules - **Single orchestrator** – `packages/forge/pipeline_orchestrator.py:create_orchestrator()` is the only entry point. - **Approvals required** – Assets cannot be upscaled, rendered, or published until approvals are logged in state. - **Atlas everywhere** – All publishing, commerce, and marketing calls must flow through OpenAI Atlas connectors. ## Stage Detail ### 1. Intake & Project Setup - **Entry point**: `orchestrator.run(initial_state)` or `python -m examples.run_full_pipeline_demo`. - **Key tasks**: normalize briefs, allocate `project_id`, seed logs, capture governance requirements. - **Outputs**: base state with `initial_request`, `project_manifest`, and audit trail entries. ### 2. Planning & Styling - **Agents**: `SageAgent`, `DirectorAgent`. - **Responsibilities**: produce `story_analysis`, emotional arcs, and a style bible. All artifacts attach to state for downstream tooling. - **Notes**: Planning runs deterministically so repeated executions keep the same scene slugs. ### 3. Visual Development - **Agents**: `VisualiserAgent`, `CinematographerAgent`, `dp.run`. - **Responsibilities**: translate story analysis into prompt matrices, camera plans, and a canonical `shot_list`. - **Deliverables**: deterministic scene slugs (e.g., `scene-001`) used throughout approvals and storage. ### 4. Asset Generation & Human Review - **Tools**: Ideogram integration via `packages/app/tools`. - **Flow**: generate image candidates, push them for human review, capture selections in `data//approvals.json`. - **Gate**: the orchestrator calls `require_approval` and halts until every scene has an approved asset. ### 5. Enhancement & Assembly - **Tools**: `MagnificBrowser`, `assemble_full_story`, `compose_book`. - **Responsibilities**: upscale approved art, render Kling shots (with Atlas fallback retrieval), stitch master videos and promos, compose EPUB and print-ready PDFs. - **Outputs**: `images_upscaled/`, `video/master/`, `ebook/`, and associated manifests. ### 6. Gate B, Publishing & Commerce Packaging - **Agents**: `PublishingAgent`, `PipelineOrchestrator.approval_gate_node`, `DistributionAgent`. - **Atlas connectors**: KDP, Apple Books, Google Play Books, Shopify. - **Tasks**: package deliverables, generate `state/approval_sheet.json` for the Interactive Design Sheet UI, accept decisions via `/api/approvals/submit`, regenerate Shopify manifests/CSV, and optionally stage drafts with `scripts/shopify_stage_listings.py`. - **Rule**: Distribution stays paused (`distribution_requires_approval=True`) until every required candidate is approved. Change requests flow into `state/change_requests.json` and trigger engine regeneration. ### 7. Marketing & Analytics - **Driver**: `packages/app/ops/marketing_blitz.py` with Atlas marketing connectors. - **Scope**: launch ad campaigns, schedule social posts, trigger email flows, and sync KPI dashboards. - **Outputs**: marketing artifacts in `final_output//marketing/` and Atlas campaign telemetry written back to state. ## Operational Commands | Command | Purpose | | --- | --- | | `python -m examples.run_full_pipeline_demo` | Execute the full pipeline with sample inputs. | `python pipeline.py plan --project ` | Inspect planning output and approval status. | `python pipeline.py approve --project ` | Record approvals from the terminal (writes `approvals.json`). | `python pipeline.py run-publish --project ` | Re-run only the publishing branch once assets are approved. | `python pipeline.py run-video --project ` | Re-render video branch using cached approvals and upscales. | `python pipeline.py approve-and-upload --project ` | Trigger Atlas uploads after manual verification. ## Artifacts & Manifests - `data//approvals.json` – Canonical record of human-approved assets. - `data//images/` – Raw Ideogram outputs. - `data//images_upscaled/` – Magnific outputs managed by Atlas. - `data//video/` – Kling renders and assembled promos. - `final_output//ebook/` – EPUB, print PDF, cover. - `state/approval_sheet.json` – Latest Gate B candidate sheet consumed by the approvals UI. - `state/approval_decisions.json` – Persisted approval/needs changes/reject decisions. - `state/change_requests.json` – Outstanding regeneration tasks created by Gate B. - `state/spec_reports/*.json` – Fulfilment template compliance reports for reviewer download. - `manifests/shopify_listing.jsonl` – Shopify manifest regenerated after approvals. - `exports/shopify_products.csv` – Shopify CSV export aligned with the manifest. - `final_output//marketing/` – Campaign copy, schedules, and KPI exports. ## Change Management 1. File an issue or ADR describing the enhancement and the stage it extends. 2. Update the Mermaid diagram here and in `README.md` to show the new sub-step. 3. Prove compliance with human approval and Atlas routing in tests (`tests/test_pipeline_orchestrator.py`). 4. Update this document, `README.md`, and `docs/ARCHITECTURE.md` **before** merging code. Enhancements that attempt to fork the pipeline, bypass approvals, or perform direct (non-Atlas) uploads will be rejected.

Process Optimization

🎯 COMPLETE FEATURE LIST & IMPROVEMENTS

Pipeline Orchestration