Pipeline Orchestration

This document defines the final, locked production pipeline. The flow below is the only approved sequence for transforming a creative brief into publishable media, commerce assets, and marketing campaigns. Any enhancement must plug into one of these stages and preserve the human approval checkpoint and Atlas-only distribution posture.

End-to-End Flow

flowchart LR
    Brief["Creative Brief"] --> Intake["Intake & Project Setup"]
    Intake --> Planning["Sage + Director Planning"]
    Planning --> Visuals["Visuals (Character DNA)"]
    Visuals --> Shots["Cinematographer & DP Shotlist"]
    Shots --> Images["Ideogram + Flux Illustration"]
    Images --> Approvals{{"Human Approval Gate"}}
    Approvals --> Magnific["Atlas-Managed Magnific Upscale"]
    Magnific --> Kling["Atlas-Managed Kling Renders"]
    Kling --> Composer["Dynamic Layout + Purchase Psychology"]
    Composer --> Commerce["Commerce Sheet (Approval Gate)"]
    Commerce --> Shopify["Shopify & Distribution (Atlas)"]
    Shopify --> Marketing["Atlas Marketing & Social Ops"]

Neuro-Aesthetic & Commerce Enhancements

• CharacterConsistencyEngine extends the Visuals stage with studio DNA prompts for Ideogram + Flux while preserving the existing marketing writer agent for copy.
• DynamicBookComposer runs immediately after the human approval gate, deriving saliency-aware layout zones that respect the locked KDP specs before the commerce branch executes.
• PurchasePsychologyOptimizer turns the approved illustration stack into conversion-ready frames consumed by marketing nodes and Atlas promo automation.
• StyleFingerprintEngine + OmniChannelPublisher attach a reusable brand fingerprint and omni-channel preview manifests to the product-sheet approval gate so Shopify/Atlas flows remain human-controlled.
• Tooling remains identical: imagery via Ideogram + Flux, Kling via API with Atlas fallback, Magnific via Atlas adapters, and Atlas continues to be the only browser automation surface.

Immutable Rules

• Single orchestrator – packages/forge/pipeline_orchestrator.py:create_orchestrator() is the only entry point.
• Approvals required – Assets cannot be upscaled, rendered, or published until approvals are logged in state.
• Atlas everywhere – All publishing, commerce, and marketing calls must flow through OpenAI Atlas connectors.

Stage Detail

1. Intake & Project Setup

• Entry point: orchestrator.run(initial_state) or python -m examples.run_full_pipeline_demo.
• Key tasks: normalize briefs, allocate project_id, seed logs, capture governance requirements.
• Outputs: base state with initial_request, project_manifest, and audit trail entries.

2. Planning & Styling

• Agents: SageAgent, DirectorAgent.
• Responsibilities: produce story_analysis, emotional arcs, and a style bible. All artifacts attach to state for downstream tooling.
• Notes: Planning runs deterministically so repeated executions keep the same scene slugs.

3. Visual Development

• Agents: VisualiserAgent, CinematographerAgent, dp.run.
• Responsibilities: translate story analysis into prompt matrices, camera plans, and a canonical shot_list.
• Deliverables: deterministic scene slugs (e.g., scene-001) used throughout approvals and storage.

4. Asset Generation & Human Review

• Tools: Ideogram integration via packages/app/tools.
• Flow: generate image candidates, push them for human review, capture selections in data/<project_id>/approvals.json.
• Gate: the orchestrator calls require_approval and halts until every scene has an approved asset.

5. Enhancement & Assembly

• Tools: MagnificBrowser, assemble_full_story, compose_book.
• Responsibilities: upscale approved art, render Kling shots (with Atlas fallback retrieval), stitch master videos and promos, compose EPUB and print-ready PDFs.
• Outputs: images_upscaled/, video/master/, ebook/, and associated manifests.

6. Gate B, Publishing & Commerce Packaging

• Agents: PublishingAgent, PipelineOrchestrator.approval_gate_node, DistributionAgent.
• Atlas connectors: KDP, Apple Books, Google Play Books, Shopify.
• Tasks: package deliverables, generate state/approval_sheet.json for the Interactive Design Sheet UI, accept decisions via /api/approvals/submit, regenerate Shopify manifests/CSV, and optionally stage drafts with scripts/shopify_stage_listings.py.
• Rule: Distribution stays paused (distribution_requires_approval=True) until every required candidate is approved. Change requests flow into state/change_requests.json and trigger engine regeneration.

7. Marketing & Analytics

• Driver: packages/app/ops/marketing_blitz.py with Atlas marketing connectors.
• Scope: launch ad campaigns, schedule social posts, trigger email flows, and sync KPI dashboards.
• Outputs: marketing artifacts in final_output/<project_id>/marketing/ and Atlas campaign telemetry written back to state.

Operational Commands

Command	Purpose
python -m examples.run_full_pipeline_demo	Execute the full pipeline with sample inputs.
python pipeline.py plan --project <id>	Inspect planning output and approval status.
python pipeline.py approve --project <id>	Record approvals from the terminal (writes approvals.json).
python pipeline.py run-publish --project <id>	Re-run only the publishing branch once assets are approved.
python pipeline.py run-video --project <id>	Re-render video branch using cached approvals and upscales.
python pipeline.py approve-and-upload --project <id>	Trigger Atlas uploads after manual verification.

Artifacts & Manifests

• data/<project_id>/approvals.json – Canonical record of human-approved assets.
• data/<project_id>/images/ – Raw Ideogram outputs.
• data/<project_id>/images_upscaled/ – Magnific outputs managed by Atlas.
• data/<project_id>/video/ – Kling renders and assembled promos.
• final_output/<project_id>/ebook/ – EPUB, print PDF, cover.
• state/approval_sheet.json – Latest Gate B candidate sheet consumed by the approvals UI.
• state/approval_decisions.json – Persisted approval/needs changes/reject decisions.
• state/change_requests.json – Outstanding regeneration tasks created by Gate B.
• state/spec_reports/*.json – Fulfilment template compliance reports for reviewer download.
• manifests/shopify_listing.jsonl – Shopify manifest regenerated after approvals.
• exports/shopify_products.csv – Shopify CSV export aligned with the manifest.
• final_output/<project_id>/marketing/ – Campaign copy, schedules, and KPI exports.

Change Management

1. File an issue or ADR describing the enhancement and the stage it extends.
2. Update the Mermaid diagram here and in README.md to show the new sub-step.
3. Prove compliance with human approval and Atlas routing in tests (tests/test_pipeline_orchestrator.py).
4. Update this document, README.md, and docs/ARCHITECTURE.md before merging code.

Enhancements that attempt to fork the pipeline, bypass approvals, or perform direct (non-Atlas) uploads will be rejected.