Neura Market ASCII Art Video Production Pipeline Reference

Overview

This reference documents the production pipeline for creating ASCII art videos from various inputs, including video, audio, images, and generative seeds. The pipeline outputs MP4, GIF, or image sequences. It covers video-to-ASCII conversion, audio-reactive visualizers, generative animations, hybrid video+audio compositions, text/lyrics overlays, and real-time terminal rendering.

Use this pipeline when a user requests ASCII video, text art video, terminal-style video, character art animation, retro text visualization, audio visualizer in ASCII, converting video to ASCII art, matrix-style effects, or any animated ASCII output.

Pipeline Architecture

The core pipeline consists of six stages executed in order:

INPUT → ANALYZE → SCENE_FN → TONEMAP → SHADE → ENCODE

INPUT: Load source material (video file, audio file, images, text/SRT, or seed parameters).
ANALYZE: Extract features (audio FFT, video luminance, or synthetic data).
SCENE_FN: Execute per-scene function that returns a canvas (uint8 H,W,3).
TONEMAP: Apply adaptive brightness normalization.
SHADE: Run post-processing through ShaderChain and FeedbackBuffer.
ENCODE: Write frames to output format using ffmpeg pipes.

Prerequisites

Python 3.10+ installed
NumPy installed
SciPy installed (for audio modes)
Pillow (PIL) installed
ffmpeg CLI installed and accessible in PATH
ElevenLabs API key (optional, for TTS narration mode)
OpenCV installed (optional, for video frame sampling and edge detection)
Access to reference files: references/architecture.md, references/composition.md, references/effects.md, references/shaders.md, references/scenes.md, references/inputs.md, references/optimization.md, references/troubleshooting.md
Input source material: video file, audio file, images, text/SRT, or seed parameters (depending on mode)
Sufficient RAM and CPU cores for parallel rendering (auto-detected for quality profile)

Capabilities

The pipeline supports the following capabilities:

Convert video to colored ASCII character video (MP4, GIF, image sequence)
Audio-reactive music visualizers driven by audio features (FFT, beats, bands)
Generative ASCII art animations from seed parameters (no input)
Hybrid video+audio reactive with audio-reactive overlays on ASCII video
Timed text/lyrics overlays with visual effects (from audio + SRT file)
TTS narration with typed text (text quotes + ElevenLabs API)
Multi-grid composition with per-layer character palettes, color strategies, background textures, primary effects, particles, shader moods, grid densities, coordinate spaces, feedback, masking, and transitions
Per-scene variation: different background effects, character palettes, color strategies, shader intensities, particle types per section
Project-specific invention: custom character palettes, background effects, color palettes, particle character sets, novel transitions
Adaptive brightness normalization via percentile-based tonemap (not linear multipliers)
Post-processing via ShaderChain and FeedbackBuffer
Parallel rendering of clips using concurrent.futures and ffmpeg pipes
Hardware detection for auto-setting quality profile
Output formats: MP4 (default), GIF (640x360 @ 15fps), PNG sequence
Resolutions: landscape 1920x1080 (default), portrait 1080x1920, square 1080x1080 @ 24fps
Creative divergence strategies: Forced Connections, Conceptual Blending, Oblique Strategies for experimental output

Procedures

ASCII Video Production Pipeline

Follow these steps in order for every project.

Step 1: Creative Vision

Before writing any code, articulate the following:

Mood/atmosphere
Visual story over duration
Color world (warm/cool/monochrome/neon/earth tones, dominant hue)
Character texture (dense/sparse/organic/geometric)
What makes THIS different
Emotional arc (how scenes progress)

Map the user prompt to aesthetic choices.

Step 2: Technical Design

Choose the mode: Video-to-ASCII, Audio-reactive, Generative, Hybrid, Lyrics/text, or TTS narration.

Set the resolution: landscape 1920x1080 (default), portrait 1080x1920, or square 1080x1080 @ 24fps.

Auto-detect cores and RAM for quality profile.

Map timestamps to scene functions, each with its own effect, palette, color, and shader configuration.

Choose the output format: MP4 (default), GIF (640x360 @ 15fps), or PNG sequence.

Step 3: Build the Script

Create a single Python file with these components:

Hardware detection + quality profile
Input loader (mode-dependent)
Feature analyzer (audio FFT, video luminance, or synthetic)
Grid + renderer (multi-density grids with bitmap cache)
Character palettes (multiple per project)
Color system (HSV + discrete RGB + harmony generation)
Scene functions (each returns canvas uint8 H,W,3)
Tonemap (adaptive brightness normalization)
Shader pipeline (ShaderChain + FeedbackBuffer)
Scene table + dispatcher (time to scene function + config)
Parallel encoder (N-worker clip rendering with ffmpeg pipes)
Main (orchestrate full pipeline)

Step 4: Quality Verification

Render single test frames at key timestamps before full render.

Check brightness: canvas.mean() > 8 for all ASCII content. If dark, lower gamma.

Check visual coherence: all scenes should feel like the same video.

Check creative vision: output must match Step 1 concept. If generic, go back.

Parameters

Mode

Meaning: Type of input and processing pipeline: Video-to-ASCII, Audio-reactive, Generative, Hybrid, Lyrics/text, TTS narration
Required: Yes

Resolution

Meaning: Output video dimensions: landscape 1920x1080 (default), portrait 1080x1920, square 1080x1080
Required: No (default 1920x1080)

Framerate

Meaning: Frames per second for output video (default 24fps for MP4, 15fps for GIF)
Required: No (default 24fps)

Output format

Meaning: MP4 (default), GIF (640x360 @ 15fps), PNG sequence
Required: No (default MP4)

Gamma

Meaning: Gamma value for tonemap: default 0.75, solarize 0.55, posterize 0.50, bright scenes 0.85
Required: No (default 0.75)

Character palette

Meaning: Set of ASCII characters used for rendering (density ramps, block elements, symbols, scripts like katakana/Greek/runes/braille, project-specific)
Required: Yes (per scene)

Color strategy

Meaning: Color system: HSV, OKLAB/OKLCH, discrete RGB palettes, auto-generated harmony, monochrome, temperature
Required: Yes (per scene)

Background texture

Meaning: Background generation method: sine fields, fBM noise, domain warp, voronoi, reaction-diffusion, cellular automata, video
Required: Yes (per scene)

Primary effects

Meaning: Main visual effect: rings, spirals, tunnel, vortex, waves, interference, aurora, fire, SDFs, strange attractors
Required: Yes (per scene)

Particles

Meaning: Particle system type: sparks, snow, rain, bubbles, runes, orbits, flocking boids, flow-field followers, trails
Required: No (per scene)

Shader mood

Meaning: Post-processing shader style: retro CRT, clean modern, glitch art, cinematic, dreamy, industrial, psychedelic
Required: No (per scene)

Grid density

Meaning: Character cell size: xs(8px) through xxl(40px), mixed per layer
Required: Yes (per scene)

Coordinate space

Meaning: Coordinate transform: Cartesian, polar, tiled, rotated, fisheye, Möbius, domain-warped
Required: No (per scene)

Feedback

Meaning: Feedback buffer effect: zoom tunnel, rainbow trails, ghostly echo, rotating mandala, color evolution
Required: No (per scene)

Masking

Meaning: Mask type: circle, ring, gradient, text stencil, animated iris/wipe/dissolve
Required: No (per scene)

Transitions

Meaning: Transition between scenes: crossfade, wipe, dissolve, glitch cut, iris, mask-based reveal
Required: No (per scene)

Seed params

Meaning: Seed parameters for generative mode (no input)
Required: Only for Generative mode

Audio file

Meaning: Input audio file for Audio-reactive, Hybrid, Lyrics/text, or TTS narration modes
Required: For Audio-reactive, Hybrid, Lyrics/text, TTS narration modes

Video file

Meaning: Input video file for Video-to-ASCII or Hybrid modes
Required: For Video-to-ASCII, Hybrid modes

Text/SRT

Meaning: Timed text file for Lyrics/text mode
Required: For Lyrics/text mode

Text quotes

Meaning: Text quotes for TTS narration mode
Required: For TTS narration mode

TTS API

Meaning: ElevenLabs API for TTS narration mode
Required: For TTS narration mode (optional)

Tonemap Function

The tonemap function performs adaptive brightness normalization using percentile-based scaling. Never use canvas * N multipliers, as they clip highlights.

def tonemap(canvas, gamma=0.75):
    f = canvas.astype(np.float32)
    lo, hi = np.percentile(f[::4, ::4], [1, 99.5])
    if hi - lo < 10: hi = lo + 10
    f = np.clip((f - lo) / (hi - lo), 0, 1) ** gamma
    return (f * 255).astype(np.uint8)

The pipeline order is: scene_fn() → tonemap() → FeedbackBuffer → ShaderChain → ffmpeg.

Per-scene gamma values: default 0.75, solarize 0.55, posterize 0.50, bright scenes 0.85.

Use screen blend (not overlay) for dark layers.

Constraints and Caveats

Brightness

Use tonemap() with percentile-based normalization. Never use canvas * N multipliers, which clip highlights.

Font Cell Height on macOS

textbbox() returns wrong height. Use font.getmetrics(): cell_height = ascent + descent.

ffmpeg Pipe Deadlock

Never use stderr=subprocess.PIPE with long-running ffmpeg. The buffer fills at 64KB and deadlocks. Redirect to a file instead.

Font Compatibility

Not all Unicode characters render in all fonts. Validate palettes at initialization by rendering each character and checking for blank output.

Per-Clip Architecture

For segmented videos (quotes, scenes, chapters), render each as a separate clip file for parallel rendering and selective re-rendering.

Creative Standard

First-render excellence is required. No revision rounds. Output must be visually striking, not generic or "AI-generated ASCII art."

Cohesive Aesthetic

All scenes must feel connected by shared color temperature, related character palettes, and consistent motion vocabulary. Never use random different effects per scene.

Dense, Layered, Considered

Every frame should reward viewing. Never use flat black backgrounds. Always use multi-grid composition. Always include per-scene variation. Always use intentional color.

Per-Section Variation

Never use the same configuration for the entire video. For each scene: use a different background effect (or compose 2-3), a different character palette, a different color strategy (or at minimum a different hue), vary shader intensity, and use different particle types if active.

Project-Specific Invention

For every project, invent at least one custom character palette, custom background effect, custom color palette, custom particle character set, or novel transition.

Performance Targets

Feature extraction: 1-5ms
Effect function: 2-15ms
Character render: 80-150ms (bottleneck)
Shader pipeline: 5-25ms
Total: ~100-200ms/frame

No GPU Required

The pipeline runs on CPU. Each project is a single self-contained Python script.

Stack

Python 3.10+, NumPy, SciPy (for audio modes), Pillow, ffmpeg CLI, concurrent.futures, ElevenLabs API (optional), OpenCV (optional).

Creative Divergence Strategies

Use Forced Connections, Conceptual Blending, or Oblique Strategies only when the user requests experimental, creative, or unique output.

Failure Modes

Dark Output

canvas.mean() <= 8. Caused by not using tonemap() or using linear multipliers that clip highlights.

Generic-Looking Output

Flat, "AI-generated ASCII art" appearance. Caused by not following the creative standard (first-render excellence, cohesive aesthetic, per-scene variation, project-specific invention).

Incoherent Video

Scenes feel disconnected. Caused by not maintaining shared color temperature, related character palettes, and consistent motion vocabulary across scenes.

ffmpeg Deadlock

Long-running ffmpeg with stderr=subprocess.PIPE fills buffer at 64KB.

Font Rendering Issues

Blank characters or wrong cell height. Caused by font incompatibility or using textbbox() on macOS instead of font.getmetrics().

Performance Bottlenecks

Character render exceeding 150ms budget. Caused by inefficient grid rendering or lack of bitmap caching.

Brightness Clipping

Using canvas * N multipliers instead of adaptive tonemap.

Missing Per-Scene Variation

Entire video uses the same configuration. Violates aesthetic standards.

No Project-Specific Invention

Output feels like a catalog selection rather than original creation.

Examples

Video-to-ASCII

Convert a video file into an ASCII recreation of the source footage.

Audio-Reactive

Generate generative visuals driven by audio features from an audio file.

Generative

Create procedural ASCII animation from seed parameters with no input.

Hybrid

Produce ASCII video from a video file with audio-reactive overlays.

Lyrics/Text

Generate timed text with visual effects from an audio file and SRT file.

TTS Narration

Create a narrated testimonial or quote video with typed text from text quotes and the ElevenLabs API.

Creative Divergence

Use Forced Connections (e.g., map erosion from geology to gradual reveal in ASCII), Conceptual Blending (e.g., ocean waves + sheet music), or Oblique Strategies (e.g., "Honor thy error as a hidden intention") for experimental output.

Reference Files

The following reference files provide detailed implementation guidance:

references/architecture.md — Pipeline architecture and component design
references/composition.md — Multi-grid composition and layering
references/effects.md — Primary effects and background textures
references/shaders.md — Shader pipeline and post-processing
references/scenes.md — Scene function patterns and dispatchers
references/inputs.md — Input loading and feature extraction
references/optimization.md — Performance tuning and caching
references/troubleshooting.md — Common issues and solutions

Key Components

`Renderer`

The main rendering class that manages grid rendering, character placement, and color application.

`SCENES`

A table mapping timestamps to scene functions and their configuration.

`render_clip()`

Function that renders a single clip segment, used for parallel rendering.

`_render_vf()`

Internal function for rendering video frames.

`blend_canvas()`

Function for blending multiple canvas layers.

`PixelBlendStack`

A stack-based blending system for compositing layers.

`_apply_shader_step()`

Internal function for applying a single shader step in the pipeline.