A new open-source project called h3-metal delivers a native Metal implementation of MiniMax-H3, the text-to-video model, for Apple Silicon. The release includes a tutorial, performance optimizations, and explicit quality and speed tradeoffs. Prompt-to-video and prompt-to-audio, first and last frame conditioning, and ordered Ref2VA references all work end to end. Current work focuses on incremental H3-specific Metal performance and memory optimization on M3 Max and M5 Max hardware.
The project is built as a sequence of working vertical slices. Those slices cover host and model metadata, Metal block parity, prompt encoding, prompt-to-video and prompt-to-audio, first and last frame conditioning, and ordered references. The tutorial assumes a Hugging Face snapshot in ./MiniMax-H3 and FFmpeg and FFprobe on PATH. The first command builds the project and inspects the model layout: make -j8; mkdir -p outputs; ./h3 , info -d ./MiniMax-H3. The , info flag checks the model layout and prints the selected Metal device without mapping weights or generating media.
Without the -p flag, the binary starts an Iris-style interactive session. The command ./h3 -d ./MiniMax-H3 , width 512 , height 512 , steps 6 launches that session. The session keeps BF16 prompt conditioning, the prepared DiT, and the video decoder in memory for reuse. Interactive commands include !status, !seed random, !seconds 2, !show, !save output.mp4, !cache, and !help. First and last frame conditioning uses !first opening.png, !last ending.png, !first clear, and !last clear. Ref2VA references use !ref-image PATH, !refs to list order, !ref-remove N, and !refs clear. Ref2VA references cannot be mixed with !first and !last anchors.
Building a First Fast Video
A validated balanced preset generates 22 frames at 24 fps, roughly 0.92 seconds of video. The preset displays the middle-video frame after each denoising transition. An example command demonstrates the balanced path: ./h3 , profile -d ./MiniMax-H3 -p "A red fox walks through fresh snow in a pine forest. Medium tracking shot, natural winter light, realistic fur, soft footsteps and wind." , width 512 , height 512 , frames 22 , steps 20 , layers 45 , reuse 2 , show -o outputs/fox-fast.mp4. The , steps 20 flag performs 20 denoising passes. The , reuse 2 flag computes 11 fresh denoiser velocities instead of 20 and extrapolates the skipped transitions. The , layers 45 flag runs 45 of the 50 transformer blocks.
The , show flag supports Kitty and Ghostty, plus iTerm2, WezTerm, and Konsole graphical protocols. It loads a resident preview VAE, displays a representative middle-video frame after every Euler transition, and then shows all final frames. Display dimensions default to 2x for Retina displays; , zoom 1 targets non-HiDPI displays. The , show flag adds preview decode time and roughly 10 GiB of temporary model residency. The , profile flag does not change the generation path.
The first invocation pays model loading and filesystem-cache costs. The workload is sensitive to thermal throttling, so comparisons should use repeated runs. A short iteration command is , steps 4 , layers 50 , reuse 1 , show. The , steps N flag always means exactly N denoising passes. Four through seven passes use the same schedule that won a low-budget comparison. Increasing from 4 to 7 steps improves detail and motion. Keep , reuse 1 at small budgets.
Tail-heavy schedules were evaluated and rejected. They preserved too few early composition updates, producing woven texture, weak motion, or clipped colors. The retained mode uses a released linear base grid with one terminal point. The four-pass result on the 512-square, 22-frame fox test measured 0.556 full-video SSIM against a 29-pass reference. An independent surfer test measured 0.547 SSIM. The four-pass denoise took roughly 3.5 seconds on an M5 Max, compared to 26.4 seconds for the reference.
Moving Toward Reference Quality
The close-quality path uses , steps 50 , layers 50 , reuse 1. The defaults are , steps 20 , layers 50 , reuse 1. Numerical pixel identity with MLX is not expected because the random-number and execution engines differ. The depicted content and motion should agree. Speed and quality controls include , steps with values 50, 20, or 4 through 7, , reuse with values 1, 2, or 3, , layers with values 50, 45, or 40, , core-reuse with values 1, 4, or 6, , token-reduction, and , render-width and , render-height with values like 384 or 320 for 512 output.
The , core-reuse flag refreshes patch and head work every step but runs the expensive core less often. The , token-reduction flag pairs horizontal video tokens inside middle blocks and may change composition. The internal canvas runs at 384x384 for 512 square output, or 320x320 for aggressive settings. The , use-int8-row-fc2 flag on M5 uses one activation scale per FC2 row and a single full-width TensorOps product. It is optional and less numerically conservative than grouped int8. The flag reduced complete denoiser forwards by roughly 2.6% in reciprocal tests. Matched four-step fox and surfer videos with int8-row-fc2 measured full-video SSIM of 0.919 and 0.828.
The , reuse and , core-reuse flags are mutually exclusive. Layer thinning can combine with either. A token reduction example uses a surfer prompt with , steps 20 , layers 45 , reuse 2 , token-reduction. Token reduction cut the 45 layers plus reuse 2 denoise profile from 16.69 to 12.60 seconds on an IT M5 Max. An aggressive preview uses , render-width 320 , render-height 320 , steps 20 , layers 40 , reuse 3. That combination produced a clean, recognizable 22-frame fox but loses fine detail and can change framing. Do not add , token-reduction to both , layers 40 and , reuse 3. That combination caused color ringing, outlines, and ghosted limbs.
A core reuse example uses , layers 45 , core-reuse 4. The , core-reuse 6 setting is only for aggressive preview. Values above 6 are not exposed because they lose subject fidelity.
Choosing Resolution and Duration
Resolution limits require width and height to be multiples of 32, at least 32, and the product must be at most 768 times 1344 pixels. H3-Base is a 768p model. Canvas guidance marks 512x512 as the safest choice. The 768x768 canvas is validated for close-quality work. The 1344x768 and 768x1344 canvases are released 768p-class. The 1024x768 and 768x1024 canvases are valid 4:3 and 3:4. The 384x384 internal canvas for 512x512 output is validated fast-quality. The 320x320 internal canvas for 512x512 output is validated aggressive. The 256x256 native canvas is a fast-preview option with automatic low-resolution RoPE adaptation.
At 256 square, H3 has an 8x8 effective spatial-token grid. H3 automatically halves spatial RoPE coordinates at exactly 256 square, which removed repeating lattice artifacts. The , use-reference-rope flag restores released and MLX coordinates for parity checks. Keep token reduction off at 256 size. Native 128 square is unsupported because the 4x4 token grid did not recover a recognizable subject.
The , render-width and , render-height flags must be set together, must match the output aspect ratio, and cannot exceed output dimensions. The model and VAE use the internal size. Terminal frames and encoded video retain the requested output size. H3 emits 24 fps and aligns frame requests upward to the legal temporal shape of 5 plus 17 times n. The , seconds N flag handles duration-oriented requests. The , frames N flag gives direct frame control. The two are mutually exclusive. Fractional seconds are accepted, converted at 24 fps, and rounded upward to the next legal H3 temporal shape. The , seconds 10 command produces 243 frames, or 10.125 seconds.
Frame counts and durations include 22 frames at 0.917 seconds, 39 frames at 1.625 seconds, 56 frames at 2.333 seconds, 107 frames at 4.458 seconds, 243 frames at 10.125 seconds, and 362 frames at 15.083 seconds. The released workflow is intended for roughly 4 to 15 second videos. A , frames 23 request rounds up to 39 frames.
Improving the Prompt and Previewing
Prompt guidance recommends stating the subject, action, setting, camera, lighting and style, and desired sound, in a Context-IR-like description. An example prompt reads: "Scene: a single red fox in a snow-covered pine forest at dawn. Action: the fox walks steadily left to right and looks toward the camera once. Camera: medium-height lateral tracking shot, 50 mm lens, stable framing. Look: photorealistic fur, cold blue ambient light, warm sunrise rim light. Audio: soft footsteps in snow, light wind through pine branches, no music." The , seed N flag controls the native random stream and defaults to 42.
The , show flag displays a representative frame after every denoising transition, then all frames. The , zoom N flag changes the display factor without resizing video. The , frames-dir DIR flag writes final callback frames as PPM files. Intermediate previews are not written there. The -o '' flag disables MP4 encoding and can combine with , frames-dir when FFmpeg is unavailable. The , profile flag reports phase wall time, Metal encoding and wait time, peak live tensor storage, cumulative allocation, and dispatch counts.
An example profiling command is: ./h3 , profile -d ./MiniMax-H3 -p "A hummingbird hovering over red flowers." , width 512 , height 512 , frames 22 , steps 20 , layers 45 , reuse 2 , frames-dir outputs/hummingbird-frames -o ' '.
Adding Image, Video, and Audio References
First and last frame anchors select the FL2VA path with , first-frame fox.png , last-frame fox-later.png. Ordered references select a distinct Ref2VA checkpoint. Reference flags include , ref-image, , ref-silent-video, , ref-video, , ref-video-audio VIDEO AUDIO, and , ref-audio. Reference flags may be repeated, and command-line order is preserved. Standalone audio must accompany an image or video reference. Audio references must be 2 to 15 seconds. At most three audio inputs are accepted, and total decoded duration is capped at 15 seconds.
Example reference commands include:
./h3 -d ./MiniMax-H3 -p "A red fox walks through fresh snow in a pine forest. Medium tracking shot, natural winter light, realistic fur." \
, width 512 , height 512 , frames 22 , steps 20 \
, ref-silent-video fox.mp4 -o outputs/fox-video-reference.mp4
That command preserves the clip's embedded audio. Another example replaces a video's soundtrack explicitly:
./h3 -d ./MiniMax-H3 -p "A surfer riding inside a sharp blue ocean wave, one rider and one white board, realistic spray." \
, width 512 , height 512 , frames 56 , steps 20 \
, ref-video fox-with-audio.mp4 -o outputs/fox-video-audio.mp4
A third example replaces audio with an explicit file:
./h3 -d ./MiniMax-H3 -p "A red fox walking through snow, realistic, tracking shot." \
, width 512 , height 512 , frames 56 , steps 20 \
, ref-video-audio silent-fox.mp4 replacement.wav \
-o outputs/fox-replaced-audio.mp4
An ordered image plus standalone audio reference uses , ref-image and , ref-audio together. A simple prompt like "A red fox walks through fresh snow in a pine forest." works for quick tests. When composing reference prompts, the guidance is to "Use the animal and setting in the reference."
Testing and Implementation Details
The make test target runs a deterministic host suite. When an MLX fixture is installed under misc/fixtures, it compiles Metal source at runtime and checks a toy H3 block against named MLX outputs. Runtime compilation is intentional and follows Iris. It does not require Xcode's optional offline Metal toolchain. The test covers the F32 diagnosis path and the production BF16 storage path. Wide BF16 matrix products and SDPA use cached MPSGraph graphs with direct Metal correctness fallbacks. The make parity target runs only Metal and MLX checks.
FFmpeg and FFprobe must be on PATH. The H3_FFMPEG and H3_FFPROBE environment variables may select explicit executables. Generated RGB24 and 32 kHz stereo F32 PCM feed through concurrent pipes. No intermediate uncompressed media file is written. The default sampler uses the released shifted video and audio schedule. The , steps flag always names the number of denoising passes, with a terminal zero added after the last pass.
Whole-denoiser reuse evaluates the first and last pass plus every requested interval, then extrapolates skipped video and audio velocities on independent schedules. With very small step counts, keep , reuse 1. The low-budget path uses a released linear base grid. That grid won against actual-video-sigma linear spacing, quadratic and cubic warps, exact 30-point tail subsets, mild power warps, zero-order held full-grid velocities, linear velocity extrapolation, and RES. Tail-heavy candidates often sharpened the subject but damaged motion or left a repetitive woven background. Sparse RES and long extrapolation intervals failed more visibly.
Layer thinning ranks the checkpoint's actual AdaLN gates while protecting structurally important first and final blocks. Unused weights and schedule tensors are not retained. The , layers 45 and , layers 40 flags reduce transformer time and unified-memory use. Core reuse holds the previous full transformer residual while refreshing patch projection and timestep-aware head. It is mutually exclusive with whole-velocity reuse.
Exact DiT Fusions
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
Every active DiT block fuses the attention residual gate with the following MLP AdaLN. The rounded BF16 residual is written exactly. The same row stays in threadgroup memory for normalization, eliminating one dispatch and one global reread. Away from token-reduction boundaries, the MLP residual gate also produces the next block's attention AdaLN and carries normalized state across the loop. The H3_DISABLE_FUSED_GATE_ADALN=1 and H3_DISABLE_FUSED_CROSS_BLOCK_ADALN=1 environment variables restore two-kernel oracles.
Final audio and video AdaLN kernels bind directly to offsets in the residual stream. This avoids two slice blits and 18.8 MiB of scratch at 512x512, or 29.4 MiB at the 864-class benchmark shape. The H3_DISABLE_FUSED_FINAL_SLICE=1 flag restores the copy-plus-AdaLN oracle at load. BF16 final heads apply AdaLN while loading 16x16 projection tiles. This preserves standalone rounding and accumulation order while removing another equally sized normalized activation. The two optimizations together save 37.5 or 58.9 MiB. The H3_DISABLE_FUSED_FINAL_HEAD=1 flag restores the offset-AdaLN-plus-linear oracle at load.
Token Reduction Internals
Token reduction pairs adjacent horizontal target-video tokens after block 3. Text, audio, conditions, and reference tokens remain exact. A complete full-resolution state is kept as a bypass. During the first ten noisy evaluations, the bypass restores before block 40. Subsequent detail-forming evaluations restore before block 30. Each token returns as its original value plus an update learned by the pair. Within-pair detail is not discarded.
The pooling kernel writes only true-pair baselines into the dense tail of an already allocated attention scratch buffer. Odd-width singleton tokens need no baseline. The full bypass uses an oversized QKV tail when it fits. A guarded dedicated fallback handles reference-heavy layouts. Common text-only canvases add no activation arena at any token-grid width. Pooling snapshots both source tokens while BF16 values are in registers, avoiding a separate full-hidden blit and a redundant source read. The entry kernel keeps each pooled row in threadgroup memory and emits the first reduced block's attention AdaLN, eliminating another global residual read.
At the restore boundary, the first full-resolution attention AdaLN is fused into expansion. A 10.5 KiB threadgroup row avoids a global residual reread while writing the exact bypass. On a thermal-balanced 512x512x22, 19-forward IT M5 Max A/B test, token reduction reduced denoise time from 39.13 to 28.06 seconds, a 28.3% improvement. Final video and audio latent relative L2 measured 5.56% and 15.14%. First, middle, and last fox frames retained one clean muzzle, coherent legs, and sharp fur. An independent surfer remained consistent with one rider and board. Token reduction changes composition, so it is opt-in rather than the close-reference default.
The H3_TOKEN_REDUCTION_BLOCKS variable can override the later 4:30 interval. The H3_TOKEN_REDUCTION_EARLY=STEPS:END variable overrides the early schedule, and 0 disables it. The H3_DISABLE_TOKEN_REDUCTION=1 flag provides an in-context exact oracle. The H3_DISABLE_FUSED_TOKEN_POOL_ADALN=1 and H3_DISABLE_FUSED_TOKEN_ADALN=1 flags independently restore two-kernel entry and exit boundaries. Token reduction composes cleanly with , layers 45 , reuse 2, reducing the profile from 16.69 to 12.60 seconds, a 24.5% marginal gain. Do not combine it with both , layers 40 and , reuse 3. A 6.47-second experiment produced chromatic ringing and ghosted limbs despite acceptable latent norms.
Internal Canvas and Video VAE
The , render-width and , render-height flags run the model and VAE on a lower same-aspect internal canvas. High-quality vImage scaling then upscales RGB frames to the requested output size. A measured 384-to-512 prompt render reduced M5 DiT time by 33% and video-VAE time by 18% while retaining a clean, recognizable photorealistic result. Both values must be multiples of 32. The exact output canvas remains the default. For square 512 output, 384 is the fast-quality point and 320 is the validated aggressive point. At 320 internal, a coherent walking fox repeated at 8.02 seconds of DiT time versus roughly 15.82 seconds natively.
Native 256 uses the same-cost spatial-RoPE adaptation. It is a fast composition preview, not a substitute for 512- or 768-class final renders. The video VAE automatically chooses a 256 to 320 pixel spatial tile from the requested canvas geometry. This minimizes repeated overlap work while keeping peak storage bounded. The H3_VAE_TILE_PIXELS=256 flag restores the original conservative tile plan.
Weight Residency and Streamed Prompt Encoding
On M5-class GPUs, persistent transformer weights map directly from safetensor shards instead of being copied into anonymous shared buffers. This keeps the 37 GiB model file-backed and reclaimable. It also slightly improves total transformer time. The M3 uses a faster copied-buffer path. The H3_ZERO_COPY_WEIGHTS=0 flag disables the M5 selection for diagnostics.
The streamed Qwen text encoder preallocates a small ring of future layer buffers and fills them on eight I/O workers while Metal executes the current layer. The default ring depth is two layers on M3 and older hardware, and three on M5, since the target machine has 128 GiB. The H3_QWEN_PREFETCH=0 flag restores the single-layer synchronous reference path. Values 1 through 8 select the worker count. The H3_QWEN_PREFETCH_DEPTH=1 through 6 flags override the ring depth.
Metal 4 and TensorOps Paths
M5 GPUs automatically use native BF16 Metal 4 and TensorOps for DiT QKV and attention-output projections at sequence lengths up to 2,048. A compact Morton schedule routes Q, K, and V directly into head-major attention inputs. This avoids three MPSGraph input transposes and is byte-identical to the portable path. It improves the complete 512x512 50-block forward by roughly 2% across repeated IT and US M5 Max runs. For 2,049 to 3,072 rows, including 864x480, two row-offset Morton dispatches preserve efficient tile geometry. They improve the complete forward by about 2% in balanced runs. Still larger sequences stay on MPSGraph.
The H3_NAX=0 flag disables TensorOps for exact A/B diagnosis. Selection is guarded at runtime and falls back to the unchanged portable library if compilation is unavailable. The H3_NAX=1 flag forces a broader native BF16 linear path. It passes the complete 50-block MLX fixture, but it is opt-in because exact-shape microbenchmarks favor its 128-row tile while full DiT runs currently favor MPSGraph scheduling. The H3_NAX=mlp flag selects a more specialized Metal 4 path. Paired FC1 gate and up TensorOps tiles apply SwiGLU in threadgroup memory and write only a 14,336-wide activated intermediate. FC2 also stays on TensorOps.
The H3_DISABLE_NAX_MLP=1 flag keeps the MPSGraph MLP in a context created this way for same-process A/B testing. The path is deliberately opt-in because scheduling depends on the OS GPU stack. On a primary macOS 26.5.2 M5 Max, the isolated real-weight MLP runs gained 1.3 to 2.0% but lost roughly 1 to 3% in the complete 50-block forward. An identical macOS 26.5 M5 Max gained 1.4% in the same-context forward A/B. The resulting 50-block velocities are close, with 1.9% video and 2.4% audio relative L2, but not byte-identical.
Specialized Projection Kernels
Narrow DiT audio and video output heads convert small released F32 weights to BF16 once. They use an Iris-derived 16x16 tiled linear directly on BF16 activations. At production 320-render geometry, isolated paired-head measurements were 2.30x faster on M3 Max and 1.83x faster on M5 Max, with relative L2 of 8.64e-4. The absolute M5 saving is roughly 0.6 ms per evaluated step. Full fox and surfer sequences remained clean, measuring 29.9 and 38.4 dB against F32-head renders. The H3_DIT_F32_FINAL=1 flag restores the close-reference head and extra activation buffers.
F32 96-to-5376 video and 32-to-5376 audio patch projections use a dedicated 16x16 cooperative tile. They retain F32 weights, inputs, and accumulation while rounding the tile result directly to BF16. Paired production-shape measurements were 1.77x faster on M3 and 1.62 to 1.78x on M5. The complete generated RGB stream is byte-identical to the scalar path. Fusing the final cast improves the 2835-row tile from 2.499 to 1.734 ms on M3 and from 1.555 to 1.186 ms on M5. It removes 38.27 or 59.66 MiB of F32 scratch at 512 or 864-class geometry. The H3_DISABLE_FUSED_PATCH_CAST=1 flag restores tiled F32 output plus a standalone cast. The H3_SCALAR_PATCH=1 flag selects a scalar diagnostic path.
The same tile binds output directly into the packed hidden stream. This removes BF16 media staging buffers and blits, saving another 19.13 or 29.83 MiB. It improves the 2835-row boundary from 1.847 to 1.730 ms on M3 and from 1.282 to 1.184 ms on M5. Contiguous T2VA uses byte offsets. FL2VA and Ref2VA use compact destination-row maps so each modality remains one large dispatch. A complete six-segment Ref2VA M5 ABBA remained byte-identical and improved from 5.067 to 5.033 seconds per measured forward pair. The H3_DISABLE_FUSED_PATCH_PACK=1 flag restores staging buffers and packing blits.
Scheduling and Activation Memory
The DiT core splits into two ordered Metal command buffers. GPU execution of the first part overlaps CPU encoding of the second. Thermal-balanced ABBA measurements select a 60%-depth split on M5, with 30 of 50, 27 of 45, and 24 of 40. The gains range from 0.5 to 1.8%. The M3 automatically splits only the validated 30 of 50 case, which is 1.2% faster, because 24 of 40 regressed. Operation order and generated bytes are unchanged. The H3_DIT_COMMAND_BLOCKS=0 flag restores one command buffer. Values 1 through 50 override the split.
DiT activation buffers follow actual intra-block lifetimes. The QKV projection arena is reused first for attention heads, then for the normalized MLP input. The current attention-output arena becomes the MLP output after the branch is consumed. This removes 61.25 MiB at 512-class geometry and 99.63 MiB at 864-class geometry without changing dispatches or arithmetic. The H3_DISABLE_DIT_ACTIVATION_ALIAS=1 flag restores separate diagnostic buffers.
MPSGraph tensor-data wrappers for immutable DiT weights and biases are retained with resident buffers. This avoids rebuilding binding metadata for every block and denoiser evaluation without copying tensor storage. Measured ABBA gains are 1.6% on M3 Max and 0.4 to 1.1% on M5 Max. Activation wrappers stay transient because retaining them regressed M5. Outputs are byte-identical. The H3_DISABLE_GRAPH_DATA_CACHE=1 flag restores transient wrappers for all tensors.
On M3 and older hardware, four MPSGraph segments in each DiT block reuse one MPSCommandBuffer wrapper for the shared underlying Metal command buffer. Repeated thermal-balanced runs measured 1.0 to 1.6% faster on M3 Max. The M5 is neutral and retains fresh wrappers. The H3_REUSE_MPS_COMMAND=0 or 1 flags override automatic selection. Results are byte-identical.
On M5, the serving Euler sampler keeps patch-packed F32 latents and cached BF16 velocities in Metal buffers. Each selected denoiser refresh completes before the next is encoded. This avoids MPSGraph back-pressure while removing all intermediate latent and velocity readbacks and repacking. Two warm eight-run A/B sequences measured small 0.1% and 0.3% gains with byte-identical final latents. It saves roughly 16 bytes of transient host state per video-latent element, about 136 MB at 768p shape. M3 and older GPUs retain the CPU sampler by default. The H3_CPU_SAMPLER=1 flag restores it on M5. The H3_GPU_SAMPLER=1 flag selects the GPU-state path explicitly. The H3_GPU_SAMPLER_WINDOW=0 flag enables a slower unbounded encode-ahead diagnostic.
Checkpoint Layout and Media Pipeline
The released checkpoint stores DiT QKV rows interleaved per attention head. Native Metal consumes this directly in the fused QK-normalization and RoPE kernel. This avoids a checkpoint transpose and extra RAM. An earlier identity interpretation was the cause of noisy diagnostic outputs. The public generation path decodes the joint audio latent with streamed native BigVGAN and AudioVAE. It writes synchronized H.264 plus 32 kHz stereo AAC. The native waveform agrees with the corrected MLX oracle to relative L2 of 6.94e-5.
The , first-frame, , last-frame, and their combination use the released visual VAE encoder, the Qwen3-VL vision tower, and a three-deepstack multimodal presentation. They use 0.999 condition augmentation and fixed condition rows in the native DiT. The first image stretches to the target canvas. The last image is aspect-cover scaled and center cropped, matching the reference implementation. The , ref-image flag selects the distinct Ref2VA transformer, preserves ordered picture presentation, and uses the released down-only aspect-preserving reference canvas.
The , ref-silent-video flag additionally performs bounded 24 fps decoding, the visual VAE's causal ceil of T over 4 compression, two-frame Qwen sampling, and timestamped video presentation. The , ref-video flag preserves the embedded soundtrack. The , ref-video-audio VIDEO AUDIO flag supplies an explicit replacement. The , ref-audio flag appends an ordered standalone clip. Reference audio decodes as 32 kHz stereo F32, is encoded by the native AudioVAE posterior-mean path, and is mixed as 0.999 clean latent plus 0.001 seeded noise. It is pinned to the audio condition timestep 1.0 and packed as width-32 rows on the same rotary timeline as visual references.
Audio inputs must be 2 to 15 seconds. At most three are accepted, and total decoded duration is capped at 15 seconds. Standalone audio must be combined with an image or video reference. The native audio encoder matches the corrected MLX oracle at relative L2 of 3.59e-6 on a real two-second stereo fixture. The correction matters because the original MLX reshape interleaved left and right samples. The official PyTorch and SGLang path folds intact stereo channels into the batch dimension. On a 128 GB M5 Max, clean end-to-end image plus audio and embedded-video plus audio renders completed in 74.58 and 76.99 seconds respectively. Each used roughly 40.1 GB peak physical footprint with zero swaps.
Profiling and Diagnostic Paths
The , profile flag reports each Metal-backed phase separately. It includes wall time, CPU-side command encoding, complete commit-to-fence wait, root-command GPU timestamps, peak live tensor storage, cumulative allocation, and dispatch counts. The wait measurement is complete command turnaround. The root GPU timestamp alone can omit child buffers scheduled internally by MPSGraph and is labeled accordingly.
The DiT fast path evaluates each BF16 fc1, SwiGLU, fc2 block as one cached graph. This avoids separate graph boundaries and persistent intermediate tensors. The H3_DISABLE_FUSED_MLP=1 flag retains close-reference operation boundaries for numerical diagnosis. On supported M5 Metal 4 TensorOps hardware, the native int8 MLP engine is the default. It dynamically quantizes activations and uses per-output-channel weight scales. The sensitive FC2 input gets one scale per 1,024 channels. The selected FC2 kernel keeps scaled partial products in private cooperative fragments instead of repeatedly spilling a 32 KiB threadgroup tile.
A fixed 50-layer, 19-transition 512x512 render measured 36.30 seconds with BF16 MPS and 25.80 seconds with int8 on M5 Max. Beginning, middle, and final decoded frames retained the same subject, composition, and motion. Small edge and fur details can differ. The current diagnostic implementation retains both BF16 and int8 MLP weights only when an A/B diagnostic requests them. Normal int8 loading releases each block's BF16 FC1 and FC2 buffers after submitted quantization finishes. This reduces measured peak tensor storage to 25.9 GiB from the BF16 path's 36.4 GiB.