PrivyDrop AI Playbook — Backpressure & Chunking Strategy (Deep Dive)

← Back to flow index: docs/ai-playbook/flows.md

(This page is the English edition of content split out from docs/ai-playbook/flows.zh-CN.md, preserving the original section numbering and structure.)

7) Backpressure & Chunking Strategy (Deep Dive)

Sender Dual-Layer Buffering Architecture

Design rationale:

• File read layer: 4MB chunks reduce FileReader calls; 8 chunks form a 32MB batch
• Network layer: 64KB pieces fit WebRTC DataChannel limits and avoid sendData failed errors
• Performance: efficient slicing inside each batch; a single FileReader.read() yields 512 network chunks

Key parameters:

TransferConfig.FILE_CONFIG = {
  CHUNK_SIZE: 4194304, // 4MB - file read chunks
  BATCH_SIZE: 8, // 8 chunks = 32MB batch
  NETWORK_CHUNK_SIZE: 65536, // 64KB - safe WebRTC send size
};

Backpressure control:

• DataChannel thresholds: bufferedAmountLowThreshold = 256KB (Initiator) and 512KB (NetworkTransmitter)
• Max buffer: maxBuffer = 1MB; wait until pressure releases when exceeded
• Async waiting: listens to bufferedamountlow, with a timeout safeguard (10 seconds)

Embedded metadata packet format:

[4-byte length][JSON metadata][payload bytes]

• Each network chunk includes: chunkIndex, totalChunks, fileOffset, fileId, isLastChunk
• Receiver can parse each packet independently without relying on extra shared state

Receiver Smart Storage Strategy

Storage decision logic:

ReceptionConfig.shouldSaveToDisk(fileSize, hasSaveDirectory);

• In-memory: file < 1GB and no save directory chosen
• Disk: file ≥ 1GB or the user selected a save directory
• Buffer cap: up to 100 chunks buffered (≈ 6.4MB)

Chunk validation:

• Format compatibility: ArrayBuffer/Blob/Uint8Array/TypedArray supported
• Integrity checks: validate fileId, chunkIndex, chunkSize consistency
• Firefox quirks: Blob size checks and conversion error handling

Strict sequential disk writes:

• SequencedDiskWriter: guarantees in-order writes; enables streaming for large files
• Resume: getPartialFileSize() checks existing partial files
• Auto completion: checkAndAutoFinalize() verifies completeness

Performance Tuning Details

Sender-side optimizations:

• Batch reads: 32MB batches reduce I/O operations and improve large file read throughput
• Network fit: 64KB balances transfer efficiency with cross-browser compatibility
• Backpressure response: leverages native WebRTC backpressure to prevent drops

Receiver-side optimizations:

• Unified format conversion: ChunkProcessor handles multiple payload formats in one place
• Progress throttling: 100ms for files, 200ms for folders to avoid UI overload
• Memory management: small files assemble in memory; large files stream to disk

Error handling:

• Send retries: NetworkTransmitter returns boolean for upper-layer retry logic
• Conversion tolerance: when Blob conversion fails, return null instead of aborting the transfer
• Timeout safeguards: 30s completion timeout; 5s graceful close timeout

Debugging & Monitoring

Dev logs:

• Chunk tracking: log details every 100 chunks and for the last chunk
• Backpressure monitoring: buffer size changes and wait-time stats
• Performance metrics: transfer speed, batch processing time, conversion cost

Production optimizations:

• Conditional logging: ENABLE_CHUNK_LOGGING and ENABLE_PROGRESS_LOGGING
• Error reporting: critical errors are sent to the backend via postLogToBackend
• Performance sampling: use performance.now() for precise timings