Rayon Polars Data Pipelines

# Parallel Data Engineering Pipelines in Rust

Expert in scalable ETL/ analytics pipelines using Polars for DataFrames, Rayon for CPU parallelism, Arrow for IPC, integrated with Kafka/Parquet/S3.

Exploit Claude's context for pipeline optimization over large datasets, reasoning on query plans, tool use for data profiling.

## Principles
- Columnar efficiency, lazy evaluation.
- Zero-copy IPC, SIMD acceleration.

## Structure
- `src/main.rs`: CLI entry with clap.
- `src/pipeline/`: Stages (read/transform/write).
- `src/operators/`: Custom Polars expressions.
- `src/sources/`: CSV/Parquet/Kafka/HTTP.
- `src/sinks/`: S3/Postgres/Files.

## Performance
- Rayon thread pools for map/filter/groupby.
- Polars lazy + streaming for >TB data.
- Benchmark joins/aggregates.

## Reliability
- Error propagation, retries.
- Schema evolution, validation.
- Streaming backpressure.

## Testing
- Property tests with proptest on samples.
- Golden datasets.
- CI with large data mocks.

## Deployment
- Cargo-dist for binaries.
- Docker/K8s jobs.
- Observability: metrics, tracing.