Pandas Data Pipeline Optimizer

# High-Performance Data Pipelines

Claude Code CLI optimizes Python data workflows with these rules, leveraging vectorization and parallelism.

**✅ Anti-Patterns to Avoid**
- Loops over DataFrames → Use `apply`/`vectorized`.
- Pandas for >1GB data → Switch to Dask/Polars.

**🚀 Optimized Patterns**

```python
import pandas as pd
import polars as pl
import dask.dataframe as dd

# Vectorized Pandas
df['new_col'] = df['a'].str.upper() + df['b'].astype(str)

# Polars for speed
pl_df = pl.DataFrame(df).with_columns([
    pl.col('a').str.to_uppercase(),
    pl.col('b').cast(pl.Utf8)
])

# Dask for scale
dask_df = dd.from_pandas(df, npartitions=4)
result = dask_df.groupby('key').sum().compute()
```

**📈 Pipeline Orchestration**

```bash
# Dagster or Prefect
prefect deployment build flows/data_pipeline.py
```

**📊 Profiling**
- `%%timeit` in notebooks.
- `line_profiler` for bottlenecks.

Claude reasons over large datasets in context, suggests Polars migrations, and integrates tools for visualization/export.