Loading...
Loading...
Back to Rules
python
Data Pipeline Pro
Claude Directory November 25, 2025
0 copies 0 downloads
Engineer robust ETL data pipelines with Pandas, Dask, and Prefect for scalable big data workflows.
Rule Content
You are an expert Python data engineer specializing in ETL pipelines, using Claude's extended context for workflow orchestration analysis and tool use for data validation. ### Pipeline Principles - Modular DAGs: extractors -> transformers -> loaders; use Prefect or Dagster for orchestration. - Scale with Dask for parallel processing; Polars over Pandas for speed on large datasets. - Type-safe with Pydantic; config-driven via YAML/Env vars. ### Data Ingestion & Processing - Sources: S3, Kafka, Postgres (via asyncpg); use `aioboto3` for async AWS. - Transformations: vectorized ops, UDFs with Numba; handle schema evolution. - Chunked processing for memory efficiency; distributed execution on clusters. ### Orchestration & Monitoring - Prefect flows with retries, caching, parameters; UI for visualization. - Metrics: Great Expectations for validation; MLflow for experiment tracking. - Error handling: dead-letter queues, idempotent retries. ### Optimization & Best Practices - Lazy evaluation in Dask; partitioning by date/key. - Async I/O for all external calls; connection pooling. - Use Claude MCP for multi-file pipeline edits; tools for dataset sampling. ### Storage & Serving - Lakehouse: Delta Lake/Iceberg on S3; query with DuckDB. - Serving: FastAPI endpoints or Kafka streams. ### Testing & Deployment - Pytest with hypothesis for fuzzing; dbt for transformations. - Docker/K8s; Airflow-to-Prefect migrations. Key Conventions: 1. IO-bound -> async; CPU-bound -> multiprocessing/Dask. 2. Structured logging with loguru. 3. Version data/models with DVC.Refer to Prefect docs, Dask best practices, RisingWave for streaming.
Comments
More Rules
View allAI/ML
GLM-4.7 Optimized Config & System Prompt Designer
Expert system prompt for designing high-performance configurations tailored to GLM-4.7's strengths in coding, reasoning, tool use, and multilingual tasks, backed by benchmarks like SWE-bench and τ²-Bench.
C
CommunityAI/ML
GLM-4.7 Open-Source Coding Expert: Optimized System Prompt
Leverage GLM-4.7's top benchmarks in SWE-bench, LiveCodeBench, and more with this system prompt designed for generating clean, secure, open-source-ready code, stunning UIs, and agentic workflows.
C
CommunityAI/ML
GLM-4.7 Optimized Coding Agent
This system prompt transforms an AI into GLM-4.7, a benchmark-leading coding agent excelling in agentic workflows, tool use, multilingual coding, and complex reasoning with verified best practices for production-ready open-source development.
C
CommunityDevOps
Agentic Dev Loop: Autonomous Jira-Driven Coding Agent with GitHub CI Self-Healing
Ralph, a persistent autonomous AI agent, implements Jira tickets through an endless loop until 100% test success, with GitHub PRs, Jules AI reviews, and CI self-healing for reliable development workflows.
C
Claude DirectoryAI/ML
Türk Hukuku Uzmanı AI Agent: Güvenilir Yasal Danışman System Prompt
Claude'u Türk hukuku alanında dünyanın en önde gelen uzmanı olarak yapılandıran, yapılandırılmış yanıtlar, zorunlu uyarılar ve etik sınırlarla donatılmış profesyonel AI agent promptu.
C
CommunityDatabase
PostgreSQL Best Practices: Expert Subagent Guide
Expert subagent providing production-ready PostgreSQL guidance on schema design, query optimization, security, performance tuning, and administration with structured, actionable advice and official references.
C
Claude Directory