Neura Market
prompt logo

prompt

Free

Data Engineer agent prompt for building reliable data infrastructure

FreeFree tier
Type
Open Source
Visit Website GitHub (40 stars)

About prompt

A specialized prompt from the ai-boost/awesome-prompts collection that instructs an AI assistant to embody a Data Engineer agent. The prompt defines a detailed persona, core mission, and critical rules for designing, building, and operating data infrastructure. It covers ETL/ELT pipeline engineering with Medallion Architecture (Bronze-Silver-Gold), data platform architecture on cloud providers (Azure, AWS, GCP), data quality and reliability practices, and streaming/real-time data processing. The prompt includes technical deliverables such as Spark (PySpark + Delta Lake) code snippets, and enforces pipeline reliability standards like idempotency, schema contracts, and explicit null handling.

Key Features

Designs ETL/ELT pipelines with idempotency and observability
Implements Medallion Architecture (Bronze-Silver-Gold) with data contracts
Automates data quality checks, schema validation, and anomaly detection
Builds streaming pipelines using Kafka, Flink, Spark Structured Streaming
Provides Spark/PySpark code examples with Delta Lake
Enforces pipeline reliability standards: idempotency, explicit null handling, schema drift alerts
Architects cloud-native data lakehouses on Azure, AWS, or GCP
Establishes data lineage tracking and metadata management

Pros & Cons

Pros
  • Provides a comprehensive and detailed persona for a Data Engineer
  • Includes concrete technical standards (idempotency, schema contracts, etc.)
  • Covers both batch and streaming pipeline paradigms
  • Offers ready-to-use Spark/PySpark code snippets
  • Open-source and freely available in a curated prompt collection
Cons
  • Specific to a single prompt style; may not cover all data engineering roles or tools
  • Requires a compatible AI assistant to interpret and execute the prompt
  • The prompt is lengthy and may be too verbose for simple use cases
  • Does not include live data or production environment integration

Best For

Data pipeline engineering and architectureData platform design and optimizationData quality and reliability managementStreaming and real-time data processingTeaching data engineering best practicesAI-assisted data infrastructure development

FAQ

What is the main purpose of this prompt?
To instruct an AI assistant to act as a Data Engineer expert, covering pipeline design, platform architecture, data quality, and streaming, with detailed rules and deliverables.
What technologies are referenced in the prompt?
The prompt references Spark (PySpark + Delta Lake), Apache Kafka, Apache Flink, Azure (Fabric/Synapse/ADLS), AWS (S3/Glue/Redshift), GCP (BigQuery/GCS/Dataflow), Delta Lake, Apache Iceberg, Apache Hudi, and dbt.
Is this prompt free to use?
Yes, the prompt is part of the open-source awesome-prompts repository on GitHub and is available for free.
What is the Medallion Architecture mentioned?
It is a data design pattern that organizes data into Bronze (raw, immutable), Silver (cleansed, conformed), and Gold (business-ready, aggregated) layers, each with specific contracts.