loafer

# Loafer

> A CLI-first, declarative ETL/ELT pipeline tool driven by YAML.

## Overview

Modern data pipelines often become bogged down by repetitive boilerplate and fragile scripts. Loafer solves this by treating data movement and transformation as configuration. 

Loafer is a CLI-first tool that allows you to extract data from databases, files, and APIs, apply powerful transformations (both custom and AI-generated), and load the results into target systems — all configured within a single, highly readable YAML file.

**Key Idea:** Describe your pipeline in YAML, run it from the CLI, and let Loafer handle the execution.

## Features

- **Declarative Configuration:** Define your inputs, outputs, and processing logic in simple YAML. No complex workflow orchestrators required.
- **ETL & ELT Support:** Choose between Extract-Transform-Load (in-memory/streaming Python transformations) or Extract-Load-Transform (in-database SQL executions).
- **AI-Powered Transformations:** Optionally use natural language to auto-generate complex Python transforms or target-database SQL using modern LLMs (Gemini, OpenAI, Gemini).
- **Streaming & Batch Processing:** Automatically switches to memory-efficient streaming for large datasets based on configurable thresholds.
- **Data Quality Guards:** Built-in validation steps to catch malformed data or high null-rates before they hit your target.
- **Developer-Friendly & Extensible:** Use custom Python files for advanced transformations when declarative logic isn't enough.

## Installation

### Via pip (PyPI)
The easiest way to install Loafer is via pip:
```bash
pip install loafer-etl
```

### Via Docker (GHCR)
Loafer is officially published to the GitHub Container Registry. Available tags include specific versions (e.g. `0.2.0`, `0.2`) and `latest`.

```bash
docker pull ghcr.io/lupppig/loafer:latest
```

To run Loafer using Docker, mount your current working directory so Loafer can access your configuration and local files:
```bash
docker ru