Documentation
FreeWeb scraping at scale with LLMs
FreeFree tier
About Documentation
Reworkd uses LLMs to parse, understand, and interact with web pages, enabling users to scrape web data at scale. The platform automatically handles deduplication using composite keys when scrapers re-run, supports custom schema definition with deduplication keys, and allows exporting extracted data. Customers leverage Reworkd to extract millions of rows of data for building data-constrained products, fine-tuning domain-specific language models, and enriching existing data pipelines. The team behind Reworkd also created AgentGPT.
Key Features
LLM-powered web page parsing and understanding
Automatic data deduplication using composite keys
Customizable schema with deduplication key selection
Data export functionality
Scalable extraction of millions of rows of data
Pros & Cons
Pros
- Automatically handles deduplication when scrapers re-run
- Uses LLMs for intelligent, context-aware web parsing
- Scalable to millions of rows of extracted data
- Built by the team behind AgentGPT, indicating experience in AI agent development
Best For
Building data-constrained productsFine-tuning domain-specific language modelsEnriching existing data pipelinesWeb scraping at scale for research or analysis
FAQ
How does deduplication work in Reworkd?
Reworkd automatically handles deduplication by using a unique key (or composite key) based on the record's fields. New rows are inserted and marked as CREATE, duplicate rows are skipped, and existing keys trigger an UPDATE without duplication.
What makes a good deduplication key?
A good deduplication key must be unique for every output row, remain stable over time (avoid frequently changing fields), and be consistent across different websites. Examples include SKU/UPC or a combination of Brand + Model + Color.