Oxen

Free

Fast dataset version control

FreeFree tier

Type

Open Source

Visit Website GitHub (38 stars)

About Oxen

Oxen is a lightning fast data version control system for large datasets, designed to make versioning data as easy as versioning code. The interface mirrors git, but excels where git or git-lfs fall short. Built from the ground up for any data type, Oxen is optimized to handle repositories with millions of files and scales to terabytes of data. It supports images, audio, video, text, parquet, CSV, JSON, model weights, and more, with specialized metadata extractors for certain filetypes. Oxen offers command-line interface and bindings for Rust, Python, and HTTP, enabling seamless integration into workflows. Key capabilities include efficient indexing and syncing, native tabular data handling, change tracking over time, team collaboration via an oxen-server, and workspaces for server-side data interaction. All of this is available as free and open-source software.

Key Features

Lightning fast indexing and syncing of large datasets

Git-like commands (oxen init, add, commit, push, pull) make it easy to learn

Handles large files including images, videos, audio, text, parquet, CSV, JSON, model weights

Indexes millions of files efficiently

Native tabular data handling with ability to index and query CSV, parquet, and JSONL

Tracks changes over time to prevent data loss

Supports team collaboration via centralized oxen-server

Workspaces for interacting with data on the server

Specialized metadata extractors for file types cached in the merkle tree

Bindings for Rust, Python, and HTTP interfaces

Pros & Cons

Pros

Extremely fast indexing and syncing of large datasets
Familiar git-like interface reduces learning curve
Supports a wide variety of data formats (images, video, audio, text, tabular)
Native tabular data handling for efficient querying of CSV and parquet
Open-source and free to use
Multiple integration options: CLI, Rust, Python, HTTP
Built from scratch for performance with large-scale data

Best For

Version control for machine learning datasets (images, audio, video, text)Managing large tabular data like parquet files with millions of rowsCollaborating on data projects with teamsTracking and syncing large repositories of model weightsData science and research workflow versioning