Preprint
Computer Vision

DVID: Distributed Versioned Image-Oriented Dataservice

William T. Katz(Howard Hughes Medical Institute), Stephen M. Plaza(Howard Hughes Medical Institute)
February 5, 2019Frontiers in Neural Circuits36 citations
PDFDOI

36

Citations

5

Influential Citations

Frontiers in Neural Circuits

Venue

2019

Year

Abstract

Open-source software development has skyrocketed in part due to community tools like github.com, which allows publication of code as well as the ability to create branches and push accepted modifications back to the original repository. As the number and size of EM-based datasets increases, the connectomics community faces similar issues when we publish snapshot data corresponding to a publication. Ideally, there would be a mechanism where remote collaborators could modify branches of the data and then flexibly reintegrate results via moderated acceptance of changes. The DVID system provides a web-based connectomics API and the first steps toward such a distributed versioning approach to EM-based connectomics datasets. Through its use as the central data resource for Janelia's FlyEM team, we have integrated the concepts of distributed versioning into reconstruction workflows, allowing support for proofreader training and segmentation experiments through branched, versioned data. DVID also supports persistence to a variety of storage systems from high-speed local SSDs to cloud-based object stores, which allows its deployment on laptops as well as large servers. The tailoring of the backend storage to each type of connectomics data leads to efficient storage and fast queries. DVID is freely available as open-source software with an increasing number of supported storage options.

Analysis

Why This Paper Matters

The connectomics community faces a critical challenge: as electron microscopy (EM) datasets grow in size and number, the need for collaborative, version-controlled data management becomes paramount. Traditional snapshot-based publication of datasets hinders iterative refinement and distributed collaboration. DVID addresses this gap by introducing a distributed versioned dataservice tailored for EM-based connectomics, drawing inspiration from software version control systems like GitHub. This work is significant because it enables remote collaborators to branch data, make modifications, and reintegrate changes through a moderated workflow, mirroring best practices in open-source software development. By providing a web-based API and flexible storage backend, DVID lowers the barrier for labs of all sizes to adopt versioned data management, potentially accelerating discovery in neural circuit reconstruction.

Technical Contributions

  • Distributed Versioning: DVID introduces branching and moderated reintegration for connectomics data, allowing multiple teams to work on different versions of a dataset simultaneously.
  • Web-Based API: A RESTful API enables programmatic access to data, facilitating integration with existing reconstruction workflows and tools.
  • Flexible Storage Backend: Supports multiple storage systems (local SSDs, cloud object stores) and tailors storage to data type (e.g., images, volumes, annotations) for efficiency.
  • Deployment Scalability: Can run on laptops for small-scale experiments or large servers for production use, making it accessible to diverse research groups.
  • Open-Source Availability: Freely available software encourages community adoption and extension.

Results

The paper reports that DVID has been used as the central data resource for Janelia's FlyEM team, integrating distributed versioning into reconstruction workflows. It supports proofreader training and segmentation experiments through branched, versioned data. However, no quantitative metrics (e.g., throughput, latency, storage efficiency) or comparisons with alternative systems are provided. The results are primarily qualitative, demonstrating feasibility and practical deployment.

Significance

DVID represents an important step toward collaborative, version-controlled data management in connectomics, a field where data complexity and size are rapidly increasing. By enabling branching and reintegration, it facilitates iterative refinement of reconstructions and supports distributed teams. The flexible storage backend and open-source nature broaden its potential impact beyond connectomics to other image-oriented scientific domains (e.g., medical imaging, remote sensing). However, the lack of quantitative evaluation limits assessment of its performance relative to existing solutions. Future work should include benchmarks and comparisons to establish DVID's practical advantages.

connectomicsdistributed-versioningem-datasetsimage-oriented-dataserviceopen-sourcestorage-backendreconstruction-workflows

Related Papers