Kerchunk
Explains how Kerchunk creates cloud-friendly access to netCDF/HDF5 data by extracting metadata and enabling parallel chunk reads.
View sourceWhat this file does
Explains how Kerchunk creates cloud-friendly access to netCDF/HDF5 data by extracting metadata and enabling parallel chunk reads.
When to use it
- Working with large netCDF or HDF5 files in cloud storage
- Needing parallel or concurrent data access without full file downloads
- Combining multiple files into a single virtual dataset
- Evaluating Kerchunk's trade-offs for your workflow
Assumes this stack
tags:
- To Do
- Kerchunk
Kerchunk
!!! warning
Unsorted notes
kerchunk supports cloud-friendly access of data with specific reference to netCDF4/HDF5 files.1
How? Kerchunk
- extracts metadata in a single scan
- arranges multiple chunks from multiple files
- with dask and zarr, reads chunks in parallel and/or concurrently within a single indexible aggregate dataset
+ advantages
- supports parallel and concurrent reads
- memory efficiency
- parallel processing
- data locality
- drawbacks
- ?
How does it work?
-
Combines
fsspec,dask, andzarrone_day_data = read_chunk("10_year_data_chunked.hdf5", chunk_index=0) -
Reference file :
{ "version": 1, "shapes": {"var1": [365, 24]}, "refs": { "var1/0": ["file_1.nc", "0:24"], "var1/1": ["file_2.nc", "0:24"], // ... } }
!!! seealso
- [References specification](https://fsspec.github.io/kerchunk/spec.html)
Real world examples
- Using Kerchunk with uncompressed NetCDF 64-bit offset files: Cloud-optimized access to HYCOM Ocean Model output on AWS Open Data
- Generate references for HYCOM using kerchunk (Notebook)
Footnotes
-
Development supported by NASA fundung https://doi.org/10.6084/m9.figshare.22266433.v1 ↩
What's inside
5 sections: intro, advantages, drawbacks, how it works with code example, real-world links
Change this for your project
- Replace
"file_1.nc"and"file_2.nc"in the reference JSON with your actual file paths - Replace
"10_year_data_chunked.hdf5"in the code example with your own file name - Replace the HYCOM example links with your own dataset examples if applicable
Where it goes
Keep it in your repository where the agent or team that needs it will read it.
Related Documents
基于命题分块以增强RAG
Implements proposition chunking for RAG: decomposes documents into atomic facts, evaluates quality, and compares retrieval against standard chunking.
TileMap Chunk Manager
Divides a large tilemap into fixed-size chunks, loading and rendering only those visible to reduce memory and draw calls by over 99%.
Message Chunking for MCP stdio Transport
Explains macOS 64KB pipe write limit and provides a chunking layer for MCP stdio transport that splits and reassembles large messages transparently.
🤖 n8n AI Agent Mastery Course 2025
Serves as the main README for an n8n course repository, listing 10+ AI agent workflow projects with links to YouTube tutorials and Docker setup instructions.