Building a gridded dataset (CLI)

This page walks through building a gridded dataset from a recipe using the anemoi-datasets command line tool. For the recipe syntax itself, see Building your own datasets.

A gridded dataset stores model fields on a (possibly unstructured) spatial grid that is regular in time. On disk it is a 4-D array (dates, variables, ensembles, grid_points).

Recipe

Set output.layout to gridded in the recipe. This is the default, so it may be omitted:

dates:
  start: 2020-01-01 00:00:00
  end: 2020-12-31 18:00:00
  frequency: 6h

input:
  mars:
    # ... source definition ...

output:
  layout: gridded

One-shot creation

The simplest way to build the dataset is the create command, which runs every step in a single process:

anemoi-datasets create dataset.yaml dataset.zarr --overwrite

Before running a full build, you can generate a reduced recipe with the create-test-recipe command and build that first to check the configuration.

Incremental / parallel

For large datasets, build the dataset step by step so the loading can be split across processes, terminals or SLURM jobs. This mirrors the general incremental build workflow.

  1. Initialise the (empty) dataset. The recipe is copied into the store, so it is no longer needed by the following steps:

    anemoi-datasets init dataset.yaml dataset.zarr --overwrite

  2. Load the data in parts. Parts are numbered 1/N … N/N (1-based) and can be run in any order and in parallel; zarr handles the concurrent writes:

    anemoi-datasets load dataset.zarr --parts 1/20
anemoi-datasets load dataset.zarr --parts 2/20
# ... up to ...
anemoi-datasets load dataset.zarr --parts 20/20

    For gridded datasets, the per-group statistics are computed on the fly as each part is loaded and cached in the work directory.

  3. Finalise the dataset. This merges the partial statistics, writes the metadata and attributes, and removes the temporary files:

    anemoi-datasets finalise dataset.zarr

  4. Patch the metadata (this removes the reference to the recipe file used at init time):

    anemoi-datasets patch dataset.zarr

You can follow the progress at any time with:

anemoi-datasets inspect dataset.zarr

If temporary files are left behind, remove them with:

anemoi-datasets cleanup dataset.zarr

Additional statistics

Increment statistics (e.g. for 6h or 12h tendencies) are added with the *-additions commands, as described in Creating a dataset incrementally.

See also