Recipe

A recipe is a YAML file that describes how to build a dataset. It is composed of a list of sources and filters, and the operations to combine them. Below is an example of a recipe. The order of the entries is not important, but we recommend following the order of the example for readability.

# Dataset description (compulsory)

description: |
  Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod
  tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
  quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

  Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu
  fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident,
  sunt in culpa qui officia deserunt mollit anim id est laborum.

# The name of the dataset (optional, but recommended). This entry is ingrored by anemoi-datasets
# but use by some tools to identify the dataset. Best practice is to use hyphens, not underscores or camel case, as these names
# will be used to build URLs.

name: my-dataset

# The licence under which the dataset is released (optional, but recommended).

licence: CC-BY-4.0

# The name of the author(s) of the dataset (optional, but recommended).

attribution: ECMWF

# The range of dates covered by the dataset (compulsory for the
# gridded and tabular layouts; mutually exclusive with `base_dates:`
# and `steps:` used by the trajectories layout — see
# docs/layouts/trajectories.rst).
# The various source are called with a list of dates build from start, end and frequency.
# How the dates are used by the sources depends on the source (gridded or tabular).
# The frequency can be given in human friendly format (e.g. 6h, 1d) or in ISO 8601 format (e.g. PT6H, P1D).
# All dates are always undestood as UTC.

dates:
  start: 2000-01-01 00:00:00
  end: 2000-01-10 18:00:00
  frequency: 6h

# For the trajectories layout only, `dates:` is replaced by `base_dates:`
# (forecast initialisation times) and `steps:` (forecast lead times).
# The resulting samples are the Cartesian product base_dates x steps.
#
# base_dates:
#   start: 2000-01-01 00:00:00
#   end:   2000-01-10 00:00:00
#   frequency: 12h
#
# steps:
#   start: 6
#   end: 30
#   frequency: 6h

# This part is compulsory and describes how to build the dataset using sources and filters

input:
  pipe:
    - grib:
        path: /path/to/file.grib
    - rename:
        t2m: 2t

# An optional section describing named snipets of code that can be reused in the recipe

data_sources:
  my_source_1:
    grib:
      path: /path/to/file.grib
  my_source_2:
    netcdf:
      path: /path/to/file.nc

# List of options to control the output of the dataset (optional)

output:
  layout: gridded # The layout of the dataset: `gridded`, `tabular` or `trajectories`.
  dtype: float32 # The data type of the output variables (e.g. float32, int16, etc.)
  # chunking: {dates: 1, ensembles: 1}         # for layout: gridded
  # chunking: {base_dates: 1, steps: 1, ensembles: 1}  # for layout: trajectories
  #
  # Note: `output.order_by` is deprecated. The cube ordering is
  # hard-coded per layout and no longer read from the recipe.
  # Note: `output.flatten_grid` has been removed; flattening is always on.

# List of options to control the building process (optional)

build:
  group_by: 10 # The number of dates to process together (e.g. 10, monthly, weekly, etc.)

# List of options to control the generation of statistics (optional)
# Defaults are inferred from the dates section if not provided, using some heuristics

statistics:
  start: 2000-01-01 00:00:00
  end: 2000-01-10 18:00:00
  tendencies: [6h, 12h, 24h] # The list of tendencies to compute (e.g. 6h, 12h, 24h, etc.)

Layout-specific recipe keys

The top-level dates block depends on output.layout:

For layout: gridded (default) and layout: tabular — provide dates: {start, end, frequency, …}. base_dates: and steps: must be absent.
For layout: trajectories — provide base_dates: {...} and steps: {start, end, frequency}. dates: must be absent. See Trajectories for the full layout description.

The recipe loader validates these rules and raises a clear error if they are mismatched.

Deprecated / removed output keys

output.order_by is deprecated. The cube ordering is now hard-coded per layout (["valid_datetime", "param_level", "number"] for gridded, ["traj_point", "param_level", "number"] for trajectories). Existing recipes that set it to the default value keep parsing with a DeprecationWarning; any other value is rejected.
output.flatten_grid has been removed. Cube flattening is always on.
MARS request keys user_date and user_time are no longer accepted — use the wildcard shorthand date: "????-??-01" with an optional time: list instead. See mars.