Inference using an external graph
Anemoi is a framework for building and running machine learning models based on graph neural networks (GNNs). One of the key features of such GNNS is that they can operate on arbitrary graphs. In particular it means one can train the model on one graph, but use it in inference on another graph. This way one can transfer the model to a different domain or dataset, without any fine tuning, or even change the scope of a model. For example using a model trained as a stretched grid as a limited area model (LAM) with boundary forcings in inference.
We should caution that such transfer of the model from one graph to another is not guaranteed to lead to good results. Still, it is a powerful tool to explore generalizability of the model or to test performance before starting fine tuning through transfer learning.
The ability to do inference with an alternative graph, or more precisely
one ‘external’ to the checkpoint created in training, is supported by
anemoi-inference through the external_graph runner.
This runner, and the graph it will use, can be specified in the config file as follows:
runner:
external_graph:
graph: path/to/graph.pt
In case one wants to run a model trained on a global dataset on a graph
supported only on a limited area one needs to specify the
output_mask to be used. This mask selects the region on which the
model will forecast and triggers boundary forcings to be applied when
forecasting autoregressively towards later lead times. As in training,
also in inference the output mask orginates from an attribute of the
output nodes of the graph. It can be specified in the config file as
follows:
runner:
external_graph:
graph: path/to/graph.pt
output_mask:
nodes_name: data # name of the output nodes of the graph
attribute_name: cutout_mask # mask specifying the limited area among the output nodes
For LAM models the limited area among the input nodes of a larger
dataset is often specified by the indices_connected_nodes attribute
of the input nodes. Anemoi-inference will automatically update the
dataloader to load only data in the limited area in case the external
graph contains this attribute and was build using the same dataset as
the one in the checkpoint.
In case one wants to work with a graph that was built on another dataset than that used in training, on should specify this in the config file as well:
runner:
external_graph:
graph: path/to/graph.pt
graph_dataset: path/to/graph_dataset.zarr
# the above can be an anemoi-datasets.open_dataset argument as well,
# rather than simply a path
If you wish to run the external graph runner, but without the
anemoi-dataset configuration, some supporting arrays may need to be
updated. These can be sourced from either the graph['data'] or from
a file on disk. This is done with the update_supporting_arrays key
runner:
external_graph:
graph: path/to/graph.pt
update_supporting_arrays:
graph: # Get data from the graph
'global/cutout_mask': 'cutout_mask' # Change `global/cutout_mask` in the arrays to `cutout_mask`
file:
'global/cutout_mask': 'cutout_mask.npy' # Change `global/cutout_mask` in the arrays to `cutout_mask.npy`
It should be emphasized that by using this runner the model will be rebuilt and for this reason will differ from the model stored in the checkpoint. To avoid unexpected results, there is a default check that ensures the model used in inference has the same weights, biases and normalizer values as that stored in the checkpoint. In case of a more adventurous use-case this check can be disabled through the config as:
runner:
external_graph:
graph: path/to/graph.pt
check_state_dict: False
Warning
If you train a model on a graph using a version before 0.8 and then run inference or finetuning with anemoi-graphs version 0.8 or later, results may be inconsistent or incorrect. Previously edge directions were wrong by 90 degrees. Ensure that the same version of anemoi-graphs is used for both training and inference/finetuning to avoid incompatibility
This issue arises because changing the graph can shift the data distribution of edge attributes, which are used as inputs to the model.
Similarly, if the graph is created in a different domain or dataset, the statistics of these edge attributes may differ too, potentially causing a drift in the input data distribution and leading to degraded model performance. Therefore, it is important to be cautious with the normalisation of edge attributes when working with external graphs.
See also
Run Command - Run command documentation
Configuration - Configuration file reference
Parallel Inference - Parallel inference for large models
Environment Setup - Environment setup and version compatibility