Activations

The activations module provides custom activation layers used throughout the network.

Usage

These activation layers can be used through the layer kernels configuration system. For example, to use the Sine activation function:

layer_kernels:
  Activation:
    _target_: anemoi.models.layers.activations.Sine

For gated variants (GLU, SwiGLU, GEGLU, ReGLU), use mlp_implementation instead of layer_kernels.Activation:

processor:
  mlp_implementation: swiglu  # options: glu, swiglu, geglu, reglu
  mlp_hidden_ratio: 2.67  # recommended ratio for gated variants

Available Layers

anemoi.models.layers.activations.leaky_hardtanh(input: Tensor, min_val: float = -1.0, max_val: float = 1.0, negative_slope: float = 0.01, positive_slope: float = 0.01) → Tensor

Leaky version of hardtanh where regions outside [min_val, max_val] have small non-zero slopes.

Parameters:

  • input – Input tensor

  • min_val – Minimum value for the hardtanh region

  • max_val – Maximum value for the hardtanh region

  • negative_slope – Slope for values below min_val

  • positive_slope – Slope for values above max_val

Returns:

Tensor with leaky hardtanh applied

class anemoi.models.layers.activations.CustomRelu(threshold: float = 0.0)

Bases: Module

Custom ReLU activation function with a specified threshold.

forward(x: Tensor) → Tensor

Apply the ReLU activation with the specified threshold.

Parameters:

x (torch.Tensor) – The input tensor to process.

Returns:

The processed tensor with ReLU applied.

Return type:

torch.Tensor

class anemoi.models.layers.activations.Sine(w=1, phi=0)

Bases: Module

Sine activation function.

Periodic activation function defined as:

Equation: Sine(x, w, phi) = sin(w x + phi)

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.