Operations

Operations are blocks of YAML code that translates a list of dates into fields.

join

The join is the process of combining several sources data. Each source is expected to provide different variables at the same dates.

input:
  join:
    - source1:
        key: date
    - source2
    - ...

concat

The concatenation is the process of combining different sets of operation that handle different dates. This is typically used to build a dataset that spans several years, when the several sources are involved, each providing a different period.

input:
  concat:
  - dates:
      start: 2020-12-30 00:00:00
      end: 2021-01-01 12:00:00
      frequency: 12h

    source1:
    - args

  - dates:
      start: 2021-01-02 00:00:00
      end: 2021-01-03 12:00:00
      frequency: 12h

    source2:
    - args

pipe

The pipe is the process of transforming fields using filters. The first step of a pipe is typically a source, a join or another pipe. The following steps are filters.

input:
  pipe:
    - source
    - filter1
    - filter2
    - ...