Recording Modes

Control how entities are persisted — append, upsert, or deduplicate

The recording mode controls how observations are persisted when the feeder processes new data. Choose the mode based on how your source data changes over time.

recording:
  mode: upsert

Append

Every poll inserts a new observation row, building a time-series history for each entity.

recording:
  mode: append

Use append when:

• Entity positions change frequently (satellite tracks, ship voyages, flight paths)
• You need historical trajectory data
• You want to replay entity movement over time

Each poll creates a new row even if nothing changed. The observation history grows with every poll cycle.

Example use cases: satellites, aircraft, ships, weather balloons, GPS trackers.

Upsert

Each poll inserts a new observation or updates an existing row on a conflict against the composite key (entity_id, ts) (ON CONFLICT (entity_id, ts) DO UPDATE). When an entity’s observation timestamp stays the same across polls, the existing row is overwritten in place. When the observation timestamp advances between polls, the upsert inserts a new row – so for an entity whose timestamp changes every poll, upsert behaves like append for that entity.

recording:
  mode: upsert

Use upsert when:

• You only care about the current state, not historical positions
• Entities are relatively static (their position rarely changes)
• The observation timestamp is stable across polls so updates land on the same row
• You want to minimize storage

New data overwrites the previous observation only when the (entity_id, ts) pair matches; a changed timestamp produces an additional row rather than replacing the prior one.

Example use cases: earthquakes, weather stations, sensor readings, infrastructure status.

Dedupe

Inserts a new observation only when the content_hash changes from the last recorded value. Requires observation.content_hash to be set.

recording:
  mode: dedupe

# content_hash must be defined in observation:
observation:
  content_hash: >
    record.link

Use dedupe when:

• The source re-publishes the same content across polls
• You want to record changes but avoid duplicates
• Data arrives as a feed with overlapping entries (news articles, alerts)

The content hash is compared against the last recorded hash for each entity. If they match, no new row is written.

Example use cases: news feeds, alert bulletins, status pages, event logs.

Cache TTL

The cache section controls how long entities remain visible on the globe after the last successful fetch.

cache:
  ttl: "3600s"

Choosing a recording mode