Field Mapping

Map parsed records to entities and observations using CEL expressions

After parsing, each record is a flat or nested map of fields. Field mappings use CEL (Common Expression Language) expressions to transform records into Respondent entities and observations. Every expression receives the current record as the record variable.

Entity mapping

The entity section defines how each record becomes a domain entity. Entity mappings produce the identity and metadata for each data point.

entity:
  external_id: >
    record.id
  name: >
    record.properties.name
  metadata:
    category: >
      has(record.category) ? record.category : "unknown"

Fields

external_id CEL -> string required
Unique identifier within this layer. Must be stable across polls – the same entity must produce the same external_id every time it is fetched. This is the key used for upsert and dedupe recording modes.
name CEL -> string required
Human-readable name displayed in the UI entity detail panel.
metadata map[string]CEL -> string
Key-value pairs of additional entity data. All values must be CEL expressions that return strings. These appear in the entity detail panel and can be formatted by display.field_renderers.
⚠ Metadata values must return strings
Every metadata value must be a CEL expression that evaluates to a string. Use string() to convert numeric fields: string(record.magnitude). Returning a non-string type causes a runtime error.

External ID patterns

The external_id must be unique and deterministic. Choose a pattern based on your data:

Expression record.id
Input record.id = "eq-2024-abc123"
Output "eq-2024-abc123"
Direct field access – use when the API provides a unique ID field.
Expression string(record.eventid)
Input record.eventid = 12345
Output "12345"
Type coercion – convert numeric IDs to strings with string().
Expression record.lat + "_" + record.lon + "_" + record.date
Input record.lat = "39.7", record.lon = "-104.9", record.date = "2024-01-15"
Output "39.7_-104.9_2024-01-15"
Composite key – combine multiple fields when no single unique ID exists. Ensure the combination is stable across polls.

Observation mapping

The observation section maps each record to a spatial observation – a position-and-time snapshot attached to an entity.

observation:
  latitude: >
    double(record.lat)
  longitude: >
    double(record.lon)
  altitude: "0.0"
  timestamp: >
    unix_ms(record.time)
  velocity:
    speed: >
      has(record.speed) ? double(record.speed) : 0.0
    heading: >
      has(record.heading) ? double(record.heading) : 0.0
  metadata:
    status: >
      has(record.status) ? record.status : ""
  content_hash: ""

Position fields

latitude CEL -> double required
Latitude in decimal degrees. Must evaluate to a float64. Omit when entity_type is global_indicator.
longitude CEL -> double required
Longitude in decimal degrees. Must evaluate to a float64. Omit when entity_type is global_indicator.
altitude CEL -> double default: 0.0
Altitude in meters above sea level. Use negative values for underground or underwater entities.

Latitude and longitude patterns

Expression double(record.lat)
Input record.lat = "39.7392"
Output 39.7392
String-to-number conversion – use double() when the API returns coordinates as strings.
Expression record.geometry.coordinates[1]
Input record.geometry.coordinates = [-104.99, 39.74, 1609.0]
Output 39.74
GeoJSON array access – index [0] is longitude, [1] is latitude, [2] is altitude.
Expression record.position.latitude
Input record.position = {"latitude": 39.74, "longitude": -104.99}
Output 39.74
Nested object access – traverse nested structures with dot notation.

Timestamp field

timestamp CEL -> timestamp required
Timestamp of the observation. Must evaluate to a CEL timestamp value.

Available timestamp functions:

FunctionInputDescription
now()noneCurrent time (use as fallback only)
unix_ms(expr)integerEpoch milliseconds
unix_s(expr)integerEpoch seconds
parse_rfc3339(expr)stringRFC 3339 format: "2024-01-15T00:00:00Z"
parse_iso8601(expr)stringISO 8601 format (appends Z if no timezone)
parse_rfc2822(expr)stringRFC 2822 format (RSS feeds)
parse_datetime(expr, layout)string, stringGo time layout (e.g., "2006-01-02 15:04:05")
timestamp(expr)stringParse standard timestamp string
Expression unix_ms(record.time)
Input record.time = 1705276800000
Output 2024-01-15T00:00:00Z
Epoch milliseconds – common in APIs that return integer timestamps.
Expression parse_rfc3339(record.date)
Input record.date = "2024-01-15T12:30:00Z"
Output 2024-01-15T12:30:00Z
RFC 3339 – standard ISO timestamp format with timezone.
Expression has(record.pubDate) ? parse_rfc2822(record.pubDate) : now()
Input record.pubDate = "Mon, 15 Jan 2024 12:30:00 GMT"
Output 2024-01-15T12:30:00Z
Guarded fallback – parse the date if present, otherwise fall back to current time.

Event time fields

For time-bounded events (weather alerts, conflict events), you can specify a start and end time:

event_time CEL -> timestamp
Event start time. Must evaluate to a CEL timestamp.
event_end CEL -> timestamp
Event end time. Must evaluate to a CEL timestamp.
observation:
  timestamp: >
    parse_rfc3339(record.updated)
  event_time: >
    parse_rfc3339(record.start_time)
  event_end: >
    parse_rfc3339(record.end_time)

Velocity fields

velocity map[string]CEL -> number
Velocity components for moving entities. Keys are arbitrary names (commonly speed, heading, climb). Each value is a CEL expression that must evaluate to a number – it is stored as map[string]float64 via numeric evaluation. A component whose expression returns a non-numeric value (or errors) is silently dropped from the velocity map.
velocity:
  speed: >
    has(record.speed) ? double(record.speed) : 0.0
  heading: >
    has(record.heading) ? double(record.heading) : 0.0
  climb: >
    has(record.vertical_rate) ? double(record.vertical_rate) : 0.0
ⓘ Velocity enables smooth interpolation
When velocity data is provided along with display.icon.interpolation: true, the frontend interpolates entity positions between poll updates for smooth movement on the globe.

Observation metadata

observation.metadata map[string]CEL -> string
Per-observation metadata. Same rules as entity metadata – all values must return strings.

Content hash

content_hash CEL -> string
Hash string for deduplication. Required when recording.mode is dedupe. Should produce a unique string for each distinct observation state. A new observation is only recorded when the content hash changes.
# Dedupe by article URL -- only re-record if the URL changes
content_hash: >
  record.link

# Dedupe by composite state -- re-record if position or brightness changes
content_hash: >
  record.lat + "_" + record.lon + "_" + record.brightness

Common CEL patterns

Optional field guards

Always guard access to fields that may not exist in every record. Without has(), accessing a missing field causes a runtime error that silently skips the record.

Expression has(record.field) ? record.field : "default"
Input record = {}
Output "default"
Conditional access with fallback – returns the field value if present, otherwise a default.

Type coercion

Expression string(record.mag)
Input record.mag = 4.2
Output "4.2"
Number to string – required for all metadata values.
Expression double(record.latitude)
Input record.latitude = "39.7392"
Output 39.7392
String to double – use for coordinate fields that arrive as strings.
Expression int(record.count)
Input record.count = "42"
Output 42
String to integer – use for integer fields that arrive as strings.

Safe numeric conversion

Expression coerce_double(record.value, 0.0)
Input record.value = "not_a_number"
Output 0.0
Safe double conversion – returns the fallback value if the input cannot be parsed as a double.

Lookup tables

When you have static reference data (country codes, station metadata), define a lookup table and query it in CEL expressions:

lookup_tables:
  - name: country_codes
    key_field: code
    entries:
      - code: US
        name: "United States"
      - code: GB
        name: "United Kingdom"

entity:
  metadata:
    country_name: >
      lookup("country_codes", record.country, "name")
Expression lookup("country_codes", record.country, "name")
Input record.country = "US"
Output "United States"
Lookup table query – retrieves a field from a named lookup table by key. Returns dyn (the entry value’s native type), or a CEL error if the table, key, or field is not found.

The lookup family also includes a presence check and a default-aware variant:

FunctionReturnsDescription
lookup(table, key, field)dynRetrieve a field from a lookup table entry. Errors if the table, key, or field is missing.
has_lookup(table, key)boolReport whether key exists in the named lookup table. Returns false if the table is unknown.
lookup_or(table, key, field, default)dynLike lookup, but returns default instead of erroring when the table, key, or field is missing.
Expression has_lookup("country_codes", record.country)
Input record.country = "US"
Output true
Presence check – safe to call before lookup when a key may be absent.
Expression lookup_or("country_codes", record.country, "name", "unknown")
Input record.country = "ZZ"
Output "unknown"
Default-aware lookup – returns the supplied default rather than erroring on a missing key.

TLE orbital functions

For satellite TLE data, use SGP4 propagation functions. Each propagates the TLE to the current time. All four return 0.0 on a malformed or invalid TLE.

CEL FunctionReturnsDescription
sgp4_lat(line1, line2)doubleCurrent latitude in degrees
sgp4_lon(line1, line2)doubleCurrent longitude in degrees
sgp4_alt_m(line1, line2)doubleCurrent altitude in meters
sgp4_vel_mps(line1, line2)doubleCurrent velocity in meters per second
Expression sgp4_lat(record.line1, record.line2)
Input record.line1 = "1 25544U ..."
Output 42.35
Propagate the TLE to the current time and return the satellite’s latitude in degrees.
Edit this page on GitHub