Source Schema Reference

Complete YAML schema for source definitions -- every field, type, and default

This page documents every field in a source definition YAML file (sources.d/*.yaml). Fields are grouped by section. For transport-specific fields, see Transports.

Top-level

schema_version integer required
Schema version. Use 2 for new sources. Version 2 adds support for filtering, geo_cache, spatial crawling, entity_cache, on_demand_url, and advanced parser options. Allowed values: 1, 2.
name string required
Unique source identifier. Lowercase letters, digits, and underscores only. Must start with a letter. Max 64 characters. Pattern: ^[a-z][a-z0-9_]{0,63}$.
source_type string required
Maps to the domain SourceType for the ingest registry. Convention: same as name unless you have a reason to differ. Same naming rules as name.
layer_type string required
Groups entities on the globe. Multiple sources can feed one layer. Same naming rules as name.
display_name string required
Human-readable name for the SOURCE (shown in source listings, logs, and AI prompts). This names the source, not the layer-picker label.
layer_display_name string
Overrides the human-readable LAYER label shown in the UI layers panel. When omitted, the layer is labeled by title-casing layer_type (FormatLayerName, e.g. traffic_stations -> Traffic Stations). Set it when the title-cased layer_type loses meaning – for example a country qualifier: "Traffic Stations (Mexico)". Distinct from display_name, which names the source.
entity_type string default: geo_entity
Discriminates between geographic and non-geographic entities. geo_entity has lat/lon coordinates. global_indicator has no coordinates and is rendered as a HUD overlay panel. Allowed values: geo_entity, global_indicator.
enabled boolean default: true
Set to false to disable this source without deleting the file. When omitted or null, defaults to true.
dry_run boolean default: false
When true, the source fetches and parses data but does not persist to the database or cache. Useful for testing new sources.
labels map[string]string
Optional key-value labels for categorization and filtering. Common keys: category, priority.
filtering string
Controls how the frontend subscribes to this layer (schema_version 2 only). viewport sends camera bounds and uses GEOSEARCH. all explicitly fetches all entities. When omitted, the frontend subscribes in global mode. Allowed values: viewport, all.

Transport

transport.type string required
Transport type. Allowed values: http_poll, websocket, sse, mqtt, webhook, pubsub, grpc_stream, kafka, amqp, tcp_udp, ftp_sftp, s3_poll, nats.
transport.url string
The API endpoint URL. Supports ${VAR} for environment variable substitution (resolved from RESPONDENT_VAR). For spatial sources, use {lat} and {lon} placeholders.
transport.on_demand_url string
Alternative URL for single-point fetches (schema_version 2 only). Supports {lat} and {lon} placeholders. Must be a valid URL.
transport.method string default: GET
HTTP method. Allowed values: GET, POST.
transport.headers map[string]string
Static headers sent with every request.
transport.timeout duration
HTTP request timeout (e.g., "15s").
transport.interval duration
Polling interval – how often to re-fetch data (e.g., "60s").
transport.max_response_bytes integer
Maximum response body size in bytes. Min: 1, max: 104857600 (100 MB).

Auth

transport.auth.type string required
Authentication type. Allowed values: bearer, api_key, oauth2.
transport.auth.header string
Custom header name for api_key auth type.
transport.auth.env_var string
Environment variable name (without the RESPONDENT_ prefix). Resolved from RESPONDENT_<env_var> at runtime. Required for bearer and api_key types.
transport.auth.oauth2 object
OAuth2 token exchange configuration. Required when type is oauth2. See the Transports page for full OAuth2 field reference.

Retry

transport.retry.max_attempts integer
Number of retry attempts. Range: 0-10. Total attempts = max_attempts + 1.
transport.retry.backoff string
Backoff strategy. Allowed values: exponential, fixed.
transport.retry.initial_delay duration
Starting delay between retries (e.g., "2s").
transport.retry.max_delay duration
Maximum delay between retries (e.g., "30s").

Pagination

transport.pagination.type string required
Pagination strategy. Allowed values: page_number, offset, cursor.
transport.pagination.page_param string required
Query parameter name for the page number, offset, or cursor.
transport.pagination.size_param string
Query parameter name for page size or limit.
transport.pagination.size integer required
Items per page. Range: 1-10000.
transport.pagination.max_pages integer required
Stop after this many pages. Range: 1-1000.
transport.pagination.stop_when string
CEL expression evaluated per page. The records variable is bound to the current page array. When true, pagination stops early.
transport.pagination.cursor_path string
Dot-path to extract the next cursor value from the JSON response (for cursor type).

Parser

parser.format string required
Response format. Allowed values: json, json_table, geojson, csv, tle, xml, rss.
parser.records_path string
Dot-separated path to the array of records in the response. Max depth: 8 segments. Pattern: ^[a-zA-Z0-9_]+(\.[a-zA-Z0-9_]+){0,7}$. Omit when the response is already the array.
parser.max_records integer
Safety cap on the number of records to process. Range: 1-100000.

CSV Options

parser.csv_options.delimiter string default: ,
Field delimiter character.
parser.csv_options.has_header boolean default: false
Whether the first row contains column headers.
parser.csv_options.skip_lines integer default: 0
Number of lines to skip before parsing.
parser.csv_options.collapse_whitespace boolean default: false
Collapse consecutive whitespace in field values.
parser.csv_options.comment_prefix string
Lines starting with this string are ignored.

JSON Reshaping (schema_version 2)

parser.array_columns string[]
Map array indices to named fields. Use when the API returns arrays of arrays instead of objects (e.g., OpenSky returns [icao24, callsign, ...]). The Nth entry becomes the field name for index N.
parser.object_to_records boolean default: false
Convert a top-level JSON object {"key": {...}, ...} into an array of its values [{...}, ...]. Useful for dictionary-keyed APIs (e.g., SondeHub).
parser.object_key_field string
When object_to_records is true, inject the object key into each record under this field name.
parser.array_of_arrays boolean default: false
Treat a JSON array-of-arrays as a table where the first row is column headers and subsequent rows are data. Converts to an array of objects. Useful for APIs like NOAA SWPC.

Filter

filter string
CEL expression that must return true for a record to be processed. The record variable is bound to each parsed record. Use has() to guard optional fields. See CEL Functions for available functions.

Entity

entity.external_id string (CEL) required
CEL expression returning a unique identifier for this entity within the layer. Must be stable across polls. Examples: record.id, string(record.eventid), record.lat + "_" + record.lon.
entity.name string (CEL) required
CEL expression returning a human-readable name for display in the UI.
entity.metadata map[string]string (CEL)
Map of metadata key to CEL expression. Each expression must return a string. These appear in the entity detail panel.

Observation

observation.latitude string (CEL) required
CEL expression returning latitude as a double. Required for geo_entity, omit for global_indicator.
observation.longitude string (CEL) required
CEL expression returning longitude as a double. Required for geo_entity, omit for global_indicator.
observation.altitude string (CEL) default: 0.0
CEL expression returning altitude in meters as a double.
observation.timestamp string (CEL) required
CEL expression returning a CEL timestamp. See time functions in CEL Functions.
observation.event_time string (CEL)
CEL expression returning a timestamp for the start of a time-bounded event (e.g., weather alerts, conflict events).
observation.event_end string (CEL)
CEL expression returning a timestamp for the end of a time-bounded event.
observation.velocity map[string]string (CEL)
Map of velocity component to CEL expression. Each expression must return a string. Common keys: speed, heading, climb.
observation.metadata map[string]string (CEL)
Map of metadata key to CEL expression. Each expression must return a string.
observation.content_hash string (CEL)
CEL expression returning a unique string for deduplication. Required when recording.mode is dedupe.

Recording

recording.mode string required

How observations are persisted. Allowed values:

  • append – always insert a new row (time-series style)
  • upsert – insert or update based on entity_id (latest-only)
  • dedupe – insert only if content_hash differs from the latest observation (requires observation.content_hash)

Cache

cache.ttl duration required
TTL for the hot cache (Valkey in production, MemCache in community edition). Controls how long entities remain visible on the globe after the last fetch. Set higher than transport.interval to survive missed polls.

Entity Cache (schema_version 2)

Accumulates entities across spatial batches. Different from cache.ttl – this controls in-memory accumulation for cross-batch deduplication.

entity_cache.enabled boolean default: false
Enable entity-level caching for spatial sources.
entity_cache.key string (CEL) required
CEL expression for the cache key.
entity_cache.ttl duration required
How long to keep an entity in the accumulation cache.
entity_cache.accumulate boolean default: false
Merge entities across spatial batches.

Geo Cache

Controls geo sorted set maintenance for viewport-filtered layers. Required alongside filtering: viewport.

geo_cache.overfetch_ratio float
Fetch this multiple of members to find enough alive ones. Range: 1.0-10.0.
geo_cache.alive_ratio_threshold float
Trigger GC when fewer than this ratio of results are alive. Range: 0.1-1.0.
geo_cache.gc_batch_size integer
Max dead members to prune per query. Range: 10-10000.

Backfill (schema_version 2)

backfill.threshold integer
Trigger backfill when a spatial region has fewer entities than this threshold. Range: 1-10000.

Display

Icon

display.icon.shape string required
Icon shape for entity rendering. See Icon Shapes for the full list of 57 available shapes.
display.icon.rotatable boolean default: false
Rotate the icon based on entity heading. Enable for directional entities like aircraft and ships.
display.icon.interpolation boolean default: false
Interpolate position between updates for smooth movement.
display.icon.scale float default: 1.0
Size multiplier. Range: 0.1-5.0. 1.0 is the standard size.

Trail

display.trail.color string required
Hex color for the entity trail (e.g., "#ffffff"). Must be a valid hex color.
display.trail.width float
Trail line width. Range: 0.5-10.0.
display.trail.opacity float
Trail transparency. Range: 0.0-1.0.

Style

display.style.color string required
Entity color on the globe (e.g., "#ffffff"). Must be a valid hex color.
display.style.point_size integer required
Base point size. Range: 1-32. 8 is the standard size.

Field Renderers

An array defining how metadata fields are displayed in the entity detail panel. Renderers are sorted by priority (lower = shown first).

display.field_renderers[].keys string[] required
Metadata key(s) to match. Min length: 1. Matches keys from entity.metadata or observation.metadata.
display.field_renderers[].label string required
Display label shown in the entity detail panel.
display.field_renderers[].format.type string required
Format type. Allowed values: string, float, integer, raw.
display.field_renderers[].format.precision integer
Decimal places for float format type.
display.field_renderers[].format.prefix string
String prepended to the formatted value.
display.field_renderers[].format.suffix string
String appended to the formatted value.
display.field_renderers[].format.transform string
Text transform for string format type. Allowed values: upper, lower.
display.field_renderers[].priority integer default: 0
Sort order. Lower values are shown first.

Color By

Colors entities by the value of a metadata field. Overrides display.style.color per entity based on the matched value. When present, the client reads this map instead of applying a single base color to every entity.

display.color_by.field string required
Metadata field name whose value selects the color.
display.color_by.values map[string]string required
Map of field value to hex color (e.g., operational: "#00ff9d"). Min 1 entry. Each value must be a valid hex color.
display.color_by.default_color string
Hex color used when the field value is not present in values. Must be a valid hex color.

History

Per-layer time range limits for historical exploration. When present, overrides the server defaults (48h lookback, 24h span).

history.max_lookback duration
Maximum historical lookback (e.g., "8760h" for 1 year).
history.max_range_span duration
Maximum time range span per query (e.g., "168h" for 1 week).

Indicator

Maps a global indicator entity’s metadata fields to structured indicator values for HUD rendering. Only meaningful when entity_type is global_indicator.

indicator.summary_expr string (CEL)
CEL expression that computes a human-readable summary. Variables: level (int), metadata (map[string]string). When omitted, a default severity scale (Quiet -> Extreme) is used.

Values (array)

indicator.values[].key string
Identifier for this indicator reading.
indicator.values[].label string
Human-readable label for the reading.
indicator.values[].source_field string
Metadata key holding the raw value.
indicator.values[].unit string
Unit of measurement displayed with the value.
indicator.values[].max_level integer
Maximum severity level on the scale.
indicator.values[].level_thresholds float[]
Ascending thresholds used to compute the severity level from a raw value (when level_expr is omitted).
indicator.values[].level_expr string (CEL)
CEL expression that computes the severity level from a raw value. Variables: value (string), change_pct (string). Must return an int. When omitted, the level is computed from level_thresholds/max_level.
indicator.values[].change_source_field string
Metadata key holding the percentage change.
indicator.values[].format string
Value display format. Allowed values: scale, number, currency, percent.
indicator.values[].precision integer
Decimal places for the displayed value.
indicator.values[].prefix string
String prepended to the value (e.g., "$").

Field Mappings

A top-level array of generalized CEL mappings from source data to domain structures.

field_mappings[].source string (CEL) required
CEL expression producing the value from the source record.
field_mappings[].target string required
Destination field for the mapped value.
field_mappings[].type string required
Value type. Allowed values: timestamp, string, integer, float, boolean.

Lookup Tables

An array of static lookup tables for CEL enrichment. Loaded and indexed at config time.

lookup_tables[].name string required
Unique table identifier. Same naming rules as source name.
lookup_tables[].key_field string required
Primary key field name in each entry.
lookup_tables[].entries array
Inline entries. Each entry is a map with arbitrary fields. Use either entries or file, not both.
lookup_tables[].file string
Path to an external data file (relative to working directory). Use either entries or file, not both.
lookup_tables[].format string
File format when using file. Allowed values: json, csv.

AI

AI enrichment configuration. Optional. Requires ai.enabled: true in the server config and a configured LLM provider.

ai.enabled boolean default: false
Enable AI operations for this source.

Operations (array)

ai.operations[].name string required
Unique operation name.
ai.operations[].tags string[]
Optional tags for filtering and grouping.
ai.operations[].filter string
CEL expression to select which entities are processed.
ai.operations[].prompt string required
LLM prompt template. Supports Go template syntax with .Entity and .Observation variables.
ai.operations[].output_schema object
JSON Schema for structured LLM output.
ai.operations[].output_mapping map[string]string
Map LLM result fields to entity or observation metadata keys.
ai.operations[].output_target string default: entity
Where results are stored. Allowed values: entity, observation.
ai.operations[].max_tokens integer
Maximum tokens for the LLM response. Range: 1-128000.
ai.operations[].temperature float
LLM temperature. Range: 0.0-2.0.
ai.operations[].cache_ttl string
Cache LLM results for this duration to avoid re-processing (e.g., "24h").
ai.operations[].priority integer default: 0
Operation priority. Lower values are processed first.

Operation Output

ai.operations[].output.store_insights boolean default: false
Store results as AI insights.
ai.operations[].output.insight_type string
Insight type identifier for querying.
ai.operations[].output.websocket_push boolean default: false
Push results to the frontend via WebSocket.
ai.operations[].output.retention string
Insight retention period (e.g., "720h").
ai.operations[].output.results_path string
Dot-path to the results in the LLM response.
Edit this page on GitHub