Explore (/explore) — slice coverage probe and cold ingest

This page owns the /explore surface: URL → slice, coverage check against Postgres, branching to charts vs. cold ETL, orchestration via fire_data.ingest_runs, and status polling. It does not duplicate ETL contracts, full DDL, or generic read path prose — use the links below.

Related documentation

Probe filter args and operator guardrails

• Empty integer[] vs NULL: fire_data.probe_slice_coverage treats a non-null empty filter array as an impossible predicate (see migration 20260503210000_probe_slice_coverage_filter_code_arrays.sql). Explore therefore normalizes [] → null for all *_codes arrays before compute_slice_hash, probe_slice_coverage / _daily, and ingest_runs.slice_config (implementation: app/(main)/explore/_lib/slice-filter-probe-args.ts, applied from resolveSliceFromParams and defensively in probe RPC wrappers).

• Probe vs chart (aligned on NULL-as-ALL): Yearly coverage uses the count rule (value > 0) on matching fact rows. Both probe_slice_coverage* and get_fire_data treat an omitted / NULL axis as «only facts where that axis FK is NULL» (the all-level total), per migration 20260601120000_get_fire_data_null_as_all_breakdown_aware.sql — which superseded the earlier 20260515212522…align_alle… that had instead aligned the probe to the buggy "no predicate" read path. An «Avvik mellom dekning og lesing» banner remains as a safety net for any residual drift.

• Optional ingest cooldown: Innstillinger → API can set a cookie-backed minimum time between Utforsk-triggered ingests for the same slice_hash (explore_ingest_cooldown). When enabled and the window has not expired, triggerSliceIngest returns covered without enqueuing a new run (service-role peek at ingest_runs). This is a guardrail, not a substitute for fixing probe/ETL bugs — see ../ingestion/pipeline-slices-idempotency-and-triggers.md for durable idempotency (rows_* columns, tiering).

High-level flow

A probe is a small, targeted call that answers: for this slice (source, dimension, filter IDs, year/date window, BRASK metric, and read grain), are there usable rows in the same fact table get_fire_data will read, and is there a completed ingest_runs row for the same slice_hash? It is neither full chart SQL nor ETL.

Critical: p_granularity = year reads fact_yearly only; any sub-year value reads fact_daily only (see read-path doc and migration 20260506210000_get_fire_data_branch_granularity_sources_supports_subyear.sql). The Explore probe must use the matching RPC or coverage and cold ingest disagree: yearly rows would satisfy probe_slice_coverage while get_fire_data with granularity=day still returns zero rows (the “Ingen rader etter filtrering” silent mismatch).

flowchart TB
  subgraph rsc [Server Component]
    B{BRASK and mode=native?}
    B -->|yes| PT[getCachedBraskPassthrough + BraskPassthroughCharts]
    B -->|no| N{native-only dim?}
    N -->|yes| Live[getCachedNativeFireData + charts]
    N -->|no| G{granularity year?}
    G -->|yes| Py[probeSliceCoverage → fact_yearly]
    G -->|no| Pd[probeSliceCoverageDaily → fact_daily]
    Py --> W{windowFullyCovered?}
    Pd --> W
    W -->|yes| C[getCachedFireData + charts]
    W -->|partial heal or daily gap| H[CoveredBranch plus ExploreClient when signed in]
    W -->|missing signed in| X[ExploreClient cold path]
    W -->|missing anonymous| L[ExploreLoginPrompt]
  end
  H --> A
  X --> A[triggerSliceIngest Server Action]
  A -->|after| I["POST /api/admin/ingest grain yearly or daily"]
  I --> U[Service role upsert + ingest_runs]
  X --> S["GET /api/ingest/status/:runId polling"]
  S -->|complete| R[router.refresh]

The diagram is simplified — it omits the yearly silent-mismatch safety net, the SSB «Ingen landstall…» retry card, and BRASK kr notices. Exact predicates live in § Branching after the probe and ExploreSection in page.tsx.

BRASK passthrough (mode=native): user-toggled live Utforsk crosstab — no probe, no ingest. URL keys: pt_row, pt_col, pt_<dim> filters, plus existing metric (count | kr). Console toggle: PassthroughToggle. Charts: BraskPassthroughCharts via shared SeriesChartCard (line / area / bar / table). Standard ingested path unchanged when passthrough is off; kr requires value_kr rows in fact_* (separate ingest pass from antall).

Shipped implementation choice: cold ingest goes through a Server Action that, after enqueueing, calls the POST /api/admin/ingest route handler (same path as admin/cron), not long synchronous ETL inside the action response itself. That shares idempotency, runId, and after() / platform background work.

Code map

URL query string (current)

Each filter axis (fire_type, building_type, region, building_age) is an optional comma-separated list:

• Preferred: numeric filters.id values, e.g. fire_type=128 or fire_type=128,130.
• Legacy bookmarks: standard_code tokens expand to all matching rows on that axis for the current source + chart dimension, e.g. fire_type=DEVELOPED.

Other keys: source_id, dim_code, year_start, year_end, metric (count or kr). Optional: granularity (day | week | month | quarter | year; omitted or year → yearly branch). Optional: explore_debug=1 → server renders diagnostics with the resolved slice and raw probe JSON.

SSB Nøkkeltall (source_id=5, dim_code=7): optional cat_standard — semantic group standard_code for the Emne toolbar row (FIRE_AND_COMPENSATION, EMERGENCY_DISPATCH, CHIMNEY_AND_SWEEPING, MAN_YEARS, HIGH_RISK_OBJECTS, GROSS_OPEX, NET_OPEX). Omitted → FIRE_AND_COMPENSATION. Legacy nokkelgruppe=brann (etc.) still resolves. Cleared when leaving dim 7 or switching source. Drives which native-only cat_code rows NativeBranch passes to getCachedNativeFireData (see lib/data-model/ssb-nokkelgrupper.ts). No probe or cold ingest on this dimension.

BRIS Strømkilde (source_id=4, dim_code=8): second native-only dimension — same NativeBranch routing, no cat_standard (all categories load). fire_type and region filters work as on ingested DSB dims: the provider (lib/data-model/native/providers/source-bris-restricted.ts) fetches /restrictedaggregation per selected filter id and tags rows; no filter → one unfiltered fetch tagged NULL. Yearly only. No probe or cold ingest.

Default slice when params are missing — DEFAULT_PARAMS in page.tsx: Police (source_id=2), primary causal factor (dim_code=1), fire_type=3 (Brann i bygning — native row, not standard_code), years 2016–2025, metric=count. building_type has no default — an omitted URL param means unfiltered upstream slice (NULL p_building_type → facts where building_type_code IS NULL per target contract). Under NULL-as-ALL there is no "all buildings" filter token to select: «alle bygg» is the omitted axis. The upstream all-buildings request (BRASK Alle næringer / BRIS empty buildingTypes.ids) is ingested onto that same NULL slice.

fire_type default is fresh-visit only: paramsFromSearch applies DEFAULT_PARAMS.fire_type only when both source_id and fire_type are absent from the URL (first landing). After the user has interacted, a cleared fire_type param stays cleared — «Fjern alle» must not snap back to Brann i bygning (same pattern as building_type).

Sub-year display refinement: date_start / date_end

When the slider produces an intra-year window (e.g. Q2 2024 → Q3 2025) on a non-year granularity, both endpoints round-trip as optional YYYY-MM-DD URL params. They are consumed only by sliceParamsToFireDataQuery in app/(main)/explore/_lib/slice-params-bridge.ts, which prefers date_start / date_end over the default ${year_start}-01-01..${year_end}-12-31 window when both are set and granularity !== 'year'. slice_hash, slice_config, and ingested_year_window still key off the year rails (year_start / year_end); the grain: 'daily' ingest payloads additionally receive that date window for the actual upsert (see pipeline §10 and triggerSliceIngest in actions.ts). Year-only URLs keep working unchanged. The params are dropped on granularity=year (year window is the only intent there), clamped to the year span by normalizeParams, and omitted from the URL when the dates exactly match ${year_start}-01-01 / ${year_end}-12-31 (clean-URL invariant — see sliceParamsToQueryUpdates and ConsoleToSliceParams).

Slider rails and the dynamic upper bound

FireDataTimeRangeControl (components/fire-data/console/FireDataTimeRangeControl.tsx) follows the contract in components/fire-data/console/TIMEFRAME_UX.md: stepping finer sets the new outer bounds to the current selection translated into the finer grain (zoom in to the picked window); stepping coarser resets bounds to the full envelope for the new grain and clamps the selection into them (zoom out). Bounds are local state — prop-driven grain changes (deep-link, parent reset) re-derive them, but a self-triggered finer/coarser step keeps the bounds the component just set so the URL round-trip does not flicker the rails back to the full archive.

The slider's upper bound is dynamic, not a literal year. Today resolveSourceYearBounds.max = new Date().getUTCFullYear() for every source; if the catalog later exposes a per-source "latest year upstream actually publishes" (e.g. sources.year_max), use Math.min(sourceLatestYear, utcCalendarYear) through the same helper so the slider never paints past reality or the clock. Yearly cold ingest is unaffected — it stays unbounded by upstream per pipeline §10 (BRASK lbÅr='', BRIS full history). For the daily ingest path, triggerSliceIngest in actions.ts additionally clamps year_end to min(year_end, utcCalendarYear) before building the probe window and adapter slices (clampDailyYearEnd guardrail) so a future calendar year in the URL cannot fan out past today.

Coverage RPCs (yearly vs sub-year)

Explore picks one probe per request, aligned with ResolvedSlice.granularity and with get_fire_data.p_granularity (via sliceParamsToFireDataQuery in slice-params-bridge.ts).

fire_data.probe_slice_coverage (yearly branch)

When: granularity === 'year' (default).

Reads fire_data.fact_yearly for p_year_start–p_year_end. Filter axes are integer[] (p_fire_type_codes, …):

• NULL on an axis means only facts where that axis FK is NULL (aligned with get_fire_data — see breakdown-aware note below).
• Non-null array means column = ANY(array) (OR within the axis).
• Empty array (non-null, cardinality 0) remains an impossible predicate — Explore normalizes [] → null where appropriate before the RPC.

Definition in repo: introduced in 20260503210000_probe_slice_coverage_filter_code_arrays.sql; the live IS NULL-on-omitted-axis semantics are set by 20260601120000_get_fire_data_null_as_all_breakdown_aware.sql (see postgres-contract-rls-and-keys.md), which undid the interim 20260515212522…align_alle… that had matched the probe to the buggy "no predicate" read path.

For count, rows must have value > 0; for kr, value_kr IS NOT NULL (BRASK may have value = 0 on kr-only rows).

fire_data.probe_slice_coverage_daily (sub-year branch)

When: granularity is any of day, week, month, or quarter.

Definition in repo: 20260512010000_fact_daily_helpers_and_phase_detail.sql — function body carries the same NULL-as-ALL alignment as the yearly probe, set live by 20260601120000_get_fire_data_null_as_all_breakdown_aware.sql.

Reads fire_data.fact_daily with date between p_date_start and p_date_end. Explore passes the calendar window {year_start}-01-01 through {year_end}-12-31 (same span as the time slider). Axis and metric predicates mirror the yearly probe.

Cold ingest: triggerSliceIngest uses the same branch as ExploreSection, then POST /api/admin/ingest with grain: 'daily' and adapter slices that include date_start / date_end (see buildAdapterSlices in _lib/ingest-plan-slices.ts). Pipeline §10.3–§10.4 remain the canonical narrative for four-tier adapters; this page only documents the Explore wiring above.

App code builds RPC args in _lib/probe.ts (p_metric optional on the yearly probe for retry against an older DB). Both the Server Component and the Server Action call the matching probe before any ingest_runs INSERT.

Branching after the probe (ExploreSection)

Approximate logic (see page.tsx for exact conditions):

1. Fully covered → chart path: getCachedFireData plus any client-side narrowing in CoveredBranch. "Fully covered" depends on the branch:
   • Yearly: row_count > 0 and yearlyHealDue === false (isYearlyHealDue in probe.ts). Heal is not due when materialised min_year/max_year already contain the URL window (narrowing inside existing facts). With rows but a partial span, heal runs only while last_run_completed is null — once a run completes, an upstream gap (e.g. SSB before 2015) is accepted and the chart stands alone. With row_count === 0, heal runs only when last_run_completed is null (never-ingested slice). Yearly cold ingest still pulls the full upstream archive per slice — see pipeline §10.
   • Daily: row_count > 0 and probe_slice_coverage_daily.min_date/max_date lexicographically span [year_start-01-01, year_end-12-31]. Encoded by isDailyWindowFullyCovered in app/(main)/explore/_lib/probe.ts.
2. Partial coverage with heal / daily gap — row_count > 0 but not fully covered:
   • Yearly heal: same chart via CoveredBranch plus <ExploreClient /> when signed in (anonymous visitors see the partial chart only). The client shows a confirmation dialog before enqueueing ETL.
   • Daily: materialised date span narrower than the requested window. ExploreSection stacks chart + ingest gateway unless last_run_completed is set (upstream cannot fill the gap — chart only).
3. partial_axis (OR vs Cartesian gap): aggregate probe reports row_count > 0 (OR within an axis matches existing facts) but one or more URL Cartesian combinations from buildAdapterSlices still lack per-combo coverage (combo_present in findMissingAdapterSlices). Chart renders partial data; signed-in users get the confirmation dialog. Ingest options are not that Cartesian product: the dialog offers one trigger filter or a sibling-axis rollout (see below). Assessment lives in computeIngestPlan (_lib/ingest-plan.ts).
4. row_count === 0 (missing coverage):
   • Signed-in → <ExploreClient /> confirmation dialog, then cold ingest + polling.
   • Anonymous → <ExploreLoginPrompt /> ("Logg inn for å hente data") because ingest is auth-gated — this is an auth gate, not a verdict that upstream has no data.
   • Sub-case — completed run + zero probe rows (yearly): try yearlyReadHasMatchingRows first (silent probe/read mismatch → chart). Otherwise show an informational card and offer retry via the same confirmation gateway (signed-in) or login prompt (anon).

Explore chart read (p_breakdown_axes)

Utforsk charts series by category only — filter axes are WHERE clauses, not chart breakdown dimensions. CoveredBranch and yearlyReadHasMatchingRows call getCachedFireData with p_breakdown_axes: [] after resolveFireDataRequest, because the default resolver pushes every filtered axis into p_breakdown_axes and would return one row per (period × category × each filter value) (thousands of rows, PostgREST truncation risk). Server-side aggregation keeps one row per (period × category); client rowMatchesAxisFilters narrows to the resolved slice ids.

Cold ingest and progress polling

Confirmation gateway (before ETL)

When computeIngestPlan returns a plan, ExploreClient opens a Shadcn AlertDialog (ExploreIngestConfirmDialog) instead of auto-starting ingest. The user chooses:

1. Importer valgt filter — one adapter payload for the filter value that triggered ingest (explore:lastFilterChange in sessionStorage, mirrored to cookie for RSC; written by ConsoleShell on explore URL updates), or server-inferred trigger id as fallback.
2. Importer alle <plural akse> (N) — sequential ingest of every catalog option on that filter axis (buildAdapterSlicesForAxisRollout + filtersForAxis); other axes stay fixed at the current URL (not a Cartesian product of multi-selected filters). Hidden when N === 1. Button copy uses plural axis labels from EXPLORE_AXIS_LABEL_PLURAL_NO in _lib/ingest-plan-core.ts (e.g. «Importer alle bygningsaldre (6)»). Same behaviour as the legacy building-type leaf sequence.
3. Avbryt — router.push to explore:previousSearchParams (slice before the filter change).

Detection vs enqueue: computeIngestPlan may probe the URL Cartesian product (buildAdapterSlices) to decide whether ingest is needed and to infer the rollout axis; only the user's choice above determines which adapter payloads are passed to POST /api/admin/ingest (adapterSlices on triggerSliceIngest).

Reason copy (cold, yearly_heal, daily_gap, partial_axis, retry_zero_rows) is in _lib/explore-ingest-copy.ts. slice_config / slice_hash still reflect the full URL selection at ingest_runs INSERT time.

After confirmation

1. triggerSliceIngest (in actions.ts) verifies a signed-in user, re-runs the probe, handles concurrent visitors via the partial unique index on active slice_hash runs (see migrations for ingest_runs_active_slice), inserts or reuses run_id, and starts fetch to /api/admin/ingest inside after() with server-side x-ingest-key.
2. /api/admin/ingest updates fire_data.ingest_runs as it goes and calls revalidateTag per lib/data-model/queries.ts / the route handler (see the read-path document).
3. ExploreClient polls /api/ingest/status/[runId] every 1500 ms (constant in source), with a timeout. On complete: invalidate the ingest-monitor query and router.refresh() (not a full window.location reload).
4. Explore progress bar: Shadcn Progress is driven by computeTruthFloorPct in ingest-progress.ts: primarily phase_detail.canonical_progress when phase_detail.v === 1, otherwise slices_done / slices_total, plus a small visual floor during queue/startup.

Anonymous users can read covered slices; writes / ingest trigger require sign-in (returned as unauthorized to the client).

Sub-year branch (fact_daily)

When the URL has non-yearly granularity, charts read fact_daily (subject to sources.supports_subyear). Explore must use probe_slice_coverage_daily and must send grain: 'daily' with date_start / date_end on adapter slices so /api/admin/ingest dispatches to the daily adapters. Supported sources from Explore match the catalog: BRASK (source_id=1) and BRIS Police (source_id=2); others are blocked at ingest build time or upstream. Four-tier fan-out, phase_detail, auto-chain to yearly when needed, and undo_ingest_run routing are documented in pipeline §10.3–§10.4 — link there for adapter mechanics; update this page when Explore URL wiring, probe choice, or actions.ts dispatch changes.

ingest_runs and status JSON

The orchestration table lives in fire_data; the PostgREST client must use .schema('fire_data') before .from('ingest_runs') where relevant (lib/ingest/client.ts). Full column list, indexes, and security model: Postgres contract (linked at the top).

The status endpoint returns fields such as slices_total, slices_done, row counters, api_total / transformed_row_count when set, per_slice_metrics for batch runs, phase_detail, and error fields — see the route source and types/fire_data.database.types.ts.

slice_config.year_start / year_end vs slice_config.ingested_year_window

slice_config stores two separate year spans:

• year_start / year_end — requested intent. Written at INSERT from buildIngestSliceConfig (canonical slice JSON). These drive slice_hash, the appropriate coverage RPC (probe_slice_coverage or probe_slice_coverage_daily via the date window above), and the chart query. They reflect what the user asked for in the URL.
• ingested_year_window: { start, end } — actually stored. Written by the route handler (app/api/admin/ingest/route.ts) when the run finishes, aggregated from SliceResult.year_window_written across slices. For BRASK and BRIS yearly slices this is almost always wider than the requested intent (one POST pulls the full upstream archive — 1985+ for BRASK, 2016+ for BRIS). The Last imported list uses it to show “Requested 2024–2025 (stored 1985–2026)” when the two differ. Missing on rows from before this mechanism and when no slice produced a year-keyed window.

For coverage, level A (row_count > 0 over the requested window) is enough: a cold ingest of a yearly slice loads the full upstream span, so any later request for a sub-range of the same slice hits existing rows immediately. You do not need a per-year coverage map inside probe_slice_coverage to get that behavior.

Display granularity vs physical write grain

The Explore time-range slider walks five display granularities (year → quarter → month → week → day), but the dispatcher in actions.ts collapses everything sub-year to a single physical grain: const grain: IngestGrain = slice.granularity === 'year' ? 'yearly' : 'daily'. That means an ingest_runs row may carry slice_config.granularity = 'quarter' (the user's slider choice) while the facts live in fact_daily. The Explore monitor surfaces only the write grain (Dag / År) via writeGrainLabelFromSliceConfig in app/(main)/explore/_lib/ingest-progress.ts; it is the value rendered in the "Granularitet" column of "Sist importert" and in the "Lagringsgrain" row of the ingest-detaljer top stats. The original display label stays in slice_config.granularity_label for diagnostic JSON only. Pipeline-side rationale: see pipeline §10 ("Display granularity vs write grain").

Undo eligibility

undoIngestRun (in actions.ts) reverses any run with a non-empty ledger_events array — the same RPC handles partial ledgers from runs that errored mid-fan-out (each month's appendLedgerEvents runs after bulkUpsertFactDailyRpc succeeds, so committed rows are always represented). The UI gate in IngestRunDetailSheet.tsx is therefore (rows_added + rows_updated) > 0 AND status ∈ {complete, error} — not status === 'complete' only. A run that errored after writing 3 800 rows has the same undo affordance as a clean completion; a run that wrote zero rows (cold path short-circuit, total-equal skip, or pre-fetch failure) has no undo button because there is nothing to reverse.

Terminal phase_detail and the PhaseStrip

On status === 'complete' the route handler emits a final route_lifecycle payload with phase: 'finalize_metrics_cache' (95%) right before revalidateTag; the Status badge in the detail sheet renders the actual completion signal, so the PhaseStrip is suppressed for complete and reverted runs to avoid showing "Ferdig" twice. On error the route handler skips the lifecycle ceiling entirely, so the adapter's last brask_daily_fanout / bris_police_daily_fanout payload (with its in-progress canonical_progress and cursor: { year, month }) survives as the terminal phase_detail. The PhaseStrip renders that cursor — e.g. Henter dagdata for 2026-01 · 73% — so operators see where the run died without grepping server logs. The matching error_message carries the what (BRASK POST 500 (2026-01), surfaced by postBraskForm in lib/data-model/adapters/_braskWebForms.ts).

Maintenance

• RPC signature or probe semantics change: update migrations under supabase/migrations/, types/fire_data.database.types.ts, app/(main)/explore/_lib/probe.ts, page.tsx / actions.ts if yearly vs daily probe routing changes, and the Postgres contract if it mirrors the function.
• URL or default params change: page.tsx, catalog-options.ts, actions.ts, this file, and optionally app-routes.md.
• Cache tags or getCachedFireData change: read-path-get-fire-data-and-catalog-caching.md and app/api/admin/ingest/route.ts.
• Daily BRASK flow change: pipeline §10.3 (canonical).