Skip to content

Automatic translation — the Portuguese version is the reference. View in Portuguese

Updated on 2026-09-08

Troubleshooting: pipeline and schema

Symptom → cause → how to check → fix → prevention catalog for ingestion, the medallion pipeline (bronze/silver/gold), retry/DLQ, watermark, and schema changes.

On this page (4)

The pipeline follows the medallion pattern: Bronze (raw) → Silver (clean) → Gold (published mart). Execution is asynchronous and idempotent, with retry/backoff and DLQ — see Medallion pipeline and Scheduling, retry, and DLQ.

SymptomLikely causeHow to checkFixPrevention
Pipeline "stuck" in runninghung job / slow dependencycheck the Audit log (Pipeline category) and the run timewait for reconciliation; reprocess the sourcemonitor long runs; split large loads
Run marked as failedsource unavailable or malformed dataexpand the run detail (recorded error)fix the source and rerunvalidate "Test connection" before scheduling
Run reprocessed without duplicatingpipeline idempotencyexpected (safe retry)nothing to dorely on retry/backoff instead of retriggering manually
Source "empty" on a given dayorigin had no data in the windowcheck the origin for the periodrerun when the origin has datafreshness alert on the Gold layer
Incremental doesn't bring new rowswatermark has already moved past the datacheck the watermark columnrun a backfill of the needed windowchoose a monotonic watermark column
Expected CDC (updates/deletes)true CDC is Roadmapincremental is watermark-baseduse watermark + backfill; deletes aren't capturedsee Incremental ingestion
New column at the origin didn't show upschema drift not propagatedcompare the source schema with Bronzererun discovery/load for the sourcestandardize the schema at the origin
Column type changed and broke Silver/Goldincompatible type changecheck the transformation erroradjust the transformation/expected typeavoid changing types at the origin without notice
STRUCT/RECORD or array doesn't come out in the feednested columns aren't publishedcheck the Gold table schemapublish flat columns in the martsmodel Gold with flat columns
Job fails due to costhit the volume ceiling per querycheck the run's cost/errorreduce the scope/partition the sourcesee Cost and performance

Quick check#

  1. Open the Audit log, filter by Pipeline category and the period.
  2. Find the run and expand the detail (error/note).
  3. Cross-check with the source (did the origin have data in the window?) and with the watermark.

Expected result after the fix#

  • The run goes back to success, Bronze receives the rows, and Silver/Gold materialize without errors. The feed/dashboard starts reflecting the new data.

Related#


Last reviewed: 2026-09-08. Incremental is watermark-based; CDC (change data capture) is planned.

Related links