The pipeline follows the medallion pattern: Bronze (raw) → Silver (clean) → Gold (published mart). Execution is asynchronous and idempotent, with retry/backoff and DLQ — see Medallion pipeline and Scheduling, retry, and DLQ.
Catalog#
| Symptom | Likely cause | How to check | Fix | Prevention |
|---|---|---|---|---|
Pipeline "stuck" in running | hung job / slow dependency | check the Audit log (Pipeline category) and the run time | wait for reconciliation; reprocess the source | monitor long runs; split large loads |
Run marked as failed | source unavailable or malformed data | expand the run detail (recorded error) | fix the source and rerun | validate "Test connection" before scheduling |
| Run reprocessed without duplicating | pipeline idempotency | expected (safe retry) | nothing to do | rely on retry/backoff instead of retriggering manually |
| Source "empty" on a given day | origin had no data in the window | check the origin for the period | rerun when the origin has data | freshness alert on the Gold layer |
| Incremental doesn't bring new rows | watermark has already moved past the data | check the watermark column | run a backfill of the needed window | choose a monotonic watermark column |
| Expected CDC (updates/deletes) | true CDC is Roadmap | incremental is watermark-based | use watermark + backfill; deletes aren't captured | see Incremental ingestion |
| New column at the origin didn't show up | schema drift not propagated | compare the source schema with Bronze | rerun discovery/load for the source | standardize the schema at the origin |
| Column type changed and broke Silver/Gold | incompatible type change | check the transformation error | adjust the transformation/expected type | avoid changing types at the origin without notice |
STRUCT/RECORD or array doesn't come out in the feed | nested columns aren't published | check the Gold table schema | publish flat columns in the marts | model Gold with flat columns |
| Job fails due to cost | hit the volume ceiling per query | check the run's cost/error | reduce the scope/partition the source | see Cost and performance |
Quick check#
- Open the Audit log, filter by Pipeline category and the period.
- Find the run and expand the detail (error/note).
- Cross-check with the source (did the origin have data in the window?) and with the watermark.
Expected result after the fix#
- The run goes back to
success, Bronze receives the rows, and Silver/Gold materialize without errors. The feed/dashboard starts reflecting the new data.
Related#
- Medallion pipeline
- Incremental ingestion and backfill
- Scheduling, retry, and DLQ
- Troubleshooting: connection and credentials
Last reviewed: 2026-09-08. Incremental is watermark-based; CDC (change data capture) is planned.