Skip to content

Automatic translation — the Portuguese version is the reference. View in Portuguese

Updated on 2026-10-05

How data flows in Ingestia (overview)

The end-to-end path of a piece of data in Ingestia — from the customer's source to the Gold marts and the dashboards.

On this page (8)

This page is conceptual: it gives you the mental map. Operational detail lives in the reference pages (conectores.md, pipeline-medalhao.md) and the step-by-step guides in tutoriais/.


The idea in one sentence#

Ingestia takes data that already exists in your business (database, spreadsheet, ERP, files), keeps a raw, auditable copy, cleans and organizes that copy, and in the end delivers ready-made tables (marts) for dashboards and natural-language questions — all isolated per customer (workspace).

The path of a piece of data (the 5 stations)#

code
Source → Landing → Bronze → Silver → Gold → Dashboard / Q&A
(customer) (raw copy) (verbatim) (clean) (aggregated)
  1. Source — where the data comes from. It can be a database (PostgreSQL, MySQL, SQL Server, Oracle, BigQuery), a spreadsheet (Google Sheets), an ERP/CRM (Omie, Bling, Conta Azul, Tiny, NF-e, Salesforce, HubSpot), ads (Meta Ads, Google Ads) or files (CSV, JSON, Parquet).
  2. Landing — the landing zone. Raw files arrive in the file area (landing) before any transformation. It is the "snapshot bank" of the data as it arrived.
  3. Bronze — the data loaded verbatim (raw, as text), with no conversion, with audit stamps (_ingest_at, _source, _row). Nothing is discarded here.
  4. Silver — the clean, typed data: quality rules, deduplication and derivations (e.g. receita = preço × quantidade). Whatever fails the rules is counted and left out (the difference never disappears silently).
  5. Gold — the aggregated datamarts, ready for consumption (revenue by month, by state, by category, KPIs). This is what dashboards and Q&A read.

This three-layer design is the medallion (Bronze/Silver/Gold), a data engineering pattern. It separates "what arrived" from "what is clean" from "what is ready" — so a cleaning error never destroys the original data.

Two ways to bring in data: push and pull#

Ingestia works with two ingestion models:

ModelHow it worksSource examples
Pull ("We fetch")Ingestia connects to your source and pulls the data at the scheduled time.Database, BigQuery, ERP/CRM, ads, Google Sheets, object storage
Push ("You drop it off")You send the files to the landing and Ingestia loads them into Bronze.CSV / JSON / Parquet upload

File sources (CSV/JSON/Parquet) accept both modes: you upload the file (push) or point to an object storage and Ingestia reads from there (pull). Databases and SaaS are always pull.

How data is updated after the 1st load#

After the first load (which brings the history), each update can be:

  • Full — recalculates everything. Always correct, more expensive. It is the safe fallback.
  • Incremental by watermark — brings only what is new since the last run, using an increasing column (date/id) as a "watermark". Much cheaper.
  • Backfill — bring a larger history on demand (e.g. the last 24 months).

And how often the pipeline runs is the cadence:

  • Batch — once a day, at a set time (the default).
  • Micro-batch — every 5 or 15 minutes, bringing only what changed (Business and Scale plans). It is the near real time option. See Micro-batch.

Honesty: Ingestia's incremental load is watermark-based, not CDC. Continuous change capture (INSERT/UPDATE/DELETE read from the source's log, in real time) is not available yet. Details and limitations in conceitos-modos-de-carga.md.

Where each customer lives (per-workspace isolation)#

Each workspace has its own space, derived from a sanitized prefix:

  • Datasets: <prefix>_bronze, <prefix>_silver, <prefix>_gold.
  • Landing: landing/{prefix}/... in the file area.

A customer never reads another's dataset or bucket. This is an invariant that holds across all Data pages. Connection secrets (password, token, key) are encrypted and redacted in any log or error message.

Quick glossary#

TermMeaning
Source / Data sourceA configured connection to a source system.
ConnectorThe adapter that knows how to talk to a type of source (e.g. PostgreSQL, Meta Ads).
Landing zoneLanding area for raw files in the file area (landing) before Bronze.
MedallionThe Bronze → Silver → Gold pattern.
Mart / datamartAn aggregated table ready for consumption, in the Gold layer.
WatermarkThe highest value of an increasing column already ingested; the basis of incremental loads.
BackfillLoading a history larger than the incremental window.
PipelineThe sequence ingest → bronze → silver → gold → publish.
WorkspaceThe isolated customer; owner of its datasets and buckets.
DLQDead Letter Queue: queue of runs that failed permanently.

Where to start#

  1. Connect a source — start with a database or a spreadsheet: connect PostgreSQL · connect BigQuery · import a file.
  2. Run a pipeline — take the data all the way to Gold: full pipeline.
  3. Keep it cheap and fresh — update only the delta: incremental pipeline by watermark.

Related#


Last reviewed: 2026-09-08.

Related links