This page is conceptual: it gives you the mental map. Operational detail lives in the reference pages (
conectores.md,pipeline-medalhao.md) and the step-by-step guides intutoriais/.
The idea in one sentence#
Ingestia takes data that already exists in your business (database, spreadsheet, ERP, files), keeps a raw, auditable copy, cleans and organizes that copy, and in the end delivers ready-made tables (marts) for dashboards and natural-language questions — all isolated per customer (workspace).
The path of a piece of data (the 5 stations)#
Source → Landing → Bronze → Silver → Gold → Dashboard / Q&A
(customer) (raw copy) (verbatim) (clean) (aggregated)- Source — where the data comes from. It can be a database (PostgreSQL, MySQL, SQL Server, Oracle, BigQuery), a spreadsheet (Google Sheets), an ERP/CRM (Omie, Bling, Conta Azul, Tiny, NF-e, Salesforce, HubSpot), ads (Meta Ads, Google Ads) or files (CSV, JSON, Parquet).
- Landing — the landing zone. Raw files arrive in the file area (landing) before any transformation. It is the "snapshot bank" of the data as it arrived.
- Bronze — the data loaded verbatim (raw, as text), with no conversion, with
audit stamps (
_ingest_at,_source,_row). Nothing is discarded here. - Silver — the clean, typed data: quality rules, deduplication and
derivations (e.g.
receita = preço × quantidade). Whatever fails the rules is counted and left out (the difference never disappears silently). - Gold — the aggregated datamarts, ready for consumption (revenue by month, by state, by category, KPIs). This is what dashboards and Q&A read.
This three-layer design is the medallion (Bronze/Silver/Gold), a data engineering pattern. It separates "what arrived" from "what is clean" from "what is ready" — so a cleaning error never destroys the original data.
Two ways to bring in data: push and pull#
Ingestia works with two ingestion models:
| Model | How it works | Source examples |
|---|---|---|
| Pull ("We fetch") | Ingestia connects to your source and pulls the data at the scheduled time. | Database, BigQuery, ERP/CRM, ads, Google Sheets, object storage |
| Push ("You drop it off") | You send the files to the landing and Ingestia loads them into Bronze. | CSV / JSON / Parquet upload |
File sources (CSV/JSON/Parquet) accept both modes: you upload the file (push) or point to an object storage and Ingestia reads from there (pull). Databases and SaaS are always pull.
How data is updated after the 1st load#
After the first load (which brings the history), each update can be:
- Full — recalculates everything. Always correct, more expensive. It is the safe fallback.
- Incremental by watermark — brings only what is new since the last run,
using an increasing column (date/
id) as a "watermark". Much cheaper. - Backfill — bring a larger history on demand (e.g. the last 24 months).
And how often the pipeline runs is the cadence:
- Batch — once a day, at a set time (the default).
- Micro-batch — every 5 or 15 minutes, bringing only what changed (Business and Scale plans). It is the near real time option. See Micro-batch.
Honesty: Ingestia's incremental load is watermark-based, not CDC. Continuous change capture (INSERT/UPDATE/DELETE read from the source's log, in real time) is not available yet. Details and limitations in
conceitos-modos-de-carga.md.
Where each customer lives (per-workspace isolation)#
Each workspace has its own space, derived from a sanitized prefix:
- Datasets:
<prefix>_bronze,<prefix>_silver,<prefix>_gold. - Landing:
landing/{prefix}/...in the file area.
A customer never reads another's dataset or bucket. This is an invariant that holds across all Data pages. Connection secrets (password, token, key) are encrypted and redacted in any log or error message.
Quick glossary#
| Term | Meaning |
|---|---|
| Source / Data source | A configured connection to a source system. |
| Connector | The adapter that knows how to talk to a type of source (e.g. PostgreSQL, Meta Ads). |
| Landing zone | Landing area for raw files in the file area (landing) before Bronze. |
| Medallion | The Bronze → Silver → Gold pattern. |
| Mart / datamart | An aggregated table ready for consumption, in the Gold layer. |
| Watermark | The highest value of an increasing column already ingested; the basis of incremental loads. |
| Backfill | Loading a history larger than the incremental window. |
| Pipeline | The sequence ingest → bronze → silver → gold → publish. |
| Workspace | The isolated customer; owner of its datasets and buckets. |
| DLQ | Dead Letter Queue: queue of runs that failed permanently. |
Where to start#
- Connect a source — start with a database or a spreadsheet: connect PostgreSQL · connect BigQuery · import a file.
- Run a pipeline — take the data all the way to Gold: full pipeline.
- Keep it cheap and fresh — update only the delta: incremental pipeline by watermark.
Related#
- Connector catalog by state
- Bronze/Silver/Gold medallion pipeline
- Load modes: full, incremental by watermark and backfill
- Quality, rejected rows and quarantine
- Cost, limits and per-workspace security
Last reviewed: 2026-09-08.