What this folder is. Public documentation, written by the REAL state of the product (not by promise), of ingestia.io's Data layer: the connectors and the entire datalake pipeline — ingestion, medallion Bronze/Silver/Gold, profiling, schema drift, quality, deduplication, transformations, scheduling, retry, DLQ, cost, limits and security per workspace. Sample data is synthetic.
Pipeline operation lives in Orchestration
Versioned publishing, schedule and timezone, execution (task/attempt/log) and failure recovery have their own guides in Orquestração. The operation pages that already lived here remain at the same addresses — nothing was moved.
Looking for the exact name of a button?
This folder explains the how it works. The exact name of each button, screen by screen, is in the Screens Guide: Conexões · Pipelines · Execuções · Console de dados · Seu datalake · Consultas · Linhagem · Governança · Conectar BI.
Pages#
| Page | Covers |
|---|---|
conectores.md | Catalog of connectors by state (Available · Limited access · Planned) and by category, common SDK, contract suite, 30-day certification |
pipeline-medalhao.md | Medallion Bronze→Silver→Gold, quality, deduplication, profiling, schema drift, sanitization, transformations |
recuperar-dados-time-travel.md | View and recover a table as it was in the last 7 days (time travel) from the Data console |
ingestao-incremental-backfill.md | Full · incremental by period · incremental by key (watermark) · backfill · resume checkpoint |
agendamento-retry-dlq.md | Scheduling (cron), retry/backoff, DLQ, layer reconciliation, observability per connector |
custo-limites-seguranca.md | Cost engine · billable bytes cap · limits/quotas · scope by workspace · redaction · PII |
micro-lote-cadencia.md | Batch × micro-batch cadence (every 5 or 15 minutes), what each run reads, delay warning |
notebooks-e-clusters.md | Spark clusters and notebooks: concept, step by step and cost |
etapa-notebook-silver-gold.md | Published notebook as a Silver or Gold pipeline step |
formato-iceberg.md | Apache Iceberg write format for Silver and Gold tables |
Feature status#
The features described in this folder are available in the product, unless the page says otherwise.
In the connectors catalog each connector has one of three public states: Available (registration and use in the normal flow), Limited access (the connector page states the condition) and Planned (listed in the catalog with page and logo, but configuration is rejected with the reason). A connector only counts as "live" when it is Available.
Mandatory template (9 fields)#
Every flow or connector documented in the pages of this folder follows this template. The 9 fields describe each item:
- State — available, or the concrete limitation when there is one.
- What it does — honest description of the real behavior.
- How it works — the mechanics, in product language.
- Plan · Permission — entitlement by plan + access check + capability kill switch (``), when present.
- Cost — billed bytes / billable bytes cap. Values on screen are estimates; the amount charged is the one on the invoice.
- Limits — concrete product caps (pages, rows, bytes, columns…).
- Security — scope by workspace, secret redaction, production guard, PII handling.
- How to validate — how to confirm the behavior in your own workspace, via the console/dashboard.
- In short — what the feature delivers today.
How to confirm behavior (in the product)#
Each page brings, in the How to validate field, the step-by-step for you to confirm the behavior in your own workspace, via the console/dashboard — no technical environment required. No step depends on external network beyond your own datalake.
Cost values shown are estimates; the amount charged is the one on the invoice. Connectors labeled Available in the catalog are registered and used in the normal flow, without case-by-case approval. Only connectors with configuration rejected, listed there, require contacting support.
Security and scope (invariant across all pages)#
- Scope by workspace, always. The datalake is
<prefix>_bronze/_silver/_goldand landinglanding/{prefix}/...in the file area, withprefixderived from the workspace (sanitized to[a-zA-Z0-9_-]). A customer never reads another's dataset/bucket. - Secret never leaks. Every connector error/log goes through secret redaction — raw token, in URL (percent-encoded) and in JSON.
- Production does not fake. A path that would fall back to a mock triggers the production guard: in production it fails with 503 instead of simulating success.
- PII protected. Profiling masks PII (
•••) and suppresses min/max/distincts; a hidden column does not appear.