Skip to content

Automatic translation — the Portuguese version is the reference. View in Portuguese

INGESTIA DATA — Public Data Documentation

On this page (5)

What this folder is. Public documentation, written by the REAL state of the product (not by promise), of ingestia.io's Data layer: the connectors and the entire datalake pipeline — ingestion, medallion Bronze/Silver/Gold, profiling, schema drift, quality, deduplication, transformations, scheduling, retry, DLQ, cost, limits and security per workspace. Sample data is synthetic.


Pipeline operation lives in Orchestration

Versioned publishing, schedule and timezone, execution (task/attempt/log) and failure recovery have their own guides in Orquestração. The operation pages that already lived here remain at the same addresses — nothing was moved.

Looking for the exact name of a button?

This folder explains the how it works. The exact name of each button, screen by screen, is in the Screens Guide: Conexões · Pipelines · Execuções · Console de dados · Seu datalake · Consultas · Linhagem · Governança · Conectar BI.

Pages#

PageCovers
conectores.mdCatalog of connectors by state (Available · Limited access · Planned) and by category, common SDK, contract suite, 30-day certification
pipeline-medalhao.mdMedallion Bronze→Silver→Gold, quality, deduplication, profiling, schema drift, sanitization, transformations
recuperar-dados-time-travel.mdView and recover a table as it was in the last 7 days (time travel) from the Data console
ingestao-incremental-backfill.mdFull · incremental by period · incremental by key (watermark) · backfill · resume checkpoint
agendamento-retry-dlq.mdScheduling (cron), retry/backoff, DLQ, layer reconciliation, observability per connector
custo-limites-seguranca.mdCost engine · billable bytes cap · limits/quotas · scope by workspace · redaction · PII
micro-lote-cadencia.mdBatch × micro-batch cadence (every 5 or 15 minutes), what each run reads, delay warning
notebooks-e-clusters.mdSpark clusters and notebooks: concept, step by step and cost
etapa-notebook-silver-gold.mdPublished notebook as a Silver or Gold pipeline step
formato-iceberg.mdApache Iceberg write format for Silver and Gold tables

Feature status#

The features described in this folder are available in the product, unless the page says otherwise.

In the connectors catalog each connector has one of three public states: Available (registration and use in the normal flow), Limited access (the connector page states the condition) and Planned (listed in the catalog with page and logo, but configuration is rejected with the reason). A connector only counts as "live" when it is Available.


Mandatory template (9 fields)#

Every flow or connector documented in the pages of this folder follows this template. The 9 fields describe each item:

  1. State — available, or the concrete limitation when there is one.
  2. What it does — honest description of the real behavior.
  3. How it works — the mechanics, in product language.
  4. Plan · Permission — entitlement by plan + access check + capability kill switch (``), when present.
  5. Cost — billed bytes / billable bytes cap. Values on screen are estimates; the amount charged is the one on the invoice.
  6. Limits — concrete product caps (pages, rows, bytes, columns…).
  7. Security — scope by workspace, secret redaction, production guard, PII handling.
  8. How to validate — how to confirm the behavior in your own workspace, via the console/dashboard.
  9. In short — what the feature delivers today.

How to confirm behavior (in the product)#

Each page brings, in the How to validate field, the step-by-step for you to confirm the behavior in your own workspace, via the console/dashboard — no technical environment required. No step depends on external network beyond your own datalake.

Cost values shown are estimates; the amount charged is the one on the invoice. Connectors labeled Available in the catalog are registered and used in the normal flow, without case-by-case approval. Only connectors with configuration rejected, listed there, require contacting support.


Security and scope (invariant across all pages)#

  • Scope by workspace, always. The datalake is <prefix>_bronze / _silver / _gold and landing landing/{prefix}/... in the file area, with prefix derived from the workspace (sanitized to [a-zA-Z0-9_-]). A customer never reads another's dataset/bucket.
  • Secret never leaks. Every connector error/log goes through secret redaction — raw token, in URL (percent-encoded) and in JSON.
  • Production does not fake. A path that would fall back to a mock triggers the production guard: in production it fails with 503 instead of simulating success.
  • PII protected. Profiling masks PII (•••) and suppresses min/max/distincts; a hidden column does not appear.

Related links